OpenSSH public key material should be rotated periodically, just as X.509 certificates are. This should be done yearly via a calendar invite, as this creates a predictable event around which the rotation tasks can be scheduled. Reasons to rotate keys include:
- Security - keys used for years or decades may offer attackers opportunities to break the key, or an attacker may archive the SSH traffic in the hopes that some future attack will grant access to the private key material, thereby allowing the previous encrypted network traffic to be decrypted. Rotation limits the lifetime of keys, reducing how long a key is available to be attacked, and how much data is recoverable should the key be broken.
- Reliability - while rotating keys may break connectivity, doing so during a scheduled event offers a clean rollback plan, and the ability to update the documentation with the previously unknown connection. Without scheduled rotation, these unknown connections would only be revealed during a disaster. Cascading failures are possible: both the disaster that wipes out the public keys, and an unknown number of undocumented SSH connections must then be recovered from, increasing the time, cost, and complexity of the recovery. With rotation, the number of unknown agents is minimized, allowing documentation or automation to recreate the necessary public key relationships.
The Config::OpenSSH::Authkey Perl module offers a programatic interface to the OpenSSH authorized_keys file. This can help automate key user rotations.
User Keys
User keys can be updated via the ssh-keygen(1) command on the client, and the public key material copied via various means to any relevant servers. OpenSSH Public Key Authentication goes into more detail on setting up pulic key authentication. Public keys can also be maintained under version control or database, and installed automatically on the required hosts via custom software. This avoids the tedium and mistakes possible when a human manually copies the key material to many hosts.
While testing the new key material, note that shared connections may mask connectivity problems. Turn off connection sharing during testing, or write the rotation software such that a rollback is performed automatically should the user not be able to intervene.
$ ssh -S /dev/null server.example.org
Control socket connect(/dev/null): Socket operation on non-socket
ControlSocket /dev/null already exists, disabling multiplexing
…
server%
Host Keys
The host keys should also be rotated. This requires a restart of the sshd process. Changing the host keys will also require updates to various known_hosts files on various systems, which may break automated jobs that assume a particular host key, or are set to ask should a host key mismatch occur.
#!/bin/sh
#
# Example SSH host key rotation.
# Different vendors hide the keys in /etc/ssh or /etc
DIR=/etc/ssh
# TODO This prompts to overwrite, and offers no
# backup for rollback. This also supports SSHv1,
# which should be disabled where possible.
while read type file; do
ssh-keygen -q -N '' -t $type -f "$DIR/$file"
done <<EOF
dsa ssh_host_dsa_key
rsa1 ssh_host_key
rsa ssh_host_rsa_key
EOF
Host keys may also be updated through host lease returns, or by retiring virtual servers. This ties the lifetime of the host SSH key to that of the system, so may not be viable for servers kept beyond the lifetime of the lease (ideally no more than a few years), or for long running virutal systems.
host_keys Scalability Problems
Larger organizations with many thousands of hosts will likely not support the known_hosts file, or will replace lookups from this flat file with some other code path, for example to a local cached database that is populated from a central source. If the known_hosts is not supported, then individual hosts will have random entries for host keys, depending on what systems they connect to, and how often. Programs at sites like this may need to disable the host key checks: this increases the risk of a man-in-the-middle attack, though decreases the risk that the program will fail due to an unknown host key being presented, should the host key change through normal key rotation, or when connecting to a new server for the service.
ssh -o CheckHostIP=no,StrictHostKeyChecking=no …
As an alternative, programs that create SSH connections would use strict host key checking, though would behave properly should the connectivity be disrupted, and escalate the issue to a human to resolve. This will increase the complexity and time to write software that connects via SSH.
