For small sites, a changed Secure Shell (SSH) host key poses only minor problems, as only a few systems might be affected. At larger sites, the host key data should be maintained with configuration management, so that any SSH host key changes can be propogated throughout the site. CFEngine can maintain a global ssh_known_hosts file for OpenSSH.
Alternatives inculde storing the host keys in a directory service that clients lookup when needed. With a directory service, servers could upload new host keys after being installed or rotating the host keys, and other systems would immediately see the new keys. On the other hand, should the directory service fail, so will new SSH connections. The following method requires an additional step, plus time for CFEngine to syncronize the new ssh_known_hosts file to all systems.
module:authkey can help automate authorized_keys file management.
Host Key Collection with ssh-keyscan
The global ssh_known_hosts file should be maintained under version control. I keep the file at masterfiles/ssh/ssh_known_hosts under the repository, and use a set of scripts around ssh-keyscan(1) to add or update host keys manually. When collecting host keys from other servers, care should be taken that no attacker has supplied a malicious key. The man-in-the-middle threat can be reduced by using an authoritative name server on the system running ssh-keyscan(1). Also, have each system compare its own keys with the keys the scanning system collected, and alert if differences appear.
Scripts to assist key collection include:
- add-ssh-hostkeys - add new keys to a ssh_known_hosts file.
- expand-hl - list variations on a hostname, so known_hosts record include the short hostname, fully qualified hostname, and any other variations.
- ssh-hkutil - wrapper called by add-ssh-hostkeys and update-ssh-hostkeys.
- update-ssh-hostkeys - update existing records in a known_hosts file if changed on the remote system.
With a Makefile, make add or make update can be used to update the host key list:
# usage:
# echo hostname | make add
add:
@expand-hl | add-ssh-hostkeys ssh_known_hosts
update:
@update-ssh-hostkeys ssh_known_hosts
Alternative Collection Methods
If all systems at a site run cfservd, then the host keys could be copied between systems automatically. Other options include uploading the keys from client systems to a collection system via pre-established SSH trust or a HTTPS server.
Sites with thousands of systems should either use a directory service, or break the ssh_known_hosts down into different groups, as some systems may not need host keys for every other system at the site.
Distributing ssh_known_hosts
Use a copy statement to distribute the ssh_known_hosts file, with variables to account for vendor specific differences.
control:
macosx::
ssh_known_hosts = ( /etc/ssh_known_hosts )
freebsd|linux|openbsd::
ssh_known_hosts = ( /etc/ssh/ssh_known_hosts )
copy:
any::
${masterfiles}/etc/ssh/ssh_known_hosts
dest=${ssh_known_hosts}
mode=0444
server=${policyhost}
type=checksum backup=false
Users with old host keys in their ~/.ssh/known_hosts files will see key warnings. Old keys must be deleted in favor of the globally maintained keys. For systems not under configuration management, consider making the ssh_known_hosts information verifiable from a Transport Layer Security (TLS) encrypted website.
