Pre-Seeded Known Host to Access New Server

• Why Skipping this Step is a Good Idea
• Why Does Host Key Verification Exist
• Pre-Seed the Host Key

This post goes over how to create a new server and be able to ssh into it without needing to go through the ssh host verification manual step.

This step is annoying, error prone and tedious enough that I’m sure most of us answer “yes” to the question as fast as when reading some Terms of Services.

To skip to the solution, click here.

Why Skipping this Step is a Good Idea

As said in the preamble, this step is tedious and error prone. One needs to extract the newly generated public key from the new server, then generate the fingerprint of the key and compare it visually with the one printed by SSH.

Usually, extracting the public key requires accessing the server in some less secure way, by for example using the username/password protocol to SSH into the server. This is not ideal.

I would even argue this step is inherently insecure especially in a cloud environment because how can you be really sure this public key/fingerprint is from the server you just booted unless it’s physically in front of you?

Why Does Host Key Verification Exist

In NixOS, when OpenSSH’s systemd unit starts for the first time, it generates some SSH keys using ssh-keygen by default at /etc/ssh/ssh_host_rsa_key and /etc/ssh/ssh_host_ed25519_key.

This step is also done in most other distributions.

The location, number and flags used to generate the keys can be changed with the services.openssh.hostKeys option.

Now, let’s assume one can SSH into the server by having shared their own private key with the server. On the first attempt, SSH will prompt them with the following question:

The authenticity of host '[XXX.XXX.XXX.XXX]:22
([XXX.XXX.XXX.XXX]:22)' can't be established.
ED25519 key fingerprint is
SHA256:YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY.
Are you sure you want to continue connecting
(yes/no/[fingerprint])?

This fingerprint corresponds to the public key of one of the SSH keys generated by OpenSSH. One can see a key fingerprint with:

$ ssh-keygen -lvf <public key file>

If they say yes, the following will get printed:

Warning: Permanently added '[XXX.XXX.XXX.XXX]:22'
(ED25519) to the list of known hosts.

And indeed, a new entry is added to their ~/.ssh/known_hosts file:

$ cat ~/.ssh/known_hosts | grep XXX.XXX.XXX.XXX
[192.168.1.30]:22345 ssh-ed25519 ZZZZZ...

The whole reason is to protect them if, in the future, the host changes without them knowing about it. There are a myriad of reasons for this, some innocuous and some malevolent. Anyway, the next time they’ll try to connect, they will see this message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW.
Please contact your system administrator.
Add correct host key in /home/them/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/them/.ssh/known_hosts:1
Host key for [XXX.XXX.XXX.XXX]:22 has changed and you have requested strict checking.
Host key verification failed.

Essentially, it’s saying the fingerprint of the host’s public key changed and warns that it could be potentially a bad sign.

If they happen to know this is okay, because for example they indeed generated a new host key for some reason, getting rid of the fingerprint saved the first time and restarting the host key verification procedure is done by deleting the entry in the ~/.ssh/known_hosts file with:

ssh-keygen -R "[XXX.XXX.XXX.XXX]:22"
# Host [XXX.XXX.XXX.XXX]:22 found: line 1
/home/them/.ssh/known_hosts updated.
Original contents retained as
/home/them/.ssh/known_hosts.old

Pre-Seed the Host Key

We can generate the SSH key pair with:

$ ssh-keygen -N "" -f hostkey

Then, assuming we use SOPS with sops-nix to handle secrets, we add the content of the hostkey and hostkey.pub files to our SOPS secrets file:

hostkey:
  privateKey: XXXX...
  publicKey: YYYY...

Finally, the two NixOS snippets for the remaining work are for the server we just let SOPS populate the correct location by using the path option:

sops.secrets."hostkey/privateKey" = {
  path = "/etc/ssh/ssh_host_rsa_key";
};

And for our laptop we add it to our system known hosts:

sops.secrets."hostkey/publicKey" = {};

programs.ssh.knownHosts."myserver".publicKeyFile =
  sops.secrets."hostkey/publicKey".path;