Empty (zero byte) SSH host keys

[Benjamin Drung]

Hi,

We create virtual machine image templates by doing automated minimal
installations of different Linux distributions (via
preseed/kickstarter/autoyast). At the end of the installation, we
remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host
keys will be generated on the first boot of the image instances. This
is done by adding a "dpkg-reconfigure openssh-server" call in
/etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the init
script of sshd on the other distributions.

This leads to working SSH server running on the virtual machines most
of the times, but sometimes the SSH connection fails with "connection
reset by peer". The investigation of Debian 7 "wheezy" images showed
that these faulty machines have empty (zero byte) SSH host key files.
These files do not exist before the machines are started, but they do
exist before "dpkg-reconfigure openssh-server" is called.

So it seems that some process creates these empty SSH host key files.
Can you help to further debugging this strange behavior? Does sshd
create SSH host keys?
--
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: ***@profitbricks.com
URL: http://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Andreas Gauger, Achim Weiss.

[Damien Miller]

Post by Benjamin Drung
Hi,
We create virtual machine image templates by doing automated minimal
installations of different Linux distributions (via
preseed/kickstarter/autoyast). At the end of the installation, we
remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host
keys will be generated on the first boot of the image instances. This
is done by adding a "dpkg-reconfigure openssh-server" call in
/etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the init
script of sshd on the other distributions.
This leads to working SSH server running on the virtual machines most
of the times, but sometimes the SSH connection fails with "connection
reset by peer". The investigation of Debian 7 "wheezy" images showed
that these faulty machines have empty (zero byte) SSH host key files.
These files do not exist before the machines are started, but they do
exist before "dpkg-reconfigure openssh-server" is called.
So it seems that some process creates these empty SSH host key files.
Can you help to further debugging this strange behavior? Does sshd
create SSH host keys?

No, sshd only reads and never writes host keys. It's possible that
either ssh-keygen is failing during writing the keys out or there
is some bug in the init script that is calling it.

-d

[Nico Kadel-Garcia]

On Wed, Jan 6, 2016 at 1:08 PM, Benjamin Drung

Post by Benjamin Drung
Hi,
We create virtual machine image templates by doing automated minimal
installations of different Linux distributions (via
preseed/kickstarter/autoyast). At the end of the installation, we
remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host
keys will be generated on the first boot of the image instances. This
is done by adding a "dpkg-reconfigure openssh-server" call in
/etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the init
script of sshd on the other distributions.
This leads to working SSH server running on the virtual machines most
of the times, but sometimes the SSH connection fails with "connection
reset by peer". The investigation of Debian 7 "wheezy" images showed
that these faulty machines have empty (zero byte) SSH host key files.
These files do not exist before the machines are started, but they do
exist before "dpkg-reconfigure openssh-server" is called.
So it seems that some process creates these empty SSH host key files.
Can you help to further debugging this strange behavior? Does sshd
create SSH host keys?

It could be a filesystem issue in your image generation process. Are
you doing a "sync" before recording your image? And are you using one
of the more fragile, "optimized" filesystems like ReiserFS, which is
infamous for zeroing files?

Until you discover the issue, you *could* edit the OpenSSH init
script, or /etc/sysconfig/sshd, to clear the hostkeys if they're empty
and before new keys would be auto-generated.

[Benjamin Drung]

Post by Damien Miller

Post by Benjamin Drung
Hi,
We create virtual machine image templates by doing automated
minimal
installations of different Linux distributions (via
preseed/kickstarter/autoyast). At the end of the installation, we
remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host
keys will be generated on the first boot of the image instances.
This
is done by adding a "dpkg-reconfigure openssh-server" call in
/etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the
init
script of sshd on the other distributions.
This leads to working SSH server running on the virtual machines
most
of the times, but sometimes the SSH connection fails with
"connection
reset by peer". The investigation of Debian 7 "wheezy" images
showed
that these faulty machines have empty (zero byte) SSH host key
files.
These files do not exist before the machines are started, but they
do
exist before "dpkg-reconfigure openssh-server" is called.
So it seems that some process creates these empty SSH host key
files.
Can you help to further debugging this strange behavior? Does sshd
create SSH host keys?

No, sshd only reads and never writes host keys.

Thanks for confirming it.

Post by Damien Miller
It's possible that
either ssh-keygen is failing during writing the keys out or there
is some bug in the init script that is calling it.

The strange thing is that the empty files appear before ssh-keygen is
called (via our "dpkg-reconfigure openssh-server" call in
/etc/rc.local). I am not aware of any other ssh-keygen calls besides
our one. Who creates these empty SSH host key files?

I also checked the sysvinit script of ssh on Debian 7 "wheezy". It only
checks/creates /var/run/sshd before calling sshd. There is not host key
handling in the init script of ssh.
--
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: ***@profitbricks.com
URL: http://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Andreas Gauger, Achim Weiss.

[Benjamin Drung]

Post by Nico Kadel-Garcia
On Wed, Jan 6, 2016 at 1:08 PM, Benjamin Drung

Post by Benjamin Drung
Hi,
We create virtual machine image templates by doing automated
minimal
installations of different Linux distributions (via
preseed/kickstarter/autoyast). At the end of the installation, we
remove the SSH host keys (rm -f /etc/ssh/ssh*_key*). Fresh SSH host
keys will be generated on the first boot of the image instances.
This
is done by adding a "dpkg-reconfigure openssh-server" call in
/etc/rc.local (which calls ssh-keygen) on Debian/Ubuntu and by the
init
script of sshd on the other distributions.
This leads to working SSH server running on the virtual machines
most
of the times, but sometimes the SSH connection fails with
"connection
reset by peer". The investigation of Debian 7 "wheezy" images
showed
that these faulty machines have empty (zero byte) SSH host key
files.
These files do not exist before the machines are started, but they
do
exist before "dpkg-reconfigure openssh-server" is called.
So it seems that some process creates these empty SSH host key
files.
Can you help to further debugging this strange behavior? Does sshd
create SSH host keys?

It could be a filesystem issue in your image generation process. Are
you doing a "sync" before recording your image? And are you using one
of the more fragile, "optimized" filesystems like ReiserFS, which is
infamous for zeroing files?

All images use ext4 as filesystem.

The image creation works as following:
1) Create a raw HDD image (2 GiB in size)
2) Launch qemu to do a preseed/kickstarter/autoyast installation
2b) as part of the installation remove the SSH host keys (via a
late_command or post command)
2c) wait for the installation to finish / exit of qemu
3) mount the raw image
4) copy the installer logs and remove them
5) unmout the raw image
6) run zerofree on the partition
7) convert the raw image to qcow2 and upload/distribute it

When an instance is created from this template, following steps are
done:

1) create a block device (could be larger than the image)
2) copy the template to the block device
3) resize the root partition and add a tmpfs partition
4) mount the resized partition
5) (optionally) inject a authorized_key file
6) set the root password
7) verify that no SSH host keys are on this partition
8) unmount the partition
9) launch qemu

Post by Nico Kadel-Garcia
Until you discover the issue, you *could* edit the OpenSSH init
script, or /etc/sysconfig/sshd, to clear the hostkeys if they're
empty
and before new keys would be auto-generated.

That's exactly the workaround that we are currently testing.
--
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: ***@profitbricks.com
URL: http://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Andreas Gauger, Achim Weiss.