On automatic MAC selection in OpenSSH_6.7p1 + OpenSSL 1.0.1k

[Dimitris Diochnos]

Dear All,

I am writing this email because of some observed strange behavior of ssh. I
think this is some sort of complement of the issue that was discussed in
[1] ([2, 3, 4, 5, 6] were related to the issue).

Long story short, I have a client and a server, both running the latest
raspbian jessie with all the relevant updates in two different countries
(i.e. both OpenSSH_6.7p1 Raspbian-5+deb8u2, OpenSSL 1.0.1k 8 Jan 2015).
When I attempt to ssh from one to the other one (direction: country B -->
country A) using the command
$ ssh -p PPPP ***@host
then the connection times out after about 2 minutes and I get the error
message `Connection closed by <host>`. (So, it is not the message
`connection reset by peer` that was present in [1].)

Inspecting the configuration for the established link using `-v` I can see
that the cipher selected is `aes128-ctr`, the message authentication code
(mac) is `umac-64-***@openssh.com` and the compression is `none`. If I now
go ahead and use the command

$ ssh -m umac-64-***@openssh.com -p PPPP ***@host

quite surprisingly, the key exchange issue that I have above has
disappeared and I can connect to the remote machine from country B to
country A. Further, specifying only the cipher on command line does not
resolve the issue; i.e. using

$ ssh -c aes128-ctr -p PPPPP ***@host

the key exchange times out again. Finally, the combined

$ ssh -c aes128-ctr -m umac-64-***@openssh.com -p PPPPP ***@host

also works.


To make things even weirder, when I attempt to ssh between the two machines
in the opposite direction (country A --> country B), then the simple

$ ssh -p PPPP ***@host

command works as expected using exactly the same configuration. That is, I
do not have to specify the message authentication code (recall that both
machines are raspberry pi's running the latest raspbian jessie,
OpenSSH_6.7p1 and OpenSSL 1.0.1k).



On another note, lowering the MTU size (which was another workaround for
[1]) also allows me to pass successfully the key exchange phase in the
direction where I normally have an issue (that is, country B --> country
A). The maximum MTU size that would allow me to pass the key exchange
negotiation was 1458 (that is, with a size of 1459 the key exchange got
stuck).



My first attempt to resolve the situation was in Raspberry Pi Stack
Exchange in thread [7]. Since the issue was not resolved there, I posted
the issue in Super User Stack Exchange in thread [8]. It was this thread
that made me aware of the issue in [1] where by lowering the MTU size was a
viable solution. However, this may not be the only issue here, because, as
explained above, automatic configuration of MAC leads to failure (but then
again, only in one direction of establishing the link).

If you need more details on the output or the configurations that I am
using I am happy to provide you with all the relevant information.
Meanwhile, if you would like to read an extended version with debug
information (`ssh -vv ...`), I would suggest [8] which was written more
recently and I think has a better description of the problem.



Best regards,
Dimitris



References

[1] ssh 'connection reset by peer' problem since 5.8p1, url:
http://lists.mindrot.org/pipermail/openssh-unix-dev/2011-March/029432.html
[2] The mysterious case of broken SSH client (“connection reset by peer”),
url:
http://www.held.org.il/blog/2011/05/the-myterious-case-of-broken-ssh-client-connection-reset-by-peer/
[3] Can't login anymore: Read from socket failed: Connection reset by peer,
url: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/708493
[4] Connecting to older ssh version has cipher negotiation problem, url:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612607
[5] FS#22897 - [openssh] SSH is falling to connect to some server Read from
socket failed: Connection reset by peer, url:
https://bugs.archlinux.org/task/22897
[6] ssh client problem: Connection reset by peer [closed], url:
http://serverfault.com/questions/265244/ssh-client-problem-connection-reset-by-peer

[7] Can not ssh on remote host, url:
http://raspberrypi.stackexchange.com/questions/48089/can-not-ssh-on-remote-host
[8] Strange ssh connection issue, url:
http://superuser.com/questions/1082529/strange-ssh-connection-issue/

[Daniel Kahn Gillmor]

Post by Dimitris Diochnos
On another note, lowering the MTU size (which was another workaround for
[1]) also allows me to pass successfully the key exchange phase in the
direction where I normally have an issue (that is, country B --> country
A). The maximum MTU size that would allow me to pass the key exchange
negotiation was 1458 (that is, with a size of 1459 the key exchange got
stuck).

This is the relevant hint for your connection. It sounds like some
element along the network path from B to A is silently dropping packets
that are larger than 1458, and your network stack has not detected this
situation.

When you force the MAC algorithm to be the specific one, you are
probably making the ssh handshake negotiation packets each be small
enough to fit into the smaller MTU.

As such, i think this is a networking configuration issue, and not
something for ssh to try to fix. Maybe the fix belongs in your TCP
stack, or in your network configuration?

Sounds frustrating!

--dkg

[Dimitris Diochnos]

I can understand that some element along the path from B to A is
silently dropping packets larger than 1458. However, the other point
that I am making is that when MTU is set to the default 1500 (as
reported by ifconfig) then, the command

ssh -p PPPP ***@host

fails (selecting automatically for mac umac-64-***@openssh.com), while
the command

ssh -m umac-64-***@openssh.com -p PPPP ***@host

succeeds.

In this sense, both commands are executed with MTU=1500 but ssh does
behave differently in these two situations without me having to change
anything in my network configuration. Thus a reasonable (?) guess is
that perhaps ssh does not set all the necessary flags and options
correctly when umac-64-***@openssh.com is set automatically during the
negotiation, but does set all the necessary flags and options
correctly when the same mac is specified as a command line parameter.

Perhaps my guess is incorrect (after all I have no real knowledge on
this matter), but I thought it would be best if I report this. My
apologies if this is spamming the mailing list.

Best regards,
Dimitris


On Tue, May 31, 2016 at 12:20 PM, Daniel Kahn Gillmor

Post by Daniel Kahn Gillmor

Post by Dimitris Diochnos
On another note, lowering the MTU size (which was another workaround for
[1]) also allows me to pass successfully the key exchange phase in the
direction where I normally have an issue (that is, country B --> country
A). The maximum MTU size that would allow me to pass the key exchange
negotiation was 1458 (that is, with a size of 1459 the key exchange got
stuck).

This is the relevant hint for your connection. It sounds like some
element along the network path from B to A is silently dropping packets
that are larger than 1458, and your network stack has not detected this
situation.
When you force the MAC algorithm to be the specific one, you are
probably making the ssh handshake negotiation packets each be small
enough to fit into the smaller MTU.
As such, i think this is a networking configuration issue, and not
something for ssh to try to fix. Maybe the fix belongs in your TCP
stack, or in your network configuration?
Sounds frustrating!
--dkg

[Alex Bligh]

Post by Dimitris Diochnos
I can understand that some element along the path from B to A is
silently dropping packets larger than 1458. However, the other point
that I am making is that when MTU is set to the default 1500 (as
reported by ifconfig) then, the command
the command
succeeds.

MTU / MSS problems are notorious for being tickled only by specific
data sizes. Your first example happens to use different size packets
from the second, causing the TCP window to be within the problematic
range. A bit smaller or a bit larger packets will work. Debug
the network problem and all will be fine.

http://netalyzr.icsi.berkeley.edu/

specifically this test

http://netalyzr.icsi.berkeley.edu/info_mtu.html

is quite useful though only tests between the browser and the
site it runs on. Other tools are also available.

[Darren Tucker]

Post by Dimitris Diochnos
[...]
In this sense, both commands are executed with MTU=1500 but ssh does
behave differently in these two situations without me having to change
anything in my network configuration.

The thing ssh does differently in these two instances is that when you
don't specify -m, it uses the default MACs list which can be reasonably
long. If you look at the output of ssh -vv you'll see something like this:

debug2: kex_parse_kexinit: umac-64-***@openssh.com,umac-128-***@openssh.com,
hmac-sha2-256-***@openssh.com,hmac-sha2-512-***@openssh.com,
hmac-sha1-***@openssh.com,umac-***@openssh.com,umac-***@openssh.com
,hmac-sha2-256,hmac-sha2-512,hmac-sha1

which in my case is 214 bytes, compared to 24 bytes when you specify
umac64. You'll likely see similar behaviour if you specify the Ciphers
or KexAlgorithms.

Post by Dimitris Diochnos
Thus a reasonable (?) guess is
that perhaps ssh does not set all the necessary flags and options
negotiation

Nope, it doesn't do anything different with regard to network options and
such. You did an good job figuring out that your network is broken, but
that's what you need to fix to resolve your problem.