sshd "getpeername failed: Transport endpoint is not connected" error

[Andrey Klimentev]

Hello, everybody.

I've recently encountered a problem with OpenSSH server. Could you
help me to troubleshoot it?

I've configured 2 IP interfaces[1]: one with a public IP adress and
one with a private address. When I connect[2] through the public
interface (ens34), SSH works fine, but when I connect[3] through the
private interface (ens32), I receive a rather cryptic message on my
client - "ssh_exchange_identification: read: Connection reset by
peer". And this on my server - getpeername failed: Transport endpoint
is not connected".

I am using OpenSSH_7.1p1, OpenSSL 1.0.2e 3 Dec 2015. It is the latest
version available in Arch Linux repo.

--
[1] http://pastebin.com/8BF0N84t - interface conf and routing table
[2] http://pastebin.com/QptpeF7f - remote log and strace
[3] http://pastebin.com/niFJac46 - local log and strace

[Darren Tucker]

Post by Andrey Klimentev
Hello, everybody.
I've recently encountered a problem with OpenSSH server. Could you
help me to troubleshoot it?
I've configured 2 IP interfaces[1]: one with a public IP adress and
one with a private address. When I connect[2] through the public
interface (ens34), SSH works fine, but when I connect[3] through the
private interface (ens32), I receive a rather cryptic message on my
client - "ssh_exchange_identification: read: Connection reset by
peer".

"reset by peer" usually means the TCP connection has been torn down
via a TCP RST packet. This can be the other end crashing (although
from your logs that doesn't look to be the case here) or something
like a firewall sending a RST.

Do you have any firewall or packet filter in that network path?
iptables anti-spoofing rules?

Post by Andrey Klimentev
And this on my server - getpeername failed: Transport endpoint
is not connected".

From the failed traces:

debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
debug1: getpeername failed: Transport endpoint is not connected
debug1: get_remote_port failed

Since this happens immediately after the server reexecs itself,
another possiblity is that somehow the descriptors are being marked
close-on-exec. You can test this theory by adding "-r" to sshd's
command line to disable reexec.

[Andrey Klimentev]

Thanks for suggestion. That was, indeed, a problem on our network.
Sorry for bothering you.

BTW, "-r" option is not included in a man page, it doesn't seem to
have any effect as well.

Post by Darren Tucker

Post by Andrey Klimentev
Hello, everybody.
I've recently encountered a problem with OpenSSH server. Could you
help me to troubleshoot it?
I've configured 2 IP interfaces[1]: one with a public IP adress and
one with a private address. When I connect[2] through the public
interface (ens34), SSH works fine, but when I connect[3] through the
private interface (ens32), I receive a rather cryptic message on my
client - "ssh_exchange_identification: read: Connection reset by
peer".

"reset by peer" usually means the TCP connection has been torn down
via a TCP RST packet. This can be the other end crashing (although
from your logs that doesn't look to be the case here) or something
like a firewall sending a RST.
Do you have any firewall or packet filter in that network path?
iptables anti-spoofing rules?

Post by Andrey Klimentev
And this on my server - getpeername failed: Transport endpoint
is not connected".

debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
debug1: getpeername failed: Transport endpoint is not connected
debug1: get_remote_port failed
Since this happens immediately after the server reexecs itself,
another possiblity is that somehow the descriptors are being marked
close-on-exec. You can test this theory by adding "-r" to sshd's
command line to disable reexec.
--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

[Darren Tucker]

On Fri, Dec 11, 2015 at 6:23 PM, Andrey Klimentev

Post by Andrey Klimentev
Thanks for suggestion. That was, indeed, a problem on our network.
Sorry for bothering you.
BTW, "-r" option is not included in a man page,

We consider -r (and -R which is also used by the reexec code) to be
implementation details that end users should not rely on.

Post by Andrey Klimentev
it doesn't seem to have any effect as well.

It does, but the effect is not user visible. Look for
"send_rexec_state: done" in the debug output with and without -r.

[Nico Kadel-Garcia]

Post by Darren Tucker
On Fri, Dec 11, 2015 at 6:23 PM, Andrey Klimentev

Post by Andrey Klimentev
Thanks for suggestion. That was, indeed, a problem on our network.
Sorry for bothering you.
BTW, "-r" option is not included in a man page,

We consider -r (and -R which is also used by the reexec code) to be
implementation details that end users should not rely on.

So you refuse to document it at all, and insist that admins must read
the source code? I've actually seen people do this, but it seems
unfriendly.