Troubleshooting AutoSSH

Troubleshooting AutoSSH

[L. V. Lammert]

Trying to debug a session, .. neither AUTOSSH_DEBUG nor AUTOSSH_LOGLEVEL:

$ env | grep AUTO
AUTOSSH_DEBUG=1
AUTOSSH_LOGLEVEL=7

nor -vv:

cygrunsrv -I AutoSSH -f "remote_link" -p /usr/bin/autossh -a " -vv \

change the logging info always ("Host key verification filed"); what is
the correct way to increase the log level?

	TIA!

	Lee

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Troubleshooting AutoSSH

[Andrew Schulman]

> Trying to debug a session, .. neither AUTOSSH_DEBUG nor AUTOSSH_LOGLEVEL:
> 
> $ env | grep AUTO
> AUTOSSH_DEBUG=1
> AUTOSSH_LOGLEVEL=7
> 
> nor -vv:
> 
> cygrunsrv -I AutoSSH -f "remote_link" -p /usr/bin/autossh -a " -vv \
> 
> change the logging info always ("Host key verification filed"); what is
> the correct way to increase the log level?

Your command line looks wrong.  Did it get cut off? It's missing some
required information in the -a argument to cygrunsrv.  That argument will
have to include all of the arguments you need to pass to autossh:  at least
the name of the host that you're connecting to, plus any other autossh or
ssh options.  For example, you might also need to include -i keyfile so ssh
will know what key file to load.

For installing autossh as a service, please read
/usr/share/doc/autossh/README.Cygwin.  It recommends using something like:

cygrunsrv \
  --install autossh \
  --path /usr/bin/autossh \
  --dep tcpip \
  --env "AUTOSSH_NTSERVICE=yes" \
  --args "-F /etc/autossh/ssh_config <target_host>" \
  <... other autossh args or environment settings ...>

In particular, setting AUTOSSH_NTSERVICE=yes redirects error messages to
stdout.  From there cygrunsrv will put them into the service log file, e.g.
/var/log/autossh.log, where you can inspect them.  Otherwise I think they
go into the Windows service logs, where you might still find them but
they'll be all split up and hard to find (the interface for that is
horrible).

In general, if autossh isn't working as you expect, there are three steps
to follow, in order:

(1) Get the ssh command working.
(2) Get the autossh command working.
(3) Get the service working.

Each of these steps depends on the previous ones, so if they fail, the
later ones will too.

Good luck,
Andrew


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Troubleshooting AutoSSH

[L. V. Lammert]

On Fri, 12 Jul 2013, Andrew Schulman wrote:

> > Trying to debug a session, .. neither AUTOSSH_DEBUG nor AUTOSSH_LOGLEVEL:
> >
> > $ env | grep AUTO
> > AUTOSSH_DEBUG=1
> > AUTOSSH_LOGLEVEL=7
> >
> > nor -vv:
> >
> > cygrunsrv -I AutoSSH -f "remote_link" -p /usr/bin/autossh -a " -vv \
> >
> > change the logging info always ("Host key verification filed"); what is
> > the correct way to increase the log level?
>
> Your command line looks wrong.  Did it get cut off?
>
Yes, I only included the first half where I inserted the "-vv", which does
not work. The entire command works on other systems, so that is not the
problem.

The problem is, nothing seems to raise the debug level for autossh when
starting as a service, .. [see below].

> (1) Get the ssh command working.
>
Works fine, .. keys setup.

> (2) Get the autossh command working.
>
Bingo - looks like -v *DOES* work when starting as a user! ON startup:

$ autossh -v -M 5661:6661 -N -R 4661:127.0.0.1:2206
wtadmin@nagios.winningtech.com
2013/07/12 11:10:09 autossh[5128]: checking for grace period, tries = 0
2013/07/12 11:10:09 autossh[5128]: starting ssh (count 1)
2013/07/12 11:10:09 autossh[5128]: ssh child pid is 5060
2013/07/12 11:10:09 autossh[5128]: check on child 5060
2013/07/12 11:10:09 autossh[5128]: set alarm for 600 secs
2013/07/12 11:10:09 autossh[5060]: execing /usr/bin/ssh
OpenSSH_6.2p2, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to <remote server> [<IP>] port 2206.
debug1: Connection established.
debug1: identity file /home/<admin user>/.ssh/id_rsa type 1
debug1: identity file /home/<admin user>/.ssh/id_rsa-cert type -1
debug1: identity file /home/<admin user>/.ssh/id_dsa type -1
debug1: identity file /home/<admin user>/.ssh/id_dsa-cert type -1
debug1: identity file /home/<admin user>/.ssh/id_ecdsa type -1
debug1: identity file /home/<admin user>/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.8
debug1: match: OpenSSH_5.8 pat OpenSSH_5*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA
70:5c:35:ee:86:19:23:15:32:1a:e7:d6:99:95:9a:e4
debug1: Host '[<remote server>]:2206' is known and matches the ECDSA host key.
debug1: Found key in /home/<admin user>n/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/<admin user>/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to <remote server> ([<IP>]:2206).
debug1: Local connections to LOCALHOST:5661 forwarded to remote address
127.0.0.1:6661
debug1: Local forwarding listening on ::1 port 5661.
				  ^^^^^^^^^

<connection attempt fails with "Connection closed by ::1">!!!!

It looks like the problem is that AutoSSH is binding to IPV6, *NOT*
IPv4! sshd is set to "AddressFamily inet", .. so the problem appears to be
that AutoSSH is not binding properly.

Don't see anything in the man pages or a quick search, .. how would one
force AutoSSH to bind to IPv4? IPV6 is installed on this box, but not
used.

	Thanks!!!!

	Lee

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Troubleshooting AutoSSH

[Andrew Schulman]

> debug1: Local connections to LOCALHOST:5661 forwarded to remote address
> 127.0.0.1:6661
> debug1: Local forwarding listening on ::1 port 5661.
> 				  ^^^^^^^^^
> 
> <connection attempt fails with "Connection closed by ::1">!!!!
> 
> It looks like the problem is that AutoSSH is binding to IPV6, *NOT*
> IPv4! sshd is set to "AddressFamily inet", .. so the problem appears to be
> that AutoSSH is not binding properly.
> 
> Don't see anything in the man pages or a quick search, .. how would one
> force AutoSSH to bind to IPv4? IPV6 is installed on this box, but not
> used.

Hi Lee.  Okay, that does seem to narrow it down.

You're right that autossh doesn't have any ipv4 options.  It hasn't been
updated in a few years, and I think it's just not ipv6-aware yet.

Does it help if you include the -4 switch to ssh, to force it to use ipv4
only?  It's worth a try.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Troubleshooting AutoSSH

[L. V. Lammert]

On Fri, 12 Jul 2013, Andrew Schulman wrote:

> Hi Lee.  Okay, that does seem to narrow it down.
>
> You're right that autossh doesn't have any ipv4 options.  It hasn't been
> updated in a few years, and I think it's just not ipv6-aware yet.
>
Looks like it may not be autossh - if I start sshd with a default config,
it works [ssh localhost], .. if I try ssh -4 localhost, nada!

It looks like *sshd* can only bind IPV6 - forcing it to bind IPV4 only
prevents startup.

Windows Firewall is not enabled, .. Trend Micro is installed, but I can
find no evidence it is more than virus scanner. The service user
(cyg_server) is a local Administrator, .. tried Administrator credentials,
but the service still cannot start if I try to force IPV4.

*User* level IPV4 connections (e.g. Firefox) are working, as I just did a
Cygwin reinstall to verify the situation.

Any thought on what might be blocking sshd from binding to the IPV4 stack?
FYI, this is the first machine we have used as a Domain Member, previously
we have only been running standalone; cyg_server is a local user, as is
our admin user.

	TIA!!

	Lee


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Troubleshooting AutoSSH

[Andrew Schulman]

> On Fri, 12 Jul 2013, Andrew Schulman wrote:
> 
> > Hi Lee.  Okay, that does seem to narrow it down.
> >
> > You're right that autossh doesn't have any ipv4 options.  It hasn't been
> > updated in a few years, and I think it's just not ipv6-aware yet.
> >
> Looks like it may not be autossh - if I start sshd with a default config,
> it works [ssh localhost], .. if I try ssh -4 localhost, nada!
> 
> It looks like *sshd* can only bind IPV6 - forcing it to bind IPV4 only
> prevents startup.

OK.  So does it work then to pass the -6 flag to ssh?

If sshd is only accepting ipv6, then it may be autossh's port monitoring
feature that's broken.  It uses ssh to forward some ports of its own, and
that feature might be ipv4-only.  You can turn it off by passing -M0 to
autossh.  Does that fix the problem?

If that does fix the problem, then it's not a very good solution, because
you've had to disable autossh's monitoring of your connection.  So it won't
know if your connection has stopped passing traffic, in order to restart
it.  But at least we'll have a well-defined bug to report upstream.

Andrew


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Troubleshooting AutoSSH

[L. V. Lammert]

On Fri, 12 Jul 2013, Andrew Schulman wrote:

> > It looks like *sshd* can only bind IPV6 - forcing it to bind IPV4 only
> > prevents startup.
>
> OK.  So does it work then to pass the -6 flag to ssh?
>
I did not try forcing -6 because that was the only way it would connect
(i.e. sshd would only start with an IPV6 connection, forcing IPV4 was a
non-starter).

> If sshd is only accepting ipv6, then it may be autossh's port monitoring
> feature that's broken.
>
The failure is in starting sshd, so it's something in the OpenSSH update
(15 May), or something in this Windows machine.

> It uses ssh to forward some ports of its own, and
> that feature might be ipv4-only.  You can turn it off by passing -M0 to
> autossh.  Does that fix the problem?
>
I could try, but there is no way I can test a remote connection with V6,
local connections on V6 do work when using default configuration.

Would there be any value in testing the previous package, openssh-6.2p1-2?

	Lee

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple