Timeouts using SSH wrapper when tests read /dev/stdout

Timeouts using SSH wrapper when tests read /dev/stdout

[Stafford Horne]

Hello,

For the OpenRISC port I pretty much have all tests passing but there is one last
problem that I have yet to solve.  For the following tests they timeout when using
the SSH wrapper but the succeed when logged into the host.

This is from my notes, these are the affected tests:
    elf/tst-audit14,works with stdout redirect to file: not over ssh
    elf/tst-audit15,works with stdout redirect to file: not over ssh
    elf/tst-audit16,works with stdout redirect to file: not over ssh
    libio/tst-wfile-sync,works with stdin redirect: not over ssh
    iconv/tst-iconv_prog,has redirects works on platform: not over ssh

For example (elf/tst-audit14), when running the wrapper it essentially runs:

  ssh 10.0.0.5 elf/tst-audit14 > elf/tst-audit14.out

In test elf/tst-audit14 it does a read on /dev/stdout and blocks.  It seems
since the redirect of stdout is going over the socket it does not end up in
/dev/stdout.

I have tried forcing a pseudo-terminal with 'ssh -tt' but that seems to have no
affect.

Is this expected?

Any tips would be helpful, has anyone else run into this before?  It may be just
a bug in my architecture.

-Stafford

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Stafford Horne]

I haven't thought about this much, but now I am.  I guess the issue is that
of we are redirecting stdout to a file, we can read it later.

If we are redirecting to a pipe like in ssh we cannot read again.

If thats correct should we update the tests to be able to work over ssh? Or
maybe we should change how the wrapper is invoked for redirects.

On Thu, Sep 30, 2021, 6:32 AM Stafford Horne <shorne@gmail.com> wrote:

> Hello,
>
> For the OpenRISC port I pretty much have all tests passing but there is
> one last
> problem that I have yet to solve.  For the following tests they timeout
> when using
> the SSH wrapper but the succeed when logged into the host.
>
> This is from my notes, these are the affected tests:
>     elf/tst-audit14,works with stdout redirect to file: not over ssh
>     elf/tst-audit15,works with stdout redirect to file: not over ssh
>     elf/tst-audit16,works with stdout redirect to file: not over ssh
>     libio/tst-wfile-sync,works with stdin redirect: not over ssh
>     iconv/tst-iconv_prog,has redirects works on platform: not over ssh
>
> For example (elf/tst-audit14), when running the wrapper it essentially
> runs:
>
>   ssh 10.0.0.5 elf/tst-audit14 > elf/tst-audit14.out
>
> In test elf/tst-audit14 it does a read on /dev/stdout and blocks.  It seems
> since the redirect of stdout is going over the socket it does not end up in
> /dev/stdout.
>
> I have tried forcing a pseudo-terminal with 'ssh -tt' but that seems to
> have no
> affect.
>
> Is this expected?
>
> Any tips would be helpful, has anyone else run into this before?  It may
> be just
> a bug in my architecture.
>
> -Stafford
>

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Adhemerval Zanella]

On 29/09/2021 23:46, Stafford Horne via Libc-alpha wrote:
> I haven't thought about this much, but now I am.  I guess the issue is that
> of we are redirecting stdout to a file, we can read it later.
> 
> If we are redirecting to a pipe like in ssh we cannot read again.
> 
> If thats correct should we update the tests to be able to work over ssh? Or
> maybe we should change how the wrapper is invoked for redirects.
> 


I would say that if we can fix it on the wrapper it would be better, since
it will less constraint on how we create tests.

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Stafford Horne]

On Thu, Sep 30, 2021 at 08:47:02AM -0300, Adhemerval Zanella wrote:
> 
> 
> On 29/09/2021 23:46, Stafford Horne via Libc-alpha wrote:
> > I haven't thought about this much, but now I am.  I guess the issue is that
> > of we are redirecting stdout to a file, we can read it later.
> > 
> > If we are redirecting to a pipe like in ssh we cannot read again.
> > 
> > If thats correct should we update the tests to be able to work over ssh? Or
> > maybe we should change how the wrapper is invoked for redirects.
> > 
> 
> 
> I would say that if we can fix it on the wrapper it would be better, since
> it will less constraint on how we create tests.

Alright, I am having a look to see what I can do.

-Stafford

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Szabolcs Nagy]

The 09/30/2021 21:12, Stafford Horne via Libc-alpha wrote:
> On Thu, Sep 30, 2021 at 08:47:02AM -0300, Adhemerval Zanella wrote:
> > 
> > 
> > On 29/09/2021 23:46, Stafford Horne via Libc-alpha wrote:
> > > I haven't thought about this much, but now I am.  I guess the issue is that
> > > of we are redirecting stdout to a file, we can read it later.
> > > 
> > > If we are redirecting to a pipe like in ssh we cannot read again.
> > > 
> > > If thats correct should we update the tests to be able to work over ssh? Or
> > > maybe we should change how the wrapper is invoked for redirects.
> > > 
> > 
> > 
> > I would say that if we can fix it on the wrapper it would be better, since
> > it will less constraint on how we create tests.
> 
> Alright, I am having a look to see what I can do.

it may be an ssh server side issue, i have seen similar
with dropbear sshd:

https://lists.ucc.gu.uwa.edu.au/pipermail/dropbear/2020q2/002230.html

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Andreas Schwab]

On Sep 30 2021, Adhemerval Zanella via Libc-alpha wrote:

> On 29/09/2021 23:46, Stafford Horne via Libc-alpha wrote:
>> I haven't thought about this much, but now I am.  I guess the issue is that
>> of we are redirecting stdout to a file, we can read it later.
>> 
>> If we are redirecting to a pipe like in ssh we cannot read again.
>> 
>> If thats correct should we update the tests to be able to work over ssh? Or
>> maybe we should change how the wrapper is invoked for redirects.
>> 
>
>
> I would say that if we can fix it on the wrapper it would be better, since
> it will less constraint on how we create tests.

The redirection needs to be performed on the target host, but the
wrapper doesn't know about the rediction, since it is already performed
by the parent shell.  The wrapper needs to see the original command line
and evaluate it on the target host.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Stafford Horne]

On Thu, Sep 30, 2021 at 04:17:56PM +0200, Andreas Schwab wrote:
> On Sep 30 2021, Adhemerval Zanella via Libc-alpha wrote:
> 
> > On 29/09/2021 23:46, Stafford Horne via Libc-alpha wrote:
> >> I haven't thought about this much, but now I am.  I guess the issue is that
> >> of we are redirecting stdout to a file, we can read it later.
> >> 
> >> If we are redirecting to a pipe like in ssh we cannot read again.
> >> 
> >> If thats correct should we update the tests to be able to work over ssh? Or
> >> maybe we should change how the wrapper is invoked for redirects.
> >> 
> >
> >
> > I would say that if we can fix it on the wrapper it would be better, since
> > it will less constraint on how we create tests.
> 
> The redirection needs to be performed on the target host, but the
> wrapper doesn't know about the rediction, since it is already performed
> by the parent shell.  The wrapper needs to see the original command line
> and evaluate it on the target host.

Exactly,

I have done some basic tests and it works fine over ssh if the redirection is
passed to the target.  i.e.:

    ssh 10.0.0.5 /bin/sh -c "/home/shorne/work/gnu-toolchain/build-glibc/elf/ld-linux-or1k.so.1
        --library-path xxx
        /home/shorne/work/gnu-toolchain/build-glibc/elf/tst-audit14 >
        /home/shorne/work/gnu-toolchain/build-glibc/elf/tst-audit14.out"

But that is not what is passed to the wrapper.

I think I have traced this down to the make file bit where this is happening:

Rules:

    define make-test-out
    $(if $($*-ENV-only),$(test-wrapper-env-only) $($*-ENV-only),\
	 $(test-wrapper-env) $(run-program-env) $($*-ENV)) \
    $(host-test-program-cmd) $($*-ARGS)
    endef
    $(objpfx)%.out: %.input $(objpfx)%
	    $(make-test-out) > $@ < $(word 1,$^); \
	    $(evaluate-test)
    $(objpfx)%.out: /dev/null $(objpfx)%    # Make it 2nd arg for canned sequence.
	    $(make-test-out) > $@; \
	    $(evaluate-test)

For .out and .input the redirections are made on the test harness, not passed in
as args.  So there is no chance for the wrapper to interpret the redirections as
it stands now.

I am looking at options to allow the wrapper to see the redirections, either by
using *-ARGS for these kind of tests or by changing the "*.out" rule.

-Stafford

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Joseph Myers]

On Fri, 1 Oct 2021, Stafford Horne via Libc-alpha wrote:

> I am looking at options to allow the wrapper to see the redirections, either by
> using *-ARGS for these kind of tests or by changing the "*.out" rule.

I suggest changing the tests to avoid depending on reading stdout (the 
output could be checked by a separate test, for example - there are a few 
tests like that with dependencies on .out files).  The requirements for 
wrappers are deliberately simple (for example, they don't try to preserve 
environment variables - those must be explicitly passed to 
$(test-wrapper-env)).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Stafford Horne]

On Thu, Sep 30, 2021 at 10:31:15PM +0000, Joseph Myers wrote:
> On Fri, 1 Oct 2021, Stafford Horne via Libc-alpha wrote:
> 
> > I am looking at options to allow the wrapper to see the redirections, either by
> > using *-ARGS for these kind of tests or by changing the "*.out" rule.
> 
> I suggest changing the tests to avoid depending on reading stdout (the 
> output could be checked by a separate test, for example - there are a few 
> tests like that with dependencies on .out files).  The requirements for 
> wrappers are deliberately simple (for example, they don't try to preserve 
> environment variables - those must be explicitly passed to 
> $(test-wrapper-env)).

That makes sense to move this out to use the evaluate-test phase.

I did get something working by hacking the wrapper, but this test doesnt work, I
haven't looked at it much but it depends on stdin.

 libio/tst-wfile-sync


Quick fix to make stdout a file.  The proper fix would be to make the stdout
creation run on the host.  Also I am not sure if it's ok to depend on 'mktemp',
'cat' and 'rm' on the host.  But I guess we do expect coreutils so maybe that's ok.


diff --git a/scripts/cross-test-ssh.sh b/scripts/cross-test-ssh.sh
index a839e22dd0..cd5920b866 100755
--- a/scripts/cross-test-ssh.sh
+++ b/scripts/cross-test-ssh.sh
@@ -150,9 +150,11 @@ export GLIBC_TEST_ALLOW_TIME_SETTING=1
 ${command}"
 fi

+stdout=`mktemp --dry-run --suffix=.stdout`
+
 # HOST's sshd simply concatenates its arguments with spaces and
 # passes them to some shell.  We want to force the use of /bin/sh,
 # so we need to re-quote the whole command to ensure it appears as
 # the sole argument of the '-c' option.
 full_command="$(bourne_quote "${command}")"
-$ssh "$host" /bin/sh -c "$full_command"
+$ssh "$host" /bin/sh -c "$full_command > $stdout && cat $stdout && rm $stdout"

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Joseph Myers]

On Sat, 2 Oct 2021, Stafford Horne via Libc-alpha wrote:

> +stdout=`mktemp --dry-run --suffix=.stdout`

mktemp --dry-run is unsafe (i.e., actually using the output of that 
command in a redirection is unsafe) and so should not be used at all.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Timeouts using SSH wrapper when tests read /dev/stdout

[Stafford Horne]

On Mon, Oct 04, 2021 at 05:03:01PM +0000, Joseph Myers wrote:
> On Sat, 2 Oct 2021, Stafford Horne via Libc-alpha wrote:
> 
> > +stdout=`mktemp --dry-run --suffix=.stdout`
> 
> mktemp --dry-run is unsafe (i.e., actually using the output of that 
> command in a redirection is unsafe) and so should not be used at all.

Yup, I am aware of that, that's why I mentioned it was a hack, I could have been
more clear about why.  If we wanted it to work we would want to have the temp
file created by mktemp on the target host.

Though, I am now tryig to change the tests to not depend on stdout being a file
as you suggested.

-Stafford