DejaGnu/GCC testsuite behavior regarding multiple 'dg-do'

DejaGnu/GCC testsuite behavior regarding multiple 'dg-do'

[Thomas Schwinge]

Hi!

Comparing DejaGnu/GCC testsuite '*.sum' files between two systems ("old"
vs. "new") that ought to return identical results, I found that they
didn't:

    @@ -75032,6 +75023,7 @@ PASS: g++.dg/expr/bitfield4.C  -std=c++98 (test for excess errors)
     PASS: g++.dg/expr/bitfield5.C  -std=c++14  (test for warnings, line 12)
     PASS: g++.dg/expr/bitfield5.C  -std=c++14  (test for warnings, line 16)
     PASS: g++.dg/expr/bitfield5.C  -std=c++14 (test for excess errors)
    +PASS: g++.dg/expr/bitfield5.C  -std=c++14 execution test
     PASS: g++.dg/expr/bitfield5.C  -std=c++17  (test for errors, line 12)
     PASS: g++.dg/expr/bitfield5.C  -std=c++17  (test for errors, line 16)
     PASS: g++.dg/expr/bitfield5.C  -std=c++17 (test for excess errors)
    @@ -75041,6 +75033,7 @@ PASS: g++.dg/expr/bitfield5.C  -std=c++2a (test for excess errors)
     PASS: g++.dg/expr/bitfield5.C  -std=c++98  (test for warnings, line 12)
     PASS: g++.dg/expr/bitfield5.C  -std=c++98  (test for warnings, line 16)
     PASS: g++.dg/expr/bitfield5.C  -std=c++98 (test for excess errors)
    +PASS: g++.dg/expr/bitfield5.C  -std=c++98 execution test
     PASS: g++.dg/expr/bitfield6.C  -std=c++14  (test for warnings, line 10)
     PASS: g++.dg/expr/bitfield6.C  -std=c++14 (test for excess errors)
     PASS: g++.dg/expr/bitfield6.C  -std=c++17  (test for errors, line 10)

..., and several more like that.

'g++.dg/expr/bitfield5.C':

    // PR c++/30274
    // { dg-do run { target c++14_down } }
    // { dg-do compile { target c++17 } }
    [...]
      s.x++; // { dg-warning "5:use of an operand of type .bool. in .operator\\+\\+. is deprecated" "" { target { ! c++17 } } }
      // { dg-error "forbidden" "" { target c++17 } .-1 }
    [...]

So, this is meant to be an execution test for C++14 and older, and a
compile test for C++17 and newer.  This pattern seems to be recognized on
the "new" system, but not on the "old" system, where execution tests
aren't being run, only compile tests.

The "old" system, powerpc64le-unknown-linux-gnu Ubuntu 14.04, has DejaGnu
1.5-3ubuntu1.  Also reproduced on a x86_64-pc-linux-gnu Ubuntu 12.10
system with DejaGnu 1.5-3.

The "new" system, powerpc64le-unknown-linux-gnu Ubuntu 18.04, has DejaGnu
1.6.1-1.  Also reproduced on a x86_64-pc-linux-gnu Ubuntu 18.04 system
with DejaGnu 1.6.1-1.  And: also reproduced on the "old" system, when
instead of system-provided DejaGnu 1.5-3ubuntu1 manually running with
Ubuntu 18.04 DejaGnu 1.6.1-1.

Relevant (but haven't verified) seems to be the following DejaGnu commit:

    commit 569f8718b534a2cd9511a7d640352eb0126ff492
    Author: Dominik Vogt <vogt@linux.vnet.ibm.com>
    Date:   Mon Mar 28 17:31:07 2016 +1100

            * dg.exp (dg-do): Do not change the previously selected action if
            a de-selected dg-do is encountered.

Discussed in
<http://mid.mail-archive.com/20160308132910.GA30210@linux.vnet.ibm.com>,
and this indeed got included for DejaGnu 1.6.

Looking through the GCC testuite for test case files with more than one
'dg-do', the confusing thing is that this pattern has already been used
for a very long time.  See, for example, commit
5bdf05c8743e7486521ce3a3981ac3e6e7850ad0 (r143350, 2009-01-13):

    -/* { dg-do run { target powerpc*-*-* } } */
    +/* { dg-do run { target { powerpc*-*-* && vmx_hw } } } */
    +/* { dg-do compile { target { powerpc*-*-* && { ! vmx_hw } } } } */

I have not found any evidence in DejaGnu master branch that this not
working would've been a "recent" DejaGnu regression (and then fixed for
DejaGnu 1.6) -- so do we have to assume that this never worked as
intended back then?

I have not found an easy way to express different 'do-what-keyword'
within one test case file that work with old and new DejaGnu.

Per our "Prerequisites for GCC" installation documentation, we currently
require DejaGnu 1.4.4.  Advancing that to 1.6 is probably out of
question, given that it has "just" been released (four years ago).

As the failure mode with old DejaGnu is "benign" (only causes missing
execution testing), we could simply move on, and accept non-reproducible
results between different DejaGnu versions?  Kind of lame...  ;-|


Grüße
 Thomas
-----------------
Mentor Graphics (Deutschland) GmbH, Arnulfstraße 201, 80634 München / Germany
Registergericht München HRB 106955, Geschäftsführer: Thomas Heurung, Alexander Walter

dejagnu version update?

[Mike Stump]

I've changed the subject to match the 2015, 2017 and 2018 email threads.

On May 13, 2020, at 3:26 AM, Thomas Schwinge <thomas@codesourcery.com> wrote:
> 
> Comparing DejaGnu/GCC testsuite '*.sum' files between two systems ("old"
> vs. "new") that ought to return identical results, I found that they
> didn't:

> I have not found any evidence in DejaGnu master branch that this not
> working would've been a "recent" DejaGnu regression (and then fixed for
> DejaGnu 1.6) -- so do we have to assume that this never worked as
> intended back then?

Likely not.

> Per our "Prerequisites for GCC" installation documentation, we currently
> require DejaGnu 1.4.4.  Advancing that to 1.6 is probably out of
> question, given that it has "just" been released (four years ago).

:-)  A user that wants full coverage should use 1.6, apparently.

> As the failure mode with old DejaGnu is "benign" (only causes missing
> execution testing), we could simply move on, and accept non-reproducible
> results between different DejaGnu versions?  Kind of lame...  ;-|

An ugly wart to be sure.

So, now that ubuntu 20.04 is out and RHEL 8 is out, and they both contain 6, and SLES has 6 and since we've been sitting at 1.4.4 for so long, anyone want to not update dejagnu to require 1.6?

I had previously approved the update to 1.5.3, but no one really wanted it as no one updated the requirement.  Let's have the 1.6 discussion.  I'm not only inclined to up to 1.6, but to actually edit it in this time.

Anyone strongly against?  Why?

Re: dejagnu version update?

[Jonathan Wakely]

On Wed, 13 May 2020 at 18:19, Mike Stump via Gcc <gcc@gcc.gnu.org> wrote:
>
> I've changed the subject to match the 2015, 2017 and 2018 email threads.
>
> On May 13, 2020, at 3:26 AM, Thomas Schwinge <thomas@codesourcery.com> wrote:
> >
> > Comparing DejaGnu/GCC testsuite '*.sum' files between two systems ("old"
> > vs. "new") that ought to return identical results, I found that they
> > didn't:
>
> > I have not found any evidence in DejaGnu master branch that this not
> > working would've been a "recent" DejaGnu regression (and then fixed for
> > DejaGnu 1.6) -- so do we have to assume that this never worked as
> > intended back then?
>
> Likely not.
>
> > Per our "Prerequisites for GCC" installation documentation, we currently
> > require DejaGnu 1.4.4.  Advancing that to 1.6 is probably out of
> > question, given that it has "just" been released (four years ago).
>
> :-)  A user that wants full coverage should use 1.6, apparently.

As documented at
https://gcc.gnu.org/onlinedocs/libstdc++/manual/test.html#test.run.permutations
anything older than 1.5.3 causes problems for libstdc++ (and probably
the rest of GCC) because the options in --target_board get placed
after the options in dg-options. If the test depends on the options in
dg-options to work properly it might fail. For example, a test that
has { dg-options "-O2" } and fails without optimisation would FAIL if
you use --target_board=unix/-O0 with dejagnu 1.5.


> > As the failure mode with old DejaGnu is "benign" (only causes missing
> > execution testing), we could simply move on, and accept non-reproducible
> > results between different DejaGnu versions?  Kind of lame...  ;-|
>
> An ugly wart to be sure.
>
> So, now that ubuntu 20.04 is out and RHEL 8 is out, and they both contain 6, and SLES has 6 and since we've been sitting at 1.4.4 for so long, anyone want to not update dejagnu to require 1.6?

There are still lots of older systems in use for GCC dev, like all the
POWER servers in the compile farm (but I've put a recent dejagnu in
/opt/cfarm on some of them).

> I had previously approved the update to 1.5.3, but no one really wanted it as no one updated the requirement.  Let's have the 1.6 discussion.  I'm not only inclined to up to 1.6, but to actually edit it in this time.

Would the tests actually refuse to run with an older version?

> Anyone strongly against?  Why?

I'm in favour of requiring 1.5.3 or later, so 1.6 would be OK for me.

Re: dejagnu version update?

[Rob Savoye]

On 5/13/20 10:51 AM, Mike Stump wrote:

> So, now that ubuntu 20.04 is out and RHEL 8 is out, and they both
> contain 6, and SLES has 6 and since we've been sitting at 1.4.4 for
> so long, anyone want to not update dejagnu to require 1.6?

  We do still find and fix bugs occasionally. :-) And 1.4.4 was released
16 years ago... Linaro has been using the most recent release for years,
so I think it's a safe upgrade.

	- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Rainer Orth]

Jonathan Wakely via Gcc <gcc@gcc.gnu.org> writes:

>> I had previously approved the update to 1.5.3, but no one really wanted
>> it as no one updated the requirement.  Let's have the 1.6 discussion.
>> I'm not only inclined to up to 1.6, but to actually edit it in this time.
>
> Would the tests actually refuse to run with an older version?
>
>> Anyone strongly against?  Why?
>
> I'm in favour of requiring 1.5.3 or later, so 1.6 would be OK for me.

If we go beyond 1.5.x, we need to go all the way up to 1.6.2: 1.6 and
1.6.1 have an ugly bug that can miss timeouts, causing tests to hang
indefinitely until one manually kills them.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: dejagnu version update?

[Maciej W. Rozycki]

On Wed, 13 May 2020, Rainer Orth wrote:

> > I'm in favour of requiring 1.5.3 or later, so 1.6 would be OK for me.
> 
> If we go beyond 1.5.x, we need to go all the way up to 1.6.2: 1.6 and
> 1.6.1 have an ugly bug that can miss timeouts, causing tests to hang
> indefinitely until one manually kills them.

 Would you mind sharing a reference such as a DejaGNU Git commit ID of the 
fix for the bug you mean?

 Versions 1.6 and 1.6.1 seem ubiquitous and coincidentally earlier this 
very week I have been chasing a phenomenon with Expect's `wait' semantics 
causing a reliable hang in remote testing if `ssh' to the target board 
stops responding for whatever reason.  I have come up with a solution 
(that I'd be happy to upstream, except that DejaGNU maintenance seems to 
have been dead for like a year now), which I have also confirmed to be 
required with current DejaGNU Git master so it must be a different one, 
and I would like to know how it might be related to the bug you mention.

  Maciej

Re: dejagnu version update?

[Christophe Lyon]

On Wed, 13 May 2020 at 19:44, Jonathan Wakely via Gcc <gcc@gcc.gnu.org> wrote:
>
> On Wed, 13 May 2020 at 18:19, Mike Stump via Gcc <gcc@gcc.gnu.org> wrote:
> >
> > I've changed the subject to match the 2015, 2017 and 2018 email threads.
> >
> > On May 13, 2020, at 3:26 AM, Thomas Schwinge <thomas@codesourcery.com> wrote:
> > >
> > > Comparing DejaGnu/GCC testsuite '*.sum' files between two systems ("old"
> > > vs. "new") that ought to return identical results, I found that they
> > > didn't:
> >
> > > I have not found any evidence in DejaGnu master branch that this not
> > > working would've been a "recent" DejaGnu regression (and then fixed for
> > > DejaGnu 1.6) -- so do we have to assume that this never worked as
> > > intended back then?
> >
> > Likely not.
> >
> > > Per our "Prerequisites for GCC" installation documentation, we currently
> > > require DejaGnu 1.4.4.  Advancing that to 1.6 is probably out of
> > > question, given that it has "just" been released (four years ago).
> >
> > :-)  A user that wants full coverage should use 1.6, apparently.
>
> As documented at
> https://gcc.gnu.org/onlinedocs/libstdc++/manual/test.html#test.run.permutations
> anything older than 1.5.3 causes problems for libstdc++ (and probably
> the rest of GCC) because the options in --target_board get placed
> after the options in dg-options. If the test depends on the options in
> dg-options to work properly it might fail. For example, a test that
> has { dg-options "-O2" } and fails without optimisation would FAIL if
> you use --target_board=unix/-O0 with dejagnu 1.5.
>
I think that was commit:
http://git.savannah.gnu.org/gitweb/?p=dejagnu.git;a=commitdiff;h=5256bd82343000c76bc0e48139003f90b6184347
which for sure was a major change (though I don't see it documented in
dejagnu/NEWS file)

>
> > > As the failure mode with old DejaGnu is "benign" (only causes missing
> > > execution testing), we could simply move on, and accept non-reproducible
> > > results between different DejaGnu versions?  Kind of lame...  ;-|
> >
> > An ugly wart to be sure.
> >
> > So, now that ubuntu 20.04 is out and RHEL 8 is out, and they both contain 6, and SLES has 6 and since we've been sitting at 1.4.4 for so long, anyone want to not update dejagnu to require 1.6?
>
> There are still lots of older systems in use for GCC dev, like all the
> POWER servers in the compile farm (but I've put a recent dejagnu in
> /opt/cfarm on some of them).
>
> > I had previously approved the update to 1.5.3, but no one really wanted it as no one updated the requirement.  Let's have the 1.6 discussion.  I'm not only inclined to up to 1.6, but to actually edit it in this time.
>
> Would the tests actually refuse to run with an older version?
>
> > Anyone strongly against?  Why?
>
> I'm in favour of requiring 1.5.3 or later, so 1.6 would be OK for me.

Re: dejagnu version update?

[Jonathan Wakely]

On Thu, 14 May 2020 at 07:45, Christophe Lyon wrote:
>
> On Wed, 13 May 2020 at 19:44, Jonathan Wakely via Gcc <gcc@gcc.gnu.org> wrote:
> >
> > On Wed, 13 May 2020 at 18:19, Mike Stump via Gcc <gcc@gcc.gnu.org> wrote:
> > >
> > > I've changed the subject to match the 2015, 2017 and 2018 email threads.
> > >
> > > On May 13, 2020, at 3:26 AM, Thomas Schwinge <thomas@codesourcery.com> wrote:
> > > >
> > > > Comparing DejaGnu/GCC testsuite '*.sum' files between two systems ("old"
> > > > vs. "new") that ought to return identical results, I found that they
> > > > didn't:
> > >
> > > > I have not found any evidence in DejaGnu master branch that this not
> > > > working would've been a "recent" DejaGnu regression (and then fixed for
> > > > DejaGnu 1.6) -- so do we have to assume that this never worked as
> > > > intended back then?
> > >
> > > Likely not.
> > >
> > > > Per our "Prerequisites for GCC" installation documentation, we currently
> > > > require DejaGnu 1.4.4.  Advancing that to 1.6 is probably out of
> > > > question, given that it has "just" been released (four years ago).
> > >
> > > :-)  A user that wants full coverage should use 1.6, apparently.
> >
> > As documented at
> > https://gcc.gnu.org/onlinedocs/libstdc++/manual/test.html#test.run.permutations
> > anything older than 1.5.3 causes problems for libstdc++ (and probably
> > the rest of GCC) because the options in --target_board get placed
> > after the options in dg-options. If the test depends on the options in
> > dg-options to work properly it might fail. For example, a test that
> > has { dg-options "-O2" } and fails without optimisation would FAIL if
> > you use --target_board=unix/-O0 with dejagnu 1.5.
> >
> I think that was commit:
> http://git.savannah.gnu.org/gitweb/?p=dejagnu.git;a=commitdiff;h=5256bd82343000c76bc0e48139003f90b6184347
> which for sure was a major change (though I don't see it documented in
> dejagnu/NEWS file)


Yes, that's the one.

Re: dejagnu version update?

[Rainer Orth]

Hi Maciej,

> On Wed, 13 May 2020, Rainer Orth wrote:
>
>> > I'm in favour of requiring 1.5.3 or later, so 1.6 would be OK for me.
>> 
>> If we go beyond 1.5.x, we need to go all the way up to 1.6.2: 1.6 and
>> 1.6.1 have an ugly bug that can miss timeouts, causing tests to hang
>> indefinitely until one manually kills them.
>
>  Would you mind sharing a reference such as a DejaGNU Git commit ID of the 
> fix for the bug you mean?
>
>  Versions 1.6 and 1.6.1 seem ubiquitous and coincidentally earlier this 
> very week I have been chasing a phenomenon with Expect's `wait' semantics 
> causing a reliable hang in remote testing if `ssh' to the target board 
> stops responding for whatever reason.  I have come up with a solution 
> (that I'd be happy to upstream, except that DejaGNU maintenance seems to 
> have been dead for like a year now), which I have also confirmed to be 
> required with current DejaGNU Git master so it must be a different one, 
> and I would like to know how it might be related to the bug you mention.

I believe it's

commit b4e61f85ae26d215e8ad5d6e9fcda6c0745096a1
Author: Richard Biener <rguenther@suse.de>
Date:   Fri Jul 6 21:35:29 2018 +1000

            * lib/remote.exp (close_wait_program): Use separate kill command
            for each pid.

Richard pointed me at the 1.6.2 release when I mentioned the issue
somewhere on a gcc list, and it's his patch, so it's likely to be the
one.

It went into DejaGnu immediately after the 1.6.1. release ;-(

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: dejagnu version update?

[Rob Savoye]

On 5/14/20 8:08 AM, Rainer Orth wrote:

>> stops responding for whatever reason.  I have come up with a solution 
>> (that I'd be happy to upstream, except that DejaGNU maintenance seems to 
>> have been dead for like a year now), which I have also confirmed to be 
>> required with current DejaGNU Git master so it must be a different one, 
>> and I would like to know how it might be related to the bug you mention.

  I feel I need to bring up the issue that DejaGnu is 30 years old, and
it's two maintainers are either trying to pay bills, or trying to
retire, or both... This problem will effect more projects in the future,
not just DejaGnu. I'd love to see if anyone would like to become a
co-maintainer, who preferably will be actively working for a few decades
still. I think most people on these lists make their income from working
on the toolchain, but some of us are still volunteers... and getting
older every day...

  There is a patch backlog neither of us have even looked at, sorry. I'm
willing to put some time into this, but I think you all realize the time
involved to adequately test this. I'm not sure I have enough disk space. :-)

  Personally, I tried to find funding to refactor DejaGnu in Python,
since Tcl is unmaintained too, but nobody was interested.

	- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Maciej W. Rozycki]

Hi Rainer,

> >  Versions 1.6 and 1.6.1 seem ubiquitous and coincidentally earlier this 
> > very week I have been chasing a phenomenon with Expect's `wait' semantics 
> > causing a reliable hang in remote testing if `ssh' to the target board 
> > stops responding for whatever reason.  I have come up with a solution 
> > (that I'd be happy to upstream, except that DejaGNU maintenance seems to 
> > have been dead for like a year now), which I have also confirmed to be 
> > required with current DejaGNU Git master so it must be a different one, 
> > and I would like to know how it might be related to the bug you mention.
> 
> I believe it's
> 
> commit b4e61f85ae26d215e8ad5d6e9fcda6c0745096a1
> Author: Richard Biener <rguenther@suse.de>
> Date:   Fri Jul 6 21:35:29 2018 +1000
> 
>             * lib/remote.exp (close_wait_program): Use separate kill command
>             for each pid.
> 
> Richard pointed me at the 1.6.2 release when I mentioned the issue
> somewhere on a gcc list, and it's his patch, so it's likely to be the
> one.

 Thanks; I have seen it and even backported it to my test environment.  
It does not affect my scenario at all however as the kill commands get 
killed just as Expect's `wait' succeeds and then the subsequent TCL 
`close' invocation made on the pipeline in the caller hangs on wait(2) 
indefinitely.  So it's all the same area, but different modes of failure.  
I'll wrap up on my patches then and post them soon.

> It went into DejaGnu immediately after the 1.6.1. release ;-(

 Well, 1.6.2 has been out for a while now so I guess distros might well 
upgrade.  I think it's one of those packages there is not need with to 
wait for code to settle, as it's for core developers' use.  OTOH it's not 
a big deal for the intended users to check out and install the top of 
trunk version anyway.

  Maciej

Re: dejagnu version update?

[David Edelsohn]

On Thu, May 14, 2020 at 12:00 PM Rob Savoye <rob@welcomehome.org> wrote:
>
> On 5/14/20 8:08 AM, Rainer Orth wrote:
>
> >> stops responding for whatever reason.  I have come up with a solution
> >> (that I'd be happy to upstream, except that DejaGNU maintenance seems to
> >> have been dead for like a year now), which I have also confirmed to be
> >> required with current DejaGNU Git master so it must be a different one,
> >> and I would like to know how it might be related to the bug you mention.
>
>   I feel I need to bring up the issue that DejaGnu is 30 years old, and
> it's two maintainers are either trying to pay bills, or trying to
> retire, or both... This problem will effect more projects in the future,
> not just DejaGnu. I'd love to see if anyone would like to become a
> co-maintainer, who preferably will be actively working for a few decades
> still. I think most people on these lists make their income from working
> on the toolchain, but some of us are still volunteers... and getting
> older every day...
>
>   There is a patch backlog neither of us have even looked at, sorry. I'm
> willing to put some time into this, but I think you all realize the time
> involved to adequately test this. I'm not sure I have enough disk space. :-)
>
>   Personally, I tried to find funding to refactor DejaGnu in Python,
> since Tcl is unmaintained too, but nobody was interested.

I greatly appreciate all of the work that you have invested in the
infrastructure and your continued assistance.

Have you approached the Linux Foundation Core Infrastructure
Initiative for funding for both DejaGNU maintenance (patch backlog)
and refactoring DejaGNU in Python efforts?

Thanks, David

Re: dejagnu version update?

[Rob Savoye]

On 5/14/20 10:08 AM, David Edelsohn wrote:

> Have you approached the Linux Foundation Core Infrastructure
> Initiative for funding for both DejaGNU maintenance (patch backlog)
> and refactoring DejaGNU in Python efforts?

  Not that team, the folks I talked to thought I was crazy for wanting
to refactor it. :-) That's been the usual answer from anyone. I even
talked about this at GNU Tools Cauldron once. Seriously though, it'd be
great to analyze the current code base, write an actual design document,
clean internal APIs, and build something we can use for another 30
years. I'd start by writing a solid expect module for Python, and then
embedding Tcl in Python as a comparability layer. I understand it'd be a
huge project, which is why I haven't done this as a volunteer.

  I think there has been a bit of a leave it alone so it stays stable
attitude... Bugs in the test framework effect our ability to work on the
toolchain efficiently, but other than the handful of people here, nobody
cares. Forgetting the refactoring daydream, testing patches and doing
releases still needs to be done, so I think we need a long-term solution
and fresh energy.

	- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Tom Tromey]

>>>>> "Rob" == Rob Savoye <rob@welcomehome.org> writes:

Rob>   Not that team, the folks I talked to thought I was crazy for wanting
Rob> to refactor it. :-)

I don't think refactoring dejagnu is crazy, but I think it's pretty hard
to imagine rewriting the gdb test suite in Python.  It's 260 KLOC.

Tom

Re: dejagnu version update?

[Maciej W. Rozycki]

On Thu, 14 May 2020, Rob Savoye wrote:

>   Personally, I tried to find funding to refactor DejaGnu in Python,
> since Tcl is unmaintained too, but nobody was interested.

 Thank you for your contribution to DejaGnu over the years and for your 
input on this occasion.

 However it does not appear to me to be right that Tcl is not maintained, 
as a stable 8.6.10 release has been made as recently as half a year ago. 
And then current development appears ongoing, ferociously indeed, with the 
last check in literally today (barring my time zone), as indicated here: 
<https://core.tcl-lang.org/tcl/timeline>.

 Other than that what would be the technical advantage from rewriting 
DejaGnu in Python (/me asking as a Python ignoramus)?

  Maciej

Re: dejagnu version update?

[Rob Savoye]

On 5/14/20 5:34 PM, Maciej W. Rozycki wrote:

> And then current development appears ongoing, ferociously indeed, with the 
> last check in literally today (barring my time zone), as indicated here: 
> <https://core.tcl-lang.org/tcl/timeline>.

  It's obvious I haven't been paying attention, so much for my
retirement... :-)  That's awesome actually, and it does look active.

>  Other than that what would be the technical advantage from rewriting 
> DejaGnu in Python (/me asking as a Python ignoramus)?

  The main thought was more towards an actual design and clean APIs, and
assuming Tcl/Expect was unmaintained, needed to use something that'll be
around for a few more decades. Assuming Tcl/Expect are maintained
forever, that's a bit of a moot point. Everybody has always complained
about using Tcl in DejaGnu. Python is just much more commonly used in
the current century.

  Right now working through patches is probably more important. :-)
There's zero patches on the DejaGnu savannah site, so I'd ask anybody to
submit them so I don't have to dig them out of email archives.

	- rob -

Re: dejagnu version update?

[Jacob Bachmeyer]

Rob Savoye wrote:
> On 5/14/20 10:08 AM, David Edelsohn wrote:
>   
>> Have you approached the Linux Foundation Core Infrastructure
>> Initiative for funding for both DejaGNU maintenance (patch backlog)
>> and refactoring DejaGNU in Python efforts?
>>     
>
>   Not that team, the folks I talked to thought I was crazy for wanting
> to refactor it. :-) That's been the usual answer from anyone. I even
> talked about this at GNU Tools Cauldron once. Seriously though, it'd be
> great to analyze the current code base, write an actual design document,
> clean internal APIs, and build something we can use for another 30
> years.
This is very close to the volunteer work I have been doing on DejaGNU:  
just documenting the existing API would be a major step forwards, with 
follow-up plans to slowly move DejaGNU to more modern Tcl.  Using modern 
Tcl features, like namespaces (introduced in Tcl 8.0) and slave 
interpreters (introduced in Tcl 7.6) would enable significant clean up 
and possibly even simultaneous support for multiple testsuite API 
versions, allowing even backwards-incompatible changes by preserving 
support for the old API and requiring the new API to be requested.

>  I'd start by writing a solid expect module for Python, and then
> embedding Tcl in Python as a comparability layer. I understand it'd be a
> huge project, which is why I haven't done this as a volunteer.
>   
Amusingly, I believe both of those may already be in CPAN for Perl 5.  :-)

-- Jacob

Re: dejagnu version update?

[Mike Stump]

On May 14, 2020, at 11:11 AM, Tom Tromey <tom@tromey.com> wrote:
> 
>>>>>> "Rob" == Rob Savoye <rob@welcomehome.org> writes:
> 
> Rob>   Not that team, the folks I talked to thought I was crazy for wanting
> Rob> to refactor it. :-)
> 
> I don't think refactoring dejagnu is crazy, but I think it's pretty hard
> to imagine rewriting the gdb test suite in Python.  It's 260 KLOC.

So, TCL is subject to being easy to parse, and if you can reliable move each feature to a new system with a re-engineering style system that is complete enough to handle converting code from TCL to python, for example; one merely needs to complete the work for a few of the odd corner cases one might use.  At some point, I do think as an industry, we do need tools to migrate code from system to system, updating the language used.  C++ may well fall outside of the possibility for the next 30-90 years, but TCL, lisp and python might not be so unreasonable in a shorter timeframe.  I one saw someone convert TCL into lisp I think it was, which I thought was neat.

One day, would be nice if language implementors and designers implemented conversions into and out of their language from _the_ re-engineering toolkit as they did their language.  10 or 30 years after they decide, oh, no more support for you, you're dead, you can then migrate to the next new wiz bang language.

Yes, I say this all, even knowing that people can't even do the python 2.7 -> 3.x conversion program yet.

Anyway, love to have software that can move code wholesale.  Love to move the testsuite into a new language.

Re: dejagnu version update?

[Rob Savoye]

On 5/15/20 6:22 PM, Mike Stump wrote:

> Anyway, love to have software that can move code wholesale.  Love to move the testsuite into a new language.

  All it needs is funding. :-) What GDB needs is expect, not Tcl. Most
of the GDB testsuite is just expect pattern matching from the shell.
That's the entire reason I choose Tcl as it already had expect support.
Expect was necessary functionality for GDB testing. For GCC &  Binutils,
Expect is only used for remote testing support.

  As it's possible to embed Tcl in other programs, the idea was to use
an embedded Tcl interpreter when needed during a transition period. It's
mostly just the framework itself that would need to be refactored into
Python. There is also a large amount of code in gcc/testsuite that
should probably be in core DejaGnu too. That would be a large component
in analyzing existing code to write a true design doc. The best part is
now we have large toolchain testsuites to use to test DejaGnu changes.

 A one point we thought DejaGnu would be used for other projects, but I
think it's obviously that these days it's  only used for GNU toolchain
testing.

  I'm making progress on setting up a development environment to test
patches. I use my ABE tool to build toolchains, had to fix some bugs
(and add PI support) first.

	- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Maciej W. Rozycki]

On Thu, 14 May 2020, Rob Savoye wrote:

>   Right now working through patches is probably more important. :-)
> There's zero patches on the DejaGnu savannah site, so I'd ask anybody to
> submit them so I don't have to dig them out of email archives.

 I have reposted the single patch I have had outstanding.  Please review.  
I have a couple of new ones down the line.

 Overall perhaps a patch management system might be good having to make 
chasing patches easier, such as patchwork, and we already use Git, so we 
are set for using that tool.  Siddhesh has recently installed version 2.0 
for glibc and migrated all the old patchwork data, and might be able to 
give us some input.

 Siddhesh, would you care to tell us how much effort it would be to set up 
fresh patchwork?  The patch traffic is surely much lower with DejaGnu than 
it is with glibc, and there would be no data to migrate (but we might want 
to feed a couple of months' back worth of mailing list traffic).

 Just a suggestion.  See: <https://patchwork.sourceware.org/> to get a 
feel.

  Maciej

Re: dejagnu version update?

[Siddhesh Poyarekar]

On 17/05/20 05:15, Maciej W. Rozycki wrote:
>  Siddhesh, would you care to tell us how much effort it would be to set up 
> fresh patchwork?  The patch traffic is surely much lower with DejaGnu than 
> it is with glibc, and there would be no data to migrate (but we might want 
> to feed a couple of months' back worth of mailing list traffic).
> 

Not much on my end, I only need to create a project on sourceware.
Dejagnu admins will have to add the patchwork email address to their
mailing list from the back end and then you'd have to point me or Carlos
to the email archive to seed the project.

That and nominate a maintainer or two for the project on patchwork to
carry out regular tasks.

Siddhesh

Re: dejagnu version update?

[Rob Savoye]

On 5/16/20 5:45 PM, Maciej W. Rozycki wrote:

>  Overall perhaps a patch management system might be good having to make 
> chasing patches easier, such as patchwork, and we already use Git, so we 

 As an old GNU project, we're required to use what the FSF prefers,
which is on savannah. https://savannah.gnu.org/patch/?group=dejagnu, Our
bug tracker is there their too. We've used that for a long time. Yes,
patches in email are harder to track.

> fresh patchwork?  The patch traffic is surely much lower with DejaGnu than 
> it is with glibc, and there would be no data to migrate (but we might want 
> to feed a couple of months' back worth of mailing list traffic).

  I'm now building up the infrastructure to properly test patches, but
it's not enough to do the next release. All I have these days is my
laptop and a PI B3+. I'd need access to more hardware as some of the
patches effect cross testing, or get others to test the release candidates.

  Much of the problems with cross testing are often obscure timing
problems. It's amazing how sometimes a minor unrelated change changes
the timing and things break... To do a release properly requires
duplicating that level of infrastructure for at least several targets
and several toolchain release, and built on more than one GNU/Linux distro.

 It'll take most of the week to really get a good base setup with
baseline test results, but some of the patches like the DejaGnu
testsuite ones will go in first since they don't effect the toolchain.

  Jacob already added 9 patches to our site. I'm still building cross
compilers since some of his patches effect cross testing. I did add ADA
to my builds, which isn't a normal build default, since I thought some
of the patches for ADA.

	- rob -
---
https://www.senecass.com

Re: dejagnu version update?

[Andrew Pinski]

On Sat, May 16, 2020 at 9:02 PM Rob Savoye <rob@welcomehome.org> wrote:
>
> On 5/16/20 5:45 PM, Maciej W. Rozycki wrote:
>
> >  Overall perhaps a patch management system might be good having to make
> > chasing patches easier, such as patchwork, and we already use Git, so we
>
>  As an old GNU project, we're required to use what the FSF prefers,
> which is on savannah. https://savannah.gnu.org/patch/?group=dejagnu, Our
> bug tracker is there their too. We've used that for a long time. Yes,
> patches in email are harder to track.
>
> > fresh patchwork?  The patch traffic is surely much lower with DejaGnu than
> > it is with glibc, and there would be no data to migrate (but we might want
> > to feed a couple of months' back worth of mailing list traffic).
>
>   I'm now building up the infrastructure to properly test patches, but
> it's not enough to do the next release. All I have these days is my
> laptop and a PI B3+. I'd need access to more hardware as some of the
> patches effect cross testing, or get others to test the release candidates.

You should be able to use the gcc compilefarm too for this purpose
(https://gcc.gnu.org/wiki/CompileFarm).

Thanks,
Andrew

>
>   Much of the problems with cross testing are often obscure timing
> problems. It's amazing how sometimes a minor unrelated change changes
> the timing and things break... To do a release properly requires
> duplicating that level of infrastructure for at least several targets
> and several toolchain release, and built on more than one GNU/Linux distro.
>
>  It'll take most of the week to really get a good base setup with
> baseline test results, but some of the patches like the DejaGnu
> testsuite ones will go in first since they don't effect the toolchain.
>
>   Jacob already added 9 patches to our site. I'm still building cross
> compilers since some of his patches effect cross testing. I did add ADA
> to my builds, which isn't a normal build default, since I thought some
> of the patches for ADA.
>
>         - rob -
> ---
> https://www.senecass.com

Re: dejagnu version update?

[Maciej W. Rozycki]

On Sat, 16 May 2020, Rob Savoye wrote:

> >  Overall perhaps a patch management system might be good having to make 
> > chasing patches easier, such as patchwork, and we already use Git, so we 
> 
>  As an old GNU project, we're required to use what the FSF prefers,
> which is on savannah. https://savannah.gnu.org/patch/?group=dejagnu,

 Hmm, as a youngster who's been around for twenty-something years only (my 
first patch was for glibc, back in 1998) I haven't even heard of this 
patch service before.  It doesn't appear linked to our mailing list either 
and instead you need to go through the hoops of a web interface (and open 
an account first) to submit a change.

> Our
> bug tracker is there their too. We've used that for a long time. Yes,
> patches in email are harder to track.

 That's precisely what patchworks is for, which has been used to various 
extent for the GNU C library, GCC and GDB already.  All of which projects 
are of similar vintage to DejaGnu and I reckon rather important for the 
GNU project.  Given that our main patch submission channel is e-mail, 
what's the point in using a system that does not accept one?

  Maciej

Re: dejagnu version update?

[Rob Savoye]

On 5/17/20 1:43 PM, Maciej W. Rozycki wrote:

> patch service before.  It doesn't appear linked to our mailing list either 
> and instead you need to go through the hoops of a web interface (and open 
> an account first) to submit a change.

  From what I remember, it was decided the GNU toolchain (minus DejaGnu)
that the savannah infrastructure was insufficient. Some of the others
here probably remember better. GNU projects are different when it's not
part of the toolchain. My Gnash project uses savannah heavily. But you
are right, it forces you into a webbie world, and email is mostly for
notifications.

>  That's precisely what patchworks is for, which has been used to various 
> extent for the GNU C library, GCC and GDB already.  All of which projects 
> are of similar vintage to DejaGnu and I reckon rather important for the 
> GNU project.  Given that our main patch submission channel is e-mail, 
> what's the point in using a system that does not accept one?

  So maybe the discussion point is if we want to consider DejaGnu part
of the toolchain, and use the same infrastructure as the other tools. I have
no problem with that. DejaGnu doesn't change much, and there were only 2
of us working on it, so we just never thought about it.

	- rob -

Re: dejagnu version update?

[Rob Savoye]

  I processed the patch backlog for DejaGnu, and have gone through the
bug list. It'd be nice if somebody could try master with a more complex
environment, etc... if I'm going to push out a release. For cross
testing all I have is a PI and QEMU.

	- rob -

Re: dejagnu version update?

[Maciej W. Rozycki]

On Tue, 26 May 2020, Rob Savoye wrote:

>   I processed the patch backlog for DejaGnu, and have gone through the
> bug list. It'd be nice if somebody could try master with a more complex
> environment, etc... if I'm going to push out a release. For cross
> testing all I have is a PI and QEMU.

 I'll run some RISC-V remote GCC/GDB testing and compare results for 
DejaGnu 1.6/1.6.1 vs trunk.  It will take several days though, as it takes 
many hours to go through these testsuite runs.

  Maciej

Re: dejagnu version update?

[Rob Savoye]

On 5/26/20 7:20 PM, Maciej W. Rozycki wrote:

>  I'll run some RISC-V remote GCC/GDB testing and compare results for 
> DejaGnu 1.6/1.6.1 vs trunk.  It will take several days though, as it takes 
> many hours to go through these testsuite runs.

  That'd be great. I'd rather push out a stable release, than have to
fix bugs right after it gets released.

	- rob -

Re: dejagnu version update?

[Maciej W. Rozycki]

On Tue, 26 May 2020, Rob Savoye wrote:

> >  I'll run some RISC-V remote GCC/GDB testing and compare results for 
> > DejaGnu 1.6/1.6.1 vs trunk.  It will take several days though, as it takes 
> > many hours to go through these testsuite runs.
> 
>   That'd be great. I'd rather push out a stable release, than have to
> fix bugs right after it gets released.

 I have completed GCC and GDB testing now with the `riscv64-linux-gnu' 
target and the `x86_64-linux-gnu' host.

 GDB results obtained with real hardware are looking good, with the usual 
fluctuation due to intermittent failures caused by races in test cases:

@@ -63343,8 +63344,8 @@

 		=== gdb Summary ===

-# of expected passes		59902
-# of unexpected failures	778
+# of expected passes		59900
+# of unexpected failures	781
 # of unexpected successes	2
 # of expected failures		48
 # of unknown successes		5

 Similarly GCC test results taken with QEMU in the user emulation mode 
show a couple of intermittent discrepancies (not present on a rerun) 
within the libgo testsuite only:

@@ -912,12 +912,12 @@ Schedule of variations:

 Running target qemu-user-lp64d
 Running .../libgo/testsuite/libgo.testmain/testmain.exp ...
-FAIL: database/sql
+PASS: database/sql

 		=== libgo Summary ===

-# of unexpected failures	1
-Test run by macro on Thu Jun  4 04:29:42 2020
+# of expected passes		1
+Test run by macro on Wed Jun  3 01:38:10 2020
 Target is riscv64-unknown-linux-gnu
 Host   is riscv64-unknown-linux-gnu
 Build  is x86_64-pc-linux-gnu

and:

@@ -2239,12 +2239,12 @@ Schedule of variations:

 Running target qemu-user-lp64d
 Running .../libgo/testsuite/libgo.testmain/testmain.exp ...
-PASS: net/http/pprof
+FAIL: net/http/pprof

 		=== libgo Summary ===

-# of expected passes		1
-Test run by macro on Thu Jun  4 02:42:17 2020
+# of unexpected failures	1
+Test run by macro on Tue Jun  2 23:54:18 2020
 Target is riscv64-unknown-linux-gnu
 Host   is riscv64-unknown-linux-gnu
 Build  is x86_64-pc-linux-gnu

(the testsuite is run in a peculiar manner in the remote case, hence the 
individual per-test summaries) and are otherwise identical.

 However native `x86_64-linux-gnu' GDB testing does not work at all; all 
it gets is:

get_compiler_info: default_target_compile: No compiler to compile with
get_compiler_info: compiler_info not provided
get_compiler_info: got unexpected diagnostics
get_compiler_info: unknown
gdb compile failed, default_target_compile: No compiler to compile with

Conversely all things being equal DejaGnu 1.6 finds native `/usr/bin/gcc' 
just fine:

get_compiler_info: set compiler_info "unknown"
get_compiler_info: set compiler_info [join {gcc 9 2 1} -]
get_compiler_info: gcc-9-2-1

and in the verbose mode also:

Checking /home/macro/bin for gcc
Checking /home/macro/bin for gcc
Checking /usr/local/bin for gcc
Checking /usr/bin for gcc
Choosing /usr/bin/gcc

I ran a quick bisection and the culprit turned out to be:

ba60272a5ac6f6a7012acca03f596a6ed003f044 is the first bad commit
commit ba60272a5ac6f6a7012acca03f596a6ed003f044
Author: Jacob Bachmeyer <jcb62281+dev@gmail.com>
Date:   Mon May 25 08:40:46 2020 -0600

     Establish a default C compiler by evaluating [find_gcc] if no other compiler is given.

 So this regression has to be fixed before any new release is made.

 HTH,

  Maciej

Re: dejagnu version update?

[Jacob Bachmeyer]

Maciej W. Rozycki wrote:
> On Tue, 26 May 2020, Rob Savoye wrote:
>   
>>>  I'll run some RISC-V remote GCC/GDB testing and compare results for 
>>> DejaGnu 1.6/1.6.1 vs trunk.  It will take several days though, as it takes 
>>> many hours to go through these testsuite runs.
>>>       
>>   That'd be great. I'd rather push out a stable release, than have to
>> fix bugs right after it gets released.
>>     
>
>  I have completed GCC and GDB testing now with the `riscv64-linux-gnu' 
> target and the `x86_64-linux-gnu' host.
>
>  GDB results obtained with real hardware are looking good, with the usual 
> fluctuation due to intermittent failures caused by races in test cases:
>
> [...]
>
>  Similarly GCC test results taken with QEMU in the user emulation mode 
> show a couple of intermittent discrepancies (not present on a rerun) 
> within the libgo testsuite only:
>
> [...]
>
>  However native `x86_64-linux-gnu' GDB testing does not work at all; all 
> it gets is:
>
> get_compiler_info: default_target_compile: No compiler to compile with
> get_compiler_info: compiler_info not provided
> get_compiler_info: got unexpected diagnostics
> get_compiler_info: unknown
> gdb compile failed, default_target_compile: No compiler to compile with
>
> Conversely all things being equal DejaGnu 1.6 finds native `/usr/bin/gcc' 
> just fine:
>
> get_compiler_info: set compiler_info "unknown"
> get_compiler_info: set compiler_info [join {gcc 9 2 1} -]
> get_compiler_info: gcc-9-2-1
>
> and in the verbose mode also:
>
> Checking /home/macro/bin for gcc
> Checking /home/macro/bin for gcc
> Checking /usr/local/bin for gcc
> Checking /usr/bin for gcc
> Choosing /usr/bin/gcc
>
> I ran a quick bisection and the culprit turned out to be:
>
> ba60272a5ac6f6a7012acca03f596a6ed003f044 is the first bad commit
> commit ba60272a5ac6f6a7012acca03f596a6ed003f044
> Author: Jacob Bachmeyer <jcb62281+dev@gmail.com>
> Date:   Mon May 25 08:40:46 2020 -0600
>
>      Establish a default C compiler by evaluating [find_gcc] if no other compiler is given.
>
>  So this regression has to be fixed before any new release is made.
>   

I will look into this.  So far, I have confirmed that the problem does 
occur and that setting the "compiler" board_info key in 
baseboards/unix.exp seems to avoid it.  It looks like the default is not 
being selected in all cases where it should be used.  I still need to 
find the exact problem to write a regression test to isolate this bug 
and make it stay squashed.  Please submit a bug report to 
<bug-dejagnu@gnu.org> for this issue.


Side note to Rob:  should I base the fix for this on the 
"new-api-for-1.6.3" branch or should we have a merge commit for that 
later?  (Or should I go ahead and merge that branch now?)

-- Jacob

Re: dejagnu version update? [CORRECTION: not a regression in DejaGnu; GDB testsuite bug]

[Jacob Bachmeyer]

Jacob Bachmeyer wrote:
> Maciej W. Rozycki wrote:
>> [...]
>>
>> I ran a quick bisection and the culprit turned out to be:
>>
>> ba60272a5ac6f6a7012acca03f596a6ed003f044 is the first bad commit
>> commit ba60272a5ac6f6a7012acca03f596a6ed003f044
>> Author: Jacob Bachmeyer <jcb62281+dev@gmail.com>
>> Date:   Mon May 25 08:40:46 2020 -0600
>>
>>      Establish a default C compiler by evaluating [find_gcc] if no 
>> other compiler is given.
>>
>>  So this regression has to be fixed before any new release is made.
>
> I will look into this.  So far, I have confirmed that the problem does 
> occur and that setting the "compiler" board_info key in 
> baseboards/unix.exp seems to avoid it.  It looks like the default is 
> not being selected in all cases where it should be used.  I still need 
> to find the exact problem to write a regression test to isolate this 
> bug and make it stay squashed.

After further efforts, and a few hours wasted trying to figure out why 
additional tracing code added to default_target_compile was not 
producing output, I eventually decided to just make 
default_target_compile throw a Tcl error -- and I did not get a Tcl 
error when running the tests...

That is "very interesting" and a quick grep -R reveals the culprit:  the 
GDB testsuite is overriding default_target_compile in 
gdb/testsuite/lib/future.exp.  Comments indicate that that file was 
intended to temporarily bundle certain features with the GDB testsuite 
that had not yet been merged into upstream DejaGnu -- in 2004.  Now 
DejaGnu core is continuing to develop, leaving the old code copied into 
the GDB testsuite broken, and making this NOTOURBUG.

In short, this is not a regression in DejaGnu; this is code duplicated 
into the GDB testsuite that was slated for removal from that location 
years ago and needs to be removed from the GDB testsuite, or at least 
made conditional on running under a version of DejaGnu old enough to 
require it, if such versions are still supported for running the GDB 
testsuite.  If that code has added features not present in upstream 
DejaGnu over the years, now is the time to get those patches in.


-- Jacob

Re: dejagnu version update? [CORRECTION: not a regression in DejaGnu; GDB testsuite bug]

[Maciej W. Rozycki]

[cc-ing Joel as the originator, and <gdb@sourceware.org>]

On Tue, 9 Jun 2020, Jacob Bachmeyer wrote:

> >> I ran a quick bisection and the culprit turned out to be:
> >>
> >> ba60272a5ac6f6a7012acca03f596a6ed003f044 is the first bad commit
> >> commit ba60272a5ac6f6a7012acca03f596a6ed003f044
> >> Author: Jacob Bachmeyer <jcb62281+dev@gmail.com>
> >> Date:   Mon May 25 08:40:46 2020 -0600
> >>
> >>      Establish a default C compiler by evaluating [find_gcc] if no 
> >> other compiler is given.
> >>
> >>  So this regression has to be fixed before any new release is made.
> >
> > I will look into this.  So far, I have confirmed that the problem does 
> > occur and that setting the "compiler" board_info key in 
> > baseboards/unix.exp seems to avoid it.  It looks like the default is 
> > not being selected in all cases where it should be used.  I still need 
> > to find the exact problem to write a regression test to isolate this 
> > bug and make it stay squashed.
> 
> After further efforts, and a few hours wasted trying to figure out why 
> additional tracing code added to default_target_compile was not 
> producing output, I eventually decided to just make 
> default_target_compile throw a Tcl error -- and I did not get a Tcl 
> error when running the tests...
> 
> That is "very interesting" and a quick grep -R reveals the culprit:  the 
> GDB testsuite is overriding default_target_compile in 
> gdb/testsuite/lib/future.exp.  Comments indicate that that file was 
> intended to temporarily bundle certain features with the GDB testsuite 
> that had not yet been merged into upstream DejaGnu -- in 2004.  Now 
> DejaGnu core is continuing to develop, leaving the old code copied into 
> the GDB testsuite broken, and making this NOTOURBUG.

 Well, as the name suggests (regrettably there's no adequate documentation 
there) this procedure is there to be overridden.  The way it's done in GDB 
might perhaps be non-standard, as the standard way AFAICS would be by 
providing a `gdb_compile' handler, but I think this is tangential, and the 
chosen solution is there possibly because DejaGnu had no `${tool}_compile' 
knob back then (I haven't checked).  That does not really matter though, 
as the net effect would be the same.

> In short, this is not a regression in DejaGnu; this is code duplicated 
> into the GDB testsuite that was slated for removal from that location 
> years ago and needs to be removed from the GDB testsuite, or at least 
> made conditional on running under a version of DejaGnu old enough to 
> require it, if such versions are still supported for running the GDB 
> testsuite.  If that code has added features not present in upstream 
> DejaGnu over the years, now is the time to get those patches in.

 Well, it clearly is a functional regression as the interface has changed, 
even if not a formal one, and I would consider it a release stopper.

 As it happens there are about 2 users of DejaGnu there, which means any 
fatal regression would have been easily caught in the course of change 
verification (and running binutils/GDB and GCC test suites natively is 
about as easy as it can be: `configure && make && make check'), and indeed 
should have, and then sorted with the respective communities if indeed the 
interface change is unavoidable, with a transition period so that the 
users can prepare changes to their frameworks, including backports to 
various release branches if applicable, before the old interface is 
removed from DejaGnu.  My suggestion would be to revert the change, get 
the details sorted with GDB people and only reapply it when all parties 
are ready. 

 Anyway, I have completed the verification I have volunteered to do and 
therefore I'm done with my part.  Please sort it between the communities.  
I have other stuff to do I have committed to.

  Maciej

GDB testsuite overrides default_target_compile and breaks

[Jacob Bachmeyer]

[previous Subject:  Re: dejagnu version update? [CORRECTION: not a 
regression in DejaGnu; GDB testsuite bug] ]
[adding Tom Tromey to CC list per advice from Rob Savoye]
[adding main DejaGnu mailing list to CC list]

In brief for those new to this, the GDB testsuite is currently broken 
when run with the current DejaGnu Git master.  The problem was initially 
reported off-list and traced to a copy of default_target_compile in the 
GDB testsuite.  A recent change to DejaGnu internals included a change 
to that procedure upstream.  Cleanup enabled by that change breaks the 
old copy of the code in the GDB testsuite.

What is needed in brief:
(1) Merge the features of default_target_compile that the GDB testsuite 
depends on upstream for 1.6.3.
(2) Find the actual extensibility that GDB needs here and add that 
support to the default_target_compile rewrite slated for 1.6.4.

Maciej W. Rozycki wrote:
> [cc-ing Joel as the originator, and <gdb@sourceware.org>]
>
> On Tue, 9 Jun 2020, Jacob Bachmeyer wrote:
>   
>>>> I ran a quick bisection and the culprit turned out to be:
>>>>
>>>> ba60272a5ac6f6a7012acca03f596a6ed003f044 is the first bad commit
>>>> commit ba60272a5ac6f6a7012acca03f596a6ed003f044
>>>> Author: Jacob Bachmeyer <jcb62281+dev@gmail.com>
>>>> Date:   Mon May 25 08:40:46 2020 -0600
>>>>
>>>>      Establish a default C compiler by evaluating [find_gcc] if no 
>>>> other compiler is given.
>>>>
>>>>  So this regression has to be fixed before any new release is made.
>>>>         
>>> I will look into this.  So far, I have confirmed that the problem does 
>>> occur and that setting the "compiler" board_info key in 
>>> baseboards/unix.exp seems to avoid it.  It looks like the default is 
>>> not being selected in all cases where it should be used.  I still need 
>>> to find the exact problem to write a regression test to isolate this 
>>> bug and make it stay squashed.
>>>       
>> After further efforts, and a few hours wasted trying to figure out why 
>> additional tracing code added to default_target_compile was not 
>> producing output, I eventually decided to just make 
>> default_target_compile throw a Tcl error -- and I did not get a Tcl 
>> error when running the tests...
>>
>> That is "very interesting" and a quick grep -R reveals the culprit:  the 
>> GDB testsuite is overriding default_target_compile in 
>> gdb/testsuite/lib/future.exp.  Comments indicate that that file was 
>> intended to temporarily bundle certain features with the GDB testsuite 
>> that had not yet been merged into upstream DejaGnu -- in 2004.  Now 
>> DejaGnu core is continuing to develop, leaving the old code copied into 
>> the GDB testsuite broken, and making this NOTOURBUG.
>>     
>
>  Well, as the name suggests (regrettably there's no adequate documentation 
> there) this procedure is there to be overridden.

No, *_hook indicates a procedure that is there to be overridden, but not 
by testsuites.  Overriding procedures inside DejaGnu is generally for 
testing lab customization, because a site can update their init files 
when they update DejaGnu, but testsuites need to work with multiple 
versions of DejaGnu.

>   The way it's done in GDB 
> might perhaps be non-standard, as the standard way AFAICS would be by 
> providing a `gdb_compile' handler, but I think this is tangential, and the 
> chosen solution is there possibly because DejaGnu had no `${tool}_compile' 
> knob back then (I haven't checked).  That does not really matter though, 
> as the net effect would be the same.
>   

DejaGnu does not have a ${tool}_compile -- that would be a legitimate 
procedure for a testsuite to define and use, and I believe that the GDB 
testsuite does so.  There is a ${target}_compile customization point 
that is intended for testing labs to override for specific target 
boards.  (It is not quite correctly handled in target_compile at the 
moment, but this is slated to be fixed for 1.6.4.)  The 
default_target_compile procedure is intended to be the default handler 
for target_compile for targets that do not have their own special 
compilation procedures.

>> In short, this is not a regression in DejaGnu; this is code duplicated 
>> into the GDB testsuite that was slated for removal from that location 
>> years ago and needs to be removed from the GDB testsuite, or at least 
>> made conditional on running under a version of DejaGnu old enough to 
>> require it, if such versions are still supported for running the GDB 
>> testsuite.  If that code has added features not present in upstream 
>> DejaGnu over the years, now is the time to get those patches in.
>>     
>
>  Well, it clearly is a functional regression as the interface has changed, 
> even if not a formal one, and I would consider it a release stopper.
>   

The changes were to an internal component and broke an out-of-tree copy 
of same.  I have to draw a line somewhere, and "monkeypatching the core 
is not supported and can break" is necessary for DejaGnu to develop at 
all.  Add that comments in gdb/testsuite/lib/future.exp clearly state 
(from 2004! 16 years ago!) that that entire file was supposed to only 
temporarily exist until its features were merged upstream and I say that 
now is the time to pick that dropped ball back up and get those features 
merged upstream.

>  As it happens there are about 2 users of DejaGnu there, which means any 
> fatal regression would have been easily caught in the course of change 
> verification (and running binutils/GDB and GCC test suites natively is 
> about as easy as it can be: `configure && make && make check'), and indeed 
> should have, and then sorted with the respective communities if indeed the 
> interface change is unavoidable, with a transition period so that the 
> users can prepare changes to their frameworks, including backports to 
> various release branches if applicable, before the old interface is 
> removed from DejaGnu.

There will certainly be a lengthy deprecation process for the planned 
future API cleanups; indeed, the difference between the last release in 
1.x and 2.0 is planned to be the removal of compatibility shims, but 
that is still very far off.

>   My suggestion would be to revert the change, get 
> the details sorted with GDB people and only reapply it when all parties 
> are ready.
>   

The actual change that broke the out-of-tree copy of 
default_target_compile was cleaning up the board files to remove the 
now-optional definition of the "compiler" board_info key.  (Some 
testsuites do not need to compile code and running find_gcc emits 
useless "verbose noise" in those cases.)  A patch to revert this clean 
up (but not the addition of [find_gcc] as default in 
default_target_compile) may be reasonable on the release branches for 
the remaining 1.6.x releases, since it would not significantly affect 
the core, but the change stays on the master branch.  Monkeypatching the 
DejaGnu core is not supported and cannot be supported long-term.  (In 
the future, default_target_compile may eventually disappear from the 
global namespace as seen by testsuites.  If GDB continues to override 
it, the rename will end up deleting the compatibility shim while the 
internal calls still go to the internal copy of the procedure.  Then it 
all blows up with a Tcl error in DejaGnu 2.0 when the compatibility shim 
is removed.)

GDB seems to have also extended default_target_compile, so a discussion 
on this is needed because there seems to be a need for some kind of 
language extensibility feature to allow new language-default-selector 
options to be defined without overriding the entire 
default_target_compile procedure (which could also break in a test lab 
that uses a custom ${target}_compile procedure for a certain board).  
Since default_target_compile is slated to be rewritten between 1.6.3 and 
1.6.4, now is a very good time to get additional features that it needs 
to have into the DejaGnu core.  The API is now documented as 
"target_compile" in Git master.  Backwards-compatible extensions (like 
adding new language default options) are possible, of course.

If I understand correctly, merging the features currently carried in 
gdb/testsuite/lib/future.exp upstream will cause all existing GDB 
releases to correctly use the upstream version of 
default_target_compile, making the partial reversion patch unnecessary.

>  Anyway, I have completed the verification I have volunteered to do and 
> therefore I'm done with my part.

Thank you for your help.  [To Maciej W. Rozycki:  The requests for 
further efforts in this email are for other people who work more closely 
with GDB.]



-- Jacob

Re: dejagnu version update?

[Christophe Lyon]

[-- Attachment #1: Type: text/plain, Size: 1196 bytes --]

Hi,

On Wed, 27 May 2020 at 03:58, Rob Savoye <rob@welcomehome.org> wrote:
>
> On 5/26/20 7:20 PM, Maciej W. Rozycki wrote:
>
> >  I'll run some RISC-V remote GCC/GDB testing and compare results for
> > DejaGnu 1.6/1.6.1 vs trunk.  It will take several days though, as it takes
> > many hours to go through these testsuite runs.
>
>   That'd be great. I'd rather push out a stable release, than have to
> fix bugs right after it gets released.
>
>         - rob -


I ran our GCC validation harness using dejagnu master branch and
compared to the results we get using our linaro-local/stable branch
(https://git.linaro.org/toolchain/dejagnu.git/)

I noticed that we'd need to add patches (1) and (2) at least.

Patch (1) enables us to run tests on aarch64-elf using Arm's Foundation Model.

Patch (2) was posted in 2016:
https://lists.gnu.org/archive/html/dejagnu/2016-03/msg00005.html.
It fixes problems with tests output patterns (in fortran, ubsan and asan tests)

Patch (3) was posted in 2016 too:
https://lists.gnu.org/archive/html/dejagnu/2016-03/msg00034.html
I'm not 100% sure it made a difference in these test runs because we
still see some random failures anyway.

Thanks,

Christophe

[-- Attachment #2: 0002-Keep-trailing-newline-in-remote-execution-output.patch --]
[-- Type: text/x-patch, Size: 1452 bytes --]

From 382440f145811eeb3e85d0e57d9b8aa5418d1e80 Mon Sep 17 00:00:00 2001
From: Yvan Roux <yvan.roux@linaro.org>
Date: Mon, 25 Apr 2016 11:09:52 +0200
Subject: [PATCH 2/3] Keep trailing newline in remote execution output.

	* lib/rsh.exp (rsh_exec): Don't remove trailing newline.
	* lib/ssh.exp (rsh_exec): Likewise.

Change-Id: I2368c18729c4bff9ee87d9163b1c8f4b0ad7f4d8
---
 lib/rsh.exp | 3 ---
 lib/ssh.exp | 3 ---
 2 files changed, 6 deletions(-)

diff --git a/lib/rsh.exp b/lib/rsh.exp
index 5b583c6..43f5430 100644
--- a/lib/rsh.exp
+++ b/lib/rsh.exp
@@ -283,8 +283,5 @@ proc rsh_exec { boardname program pargs inp outp } {
 	return [list -1 "Couldn't parse $RSH output, $output."]
     }
     regsub "XYZ(\[0-9\]*)ZYX\n?" $output "" output
-    # Delete one trailing \n because that is what `exec' will do and we want
-    # to behave identical to it.
-    regsub "\n$" $output "" output
     return [list [expr {$status != 0}] $output]
 }
diff --git a/lib/ssh.exp b/lib/ssh.exp
index 0cf0f8d..a72f794 100644
--- a/lib/ssh.exp
+++ b/lib/ssh.exp
@@ -194,9 +194,6 @@ proc ssh_exec { boardname program pargs inp outp } {
 	return [list -1 "Couldn't parse $SSH output, $output."]
     }
     regsub "XYZ(\[0-9\]*)ZYX\n?" $output "" output
-    # Delete one trailing \n because that is what `exec' will do and we want
-    # to behave identical to it.
-    regsub "\n$" $output "" output
     return [list [expr {$status != 0}] $output]
 }
 
-- 
2.7.4


[-- Attachment #3: 0001-Support-using-QEMU-in-local-remote-testing-using-def.patch --]
[-- Type: text/x-patch, Size: 1545 bytes --]

From 1e5110d99ac8bac61e97bbdb4bb78ca72adb7e4e Mon Sep 17 00:00:00 2001
From: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
Date: Tue, 28 Jun 2016 09:40:01 +0100
Subject: [PATCH 1/3] Support using QEMU in local/remote testing using default
 "unix" board

If the board file defines "exec_shell", prepend it before the local or
remote command.

Change-Id: Ib3ff96126c4c96e4e7f8898609d0fce6faf803ef
---
 config/unix.exp | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/config/unix.exp b/config/unix.exp
index 2e38454..dc3f781 100644
--- a/config/unix.exp
+++ b/config/unix.exp
@@ -78,6 +78,11 @@ proc unix_load { dest prog args } {
 	verbose -log "Setting LD_LIBRARY_PATH to $ld_library_path:$orig_ld_library_path" 2
 	verbose -log "Execution timeout is: $test_timeout" 2
 
+	# Prepend shell name (e.g., qemu emulator) to the command.
+	if {[board_info $dest exists exec_shell]} {
+	    set command "[board_info $dest exec_shell] $command"
+	}
+
 	set id [remote_spawn $dest $command "readonly"]
 	if { $id < 0 } {
 	    set output "remote_spawn failed"
@@ -119,6 +124,14 @@ proc unix_load { dest prog args } {
 		return [list "unresolved" ""]
 	    }
 	}
+
+	# Prepend shell name (e.g., qemu emulator) to the command.
+	if {[board_info $dest exists exec_shell]} {
+	    set remotecmd "[board_info $dest exec_shell] $remotefile"
+	} else {
+	    set remotecmd "$remotefile"
+	}
+
 	set status [remote_exec $dest $remotefile $parg $inp]
 	remote_file $dest delete $remotefile.o $remotefile
 	if { [lindex $status 0] < 0 } {
-- 
2.7.4


[-- Attachment #4: 0003-2018-09-28-Christophe-Lyon-christophe.lyon-linaro.or.patch --]
[-- Type: text/x-patch, Size: 1171 bytes --]

From b6a3e52aec69146e930d85b84a81b1e059f2ffe5 Mon Sep 17 00:00:00 2001
From: Christophe Lyon <christophe.lyon@linaro.org>
Date: Fri, 28 Sep 2018 08:26:02 +0000
Subject: [PATCH 3/3] 2018-09-28  Christophe Lyon  <christophe.lyon@linaro.org>

	* lib/ssh.exp (ssh_exec): Redirect stderr to stdout on the remote
	machine, to avoid race conditions.

Change-Id: Ie0613a85fa990484fda41b13738025edf7477a62
---
 lib/ssh.exp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/ssh.exp b/lib/ssh.exp
index a72f794..3c7b840 100644
--- a/lib/ssh.exp
+++ b/lib/ssh.exp
@@ -171,7 +171,7 @@ proc ssh_exec { boardname program pargs inp outp } {
 
     # We use && here, as otherwise the echo always works, which makes it look
     # like execution succeeded when in reality it failed.
-    set ret [local_exec "$SSH $ssh_useropts $ssh_user$hostname sh -c '$program $pargs && echo XYZ\\\${?}ZYX \\; rm -f $program'" $inp $outp $timeout]
+    set ret [local_exec "$SSH $ssh_useropts $ssh_user$hostname sh -c '$program $pargs 2>&1 && echo XYZ\\\${?}ZYX \\; rm -f $program'" $inp $outp $timeout]
     set status [lindex $ret 0]
     set output [lindex $ret 1]
 
-- 
2.7.4