Replacing DejaGNU

Replacing DejaGNU

[MCC CS]

Hi all,

I've been running the testsuite on my macOS, on which
it is especially unbearable. I want to (at least try to)
rewrite a DejaGNU replacement accepting the same
syntax and having no dependency, should therefore
be faster. I was wondering if there have been any
attempts on this? Knowing what went wrong would
help me. What I'll try to code will be a GCC-specific
multicore test suite runner in C (without using a
Makefile)

Regards

Re: Replacing DejaGNU

[Rainer Orth]

"MCC CS" <mcccs@gmx.com> writes:

> I've been running the testsuite on my macOS, on which
> it is especially unbearable. I want to (at least try to)

that problem may well be macOS specific: since at least macOS 10.13
(maybe even 10.12; cannot currently tell for certain) make -jN check
times on my Mac mini skyrocketed with between 60 and 80% system time.
It seems this is due to lock contention on one specific kernel lock, but
I haven't been able to find out more yet.

There's no such problem on other targets, not even e.g. on Mac OS X 10.7.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: Replacing DejaGNU

[Richard Biener]

On Mon, Jan 14, 2019 at 2:54 PM Rainer Orth <ro@cebitec.uni-bielefeld.de> wrote:
>
> "MCC CS" <mcccs@gmx.com> writes:
>
> > I've been running the testsuite on my macOS, on which
> > it is especially unbearable. I want to (at least try to)
>
> that problem may well be macOS specific: since at least macOS 10.13
> (maybe even 10.12; cannot currently tell for certain) make -jN check
> times on my Mac mini skyrocketed with between 60 and 80% system time.
> It seems this is due to lock contention on one specific kernel lock, but
> I haven't been able to find out more yet.
>
> There's no such problem on other targets, not even e.g. on Mac OS X 10.7.

If I would take a guess then it's security checks (verifying signatures
for each process invocation?).  IIRC you can disable this system-wide
somehow (of course that's not recommended).

Richard.

>         Rainer
>
> --
> -----------------------------------------------------------------------------
> Rainer Orth, Center for Biotechnology, Bielefeld University

Re: Replacing DejaGNU

[MCC CS]

Thank you for the quick replies. I was inspired by
https://gcc.gnu.org/ml/gcc-help/2012-04/msg00223.html
but it seems, according to your comments, that was outdated.
The problem on my Mac was each of the processes used no more
than 10% of a core. Now I know that it's not so inefficient
on other platforms, but I might try rewriting it in the future
if I have time, as I believe there's still some room for optimizations.
 
Thanks

Sent: Monday, January 14, 2019 at 4:57 PM
From: "Richard Biener" <richard.guenther@gmail.com>
To: "Rainer Orth" <ro@cebitec.uni-bielefeld.de>
Cc: "MCC CS" <mcccs@gmx.com>, "GCC Development" <gcc@gcc.gnu.org>
Subject: Re: Replacing DejaGNU
On Mon, Jan 14, 2019 at 2:54 PM Rainer Orth <ro@cebitec.uni-bielefeld.de> wrote:
>
> "MCC CS" <mcccs@gmx.com> writes:
>
> > I've been running the testsuite on my macOS, on which
> > it is especially unbearable. I want to (at least try to)
>
> that problem may well be macOS specific: since at least macOS 10.13
> (maybe even 10.12; cannot currently tell for certain) make -jN check
> times on my Mac mini skyrocketed with between 60 and 80% system time.
> It seems this is due to lock contention on one specific kernel lock, but
> I haven't been able to find out more yet.
>
> There's no such problem on other targets, not even e.g. on Mac OS X 10.7.

If I would take a guess then it's security checks (verifying signatures
for each process invocation?). IIRC you can disable this system-wide
somehow (of course that's not recommended).

Richard.

> Rainer
>
> --
> -----------------------------------------------------------------------------
> Rainer Orth, Center for Biotechnology, Bielefeld University

Re: Replacing DejaGNU

[Jakub Jelinek]

On Mon, Jan 14, 2019 at 03:15:05PM +0100, MCC CS wrote:
> Thank you for the quick replies. I was inspired by
> https://gcc.gnu.org/ml/gcc-help/2012-04/msg00223.html
> but it seems, according to your comments, that was outdated.

Since then the parallelization has been changed, since 2014 all the
instances run the same set of tests and communicate together which one
picks which test, see http://gcc.gnu.org/r215273 .

So, I don't really see what would help you replacing the testsuite
framework, moreover, we have like 400k tests now and many of them
use simpler or more complicated tcl expressions in them, including almost
2.5MB of pure tcl code.  Replacing it with something different and
incompatible is lots of work, especially when all you want is work around a
bug in some broken OS.

	Jakub

Re: Replacing DejaGNU

[Paolo Carlini]

Hi,

On 14/01/19 15:35, Jakub Jelinek wrote:
> On Mon, Jan 14, 2019 at 03:15:05PM +0100, MCC CS wrote:
>> Thank you for the quick replies. I was inspired by
>> https://gcc.gnu.org/ml/gcc-help/2012-04/msg00223.html
>> but it seems, according to your comments, that was outdated.
> So, I don't really see what would help you replacing the testsuite
> framework, moreover, we have like 400k tests now and many of them
> use simpler or more complicated tcl expressions in them, including almost
> 2.5MB of pure tcl code.  Replacing it with something different and
> incompatible is lots of work, especially when all you want is work around a
> bug in some broken OS.

I'm not an expert but certainly there are long standing issues with 
DejaGNU, well beyond perfornance, right? I remember Mark Mitchell doing 
some work in this area which, as far as I can remember, had nothing to 
do with performance per se. And, well, some of these issues are obvious 
to explain, like not being able to check for *duplicate* error messages. 
I remember briefly discussing this with Dodji in Manchester.

Just wanted to make sure this kind of public discussion isn't completely 
suppressed.

Paolo.

Re: Replacing DejaGNU

[David Edelsohn]

On Mon, Jan 14, 2019 at 9:51 AM Paolo Carlini <paolo.carlini@oracle.com> wrote:
>
> Hi,
>
> On 14/01/19 15:35, Jakub Jelinek wrote:
> > On Mon, Jan 14, 2019 at 03:15:05PM +0100, MCC CS wrote:
> >> Thank you for the quick replies. I was inspired by
> >> https://gcc.gnu.org/ml/gcc-help/2012-04/msg00223.html
> >> but it seems, according to your comments, that was outdated.
> > So, I don't really see what would help you replacing the testsuite
> > framework, moreover, we have like 400k tests now and many of them
> > use simpler or more complicated tcl expressions in them, including almost
> > 2.5MB of pure tcl code.  Replacing it with something different and
> > incompatible is lots of work, especially when all you want is work around a
> > bug in some broken OS.
>
> I'm not an expert but certainly there are long standing issues with
> DejaGNU, well beyond perfornance, right? I remember Mark Mitchell doing
> some work in this area which, as far as I can remember, had nothing to
> do with performance per se. And, well, some of these issues are obvious
> to explain, like not being able to check for *duplicate* error messages.
> I remember briefly discussing this with Dodji in Manchester.
>
> Just wanted to make sure this kind of public discussion isn't completely
> suppressed.

A few years ago, Rob Savoye mentioned that he had a plan for a replacement.

- David

Re: Replacing DejaGNU

[Jakub Jelinek]

On Mon, Jan 14, 2019 at 03:50:32PM +0100, Paolo Carlini wrote:
> On 14/01/19 15:35, Jakub Jelinek wrote:
> > On Mon, Jan 14, 2019 at 03:15:05PM +0100, MCC CS wrote:
> > > Thank you for the quick replies. I was inspired by
> > > https://gcc.gnu.org/ml/gcc-help/2012-04/msg00223.html
> > > but it seems, according to your comments, that was outdated.
> > So, I don't really see what would help you replacing the testsuite
> > framework, moreover, we have like 400k tests now and many of them
> > use simpler or more complicated tcl expressions in them, including almost
> > 2.5MB of pure tcl code.  Replacing it with something different and
> > incompatible is lots of work, especially when all you want is work around a
> > bug in some broken OS.
> 
> I'm not an expert but certainly there are long standing issues with DejaGNU,
> well beyond perfornance, right? I remember Mark Mitchell doing some work in
> this area which, as far as I can remember, had nothing to do with
> performance per se. And, well, some of these issues are obvious to explain,
> like not being able to check for *duplicate* error messages. I remember
> briefly discussing this with Dodji in Manchester.

I think several testcases check for duplicate error messages, the regexp is
against the whole text, so you can just check if it occurs more than once
there.

> Just wanted to make sure this kind of public discussion isn't completely
> suppressed.

I don't want to suppress any discussion, all I wanted to say is that
replacing the testsuite framework is a multi-year project and if it is done,
we'd need to decide on the benefits and disadvantages (even if it has a
superset of the features current framework has, having each developer learn
a new framework is non-trivial cost too).

	Jakub

Re: Replacing DejaGNU

[Paolo Carlini]

Hi,

On 14/01/19 17:28, Jakub Jelinek wrote:
> I think several testcases check for duplicate error messages, the regexp is
> against the whole text, so you can just check if it occurs more than once
> there.

This is essentially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=30612

Paolo.

Re: Replacing DejaGNU

[Jeff Law]

On 1/14/19 9:39 AM, Paolo Carlini wrote:
> Hi,
> 
> On 14/01/19 17:28, Jakub Jelinek wrote:
>> I think several testcases check for duplicate error messages, the
>> regexp is
>> against the whole text, so you can just check if it occurs more than once
>> there.
> 
> This is essentially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=30612
Anyone working in this space should probably look at Ian's blogpost.

https://www.airs.com/blog/archives/499

jeff

Re: Replacing DejaGNU

[Paolo Carlini]

Hi Jeff,

On 14/01/19 17:43, Jeff Law wrote:
> On 1/14/19 9:39 AM, Paolo Carlini wrote:
>> Hi,
>>
>> On 14/01/19 17:28, Jakub Jelinek wrote:
>>> I think several testcases check for duplicate error messages, the
>>> regexp is
>>> against the whole text, so you can just check if it occurs more than once
>>> there.
>> This is essentially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=30612
> Anyone working in this space should probably look at Ian's blogpost.
>
> https://www.airs.com/blog/archives/499

Thanks for the pointer. The fourth line is already rather encouraging ;)

Paolo.

Re: Replacing DejaGNU

[Joseph Myers]

On Mon, 14 Jan 2019, Jeff Law wrote:

> On 1/14/19 9:39 AM, Paolo Carlini wrote:
> > Hi,
> > 
> > On 14/01/19 17:28, Jakub Jelinek wrote:
> >> I think several testcases check for duplicate error messages, the
> >> regexp is
> >> against the whole text, so you can just check if it occurs more than once
> >> there.
> > 
> > This is essentially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=30612
> Anyone working in this space should probably look at Ian's blogpost.
> 
> https://www.airs.com/blog/archives/499

Also in the July 2013 discussion (starting at 
<https://gcc.gnu.org/ml/gcc/2013-07/msg00398.html> I listed a series of 
other issues with DejaGnu - especially that neither DejaGnu nor the test 
code in GCC should need to know anything at all about how to make pieces 
of an uninstalled toolchain find each other, that should be entirely the 
responsibility of "make install".

To the issues added there I should also add: the DejaGnu board file 
interface is insufficiently expressive.  Specifically, it only supports 
tests exiting with a DejaGnu status rather than an arbitrary exit code, 
which is an issue if you want to use DejaGnu board files with a 
non-DejaGnu testsuite (so that the board interaction code only has to be 
written once) - it's perfectly possible to construct a DejaGnu "testsuite" 
that takes externally provided tests to run on the board (and for that 
matter to compile, if you wish to reuse ldscript settings from the board 
file by compiling through DejaGnu), and so use a board file for a 
non-DejaGnu testsuite, but that testsuite can't get back any extra 
information it might want from test exit statuses.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Replacing DejaGNU

[Iain Sandoe]

> On 14 Jan 2019, at 13:53, Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> wrote:
> 
> "MCC CS" <mcccs@gmx.com> writes:
> 
>> I've been running the testsuite on my macOS, on which
>> it is especially unbearable. I want to (at least try to)
> 
> that problem may well be macOS specific: since at least macOS 10.13
> (maybe even 10.12; cannot currently tell for certain) make -jN check
> times on my Mac mini skyrocketed with between 60 and 80% system time.
> It seems this is due to lock contention on one specific kernel lock, but
> I haven't been able to find out more yet.

this PR mentions the compilation, but it’ even more apparent on test.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84257

* Assuming SIP is disabled.

Some testing suggests that each DYLD_LIBRARY_PATH entry adds around 2ms to each exe launch.
So .. when you’re doing something that’s a lot of work per launch, not much is seen - but when you’re doing things with a huge number of exe launches - e.g. configuring or running the test suite, it bites.

A work-around is to remove the RPATH_ENVAR variable setting in the top level Makefile.in (which actually has the same effect as running things with SIP enabled)

=== Possible solution (partial hacks locally, not ready for posting)

My current investigations (targeted at GCC 10 time frame, even if they are subsequently back-ported) is to replace all use of absolute pathnames in GCC libraries with @rpath/xxx and figure out a way to get the compiler to auto-add the relevant rpaths to exes (so that a fixed installation of GCC behaves the same way as it does currently).

=== DejaGNU on macOS...

DejaGNU / expect are not fantastic on macOS, even given the comments above - it’s true.  Writing an interpreter/funnel for the testsuite has crossed my mind more than once. 

However, I suspect it’s a large job, and it might be more worth investing any available effort in debugging the slowness in expect/dejaGNU - especially the lock contention that Rainer mentions.

> There's no such problem on other targets, not even e.g. on Mac OS X 10.7.

indeed.

Iain

Re: Replacing DejaGNU

[Rainer Orth]

Hi Iain,

>> On 14 Jan 2019, at 13:53, Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> wrote:
>> 
>> "MCC CS" <mcccs@gmx.com> writes:
>> 
>>> I've been running the testsuite on my macOS, on which
>>> it is especially unbearable. I want to (at least try to)
>> 
>> that problem may well be macOS specific: since at least macOS 10.13
>> (maybe even 10.12; cannot currently tell for certain) make -jN check
>> times on my Mac mini skyrocketed with between 60 and 80% system time.
>> It seems this is due to lock contention on one specific kernel lock, but
>> I haven't been able to find out more yet.
>
> this PR mentions the compilation, but it’ even more apparent on test.
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84257
>
> * Assuming SIP is disabled.
>
> Some testing suggests that each DYLD_LIBRARY_PATH entry adds around 2ms to
> each exe launch.
> So .. when you’re doing something that’s a lot of work per launch, not much
> is seen - but when you’re doing things with a huge number of exe launches -
> e.g. configuring or running the test suite, it bites.
>
> A work-around is to remove the RPATH_ENVAR variable setting in the top
> level Makefile.in (which actually has the same effect as running things
> with SIP enabled)

this change alone helped tremendously: a bootstrap on macOS 10.14 on
20181103 took

   180041.05 real     96489.89 user    180864.44 sys

while the current one was only

    44886.30 real     74101.86 user     36879.75 sys

However, not unexpectedly quite a number of new failures occur,
e.g. many (all?) plugin tests FAIL with

cc1: error: cannot load plugin ./selfassign.so
   dlopen(./selfassign.so, 10): Symbol not found: __ZdlPvm
  Referenced from: ./selfassign.so
  Expected in: /usr/lib/libstdc++.6.dylib
 in ./selfassign.so
compiler exited with status 1

I'll still have to check which are affected this way.

> === DejaGNU on macOS...
>
> DejaGNU / expect are not fantastic on macOS, even given the comments above
> - it’s true.  Writing an interpreter/funnel for the testsuite has crossed
> my mind more than once.
>
> However, I suspect it’s a large job, and it might be more worth investing
> any available effort in debugging the slowness in expect/dejaGNU -
> especially the lock contention that Rainer mentions.

Indeed: I found it when trying to investigate the high system time with
lockstat.  However, I don't know a way how to relate the lock address
mentioned there to some lock in the darwin sources.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: Replacing DejaGNU

[Iain Sandoe]

Hey Rainer,

> On 15 Jan 2019, at 17:27, Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> wrote:

>>> On 14 Jan 2019, at 13:53, Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> wrote:
>>> 
>>> "MCC CS" <mcccs@gmx.com> writes:
>>> 
>>>> I've been running the testsuite on my macOS, on which
>>>> it is especially unbearable. I want to (at least try to)
>>> 
>>> that problem may well be macOS specific: since at least macOS 10.13
>>> (maybe even 10.12; cannot currently tell for certain) make -jN check
>>> times on my Mac mini skyrocketed with between 60 and 80% system time.
>>> It seems this is due to lock contention on one specific kernel lock, but
>>> I haven't been able to find out more yet.
>> 
>> this PR mentions the compilation, but it’ even more apparent on test.
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84257
>> 
>> * Assuming SIP is disabled.
>> 
>> Some testing suggests that each DYLD_LIBRARY_PATH entry adds around 2ms to
>> each exe launch.
>> So .. when you’re doing something that’s a lot of work per launch, not much
>> is seen - but when you’re doing things with a huge number of exe launches -
>> e.g. configuring or running the test suite, it bites.
>> 
>> A work-around is to remove the RPATH_ENVAR variable setting in the top
>> level Makefile.in (which actually has the same effect as running things
>> with SIP enabled)
> 
> this change alone helped tremendously: a bootstrap on macOS 10.14 on
> 20181103 took

>   180041.05 real     96489.89 user    180864.44 sys
> 
> while the current one was only
> 
>    44886.30 real     74101.86 user     36879.75 sys
> 
> However, not unexpectedly quite a number of new failures occur,
> e.g. many (all?) plugin tests FAIL with
> 
> cc1: error: cannot load plugin ./selfassign.so
>   dlopen(./selfassign.so, 10): Symbol not found: __ZdlPvm
>  Referenced from: ./selfassign.so
>  Expected in: /usr/lib/libstdc++.6.dylib
> in ./selfassign.so
> compiler exited with status 1
> 
> I'll still have to check which are affected this way.

I’m afraid that with this (or with SIP enabled) “uninstalled testing” can’t work,
the libraries have to be found from their intended installed path,
so you have to “make install && make check” 

** and remember to delete the install before building the next revision...

>> === DejaGNU on macOS...
>> 
>> DejaGNU / expect are not fantastic on macOS, even given the comments above
>> - it’s true.  Writing an interpreter/funnel for the testsuite has crossed
>> my mind more than once.
>> 
>> However, I suspect it’s a large job, and it might be more worth investing
>> any available effort in debugging the slowness in expect/dejaGNU -
>> especially the lock contention that Rainer mentions.
> 
> Indeed: I found it when trying to investigate the high system time with
> lockstat.  However, I don't know a way how to relate the lock address
> mentioned there to some lock in the darwin sources.

Well.. let’s take this offline - or park it in a BZ somewhere, if you can be more 
specific - would be happy to poke at it a bit :  if it’s a genuine OS bug,
 we can file a radar - but that doesn’t help the system versions out of support.
(and there’s enough useful h/w out there that’s tied to 10.11 etc)

Iain

Re: Replacing DejaGNU

[Jim Wilson]

On 1/14/19 5:44 AM, MCC CS wrote:
> I've been running the testsuite on my macOS, on which
> it is especially unbearable. I want to (at least try to)
> rewrite a DejaGNU replacement accepting the same
> syntax and having no dependency, should therefore
> be faster. I was wondering if there have been any
> attempts on this?

CodeSourcery wrote one called qmtest, but there apparently hasn't been 
any work done on it in a while.  Joseph Myers indirectly referred to it. 
  You can find a copy here
     https://github.com/MentorEmbedded/qmtest

It used to be possible to run the gcc testsuite using qmtest, but I 
don't know the current status.  I do see that there is still a 
qmtest-g++ makefile rule for running the G++ testsuite via qmtest 
though.  You could try that and see if it still works.

There is so much stuff that depends on dejagnu that replacing it will be 
difficult.

Jim

Re: Replacing DejaGNU

[Joseph Myers]

On Mon, 14 Jan 2019, Jim Wilson wrote:

> CodeSourcery wrote one called qmtest, but there apparently hasn't been any
> work done on it in a while.  Joseph Myers indirectly referred to it.  You can
> find a copy here
>     https://github.com/MentorEmbedded/qmtest

Note that's a poor git-svn conversion, missing most of the history 
(everything before a repository rearrangement, and everything outside of 
QMTest proper - in particular, the separate qmtest_gcc that provided 
various things needed for the GCC testsuite).  If anyone seriously wants 
to do things with QMTest now, I should do a proper reposurgeon conversion 
of the full history (of qmtest and qmtest_gcc, and properly of qmtc from 
the time it was in a separate repository before being integrated into 
qmtest) to replace it.

> It used to be possible to run the gcc testsuite using qmtest, but I don't know
> the current status.  I do see that there is still a qmtest-g++ makefile rule
> for running the G++ testsuite via qmtest though.  You could try that and see
> if it still works.

README.QMTEST was removed in 2011 as the support was long bit-rotten.  I'm 
not sure why the actual bitrotten makefile support wasn't removed as well.

-- 
Joseph S. Myers
joseph@codesourcery.com