How to properly build and run testsuite?

How to properly build and run testsuite?

[Andrew Dean via gcc]

I'm curious what other people are doing, because I'm never able to match the results that get reported to the test-results list. I created a brand new virtual machine running Ubuntu 18.04 (x86_64), installed the prereqs as listed here:  https://gcc.gnu.org/install/prerequisites.html, created the repo following the "Getting Started - Read Only" instructions listed here: https://gcc.gnu.org/wiki/GitMirror, then ran these commands from my build folder.
configure --disable-multilib --prefix=/home/adean/install
make
make check -k

As an example, the gcc summary for me (10.0.0 20191120) shows 
# of unexpected failures        85
# of unexpected successes       35

Whereas the most recent reported results (10.0.0 20191118) show only 2 unexpected failures and no unexpected successes in the gcc summary.

Is it really just because I'm two days newer that ~120 regressions entered the picture (unlikely) or am I doing something wrong on my machine?

Thanks,
Andrew

Re: How to properly build and run testsuite?

[Jonathan Wakely]

On Thu, 21 Nov 2019 at 19:11, Andrew Dean via gcc <gcc@gcc.gnu.org> wrote:
>
> I'm curious what other people are doing, because I'm never able to match the results that get reported to the test-results list. I created a brand new virtual machine running Ubuntu 18.04 (x86_64), installed the prereqs as listed here:  https://gcc.gnu.org/install/prerequisites.html, created the repo following the "Getting Started - Read Only" instructions listed here: https://gcc.gnu.org/wiki/GitMirror, then ran these commands from my build folder.
> configure --disable-multilib --prefix=/home/adean/install
> make
> make check -k
>
> As an example, the gcc summary for me (10.0.0 20191120) shows
> # of unexpected failures        85
> # of unexpected successes       35

That doesn't seem too bad.

> Whereas the most recent reported results (10.0.0 20191118) show only 2 unexpected failures and no unexpected successes in the gcc summary.

Which results are you looking at?
Two failures sounds very low, it's probably not running the guality
tests which usually fail.

> Is it really just because I'm two days newer that ~120 regressions entered the picture (unlikely) or am I doing something wrong on my machine?
>
> Thanks,
> Andrew
>

RE: How to properly build and run testsuite?

[Andrew Dean via gcc]

> > Whereas the most recent reported results (10.0.0 20191118) show only 2
> unexpected failures and no unexpected successes in the gcc summary.
> 
> Which results are you looking at?
> Two failures sounds very low, it's probably not running the guality tests which
> usually fail.
> 
I searched the mailing list for x86_64-pc-linux-gnu to make sure I was comparing apples to apples, and this was the most recent report: https://gcc.gnu.org/ml/gcc-testresults/2019-11/msg01190.html

Re: How to properly build and run testsuite?

[Jonathan Wakely]

On Thu, 21 Nov 2019 at 21:30, Andrew Dean <Andrew.Dean@microsoft.com> wrote:
>
> > > Whereas the most recent reported results (10.0.0 20191118) show only 2
> > unexpected failures and no unexpected successes in the gcc summary.
> >
> > Which results are you looking at?
> > Two failures sounds very low, it's probably not running the guality tests which
> > usually fail.
> >
> I searched the mailing list for x86_64-pc-linux-gnu to make sure I was comparing apples to apples, and this was the most recent report: https://gcc.gnu.org/ml/gcc-testresults/2019-11/msg01190.html

Yes, I thought you might be looking at that one. That has:
# of unsupported tests 7126
which seems high. Here's a more typical set of results:
https://gcc.gnu.org/ml/gcc-testresults/2019-11/msg01255.html

The one you looked at is built on an EC2 instance, so might be missing
something (GDB?) needed for the guality tests.

So I don't think you're doing anything wrong, you just got unlucky and
looked at one which skips most of the tests that are failing for you.

RE: How to properly build and run testsuite?

[Andrew Dean via gcc]

> > > > Whereas the most recent reported results (10.0.0 20191118) show
> > > > only 2
> > > unexpected failures and no unexpected successes in the gcc summary.
> > >
> > > Which results are you looking at?
> > > Two failures sounds very low, it's probably not running the guality
> > > tests which usually fail.
> > >
> > I searched the mailing list for x86_64-pc-linux-gnu to make sure I was
> > comparing apples to apples, and this was the most recent report:
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.
> > gnu.org%2Fml%2Fgcc-testresults%2F2019-
> 11%2Fmsg01190.html&data=02%7
> >
> C01%7CAndrew.Dean%40microsoft.com%7Cb89abb6ea9f34a0916d908d76ed0
> 3783%7
> >
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637099712871926073
> &sda
> >
> ta=KooYFswCjKMaV%2FQ3D6tc03WAej8MfQ4Zl9H9kjtR%2B6Y%3D&rese
> rved=0
> 
> Yes, I thought you might be looking at that one. That has:
> # of unsupported tests 7126
> which seems high. Here's a more typical set of results:
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu
> .org%2Fml%2Fgcc-testresults%2F2019-
> 11%2Fmsg01255.html&data=02%7C01%7CAndrew.Dean%40microsoft.co
> m%7Cb89abb6ea9f34a0916d908d76ed03783%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637099712871926073&sdata=ujcoGkSL45wodz
> A50F7NvlyHPGzt8MKnTQbUzye3JIw%3D&reserved=0
> 
> The one you looked at is built on an EC2 instance, so might be missing
> something (GDB?) needed for the guality tests.
> 
> So I don't think you're doing anything wrong, you just got unlucky and looked at
> one which skips most of the tests that are failing for you.

Thanks for your help!