Writing automated tests for the GCC driver

Writing automated tests for the GCC driver

[Giuliano Belinassi]

Hi, all.

GCC have a extensive testsuite, that is no news at all. However they are
focused on the compiler (cc1*) or in libraries, and I can't find tests
related to the GCC driver.

Are there tests to the GCC driver? If yes, is there any docs about how
to write them?

Thank you,
Giuliano.

Re: Writing automated tests for the GCC driver

[Richard Biener]

On Thu, May 21, 2020 at 11:00 PM Giuliano Belinassi via Gcc
<gcc@gcc.gnu.org> wrote:
>
> Hi, all.
>
> GCC have a extensive testsuite, that is no news at all. However they are
> focused on the compiler (cc1*) or in libraries, and I can't find tests
> related to the GCC driver.
>
> Are there tests to the GCC driver? If yes, is there any docs about how
> to write them?

I think all tests to the driver eventually exercise a compiler in the end.
One obvious I can find is gcc.dg/driver-specs.c which tests
-specs=not-a-file properly diagnoses the missing file.

So the question back would be what kind of "driver" tests do you have?
That is, what makes them not, for example, "C compiler driven by driver" tests?

Thanks,
Richard.

> Thank you,
> Giuliano.

Re: Writing automated tests for the GCC driver

[Giuliano Belinassi]

Hi,

On 05/22, Richard Biener wrote:
> On Thu, May 21, 2020 at 11:00 PM Giuliano Belinassi via Gcc
> <gcc@gcc.gnu.org> wrote:
> >
> > Hi, all.
> >
> > GCC have a extensive testsuite, that is no news at all. However they are
> > focused on the compiler (cc1*) or in libraries, and I can't find tests
> > related to the GCC driver.
> >
> > Are there tests to the GCC driver? If yes, is there any docs about how
> > to write them?
> 
> I think all tests to the driver eventually exercise a compiler in the end.
> One obvious I can find is gcc.dg/driver-specs.c which tests
> -specs=not-a-file properly diagnoses the missing file.

Yes, but that does not cover all driver features.

> 
> So the question back would be what kind of "driver" tests do you have?
> That is, what makes them not, for example, "C compiler driven by driver" tests?

GCC driver supports several modes which are required for bootstrapping
but there is no quick automated test for it. For instance.

gcc a.c b.o -o a.out
gcc a.c b.c
gcc a.S

and so on. So if you do some radical change to the GCC driver, making
sure everything is correct get somewhat painful because you have to do
a clean bootstrap and find out what is going wrong. If we had a
dedicated testsuite for that, bootstrap could be avoided for
development and only done at a last time, reducing development time.

Thank you,
Giuliano.

> 
> Thanks,
> Richard.
> 
> > Thank you,
> > Giuliano.

Re: Writing automated tests for the GCC driver

[Richard Biener]

On Mon, May 25, 2020 at 4:37 PM Giuliano Belinassi
<giuliano.belinassi@usp.br> wrote:
>
> Hi,
>
> On 05/22, Richard Biener wrote:
> > On Thu, May 21, 2020 at 11:00 PM Giuliano Belinassi via Gcc
> > <gcc@gcc.gnu.org> wrote:
> > >
> > > Hi, all.
> > >
> > > GCC have a extensive testsuite, that is no news at all. However they are
> > > focused on the compiler (cc1*) or in libraries, and I can't find tests
> > > related to the GCC driver.
> > >
> > > Are there tests to the GCC driver? If yes, is there any docs about how
> > > to write them?
> >
> > I think all tests to the driver eventually exercise a compiler in the end.
> > One obvious I can find is gcc.dg/driver-specs.c which tests
> > -specs=not-a-file properly diagnoses the missing file.
>
> Yes, but that does not cover all driver features.
>
> >
> > So the question back would be what kind of "driver" tests do you have?
> > That is, what makes them not, for example, "C compiler driven by driver" tests?
>
> GCC driver supports several modes which are required for bootstrapping
> but there is no quick automated test for it. For instance.
>
> gcc a.c b.o -o a.out
> gcc a.c b.c
> gcc a.S
>
> and so on. So if you do some radical change to the GCC driver, making
> sure everything is correct get somewhat painful because you have to do
> a clean bootstrap and find out what is going wrong. If we had a
> dedicated testsuite for that, bootstrap could be avoided for
> development and only done at a last time, reducing development time.

Hmm, indeed.  I don't think we have specific tests for all variants above.
Most of them get exercised one way or another of course but the testsuite
usually separates the compile and link steps.

I would think writing a dedicated dejagnu .exp file would be possible here,
sth in a new gcc.dg/driver/, have a driver.exp driving things and using
test files in that directory.  Sharing stuff from testsuite/lib to find the
driver and figure how to invoke it, gcc-dg.exp has gcc-dg-test
which might be enough to drive things.

> Thank you,
> Giuliano.
>
> >
> > Thanks,
> > Richard.
> >
> > > Thank you,
> > > Giuliano.

Re: Writing automated tests for the GCC driver

[Alexandre Oliva]

Hello, Giuliano,

On May 25, 2020, Giuliano Belinassi via Gcc <gcc@gcc.gnu.org> wrote:

> gcc a.c b.o -o a.out
> gcc a.c b.c
> gcc a.S

> and so on. So if you do some radical change to the GCC driver, making
> sure everything is correct get somewhat painful because you have to do
> a clean bootstrap and find out what is going wrong.

I'm about to install a patch that introduces significant changes to the
GCC driver and a large number of tests that covers some of the above.

Its main focus was to check that aux outputs got named as per the
changes introduced in the patch, but I added some baseline tests that
would cover some of the above.

Feel free to extend gcc/testsuite/gcc.misc-tests/outputs.exp to cover
other such baseline cases you may think of.

-- 
Alexandre Oliva, freedom fighter    he/him    https://FSFLA.org/blogs/lxo/
Free Software Evangelist              Stallman was right, but he's left :(
GNU Toolchain Engineer           Live long and free, and prosper ethically