Re: testsuite: introduce hostedlib effective target

Re: testsuite: introduce hostedlib effective target

[Alexandre Oliva]

[adding libstdc++@]

On Nov  5, 2023, Mike Stump <mikestump@comcast.net> wrote:

> Ick.

Indeed ;-)

> I wish there were fewer changed lines and not 1 per test
> case. It feels like we've painted ourselves into a corner.

The libstdc++ testsuite took a different approach, detecting missing
headers (and libraries?) at error pruning time, and xfailing the tests,
which seems to be more in line with what you are looking for.

That approach, though more expedient, seems more fragile to me, in that
an actual bug that caused headers to go missing would cause tests to be
silently skipped rather than fail.

I expect the set of headers, and thus of affected tests, won't by very
dynamic, so it's kind of a one-shot change.

Of course new tests might be added that rely on such headers, and would
likely go unnoticed until someone tries them on a non-hosted libstdc++.
We could alleviate this if libstdc++ headers that are not installed on
hosted systems issued a warning (conditional on some macro defined by
the testsuite, say -D_GLIBCXX_WARN_HOSTED_ONLY).  For tests aimed
exclusively at hosted libstdc++, we'd then use a dg directive that both
implied this requirement, and changed the macro definition to suppress
the warning.  Then anyone who added a testcase that included hosted
headers without indicating its hostedlib requirement would get a fail
even when testing with a hosted libstdc++.

-- 
Alexandre Oliva, happy hacker            https://FSFLA.org/blogs/lxo/
   Free Software Activist                   GNU Toolchain Engineer
More tolerance and less prejudice are key for inclusion and diversity
Excluding neuro-others for not behaving ""normal"" is *not* inclusive

Re: testsuite: introduce hostedlib effective target

[Jonathan Wakely]

On Tue, 7 Nov 2023 at 10:04, Alexandre Oliva <oliva@adacore.com> wrote:
>
> [adding libstdc++@]
>
> On Nov  5, 2023, Mike Stump <mikestump@comcast.net> wrote:
>
> > Ick.
>
> Indeed ;-)
>
> > I wish there were fewer changed lines and not 1 per test
> > case. It feels like we've painted ourselves into a corner.
>
> The libstdc++ testsuite took a different approach, detecting missing
> headers (and libraries?) at error pruning time, and xfailing the tests,
> which seems to be more in line with what you are looking for.
>
> That approach, though more expedient, seems more fragile to me, in that
> an actual bug that caused headers to go missing would cause tests to be
> silently skipped rather than fail.

I don't think we XFAIL based on missing headers. We XFAIL based on a
specific #error message in certain headers.

If a header goes missing, we'll still XFAIL.

>
> I expect the set of headers, and thus of affected tests, won't by very
> dynamic, so it's kind of a one-shot change.
>
> Of course new tests might be added that rely on such headers, and would
> likely go unnoticed until someone tries them on a non-hosted libstdc++.

Since GCC 13 you don't need to build a non-hosted libstdc++ to test
it, you can just add -ffreestanding to the runtestflags.

> We could alleviate this if libstdc++ headers that are not installed on
> hosted systems issued a warning (conditional on some macro defined by
> the testsuite, say -D_GLIBCXX_WARN_HOSTED_ONLY).

That's exactly what happens (except #error not #warning) when you
compile with -ffreestanding.

>  For tests aimed
> exclusively at hosted libstdc++, we'd then use a dg directive that both
> implied this requirement, and changed the macro definition to suppress
> the warning.  Then anyone who added a testcase that included hosted
> headers without indicating its hostedlib requirement would get a fail
> even when testing with a hosted libstdc++.

I don't think we need to add checks for a new macro and then use that
when testing, you can just test with -ffreestanding instead. This
already works today.

Re: testsuite: introduce hostedlib effective target

[Jonathan Wakely]

On Tue, 7 Nov 2023 at 10:18, Jonathan Wakely <jwakely@redhat.com> wrote:
>
> On Tue, 7 Nov 2023 at 10:04, Alexandre Oliva <oliva@adacore.com> wrote:
> >
> > [adding libstdc++@]
> >
> > On Nov  5, 2023, Mike Stump <mikestump@comcast.net> wrote:
> >
> > > Ick.
> >
> > Indeed ;-)
> >
> > > I wish there were fewer changed lines and not 1 per test
> > > case. It feels like we've painted ourselves into a corner.
> >
> > The libstdc++ testsuite took a different approach, detecting missing
> > headers (and libraries?) at error pruning time, and xfailing the tests,
> > which seems to be more in line with what you are looking for.
> >
> > That approach, though more expedient, seems more fragile to me, in that
> > an actual bug that caused headers to go missing would cause tests to be
> > silently skipped rather than fail.
>
> I don't think we XFAIL based on missing headers. We XFAIL based on a
> specific #error message in certain headers.
>
> If a header goes missing, we'll still XFAIL.
>
> >
> > I expect the set of headers, and thus of affected tests, won't by very
> > dynamic, so it's kind of a one-shot change.
> >
> > Of course new tests might be added that rely on such headers, and would
> > likely go unnoticed until someone tries them on a non-hosted libstdc++.
>
> Since GCC 13 you don't need to build a non-hosted libstdc++ to test
> it, you can just add -ffreestanding to the runtestflags.
>
> > We could alleviate this if libstdc++ headers that are not installed on
> > hosted systems issued a warning (conditional on some macro defined by
> > the testsuite, say -D_GLIBCXX_WARN_HOSTED_ONLY).
>
> That's exactly what happens (except #error not #warning) when you
> compile with -ffreestanding.
>
> >  For tests aimed
> > exclusively at hosted libstdc++, we'd then use a dg directive that both
> > implied this requirement, and changed the macro definition to suppress
> > the warning.  Then anyone who added a testcase that included hosted
> > headers without indicating its hostedlib requirement would get a fail
> > even when testing with a hosted libstdc++.
>
> I don't think we need to add checks for a new macro and then use that
> when testing, you can just test with -ffreestanding instead. This
> already works today.

Ah, reading back in the thread for  the context I missed, I see that
you're specifically testing a --disable-hosted-libstdcxx build. In
that case some headers really will be absent, not just
present-with-#error. But I am still not concerned about failing to
notice if a header goes unintentionally missing, because the libstdc++
testsuite will still notice that.

We don't prune based on "no such header" errors, so would still get
FAILs for those tests that depend on headers which are supposed to be
present for freestanding.

Re: testsuite: introduce hostedlib effective target

[Jonathan Wakely]

On Tue, 7 Nov 2023 at 10:24, Jonathan Wakely <jwakely@redhat.com> wrote:
>
> On Tue, 7 Nov 2023 at 10:18, Jonathan Wakely <jwakely@redhat.com> wrote:
> >
> > On Tue, 7 Nov 2023 at 10:04, Alexandre Oliva <oliva@adacore.com> wrote:
> > >
> > > [adding libstdc++@]
> > >
> > > On Nov  5, 2023, Mike Stump <mikestump@comcast.net> wrote:
> > >
> > > > Ick.
> > >
> > > Indeed ;-)
> > >
> > > > I wish there were fewer changed lines and not 1 per test
> > > > case. It feels like we've painted ourselves into a corner.
> > >
> > > The libstdc++ testsuite took a different approach, detecting missing
> > > headers (and libraries?) at error pruning time, and xfailing the tests,
> > > which seems to be more in line with what you are looking for.
> > >
> > > That approach, though more expedient, seems more fragile to me, in that
> > > an actual bug that caused headers to go missing would cause tests to be
> > > silently skipped rather than fail.
> >
> > I don't think we XFAIL based on missing headers. We XFAIL based on a
> > specific #error message in certain headers.
> >
> > If a header goes missing, we'll still XFAIL.
> >
> > >
> > > I expect the set of headers, and thus of affected tests, won't by very
> > > dynamic, so it's kind of a one-shot change.
> > >
> > > Of course new tests might be added that rely on such headers, and would
> > > likely go unnoticed until someone tries them on a non-hosted libstdc++.
> >
> > Since GCC 13 you don't need to build a non-hosted libstdc++ to test
> > it, you can just add -ffreestanding to the runtestflags.
> >
> > > We could alleviate this if libstdc++ headers that are not installed on
> > > hosted systems issued a warning (conditional on some macro defined by
> > > the testsuite, say -D_GLIBCXX_WARN_HOSTED_ONLY).
> >
> > That's exactly what happens (except #error not #warning) when you
> > compile with -ffreestanding.
> >
> > >  For tests aimed
> > > exclusively at hosted libstdc++, we'd then use a dg directive that both
> > > implied this requirement, and changed the macro definition to suppress
> > > the warning.  Then anyone who added a testcase that included hosted
> > > headers without indicating its hostedlib requirement would get a fail
> > > even when testing with a hosted libstdc++.
> >
> > I don't think we need to add checks for a new macro and then use that
> > when testing, you can just test with -ffreestanding instead. This
> > already works today.
>
> Ah, reading back in the thread for  the context I missed, I see that
> you're specifically testing a --disable-hosted-libstdcxx build. In
> that case some headers really will be absent, not just
> present-with-#error. But I am still not concerned about failing to
> notice if a header goes unintentionally missing, because the libstdc++
> testsuite will still notice that.
>
> We don't prune based on "no such header" errors, so would still get
> FAILs for those tests that depend on headers which are supposed to be
> present for freestanding.

An alternative approach for the g++ testsuite would be to provide a
set of dummy headers for the non-freestanding ones, so that all the
hosted-only headers are provided by the testsuite itself, but consist
of a single line:

#error not available in freestanding

Then match on that and XFAIL. So the individual tests themselves
wouldn't need the dg-skip-if added to them, they would just
automatically XFAIL if they use a hosted-only header.

The difficulty would be where to add those dummy headers for
<iostream>, <cstdio> etc. so that they're only found when testing a
non-hosted build. Maybe libstdc++ could provide them in the build dir
for the purposes of the testsuite, but not install them?

Re: testsuite: introduce hostedlib effective target

[Alexandre Oliva]

On Nov  7, 2023, Jonathan Wakely <jwakely@redhat.com> wrote:

> An alternative approach for the g++ testsuite would be to provide a
> set of dummy headers for the non-freestanding ones, so that all the
> hosted-only headers are provided by the testsuite itself, but consist
> of a single line:

> #error not available in freestanding

> Then match on that and XFAIL. So the individual tests themselves
> wouldn't need the dg-skip-if added to them, they would just
> automatically XFAIL if they use a hosted-only header.

*nod*.  That wouldn't cover all the circumstances, alas: there are tests
that fail in freestanding mode not because of headers, but because
-fcontracts (currently?) links libstdc++exp in, and that library is not
even built in freestanding mode.

> The difficulty would be where to add those dummy headers for
> <iostream>, <cstdio> etc. so that they're only found when testing a
> non-hosted build. Maybe libstdc++ could provide them in the build dir
> for the purposes of the testsuite, but not install them?

We run install-tree testing, so that wouldn't quite work for us.  If the
headers are in some subdirectory in the source tree, that we (or the
testsuite machinery) would just add to the -I set, that would help.

-- 
Alexandre Oliva, happy hacker            https://FSFLA.org/blogs/lxo/
   Free Software Activist                   GNU Toolchain Engineer
More tolerance and less prejudice are key for inclusion and diversity
Excluding neuro-others for not behaving ""normal"" is *not* inclusive

Re: testsuite: introduce hostedlib effective target

[Jonathan Wakely]

On Wed, 8 Nov 2023 at 15:30, Alexandre Oliva <oliva@adacore.com> wrote:
>
> On Nov  7, 2023, Jonathan Wakely <jwakely@redhat.com> wrote:
>
> > An alternative approach for the g++ testsuite would be to provide a
> > set of dummy headers for the non-freestanding ones, so that all the
> > hosted-only headers are provided by the testsuite itself, but consist
> > of a single line:
>
> > #error not available in freestanding
>
> > Then match on that and XFAIL. So the individual tests themselves
> > wouldn't need the dg-skip-if added to them, they would just
> > automatically XFAIL if they use a hosted-only header.
>
> *nod*.  That wouldn't cover all the circumstances, alas: there are tests
> that fail in freestanding mode not because of headers, but because
> -fcontracts (currently?) links libstdc++exp in, and that library is not
> even built in freestanding mode.

Hmm, yes, that seems like a bug. Either we should provide
libstdc++exp.a for freestanding builds (with a simplified contract
violation handler that doesn't print to stdout), or the front end
should not add -lstdc++exp when -ffreestanding is used (which would
require teh user, or the testsuite in your case, to provide a custom
contract violation handler), or it should be an error to use
-fcontracts and -ffreestanding together.

The libstdc++-v3/src/experimental/contract.cc file *already* supports
freestanding, we just don't actually build it for freestanding. We can
do that.

Re: testsuite: introduce hostedlib effective target

[Jonathan Wakely]

On Wed, 8 Nov 2023 at 15:48, Jonathan Wakely <jwakely@redhat.com> wrote:
>
> On Wed, 8 Nov 2023 at 15:30, Alexandre Oliva <oliva@adacore.com> wrote:
> >
> > On Nov  7, 2023, Jonathan Wakely <jwakely@redhat.com> wrote:
> >
> > > An alternative approach for the g++ testsuite would be to provide a
> > > set of dummy headers for the non-freestanding ones, so that all the
> > > hosted-only headers are provided by the testsuite itself, but consist
> > > of a single line:
> >
> > > #error not available in freestanding
> >
> > > Then match on that and XFAIL. So the individual tests themselves
> > > wouldn't need the dg-skip-if added to them, they would just
> > > automatically XFAIL if they use a hosted-only header.
> >
> > *nod*.  That wouldn't cover all the circumstances, alas: there are tests
> > that fail in freestanding mode not because of headers, but because
> > -fcontracts (currently?) links libstdc++exp in, and that library is not
> > even built in freestanding mode.
>
> Hmm, yes, that seems like a bug. Either we should provide
> libstdc++exp.a for freestanding builds (with a simplified contract
> violation handler that doesn't print to stdout), or the front end
> should not add -lstdc++exp when -ffreestanding is used (which would
> require teh user, or the testsuite in your case, to provide a custom
> contract violation handler), or it should be an error to use
> -fcontracts and -ffreestanding together.
>
> The libstdc++-v3/src/experimental/contract.cc file *already* supports
> freestanding, we just don't actually build it for freestanding. We can
> do that.

Which might be as simple as:

--- a/libstdc++-v3/src/Makefile.am
+++ b/libstdc++-v3/src/Makefile.am
@@ -34,14 +34,13 @@ backtrace_dir = libbacktrace
else
backtrace_dir =
endif
-
-experimental_dir = experimental
else
filesystem_dir =
backtrace_dir =
-experimental_dir =
endif

+experimental_dir = experimental
+
## Keep this list sync'd with acinclude.m4:GLIBCXX_CONFIGURE.
SUBDIRS = c++98 c++11 c++17 c++20 c++23 \
       $(filesystem_dir) $(backtrace_dir) $(experimental_dir)