Re: testsuite: introduce hostedlib effective target

[Jonathan Wakely <jwakely@redhat.com>]

On Tue, 7 Nov 2023 at 10:24, Jonathan Wakely <jwakely@redhat.com> wrote:
>
> On Tue, 7 Nov 2023 at 10:18, Jonathan Wakely <jwakely@redhat.com> wrote:
> >
> > On Tue, 7 Nov 2023 at 10:04, Alexandre Oliva <oliva@adacore.com> wrote:
> > >
> > > [adding libstdc++@]
> > >
> > > On Nov  5, 2023, Mike Stump <mikestump@comcast.net> wrote:
> > >
> > > > Ick.
> > >
> > > Indeed ;-)
> > >
> > > > I wish there were fewer changed lines and not 1 per test
> > > > case. It feels like we've painted ourselves into a corner.
> > >
> > > The libstdc++ testsuite took a different approach, detecting missing
> > > headers (and libraries?) at error pruning time, and xfailing the tests,
> > > which seems to be more in line with what you are looking for.
> > >
> > > That approach, though more expedient, seems more fragile to me, in that
> > > an actual bug that caused headers to go missing would cause tests to be
> > > silently skipped rather than fail.
> >
> > I don't think we XFAIL based on missing headers. We XFAIL based on a
> > specific #error message in certain headers.
> >
> > If a header goes missing, we'll still XFAIL.
> >
> > >
> > > I expect the set of headers, and thus of affected tests, won't by very
> > > dynamic, so it's kind of a one-shot change.
> > >
> > > Of course new tests might be added that rely on such headers, and would
> > > likely go unnoticed until someone tries them on a non-hosted libstdc++.
> >
> > Since GCC 13 you don't need to build a non-hosted libstdc++ to test
> > it, you can just add -ffreestanding to the runtestflags.
> >
> > > We could alleviate this if libstdc++ headers that are not installed on
> > > hosted systems issued a warning (conditional on some macro defined by
> > > the testsuite, say -D_GLIBCXX_WARN_HOSTED_ONLY).
> >
> > That's exactly what happens (except #error not #warning) when you
> > compile with -ffreestanding.
> >
> > >  For tests aimed
> > > exclusively at hosted libstdc++, we'd then use a dg directive that both
> > > implied this requirement, and changed the macro definition to suppress
> > > the warning.  Then anyone who added a testcase that included hosted
> > > headers without indicating its hostedlib requirement would get a fail
> > > even when testing with a hosted libstdc++.
> >
> > I don't think we need to add checks for a new macro and then use that
> > when testing, you can just test with -ffreestanding instead. This
> > already works today.
>
> Ah, reading back in the thread for  the context I missed, I see that
> you're specifically testing a --disable-hosted-libstdcxx build. In
> that case some headers really will be absent, not just
> present-with-#error. But I am still not concerned about failing to
> notice if a header goes unintentionally missing, because the libstdc++
> testsuite will still notice that.
>
> We don't prune based on "no such header" errors, so would still get
> FAILs for those tests that depend on headers which are supposed to be
> present for freestanding.

An alternative approach for the g++ testsuite would be to provide a
set of dummy headers for the non-freestanding ones, so that all the
hosted-only headers are provided by the testsuite itself, but consist
of a single line:

#error not available in freestanding

Then match on that and XFAIL. So the individual tests themselves
wouldn't need the dg-skip-if added to them, they would just
automatically XFAIL if they use a hosted-only header.

The difficulty would be where to add those dummy headers for
<iostream>, <cstdio> etc. so that they're only found when testing a
non-hosted build. Maybe libstdc++ could provide them in the build dir
for the purposes of the testsuite, but not install them?