Basic requirements for supporting OS and machine ports.

Basic requirements for supporting OS and machine ports.

[Carlos O'Donell]

TLDR; All supported OS and machine ports should have build-many-glibcs.py
and buildbot support.

Details;

We have lots of cleanup to do in glibc to facilitate future novel work and
reduce overall maintenance.

Florian and Adhemerval's discussion about pipe2 cleanups [1] have reminded 
me that we must have a discussion about basic requirements for supporting OS and
machine ports.

In order for maintainers to do effective cleanup work they need a high confidence
that they don't break things across all the supported machine and OS
combinations.

To that end the maintainers need a way to test their patches in a low latency
manner that does not involve waiting for busy machine maintainers to review.
Machine maintainer review is still critically important, but for some category
of changes it is not strictly required, particularly when it comes to mechanical
changes.

Joseph Myers' build-many-glibcs.py script has quickly become the defacto standard
for supporting compile testing of all the machine ports.

Tulio's work on a community supported buildbot master is done [2], and we need to
keep adding bots.

In light of this enabling work I suggest the community commit to the following
basic requirements for supporting OS and machine ports:

(a) If your OS and machine port has build-many-glibcs.py support then all
    maintainers will strive to ensure that their changes do not break those
    OS and machine combinations supported by the script, and will work to
    correct any such past breakage.

(b) If your OS and machine port have a buildbot slave, then all maintainers
    will strive to ensure that their changes do not introduce breakage that
    shows up on those buildbots, and will work to correct any such breakage.

    In GNU Cauldron 2016 during the buildbot discussion there was consensus
    among those present that buildbots were a real requirement, either real
    or virtual (qemu).

(c) If your OS and machine port have neither build-many-glibcs.py nor a buildbot
    then machine maintainers are *not* responsible for ensuring anything beyond
    a sensible and logically consistent change to those OS and machine ports
    that do not meet the basic requirements. This can include creating stubbed out
    files for maintainers to eventually fill in with implementation details.

At present this means that NaCL and i686 GNU/Hurd do not meet basic requirements.

Likewise those machines that don't build do not currently meet basic requirements
until their machine maintainers fix the build issues (hppa).

Comments?

-- 
Cheers,
Carlos.

[1] https://www.sourceware.org/ml/libc-alpha/2017-04/msg00237.html
[2] http://www.gnu.org/software/libc/waterfall

Re: Basic requirements for supporting OS and machine ports.

[Florian Weimer]

On 04/18/2017 03:47 PM, Carlos O'Donell wrote:
> (a) If your OS and machine port has build-many-glibcs.py support then all
>      maintainers will strive to ensure that their changes do not break those
>      OS and machine combinations supported by the script, and will work to
>      correct any such past breakage.

I don't want to see running build-many-glibcs.py as a requirement for 
posting patches.  The main reason for that is that it's rather expensive 
(either you need to be very patient, or you need a fairly big machine).

What do you think about of out-of-tree GCC ports?  Can we support 
targets which are not included in any upstream GCC version, only patches 
on top of an upstream version which already fell out of support?

Thanks,
Florian

Re: Basic requirements for supporting OS and machine ports.

[H.J. Lu]

On Tue, Apr 18, 2017 at 9:49 AM, Florian Weimer <fweimer@redhat.com> wrote:
> On 04/18/2017 03:47 PM, Carlos O'Donell wrote:
>>
>> (a) If your OS and machine port has build-many-glibcs.py support then all
>>      maintainers will strive to ensure that their changes do not break
>> those
>>      OS and machine combinations supported by the script, and will work to
>>      correct any such past breakage.
>
>
> I don't want to see running build-many-glibcs.py as a requirement for
> posting patches.  The main reason for that is that it's rather expensive
> (either you need to be very patient, or you need a fairly big machine).

I agree.

> What do you think about of out-of-tree GCC ports?  Can we support targets
> which are not included in any upstream GCC version, only patches on top of
> an upstream version which already fell out of support?
>
> Thanks,
> Florian



-- 
H.J.

Re: Basic requirements for supporting OS and machine ports.

[Adhemerval Zanella]

On 18/04/2017 13:49, Florian Weimer wrote:
> On 04/18/2017 03:47 PM, Carlos O'Donell wrote:
>> (a) If your OS and machine port has build-many-glibcs.py support then all
>>      maintainers will strive to ensure that their changes do not break those
>>      OS and machine combinations supported by the script, and will work to
>>      correct any such past breakage.
> 
> I don't want to see running build-many-glibcs.py as a requirement for posting patches.  The main reason for that is that it's rather expensive (either you need to be very patient, or you need a fairly big machine).
> 
> What do you think about of out-of-tree GCC ports?  Can we support targets which are not included in any upstream GCC version, only patches on top of an upstream version which already fell out of support?
> 

My view is rather we should aim to have at least a way to proper build
a toolchain to run the basic build tests in the absence of a real or
simulated system.  I also do not think to require use build-many-glibcs.py
as requirements if you have access to a native system where you can get
meaningful check results.  As Carlos pointed out in the (c) requirement,
if a developer can not even validate a build break with current tooling
(if any), then it is up to system maintainer to keep things on track.

Also if the idea is the support out-of-tree GCC ports, we need to solve
the problem of out of date support. As we increase the minimum required
binutils and gcc version for future releases, should it prevent to do so?
Or will we have different requirements for different architectures (as we
have now for kernel support)?  I tend to view out-of-tree ports to be
cumbersome in long-term development and to generate more issue as its
usage is not actively pursuit.  

Re: Basic requirements for supporting OS and machine ports.

[Torvald Riegel]

On Tue, 2017-04-18 at 09:47 -0400, Carlos O'Donell wrote:
> (c) If your OS and machine port have neither build-many-glibcs.py nor a buildbot
>     then machine maintainers are *not* responsible for ensuring anything beyond

I suppose this should be all maintainers, not just machine maintainers.

Re: Basic requirements for supporting OS and machine ports.

[Carlos O'Donell]

On 04/18/2017 12:49 PM, Florian Weimer wrote:
> On 04/18/2017 03:47 PM, Carlos O'Donell wrote:
>> (a) If your OS and machine port has build-many-glibcs.py support
>> then all maintainers will strive to ensure that their changes do
>> not break those OS and machine combinations supported by the
>> script, and will work to correct any such past breakage.
> 
> I don't want to see running build-many-glibcs.py as a requirement for
> posting patches.  The main reason for that is that it's rather
> expensive (either you need to be very patient, or you need a fairly
> big machine).

I agree that we should not require running build-many-glibcs.py per
posted patch. It takes too long to run.

I consciously worded the text of (a) so as not to specifically say you
had to run build-many-glibcs.py in order to post a patch, though doing
so on every post is the safest way to meet the requirements of (a).

My intent is to make it the responsibility of the maintainers to avoid
breakage of the maintained machines and OS combinations and own the
responsibility for fixing it if their patch broke the build (using the
json configuration to work out exactly what component combination is
failing).

> What do you think about of out-of-tree GCC ports?  Can we support
> targets which are not included in any upstream GCC version, only
> patches on top of an upstream version which already fell out of
> support?

If your port doesn't have a working upstream GCC then I would argue
it falls into (c) and no maintainer need do anything special
beyond a sensible and logical change. The machine or OS maintainer is
responsible for bringing GCC into a buildable state.

So yes we can "support" out-of-tree GCC ports, but the support is
only as outlined in (c).

Does that answer your question?

-- 
Cheers,
Carlos.

Re: Basic requirements for supporting OS and machine ports.

[Florian Weimer]

On 04/18/2017 10:14 PM, Carlos O'Donell wrote:
> Does that answer your question?

It does, thanks.

Florian

Re: Basic requirements for supporting OS and machine ports.

[Carlos O'Donell]

On 04/18/2017 01:33 PM, Adhemerval Zanella wrote:
> 
> 
> On 18/04/2017 13:49, Florian Weimer wrote:
>> On 04/18/2017 03:47 PM, Carlos O'Donell wrote:
>>> (a) If your OS and machine port has build-many-glibcs.py support
>>> then all maintainers will strive to ensure that their changes do
>>> not break those OS and machine combinations supported by the
>>> script, and will work to correct any such past breakage.
>> 
>> I don't want to see running build-many-glibcs.py as a requirement
>> for posting patches.  The main reason for that is that it's rather
>> expensive (either you need to be very patient, or you need a fairly
>> big machine).
>> 
>> What do you think about of out-of-tree GCC ports?  Can we support
>> targets which are not included in any upstream GCC version, only
>> patches on top of an upstream version which already fell out of
>> support?
>> 
> 
> My view is rather we should aim to have at least a way to proper
> build a toolchain to run the basic build tests in the absence of a
> real or simulated system.  I also do not think to require use
> build-many-glibcs.py as requirements if you have access to a native
> system where you can get meaningful check results.  As Carlos pointed
> out in the (c) requirement, if a developer can not even validate a
> build break with current tooling (if any), then it is up to system
> maintainer to keep things on track.

I don't want to require build-many-glibcs.py be used as the method
for testing your patches, but I do want to require that for a machine
and OS to be supported, that you must have build-many-glibcs.py working,
and that way we can hold maintainers responsible for fixing issues it
detects.

I write "strive to ensure their changes do not break those..." which
can include just doing a native build, and not using build-many-glibcs.py.

> Also if the idea is the support out-of-tree GCC ports, we need to
> solve the problem of out of date support. As we increase the minimum
> required binutils and gcc version for future releases, should it
> prevent to do so? Or will we have different requirements for
> different architectures (as we have now for kernel support)?  I tend
> to view out-of-tree ports to be cumbersome in long-term development
> and to generate more issue as its usage is not actively pursuit.

An out-of-tree GCC port gets (c)-level support.

You can't have a working build-many-glibcs.py without a checked in
working version of GCC from upstream.

I will not support extending build-many-glibcs.py to apply patches
to the checked out trees. Or to put it another way: We will not be
duplicating crosstool-NG in glibc :-)

-- 
Cheers,
Carlos.

Re: Basic requirements for supporting OS and machine ports.

[Carlos O'Donell]

On 04/18/2017 01:15 PM, H.J. Lu wrote:
> On Tue, Apr 18, 2017 at 9:49 AM, Florian Weimer <fweimer@redhat.com> wrote:
>> On 04/18/2017 03:47 PM, Carlos O'Donell wrote:
>>>
>>> (a) If your OS and machine port has build-many-glibcs.py support then all
>>>      maintainers will strive to ensure that their changes do not break
>>> those
>>>      OS and machine combinations supported by the script, and will work to
>>>      correct any such past breakage.
>>
>>
>> I don't want to see running build-many-glibcs.py as a requirement for
>> posting patches.  The main reason for that is that it's rather expensive
>> (either you need to be very patient, or you need a fairly big machine).
> 
> I agree.

Thanks for the feedback.

Hopefully my comments downthread make it clear that I don't intend for
anyone to run build-many-glibcs.py for each commit, but that if it breaks
that we are accountable to determining why it broke because it's a minimum
requirement for supporting an OS or a machine port.

To give an example: You checkin a patch you only tested on ppc64le and
AArch64 and it breaks build-many-glibcs.py for s390x. We would expect
that the committer of that change is on the hook for figuring out why
s390x broke and fixing it or reverting the change.

-- 
Cheers,
Carlos.

Re: Basic requirements for supporting OS and machine ports.

[Carlos O'Donell]

On 04/18/2017 04:03 PM, Torvald Riegel wrote:
> On Tue, 2017-04-18 at 09:47 -0400, Carlos O'Donell wrote:
>> (c) If your OS and machine port have neither build-many-glibcs.py nor a buildbot
>>     then machine maintainers are *not* responsible for ensuring anything beyond
> 
> I suppose this should be all maintainers, not just machine maintainers.

Yes, that's right. It should read:

(c) If your OS and machine port have neither build-many-glibcs.py nor a buildbot
    then maintainers are *not* responsible for ensuring anything beyond
    a sensible and logically consistent change to those OS and machine ports
    that do not meet the basic requirements. This can include creating stubbed out
    files for machine maintainers to eventually fill in with implementation details.

-- 
Cheers,
Carlos.

Re: Basic requirements for supporting OS and machine ports.

[Joseph Myers]

On Tue, 18 Apr 2017, Carlos O'Donell wrote:

> (a) If your OS and machine port has build-many-glibcs.py support then all
>     maintainers will strive to ensure that their changes do not break those
>     OS and machine combinations supported by the script, and will work to
>     correct any such past breakage.

Note that where there are pre-existing testsuite failures / testsuite 
build failures (ia64; microblaze; also sh when using GCC trunk, because of 
GCC bugs 78459, 78460; hppa before the build broke), it won't be 
immediately visible from a build-many-glibcs.py run that *more* test 
failures appeared (although examination of the logs will show exactly what 
failures there were, unless the testsuite failed to build part way 
through).

> (b) If your OS and machine port have a buildbot slave, then all maintainers
>     will strive to ensure that their changes do not introduce breakage that
>     shows up on those buildbots, and will work to correct any such breakage.

Buildbots don't provide a way for people to test their patches on those 
buildbots.  Also, I'm not aware of anyone monitoring the bots for 
regressions, they don't generally have a clean state of expected results 
at present, and they still have the mysterious "make[1]: *** [tests] Error 
1" at the end of the test run with no previous errors reported that no-one 
has tracked down (I've suggested a bot owner should set it up to run make 
with appropriate --debug options and study the output to try to find more 
information about the cause).  So I don't think buildbot is ready for any 
such expectations on maintainers.

(We also don't have a way for buildbots to send test results to 
libc-testresults, but that would be a peripheral issue if someone were 
monitoring the results for regressions and they actually reliably produced 
results for each testsuite run instead of mysterious errors.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Basic requirements for supporting OS and machine ports.

[Tulio Magno Quites Machado Filho]

Joseph Myers <joseph@codesourcery.com> writes:

> On Tue, 18 Apr 2017, Carlos O'Donell wrote:
>
>> (b) If your OS and machine port have a buildbot slave, then all maintainers
>>     will strive to ensure that their changes do not introduce breakage that
>>     shows up on those buildbots, and will work to correct any such breakage.
>
> Buildbots don't provide a way for people to test their patches on those 
> buildbots.

This is disabled right now, but we can enable it if we think that's
important:
http://docs.buildbot.net/0.8.9/manual/cfg-schedulers.html#try-schedulers

We have to confirm with BuildSlave owners if they agree with this feature.

However, I think having public logs of the failures already help to find
regressions.

> Also, I'm not aware of anyone monitoring the bots for 
> regressions,

I am doing that on POWER.  ;-)

> (We also don't have a way for buildbots to send test results to 
> libc-testresults, but that would be a peripheral issue if someone were 
> monitoring the results for regressions and they actually reliably produced 
> results for each testsuite run instead of mysterious errors.)

We can also configure that:
http://docs.buildbot.net/0.8.9/manual/cfg-statustargets.html#mailnotifier

-- 
Tulio Magno

Re: Basic requirements for supporting OS and machine ports.

[Joseph Myers]

On Wed, 19 Apr 2017, Tulio Magno Quites Machado Filho wrote:

> However, I think having public logs of the failures already help to find
> regressions.

But do we have logs that show the contents of the .out files for failed 
tests rather than just which tests failed?

> > (We also don't have a way for buildbots to send test results to 
> > libc-testresults, but that would be a peripheral issue if someone were 
> > monitoring the results for regressions and they actually reliably produced 
> > results for each testsuite run instead of mysterious errors.)
> 
> We can also configure that:
> http://docs.buildbot.net/0.8.9/manual/cfg-statustargets.html#mailnotifier

Well, we need a glibc-specific script to collect whatever information we 
think is relevant from the results of a test run for sending to 
libc-testresults (for use for both manual and buildbot builds).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Basic requirements for supporting OS and machine ports.

[Tulio Magno Quites Machado Filho]

Joseph Myers <joseph@codesourcery.com> writes:

> On Wed, 19 Apr 2017, Tulio Magno Quites Machado Filho wrote:
>
>> However, I think having public logs of the failures already help to find
>> regressions.

Not yet, but this is also possible.  We can create a tarball with all of the
files we want and upload it to the master:
http://docs.buildbot.net/0.8.9/manual/cfg-buildsteps.html#step-FileUpload

> But do we have logs that show the contents of the .out files for failed 
> tests rather than just which tests failed?
>
>> > (We also don't have a way for buildbots to send test results to 
>> > libc-testresults, but that would be a peripheral issue if someone were 
>> > monitoring the results for regressions and they actually reliably produced 
>> > results for each testsuite run instead of mysterious errors.)
>> 
>> We can also configure that:
>> http://docs.buildbot.net/0.8.9/manual/cfg-statustargets.html#mailnotifier
>
> Well, we need a glibc-specific script to collect whatever information we 
> think is relevant from the results of a test run for sending to 
> libc-testresults (for use for both manual and buildbot builds).

Usually BuildBot would notify a build failed and send a link to the place
where you could get more information, including log files.  So it wouldn't
be necessary to send all of that data to the mailing list.

But it seems you want a much broader solution.  Could you elaborate what you'd
like to see in those messages?

-- 
Tulio Magno

Re: Basic requirements for supporting OS and machine ports.

[Joseph Myers]

On Thu, 20 Apr 2017, Tulio Magno Quites Machado Filho wrote:

> > Well, we need a glibc-specific script to collect whatever information we 
> > think is relevant from the results of a test run for sending to 
> > libc-testresults (for use for both manual and buildbot builds).
> 
> Usually BuildBot would notify a build failed and send a link to the place
> where you could get more information, including log files.  So it wouldn't
> be necessary to send all of that data to the mailing list.

I think buildbot logs have disappeared at least once when the bot moved 
from one system to another.  That's a clear reason to send the key 
information to the mailing list, so it stays available.

> But it seems you want a much broader solution.  Could you elaborate what you'd
> like to see in those messages?

Key information about the compile-time configuration of glibc, such as 
configure options, glibc host, GCC and binutils versions, glibc commit 
being tested.

Key information about the test configuration, such as the run-built-tests 
value and the kernel version on the host used for testing if 
run-build-tests is yes.

A list of the tests with unexpected results (anything that's not PASS or 
XFAIL) and the numbers of each result, as at the end of the output of 
"make check".

I think that's the basic minimum.  It would be expected that when doing 
pre-release testing for an architecture you send the results to 
libc-testresults and include a link on the per-release wiki page.  Bots, 
whether or not linked to the buildbot master, would also send results to 
libc-testresults.

It should be possible for people to grep a libc-testresults mailbox to see 
"is this test failing for other people, and if so, in what 
configurations?".  (Which is a use I make of gcc-testresults from time to 
time.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Basic requirements for supporting OS and machine ports.

[Joseph Myers]

On Fri, 21 Apr 2017, Carlos O'Donell wrote:

> So it sounds like you don't object to the basic requirements, just that
> supporting the basic requirements needs more infrastructure work.
> 
> Could we then just agree on *one* basic requirement? That the machine
> and OS have build-many-glibcs.py support?

I don't object to a requirement of the form: new (machine, OS, ABI) 
configurations are expected to have such support contributed when the 
configuration support is added, and without such support there is no 
expectation that developers will help fix any build or build-time-test 
problems their patches may cause on the configurations lacking such 
support.

This does not impose an ordering in which people must get support for 
their (machine, OS, ABI) into various toolchain / kernel components, but 
if e.g. the glibc support goes in before the kernel support, the build 
with build-many-glibcs.py won't actually work and so there is no 
expectation of people fixing build problems until all the relevant support 
is upstream.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Basic requirements for supporting OS and machine ports.

[Florian Weimer]

* Joseph Myers:

> Well, we need a glibc-specific script to collect whatever information we 
> think is relevant from the results of a test run for sending to 
> libc-testresults (for use for both manual and buildbot builds).

Do you agree that the build bots are too difficult to use right now?

I found it very hard to interpret the results, but my difficulties
baffle others.

Re: Basic requirements for supporting OS and machine ports.

[Carlos O'Donell]

On 04/20/2017 12:11 PM, Joseph Myers wrote:
> I think that's the basic minimum.  It would be expected that when doing 
> pre-release testing for an architecture you send the results to 
> libc-testresults and include a link on the per-release wiki page.  Bots, 
> whether or not linked to the buildbot master, would also send results to 
> libc-testresults.

So it sounds like you don't object to the basic requirements, just that
supporting the basic requirements needs more infrastructure work.

Could we then just agree on *one* basic requirement? That the machine
and OS have build-many-glibcs.py support?

We could defer the buildbot support until we get more pieces in place.

-- 
Cheers,
Carlos.

Re: Basic requirements for supporting OS and machine ports.

[Joseph Myers]

On Fri, 21 Apr 2017, Florian Weimer wrote:

> * Joseph Myers:
> 
> > Well, we need a glibc-specific script to collect whatever information we 
> > think is relevant from the results of a test run for sending to 
> > libc-testresults (for use for both manual and buildbot builds).
> 
> Do you agree that the build bots are too difficult to use right now?
> 
> I found it very hard to interpret the results, but my difficulties
> baffle others.

I think the mysterious "make[1]: *** [tests] Error 1" at the end of a 
large proportion of test runs, with no previous errors reported, is pretty 
fatal to simple use of the bots to see what tests are failing.  Someone 
running a bot showing that issue needs to debug and fix it.

Even with that fixed, it would be a lot better if the bot showed failures 
only when there were regressions - that is, if any issues causing known 
test failures were fixed, or the tests in question made more robust if 
they aren't actually showing glibc bugs, or XFAILed if applicable.

-- 
Joseph S. Myers
joseph@codesourcery.com