GCC Buildbot Update

GCC Buildbot Update

[Paulo Matos]

Hello,

Apologies for the delay on the update. It was my plan to do an update on
a monthly basis but it slipped by a couple of weeks.

The current status is:

*Workers:*

- x86_64

2 workers from CF (gcc16 and gcc20) up and running;
1 worker from my farm (jupiter-F26) up and running;

2 broken CF (gcc75 and gcc76) - the reason for the brokenness is that
the machines work well but all outgoing ports except the git port is
open (9418 if not mistaken). This means that not only we cannot svn co
gcc but we can't connect a worker to the master through port 9918. I
have contacted the cf admin but the reply was that nothing can be done
as they don't really own the machine. They seemed to have relayed the
request to the machine owners.

- aarch64

I got an email suggesting I add some aarch64 workers so I did:
4 workers from CF (gcc113, gcc114, gcc115 and gcc116);

*Builds:*

As before we have the full build and the incremental build. Both enabled
for x86_64 and aarch64, except they are currently failing for aarch64
(more on that later).

The full build is triggered on Daily bump commit and the incremental
build is triggered for each commit.

The problem with this setup is that the incremental builder takes too
long to run the tests. Around 1h30m on CF machines for x86_64.

Segher Boessenkool sent me a patch to disable guality and prettyprinters
which coupled with --disable-gomp at configure time was supposed to make
things much faster. I have added this as the Fast builder, except this
is failing during the test runs:
unable to alloc 389376 bytes
/bin/bash: line 21: 32472 Aborted                 `if [ -f
${srcdir}/../dejagnu/runtest ] ; then echo ${srcdir}/../dejagnu/runtest
; else echo runtest; fi` --tool gcc
/bin/bash: fork: Cannot allocate memory
make[3]: [check-parallel-gcc] Error 254 (ignored)
make[3]: execvp: /bin/bash: Cannot allocate memory
make[3]: [check-parallel-gcc_1] Error 127 (ignored)
make[3]: execvp: /bin/bash: Cannot allocate memory
make[3]: [check-parallel-gcc_1] Error 127 (ignored)
make[3]: execvp: /bin/bash: Cannot allocate memory
make[3]: *** [check-parallel-gcc_1] Error 127


However, something interesting is happening here since the munin
interface for gcc16 doesn't show the machine running out of memory:
https://cfarm.tetaneutral.net/munin/gccfarm/gcc16/memory.html
(something confirmed by the cf admins)

The aarch64 build is failing as mentioned earlier. If you check the logs:
https://gcc-buildbot.linki.tools/#/builders/5/builds/10
the problem seems to be the assembler issuing:
Assembler messages:
Error: unknown architecture `armv8.1-a'
Error: unrecognized option -march=armv8.1-a


If I go to the machines and check the versions I get:
pmatos@gcc115:~/gcc-8-20171203_BUILD$ as --version
GNU assembler (GNU Binutils for Ubuntu) 2.24
Copyright 2013 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `aarch64-linux-gnu'.

pmatos@gcc115:~/gcc-8-20171203_BUILD$ gcc --version
gcc (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

pmatos@gcc115:~/gcc-8-20171203_BUILD$ as -march=armv8.1-a
Assembler messages:
Error: unknown architecture `armv8.1-a'

Error: unrecognized option -march=armv8.1-a

However, if I run the a compiler build manually with just:

$ configure --disable-multilib
$ nice -n 19 make -j4 all

This compiles just fine. So I am at the moment attempting to investigate
what might cause the difference between what buildbot does and what I do
through ssh.

*Reporters:*

There is a single reporter which is a irc bot currently silent.

*Regression analysis:*

This is one of the most important issues to tackle and I have a solution
in a branch regression-testing :
https://github.com/LinkiTools/gcc-buildbot/tree/regression-testing

using jamais-vu from David Malcolm to analyze the regressions.
It needs some more testing and I should be able to get it working still
this year.

*LNT:*

I had mentioned I wanted to setup an interface which would allow for
easy visibility of test failures, time taken to build/test, etc.
Initially I thought a stack of influx+grafana would be a good idea, but
was pointed out to using LNT as presented by James Greenhalgh in the GNU
Cauldron. I have setup LNT (soon to be available under
https://gcc-lnt.linki.tools) and contacted James to learn more about the
setup. As it turns out James is just using it for benchmarking results
and out-of-the-box only seems to support the LLVM testing infrastructure
so getting GCC results in there might take a bit more of scripting and
plumbing.

I will probably take the same route and set it up first for the
benchmarking results and then try to get the gcc test results there as well.

*TODO:*

So on my todo list for the next iteration I have:

- analysis of out-of-memory issues in CF for Fast builders;
- analysis of aarch64 build failure;
- merge regression testing verification branch into master and deploy
into production;
  - this should then trigger the irc bot reporter for regressions;
- open up LNT for benchmarking and add benchmarking job for x64_64 using
csibe (as an initial proof of concept);

If you have any wishes, questions, want to offer some fast machines to
have workers on or if you know what's wrong with the Fast builder or the
aarch64 machines, please let me know.

I hope to send another update in about a months time.

Kind regards,
-- 
Paulo Matos

Re: GCC Buildbot Update

[David Malcolm]

On Thu, 2017-12-14 at 09:56 +0100, Paulo Matos wrote:
> Hello,
> 
> Apologies for the delay on the update. It was my plan to do an update
> on
> a monthly basis but it slipped by a couple of weeks.

Thanks for working on this.

> The current status is:
> 
> *Workers:*

[...snip...]

> *Builds:*

[...snip...]

Looking at some of the red blobs in e.g. the grid view there seem to be
a few failures in the initial "update gcc trunk repo" step of the form:

svn: Working copy '.' locked
svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for
details)

https://gcc-lnt.linki.tools/#/builders/3/builds/388/steps/0/logs/stdio

Is there a bug-tracking location for the buildbot?
Presumably:
  https://github.com/LinkiTools/gcc-buildbot/issues
?

*Reporters:*
> 
> There is a single reporter which is a irc bot currently silent.
> 
> *Regression analysis:*
> 
> This is one of the most important issues to tackle and I have a
> solution
> in a branch regression-testing :
> https://github.com/LinkiTools/gcc-buildbot/tree/regression-testing
> 
> using jamais-vu from David Malcolm to analyze the regressions.
> It needs some more testing and I should be able to get it working
> still
> this year.

I actually found a serious bug in jamais-vu yesterday - it got confused
by  multiple .sum lines for the same source line e.g. from multiple
"dg-" directives that all specify a particular line).  For example,
when testing one of my patches, of the 3 tests reporting as
  "c-c++-common/pr83059.c  -std=c++11  (test for warnings, line 7)"
one of the 3 PASS results became a FAIL.  jv correctly reported that
new FAILs had occurred, but wouldn't identify them, and mistakenly
reported that new PASSes has occurred also.

I've fixed that now; to do so I've done some refactoring and added a
testsuite.

It looks like you're capturing the textual output from "jv compare" and
using the exit code.  Would you prefer to import "jv" as a python
module and use some kind of API?  Or a different output format?

If you file pull request(s) for the changes you've made in your copy of
jamais-vu, I can take at look at merging them.

[...]

> I hope to send another update in about a months time.
> 
> Kind regards,

Thanks again for your work on this
Dave

Re: GCC Buildbot Update

[Christophe Lyon]

On 14 December 2017 at 09:56, Paulo Matos <pmatos@linki.tools> wrote:
> Hello,
>
> Apologies for the delay on the update. It was my plan to do an update on
> a monthly basis but it slipped by a couple of weeks.
>
Hi,

Thanks for the update!


> The current status is:
>
> *Workers:*
>
> - x86_64
>
> 2 workers from CF (gcc16 and gcc20) up and running;
> 1 worker from my farm (jupiter-F26) up and running;
>
> 2 broken CF (gcc75 and gcc76) - the reason for the brokenness is that
> the machines work well but all outgoing ports except the git port is
> open (9418 if not mistaken). This means that not only we cannot svn co
> gcc but we can't connect a worker to the master through port 9918. I
> have contacted the cf admin but the reply was that nothing can be done
> as they don't really own the machine. They seemed to have relayed the
> request to the machine owners.
>
> - aarch64
>
> I got an email suggesting I add some aarch64 workers so I did:
> 4 workers from CF (gcc113, gcc114, gcc115 and gcc116);
>
Great, I thought the CF machines were reserved for developpers.
Good news you could add builders on them.

> *Builds:*
>
> As before we have the full build and the incremental build. Both enabled
> for x86_64 and aarch64, except they are currently failing for aarch64
> (more on that later).
>
> The full build is triggered on Daily bump commit and the incremental
> build is triggered for each commit.
>
> The problem with this setup is that the incremental builder takes too
> long to run the tests. Around 1h30m on CF machines for x86_64.
>
> Segher Boessenkool sent me a patch to disable guality and prettyprinters
> which coupled with --disable-gomp at configure time was supposed to make
> things much faster. I have added this as the Fast builder, except this
> is failing during the test runs:
> unable to alloc 389376 bytes
> /bin/bash: line 21: 32472 Aborted                 `if [ -f
> ${srcdir}/../dejagnu/runtest ] ; then echo ${srcdir}/../dejagnu/runtest
> ; else echo runtest; fi` --tool gcc
> /bin/bash: fork: Cannot allocate memory
> make[3]: [check-parallel-gcc] Error 254 (ignored)
> make[3]: execvp: /bin/bash: Cannot allocate memory
> make[3]: [check-parallel-gcc_1] Error 127 (ignored)
> make[3]: execvp: /bin/bash: Cannot allocate memory
> make[3]: [check-parallel-gcc_1] Error 127 (ignored)
> make[3]: execvp: /bin/bash: Cannot allocate memory
> make[3]: *** [check-parallel-gcc_1] Error 127
>
>
> However, something interesting is happening here since the munin
> interface for gcc16 doesn't show the machine running out of memory:
> https://cfarm.tetaneutral.net/munin/gccfarm/gcc16/memory.html
> (something confirmed by the cf admins)
>
> The aarch64 build is failing as mentioned earlier. If you check the logs:
> https://gcc-buildbot.linki.tools/#/builders/5/builds/10
> the problem seems to be the assembler issuing:
> Assembler messages:
> Error: unknown architecture `armv8.1-a'
> Error: unrecognized option -march=armv8.1-a
>
>
> If I go to the machines and check the versions I get:
> pmatos@gcc115:~/gcc-8-20171203_BUILD$ as --version
> GNU assembler (GNU Binutils for Ubuntu) 2.24
> Copyright 2013 Free Software Foundation, Inc.
> This program is free software; you may redistribute it under the terms of
> the GNU General Public License version 3 or later.
> This program has absolutely no warranty.
> This assembler was configured for a target of `aarch64-linux-gnu'.
>
> pmatos@gcc115:~/gcc-8-20171203_BUILD$ gcc --version
> gcc (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3) 4.8.4
> Copyright (C) 2013 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> pmatos@gcc115:~/gcc-8-20171203_BUILD$ as -march=armv8.1-a
> Assembler messages:
> Error: unknown architecture `armv8.1-a'
>
> Error: unrecognized option -march=armv8.1-a
>
> However, if I run the a compiler build manually with just:
>
> $ configure --disable-multilib
> $ nice -n 19 make -j4 all
>
> This compiles just fine. So I am at the moment attempting to investigate
> what might cause the difference between what buildbot does and what I do
> through ssh.
>
I suspect you are hitting a bug introduced recently, and fixed by:
https://gcc.gnu.org/ml/gcc-patches/2017-12/msg00434.html

> *Reporters:*
>
> There is a single reporter which is a irc bot currently silent.
>
> *Regression analysis:*
>
> This is one of the most important issues to tackle and I have a solution
> in a branch regression-testing :
> https://github.com/LinkiTools/gcc-buildbot/tree/regression-testing
>
> using jamais-vu from David Malcolm to analyze the regressions.
> It needs some more testing and I should be able to get it working still
> this year.
>
Great

> *LNT:*
>
> I had mentioned I wanted to setup an interface which would allow for
> easy visibility of test failures, time taken to build/test, etc.
> Initially I thought a stack of influx+grafana would be a good idea, but
> was pointed out to using LNT as presented by James Greenhalgh in the GNU
> Cauldron. I have setup LNT (soon to be available under
> https://gcc-lnt.linki.tools) and contacted James to learn more about the
> setup. As it turns out James is just using it for benchmarking results
> and out-of-the-box only seems to support the LLVM testing infrastructure
> so getting GCC results in there might take a bit more of scripting and
> plumbing.
>
> I will probably take the same route and set it up first for the
> benchmarking results and then try to get the gcc test results there as well.
>
> *TODO:*
>
> So on my todo list for the next iteration I have:
>
> - analysis of out-of-memory issues in CF for Fast builders;
> - analysis of aarch64 build failure;
> - merge regression testing verification branch into master and deploy
> into production;
>   - this should then trigger the irc bot reporter for regressions;
> - open up LNT for benchmarking and add benchmarking job for x64_64 using
> csibe (as an initial proof of concept);
>
> If you have any wishes, questions, want to offer some fast machines to
> have workers on or if you know what's wrong with the Fast builder or the
> aarch64 machines, please let me know.
>
> I hope to send another update in about a months time.
>
Thanks,

Christophe

> Kind regards,
> --
> Paulo Matos

Re: GCC Buildbot Update

[Markus Trippelsdorf]

On 2017.12.14 at 21:32 +0100, Christophe Lyon wrote:
> On 14 December 2017 at 09:56, Paulo Matos <pmatos@linki.tools> wrote:
> > I got an email suggesting I add some aarch64 workers so I did:
> > 4 workers from CF (gcc113, gcc114, gcc115 and gcc116);
> >
> Great, I thought the CF machines were reserved for developpers.
> Good news you could add builders on them.

I don't think this is good news at all. 

Once a buildbot runs on a CF machine it immediately becomes impossible
to do any meaningful measurement on that machine. That is mainly because
of the random I/O (untar, rm -fr, etc.) of the bot. As a result variance
goes to the roof and all measurements drown in noise.

So it would be good if there was a strict separation of machines used
for bots and machines used by humans. In other words bots should only
run on dedicated machines.

-- 
Markus

Re: GCC Buildbot Update

[Paulo Matos]

On 14/12/17 12:39, David Malcolm wrote:
> 
> Looking at some of the red blobs in e.g. the grid view there seem to be
> a few failures in the initial "update gcc trunk repo" step of the form:
> 
> svn: Working copy '.' locked
> svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for
> details)
> 

Yes, that's a big annoyance and a reason I have thought about moving to
using the git mirror, however that would probably bring other issues so
I am holding off. I need to add a reporter so that if it fails I am
notified by email and mobile phone.

This happens when there's a timeout from a server _during_ a
checkout/update (the svn repo unfortunately times out way too often). I
thought about doing an svn cleanup before each checkout but read it's
not good practice. If you have any suggestions on this please let me know.

> https://gcc-lnt.linki.tools/#/builders/3/builds/388/steps/0/logs/stdio
> 

Apologies, https://gcc-lnt.linki.tools is currently incorrectly
forwarding you to https://gcc-buildbot.linki.tools. I meant to have it
return an error until I open that up.

> Is there a bug-tracking location for the buildbot?
> Presumably:
>   https://github.com/LinkiTools/gcc-buildbot/issues
> ?
> 

That's correct.

> I actually found a serious bug in jamais-vu yesterday - it got confused
> by  multiple .sum lines for the same source line e.g. from multiple
> "dg-" directives that all specify a particular line).  For example,
> when testing one of my patches, of the 3 tests reporting as
>   "c-c++-common/pr83059.c  -std=c++11  (test for warnings, line 7)"
> one of the 3 PASS results became a FAIL.  jv correctly reported that
> new FAILs had occurred, but wouldn't identify them, and mistakenly
> reported that new PASSes has occurred also.
> 
> I've fixed that now; to do so I've done some refactoring and added a
> testsuite.
>

Perfect, thank you very much for this work.

> It looks like you're capturing the textual output from "jv compare" and
> using the exit code.  Would you prefer to import "jv" as a python
> module and use some kind of API?  Or a different output format?
> 

Well, I am using a fork of it which I converted to Python3. Would you be
open to convert yours to Python3? The reason I am doing this is because
all other Python software I have and the buildbot use Python3.

I would also prefer to have some json format or something but when I
looked at it, the software was just printing to stdout and I didn't want
to spend too much time implementing it, so I thought parsing the output
was just easier.

> If you file pull request(s) for the changes you've made in your copy of
> jamais-vu, I can take at look at merging them.
>

Happy to do so...
Will merge your changes into my fork first then.

Kind regards,
-- 
Paulo Matos

Re: GCC Buildbot Update

[Paulo Matos]

On 14/12/17 21:32, Christophe Lyon wrote:
> Great, I thought the CF machines were reserved for developpers.
> Good news you could add builders on them.
> 

Oh. I have seen similar things happening on CF machines so I thought it
was not a problem. I have never specifically asked for permission.

>> pmatos@gcc115:~/gcc-8-20171203_BUILD$ as -march=armv8.1-a
>> Assembler messages:
>> Error: unknown architecture `armv8.1-a'
>>
>> Error: unrecognized option -march=armv8.1-a
>>
>> However, if I run the a compiler build manually with just:
>>
>> $ configure --disable-multilib
>> $ nice -n 19 make -j4 all
>>
>> This compiles just fine. So I am at the moment attempting to investigate
>> what might cause the difference between what buildbot does and what I do
>> through ssh.
>>
> I suspect you are hitting a bug introduced recently, and fixed by:
> https://gcc.gnu.org/ml/gcc-patches/2017-12/msg00434.html
> 

Wow, that's really useful. Thanks for letting me know.

-- 
Paulo Matos

Re: GCC Buildbot Update

[Paulo Matos]

On 15/12/17 08:42, Markus Trippelsdorf wrote:
> 
> I don't think this is good news at all. 
> 

As I pointed out in a reply to Chris, I haven't seeked permission but I
am pretty sure something similar runs in the CF machines from other
projects.

The downside is that if we can't use the CF, I have no extra machines to
run the buildbot on.

> Once a buildbot runs on a CF machine it immediately becomes impossible
> to do any meaningful measurement on that machine. That is mainly because
> of the random I/O (untar, rm -fr, etc.) of the bot. As a result variance
> goes to the roof and all measurements drown in noise.
> 
> So it would be good if there was a strict separation of machines used
> for bots and machines used by humans. In other words bots should only
> run on dedicated machines.
> 

I understand your concern though. Do you know who this issue could be
raised with? FSF?

-- 
Paulo Matos

Re: GCC Buildbot Update

[Christophe Lyon]

On 15 December 2017 at 10:19, Paulo Matos <pmatos@linki.tools> wrote:
>
>
> On 14/12/17 21:32, Christophe Lyon wrote:
>> Great, I thought the CF machines were reserved for developpers.
>> Good news you could add builders on them.
>>
>
> Oh. I have seen similar things happening on CF machines so I thought it
> was not a problem. I have never specifically asked for permission.
>
>>> pmatos@gcc115:~/gcc-8-20171203_BUILD$ as -march=armv8.1-a
>>> Assembler messages:
>>> Error: unknown architecture `armv8.1-a'
>>>
>>> Error: unrecognized option -march=armv8.1-a
>>>
>>> However, if I run the a compiler build manually with just:
>>>
>>> $ configure --disable-multilib
>>> $ nice -n 19 make -j4 all
>>>
>>> This compiles just fine. So I am at the moment attempting to investigate
>>> what might cause the difference between what buildbot does and what I do
>>> through ssh.
>>>
>> I suspect you are hitting a bug introduced recently, and fixed by:
>> https://gcc.gnu.org/ml/gcc-patches/2017-12/msg00434.html
>>
>
> Wow, that's really useful. Thanks for letting me know.
>
And the patch was committed last night (r255659), so maybe your builds now work?

> --
> Paulo Matos

Re: GCC Buildbot Update

[Paulo Matos]

On 15/12/17 10:21, Christophe Lyon wrote:
> And the patch was committed last night (r255659), so maybe your builds now work?
> 

Forgot to mention that. Yes, it built!
https://gcc-buildbot.linki.tools/#/builders/5

-- 
Paulo Matos

Re: GCC Buildbot Update

[Markus Trippelsdorf]

On 2017.12.15 at 10:21 +0100, Paulo Matos wrote:
> 
> 
> On 15/12/17 08:42, Markus Trippelsdorf wrote:
> > 
> > I don't think this is good news at all. 
> > 
> 
> As I pointed out in a reply to Chris, I haven't seeked permission but I
> am pretty sure something similar runs in the CF machines from other
> projects.
> 
> The downside is that if we can't use the CF, I have no extra machines to
> run the buildbot on.
> 
> > Once a buildbot runs on a CF machine it immediately becomes impossible
> > to do any meaningful measurement on that machine. That is mainly because
> > of the random I/O (untar, rm -fr, etc.) of the bot. As a result variance
> > goes to the roof and all measurements drown in noise.
> > 
> > So it would be good if there was a strict separation of machines used
> > for bots and machines used by humans. In other words bots should only
> > run on dedicated machines.
> > 
> 
> I understand your concern though. Do you know who this issue could be
> raised with? FSF?

I think the best place would be the CF user mailing list
<cfarm-users@lists.tetaneutral.net>.
(All admins and users should be subscribed.)

-- 
Markus

Re: GCC Buildbot Update

[David Malcolm]

On Fri, 2017-12-15 at 10:16 +0100, Paulo Matos wrote:
> 
> On 14/12/17 12:39, David Malcolm wrote:

[...]

> > It looks like you're capturing the textual output from "jv compare"
> > and
> > using the exit code.  Would you prefer to import "jv" as a python
> > module and use some kind of API?  Or a different output format?
> > 
> 
> Well, I am using a fork of it which I converted to Python3. Would you
> be
> open to convert yours to Python3? The reason I am doing this is
> because
> all other Python software I have and the buildbot use Python3.

Done.

I found and fixed some more bugs, also (introduced during my
refactoring, sigh...)

> I would also prefer to have some json format or something but when I
> looked at it, the software was just printing to stdout and I didn't
> want
> to spend too much time implementing it, so I thought parsing the
> output
> was just easier.

I can add JSON output (or whatever), but I need to get back to gcc 8
work, so if the stdout output is good enough for now, let's defer
output changes.

> > If you file pull request(s) for the changes you've made in your
> > copy of
> > jamais-vu, I can take at look at merging them.
> > 
> 
> Happy to do so...
> Will merge your changes into my fork first then.
> 
> Kind regards,

Re: GCC Buildbot Update

[Segher Boessenkool]

On Fri, Dec 15, 2017 at 08:42:18AM +0100, Markus Trippelsdorf wrote:
> On 2017.12.14 at 21:32 +0100, Christophe Lyon wrote:
> > On 14 December 2017 at 09:56, Paulo Matos <pmatos@linki.tools> wrote:
> > > I got an email suggesting I add some aarch64 workers so I did:
> > > 4 workers from CF (gcc113, gcc114, gcc115 and gcc116);
> > >
> > Great, I thought the CF machines were reserved for developpers.
> > Good news you could add builders on them.
> 
> I don't think this is good news at all. 
> 
> Once a buildbot runs on a CF machine it immediately becomes impossible
> to do any meaningful measurement on that machine. That is mainly because
> of the random I/O (untar, rm -fr, etc.) of the bot. As a result variance
> goes to the roof and all measurements drown in noise.

Automated runs should not use an unreasonable amount of resources (and
neither should manual runs, but the bar for automated things lies much
lower, since they are more annoying).

All the cfarm machines are shared resources.  Benchmarking on them will
not work no matter what.  And being a shared resource means all users
have to share and be mindful of others.

> So it would be good if there was a strict separation of machines used
> for bots and machines used by humans. In other words bots should only
> run on dedicated machines.

The aarch64 builds should probably not use all of gcc113..gcc116.

We do not have enough resources to dedicate machines to bots.


Segher

Re: GCC Buildbot Update

[Paulo Matos]

On 15/12/17 15:29, David Malcolm wrote:
> On Fri, 2017-12-15 at 10:16 +0100, Paulo Matos wrote:
>>
>> On 14/12/17 12:39, David Malcolm wrote:
> 
> [...]
> 
>>> It looks like you're capturing the textual output from "jv compare"
>>> and
>>> using the exit code.  Would you prefer to import "jv" as a python
>>> module and use some kind of API?  Or a different output format?
>>>
>>
>> Well, I am using a fork of it which I converted to Python3. Would you
>> be
>> open to convert yours to Python3? The reason I am doing this is
>> because
>> all other Python software I have and the buildbot use Python3.
> 
> Done.
> 
> I found and fixed some more bugs, also (introduced during my
> refactoring, sigh...)
> 

That's great. Thank you very much for this work.

>> I would also prefer to have some json format or something but when I
>> looked at it, the software was just printing to stdout and I didn't
>> want
>> to spend too much time implementing it, so I thought parsing the
>> output
>> was just easier.
> 
> I can add JSON output (or whatever), but I need to get back to gcc 8
> work, so if the stdout output is good enough for now, let's defer
> output changes.
> 

Agree, for now I can use what I already have to read the output of jv.
I think I can now delete my fork and just use upstream jv as a submodule.

-- 
Paulo Matos

Re: GCC Buildbot Update

[Paulo Matos]

On 15/12/17 18:05, Segher Boessenkool wrote:
> All the cfarm machines are shared resources.  Benchmarking on them will
> not work no matter what.  And being a shared resource means all users
> have to share and be mindful of others.
> 

Yes, we'll definitely need better machines for benchmarking. Something I
haven't thought of yet.

>> So it would be good if there was a strict separation of machines used
>> for bots and machines used by humans. In other words bots should only
>> run on dedicated machines.
> 
> The aarch64 builds should probably not use all of gcc113..gcc116.
>
> We do not have enough resources to dedicate machines to bots.
>

I have disabled gcc116.

Thanks,
-- 
Paulo Matos

Re: GCC Buildbot Update

[Paulo Matos]

On 15/12/17 10:21, Christophe Lyon wrote:
> On 15 December 2017 at 10:19, Paulo Matos <pmatos@linki.tools> wrote:
>>
>>
>> On 14/12/17 21:32, Christophe Lyon wrote:
>>> Great, I thought the CF machines were reserved for developpers.
>>> Good news you could add builders on them.
>>>
>>
>> Oh. I have seen similar things happening on CF machines so I thought it
>> was not a problem. I have never specifically asked for permission.
>>
>>>> pmatos@gcc115:~/gcc-8-20171203_BUILD$ as -march=armv8.1-a
>>>> Assembler messages:
>>>> Error: unknown architecture `armv8.1-a'
>>>>
>>>> Error: unrecognized option -march=armv8.1-a
>>>>
>>>> However, if I run the a compiler build manually with just:
>>>>
>>>> $ configure --disable-multilib
>>>> $ nice -n 19 make -j4 all
>>>>
>>>> This compiles just fine. So I am at the moment attempting to investigate
>>>> what might cause the difference between what buildbot does and what I do
>>>> through ssh.
>>>>
>>> I suspect you are hitting a bug introduced recently, and fixed by:
>>> https://gcc.gnu.org/ml/gcc-patches/2017-12/msg00434.html
>>>
>>
>> Wow, that's really useful. Thanks for letting me know.
>>
> And the patch was committed last night (r255659), so maybe your builds now work?
> 

On some machines, in incremental builds I still seeing this:
Assembler messages:
Error: unknown architectural extension `lse'
Error: unrecognized option -march=armv8-a+lse
make[4]: *** [load_1_1_.lo] Error 1
make[4]: *** Waiting for unfinished jobs....

Looks related... the only strange thing happening is that this doesn't
happen in full builds.

-- 
Paulo Matos

Re: GCC Buildbot Update

[Christophe Lyon]

On 20 December 2017 at 09:31, Paulo Matos <pmatos@linki.tools> wrote:
>
>
> On 15/12/17 10:21, Christophe Lyon wrote:
>> On 15 December 2017 at 10:19, Paulo Matos <pmatos@linki.tools> wrote:
>>>
>>>
>>> On 14/12/17 21:32, Christophe Lyon wrote:
>>>> Great, I thought the CF machines were reserved for developpers.
>>>> Good news you could add builders on them.
>>>>
>>>
>>> Oh. I have seen similar things happening on CF machines so I thought it
>>> was not a problem. I have never specifically asked for permission.
>>>
>>>>> pmatos@gcc115:~/gcc-8-20171203_BUILD$ as -march=armv8.1-a
>>>>> Assembler messages:
>>>>> Error: unknown architecture `armv8.1-a'
>>>>>
>>>>> Error: unrecognized option -march=armv8.1-a
>>>>>
>>>>> However, if I run the a compiler build manually with just:
>>>>>
>>>>> $ configure --disable-multilib
>>>>> $ nice -n 19 make -j4 all
>>>>>
>>>>> This compiles just fine. So I am at the moment attempting to investigate
>>>>> what might cause the difference between what buildbot does and what I do
>>>>> through ssh.
>>>>>
>>>> I suspect you are hitting a bug introduced recently, and fixed by:
>>>> https://gcc.gnu.org/ml/gcc-patches/2017-12/msg00434.html
>>>>
>>>
>>> Wow, that's really useful. Thanks for letting me know.
>>>
>> And the patch was committed last night (r255659), so maybe your builds now work?
>>
>
> On some machines, in incremental builds I still seeing this:
> Assembler messages:
> Error: unknown architectural extension `lse'
> Error: unrecognized option -march=armv8-a+lse
> make[4]: *** [load_1_1_.lo] Error 1
> make[4]: *** Waiting for unfinished jobs....
>
> Looks related... the only strange thing happening is that this doesn't
> happen in full builds.
>

The recent fix changed the Makefile and configure script in libatomic.
I guess that if your incremental builds does not run configure, it's
still using old Makefiles, and old options.


> --
> Paulo Matos

Re: GCC Buildbot Update

[Paulo Matos]

On 20/12/17 10:51, Christophe Lyon wrote:
> 
> The recent fix changed the Makefile and configure script in libatomic.
> I guess that if your incremental builds does not run configure, it's
> still using old Makefiles, and old options.
> 
> 
You're right. I guess incremental builds should always call configure,
just in case.

Thanks,
-- 
Paulo Matos

Re: GCC Buildbot Update

[Christophe Lyon]

On 20 December 2017 at 11:02, Paulo Matos <pmatos@linki.tools> wrote:
>
>
> On 20/12/17 10:51, Christophe Lyon wrote:
>>
>> The recent fix changed the Makefile and configure script in libatomic.
>> I guess that if your incremental builds does not run configure, it's
>> still using old Makefiles, and old options.
>>
>>
> You're right. I guess incremental builds should always call configure,
> just in case.
>

Maybe, but this does not always work. Sometimes, I have to rm -rf $builddir


> Thanks,
> --
> Paulo Matos

Re: GCC Buildbot Update

[James Greenhalgh]

On Wed, Dec 20, 2017 at 10:02:45AM +0000, Paulo Matos wrote:
> 
> 
> On 20/12/17 10:51, Christophe Lyon wrote:
> > 
> > The recent fix changed the Makefile and configure script in libatomic.
> > I guess that if your incremental builds does not run configure, it's
> > still using old Makefiles, and old options.
> > 
> > 
> You're right. I guess incremental builds should always call configure,
> just in case.

For my personal bisect scripts I try an incremental build, with a
full rebuild as a fallback on failure.

That gives me the benefits of an incremental build most of the time (I
don't have stats on how often) with an automated approach to keeping things
going where there are issues.

Note that there are rare cases where depencies are missed in the toolchain
and an incremental build will give you a toolchain with undefined
behaviour, as one compilation unit takes a new definition of a
struct/interface and the other sits on an outdated compile from the
previous build.

I don't have a good way to detect these.

Thanks,
James

Re: GCC Buildbot Update

[Paulo Matos]

On 20/12/17 12:48, James Greenhalgh wrote:
> On Wed, Dec 20, 2017 at 10:02:45AM +0000, Paulo Matos wrote:
>>
>>
>> On 20/12/17 10:51, Christophe Lyon wrote:
>>>
>>> The recent fix changed the Makefile and configure script in libatomic.
>>> I guess that if your incremental builds does not run configure, it's
>>> still using old Makefiles, and old options.
>>>
>>>
>> You're right. I guess incremental builds should always call configure,
>> just in case.
> 
> For my personal bisect scripts I try an incremental build, with a
> full rebuild as a fallback on failure.
> 
> That gives me the benefits of an incremental build most of the time (I
> don't have stats on how often) with an automated approach to keeping things
> going where there are issues.
> 
> Note that there are rare cases where depencies are missed in the toolchain
> and an incremental build will give you a toolchain with undefined
> behaviour, as one compilation unit takes a new definition of a
> struct/interface and the other sits on an outdated compile from the
> previous build.
> 
> I don't have a good way to detect these.
> 

That's definitely a shortcoming of incremental builds. Unfortunately we
cannot cope with full builds for each commit (even for incremental
builds we'll need an alternative soon). So I will implement the same
strategy of full build if incremental fails, I think.

With respect with regards to incremental builds with undefined behaviour
that probably means that dependencies are incorrectly calculated. It
would be great to sort these out. If we could detect that there are
issues with the incremental build we could then try to understand which
dependencies were not properly calculated. Just a guess, however
implementing this might take awhile and would obviously need a lot more
resources than we have available now.

-- 
Paulo Matos

jamais-vu can now ignore renumbering of source lines in dg output (Re: GCC Buildbot Update)

[David Malcolm]

On Sat, 2017-12-16 at 12:06 +0100, Paulo Matos wrote:
> 
> On 15/12/17 15:29, David Malcolm wrote:
> > On Fri, 2017-12-15 at 10:16 +0100, Paulo Matos wrote:
> > > 
> > > On 14/12/17 12:39, David Malcolm wrote:
> > 
> > [...]
> > 
> > > > It looks like you're capturing the textual output from "jv
> > > > compare"
> > > > and
> > > > using the exit code.  Would you prefer to import "jv" as a
> > > > python
> > > > module and use some kind of API?  Or a different output format?
> > > > 
> > > 
> > > Well, I am using a fork of it which I converted to Python3. Would
> > > you
> > > be
> > > open to convert yours to Python3? The reason I am doing this is
> > > because
> > > all other Python software I have and the buildbot use Python3.
> > 
> > Done.
> > 
> > I found and fixed some more bugs, also (introduced during my
> > refactoring, sigh...)
> > 
> 
> That's great. Thank you very much for this work.
> 
> > > I would also prefer to have some json format or something but
> > > when I
> > > looked at it, the software was just printing to stdout and I
> > > didn't
> > > want
> > > to spend too much time implementing it, so I thought parsing the
> > > output
> > > was just easier.
> > 
> > I can add JSON output (or whatever), but I need to get back to gcc
> > 8
> > work, so if the stdout output is good enough for now, let's defer
> > output changes.
> > 
> 
> Agree, for now I can use what I already have to read the output of
> jv.
> I think I can now delete my fork and just use upstream jv as a
> submodule.

I've added a new feature to jamais-vu (as of
77849e2809ca9a049d5683571e27ebe190977fa8): it can now ignore test
results that merely changed line number.  

For example, if the old .sum file has a:

  PASS: g++.dg/diagnostic/param-type-mismatch.C  -std=gnu++11  (test for errors, line 106)

and the new .sum file has a:

  PASS: g++.dg/diagnostic/param-type-mismatch.C  -std=gnu++11  (test for errors, line 103)

and diffing the source trees reveals that line 106 became line 103, the
change won't be reported by "jv compare".

It also does it for dg-{begin|end}-multiline-output.

It will report them if the outcome changed (e.g. from PASS to FAIL).

To do this filtering, jv needs access to the old and new source trees,
so it can diff the pertinent source files, so "jv compare" has gained
the optional arguments
  --old-source-path=
and
  --new-source-path=
See the example in the jv Makefile for more info.  If they're not
present, it should work as before (without being able to do the above
filtering).

Is this something that the buildbot can use?

Dave

Re: jamais-vu can now ignore renumbering of source lines in dg output (Re: GCC Buildbot Update)

[Paulo Matos]

On 24/01/18 20:20, David Malcolm wrote:
> 
> I've added a new feature to jamais-vu (as of
> 77849e2809ca9a049d5683571e27ebe190977fa8): it can now ignore test
> results that merely changed line number.  
> 
> For example, if the old .sum file has a:
> 
>   PASS: g++.dg/diagnostic/param-type-mismatch.C  -std=gnu++11  (test for errors, line 106)
> 
> and the new .sum file has a:
> 
>   PASS: g++.dg/diagnostic/param-type-mismatch.C  -std=gnu++11  (test for errors, line 103)
> 
> and diffing the source trees reveals that line 106 became line 103, the
> change won't be reported by "jv compare".
> 
> It also does it for dg-{begin|end}-multiline-output.
> 
> It will report them if the outcome changed (e.g. from PASS to FAIL).
> 
> To do this filtering, jv needs access to the old and new source trees,
> so it can diff the pertinent source files, so "jv compare" has gained
> the optional arguments
>   --old-source-path=
> and
>   --new-source-path=
> See the example in the jv Makefile for more info.  If they're not
> present, it should work as before (without being able to do the above
> filtering).
> 
> Is this something that the buildbot can use?
> 

Hi David,

Thanks for the amazing improvements.
I will take a look at them on Monday. I have a lot of work at the moment
so I decided to take 1/5 of my week (usually Monday) to work on buildbot
so I will definitely get it integrated on Monday and hopefully have
something to say afterwards.

Thanks for keeping me up-to-date with these changes.

-- 
Paulo Matos

Re: jamais-vu can now ignore renumbering of source lines in dg output (Re: GCC Buildbot Update)

[Paulo Matos]

On 24/01/18 20:20, David Malcolm wrote:
> 
> I've added a new feature to jamais-vu (as of
> 77849e2809ca9a049d5683571e27ebe190977fa8): it can now ignore test
> results that merely changed line number.  
> 
> For example, if the old .sum file has a:
> 
>   PASS: g++.dg/diagnostic/param-type-mismatch.C  -std=gnu++11  (test for errors, line 106)
> 
> and the new .sum file has a:
> 
>   PASS: g++.dg/diagnostic/param-type-mismatch.C  -std=gnu++11  (test for errors, line 103)
> 
> and diffing the source trees reveals that line 106 became line 103, the
> change won't be reported by "jv compare".
> 
> It also does it for dg-{begin|end}-multiline-output.
> 
> It will report them if the outcome changed (e.g. from PASS to FAIL).
> 
> To do this filtering, jv needs access to the old and new source trees,
> so it can diff the pertinent source files, so "jv compare" has gained
> the optional arguments
>   --old-source-path=
> and
>   --new-source-path=
> See the example in the jv Makefile for more info.  If they're not
> present, it should work as before (without being able to do the above
> filtering).


Hi,

I am looking at this today and I noticed that having the source file for
all recent GCC revisions is costly in terms of time (if we wish to
compress them) and space (for storage). I was instead thinking that jv
could calculate the differences offline using pysvn and the old and new
revision numbers.

I have started implementing this in my port. Would you consider merging it?

-- 
Paulo Matos

Re: jamais-vu can now ignore renumbering of source lines in dg output (Re: GCC Buildbot Update)

[David Malcolm]

On Mon, 2018-01-29 at 14:55 +0100, Paulo Matos wrote:
> 
> On 24/01/18 20:20, David Malcolm wrote:
> > 
> > I've added a new feature to jamais-vu (as of
> > 77849e2809ca9a049d5683571e27ebe190977fa8): it can now ignore test
> > results that merely changed line number.  
> > 
> > For example, if the old .sum file has a:
> > 
> >   PASS: g++.dg/diagnostic/param-type-mismatch.C  -
> > std=gnu++11  (test for errors, line 106)
> > 
> > and the new .sum file has a:
> > 
> >   PASS: g++.dg/diagnostic/param-type-mismatch.C  -
> > std=gnu++11  (test for errors, line 103)
> > 
> > and diffing the source trees reveals that line 106 became line 103,
> > the
> > change won't be reported by "jv compare".
> > 
> > It also does it for dg-{begin|end}-multiline-output.
> > 
> > It will report them if the outcome changed (e.g. from PASS to
> > FAIL).
> > 
> > To do this filtering, jv needs access to the old and new source
> > trees,
> > so it can diff the pertinent source files, so "jv compare" has
> > gained
> > the optional arguments
> >   --old-source-path=
> > and
> >   --new-source-path=
> > See the example in the jv Makefile for more info.  If they're not
> > present, it should work as before (without being able to do the
> > above
> > filtering).
> 
> 
> Hi,
> 
> I am looking at this today and I noticed that having the source file
> for
> all recent GCC revisions is costly in terms of time (if we wish to
> compress them) and space (for storage). I was instead thinking that
> jv
> could calculate the differences offline using pysvn and the old and
> new
> revision numbers.

Note that access to the source files is optional - jv doesn't need
them, it just helps for the particular situation described above.

> I have started implementing this in my port. Would you consider
> merging it?

Sounds reasonable - though bear in mind that gcc might be switching to
git at some point.

Send a pull request (I've turned on travis CI on the github repository,
so pull requests now automatically get tested on a bunch of different
Python 3 versions).

Thanks
Dave

Re: jamais-vu can now ignore renumbering of source lines in dg output (Re: GCC Buildbot Update)

[Paulo Matos]

On 29/01/18 15:19, David Malcolm wrote:
>>
>> Hi,
>>
>> I am looking at this today and I noticed that having the source file
>> for
>> all recent GCC revisions is costly in terms of time (if we wish to
>> compress them) and space (for storage). I was instead thinking that
>> jv
>> could calculate the differences offline using pysvn and the old and
>> new
>> revision numbers.
> 
> Note that access to the source files is optional - jv doesn't need
> them, it just helps for the particular situation described above.
> 

I understand but it would be great to have line number filtering.

>> I have started implementing this in my port. Would you consider
>> merging it?
> 
> Sounds reasonable - though bear in mind that gcc might be switching to
> git at some point.
> 

Yes, I know... but... if we wait for that to happen to implement
something... :)

> Send a pull request (I've turned on travis CI on the github repository,
> so pull requests now automatically get tested on a bunch of different
> Python 3 versions).
> 

Sure.

-- 
Paulo Matos