* duplicate arm test results?
@ 2020-09-22 15:02 Martin Sebor
2020-09-22 15:15 ` Christophe Lyon
0 siblings, 1 reply; 18+ messages in thread
From: Martin Sebor @ 2020-09-22 15:02 UTC (permalink / raw)
To: Christophe Lyon, gcc mailing list
Hi Christophe,
While checking recent test results I noticed many posts with results
for various flavors of arm that at high level seem like duplicates
of one another.
For example, the batch below all have the same title, but not all
of the contents are the same. The details (such as test failures)
on some of the pages are different.
Can you help explain the differences? Is there a way to avoid
the duplication?
Thanks
Martin
Results for 11.0.0 20200922 (experimental) [master revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
arm-none-eabi Christophe LYON
Results for 11.0.0 20200922 (experimental) [master revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
arm-none-eabi Christophe LYON
Results for 11.0.0 20200922 (experimental) [master revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
arm-none-eabi Christophe LYON
Results for 11.0.0 20200922 (experimental) [master revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
arm-none-eabi Christophe LYON
Results for 11.0.0 20200922 (experimental) [master revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
arm-none-eabi Christophe LYON
Results for 11.0.0 20200922 (experimental) [master revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
arm-none-eabi Christophe LYON
Results for 11.0.0 20200922 (experimental) [master revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
arm-none-eabi Christophe LYON
Results for 11.0.0 20200922 (experimental) [master revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
arm-none-eabi Christophe LYON
Results for 11.0.0 20200922 (experimental) [master revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
arm-none-eabi Christophe LYON
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-22 15:02 duplicate arm test results? Martin Sebor
@ 2020-09-22 15:15 ` Christophe Lyon
2020-09-22 23:47 ` Martin Sebor
0 siblings, 1 reply; 18+ messages in thread
From: Christophe Lyon @ 2020-09-22 15:15 UTC (permalink / raw)
To: Martin Sebor; +Cc: gcc mailing list
On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
>
> Hi Christophe,
>
> While checking recent test results I noticed many posts with results
> for various flavors of arm that at high level seem like duplicates
> of one another.
>
> For example, the batch below all have the same title, but not all
> of the contents are the same. The details (such as test failures)
> on some of the pages are different.
>
> Can you help explain the differences? Is there a way to avoid
> the duplication?
>
Sure, I am aware that many results look the same...
If you look at the top of the report (~line 5), you'll see:
Running target myarm-sim
Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
For all of these, the first line of the report is:
LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
TARGET=arm-none-eabi CPU=default FPU=default MODE=default
I have other combinations where I override the configure flags, eg:
LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
I tried to see if I could fit something in the subject line, but that
didn't seem convenient (would be too long, and I fear modifying the
awk script....)
I think HJ generates several "running targets" in the same log, I run
them separately to benefit from the compute farm I have access to.
Christophe
> Thanks
> Martin
>
> Results for 11.0.0 20200922 (experimental) [master revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> arm-none-eabi Christophe LYON
>
> Results for 11.0.0 20200922 (experimental) [master revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> arm-none-eabi Christophe LYON
> Results for 11.0.0 20200922 (experimental) [master revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> arm-none-eabi Christophe LYON
> Results for 11.0.0 20200922 (experimental) [master revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> arm-none-eabi Christophe LYON
> Results for 11.0.0 20200922 (experimental) [master revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> arm-none-eabi Christophe LYON
> Results for 11.0.0 20200922 (experimental) [master revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> arm-none-eabi Christophe LYON
> Results for 11.0.0 20200922 (experimental) [master revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> arm-none-eabi Christophe LYON
> Results for 11.0.0 20200922 (experimental) [master revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> arm-none-eabi Christophe LYON
> Results for 11.0.0 20200922 (experimental) [master revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> arm-none-eabi Christophe LYON
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-22 15:15 ` Christophe Lyon
@ 2020-09-22 23:47 ` Martin Sebor
2020-09-23 8:54 ` Christophe Lyon
0 siblings, 1 reply; 18+ messages in thread
From: Martin Sebor @ 2020-09-22 23:47 UTC (permalink / raw)
To: Christophe Lyon; +Cc: gcc mailing list
On 9/22/20 9:15 AM, Christophe Lyon wrote:
> On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
>>
>> Hi Christophe,
>>
>> While checking recent test results I noticed many posts with results
>> for various flavors of arm that at high level seem like duplicates
>> of one another.
>>
>> For example, the batch below all have the same title, but not all
>> of the contents are the same. The details (such as test failures)
>> on some of the pages are different.
>>
>> Can you help explain the differences? Is there a way to avoid
>> the duplication?
>>
>
> Sure, I am aware that many results look the same...
>
>
> If you look at the top of the report (~line 5), you'll see:
> Running target myarm-sim
> Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
> Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
> Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
> Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
> Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
> Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
>
> For all of these, the first line of the report is:
> LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
> TARGET=arm-none-eabi CPU=default FPU=default MODE=default
>
> I have other combinations where I override the configure flags, eg:
> LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
> r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
> TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
>
> I tried to see if I could fit something in the subject line, but that
> didn't seem convenient (would be too long, and I fear modifying the
> awk script....)
Without some indication of a difference in the title there's no way
to know what result to look at, and checking all of them isn't really
practical. The duplication (and the sheer number of results) also
make it more difficult to find results for targets other than arm-*.
There are about 13,000 results for September and over 10,000 of those
for arm-* alone. It's good to have data but when there's this much
of it, and when the only form of presentation is as a running list,
it's too cumbersome to work with.
Martin
>
> I think HJ generates several "running targets" in the same log, I run
> them separately to benefit from the compute farm I have access to.
>
> Christophe
>
>> Thanks
>> Martin
>>
>> Results for 11.0.0 20200922 (experimental) [master revision
>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>> arm-none-eabi Christophe LYON
>>
>> Results for 11.0.0 20200922 (experimental) [master revision
>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>> arm-none-eabi Christophe LYON
>> Results for 11.0.0 20200922 (experimental) [master revision
>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>> arm-none-eabi Christophe LYON
>> Results for 11.0.0 20200922 (experimental) [master revision
>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>> arm-none-eabi Christophe LYON
>> Results for 11.0.0 20200922 (experimental) [master revision
>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>> arm-none-eabi Christophe LYON
>> Results for 11.0.0 20200922 (experimental) [master revision
>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>> arm-none-eabi Christophe LYON
>> Results for 11.0.0 20200922 (experimental) [master revision
>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>> arm-none-eabi Christophe LYON
>> Results for 11.0.0 20200922 (experimental) [master revision
>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>> arm-none-eabi Christophe LYON
>> Results for 11.0.0 20200922 (experimental) [master revision
>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>> arm-none-eabi Christophe LYON
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-22 23:47 ` Martin Sebor
@ 2020-09-23 8:54 ` Christophe Lyon
2020-09-23 9:22 ` Richard Sandiford
2020-09-23 15:33 ` Martin Sebor
0 siblings, 2 replies; 18+ messages in thread
From: Christophe Lyon @ 2020-09-23 8:54 UTC (permalink / raw)
To: Martin Sebor; +Cc: gcc mailing list
On Wed, 23 Sep 2020 at 01:47, Martin Sebor <msebor@gmail.com> wrote:
>
> On 9/22/20 9:15 AM, Christophe Lyon wrote:
> > On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
> >>
> >> Hi Christophe,
> >>
> >> While checking recent test results I noticed many posts with results
> >> for various flavors of arm that at high level seem like duplicates
> >> of one another.
> >>
> >> For example, the batch below all have the same title, but not all
> >> of the contents are the same. The details (such as test failures)
> >> on some of the pages are different.
> >>
> >> Can you help explain the differences? Is there a way to avoid
> >> the duplication?
> >>
> >
> > Sure, I am aware that many results look the same...
> >
> >
> > If you look at the top of the report (~line 5), you'll see:
> > Running target myarm-sim
> > Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
> > Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
> > Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
> > Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
> > Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
> > Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
> > Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
> > Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
> >
> > For all of these, the first line of the report is:
> > LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
> > r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
> > TARGET=arm-none-eabi CPU=default FPU=default MODE=default
> >
> > I have other combinations where I override the configure flags, eg:
> > LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
> > r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
> > TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
> >
> > I tried to see if I could fit something in the subject line, but that
> > didn't seem convenient (would be too long, and I fear modifying the
> > awk script....)
>
> Without some indication of a difference in the title there's no way
> to know what result to look at, and checking all of them isn't really
> practical. The duplication (and the sheer number of results) also
> make it more difficult to find results for targets other than arm-*.
> There are about 13,000 results for September and over 10,000 of those
> for arm-* alone. It's good to have data but when there's this much
> of it, and when the only form of presentation is as a running list,
> it's too cumbersome to work with.
>
To help me track & report regressions, I build higher level reports like:
https://people.linaro.org/~christophe.lyon/cross-validation/gcc/trunk/0latest/report-build-info.html
where it's more obvious what configurations are tested.
Each line of such reports can send a message to gcc-testresults.
I can control when such emails are sent, independently for each line:
- never
- for daily bump
- for each validation
So, I can easily reduce the amount of emails (by disabling them for
some configurations),
but that won't make the subject more informative.
I included the short revision (rXX-YYYY) in the title to make it clearer.
The number of configurations has grown over time because we regularly
found regressions
in configurations not tested previously.
I can probably easily add the values of --with-cpu, --with-fpu,
--with-mode and RUNTESTFLAGS
as part of the [<branch> revision rXX-YYYY-ZZZZZ] string in the title,
would that help?
I fear that's going to make very long subject lines.
It would probably be cleaner to update test_summary such that it adds
more info as part of $host
(as in "... testsuite on $host"), so that it grabs useful configure
parameters and runtestflags, however
this would be more controversial.
Christophe
> Martin
>
> >
> > I think HJ generates several "running targets" in the same log, I run
> > them separately to benefit from the compute farm I have access to.
> >
> > Christophe
> >
> >> Thanks
> >> Martin
> >>
> >> Results for 11.0.0 20200922 (experimental) [master revision
> >> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >> arm-none-eabi Christophe LYON
> >>
> >> Results for 11.0.0 20200922 (experimental) [master revision
> >> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >> arm-none-eabi Christophe LYON
> >> Results for 11.0.0 20200922 (experimental) [master revision
> >> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >> arm-none-eabi Christophe LYON
> >> Results for 11.0.0 20200922 (experimental) [master revision
> >> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >> arm-none-eabi Christophe LYON
> >> Results for 11.0.0 20200922 (experimental) [master revision
> >> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >> arm-none-eabi Christophe LYON
> >> Results for 11.0.0 20200922 (experimental) [master revision
> >> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >> arm-none-eabi Christophe LYON
> >> Results for 11.0.0 20200922 (experimental) [master revision
> >> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >> arm-none-eabi Christophe LYON
> >> Results for 11.0.0 20200922 (experimental) [master revision
> >> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >> arm-none-eabi Christophe LYON
> >> Results for 11.0.0 20200922 (experimental) [master revision
> >> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >> arm-none-eabi Christophe LYON
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 8:54 ` Christophe Lyon
@ 2020-09-23 9:22 ` Richard Sandiford
2020-09-23 10:20 ` Jakub Jelinek
2020-09-23 15:33 ` Martin Sebor
1 sibling, 1 reply; 18+ messages in thread
From: Richard Sandiford @ 2020-09-23 9:22 UTC (permalink / raw)
To: Christophe Lyon via Gcc
Christophe Lyon via Gcc <gcc@gcc.gnu.org> writes:
> On Wed, 23 Sep 2020 at 01:47, Martin Sebor <msebor@gmail.com> wrote:
>>
>> On 9/22/20 9:15 AM, Christophe Lyon wrote:
>> > On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
>> >>
>> >> Hi Christophe,
>> >>
>> >> While checking recent test results I noticed many posts with results
>> >> for various flavors of arm that at high level seem like duplicates
>> >> of one another.
>> >>
>> >> For example, the batch below all have the same title, but not all
>> >> of the contents are the same. The details (such as test failures)
>> >> on some of the pages are different.
>> >>
>> >> Can you help explain the differences? Is there a way to avoid
>> >> the duplication?
>> >>
>> >
>> > Sure, I am aware that many results look the same...
>> >
>> >
>> > If you look at the top of the report (~line 5), you'll see:
>> > Running target myarm-sim
>> > Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
>> > Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
>> > Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
>> > Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
>> > Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
>> > Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
>> > Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
>> > Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
>> >
>> > For all of these, the first line of the report is:
>> > LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
>> > r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
>> > TARGET=arm-none-eabi CPU=default FPU=default MODE=default
>> >
>> > I have other combinations where I override the configure flags, eg:
>> > LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
>> > r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
>> > TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
>> >
>> > I tried to see if I could fit something in the subject line, but that
>> > didn't seem convenient (would be too long, and I fear modifying the
>> > awk script....)
>>
>> Without some indication of a difference in the title there's no way
>> to know what result to look at, and checking all of them isn't really
>> practical. The duplication (and the sheer number of results) also
>> make it more difficult to find results for targets other than arm-*.
>> There are about 13,000 results for September and over 10,000 of those
>> for arm-* alone. It's good to have data but when there's this much
>> of it, and when the only form of presentation is as a running list,
>> it's too cumbersome to work with.
>>
>
> To help me track & report regressions, I build higher level reports like:
> https://people.linaro.org/~christophe.lyon/cross-validation/gcc/trunk/0latest/report-build-info.html
> where it's more obvious what configurations are tested.
>
> Each line of such reports can send a message to gcc-testresults.
>
> I can control when such emails are sent, independently for each line:
> - never
> - for daily bump
> - for each validation
>
> So, I can easily reduce the amount of emails (by disabling them for
> some configurations),
> but that won't make the subject more informative.
> I included the short revision (rXX-YYYY) in the title to make it clearer.
>
> The number of configurations has grown over time because we regularly
> found regressions
> in configurations not tested previously.
Yeah. And thanks for doing this. It's caught all sorts of things
that we didn't see previously.
> I can probably easily add the values of --with-cpu, --with-fpu,
> --with-mode and RUNTESTFLAGS
> as part of the [<branch> revision rXX-YYYY-ZZZZZ] string in the title,
> would that help?
> I fear that's going to make very long subject lines.
>
> It would probably be cleaner to update test_summary such that it adds
> more info as part of $host
> (as in "... testsuite on $host"), so that it grabs useful configure
> parameters and runtestflags, however
> this would be more controversial.
FWIW, taking a recent arbitrary example:
Results for 8.4.1 20200918 [releases/gcc-8 revision r8-10517-gbbb72c2ccc271541e0d1feb25d2256d47041df59] (GCC) testsuite on arm-none-linux-gnueabihf
For the subject line, I think we can drop the branch name if it's the
obvious one for the version number. And we could drop the git hash and
just use the rN-MMMM id. The “(GCC) testsuite” part seems like noise too;
AFAICT that's a de facto invariant string for everyone who's posted this
month.
So that would give:
Results for 8.4.1 20200918 [r8-10517] on arm-none-linux-gnueabihf
and hopefully free up some space at the end for the kind of thing
you mention.
Thanks,
Richard
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 9:22 ` Richard Sandiford
@ 2020-09-23 10:20 ` Jakub Jelinek
2020-09-23 10:26 ` Richard Earnshaw
2020-09-23 10:37 ` Andreas Schwab
0 siblings, 2 replies; 18+ messages in thread
From: Jakub Jelinek @ 2020-09-23 10:20 UTC (permalink / raw)
To: Christophe Lyon via Gcc, Martin Sebor, Christophe Lyon,
richard.sandiford
On Wed, Sep 23, 2020 at 10:22:52AM +0100, Richard Sandiford wrote:
> So that would give:
>
> Results for 8.4.1 20200918 [r8-10517] on arm-none-linux-gnueabihf
>
> and hopefully free up some space at the end for the kind of thing
> you mention.
Even that 8.4.1 20200918 is redundant, r8-10517 uniquely and shortly
identifies both the branch and commit.
So just
Results for r8-10517 on ...
and in ... also include something that uniquely identifies the
configuration.
Jakub
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 10:20 ` Jakub Jelinek
@ 2020-09-23 10:26 ` Richard Earnshaw
2020-09-23 12:25 ` Christophe Lyon
2020-09-23 10:37 ` Andreas Schwab
1 sibling, 1 reply; 18+ messages in thread
From: Richard Earnshaw @ 2020-09-23 10:26 UTC (permalink / raw)
To: Jakub Jelinek, Christophe Lyon via Gcc, Martin Sebor,
Christophe Lyon, richard.sandiford
On 23/09/2020 11:20, Jakub Jelinek via Gcc wrote:
> On Wed, Sep 23, 2020 at 10:22:52AM +0100, Richard Sandiford wrote:
>> So that would give:
>>
>> Results for 8.4.1 20200918 [r8-10517] on arm-none-linux-gnueabihf
>>
>> and hopefully free up some space at the end for the kind of thing
>> you mention.
>
> Even that 8.4.1 20200918 is redundant, r8-10517 uniquely and shortly
> identifies both the branch and commit.
> So just
> Results for r8-10517 on ...
> and in ... also include something that uniquely identifies the
> configuration.
>
> Jakub
>
I was thinking similarly, but then realised anyone using snapshots
rather than git might not have that information.
If that's not the case, however, then simplifying this would be a great
idea.
On the other hand, I use subject filters in my mail to steer results to
a separate folder, so I do need some invariant key in the subject line
that is sufficient to match without (too many) false positives.
R.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 10:20 ` Jakub Jelinek
2020-09-23 10:26 ` Richard Earnshaw
@ 2020-09-23 10:37 ` Andreas Schwab
2020-09-23 10:43 ` Jakub Jelinek
1 sibling, 1 reply; 18+ messages in thread
From: Andreas Schwab @ 2020-09-23 10:37 UTC (permalink / raw)
To: Jakub Jelinek via Gcc
Cc: Martin Sebor, Christophe Lyon, richard.sandiford, Jakub Jelinek
On Sep 23 2020, Jakub Jelinek via Gcc wrote:
> Even that 8.4.1 20200918 is redundant, r8-10517 uniquely and shortly
> identifies both the branch and commit.
But it requires a repository to identify. Often, redundant information
is useful.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 10:37 ` Andreas Schwab
@ 2020-09-23 10:43 ` Jakub Jelinek
2020-09-23 10:52 ` Andreas Schwab
0 siblings, 1 reply; 18+ messages in thread
From: Jakub Jelinek @ 2020-09-23 10:43 UTC (permalink / raw)
To: Andreas Schwab; +Cc: Gcc
On Wed, Sep 23, 2020 at 12:37:52PM +0200, Andreas Schwab wrote:
> On Sep 23 2020, Jakub Jelinek via Gcc wrote:
>
> > Even that 8.4.1 20200918 is redundant, r8-10517 uniquely and shortly
> > identifies both the branch and commit.
>
> But it requires a repository to identify. Often, redundant information
> is useful.
To identify what? What one usually does with gcc-testresults is look where
something regressed, for that because the short ids are monotonically
increasing, it is easy to find what is before and what is after.
And once the regression is narrowed to a particular commit (or some range of
them if not everything is covered), one needs the repository anyway,
version and date doesn't tell anything interesting, what one cares about is
what actually changed (and why).
Jakub
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 10:43 ` Jakub Jelinek
@ 2020-09-23 10:52 ` Andreas Schwab
0 siblings, 0 replies; 18+ messages in thread
From: Andreas Schwab @ 2020-09-23 10:52 UTC (permalink / raw)
To: Jakub Jelinek; +Cc: Gcc
On Sep 23 2020, Jakub Jelinek wrote:
> On Wed, Sep 23, 2020 at 12:37:52PM +0200, Andreas Schwab wrote:
>> On Sep 23 2020, Jakub Jelinek via Gcc wrote:
>>
>> > Even that 8.4.1 20200918 is redundant, r8-10517 uniquely and shortly
>> > identifies both the branch and commit.
>>
>> But it requires a repository to identify. Often, redundant information
>> is useful.
>
> To identify what?
For example, how far apart they are.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 10:26 ` Richard Earnshaw
@ 2020-09-23 12:25 ` Christophe Lyon
2020-09-23 12:32 ` David Edelsohn
0 siblings, 1 reply; 18+ messages in thread
From: Christophe Lyon @ 2020-09-23 12:25 UTC (permalink / raw)
To: Richard Earnshaw
Cc: Jakub Jelinek, Christophe Lyon via Gcc, Martin Sebor, Richard Sandiford
On Wed, 23 Sep 2020 at 12:26, Richard Earnshaw
<Richard.Earnshaw@foss.arm.com> wrote:
>
> On 23/09/2020 11:20, Jakub Jelinek via Gcc wrote:
> > On Wed, Sep 23, 2020 at 10:22:52AM +0100, Richard Sandiford wrote:
> >> So that would give:
> >>
> >> Results for 8.4.1 20200918 [r8-10517] on arm-none-linux-gnueabihf
> >>
> >> and hopefully free up some space at the end for the kind of thing
> >> you mention.
> >
> > Even that 8.4.1 20200918 is redundant, r8-10517 uniquely and shortly
> > identifies both the branch and commit.
> > So just
> > Results for r8-10517 on ...
> > and in ... also include something that uniquely identifies the
> > configuration.
> >
> > Jakub
> >
>
> I was thinking similarly, but then realised anyone using snapshots
> rather than git might not have that information.
>
> If that's not the case, however, then simplifying this would be a great
> idea.
>
> On the other hand, I use subject filters in my mail to steer results to
> a separate folder, so I do need some invariant key in the subject line
> that is sufficient to match without (too many) false positives.
>
I always assumed there was a required format for the title/email
contents, is that documented somewhere?
There must be a smart filter to avoid spam, doesn't it require some
"keywords" in the title for instance?
Same question for the gcc-regression list: is there a mandatory format?
Thanks,
Christophe
> R.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 12:25 ` Christophe Lyon
@ 2020-09-23 12:32 ` David Edelsohn
2020-09-23 15:53 ` Christophe Lyon
0 siblings, 1 reply; 18+ messages in thread
From: David Edelsohn @ 2020-09-23 12:32 UTC (permalink / raw)
To: Christophe Lyon; +Cc: Richard Earnshaw, Jakub Jelinek, Christophe Lyon via Gcc
On Wed, Sep 23, 2020 at 8:26 AM Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
>
> On Wed, 23 Sep 2020 at 12:26, Richard Earnshaw
> <Richard.Earnshaw@foss.arm.com> wrote:
> >
> > On 23/09/2020 11:20, Jakub Jelinek via Gcc wrote:
> > > On Wed, Sep 23, 2020 at 10:22:52AM +0100, Richard Sandiford wrote:
> > >> So that would give:
> > >>
> > >> Results for 8.4.1 20200918 [r8-10517] on arm-none-linux-gnueabihf
> > >>
> > >> and hopefully free up some space at the end for the kind of thing
> > >> you mention.
> > >
> > > Even that 8.4.1 20200918 is redundant, r8-10517 uniquely and shortly
> > > identifies both the branch and commit.
> > > So just
> > > Results for r8-10517 on ...
> > > and in ... also include something that uniquely identifies the
> > > configuration.
> > >
> > > Jakub
> > >
> >
> > I was thinking similarly, but then realised anyone using snapshots
> > rather than git might not have that information.
> >
> > If that's not the case, however, then simplifying this would be a great
> > idea.
> >
> > On the other hand, I use subject filters in my mail to steer results to
> > a separate folder, so I do need some invariant key in the subject line
> > that is sufficient to match without (too many) false positives.
> >
>
> I always assumed there was a required format for the title/email
> contents, is that documented somewhere?
> There must be a smart filter to avoid spam, doesn't it require some
> "keywords" in the title for instance?
>
> Same question for the gcc-regression list: is there a mandatory format?
The format is generated by contrib/test_summary.
- David
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 8:54 ` Christophe Lyon
2020-09-23 9:22 ` Richard Sandiford
@ 2020-09-23 15:33 ` Martin Sebor
2020-09-23 15:50 ` Christophe Lyon
1 sibling, 1 reply; 18+ messages in thread
From: Martin Sebor @ 2020-09-23 15:33 UTC (permalink / raw)
To: Christophe Lyon; +Cc: gcc mailing list
On 9/23/20 2:54 AM, Christophe Lyon wrote:
> On Wed, 23 Sep 2020 at 01:47, Martin Sebor <msebor@gmail.com> wrote:
>>
>> On 9/22/20 9:15 AM, Christophe Lyon wrote:
>>> On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
>>>>
>>>> Hi Christophe,
>>>>
>>>> While checking recent test results I noticed many posts with results
>>>> for various flavors of arm that at high level seem like duplicates
>>>> of one another.
>>>>
>>>> For example, the batch below all have the same title, but not all
>>>> of the contents are the same. The details (such as test failures)
>>>> on some of the pages are different.
>>>>
>>>> Can you help explain the differences? Is there a way to avoid
>>>> the duplication?
>>>>
>>>
>>> Sure, I am aware that many results look the same...
>>>
>>>
>>> If you look at the top of the report (~line 5), you'll see:
>>> Running target myarm-sim
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
>>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
>>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
>>>
>>> For all of these, the first line of the report is:
>>> LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
>>> TARGET=arm-none-eabi CPU=default FPU=default MODE=default
>>>
>>> I have other combinations where I override the configure flags, eg:
>>> LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
>>> r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
>>> TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
>>>
>>> I tried to see if I could fit something in the subject line, but that
>>> didn't seem convenient (would be too long, and I fear modifying the
>>> awk script....)
>>
>> Without some indication of a difference in the title there's no way
>> to know what result to look at, and checking all of them isn't really
>> practical. The duplication (and the sheer number of results) also
>> make it more difficult to find results for targets other than arm-*.
>> There are about 13,000 results for September and over 10,000 of those
>> for arm-* alone. It's good to have data but when there's this much
>> of it, and when the only form of presentation is as a running list,
>> it's too cumbersome to work with.
>>
>
> To help me track & report regressions, I build higher level reports like:
> https://people.linaro.org/~christophe.lyon/cross-validation/gcc/trunk/0latest/report-build-info.html
> where it's more obvious what configurations are tested.
That looks awesome! The regression indicator looks especially
helpful. I really wish we had an overview like this for all
results. I've been thinking about writing a script to scrape
gcc-testresults and format an HTML table kind of like this for
years. With that, the number of posts sent to the list wouldn't
be a problem (at least not for those using the page). But it
would require settling on a standard format for the basic
parameters of each run.
>
> Each line of such reports can send a message to gcc-testresults.
>
> I can control when such emails are sent, independently for each line:
> - never
> - for daily bump
> - for each validation
>
> So, I can easily reduce the amount of emails (by disabling them for
> some configurations),
> but that won't make the subject more informative.
> I included the short revision (rXX-YYYY) in the title to make it clearer.
>
> The number of configurations has grown over time because we regularly
> found regressions
> in configurations not tested previously.
>
> I can probably easily add the values of --with-cpu, --with-fpu,
> --with-mode and RUNTESTFLAGS
> as part of the [<branch> revision rXX-YYYY-ZZZZZ] string in the title,
> would that help?
> I fear that's going to make very long subject lines.
>
> It would probably be cleaner to update test_summary such that it adds
> more info as part of $host
> (as in "... testsuite on $host"), so that it grabs useful configure
> parameters and runtestflags, however
> this would be more controversial.
Until a way to present summaries is available, would grouping
the results of multiple runs in the same "basic configuration"
(for some definition of basic) in the same post work for you?
Martin
>
> Christophe
>
>> Martin
>>
>>>
>>> I think HJ generates several "running targets" in the same log, I run
>>> them separately to benefit from the compute farm I have access to.
>>>
>>> Christophe
>>>
>>>> Thanks
>>>> Martin
>>>>
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>>
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 15:33 ` Martin Sebor
@ 2020-09-23 15:50 ` Christophe Lyon
2020-09-24 12:12 ` Christophe Lyon
0 siblings, 1 reply; 18+ messages in thread
From: Christophe Lyon @ 2020-09-23 15:50 UTC (permalink / raw)
To: Martin Sebor; +Cc: gcc mailing list
On Wed, 23 Sep 2020 at 17:33, Martin Sebor <msebor@gmail.com> wrote:
>
> On 9/23/20 2:54 AM, Christophe Lyon wrote:
> > On Wed, 23 Sep 2020 at 01:47, Martin Sebor <msebor@gmail.com> wrote:
> >>
> >> On 9/22/20 9:15 AM, Christophe Lyon wrote:
> >>> On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
> >>>>
> >>>> Hi Christophe,
> >>>>
> >>>> While checking recent test results I noticed many posts with results
> >>>> for various flavors of arm that at high level seem like duplicates
> >>>> of one another.
> >>>>
> >>>> For example, the batch below all have the same title, but not all
> >>>> of the contents are the same. The details (such as test failures)
> >>>> on some of the pages are different.
> >>>>
> >>>> Can you help explain the differences? Is there a way to avoid
> >>>> the duplication?
> >>>>
> >>>
> >>> Sure, I am aware that many results look the same...
> >>>
> >>>
> >>> If you look at the top of the report (~line 5), you'll see:
> >>> Running target myarm-sim
> >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
> >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
> >>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
> >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
> >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
> >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
> >>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
> >>> Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
> >>>
> >>> For all of these, the first line of the report is:
> >>> LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
> >>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
> >>> TARGET=arm-none-eabi CPU=default FPU=default MODE=default
> >>>
> >>> I have other combinations where I override the configure flags, eg:
> >>> LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
> >>> r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
> >>> TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
> >>>
> >>> I tried to see if I could fit something in the subject line, but that
> >>> didn't seem convenient (would be too long, and I fear modifying the
> >>> awk script....)
> >>
> >> Without some indication of a difference in the title there's no way
> >> to know what result to look at, and checking all of them isn't really
> >> practical. The duplication (and the sheer number of results) also
> >> make it more difficult to find results for targets other than arm-*.
> >> There are about 13,000 results for September and over 10,000 of those
> >> for arm-* alone. It's good to have data but when there's this much
> >> of it, and when the only form of presentation is as a running list,
> >> it's too cumbersome to work with.
> >>
> >
> > To help me track & report regressions, I build higher level reports like:
> > https://people.linaro.org/~christophe.lyon/cross-validation/gcc/trunk/0latest/report-build-info.html
> > where it's more obvious what configurations are tested.
>
> That looks awesome! The regression indicator looks especially
> helpful. I really wish we had an overview like this for all
> results. I've been thinking about writing a script to scrape
> gcc-testresults and format an HTML table kind of like this for
> years. With that, the number of posts sent to the list wouldn't
> be a problem (at least not for those using the page). But it
> would require settling on a standard format for the basic
> parameters of each run.
>
It's probably easier to detect regressions and format reports from the
.sum files rather than extracting them from the mailing-list.
But your approach has the advantage that you can detect regressions
from reports sent by other people, not only by you.
> >
> > Each line of such reports can send a message to gcc-testresults.
> >
> > I can control when such emails are sent, independently for each line:
> > - never
> > - for daily bump
> > - for each validation
> >
> > So, I can easily reduce the amount of emails (by disabling them for
> > some configurations),
> > but that won't make the subject more informative.
> > I included the short revision (rXX-YYYY) in the title to make it clearer.
> >
> > The number of configurations has grown over time because we regularly
> > found regressions
> > in configurations not tested previously.
> >
> > I can probably easily add the values of --with-cpu, --with-fpu,
> > --with-mode and RUNTESTFLAGS
> > as part of the [<branch> revision rXX-YYYY-ZZZZZ] string in the title,
> > would that help?
> > I fear that's going to make very long subject lines.
> >
> > It would probably be cleaner to update test_summary such that it adds
> > more info as part of $host
> > (as in "... testsuite on $host"), so that it grabs useful configure
> > parameters and runtestflags, however
> > this would be more controversial.
>
> Until a way to present summaries is available, would grouping
> the results of multiple runs in the same "basic configuration"
> (for some definition of basic) in the same post work for you?
>
That's not convenient for me at the moment: each build+make check runs
on a different server in a scratch area. It sends its results, saves
the logs and everything else is discarded.
After that I have a pass to compute regressions once all .sum are
available, and that's when I build the HTML reports you saw.
It's not terribly hard to reorganize, but it does require some work
and probably some disruption. I tend to try to make sure the reports
and results are still generated while I make changes to the scripts
:-)
In the meantime, I am updating the title format following the
suggestions from Richard & Jakub. Hopefully this will be in place
quite soon, after the currently-running validations have completed.
Thanks,
Christophe
> Martin
>
> >
> > Christophe
> >
> >> Martin
> >>
> >>>
> >>> I think HJ generates several "running targets" in the same log, I run
> >>> them separately to benefit from the compute farm I have access to.
> >>>
> >>> Christophe
> >>>
> >>>> Thanks
> >>>> Martin
> >>>>
> >>>> Results for 11.0.0 20200922 (experimental) [master revision
> >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >>>> arm-none-eabi Christophe LYON
> >>>>
> >>>> Results for 11.0.0 20200922 (experimental) [master revision
> >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >>>> arm-none-eabi Christophe LYON
> >>>> Results for 11.0.0 20200922 (experimental) [master revision
> >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >>>> arm-none-eabi Christophe LYON
> >>>> Results for 11.0.0 20200922 (experimental) [master revision
> >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >>>> arm-none-eabi Christophe LYON
> >>>> Results for 11.0.0 20200922 (experimental) [master revision
> >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >>>> arm-none-eabi Christophe LYON
> >>>> Results for 11.0.0 20200922 (experimental) [master revision
> >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >>>> arm-none-eabi Christophe LYON
> >>>> Results for 11.0.0 20200922 (experimental) [master revision
> >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >>>> arm-none-eabi Christophe LYON
> >>>> Results for 11.0.0 20200922 (experimental) [master revision
> >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >>>> arm-none-eabi Christophe LYON
> >>>> Results for 11.0.0 20200922 (experimental) [master revision
> >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> >>>> arm-none-eabi Christophe LYON
> >>
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 12:32 ` David Edelsohn
@ 2020-09-23 15:53 ` Christophe Lyon
2020-09-23 18:22 ` Jonathan Wakely
0 siblings, 1 reply; 18+ messages in thread
From: Christophe Lyon @ 2020-09-23 15:53 UTC (permalink / raw)
To: David Edelsohn; +Cc: Richard Earnshaw, Jakub Jelinek, Christophe Lyon via Gcc
On Wed, 23 Sep 2020 at 14:33, David Edelsohn <dje.gcc@gmail.com> wrote:
>
> On Wed, Sep 23, 2020 at 8:26 AM Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
> >
> > On Wed, 23 Sep 2020 at 12:26, Richard Earnshaw
> > <Richard.Earnshaw@foss.arm.com> wrote:
> > >
> > > On 23/09/2020 11:20, Jakub Jelinek via Gcc wrote:
> > > > On Wed, Sep 23, 2020 at 10:22:52AM +0100, Richard Sandiford wrote:
> > > >> So that would give:
> > > >>
> > > >> Results for 8.4.1 20200918 [r8-10517] on arm-none-linux-gnueabihf
> > > >>
> > > >> and hopefully free up some space at the end for the kind of thing
> > > >> you mention.
> > > >
> > > > Even that 8.4.1 20200918 is redundant, r8-10517 uniquely and shortly
> > > > identifies both the branch and commit.
> > > > So just
> > > > Results for r8-10517 on ...
> > > > and in ... also include something that uniquely identifies the
> > > > configuration.
> > > >
> > > > Jakub
> > > >
> > >
> > > I was thinking similarly, but then realised anyone using snapshots
> > > rather than git might not have that information.
> > >
> > > If that's not the case, however, then simplifying this would be a great
> > > idea.
> > >
> > > On the other hand, I use subject filters in my mail to steer results to
> > > a separate folder, so I do need some invariant key in the subject line
> > > that is sufficient to match without (too many) false positives.
> > >
> >
> > I always assumed there was a required format for the title/email
> > contents, is that documented somewhere?
> > There must be a smart filter to avoid spam, doesn't it require some
> > "keywords" in the title for instance?
> >
> > Same question for the gcc-regression list: is there a mandatory format?
>
> The format is generated by contrib/test_summary.
That's true for gcc-testresults, and I was wondering what would happen
if I modify test_summary? Does some mail-filter need fixing too?
Regarding gcc-regression, I think only Intel guys send messages there
(https://gcc.gnu.org/pipermail/gcc-regression/)
and they use different formats, hence I'm curious about the constraints.
>
> - David
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 15:53 ` Christophe Lyon
@ 2020-09-23 18:22 ` Jonathan Wakely
0 siblings, 0 replies; 18+ messages in thread
From: Jonathan Wakely @ 2020-09-23 18:22 UTC (permalink / raw)
To: Christophe Lyon
Cc: David Edelsohn, Jakub Jelinek, Christophe Lyon via Gcc, Richard Earnshaw
On Wed, 23 Sep 2020 at 16:55, Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
>
> On Wed, 23 Sep 2020 at 14:33, David Edelsohn <dje.gcc@gmail.com> wrote:
> >
> > On Wed, Sep 23, 2020 at 8:26 AM Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
> > >
> > > On Wed, 23 Sep 2020 at 12:26, Richard Earnshaw
> > > <Richard.Earnshaw@foss.arm.com> wrote:
> > > >
> > > > On 23/09/2020 11:20, Jakub Jelinek via Gcc wrote:
> > > > > On Wed, Sep 23, 2020 at 10:22:52AM +0100, Richard Sandiford wrote:
> > > > >> So that would give:
> > > > >>
> > > > >> Results for 8.4.1 20200918 [r8-10517] on arm-none-linux-gnueabihf
> > > > >>
> > > > >> and hopefully free up some space at the end for the kind of thing
> > > > >> you mention.
> > > > >
> > > > > Even that 8.4.1 20200918 is redundant, r8-10517 uniquely and shortly
> > > > > identifies both the branch and commit.
> > > > > So just
> > > > > Results for r8-10517 on ...
> > > > > and in ... also include something that uniquely identifies the
> > > > > configuration.
> > > > >
> > > > > Jakub
> > > > >
> > > >
> > > > I was thinking similarly, but then realised anyone using snapshots
> > > > rather than git might not have that information.
> > > >
> > > > If that's not the case, however, then simplifying this would be a great
> > > > idea.
> > > >
> > > > On the other hand, I use subject filters in my mail to steer results to
> > > > a separate folder, so I do need some invariant key in the subject line
> > > > that is sufficient to match without (too many) false positives.
> > > >
> > >
> > > I always assumed there was a required format for the title/email
> > > contents, is that documented somewhere?
> > > There must be a smart filter to avoid spam, doesn't it require some
> > > "keywords" in the title for instance?
> > >
> > > Same question for the gcc-regression list: is there a mandatory format?
> >
> > The format is generated by contrib/test_summary.
>
> That's true for gcc-testresults, and I was wondering what would happen
> if I modify test_summary? Does some mail-filter need fixing too?
There is no filter. Spam is handled by spamassassin, not hardcoded rules.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-23 15:50 ` Christophe Lyon
@ 2020-09-24 12:12 ` Christophe Lyon
2020-10-05 7:26 ` Christophe Lyon
0 siblings, 1 reply; 18+ messages in thread
From: Christophe Lyon @ 2020-09-24 12:12 UTC (permalink / raw)
To: Martin Sebor; +Cc: gcc mailing list
On Wed, 23 Sep 2020 at 17:50, Christophe Lyon
<christophe.lyon@linaro.org> wrote:
>
> On Wed, 23 Sep 2020 at 17:33, Martin Sebor <msebor@gmail.com> wrote:
> >
> > On 9/23/20 2:54 AM, Christophe Lyon wrote:
> > > On Wed, 23 Sep 2020 at 01:47, Martin Sebor <msebor@gmail.com> wrote:
> > >>
> > >> On 9/22/20 9:15 AM, Christophe Lyon wrote:
> > >>> On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
> > >>>>
> > >>>> Hi Christophe,
> > >>>>
> > >>>> While checking recent test results I noticed many posts with results
> > >>>> for various flavors of arm that at high level seem like duplicates
> > >>>> of one another.
> > >>>>
> > >>>> For example, the batch below all have the same title, but not all
> > >>>> of the contents are the same. The details (such as test failures)
> > >>>> on some of the pages are different.
> > >>>>
> > >>>> Can you help explain the differences? Is there a way to avoid
> > >>>> the duplication?
> > >>>>
> > >>>
> > >>> Sure, I am aware that many results look the same...
> > >>>
> > >>>
> > >>> If you look at the top of the report (~line 5), you'll see:
> > >>> Running target myarm-sim
> > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
> > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
> > >>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
> > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
> > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
> > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
> > >>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
> > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
> > >>>
> > >>> For all of these, the first line of the report is:
> > >>> LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
> > >>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
> > >>> TARGET=arm-none-eabi CPU=default FPU=default MODE=default
> > >>>
> > >>> I have other combinations where I override the configure flags, eg:
> > >>> LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
> > >>> r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
> > >>> TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
> > >>>
> > >>> I tried to see if I could fit something in the subject line, but that
> > >>> didn't seem convenient (would be too long, and I fear modifying the
> > >>> awk script....)
> > >>
> > >> Without some indication of a difference in the title there's no way
> > >> to know what result to look at, and checking all of them isn't really
> > >> practical. The duplication (and the sheer number of results) also
> > >> make it more difficult to find results for targets other than arm-*.
> > >> There are about 13,000 results for September and over 10,000 of those
> > >> for arm-* alone. It's good to have data but when there's this much
> > >> of it, and when the only form of presentation is as a running list,
> > >> it's too cumbersome to work with.
> > >>
> > >
> > > To help me track & report regressions, I build higher level reports like:
> > > https://people.linaro.org/~christophe.lyon/cross-validation/gcc/trunk/0latest/report-build-info.html
> > > where it's more obvious what configurations are tested.
> >
> > That looks awesome! The regression indicator looks especially
> > helpful. I really wish we had an overview like this for all
> > results. I've been thinking about writing a script to scrape
> > gcc-testresults and format an HTML table kind of like this for
> > years. With that, the number of posts sent to the list wouldn't
> > be a problem (at least not for those using the page). But it
> > would require settling on a standard format for the basic
> > parameters of each run.
> >
>
> It's probably easier to detect regressions and format reports from the
> .sum files rather than extracting them from the mailing-list.
> But your approach has the advantage that you can detect regressions
> from reports sent by other people, not only by you.
>
>
> > >
> > > Each line of such reports can send a message to gcc-testresults.
> > >
> > > I can control when such emails are sent, independently for each line:
> > > - never
> > > - for daily bump
> > > - for each validation
> > >
> > > So, I can easily reduce the amount of emails (by disabling them for
> > > some configurations),
> > > but that won't make the subject more informative.
> > > I included the short revision (rXX-YYYY) in the title to make it clearer.
> > >
> > > The number of configurations has grown over time because we regularly
> > > found regressions
> > > in configurations not tested previously.
> > >
> > > I can probably easily add the values of --with-cpu, --with-fpu,
> > > --with-mode and RUNTESTFLAGS
> > > as part of the [<branch> revision rXX-YYYY-ZZZZZ] string in the title,
> > > would that help?
> > > I fear that's going to make very long subject lines.
> > >
> > > It would probably be cleaner to update test_summary such that it adds
> > > more info as part of $host
> > > (as in "... testsuite on $host"), so that it grabs useful configure
> > > parameters and runtestflags, however
> > > this would be more controversial.
> >
> > Until a way to present summaries is available, would grouping
> > the results of multiple runs in the same "basic configuration"
> > (for some definition of basic) in the same post work for you?
> >
>
> That's not convenient for me at the moment: each build+make check runs
> on a different server in a scratch area. It sends its results, saves
> the logs and everything else is discarded.
> After that I have a pass to compute regressions once all .sum are
> available, and that's when I build the HTML reports you saw.
> It's not terribly hard to reorganize, but it does require some work
> and probably some disruption. I tend to try to make sure the reports
> and results are still generated while I make changes to the scripts
> :-)
>
> In the meantime, I am updating the title format following the
> suggestions from Richard & Jakub. Hopefully this will be in place
> quite soon, after the currently-running validations have completed.
>
I have updated my scripts, twice because I discovered that an empty
DEV-PHASE had a special meaning when constructing the revision
string....
So a few reports last night had just:
Results for 11.0.0 (GCC) testsuite on XXXX
as title, which can now be as long as:
Results for 8.4.1 [r8-10521 DEFMODE=arm DEFCPU=cortex-a9
DEFFPU=neon-fp16
TESTFLAGS=-mthumb/-march=armv8-a/-mfpu=crypto-neon-fp-armv8/-mfloat-
abi=hard] (GCC) testsuite on XXX
The first revision number (11.0.0 and 8.4.1 above) come from BASEVER,
but I didn't try to replace it with an empty as it seems many things
depend on it.
Does it help you?
Thanks,
Christophe
> Thanks,
>
> Christophe
>
> > Martin
> >
> > >
> > > Christophe
> > >
> > >> Martin
> > >>
> > >>>
> > >>> I think HJ generates several "running targets" in the same log, I run
> > >>> them separately to benefit from the compute farm I have access to.
> > >>>
> > >>> Christophe
> > >>>
> > >>>> Thanks
> > >>>> Martin
> > >>>>
> > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > >>>> arm-none-eabi Christophe LYON
> > >>>>
> > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > >>>> arm-none-eabi Christophe LYON
> > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > >>>> arm-none-eabi Christophe LYON
> > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > >>>> arm-none-eabi Christophe LYON
> > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > >>>> arm-none-eabi Christophe LYON
> > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > >>>> arm-none-eabi Christophe LYON
> > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > >>>> arm-none-eabi Christophe LYON
> > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > >>>> arm-none-eabi Christophe LYON
> > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > >>>> arm-none-eabi Christophe LYON
> > >>
> >
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: duplicate arm test results?
2020-09-24 12:12 ` Christophe Lyon
@ 2020-10-05 7:26 ` Christophe Lyon
0 siblings, 0 replies; 18+ messages in thread
From: Christophe Lyon @ 2020-10-05 7:26 UTC (permalink / raw)
To: Martin Sebor; +Cc: gcc mailing list
On Thu, 24 Sep 2020 at 14:12, Christophe Lyon
<christophe.lyon@linaro.org> wrote:
>
> On Wed, 23 Sep 2020 at 17:50, Christophe Lyon
> <christophe.lyon@linaro.org> wrote:
> >
> > On Wed, 23 Sep 2020 at 17:33, Martin Sebor <msebor@gmail.com> wrote:
> > >
> > > On 9/23/20 2:54 AM, Christophe Lyon wrote:
> > > > On Wed, 23 Sep 2020 at 01:47, Martin Sebor <msebor@gmail.com> wrote:
> > > >>
> > > >> On 9/22/20 9:15 AM, Christophe Lyon wrote:
> > > >>> On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
> > > >>>>
> > > >>>> Hi Christophe,
> > > >>>>
> > > >>>> While checking recent test results I noticed many posts with results
> > > >>>> for various flavors of arm that at high level seem like duplicates
> > > >>>> of one another.
> > > >>>>
> > > >>>> For example, the batch below all have the same title, but not all
> > > >>>> of the contents are the same. The details (such as test failures)
> > > >>>> on some of the pages are different.
> > > >>>>
> > > >>>> Can you help explain the differences? Is there a way to avoid
> > > >>>> the duplication?
> > > >>>>
> > > >>>
> > > >>> Sure, I am aware that many results look the same...
> > > >>>
> > > >>>
> > > >>> If you look at the top of the report (~line 5), you'll see:
> > > >>> Running target myarm-sim
> > > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
> > > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
> > > >>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
> > > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
> > > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
> > > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
> > > >>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
> > > >>> Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
> > > >>>
> > > >>> For all of these, the first line of the report is:
> > > >>> LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
> > > >>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
> > > >>> TARGET=arm-none-eabi CPU=default FPU=default MODE=default
> > > >>>
> > > >>> I have other combinations where I override the configure flags, eg:
> > > >>> LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
> > > >>> r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
> > > >>> TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
> > > >>>
> > > >>> I tried to see if I could fit something in the subject line, but that
> > > >>> didn't seem convenient (would be too long, and I fear modifying the
> > > >>> awk script....)
> > > >>
> > > >> Without some indication of a difference in the title there's no way
> > > >> to know what result to look at, and checking all of them isn't really
> > > >> practical. The duplication (and the sheer number of results) also
> > > >> make it more difficult to find results for targets other than arm-*.
> > > >> There are about 13,000 results for September and over 10,000 of those
> > > >> for arm-* alone. It's good to have data but when there's this much
> > > >> of it, and when the only form of presentation is as a running list,
> > > >> it's too cumbersome to work with.
> > > >>
> > > >
> > > > To help me track & report regressions, I build higher level reports like:
> > > > https://people.linaro.org/~christophe.lyon/cross-validation/gcc/trunk/0latest/report-build-info.html
> > > > where it's more obvious what configurations are tested.
> > >
> > > That looks awesome! The regression indicator looks especially
> > > helpful. I really wish we had an overview like this for all
> > > results. I've been thinking about writing a script to scrape
> > > gcc-testresults and format an HTML table kind of like this for
> > > years. With that, the number of posts sent to the list wouldn't
> > > be a problem (at least not for those using the page). But it
> > > would require settling on a standard format for the basic
> > > parameters of each run.
> > >
> >
> > It's probably easier to detect regressions and format reports from the
> > .sum files rather than extracting them from the mailing-list.
> > But your approach has the advantage that you can detect regressions
> > from reports sent by other people, not only by you.
> >
> >
> > > >
> > > > Each line of such reports can send a message to gcc-testresults.
> > > >
> > > > I can control when such emails are sent, independently for each line:
> > > > - never
> > > > - for daily bump
> > > > - for each validation
> > > >
> > > > So, I can easily reduce the amount of emails (by disabling them for
> > > > some configurations),
> > > > but that won't make the subject more informative.
> > > > I included the short revision (rXX-YYYY) in the title to make it clearer.
> > > >
> > > > The number of configurations has grown over time because we regularly
> > > > found regressions
> > > > in configurations not tested previously.
> > > >
> > > > I can probably easily add the values of --with-cpu, --with-fpu,
> > > > --with-mode and RUNTESTFLAGS
> > > > as part of the [<branch> revision rXX-YYYY-ZZZZZ] string in the title,
> > > > would that help?
> > > > I fear that's going to make very long subject lines.
> > > >
> > > > It would probably be cleaner to update test_summary such that it adds
> > > > more info as part of $host
> > > > (as in "... testsuite on $host"), so that it grabs useful configure
> > > > parameters and runtestflags, however
> > > > this would be more controversial.
> > >
> > > Until a way to present summaries is available, would grouping
> > > the results of multiple runs in the same "basic configuration"
> > > (for some definition of basic) in the same post work for you?
> > >
> >
> > That's not convenient for me at the moment: each build+make check runs
> > on a different server in a scratch area. It sends its results, saves
> > the logs and everything else is discarded.
> > After that I have a pass to compute regressions once all .sum are
> > available, and that's when I build the HTML reports you saw.
> > It's not terribly hard to reorganize, but it does require some work
> > and probably some disruption. I tend to try to make sure the reports
> > and results are still generated while I make changes to the scripts
> > :-)
> >
> > In the meantime, I am updating the title format following the
> > suggestions from Richard & Jakub. Hopefully this will be in place
> > quite soon, after the currently-running validations have completed.
> >
>
> I have updated my scripts, twice because I discovered that an empty
> DEV-PHASE had a special meaning when constructing the revision
> string....
>
> So a few reports last night had just:
> Results for 11.0.0 (GCC) testsuite on XXXX
> as title, which can now be as long as:
> Results for 8.4.1 [r8-10521 DEFMODE=arm DEFCPU=cortex-a9
> DEFFPU=neon-fp16
> TESTFLAGS=-mthumb/-march=armv8-a/-mfpu=crypto-neon-fp-armv8/-mfloat-
> abi=hard] (GCC) testsuite on XXX
>
> The first revision number (11.0.0 and 8.4.1 above) come from BASEVER,
> but I didn't try to replace it with an empty as it seems many things
> depend on it.
>
> Does it help you?
>
Hi,
To drastically reduce the traffic on the list, I've switched to
sending results only for "Daily bump" builds on master.
For the time being I'm still sending results for every commit (x
number of tested configurations) on release branches.
Christophe
> Thanks,
>
> Christophe
>
>
> > Thanks,
> >
> > Christophe
> >
> > > Martin
> > >
> > > >
> > > > Christophe
> > > >
> > > >> Martin
> > > >>
> > > >>>
> > > >>> I think HJ generates several "running targets" in the same log, I run
> > > >>> them separately to benefit from the compute farm I have access to.
> > > >>>
> > > >>> Christophe
> > > >>>
> > > >>>> Thanks
> > > >>>> Martin
> > > >>>>
> > > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > > >>>> arm-none-eabi Christophe LYON
> > > >>>>
> > > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > > >>>> arm-none-eabi Christophe LYON
> > > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > > >>>> arm-none-eabi Christophe LYON
> > > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > > >>>> arm-none-eabi Christophe LYON
> > > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > > >>>> arm-none-eabi Christophe LYON
> > > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > > >>>> arm-none-eabi Christophe LYON
> > > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > > >>>> arm-none-eabi Christophe LYON
> > > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > > >>>> arm-none-eabi Christophe LYON
> > > >>>> Results for 11.0.0 20200922 (experimental) [master revision
> > > >>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
> > > >>>> arm-none-eabi Christophe LYON
> > > >>
> > >
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2020-10-05 7:26 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-22 15:02 duplicate arm test results? Martin Sebor
2020-09-22 15:15 ` Christophe Lyon
2020-09-22 23:47 ` Martin Sebor
2020-09-23 8:54 ` Christophe Lyon
2020-09-23 9:22 ` Richard Sandiford
2020-09-23 10:20 ` Jakub Jelinek
2020-09-23 10:26 ` Richard Earnshaw
2020-09-23 12:25 ` Christophe Lyon
2020-09-23 12:32 ` David Edelsohn
2020-09-23 15:53 ` Christophe Lyon
2020-09-23 18:22 ` Jonathan Wakely
2020-09-23 10:37 ` Andreas Schwab
2020-09-23 10:43 ` Jakub Jelinek
2020-09-23 10:52 ` Andreas Schwab
2020-09-23 15:33 ` Martin Sebor
2020-09-23 15:50 ` Christophe Lyon
2020-09-24 12:12 ` Christophe Lyon
2020-10-05 7:26 ` Christophe Lyon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).