From: Martin Sebor <msebor@gmail.com>
To: Christophe Lyon <christophe.lyon@linaro.org>
Cc: gcc mailing list <gcc@gcc.gnu.org>
Subject: Re: duplicate arm test results?
Date: Wed, 23 Sep 2020 09:33:53 -0600 [thread overview]
Message-ID: <f68bdba4-cf72-ff4f-6989-8128a7b82f3e@gmail.com> (raw)
In-Reply-To: <CAKdteObucXOan9pD5WG-AaBVJ0VYgzZ4z73JHAzsgwm=jfH4xA@mail.gmail.com>
On 9/23/20 2:54 AM, Christophe Lyon wrote:
> On Wed, 23 Sep 2020 at 01:47, Martin Sebor <msebor@gmail.com> wrote:
>>
>> On 9/22/20 9:15 AM, Christophe Lyon wrote:
>>> On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
>>>>
>>>> Hi Christophe,
>>>>
>>>> While checking recent test results I noticed many posts with results
>>>> for various flavors of arm that at high level seem like duplicates
>>>> of one another.
>>>>
>>>> For example, the batch below all have the same title, but not all
>>>> of the contents are the same. The details (such as test failures)
>>>> on some of the pages are different.
>>>>
>>>> Can you help explain the differences? Is there a way to avoid
>>>> the duplication?
>>>>
>>>
>>> Sure, I am aware that many results look the same...
>>>
>>>
>>> If you look at the top of the report (~line 5), you'll see:
>>> Running target myarm-sim
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
>>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
>>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
>>>
>>> For all of these, the first line of the report is:
>>> LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
>>> TARGET=arm-none-eabi CPU=default FPU=default MODE=default
>>>
>>> I have other combinations where I override the configure flags, eg:
>>> LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
>>> r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
>>> TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
>>>
>>> I tried to see if I could fit something in the subject line, but that
>>> didn't seem convenient (would be too long, and I fear modifying the
>>> awk script....)
>>
>> Without some indication of a difference in the title there's no way
>> to know what result to look at, and checking all of them isn't really
>> practical. The duplication (and the sheer number of results) also
>> make it more difficult to find results for targets other than arm-*.
>> There are about 13,000 results for September and over 10,000 of those
>> for arm-* alone. It's good to have data but when there's this much
>> of it, and when the only form of presentation is as a running list,
>> it's too cumbersome to work with.
>>
>
> To help me track & report regressions, I build higher level reports like:
> https://people.linaro.org/~christophe.lyon/cross-validation/gcc/trunk/0latest/report-build-info.html
> where it's more obvious what configurations are tested.
That looks awesome! The regression indicator looks especially
helpful. I really wish we had an overview like this for all
results. I've been thinking about writing a script to scrape
gcc-testresults and format an HTML table kind of like this for
years. With that, the number of posts sent to the list wouldn't
be a problem (at least not for those using the page). But it
would require settling on a standard format for the basic
parameters of each run.
>
> Each line of such reports can send a message to gcc-testresults.
>
> I can control when such emails are sent, independently for each line:
> - never
> - for daily bump
> - for each validation
>
> So, I can easily reduce the amount of emails (by disabling them for
> some configurations),
> but that won't make the subject more informative.
> I included the short revision (rXX-YYYY) in the title to make it clearer.
>
> The number of configurations has grown over time because we regularly
> found regressions
> in configurations not tested previously.
>
> I can probably easily add the values of --with-cpu, --with-fpu,
> --with-mode and RUNTESTFLAGS
> as part of the [<branch> revision rXX-YYYY-ZZZZZ] string in the title,
> would that help?
> I fear that's going to make very long subject lines.
>
> It would probably be cleaner to update test_summary such that it adds
> more info as part of $host
> (as in "... testsuite on $host"), so that it grabs useful configure
> parameters and runtestflags, however
> this would be more controversial.
Until a way to present summaries is available, would grouping
the results of multiple runs in the same "basic configuration"
(for some definition of basic) in the same post work for you?
Martin
>
> Christophe
>
>> Martin
>>
>>>
>>> I think HJ generates several "running targets" in the same log, I run
>>> them separately to benefit from the compute farm I have access to.
>>>
>>> Christophe
>>>
>>>> Thanks
>>>> Martin
>>>>
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>>
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi Christophe LYON
>>
next prev parent reply other threads:[~2020-09-23 15:33 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-22 15:02 Martin Sebor
2020-09-22 15:15 ` Christophe Lyon
2020-09-22 23:47 ` Martin Sebor
2020-09-23 8:54 ` Christophe Lyon
2020-09-23 9:22 ` Richard Sandiford
2020-09-23 10:20 ` Jakub Jelinek
2020-09-23 10:26 ` Richard Earnshaw
2020-09-23 12:25 ` Christophe Lyon
2020-09-23 12:32 ` David Edelsohn
2020-09-23 15:53 ` Christophe Lyon
2020-09-23 18:22 ` Jonathan Wakely
2020-09-23 10:37 ` Andreas Schwab
2020-09-23 10:43 ` Jakub Jelinek
2020-09-23 10:52 ` Andreas Schwab
2020-09-23 15:33 ` Martin Sebor [this message]
2020-09-23 15:50 ` Christophe Lyon
2020-09-24 12:12 ` Christophe Lyon
2020-10-05 7:26 ` Christophe Lyon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f68bdba4-cf72-ff4f-6989-8128a7b82f3e@gmail.com \
--to=msebor@gmail.com \
--cc=christophe.lyon@linaro.org \
--cc=gcc@gcc.gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).