* Re: [testsuite] skip ARM tests with conflicting options
2011-06-08 11:47 ` Richard Earnshaw
@ 2011-06-08 16:13 ` Janis Johnson
2011-06-08 19:34 ` Mike Stump
2011-06-08 19:39 ` Mike Stump
2011-06-10 0:11 ` Janis Johnson
2 siblings, 1 reply; 14+ messages in thread
From: Janis Johnson @ 2011-06-08 16:13 UTC (permalink / raw)
To: Richard Earnshaw; +Cc: Mike Stump, Joseph S. Myers, gcc-patches
On 06/08/2011 03:39 AM, Richard Earnshaw wrote:
> On 08/06/11 03:14, Janis Johnson wrote:
>> On 06/07/2011 06:25 PM, Mike Stump wrote:
>>> On Jun 7, 2011, at 4:24 PM, Janis Johnson wrote:
>>>> On 06/07/2011 02:07 PM, Joseph S. Myers wrote:
>>>>> On Tue, 7 Jun 2011, Janis Johnson wrote:
>>>>>
>>>>>> Several tests in gcc.target/arm use dg-options with -mcpu=xxxx, which
>>>>>> causes compiler warnings or errors when the multilib flags include
>>>>>> -march=yyyy. This patch causes those tests to be skipped. It also
>>>>>> prevents gcc.target/arm/20090811-1.c from running with multilibs that
>>>>>> would override -mcpu or -mfloat-abi options specified for the test.
>>>>>
>>>>> I think you should allow compatible -march options - for example, if
>>>>> dg-options has -mcpu=cortex-a8, allow -march=armv7-a but disallow all
>>>>> other -march options.
>>>>>
>>>> Is this one OK?
>>>
>>> Not sure if the arm people want to review this or would rather I review it...
>>>
>>> Let's give the arm folks a couple days to comment, if no objections, Ok.
>>>
>>> A point of warning, eventually, you'll discover that when a compiler defaults to the argument you want to skip, that you'll needs slightly more power to skip them. darwin ran into this with things like -m64, and eventually had to do something like lp64. configure options like --with-cpu=arm9 are the sort that can change the default.
>>
>> Yes, I hope to hear from ARM people.
>>
>> On ARM, the default from --with-cpu= is overridden by -march at
>> compile so it's not a problem for this particular set of tests.
>> As I said originally, this set is the tip of the iceberg and they
>> get more difficult.
>>
>> Janis
>>
>>
>
>
> I'm worried by this whole approach of command-line checking. It works,
> just about, for testsuite variations set with target_list, but it won't
> work with options used to configure the compiler (eg --with-mode=thumb,
> or --with-cpu=...). Perhaps a better approach would be a new dg- test
> that built a trivial file with all the options and disabled the test if
> that test failed for any reason. Something like:
>
> dg-target-compatible (target, <compile|link>, additional-opts)
>
> The test is only performed if target matches the current target.
>
> I'm not sure if this is something that can be easily cached (well, it
> might be possible if you could index off additional-opts and the default
> opts), so it might be that this test has to be re-run every time there
> is a test that needs it.
>
> R.
There's a similar functionality now with effective targets that end with
"_ok"; they test with options that would be added with a later directive.
The problem is that they can't be specific enough for what the test is
looking for; arm_neon_fp16_ok, for example, currently passes when the
multilib options include -mfpu=neon, which overrides the options provided
by dg-options, and that leads to problems.
I think that a test that requires a specific option should provide that
option, but be skipped if multilib options include a conflict. This
includes -mcpu, -mfpu, -march, -mfloat-abi, -mbig-endian, and possibly
more. Those options override the defaults for the configuration, so we
shouldn't have to worry about the defaults.
The big question is whether such a test should be run for all multilibs
that might possibly pass the test, or only for default and for mulitlibs
that provide the same options. There are a lot of arm tests that use
-march but pass for a great many other -march options. In most cases
they use -march with the value for which the problem was reported.
Should those tests be run for all multilibs, with the multilib options
overriding the "defaults" for the test from dg-options, or should they
be skipped multilibs that use other values? The answer might depend on
the individual test; maybe some should be run for a large number of
multilib options to find problems with specific ones, while others can
be limited to run just once or a few times.
Janis
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [testsuite] skip ARM tests with conflicting options
2011-06-08 16:13 ` Janis Johnson
@ 2011-06-08 19:34 ` Mike Stump
2011-06-08 20:51 ` Janis Johnson
0 siblings, 1 reply; 14+ messages in thread
From: Mike Stump @ 2011-06-08 19:34 UTC (permalink / raw)
To: Janis Johnson; +Cc: Richard Earnshaw, Joseph S. Myers, gcc-patches
On Jun 8, 2011, at 8:28 AM, Janis Johnson wrote:
> The big question is whether such a test should be run for all multilibs
> that might possibly pass the test, or only for default and for mulitlibs
> that provide the same options.
Here, reasonable people may disagree. I suspect in the end, we'll have both solutions, and then individual testcases will make their own decision. A collection of testcases will tend to follow the same convention... So, for objective-c, we face the same sort of issue, and we do what we do, and that isn't necessarily going to match exactly what for example the gcc.arm does, nor I suspect are we going to change just because gcc.arm changes. I think it makes sense to cache as much as possible and skip conflicts. Taking off my testsuite maintainer hat, I think soft conflicts with defaults should mean we run it, and punch in the options we want. If there is something that prohibits that from working (hard conflict), it should be skipped. Feel free to ignore this, as I don't know that this is the best answer.
I'd like to think that dg-skip-if and dg-require-effective-target and general target selection is beefy enough to do everything we need it to, or can be made to.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [testsuite] skip ARM tests with conflicting options
2011-06-08 19:34 ` Mike Stump
@ 2011-06-08 20:51 ` Janis Johnson
0 siblings, 0 replies; 14+ messages in thread
From: Janis Johnson @ 2011-06-08 20:51 UTC (permalink / raw)
To: Mike Stump; +Cc: Richard Earnshaw, Joseph S. Myers, gcc-patches
On 06/08/2011 12:30 PM, Mike Stump wrote:
> On Jun 8, 2011, at 8:28 AM, Janis Johnson wrote:
>> The big question is whether such a test should be run for all multilibs
>> that might possibly pass the test, or only for default and for mulitlibs
>> that provide the same options.
>
> Here, reasonable people may disagree. I suspect in the end, we'll have
> both solutions, and then individual testcases will make their own decision.
> A collection of testcases will tend to follow the same convention... So, for
> objective-c, we face the same sort of issue, and we do what we do, and that isn't
> necessarily going to match exactly what for example the gcc.arm does, nor I suspect
> are we going to change just because gcc.arm changes. I think it makes sense to
> cache as much as possible and skip conflicts. Taking off my testsuite maintainer
> hat, I think soft conflicts with defaults should mean we run it, and punch in the
> options we want. If there is something that prohibits that from working (hard
> conflict), it should be skipped. Feel free to ignore this, as I don't know that
> this is the best answer.
I agree that the answer will be different for different tests.
The problem is that in the case of a "soft conflict", the multilib options
go at the end of the compile line and override the options given in the
test via dg-options. That's OK if dg-options is providing defaults for when
there is no similar option in the multilib options, but a problem if the test
depends on the flags from dg-options being used, as when a dg-final checks
for specific code generation. Then we have the choice of running the test
only with the specific values specified in the test, or allowing a range of
values, for mfpu or march or whatever; that gets trickier but we have the
tools to do it.
> I'd like to think that dg-skip-if and dg-require-effective-target and general
> target selection is beefy enough to do everything we need it to, or can be made to.
Right, it's easy to add new effective targets, I don't think we need new test
directives.
Janis
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [testsuite] skip ARM tests with conflicting options
2011-06-08 11:47 ` Richard Earnshaw
2011-06-08 16:13 ` Janis Johnson
@ 2011-06-08 19:39 ` Mike Stump
2011-06-10 0:11 ` Janis Johnson
2 siblings, 0 replies; 14+ messages in thread
From: Mike Stump @ 2011-06-08 19:39 UTC (permalink / raw)
To: Richard Earnshaw; +Cc: Janis Johnson, Joseph S. Myers, gcc-patches
On Jun 8, 2011, at 3:39 AM, Richard Earnshaw wrote:
> I'm worried by this whole approach of command-line checking.
Right, and this was essentially my point originally. Luckily there is enough beef I think behind the curtains to do everything that needs doing without worrying about adding yet more infrastructure.
> Perhaps a better approach would be a new dg- test
A new mechanism would be necessary if the existing ones turn out to be poor or unworkable. To be concrete, could you give a specific example of something that doesn't work well with the existing mechanisms?
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [testsuite] skip ARM tests with conflicting options
2011-06-08 11:47 ` Richard Earnshaw
2011-06-08 16:13 ` Janis Johnson
2011-06-08 19:39 ` Mike Stump
@ 2011-06-10 0:11 ` Janis Johnson
2011-06-10 10:12 ` Richard Earnshaw
2 siblings, 1 reply; 14+ messages in thread
From: Janis Johnson @ 2011-06-10 0:11 UTC (permalink / raw)
To: Richard Earnshaw; +Cc: Mike Stump, Joseph S. Myers, gcc-patches
On 06/08/2011 03:39 AM, Richard Earnshaw wrote:
> On 08/06/11 03:14, Janis Johnson wrote:
>> On 06/07/2011 06:25 PM, Mike Stump wrote:
>>> On Jun 7, 2011, at 4:24 PM, Janis Johnson wrote:
>>>> On 06/07/2011 02:07 PM, Joseph S. Myers wrote:
>>>>> On Tue, 7 Jun 2011, Janis Johnson wrote:
>>>>>
>>>>>> Several tests in gcc.target/arm use dg-options with -mcpu=xxxx, which
>>>>>> causes compiler warnings or errors when the multilib flags include
>>>>>> -march=yyyy. This patch causes those tests to be skipped. It also
>>>>>> prevents gcc.target/arm/20090811-1.c from running with multilibs that
>>>>>> would override -mcpu or -mfloat-abi options specified for the test.
>>>>>
>>>>> I think you should allow compatible -march options - for example, if
>>>>> dg-options has -mcpu=cortex-a8, allow -march=armv7-a but disallow all
>>>>> other -march options.
>>>>>
>>>> Is this one OK?
>>>
>>> Not sure if the arm people want to review this or would rather I review it...
>>>
>>> Let's give the arm folks a couple days to comment, if no objections, Ok.
>>>
>>> A point of warning, eventually, you'll discover that when a compiler defaults to the argument you want to skip, that you'll needs slightly more power to skip them. darwin ran into this with things like -m64, and eventually had to do something like lp64. configure options like --with-cpu=arm9 are the sort that can change the default.
>>
>> Yes, I hope to hear from ARM people.
>>
>> On ARM, the default from --with-cpu= is overridden by -march at
>> compile so it's not a problem for this particular set of tests.
>> As I said originally, this set is the tip of the iceberg and they
>> get more difficult.
>>
>> Janis
>>
>>
>
>
> I'm worried by this whole approach of command-line checking. It works,
> just about, for testsuite variations set with target_list, but it won't
> work with options used to configure the compiler (eg --with-mode=thumb,
> or --with-cpu=...). Perhaps a better approach would be a new dg- test
> that built a trivial file with all the options and disabled the test if
> that test failed for any reason. Something like:
>
> dg-target-compatible (target, <compile|link>, additional-opts)
>
> The test is only performed if target matches the current target.
>
> I'm not sure if this is something that can be easily cached (well, it
> might be possible if you could index off additional-opts and the default
> opts), so it might be that this test has to be re-run every time there
> is a test that needs it.
>
> R.
As I said earlier, the testsuite already provides effective-target support
that checks for complicated things and it's easy enough to add more.
Given that there are lots of spurious failures right now due to tests
being compiled with conflicting options, or during dg-final checks when the
expected code isn't being generated because multilib options override the
ones given in the test, is this patch to skip tests using -mcpu if the
multilibs use -march OK? I have made the change Joseph suggested to allow
-march values that do not conflict.
Expect lots more of these patches, with perhaps different solutions for
different tests.
Janis
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [testsuite] skip ARM tests with conflicting options
2011-06-10 0:11 ` Janis Johnson
@ 2011-06-10 10:12 ` Richard Earnshaw
2011-06-10 16:22 ` Janis Johnson
0 siblings, 1 reply; 14+ messages in thread
From: Richard Earnshaw @ 2011-06-10 10:12 UTC (permalink / raw)
To: Janis Johnson; +Cc: Mike Stump, Joseph S. Myers, gcc-patches
On 10/06/11 01:04, Janis Johnson wrote:
> On 06/08/2011 03:39 AM, Richard Earnshaw wrote:
>> On 08/06/11 03:14, Janis Johnson wrote:
>>> On 06/07/2011 06:25 PM, Mike Stump wrote:
>>>> On Jun 7, 2011, at 4:24 PM, Janis Johnson wrote:
>>>>> On 06/07/2011 02:07 PM, Joseph S. Myers wrote:
>>>>>> On Tue, 7 Jun 2011, Janis Johnson wrote:
>>>>>>
>>>>>>> Several tests in gcc.target/arm use dg-options with -mcpu=xxxx, which
>>>>>>> causes compiler warnings or errors when the multilib flags include
>>>>>>> -march=yyyy. This patch causes those tests to be skipped. It also
>>>>>>> prevents gcc.target/arm/20090811-1.c from running with multilibs that
>>>>>>> would override -mcpu or -mfloat-abi options specified for the test.
>>>>>>
>>>>>> I think you should allow compatible -march options - for example, if
>>>>>> dg-options has -mcpu=cortex-a8, allow -march=armv7-a but disallow all
>>>>>> other -march options.
>>>>>>
>>>>> Is this one OK?
>>>>
>>>> Not sure if the arm people want to review this or would rather I review it...
>>>>
>>>> Let's give the arm folks a couple days to comment, if no objections, Ok.
>>>>
>>>> A point of warning, eventually, you'll discover that when a compiler defaults to the argument you want to skip, that you'll needs slightly more power to skip them. darwin ran into this with things like -m64, and eventually had to do something like lp64. configure options like --with-cpu=arm9 are the sort that can change the default.
>>>
>>> Yes, I hope to hear from ARM people.
>>>
>>> On ARM, the default from --with-cpu= is overridden by -march at
>>> compile so it's not a problem for this particular set of tests.
>>> As I said originally, this set is the tip of the iceberg and they
>>> get more difficult.
>>>
>>> Janis
>>>
>>>
>>
>>
>> I'm worried by this whole approach of command-line checking. It works,
>> just about, for testsuite variations set with target_list, but it won't
>> work with options used to configure the compiler (eg --with-mode=thumb,
>> or --with-cpu=...). Perhaps a better approach would be a new dg- test
>> that built a trivial file with all the options and disabled the test if
>> that test failed for any reason. Something like:
>>
>> dg-target-compatible (target, <compile|link>, additional-opts)
>>
>> The test is only performed if target matches the current target.
>>
>> I'm not sure if this is something that can be easily cached (well, it
>> might be possible if you could index off additional-opts and the default
>> opts), so it might be that this test has to be re-run every time there
>> is a test that needs it.
>>
>> R.
>
> As I said earlier, the testsuite already provides effective-target support
> that checks for complicated things and it's easy enough to add more.
>
> Given that there are lots of spurious failures right now due to tests
> being compiled with conflicting options, or during dg-final checks when the
> expected code isn't being generated because multilib options override the
> ones given in the test, is this patch to skip tests using -mcpu if the
> multilibs use -march OK? I have made the change Joseph suggested to allow
> -march values that do not conflict.
>
> Expect lots more of these patches, with perhaps different solutions for
> different tests.
>
Don't get me wrong, I wasn't objecting to the patch as it is (it's a
useful step forward). I'm just raising a concern that I don't think
it's a total solution.
Janice, I haven't done it, but a compiler built with --with-mode=thumb
would be a useful test to run.
R.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [testsuite] skip ARM tests with conflicting options
2011-06-10 10:12 ` Richard Earnshaw
@ 2011-06-10 16:22 ` Janis Johnson
0 siblings, 0 replies; 14+ messages in thread
From: Janis Johnson @ 2011-06-10 16:22 UTC (permalink / raw)
To: Richard Earnshaw; +Cc: Mike Stump, Joseph S. Myers, gcc-patches
On 06/10/2011 03:00 AM, Richard Earnshaw wrote:
> On 10/06/11 01:04, Janis Johnson wrote:
>> On 06/08/2011 03:39 AM, Richard Earnshaw wrote:
>>> On 08/06/11 03:14, Janis Johnson wrote:
>>>> On 06/07/2011 06:25 PM, Mike Stump wrote:
>>>>> On Jun 7, 2011, at 4:24 PM, Janis Johnson wrote:
>>>>>> On 06/07/2011 02:07 PM, Joseph S. Myers wrote:
>>>>>>> On Tue, 7 Jun 2011, Janis Johnson wrote:
>>>>>>>
>>>>>>>> Several tests in gcc.target/arm use dg-options with -mcpu=xxxx, which
>>>>>>>> causes compiler warnings or errors when the multilib flags include
>>>>>>>> -march=yyyy. This patch causes those tests to be skipped. It also
>>>>>>>> prevents gcc.target/arm/20090811-1.c from running with multilibs that
>>>>>>>> would override -mcpu or -mfloat-abi options specified for the test.
>>>>>>>
>>>>>>> I think you should allow compatible -march options - for example, if
>>>>>>> dg-options has -mcpu=cortex-a8, allow -march=armv7-a but disallow all
>>>>>>> other -march options.
>>>>>>>
>>>>>> Is this one OK?
>>>>>
>>>>> Not sure if the arm people want to review this or would rather I review it...
>>>>>
>>>>> Let's give the arm folks a couple days to comment, if no objections, Ok.
>>>>>
>>>>> A point of warning, eventually, you'll discover that when a compiler defaults to the argument you want to skip, that you'll needs slightly more power to skip them. darwin ran into this with things like -m64, and eventually had to do something like lp64. configure options like --with-cpu=arm9 are the sort that can change the default.
>>>>
>>>> Yes, I hope to hear from ARM people.
>>>>
>>>> On ARM, the default from --with-cpu= is overridden by -march at
>>>> compile so it's not a problem for this particular set of tests.
>>>> As I said originally, this set is the tip of the iceberg and they
>>>> get more difficult.
>>>>
>>>> Janis
>>>>
>>>>
>>>
>>>
>>> I'm worried by this whole approach of command-line checking. It works,
>>> just about, for testsuite variations set with target_list, but it won't
>>> work with options used to configure the compiler (eg --with-mode=thumb,
>>> or --with-cpu=...). Perhaps a better approach would be a new dg- test
>>> that built a trivial file with all the options and disabled the test if
>>> that test failed for any reason. Something like:
>>>
>>> dg-target-compatible (target, <compile|link>, additional-opts)
>>>
>>> The test is only performed if target matches the current target.
>>>
>>> I'm not sure if this is something that can be easily cached (well, it
>>> might be possible if you could index off additional-opts and the default
>>> opts), so it might be that this test has to be re-run every time there
>>> is a test that needs it.
>>>
>>> R.
>>
>> As I said earlier, the testsuite already provides effective-target support
>> that checks for complicated things and it's easy enough to add more.
>>
>> Given that there are lots of spurious failures right now due to tests
>> being compiled with conflicting options, or during dg-final checks when the
>> expected code isn't being generated because multilib options override the
>> ones given in the test, is this patch to skip tests using -mcpu if the
>> multilibs use -march OK? I have made the change Joseph suggested to allow
>> -march values that do not conflict.
>>
>> Expect lots more of these patches, with perhaps different solutions for
>> different tests.
>>
>
> Don't get me wrong, I wasn't objecting to the patch as it is (it's a
> useful step forward). I'm just raising a concern that I don't think
> it's a total solution.
It isn't.
> Janice, I haven't done it, but a compiler built with --with-mode=thumb
> would be a useful test to run.
>
> R.
>
There's effective-target support for the following:
arm32
arm_vfp_ok
arm_hard_vfp_ok
arm_thumb1_ok
arm_thumb2_ok
arm_neon_ok
arm_neon_fp16_ok
arm_iwmmxt_ok
They check that adding appropriate flags (to be added by dg-options
later in the test) results in expected macros being defined and no
compiler errors or warnings. Some need additional checks for flags
from multilibs that will override options to be used in tests, as with
my changes for arm_neon_fp16_ok.
If there are tests that fail now with a compiler configured using
--with-mode=thumb because they expect arm mode then the tests should
probably be checking that arm mode is used (arm32 does that) or using
-marm.
In case it isn't obvious, I'm still learning what all the ARM options
are and what they mean, so I might say some really dumb things and hope
that you guys will set me straight.
Janis
^ permalink raw reply [flat|nested] 14+ messages in thread