Re: [PATCH] Add compile testing to glibc test framework.

[Florian Weimer <fweimer@redhat.com>]

On 06/17/2016 03:51 PM, Carlos O'Donell wrote:
> On 06/17/2016 07:27 AM, Florian Weimer wrote:
>> On 06/07/2016 07:28 PM, Joseph Myers wrote:
>>> On Tue, 7 Jun 2016, Carlos O'Donell wrote:
>>>
>>>> Previously these kinds of tests would result in the testsuite
>>>> failing to run to completion because compiler errors are treated
>>>> as harness failures.
>>>
>>> I don't actually see this as a problem - that is, I don't see why
>>> any compile failure should be hard to fix for some system-specific
>>> reason. I'd rather just add such tests as normal tests, that break
>>> the build if they fail.
>>
>> What's annoying with that is that you lose test summary reporting.
>> If this was somehow fixed (without impacting make running times,
>> please), I think there wouldn't much of an incentive to add such
>> compile-time testing.
>
> What would you like fixed? Summary reporting of failed compile
> and link?

Yes, and the existing reports for tests that ran.

> That's an interesting solution, wrap the compiles and link with
> something like evaluate-test.sh but designed to capture failed
> compiles and failed links and report them properly as a new
> kind of failed test code?

Something like that, yes.

If anyone wants to work on this: Potential future directions in this 
area involve compiling all tests -static, as PIE, as PIC, with Address 
Sanitizer, and so on, and also run them under valgrind.

(I also see considerable value in separating the test suite from the 
glibc source body, with the goal that you can compile the tests on an 
older glibc release, and check that they still work with the current 
version, but that is probably a different conversation.)

Florian