[Bug libc/14233] New: Many of tst-*.c lack any description of the assertions they test

[Bug libc/14233] New: Many of tst-*.c lack any description of the assertions they test

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=14233

             Bug #: 14233
           Summary: Many of tst-*.c lack any description of the assertions
                    they test
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: libc
        AssignedTo: unassigned@sourceware.org
        ReportedBy: bugdal@aerifal.cx
                CC: drepper.fsp@gmail.com
    Classification: Unclassified


Browsing some of the tst-*.c files in glibc, I noticed that many lack any
comments about what requirement they're testing for. I'm not sure if this is a
"bug" strictly speaking, but it seems problematic in that:

1. It's difficult to audit/debug them and determine if they're actually testing
for what they were intended to test. This is especially problematic for tests
that involve (either as the subject of the test or as a side-difficulty of
testing) race conditions, where it may not be at all obvious where the races
are.
2. If a change introduces a bug, it may be hard to determine the cause without
reverse engineering the test to see what it was testing.
3. It would be nice to know if tests are asserting behavior required by the
standards, behavior required as part of the documented or unofficial
specification of glibc, undocumented behavior that known applications expect,
and/or quality of implementation issues.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/14233] Many of tst-*.c lack any description of the assertions they test

[joseph at codesourcery dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14233

--- Comment #1 from joseph at codesourcery dot com <joseph at codesourcery dot com> 2012-06-14 18:44:45 UTC ---
On Thu, 14 Jun 2012, bugdal at aerifal dot cx wrote:

> Browsing some of the tst-*.c files in glibc, I noticed that many lack any
> comments about what requirement they're testing for. I'm not sure if this is a
> "bug" strictly speaking, but it seems problematic in that:

As with bug 13959, I think Bugzilla is not the right place for this.  
Anything involving "many" without a clear objective way to assess whether 
the source tree has been fixed for an issue is probably better suited to 
policy discussions on libc-alpha and tracking open projects on wiki pages.  
If you create your own project page then you can track for each test 
whether it meets your own standards and other people can independently 
track, if they wish, whether each test meets their standards.

> 1. It's difficult to audit/debug them and determine if they're actually testing
> for what they were intended to test. This is especially problematic for tests
> that involve (either as the subject of the test or as a side-difficulty of
> testing) race conditions, where it may not be at all obvious where the races
> are.

It's quite likely that the function of a test changes over time as the 
test itself changes.  If you wish to propose a policy against changing 
existing tests to test new things - a policy that new tests should be 
added in preference to changing an existing test - then libc-alpha is the 
right place to do so, with the wiki being used to document any resulting 
consensus.

> 2. If a change introduces a bug, it may be hard to determine the cause without
> reverse engineering the test to see what it was testing.

Patches explaining obscure aspects of a test are always welcome, but 
there's no need to repeat in English what the test already says in C or 
what is plain from the documented semantics of the functions, and it's 
inherently subjective what is an appropriate level of commenting for a 
test.

If you want better explanations in new tests, you should do *both* of the 
following: raise the issue on libc-alpha of what appropriate levels of 
explanation are (with examples of specific well-explained and 
badly-explained tests), with a view to documenting consensus on the wiki; 
review testcases in submitted patches and ask for improvements if you 
think they are inadequately explained.  Since it's subjective what's 
enough explanation, only you can review new tests for whether they meet 
your own standards of explanation, although to some extent a standard may 
be documented and illustrated by examples.

> 3. It would be nice to know if tests are asserting behavior required by the
> standards, behavior required as part of the documented or unofficial
> specification of glibc, undocumented behavior that known applications expect,
> and/or quality of implementation issues.

Again, this is open-ended and in general a test is simply helping to 
ensure that unexpected changes to glibc's behavior are detected.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/14233] Many of tst-*.c lack any description of the assertions they test

[jsm28 at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14233

Joseph Myers <jsm28 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID

--- Comment #2 from Joseph Myers <jsm28 at gcc dot gnu.org> 2012-06-24 14:52:55 UTC ---
As discussed on libc-alpha and now documented at
<http://sourceware.org/glibc/wiki/Bugzilla%20Procedures>, please take general
policy questions and open-ended projects to libc-alpha and the wiki.

If there is an objective problem with a test - for example, it has a race
condition that may cause it to fail sometimes - then a bug in Bugzilla for the
specific issue with the specific test is appropriate.  If in doubt about
whether the testsuite should be ensuring some particular behavior beyond the
relevant standards (where not documented in the glibc manual) then libc-alpha
will be the right place to raise such a question about what the glibc API
should be (likely resulting in a change to either the manual or the test, or an
agreement about which is wrong which could then result in a Bugzilla bug
pointing to that agreement).

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/14233] Many of tst-*.c lack any description of the assertions they test

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14233

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.