Some tests in gcc.c-torture rely on undefined behaviour?

Some tests in gcc.c-torture rely on undefined behaviour?

[Nicholas Nethercote]

Hi,

I've been looking at the gcc.c-torture tests, it seems some of them rely 
on undefined behaviour.  For example, 920612-1.c looks like this:

   f(j)int j;{return++j>0;}
   main(){if(f((~0U)>>1))abort();exit(0);}

AIUI, this passes the largest possible positive integer to f(), which then 
increments it, causing signed overflow, the result of which is undefined. 
The test passes if signed overflow wraps.

930529-1.c is similar -- again the maximum positive integer is 
incremented.

20020508-2.c has the following code (abridged):

   #ifndef CHAR_BIT
   #define CHAR_BIT 8
   #endif

   #define ROR(a,b) (((a) >> (b)) | ((a) << ((sizeof (a) * CHAR_BIT) - (b))))

   #define INT_VALUE ((int)0x1234)

   #define SHIFT1 4

   int i = INT_VALUE;
   int shift1 = SHIFT1;

   if (ROR (i, shift1) != ROR (INT_VALUE, SHIFT1))
     abort ();

Similarly, the left-shifting in ROR causes signed integer overflow (I 
think) and so ROR relies on undefined behaviour.  20020508-3.c is similar.

My question is:  what exactly is gcc.c-torture testing?  It seems to be 
testing more than just C standard compliance, but also certain 
undefined-but-desired behaviours, such as wrap-on-signed-overflow (at 
least in some cases).  Is that right?  If so, other C compilers could be 
correct with respect to the C standard but not pass all the tests.  I 
couldn't find any kind of README or description about the tests that 
covered this point, so I'd appreciate any explanations.

Thanks.

Nick

Re: Some tests in gcc.c-torture rely on undefined behaviour?

[Joe Buck]

On Tue, Jul 12, 2005 at 09:41:22AM -0500, Nicholas Nethercote wrote:
> I've been looking at the gcc.c-torture tests, it seems some of them rely 
> on undefined behaviour.  For example, 920612-1.c looks like this:
> 
>   f(j)int j;{return++j>0;}
>   main(){if(f((~0U)>>1))abort();exit(0);}

Wow.  It appears that it would be legal for a C compiler to optimize f() to

int f(int j) { return 1;}

since the compiler is entitled to assume that overflow does not occur.

Just the same, I don't think we necessarily want to take advantage of
every degree of freedom the standards give us (at least, not by default).

(Oh, crap; I see a massive thread re-emerging).

Re: Some tests in gcc.c-torture rely on undefined behaviour?

[Joseph S. Myers]

On Tue, 12 Jul 2005, Nicholas Nethercote wrote:

> Similarly, the left-shifting in ROR causes signed integer overflow (I think)
> and so ROR relies on undefined behaviour.  20020508-3.c is similar.

To quote implement-c.texi:

  GCC does not use the latitude given in C99 only to treat certain
  aspects of signed @samp{<<} as undefined, but this is subject to
  change.

Unless there is a shift by >= the width of the (promoted) type, this code 
is defined for GNU C.  In any case, it is defined for gnu89 mode even if 
we start treating this as undefined in C99 mode; see pr7284-1.c.

> My question is:  what exactly is gcc.c-torture testing?  It seems to be

That GNU C code compiles or executes as expected for GNU C.

> testing more than just C standard compliance, but also certain
> undefined-but-desired behaviours, such as wrap-on-signed-overflow (at least in
> some cases).  Is that right?  If so, other C compilers could be correct with

Such tests are in general bugs.  You'd have to ask Torbjorn about what the 
original purpose of the old parts of c-torture was, as that may have 
differed from the current GCC testsuite, but invalid tests should be 
removed (or, perhaps better, moved to gcc.c-torture/compile) or have 
relevant options such as -fwrapv added.

Note that gcc.c-torture/compile correctly contains many tests involving 
undefined behavior at runtime which implementations are required to 
translate in case the problem code is never executed.  These serve to test 
that e.g. uninitialized variables do not crash the compiler.

> respect to the C standard but not pass all the tests.  I couldn't find any

The GCC testsuite is a testsuite for GCC only, not for other compilers.

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: Some tests in gcc.c-torture rely on undefined behaviour?

[Ian Lance Taylor]

"Joseph S. Myers" <joseph@codesourcery.com> writes:

> Such tests are in general bugs.  You'd have to ask Torbjorn about what the 
> original purpose of the old parts of c-torture was, as that may have 
> differed from the current GCC testsuite, but invalid tests should be 
> removed (or, perhaps better, moved to gcc.c-torture/compile) or have 
> relevant options such as -fwrapv added.

The original purpose was simply to collect programs which caused the
compiler to fail.  When he saw a bug report with a test case, he
collected the test case, reduced it, and added it to the torture
tests.  The tests were simply run at various optimization levels.
Tests which caused the compiler to crash were put in compile, tests
which showed a miscompilation visible at run time were put in execute.

I would guess that 920612-1.c, at least, could just be changed to use
unsigned int, and it would continue to test whatever bug it was
testing when it was originally added.

Ian

Re: Some tests in gcc.c-torture rely on undefined behaviour?

[Diego Novillo]

On Tue, Jul 12, 2005 at 11:39:18AM -0700, Ian Lance Taylor wrote:

> I would guess that 920612-1.c, at least, could just be changed to use
> unsigned int, and it would continue to test whatever bug it was
> testing when it was originally added.
> 
The problem is somewhat more widespread now with the tree
optimizers.  In particular with old test cases.  Some of these
cases are essentially optimized into empty functions by the time
we get into the RTL passes.

We would have to audit them all and add enough external
functions, volatile markers or what-have-you to have them survive
until RTL.  Not sure if it's worth it.

Re: Some tests in gcc.c-torture rely on undefined behaviour?

[Ian Lance Taylor]

Diego Novillo <dnovillo@redhat.com> writes:

> The problem is somewhat more widespread now with the tree
> optimizers.  In particular with old test cases.  Some of these
> cases are essentially optimized into empty functions by the time
> we get into the RTL passes.

Hmmm, yeah.

> We would have to audit them all and add enough external
> functions, volatile markers or what-have-you to have them survive
> until RTL.  Not sure if it's worth it.

I'm sure it isn't.

Clearly test cases which we now understand to be invalid should be
changed or removed.  Other than that, I would vote for leaving them
alone.  Unless people are concerned about the number of tests we
have.

(To speed up running the testsuite it would probably be more useful in
any case to put some time into running test cases in parallel on
native systems.  Tcl has enough support for processes that this
shouldn't be very hard.  It would mostly be a bookkeeping issue, I
think.)

Ian

Re: Some tests in gcc.c-torture rely on undefined behaviour?

[Nicholas Nethercote]

On Tue, 12 Jul 2005, Joseph S. Myers wrote:

>> My question is:  what exactly is gcc.c-torture testing?  It seems to be
>
> That GNU C code compiles or executes as expected for GNU C.

Is there a definition for GNU C?  implement-c.texi and extend.texi have 
some information about this, are there any other sources?

Thanks.

Nick

Re: Some tests in gcc.c-torture rely on undefined behaviour?

[Robert Dewar]

Nicholas Nethercote wrote:
> Hi,
> 
> I've been looking at the gcc.c-torture tests, it seems some of them rely 
> on undefined behaviour.  For example, 920612-1.c looks like this:
> 
>   f(j)int j;{return++j>0;}
>   main(){if(f((~0U)>>1))abort();exit(0);}
> 
> AIUI, this passes the largest possible positive integer to f(), which 
> then increments it, causing signed overflow, the result of which is 
> undefined. The test passes if signed overflow wraps.
> 
> 930529-1.c is similar -- again the maximum positive integer is incremented.

i think it is fine 2 have these tests. ant gcc version failing them is suspicious