discussion of ObjC test results on Darwin(/x86) (lengthy)

discussion of ObjC test results on Darwin(/x86) (lengthy)

[Lars Sonchocky-Helldorf]

By analyzing the ObjC test results on (Open)Darwin(/x86) I've found out  
that most of the reported failures are indeed bogus failures caused by  
wrong parameters or erroneously run tests. I wish that could be  
corrected, either by correcting the parameters or by not running those  
tests where they don't apply.

At first I'll give an overview of all the results which I explain in  
detail later on.

There are two varieties of ObjC test results resulting from the two  
available ObjC runtimes on (Open)Darwin (you probably already new that).



Results for the next(or apple)-runtime:

		=== objc tests ===


Running target unix
FAIL: objc/execute/load-2.m execution,  -O0
FAIL: objc/execute/load-2.m execution,  -O1
FAIL: objc/execute/load-2.m execution,  -O2
FAIL: objc/execute/load-2.m execution,  -O3 -fomit-frame-pointer
FAIL: objc/execute/load-2.m execution,  -O3 -g
FAIL: objc/execute/load-2.m execution,  -Os
FAIL: objc/execute/load.m execution,  -O0
FAIL: objc/execute/load.m execution,  -O1
FAIL: objc/execute/load.m execution,  -O2
FAIL: objc/execute/load.m execution,  -O3 -fomit-frame-pointer
FAIL: objc/execute/load.m execution,  -O3 -g
FAIL: objc/execute/load.m execution,  -Os
FAIL: objc.dg/headers.m (test for excess errors)
FAIL: objc.dg/stret-2.m scan-assembler objc_msgSend_stret
FAIL: objc.dg/stret-2.m scan-assembler-not objc_msgSend[^_S]
FAIL: objc.dg/stret-2.m scan-assembler-not objc_msgSendSuper[^_]
FAIL: objc.dg/symtab-1.m scan-assembler  
L_OBJC_SYMBOLS.*:\\n\\t.long\\t0\\n\\t.long\\t0\\n\\t.short\\t2\\n\\t.sh 
ort\\t0\\n\\t.long\\tL_OBJC_CLASS_Derived.*\\n\\t.long\\tL_OBJC_CLASS_Ba 
se.*\\n

		=== objc Summary ===

# of expected passes		1528
# of unexpected failures	17
# of unsupported tests		1



Results for the gnu(or gcc)-runtime:

		=== objc tests ===


Running target unix/-fgnu-runtime
FAIL: objc.dg/bitfield-2.m execution test
FAIL: objc.dg/const-str-3.m (test for excess errors)
WARNING: objc.dg/const-str-3.m compilation failed to produce executable
FAIL: objc.dg/const-str-9.m (test for excess errors)
ERROR: objc.dg/const-str-9.m: error executing dg-final: couldn't open  
"const-str-9.s": no such file or directory
UNRESOLVED: objc.dg/const-str-9.m: error executing dg-final: couldn't  
open "const-str-9.s": no such file or directory
FAIL: objc.dg/encode-1.m execution test
FAIL: objc.dg/image-info.m (test for excess errors)
ERROR: objc.dg/image-info.m: error executing dg-final: couldn't open  
"image-info.s": no such file or directory
UNRESOLVED: objc.dg/image-info.m: error executing dg-final: couldn't  
open "image-info.s": no such file or directory
FAIL: objc.dg/method-4.m scan-assembler _OBJC_CLASS_REFERENCES_0
FAIL: objc.dg/selector-1.m  (test for warnings, line 22)
FAIL: objc.dg/selector-1.m (test for excess errors)
FAIL: objc.dg/stret-2.m scan-assembler objc_msgSend_stret
FAIL: objc.dg/stret-2.m scan-assembler objc_msgSendSuper_stret
FAIL: objc.dg/symtab-1.m scan-assembler  
L_OBJC_SYMBOLS.*:\\n\\t.long\\t0\\n\\t.long\\t0\\n\\t.short\\t2\\n\\t.sh 
ort\\t0\\n\\t.long\\tL_OBJC_CLASS_Derived.*\\n\\t.long\\tL_OBJC_CLASS_Ba 
se.*\\n
FAIL: objc.dg/zero-link-1.m scan-assembler objc_getClass
FAIL: objc.dg/zero-link-2.m scan-assembler _OBJC_CLASS_REFERENCES_0

		=== objc Summary ===

# of expected passes		1528
# of unexpected failures	13
# of unresolved testcases	2
# of unsupported tests		1



the used compiler was (in both cases):

/Volumes/Data/Users/lars/GCC/FSF/gcc-build/gcc/xgcc  version 4.0.0  
20041205 (experimental)


Compiler version: 4.0.0 20041205 (experimental) objc
Platform: i686-apple-darwin7.2.1
configure flags: --prefix=/tmp/gcc --enable-languages=c,c++,java,objc



And now the details, starting with the next-runtime:

FAIL: objc/execute/load-2.m execution,  -O0
FAIL: objc/execute/load-2.m execution,  -O1
FAIL: objc/execute/load-2.m execution,  -O2
FAIL: objc/execute/load-2.m execution,  -O3 -fomit-frame-pointer
FAIL: objc/execute/load-2.m execution,  -O3 -g
FAIL: objc/execute/load-2.m execution,  -Os
FAIL: objc/execute/load.m execution,  -O0
FAIL: objc/execute/load.m execution,  -O1
FAIL: objc/execute/load.m execution,  -O2
FAIL: objc/execute/load.m execution,  -O3 -fomit-frame-pointer
FAIL: objc/execute/load.m execution,  -O3 -g
FAIL: objc/execute/load.m execution,  -Os

both are execution tests that check if +load is automatically called  
before main is run, once on a single class (load.m) once on two classes  
(load-2.m). This might be the right behaviour for the gnu-runtime but  
is different for the next-runtime, which loads classes on demand (and  
uses a caching system: OBJC_EXPORT IMP  
_class_lookupMethodAndLoadCache(Class, SEL); defined in  
objc4/runtime/objc-private.h).

To make load.m work I had to change it to:

#import <objc/Protocol.h>

static int static_variable = 0;

/*
  * +load is implemented in Protocol.m (objc4/runtime/Protocol.m)
  * (which subclasses Object) but not defined in Protocol.h
  * -> you'll get a warning:
  * 'Protocol' may not respond to '+load'
  */
@interface TestClass : Protocol
{

}
+ (void) load;
@end

@implementation TestClass
+ (void) load
{
   static_variable = 1;
   [super load];
}
@end

int main (void)
{
   id test = [[TestClass alloc] init];

   if (static_variable != 1)
     {
       abort ();
     }

   return 0;
}


FAIL: objc.dg/headers.m (test for excess errors)

headers.m tries to include <Foundation/NSString.h> which is not part of  
pure Darwin/OpenDarwin, but only distributed with Mac OS X (as part of  
Cocoa).

#ifdef __NEXT_RUNTIME__
#include <Foundation/NSString.h>
#else


FAIL: objc.dg/stret-2.m scan-assembler objc_msgSend_stret
FAIL: objc.dg/stret-2.m scan-assembler-not objc_msgSend[^_S]
FAIL: objc.dg/stret-2.m scan-assembler-not objc_msgSendSuper[^_]

Maybe that's a real failure. stret-2.s containts  
'L_objc_msgSendSuper_stret$stub', '_objc_msgSendSuper_stret',  
'L_objc_msgSend$stub' and '_objc_msgSend' but not just  
'objc_msgSend_stret'. I have the .s file handy if needed. Drop me a  
line.


FAIL: objc.dg/symtab-1.m scan-assembler  
L_OBJC_SYMBOLS.*:\\n\\t.long\\t0\\n\\t.long\\t0\\n\\t.short\\t2\\n\\t.sh 
ort\\t0\\n\\t.long\\tL_OBJC_CLASS_Derived.*\\n\\t.long\\tL_OBJC_CLASS_Ba 
se.*\\n

the assembler in question looks like:

L_OBJC_SYMBOLS:
	.long	0
	.long	0
	.word	2
	.word	0
	.long	L_OBJC_CLASS_Derived
	.long	L_OBJC_CLASS_Base

I don't know if this is correct (words instead of shorts (imagine this  
in real life btw. ;-))).



and now the results for the gnu-runtime:

FAIL: objc.dg/bitfield-2.m execution test

contains (like others to follow):

/* { dg-options "-fnext-runtime -fsigned-char" } */

this clearly conflicts with -fgnu-runtime which was used to run those  
tests. the resulting command line was:

/Volumes/Data/Users/lars/GCC/FSF/gcc-build/gcc/xgcc  
-B/Volumes/Data/Users/lars/GCC/FSF/gcc-build/gcc/  
/Users/lars/GCC/FSF/gcc-4.0-20041205/gcc/testsuite/objc.dg/bitfield-2.m  
   -fnext-runtime -fsigned-char  
-I/Users/lars/GCC/FSF/gcc-4.0-20041205/gcc/testsuite/../../libobjc  
-L/Volumes/Data/Users/lars/GCC/FSF/gcc-build/i686-apple-darwin7.2.1/./ 
libobjc/.libs  -lobjc -lm   -fgnu-runtime -o ./bitfield-2.exe

btw. this test also fails if is omitted. but in the source code it says:

/* Check if bitfield ivars are correctly @encode'd when
    the NeXT runtime is used.  */

so maybe this test is obsolete for the gnu-runtime.


FAIL: objc.dg/const-str-3.m (test for excess errors)
WARNING: objc.dg/const-str-3.m compilation failed to produce executable

fails during linking with:

/usr/bin/ld: Undefined symbols:
_objc_getClass
collect2: ld returned 1 exit status

obviously objc_getClass() is not defined for the gnu-runtime which is  
used in line 41:

   memcpy(&_FooClassReference, objc_getClass("Foo"),  
sizeof(_FooClassReference));

this could be rewritten for the gnu-runtime if that test makes sense  
there

furthermore '-fnext-runtime' and '-fgnu-runtime' are conflicting here  
too.


FAIL: objc.dg/const-str-9.m (test for excess errors)
ERROR: objc.dg/const-str-9.m: error executing dg-final: couldn't open  
"const-str-9.s": no such file or directory
UNRESOLVED: objc.dg/const-str-9.m: error executing dg-final: couldn't  
open "const-str-9.s": no such file or directory

Maybe a real bug. Compilation fails with:

/Users/lars/GCC/FSF/gcc-4.0-20041205/gcc/testsuite/objc.dg/const-str- 
9.m:17: error: cannot find interface declaration for 'NXConstantString'

'-fnext-runtime' and '-fgnu-runtime' yadda yadda ...


FAIL: objc.dg/encode-1.m execution test

Fails to execute (but compiles). I don't know if this test is  
applicable to the gnu-runtime (some comment inside the source lets me  
assume something else)


FAIL: objc.dg/image-info.m (test for excess errors)
ERROR: objc.dg/image-info.m: error executing dg-final: couldn't open  
"image-info.s": no such file or directory
UNRESOLVED: objc.dg/image-info.m: error executing dg-final: couldn't  
open "image-info.s": no such file or directory

fails deliberately during compilation:

/Users/lars/GCC/FSF/gcc-4.0-20041205/gcc/testsuite/objc.dg/image- 
info.m:9:2: error: #error Feature not currently supported by the GNU  
runtime

compiler exited with status 1

because of lines 8 to 10:

#ifndef __NEXT_RUNTIME__
#error Feature not currently supported by the GNU runtime
#endif

not very gentleman like ;-)


FAIL: objc.dg/method-4.m scan-assembler _OBJC_CLASS_REFERENCES_0

containts for instance '___objc_class_ref_Object' but not  
'_OBJC_CLASS_REFERENCES_0'. Is this test applicable for the gnu-runtime?

FAIL: objc.dg/selector-1.m  (test for warnings, line 22)
FAIL: objc.dg/selector-1.m (test for excess errors)

see: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18862

FAIL: objc.dg/stret-2.m scan-assembler objc_msgSend_stret
FAIL: objc.dg/stret-2.m scan-assembler objc_msgSendSuper_stret

I think not applicable. Every  'objc_msgSend' is replaced by  
'objc_msg_lookup'. next and gnu runtime differ here in naming.


FAIL: objc.dg/symtab-1.m scan-assembler  
L_OBJC_SYMBOLS.*:\\n\\t.long\\t0\\n\\t.long\\t0\\n\\t.short\\t2\\n\\t.sh 
ort\\t0\\n\\t.long\\tL_OBJC_CLASS_Derived.*\\n\\t.long\\tL_OBJC_CLASS_Ba 
se.*\\n

the assembler in question looks like:

L_OBJC_SYMBOLS:
	.long	0
	.long	0
	.word	2
	.word	0
	.long	L_OBJC_CLASS_Derived
	.long	L_OBJC_CLASS_Base

btw: same as for the next-runtime


FAIL: objc.dg/zero-link-1.m scan-assembler objc_getClass
FAIL: objc.dg/zero-link-2.m scan-assembler _OBJC_CLASS_REFERENCES_0

the gnu-runtime works completely different for both cases (other names  
and so on). I could provide you with the .s files if you want to: drop  
me a line.



So, now somebody has any comments on that or is able to improve the  
situation?


This was a long one. hope I didn't tire anybody out. I am tired myself  
now (already 3:30 am here) ... :-)


regards, Lars