GCC 971031 on OpenServer

GCC 971031 on OpenServer

[Robert Lipe]

Jeff's dwarf2out.c patch applied.   Top level make bootstrap-lean
to build it.

--target=i586-pc-sco3.2v5.0.4

For GCC, Both ELF and COFF fail minimally and identically.

FAIL: gcov failed: couldn't execute "./gcov": no such file or directory

                === gcc Summary ===

# of expected passes            4882
# of unexpected failures        1
# of expected failures          5
# of unsupported tests          7
./negcs version egcs-2.90.15 971031 (gcc2-970802 experimental)

disturbing g++ 971031 results. defer-pop to blame?

[Robert Lipe]

> Jeff's dwarf2out.c patch applied.   Top level make bootstrap-lean
> to build it.
> 
> --target=i586-pc-sco3.2v5.0.4
> 
> For GCC, Both ELF and COFF fail minimally and identically.

Something screwy is going on here...


On the 1024 snapshot, g++ worked pretty well for both ELF and COFF.
I can't swear that I rebuilt the libraries on 1024, but I will swear
that I have this time.   I will also swear that I've done a top-level
make clean and make bootstrap-lean since applying Jeff's dwarf2out
patch.

I'm seeing different failures on g++ than I have before.  ELF and COFF
used to fail identically.  Since it's sort of wierd, I should confess
that ELF on this target uses dwarf2eh and COFF uses sjlj.

Here are the failures.  First ELF, then COFF.

FAIL: g++.benjamin/warn01.C (test for excess errors)
FAIL: g++.brendan/template9.C  Execution test
FAIL: g++.jason/2371.C  Execution test
XPASS: g++.jason/destruct3.C - (test for bogus messages, line 38)
FAIL: g++.jason/template31.C (test for excess errors)
FAIL: g++.law/arg8.C  Execution test
FAIL: g++.law/code-gen5.C  Execution test
FAIL: g++.law/cvt2.C  Execution test
FAIL: g++.law/profile1.C (test for excess errors)
XPASS: g++.mike/dyncast1.C  Execution test
XPASS: g++.mike/dyncast2.C  Execution test
FAIL: g++.mike/eh2.C  Execution test
FAIL: g++.mike/net34.C  Execution test
FAIL: g++.mike/net46.C  Execution test
FAIL: g++.mike/p658.C  Execution test
FAIL: g++.mike/p9732b.C  Execution test

                === g++ Summary ===

# of expected passes            3367
# of unexpected failures        13
# of unexpected successes       3
# of expected failures          80
# of untested testcases         6
./negcs version egcs-2.90.15 971031 (gcc2-970802 experimental)


If I hand-compile and run, say, template.9.C, I see the "PASS" output
followed by an 'illegal instruction. Core dumped'.   Even if I modify
the assembly output for that file to just make main return, I get the
same.      I tried a couple of tests manually and got similar results.

$ ./negcs /tmp/net34.C -lstdc++
(robertl) rjlhome:/play/testgcc
$ ./a.out
bar_1::k -> 1
bar_2::k -> 2
bar_1::get_k() -> 1
bar_2::get_k() -> 2
Memory fault(coredump)

I get this with or without -fno-exceptions and with any -O from 0 to 3.

If I add '-defer-pop' to most of the cases I've tried by hand, it
seems to work.   In  fact, I just wrote a script to loop through those
cases and with -defer-pop.   With -defer-pop, 8 of them pass.   Without
-defer-pop, none of them pass.


Test Run By robertl on Sun Nov  2 13:11:08 1997
Native configuration is i586-pc-sco3.2v5.0.4
FAIL: g++.benjamin/warn01.C (test for excess errors)
XPASS: g++.jason/destruct3.C - (test for bogus messages, line 38)
FAIL: g++.jason/thunk2.C (test for excess errors)
FAIL: g++.law/profile1.C (test for excess errors)
XPASS: g++.mike/dyncast1.C  Execution test
XPASS: g++.mike/dyncast2.C  Execution test
FAIL: g++.mike/eh30.C (test for excess errors)
FAIL: g++.mike/init1.C  Execution test
FAIL: g++.mike/p2736.C  Execution test
FAIL: g++.mike/p4750.C (test for excess errors)

                === g++ Summary ===

# of expected passes            3372
# of unexpected failures        7
# of unexpected successes       3
# of expected failures          81
# of untested testcases         6
./negcs version egcs-2.90.15 971031 (gcc2-970802 experimental)


The COFF test results are more consistent with what we've seen
to date on this target.   

Any ideas what's going on?

Thanx,
RJL

Re: disturbing g++ 971031 results. defer-pop to blame?

[Jeffrey A Law]

  In message < 19971102142810.21992@dgii.com >you write:
  > I'm seeing different failures on g++ than I have before.  ELF and COFF
  > used to fail identically.  Since it's sort of wierd, I should confess
  > that ELF on this target uses dwarf2eh and COFF uses sjlj.
Yup.  But, aside from the tests that actually test EH, I would expect
the run for ELF & COFF to be identical.

Did we ever come to a resolution of the multilib testing problem?

  > If I hand-compile and run, say, template.9.C, I see the "PASS" output
  > followed by an 'illegal instruction. Core dumped'.   Even if I modify
  > the assembly output for that file to just make main return, I get the
  > same.      I tried a couple of tests manually and got similar results.
Makes me wonder if something is goof-ing in the finalization code.
Can you debug it?

  > 
  > If I add '-defer-pop' to most of the cases I've tried by hand, it
  > seems to work.   In  fact, I just wrote a script to loop through those
  > cases and with -defer-pop.   With -defer-pop, 8 of them pass.   Without
  > -defer-pop, none of them pass.
Don't you mean -fno-defer-pop?

jeff

Re: disturbing g++ 971031 results. defer-pop to blame?

[Robert Lipe]

[ re: new? failures on i586-pc-sco3.2v5.0.4 ] 
>   > I'm seeing different failures on g++ than I have before.  ELF and COFF
>   > used to fail identically.  Since it's sort of wierd, I should confess
>   > that ELF on this target uses dwarf2eh and COFF uses sjlj.
> Yup.  But, aside from the tests that actually test EH, I would expect
> the run for ELF & COFF to be identical.

And a few short weeks ago, they did.

> Did we ever come to a resolution of the multilib testing problem?

Not really.   I was so elated when I saw that runlib was running the 
tests twice that I completely failed to notice until some weeks later
that it was running the elf compiler twice - once with the elf banner
and once with the coff banner.   I wasn't amused.

I'm now just running runtest twice, each with different --tool_opts flags. 

> Makes me wonder if something is goof-ing in the finalization code.
> Can you debug it?

Now is when I notice that GDB is not playing nice with EGCS ELF output.
When it rains...

>   > If I add '-defer-pop' to most of the cases I've tried by hand, it
>   > seems to work.   In  fact, I just wrote a script to loop through those
>   > cases and with -defer-pop.   With -defer-pop, 8 of them pass.   Without
>   > -defer-pop, none of them pass.
> Don't you mean -fno-defer-pop?

Boy, I was on a roll there, wasn't I?

I meant "-fdefer-pop".   -fno-defer-pop (the default) still shows the 
problem.


$ ./negcs /tmp/net34.C -lstdc++
$ ./a.out
bar_1::k -> 1
bar_2::k -> 2
bar_1::get_k() -> 1
bar_2::get_k() -> 2
Memory fault(coredump)
$ ./negcs -fdefer-pop /tmp/net34.C -lstdc++
$ ./a.out
bar_1::k -> 1
bar_2::k -> 2
bar_1::get_k() -> 1
bar_2::get_k() -> 2
$ 

This is actually ironic becuase when H.J. was having the problems that
could only be cured by -fdefer-pop, I tried them on OpenServer and
couldn't duplicate them at all.     Apparently, our stacks are just
different enough to be sensitive to whatever is going on here...

RJL

Re: disturbing g++ 971031 results. defer-pop to blame?

[Jeffrey A Law]

  In message < 19971102224718.19861@dgii.com >you write:
  > [ re: new? failures on i586-pc-sco3.2v5.0.4 ] 
  > >   > I'm seeing different failures on g++ than I have before.  ELF and COF
  > F
  > >   > used to fail identically.  Since it's sort of wierd, I should confess
  > >   > that ELF on this target uses dwarf2eh and COFF uses sjlj.
  > > Yup.  But, aside from the tests that actually test EH, I would expect
  > > the run for ELF & COFF to be identical.
  > 
  > And a few short weeks ago, they did.
Hmmm.  I guess the one thing I can think of which would play into
some of these problems is the bogus flow control graphs we used
to generate.
 
  > Not really.   I was so elated when I saw that runlib was running the 
  > tests twice that I completely failed to notice until some weeks later
  > that it was running the elf compiler twice - once with the elf banner
  > and once with the coff banner.   I wasn't amused.
Real strange since I've actually looked in the log files when I've
done H8 testing and everything seems to be in order...  Got to
be something simple/stupid we're missing.

  > Now is when I notice that GDB is not playing nice with EGCS ELF output.
  > When it rains...
Sigh.  More likely it's not playing nice with dwarf2 more so than
ELF.

  > 
  > >   > If I add '-defer-pop' to most of the cases I've tried by hand, it
  > >   > seems to work.   In  fact, I just wrote a script to loop through thos
  > e
  > >   > cases and with -defer-pop.   With -defer-pop, 8 of them pass.   Witho
  > ut
  > >   > -defer-pop, none of them pass.
  > > Don't you mean -fno-defer-pop?
  > 
  > Boy, I was on a roll there, wasn't I?
  > 
  > I meant "-fdefer-pop".   -fno-defer-pop (the default) still shows the 
  > problem.
Hmmm, -fdefer-pop is the default; the compiler will arrange for
-fno-defer-pop to be on inside EH regions and such.  Real strange.


jeff

Re: disturbing g++ 971031 results. defer-pop to blame?

[Robert Lipe]

> Real strange since I've actually looked in the log files when I've
> done H8 testing and everything seems to be in order...  Got to
> be something simple/stupid we're missing.

Probably.   I'll investigate that next.

>   > Now is when I notice that GDB is not playing nice with EGCS ELF output.
>   > When it rains...
> Sigh.  More likely it's not playing nice with dwarf2 more so than
> ELF.

Yes, and '-gdwarf1' results in unresolved externals.   Sigh.   I'm 
investigating that right now.

Let's focus on this problem right now:

>   > >   > If I add '-defer-pop' to most of the cases I've tried by hand, it
>   > >   > seems to work.   In  fact, I just wrote a script to loop through thos
>   > I meant "-fdefer-pop".   -fno-defer-pop (the default) still shows the 
>   > problem.
> Hmmm, -fdefer-pop is the default; the compiler will arrange for
> -fno-defer-pop to be on inside EH regions and such.  Real strange.

Actually, it's only the default if we're optimizing, right?  That's
what I'm led to believe from 'gcc -S' output.   That's actually
consistent with some more tests that I just ran.

Perhaps we need to make the c++ tests run with varying -O levels
as we do for the gcc tests.  After all, what's another few hours
during the build? :-)

Here's what I'm seeing.   negcs is just a script that calls the native
egcs:

G=/play/negcs
L=${G}/libraries
exec ${G}/gcc/xgcc -B${G}/gcc/ \
        -I${L}/libio/ \
        -I${L}/libstdc++/ \
        "$@"

$ ./negcs /tmp/net34.C -lstdc++ && ./a.out
bar_1::k -> 1
bar_2::k -> 2
bar_1::get_k() -> 1
bar_2::get_k() -> 2
Memory fault(coredump)
(robertl) rjlhome:/play/testgcc
$ ./negcs -O0 /tmp/net34.C -lstdc++ && ./a.out
bar_1::k -> 1
bar_2::k -> 2
bar_1::get_k() -> 1
bar_2::get_k() -> 2
Memory fault(coredump)
(robertl) rjlhome:/play/testgcc
$ ./negcs -O1 /tmp/net34.C -lstdc++ && ./a.out
bar_1::k -> 1
bar_2::k -> 2
bar_1::get_k() -> 1
bar_2::get_k() -> 2
(robertl) rjlhome:/play/testgcc
$ ./negcs -O2 /tmp/net34.C -lstdc++ && ./a.out
bar_1::k -> 1
bar_2::k -> 2
bar_1::get_k() -> 1
bar_2::get_k() -> 2
(robertl) rjlhome:/play/testgcc
$ ./negcs -O3 /tmp/net34.C -lstdc++ && ./a.out
bar_1::k -> 1
bar_2::k -> 2
bar_1::get_k() -> 1
bar_2::get_k() -> 2
$

Perhaps something is doing something bad on the stack and defer-pop
is just circumventing it.

RJL

Re: disturbing g++ 971031 results. defer-pop to blame?

[Jeffrey A Law]

  In message < 19971103000221.16739@dgii.com >you write:
  > > Hmmm, -fdefer-pop is the default; the compiler will arrange for
  > > -fno-defer-pop to be on inside EH regions and such.  Real strange.
  > 
  > Actually, it's only the default if we're optimizing, right?  That's
  > what I'm led to believe from 'gcc -S' output.   That's actually
  > consistent with some more tests that I just ran.
Oh, yes.  You're absolutely right.  Sometimes I forget that you can
build without -O :-)

  > Perhaps we need to make the c++ tests run with varying -O levels
  > as we do for the gcc tests.  After all, what's another few hours
  > during the build? :-)
I've always thought it should be an option.  But never got up the nerve
to sit down and implement it.

Any chance of just discarding symbolic debugging and debugging the exit
path at the assembly level?  Or maybe just find out where exactly it's
core dumping and try work towards a solution from there?

Jeff

Re: disturbing g++ 971031 results. defer-pop to blame?

[Robert Lipe]

>   > Now is when I notice that GDB is not playing nice with EGCS ELF output.
>   > When it rains...
> Sigh.  More likely it's not playing nice with dwarf2 more so than
> ELF.

GDB that I built from 4.14-95q4 (i486-unknown-sco3.2v5.0.0elf) handles 
the binaries in question just fine.   GDB 4.16 built from virgin FSF 
sources isn't useful on my ELF binaries.   I'll chase this later...


I build the binary in question.

./negcs -O0 -g /tmp/net34.C -lstdc++ && ./a.out



I then step through it:

(gdb)
13              int get_k() {return k;}
(gdb)
bar_1::get_k() -> 1
bar_2::get_k() -> 2

Program received signal SIGSEGV, Segmentation fault.
0x8059187 in .fini ()



If I look at the tail end of the object, the reason for our
crash is more apparent:

Disassembly of section .fini:

0805917c <_fini>:
 805917c:       e8 c7 f9 fe ff  call   8048b48 <__do_global_dtors_aux>
 8059181:       90              nop
 8059182:       90              nop
 8059183:       90              nop

08059184 <.fini>:
 8059184:       4c              decl   %esp
 8059185:       cd 04           int    $0x4
 8059187:       08 c2           orb    %al,%dl

08059188 <.fini>:
 8059188:       c2 00 00        ret    $0x0
 805918b:       90              nop


The entry in the middle looks like mush to me.

Roaming through the objects that are linked, I can see that crtn.o gives
us the return.   crtbegin.o gives us the call to __do_global_dtors_aux().
Where does the strange sequence in th emiddle come from?

If I take hte final link line:

 /play/negcs/gcc/ld -b elf -Ra,XPG4PLUS,ELF -YP,/usr/ccs/lib:/lib:/usr/lib \
        -Qn /usr/ccs/lib/crt1.o /usr/ccs/lib/values-Xa.o \
        /play/negcs/gcc/crtbegin.o -L/play/negcs/gcc -L/usr/ccs/bin \
        -L/usr/ccs/lib -L/usr/local/lib net34.o -lstdc++ \
        -lgcc -lcrypt -lgen -lc -lgcc /play/negcs/gcc/crtend.o \
        /usr/ccs/lib/crtn.o

and play with the order (add crtn.o before or after -lstdc++) I have 
reason to think that it's from something in libstdc++.   If I just
hoist -lstdc++ down to the very last (and stick another -lgcc after
it) I get a functioning executable.

Objdump --all-headers on libstdc++.a and all the .o's that go into
it shows nothing in the .fini section.

How can I find out what this is and where it's coming from?

Re: disturbing g++ 971031 results. defer-pop to blame?

[Jeffrey A Law]

  In message < 19971103011246.03918@dgii.com >you write:
  > GDB that I built from 4.14-95q4 (i486-unknown-sco3.2v5.0.0elf) handles 
  > the binaries in question just fine.   GDB 4.16 built from virgin FSF 
  > sources isn't useful on my ELF binaries.   I'll chase this later...
Sounds wise :-)

  > Program received signal SIGSEGV, Segmentation fault.
  > 0x8059187 in .fini ()
Now we're getting somewhere...

  > If I look at the tail end of the object, the reason for our
  > crash is more apparent:
  > 
  > Disassembly of section .fini:
  > 
  > 0805917c <_fini>:
  >  805917c:       e8 c7 f9 fe ff  call   8048b48 <__do_global_dtors_aux>
  >  8059181:       90              nop
  >  8059182:       90              nop
  >  8059183:       90              nop
  > 
  > 08059184 <.fini>:
  >  8059184:       4c              decl   %esp
  >  8059185:       cd 04           int    $0x4
  >  8059187:       08 c2           orb    %al,%dl
  > 
  > 08059188 <.fini>:
  >  8059188:       c2 00 00        ret    $0x0
  >  805918b:       90              nop
Hmmmm.
If you look at fini_dummy in crtstuff.c you'll see how this happened:

static void
fini_dummy ()
{
  asm (FINI_SECTION_ASM_OP);
  __do_global_dtors_aux ();
#ifdef FORCE_FINI_SECTION_ALIGN
  FORCE_FINI_SECTION_ALIGN;
#endif
  asm (TEXT_SECTION_ASM_OP);
}

We switch into the FINI section, emit the call, align the section,
then switch out -- without ever emitting a return.

So, when we return from do_global_dtors_aux, we just wander into a section,
which I'm guessing is data, not code.  Eventually something good or
bad happens...

The other possibility is the code at 0x8059184 is getting stomped on
sometime during the program execution (would assume that it's r/w which
might be true for the special initializer/finalizer sections.

I guess the first thing I'd do is determine what kind of stuff is
at 0x8059184 (code vs data) and move forward from there.

Jeff

Re: disturbing g++ 971031 results. defer-pop to blame?

[Andreas Schwab]

Jeffrey A Law <law@cygnus.com> writes:

|>   In message < 19971103011246.03918@dgii.com >you write:
|>   > GDB that I built from 4.14-95q4 (i486-unknown-sco3.2v5.0.0elf) handles 
|>   > the binaries in question just fine.   GDB 4.16 built from virgin FSF 
|>   > sources isn't useful on my ELF binaries.   I'll chase this later...
|> Sounds wise :-)

|>   > Program received signal SIGSEGV, Segmentation fault.
|>   > 0x8059187 in .fini ()
|> Now we're getting somewhere...

|>   > If I look at the tail end of the object, the reason for our
|>   > crash is more apparent:
|>   > 
|>   > Disassembly of section .fini:
|>   > 
|>   > 0805917c <_fini>:
|>   >  805917c:       e8 c7 f9 fe ff  call   8048b48 <__do_global_dtors_aux>
|>   >  8059181:       90              nop
|>   >  8059182:       90              nop
|>   >  8059183:       90              nop
|>   > 
|>   > 08059184 <.fini>:
|>   >  8059184:       4c              decl   %esp
|>   >  8059185:       cd 04           int    $0x4
|>   >  8059187:       08 c2           orb    %al,%dl
|>   > 
|>   > 08059188 <.fini>:
|>   >  8059188:       c2 00 00        ret    $0x0
|>   >  805918b:       90              nop
|> Hmmmm.
|> If you look at fini_dummy in crtstuff.c you'll see how this happened:

|> static void
|> fini_dummy ()
|> {
|>   asm (FINI_SECTION_ASM_OP);
|>   __do_global_dtors_aux ();
|> #ifdef FORCE_FINI_SECTION_ALIGN
|>   FORCE_FINI_SECTION_ALIGN;
|> #endif
|>   asm (TEXT_SECTION_ASM_OP);
|> }

|> We switch into the FINI section, emit the call, align the section,
|> then switch out -- without ever emitting a return.

This is ok, the final return is added by crtn.o which is linked in this
example at 08059188.  The invalid instructions between 08059184 and
08059188 are coming from a different object.

-- 
Andreas Schwab                                      "And now for something
schwab@issan.informatik.uni-dortmund.de              completely different"
schwab@gnu.org

Re: disturbing g++ 971031 results. defer-pop to blame?

[Robert Lipe]

>   > Program received signal SIGSEGV, Segmentation fault.
>   > 0x8059187 in .fini ()
> Now we're getting somewhere...
> 
>   > Disassembly of section .fini:
>   > 
>   > 0805917c <_fini>:
>   >  805917c:       e8 c7 f9 fe ff  call   8048b48 <__do_global_dtors_aux>
>   >  8059181:       90              nop
>   >  8059182:       90              nop
>   >  8059183:       90              nop
>   > 
>   > 08059184 <.fini>:
>   >  8059184:       4c              decl   %esp
>   >  8059185:       cd 04           int    $0x4
>   >  8059187:       08 c2           orb    %al,%dl
>   > 
>   > 08059188 <.fini>:
>   >  8059188:       c2 00 00        ret    $0x0
>   >  805918b:       90              nop
> Hmmmm.
> If you look at fini_dummy in crtstuff.c you'll see how this happened:
> 
> static void
> fini_dummy ()
> {
>   asm (FINI_SECTION_ASM_OP);
>   __do_global_dtors_aux ();
> #ifdef FORCE_FINI_SECTION_ALIGN
>   FORCE_FINI_SECTION_ALIGN;
> #endif
>   asm (TEXT_SECTION_ASM_OP);
> }
> 
> We switch into the FINI section, emit the call, align the section,
> then switch out -- without ever emitting a return.

Right.   This code never emits a return.   The code in the OS-provided
crtn.o does.   This is why it's linked last.

> So, when we return from do_global_dtors_aux, we just wander into a section,
> which I'm guessing is data, not code.  Eventually something good or
> bad happens...

I don't think we wander anywhere.   We return, execute a couple of nops
then slam into some completely bogus-looking opcodes.   Of course, if 
those bogus-looking opcodes were less bogus, we'd then execute that 
perfectly lovely ret that crtn.o gave us.

> The other possibility is the code at 0x8059184 is getting stomped on
> sometime during the program execution (would assume that it's r/w which
> might be true for the special initializer/finalizer sections.

It's definitely in the executable that way.   The results I 
included are the output of an 'objdump --disassemble'.  A 'dis'
(approximate SCO equivalent of the same, only lamer) confirms 
this.   So nobody got a chance to trounce those sections.

objdump shows that .fini is read-only:

  8 .fini         00000010  0805917c  0805917c  0001117c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE


> I guess the first thing I'd do is determine what kind of stuff is
> at 0x8059184 (code vs data) and move forward from there.

It *should be* code, becuase this is what we're going to start
executing at tear-down time .   It looks like it *is* data or 
random trash).

If nobody recognizes those bytes as something useful that just
landed in the wrong section, I suppose I'll have to find where
they're coming from.

RJL

Re: disturbing g++ 971031 results. defer-pop to blame?

[Jason Merrill]

>>>>> Jeffrey A Law <law@cygnus.com> writes:

> We switch into the FINI section, emit the call, align the section,
> then switch out -- without ever emitting a return.

The assumption is that the OS crti.o and crtn.o provide the prologue and
epilogue for _init and _fini.

Jason

Re: disturbing g++ 971031 results. defer-pop not to blame.

[Robert Lipe]

Found it.

[ disassembly showing crap in .fini deleted ] 

> If I take hte final link line:
> 
>  /play/negcs/gcc/ld -b elf -Ra,XPG4PLUS,ELF -YP,/usr/ccs/lib:/lib:/usr/lib \
>         -Qn /usr/ccs/lib/crt1.o /usr/ccs/lib/values-Xa.o \
>         /play/negcs/gcc/crtbegin.o -L/play/negcs/gcc -L/usr/ccs/bin \
>         -L/usr/ccs/lib -L/usr/local/lib net34.o -lstdc++ \
>         -lgcc -lcrypt -lgen -lc -lgcc /play/negcs/gcc/crtend.o \
>         /usr/ccs/lib/crtn.o
> 
> and play with the order (add crtn.o before or after -lstdc++) I have 
> reason to think that it's from something in libstdc++.   If I just
> hoist -lstdc++ down to the very last (and stick another -lgcc after
> it) I get a functioning executable.
> 
> Objdump --all-headers on libstdc++.a and all the .o's that go into
> it shows nothing in the .fini section.
> 
> How can I find out what this is and where it's coming from?

Ugh.  I'm going to be ill.

The stupid final link line listed above includes a '-L/usr/local/lib'
but does not include a -L/play/negcs/libraries.    This means that 
while I spent hours staring at a disassembly of 
/play/negcs/libraries/libstdc++/libstdc++.a that looked perfectly 
reasonable, the linker was using one from /usr/local/lib from 
June 2 that happens to contain crap in the .fini section.

Of course, the obvious way to fix this is to add -L/play/libraries
to the link line - but then '-mcoff' picks up the wrong libraries.

Part of me wonders why this is dependent upon the level of 
optimization and the stack layout of the net34.o object when 
the problem is "clearly" in a linked-in library, and part of 
me just wants to rewire my tests to manually include the 
appropriate libes for ELF and COFF and get on with my life.

RJL