Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[mike stump]

> Date: Thu, 4 Oct 2001 20:58:17 -0400 (EDT)
> From: John Wehle <john@feith.com>
> To: gcc@gcc.gnu.org

> This platform has failed to bootstrap here for some time due to
> bootstrap comparison errors.

:-(

> I did some digging and generated a testcase for which the native
> (stage1) cc1 reload pass generates different rtl than a cross
> compiler.

The preference would be to actually find someone that can track down
the difference all the way, using the testcase you've already found,
and actually fix the real problem.  I'd tend to guess that there is
actually a subtle non-portability issue.

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[Andreas Schwab]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

mike stump <mrs@windriver.com> writes:

|> > From: Alexandre Oliva <aoliva@redhat.com>
|> > Date: 24 Oct 2001 17:48:35 -0200
|> 
|> > But I still don't see how this would lead to bootstrap compare
|> > failures, or to differences not showing up when the compiler is run
|> > inside a debugger.
|> 
|> For much the same reason that:
|> 
|> main() {
|>        char *a[2];
|>        a[0]=strdup("Hi");
|>        a[1]=strdup("Hello");
|>        qsort (a, 2, sizeof(char*), strcmp);
|>        printf("%s\n", a[0]);
|> }
|> 
|> might print Hi, or Hello.

Or even crash.

Andreas.

-- 
Andreas Schwab                                  "And now for something
Andreas.Schwab@suse.de				completely different."
SuSE Labs, SuSE GmbH, Schanzäckerstr. 10, D-90443 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[David Edelsohn]

>>>>> Richard Henderson writes:

>> Even without the debugger involvement, the address-based hash
>> prevents deterministically repeatable behavior from GCC if the storage
>> layout of the executable is changed by relinking.

Richard> Or when cross compiled.  I agree.

Richard> Anyhow, didn't the fix get acked?

	The work-around was installed, but the fix to the hash function
has not progressed.

David

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[Richard Henderson]

On Wed, Oct 24, 2001 at 05:59:59PM -0400, David Edelsohn wrote:
> 	The debugger requires non-shareable copies of text (executable,
> shared libraries, etc.) in order to set breakpoints.  The copies may shift
> the data layout of the process.

Ug.  A broken OS.

I repeat my assertion that nothing *should* change.  Most sane
operating systems can manage to copy-on-write those portions of
the shared libraries in which breakpoints are set.

AIX of course uses that segment based shared page table thing
for the system shared library segment.  Which is nice under normal
circumstances, but lousy when debugging.  IMO what ought to happen
is that ptrace(PTRACE_POKETEXT) (or whatever AIX uses) should not
only COW the shared library page, but also COW the page tables for
the entire segment and swap out the relevant page table register.

A vain hope, I know.

> 	Even without the debugger involvement, the address-based hash
> prevents deterministically repeatable behavior from GCC if the storage
> layout of the executable is changed by relinking.

Or when cross compiled.  I agree.

Anyhow, didn't the fix get acked?


r~

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[mike stump]

> Date: Wed, 24 Oct 2001 14:48:05 -0700
> From: Richard Henderson <rth@redhat.com>
> To: mike stump <mrs@windriver.com>

> On Wed, Oct 24, 2001 at 02:42:59PM -0700, mike stump wrote:
> > > From: Alexandre Oliva <aoliva@redhat.com>
> > > Date: 24 Oct 2001 17:48:35 -0200
> > 
> > > But I still don't see how this would lead to bootstrap compare
> > > failures, or to differences not showing up when the compiler is run
> > > inside a debugger.
> > 
> > For much the same reason that:
> [...]
> > might print Hi, or Hello.

> No.  The important part is >>when .. run inside a debugger<<.

No, the important part is bootstrap failure...  That was the part I
was mainly answering.

> The important part is >>when .. run inside a debugger<<.

bootstrap failures are in some ways, more important that differing
results in gdb.

> Nothing should change when run inside a debugger except timing, and
> gcc is not timing sensitive.

I agree, in a perfect world, this is the case.  However, if gcc ever
has a bug (and sometimes, it does), say, something like:

$ cat t5.c
main() {
       char *a[2];
       printf("%d\n", a);
       a[0]=strdup("Hi");
       a[1]=strdup("Hello");
       qsort (a, 2, sizeof(char*), strcmp);
       printf("%s\n", a[0]);
}

$ a.out
-4264688
Hi.

$ gdb a.out
GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.5.1"...

NAT>run
Starting program: /tmp/a.out
-4264680

then the behavior _can_ be dependent on such things.  We _can_ agree
that nothing in gcc should be sensitive to these things.  I was
answering the question of what to watch out for and _how_ it could
possibly be the case, _given_ that it happens.

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[David Edelsohn]

>>>>> Richard Henderson writes:

Richard> Nothing should change when run inside a debugger except timing,
Richard> and gcc is not timing sensitive.

	The debugger requires non-shareable copies of text (executable,
shared libraries, etc.) in order to set breakpoints.  The copies may shift
the data layout of the process.

	Even without the debugger involvement, the address-based hash
prevents deterministically repeatable behavior from GCC if the storage
layout of the executable is changed by relinking.

David

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[Richard Henderson]

On Wed, Oct 24, 2001 at 02:42:59PM -0700, mike stump wrote:
> > From: Alexandre Oliva <aoliva@redhat.com>
> > Date: 24 Oct 2001 17:48:35 -0200
> 
> > But I still don't see how this would lead to bootstrap compare
> > failures, or to differences not showing up when the compiler is run
> > inside a debugger.
> 
> For much the same reason that:
[...]
> might print Hi, or Hello.

No.  The important part is >>when .. run inside a debugger<<.

Nothing should change when run inside a debugger except timing,
and gcc is not timing sensitive.


r~

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[mike stump]

> From: Alexandre Oliva <aoliva@redhat.com>
> Date: 24 Oct 2001 17:48:35 -0200

> But I still don't see how this would lead to bootstrap compare
> failures, or to differences not showing up when the compiler is run
> inside a debugger.

For much the same reason that:

main() {
       char *a[2];
       a[0]=strdup("Hi");
       a[1]=strdup("Hello");
       qsort (a, 2, sizeof(char*), strcmp);
       printf("%s\n", a[0]);
}

might print Hi, or Hello.

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[Alexandre Oliva]

On Oct 23, 2001, Graham Stott <grahams@redhat.com> wrote:

> I thought this had been sorted out?

> Wasn't this fixed by this patch

May be, I'm way behind in my GCC mail folders :-(

But I still don't see how this would lead to bootstrap compare
failures, or to differences not showing up when the compiler is run
inside a debugger.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[Graham Stott]

Alex,

Alexandre Oliva wrote:
> 
> On Oct  5, 2001, David Edelsohn <dje@watson.ibm.com> wrote:
> 
> >       Using a debugger on the compiler producing different output caused
> > the compiler to stop emitting different output.
> 
> Gee!  I suppose the cat is neither dead nor alive, then :-)
> 
> If you run the compiler outside the debugger, does it reliably produce
> different output?  Given that timing doesn't play a role in GCC, I
> really don't see how a debugger could affect the output of the
> program, assuming you're not issuing commands to modify the state of
> the inferior program, such as setting variables, issuing calls, etc.
> 
> This makes me wonder whether there's some kernel problem that fails to
> preserve the upper 32 bits of 64-bit registers in certain corner cases
> on context switches, or something like that.  This would explain the
> irreproducibility of the failures, and the reason why it was when
> HOST_WIDE_INT became `long long' that it started showing up.
> 
> Does this make any sense to you?
> 
> --
> Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
> Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
> CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
> Free Software Evangelist    *Please* write to mailing lists, not to me

I thought this had been sorted out?

Wasn't this fixed by this patch

Wed Oct 10 00:41:29 EDT 2001  John Wehle  (john@feith.com)

        * rs6000.c (rs6000_emit_load_toc_table): Don't
        strdup the toc label.
        (create_TOC_reference): Likewise.

Graham

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[Alexandre Oliva]

On Oct  5, 2001, David Edelsohn <dje@watson.ibm.com> wrote:

> 	Using a debugger on the compiler producing different output caused
> the compiler to stop emitting different output.

Gee!  I suppose the cat is neither dead nor alive, then :-)


If you run the compiler outside the debugger, does it reliably produce
different output?  Given that timing doesn't play a role in GCC, I
really don't see how a debugger could affect the output of the
program, assuming you're not issuing commands to modify the state of
the inferior program, such as setting variables, issuing calls, etc.

This makes me wonder whether there's some kernel problem that fails to
preserve the upper 32 bits of 64-bit registers in certain corner cases
on context switches, or something like that.  This would explain the
irreproducibility of the failures, and the reason why it was when
HOST_WIDE_INT became `long long' that it started showing up.

Does this make any sense to you?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[David Edelsohn]

>>>>> Dale Johannsen writes:

Dale> Am I correct in thinking the problem occurs in reload (i.e. the first
Dale> dump file to differ is #20, or perhaps #21?  Which?)

	The differences begin in 22.postreload.  For example, cp/spew.c
has exactly one different instruction.  GCC chooses to assign the register
by loading the global symbol instead of copying it from a register.

--- stage1/spew.c.22.postreload Sat Oct  6 00:24:09 2001
+++ stage2/spew.c.22.postreload Sat Oct  6 00:25:14 2001
@@ -4317,7 +4317,9 @@
     (nil))
 
 (insn 788 806 787 (set (reg/f:SI 12 r12 [131])
-        (reg/v/f:SI 6 r6 [85])) 289 {*movsi_internal1} (nil)
+        (mem/u:SI (plus:SI (reg:SI 2 r2)
+                (const:SI (minus:SI (symbol_ref/u:SI ("*LC..68"))
+                        (symbol_ref:SI ("*LCTOC..1"))))) 10)) 289 {*movsi_internal1} (nil)
     (expr_list:REG_EQUIV (symbol_ref:SI ("token_obstack"))
         (nil)))

Thanks, David

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[David Edelsohn]

	Thanks for the advice.  I couldn't get the compiler to produce the
bad output when being observed by a debugger, but maybe a different
version or a different person can.

	I and others have tracked this failure mode back all of the way to
the H_W_I "long long" change.  Prior to CONST_INT holding 64-bit values on
a 32-bit host, the bootstrap succeeded.  None of the 32-bit PowerPC ports
seem to show any problem.  There are not many ports where one bootstraps a
32-bit compiler with 64-bit support (requiring H_W_I to be "long long" in
the new design).

	Has anyone seen any problems for 64-bit Irix bootstrap or 64-bit
Solaris bootstrap?  I assume that those compilers continue to be built as
32-bit applications matching the AIX target.  Most 32x64 configurations
are cross-compilers not bootstrapped for comparison where one would notice
a possible common bug in GCC's "long long" support.

	I looked through the GCC TOC handling code for any parameter
mismatches, but I could not find anything which looks suspicious.

Thanks, David

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[mike stump]

> To: mike stump <mrs@windriver.com>
> cc: john@feith.com, gcc@gcc.gnu.org
> Date: Thu, 04 Oct 2001 23:00:53 -0400
> From: David Edelsohn <dje@watson.ibm.com>

> 	Using a debugger on the compiler producing different output caused
> the compiler to stop emitting different output.  Debugging the compiler
> affected the problem, which is why I think this is a memory layout, memory
> corruption, or memory address issue.

> 	I and others tried debugging tricks #1 through #MAX_INT.  It's not
> that simple.

Standard debugging trick #136 would be to take the .ii file you know
can produce different output, and the date stamp for the compiler, and
have everyone with two or more possible environments update to that
time, build up the same compiler in all possible environments they
can, and then to see which versions of the output they get, and have
people try debug it, and see if we can't find a person with two
environments that yield different outputs that they can debug in.

For example, if one has a cygwin hosted compiler, and a solaris
compiler, one can see if they match, or are different.  Also, by
surveying it, we can deduce things about the environment that is
special.  If it is just gcc-2.95.x that miscompiles reload, then we
can put in the instructions, please compile up, non-bootstrap with a
new tool chain, and then recompile using bootstrap the compiler with
itself...  in this special case.  This sort of advice, is slightly
better than just using -O1, or hacking reload to be somehow different.

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[David Edelsohn]

	Using a debugger on the compiler producing different output caused
the compiler to stop emitting different output.  Debugging the compiler
affected the problem, which is why I think this is a memory layout, memory
corruption, or memory address issue.

	I and others tried debugging tricks #1 through #MAX_INT.  It's not
that simple.

David

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[mike stump]

> 	I have been unable to cleanly bootstrap GCC on the trunk since the
> HOST_WIDE_INT use of "long long" change.

> 	I would appreciate any help in trying to track this down as I am
> not having much luck.

First, isolate the .i (or .ii file) that differs in the assembly
output.  Then, isolate the two compilers that produce this.

From there, you can put a printf in routines that are heavily used,
and print out something that doesn't change about them INSN_UUID,
__func__, lineno... from compile to compile (not addresses of rtx
structures).  Run, and see if the .s output is still failing.  If it
is, you can then run diff on the debugging output (can be easier to do
this with diff, then doing it in on online session in gdb).  At the
first difference, you work in more prints between the last one that
worked and the current one.

Or, do this in an online way with two gdb's running side by side.  Run
with -da, find first difference, break on gen_rtx or add_insn or one
of those...  Go up a level, set a break point on calling site, rerun
to that temporal instance.  I usually binary search with cont 50, cont
100, cont 200, or, cont 25, cont 13, cont 7...  then step into the
routine, watching for the different.  In a large complex function,
binary searching with next 20, next 40 and so on, until you find a
difference is fairly quick.  And then you repeat.

After doing this for a day, it becomes easier and more automatic.
Also, you learn to slant the binary seach to favor running for a
shorter period of time, instead of in the middle, as if you miss it,
it takes more time and energy to get back to the right temporal spot.


This is standard debugging trick #135.  Now, if only I could find the
web pages that describes it in all is glory detail.

Re: powerpc-ibm-aix4.3.3.0 bootstrap issue

[David Edelsohn]

	I have been unable to cleanly bootstrap GCC on the trunk since the
HOST_WIDE_INT use of "long long" change.  The compiler works, but I get
random comparison errors of various files when bootstrapping.  I observed
that the resulting code always is correct and works, but stage2 and stage3
sometimes produces code reloading a TOC entry and sometimes copy the value
already in a register.

	I tried bootstrapping from scratch with AIX Visual Age C Compiler,
gcc-2.95.3, and gcc-3.0.1.  All encountered the similar problems -- not in
the exact same files, but all showed comparison failures.  I doubt that
all versions miscompile reload.c.

	I thought that it might be caused by an error in a hashtable
lookup or a hash value that depends on an address.  It could be something
non-deterministic in reload as well.  I and others spent a lot of time
over the summer building GCC with memory checking libraries and could not
find any wild writes that would corrupt something.

	This problem does not occur for 32-bit PowerPC which uses "long"
for HOST_WIDE_INT.  Maybe it is an error in one of the 32-bit DImode
patterns implementing "long long".

	I would appreciate any help in trying to track this down as I am
not having much luck.

Thanks, David

powerpc-ibm-aix4.3.3.0 bootstrap issue

[John Wehle]

This platform has failed to bootstrap here for some time due to bootstrap
comparison errors.  I did some digging and generated a testcase for which
the native (stage1) cc1 reload pass generates different rtl than a cross
compiler.  At a quick glance the differences seem to be related to whether
a reload comes from another register or from memory.  If I simply recompile
reload.o using -O2 than the native cc1 generates the same code as the cross
compiler.  I'm about to try bootstrapping the compiler using -O2.

It sounds like reload.c is being miscompiled by the installed compiler
(in this case gcc 2.95.3).  Do we want to hack reload.c so that bootstrapping
using gcc 2.95.3 works, or simply document that -O2 is required (assuming
I have success)?

-- John
-------------------------------------------------------------------------
|   Feith Systems  |   Voice: 1-215-646-8000  |  Email: john@feith.com  |
|    John Wehle    |     Fax: 1-215-540-5495  |                         |
-------------------------------------------------------------------------