[Bug jit/64722] New: On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[Bug jit/64722] New: On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

            Bug ID: 64722
           Summary: On 2nd time libgccjit is run in-process on i686,
                    generated code clobbers %ebx register
           Product: gcc
           Version: 5.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: jit
          Assignee: dmalcolm at gcc dot gnu.org
          Reporter: dmalcolm at gcc dot gnu.org
             Build: i686-pc-linux-gnu

I'm seeing a crash on i686 for various test cases, each
within "main", getting ready to begin the *3rd* iteration of the test.

  FAIL: test-array-as-pointer.c.exe killed: 37097 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-calling-external-function.c.exe killed: 37185 exp8 0 0 CHILDKILLED
SIGSEGV {segmentation violation}
  FAIL: test-combination.c.exe killed: 37273 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-expressions.c.exe killed: 38499 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-fibonacci.c.exe killed: 38587 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-functions.c.exe killed: 38627 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-hello-world.c.exe killed: 38695 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-nested-contexts.c.exe killed: 38831 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-operator-overloading.cc.exe killed: 38919 exp8 0 0 CHILDKILLED
SIGSEGV {segmentation violation}
  FAIL: test-quadratic.c.exe killed: 38959 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-quadratic.cc.exe killed: 39000 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-reading-struct.c.exe killed: 39040 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}
  FAIL: test-threads.c.exe killed: 39176 exp8 0 0 CHILDKILLED SIGSEGV
{segmentation violation}

All of them crash here in main, trying to read "argv", apparently something has
trashed %ebx:

(gdb) disassemble main
Dump of assembler code for function main:
   0x080499c0 <+0>:    lea    0x4(%esp),%ecx
   0x080499c4 <+4>:    and    $0xfffffff0,%esp
   0x080499c7 <+7>:    pushl  -0x4(%ecx)
   0x080499ca <+10>:    push   %ebp
   0x080499cb <+11>:    mov    %esp,%ebp
   0x080499cd <+13>:    push   %ebx
   0x080499ce <+14>:    push   %ecx
   0x080499cf <+15>:    sub    $0x10,%esp
   0x080499d2 <+18>:    mov    %ecx,%ebx
   0x080499d4 <+20>:    movl   $0x1,-0xc(%ebp)
   0x080499db <+27>:    jmp    0x8049a25 <main+101>
=> 0x080499dd <+29>:    mov    0x4(%ebx),%eax
   0x080499e0 <+32>:    mov    (%eax),%eax
   0x080499e2 <+34>:    sub    $0xc,%esp
   0x080499e5 <+37>:    push   %eax
   0x080499e6 <+38>:    call   0x8049982 <extract_progname>
   0x080499eb <+43>:    add    $0x10,%esp
   0x080499ee <+46>:    sub    $0x8,%esp
   0x080499f1 <+49>:    push   $0x5
   0x080499f3 <+51>:    pushl  -0xc(%ebp)
   0x080499f6 <+54>:    push   %eax
   0x080499f7 <+55>:    push   $0x804a06b
   0x080499fc <+60>:    push   $0x400
   0x08049a01 <+65>:    push   $0x804c340
   0x08049a06 <+70>:    call   0x8048fc0 <snprintf@plt>
   0x08049a0b <+75>:    add    $0x20,%esp
   0x08049a0e <+78>:    mov    0x4(%ebx),%eax
   0x08049a11 <+81>:    mov    (%eax),%eax
   0x08049a13 <+83>:    sub    $0x8,%esp
   0x08049a16 <+86>:    push   $0x0
   0x08049a18 <+88>:    push   %eax
   0x08049a19 <+89>:    call   0x80498be <test_jit>
   0x08049a1e <+94>:    add    $0x10,%esp
   0x08049a21 <+97>:    addl   $0x1,-0xc(%ebp)
   0x08049a25 <+101>:    cmpl   $0x5,-0xc(%ebp)
   0x08049a29 <+105>:    jle    0x80499dd <main+29>
   0x08049a2b <+107>:    call   0x8049335 <totals>
   0x08049a30 <+112>:    mov    $0x0,%eax
   0x08049a35 <+117>:    lea    -0x8(%ebp),%esp
   0x08049a38 <+120>:    pop    %ecx
   0x08049a39 <+121>:    pop    %ebx
   0x08049a3a <+122>:    pop    %ebp
   0x08049a3b <+123>:    lea    -0x4(%ecx),%esp
   0x08049a3e <+126>:    ret
End of assembler dump.

Investigating test-hello-world.c and the generated "hello_world":
(gdb) break hello_world

On iteration 1:
(gdb) disassemble hello_world
Dump of assembler code for function hello_world:
   0xf671a510 <+0>:    push   %ebp
   0xf671a511 <+1>:    mov    %esp,%ebp
   0xf671a513 <+3>:    push   %ebx
=> 0xf671a514 <+4>:    sub    $0x4,%esp
   0xf671a517 <+7>:    call   0xf671a53d <__x86.get_pc_thunk.ax>
   0xf671a51c <+12>:    add    $0x1ae4,%eax
   0xf671a521 <+17>:    sub    $0x8,%esp
   0xf671a524 <+20>:    pushl  0x8(%ebp)
   0xf671a527 <+23>:    lea    -0x1aa8(%eax),%edx
   0xf671a52d <+29>:    push   %edx
   0xf671a52e <+30>:    mov    %eax,%ebx
   0xf671a530 <+32>:    call   0xf671a3b0 <printf@plt>
   0xf671a535 <+37>:    add    $0x10,%esp
   0xf671a538 <+40>:    mov    -0x4(%ebp),%ebx
   0xf671a53b <+43>:    leave  
   0xf671a53c <+44>:    ret    
End of assembler dump.

but on iteration 2:
(gdb) cont
ITERATION 2
Breakpoint 1, 0xf671a516 in hello_world (name=0x804a35a "world") from
/tmp/libgccjit-9zApb1/fake.so
(gdb) disassemble 
Dump of assembler code for function hello_world:
   0xf671a510 <+0>:    push   %ebp
   0xf671a511 <+1>:    mov    %esp,%ebp
   0xf671a513 <+3>:    sub    $0x8,%esp
=> 0xf671a516 <+6>:    call   0xf671a539 <__x86.get_pc_thunk.ax>
   0xf671a51b <+11>:    add    $0x1ae5,%eax
   0xf671a520 <+16>:    sub    $0x8,%esp
   0xf671a523 <+19>:    pushl  0x8(%ebp)
   0xf671a526 <+22>:    lea    -0x1aac(%eax),%edx
   0xf671a52c <+28>:    push   %edx
   0xf671a52d <+29>:    mov    %eax,%ebx
   0xf671a52f <+31>:    call   0xf671a3b0 <printf@plt>
   0xf671a534 <+36>:    add    $0x10,%esp
   0xf671a537 <+39>:    leave  
   0xf671a538 <+40>:    ret    
End of assembler dump.

If I'm reading this right, iteration 1's generated "hello_world" saves and
restores %ebx, but iteration 2's version of it is clobbering %ebx.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #2 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Indeed, adding this to toplev.c fixes the issue:
@@ -2166,7 +2169,8 @@ toplev::finalize (void)
 {
   rtl_initialized = false;
   this_target_rtl->target_specific_initialized = false;
+  this_target_rtl->x_pic_offset_table_rtx = NULL;

   /* Needs to be called before cgraph_c_finalize since it uses symtab.  */
   ipa_reference_c_finalize ();

fixes the issue; 5 iterations of test-hello-world.c run to completion.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #1 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
After some debugging, root cause appears to be in initialization of
pic_offset_table_rtx.

In i386.h:
/* Register to hold the addressing base for position independent
   code access to data items.  We don't use PIC pointer for 64bit
   mode.  Define the regnum to dummy value to prevent gcc from
   pessimizing code dealing with EBX.

   To avoid clobbering a call-saved register unnecessarily, we renumber
   the pic register when possible.  The change is visible after the
   prologue has been emitted.  */

#define REAL_PIC_OFFSET_TABLE_REGNUM  (TARGET_64BIT ? R15_REG : BX_REG)

#define PIC_OFFSET_TABLE_REGNUM                        \
  ((TARGET_64BIT && (ix86_cmodel == CM_SMALL_PIC            \
                     || TARGET_PECOFF))                    \
   || !flag_pic                                \
   ? INVALID_REGNUM                            \
   : pic_offset_table_rtx                        \
     ? INVALID_REGNUM                            \
     : REAL_PIC_OFFSET_TABLE_REGNUM)

and in emit-rtl.c:init_emit_regs:
  if ((unsigned) PIC_OFFSET_TABLE_REGNUM != INVALID_REGNUM)
    pic_offset_table_rtx = gen_raw_REG (Pmode, PIC_OFFSET_TABLE_REGNUM);
  else
    pic_offset_table_rtx = NULL_RTX;

Adding some debugging prints there:
--- a/gcc/emit-rtl.c
+++ b/gcc/emit-rtl.c
@@ -5872,10 +5872,18 @@ init_emit_regs (void)
     = gen_raw_REG (Pmode, RETURN_ADDRESS_POINTER_REGNUM);
 #endif

+  debug (pic_offset_table_rtx);
   if ((unsigned) PIC_OFFSET_TABLE_REGNUM != INVALID_REGNUM)
-    pic_offset_table_rtx = gen_raw_REG (Pmode, PIC_OFFSET_TABLE_REGNUM);
+    {
+      printf ("calling gen_raw_REG\n");
+      pic_offset_table_rtx = gen_raw_REG (Pmode, PIC_OFFSET_TABLE_REGNUM);
+    }
   else
-    pic_offset_table_rtx = NULL_RTX;
+    {
+      printf ("not calling gen_raw_REG\n");
+      pic_offset_table_rtx = NULL_RTX;
+    }
+  debug (pic_offset_table_rtx);

   for (i = 0; i < (int) MAX_MACHINE_MODE; i++)
     {

For 32-bit PIC, there appears to be some circular logic in this
initialization; if this has already been called, then
pic_offset_table_rtx is non-NULL, so PIC_OFFSET_TABLE_REGNUM
is INVALID_REGNUM, and so on reinitialization, it transitions back to
NULL.

Indeed, with test-empty.c, we see the NULL/non-NULL-ness of
pic_offset_table_rtx toggle at each iteration:
ITERATION 1
        PASSED: test-empty.c.exe iteration 1 of 5: set_up_logging: logfile is
non-null
        NOTE: test-empty.c.exe iteration 1 of 5: writing reproducer to
./test-empty.c.exe.reproducer.c
 test-empty.c.exe iteration 1 of 5: writing reproducer to .
<nil>
calling gen_raw_REG
(reg:SI 3 bx)
intermediate files written to /tmp/libgccjit-n2vMx2
        PASSED: test-empty.c.exe iteration 1 of 5: verify_code: result is
non-null

ITERATION 2
        PASSED: test-empty.c.exe iteration 2 of 5: set_up_logging: logfile is
non-null
        NOTE: test-empty.c.exe iteration 2 of 5: writing reproducer to
./test-empty.c.exe.reproducer.c
 test-empty.c.exe iteration 2 of 5: writing reproducer to .
(reg:SI 3 bx)
not calling gen_raw_REG
<nil>
intermediate files written to /tmp/libgccjit-qF4SVX
        PASSED: test-empty.c.exe iteration 2 of 5: verify_code: result is
non-null

ITERATION 3
        PASSED: test-empty.c.exe iteration 3 of 5: set_up_logging: logfile is
non-null
        NOTE: test-empty.c.exe iteration 3 of 5: writing reproducer to
./test-empty.c.exe.reproducer.c
 test-empty.c.exe iteration 3 of 5: writing reproducer to .
<nil>
calling gen_raw_REG
(reg:SI 3 bx)
intermediate files written to /tmp/libgccjit-mw8kAT
        PASSED: test-empty.c.exe iteration 3 of 5: verify_code: result is
non-null

ITERATION 4
        PASSED: test-empty.c.exe iteration 4 of 5: set_up_logging: logfile is
non-null
        NOTE: test-empty.c.exe iteration 4 of 5: writing reproducer to
./test-empty.c.exe.reproducer.c
 test-empty.c.exe iteration 4 of 5: writing reproducer to .
(reg:SI 3 bx)
not calling gen_raw_REG
<nil>
intermediate files written to /tmp/libgccjit-h63OvP
        PASSED: test-empty.c.exe iteration 4 of 5: verify_code: result is
non-null

ITERATION 5
        PASSED: test-empty.c.exe iteration 5 of 5: set_up_logging: logfile is
non-null
        NOTE: test-empty.c.exe iteration 5 of 5: writing reproducer to
./test-empty.c.exe.reproducer.c
 test-empty.c.exe iteration 5 of 5: writing reproducer to .
<nil>
calling gen_raw_REG
(reg:SI 3 bx)
intermediate files written to /tmp/libgccjit-hzBrIL
        PASSED: test-empty.c.exe iteration 5 of 5: verify_code: result is
non-null

i.e. on odd-numbered iterations it transitions from NULL to reg bx
on even-numbered iterations it transitions from non-NULL back to NULL.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ienkovich at gcc dot gnu.org,
                   |                            |jakub at gcc dot gnu.org

--- Comment #3 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
But then wonder if/how target_reinit works for i?86 32-bit.
Perhaps pic_offset_table_rtx should be cleared in init_emit_regs before
computing it?
  pic_offset_table_rtx = NULL_RTX;
  if ((unsigned) PIC_OFFSET_TABLE_REGNUM != INVALID_REGNUM)
    pic_offset_table_rtx = gen_raw_REG (Pmode, PIC_OFFSET_TABLE_REGNUM);

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[enkovich.gnu at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

Ilya Enkovich <enkovich.gnu at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |enkovich.gnu at gmail dot com

--- Comment #4 from Ilya Enkovich <enkovich.gnu at gmail dot com> ---
(In reply to Jakub Jelinek from comment #3)
> But then wonder if/how target_reinit works for i?86 32-bit.
> Perhaps pic_offset_table_rtx should be cleared in init_emit_regs before
> computing it?
>   pic_offset_table_rtx = NULL_RTX;
>   if ((unsigned) PIC_OFFSET_TABLE_REGNUM != INVALID_REGNUM)
>     pic_offset_table_rtx = gen_raw_REG (Pmode, PIC_OFFSET_TABLE_REGNUM);

Clearing pic_offset_table_rtx here would mean PIC_OFFSET_TABLE_REGNUM tranfroms
into EBX and pic_offset_table_rtx is initialized with EBX which is not what we
want.  Probably we just shouldn't try to initialize pic_offset_table_rtx with a
hard reg in case target assumes pseudo pic reg?

diff --git a/gcc/emit-rtl.c b/gcc/emit-rtl.c
index df85366..51ef3a5 100644
--- a/gcc/emit-rtl.c
+++ b/gcc/emit-rtl.c
@@ -5872,7 +5872,8 @@ init_emit_regs (void)
     = gen_raw_REG (Pmode, RETURN_ADDRESS_POINTER_REGNUM);
 #endif

-  if ((unsigned) PIC_OFFSET_TABLE_REGNUM != INVALID_REGNUM)
+  if (!targetm.use_pseudo_pic_reg ()
+      && (unsigned) PIC_OFFSET_TABLE_REGNUM != INVALID_REGNUM)
     pic_offset_table_rtx = gen_raw_REG (Pmode, PIC_OFFSET_TABLE_REGNUM);
   else
     pic_offset_table_rtx = NULL_RTX;

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #5 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Can you explain it?  Usually when this function is called, pic_offset_table_rtx
is NULL and your i386.h macro relies on that.
When initializing default target during initialization it is NULL of course,
and apparently even in target_reinit, when it is called with freshly cleared
heap object for the non-default target.
It is just when jit calls the initialization again without clearing all the
variables...
So I believe my proposed change is correct.

In any case, perhaps jit shouldn't reinitialize everything all the time, at
least if the compilation options don't change.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #6 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Also, if you keep pic_offset_table_rtx NULL, then PIC_OFFSET_TABLE_REGNUM will
be 3 instead of -1 in other places.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #7 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
(In reply to Jakub Jelinek from comment #3)
> But then wonder if/how target_reinit works for i?86 32-bit.
> Perhaps pic_offset_table_rtx should be cleared in init_emit_regs before
> computing it?
>   pic_offset_table_rtx = NULL_RTX;
>   if ((unsigned) PIC_OFFSET_TABLE_REGNUM != INVALID_REGNUM)
>     pic_offset_table_rtx = gen_raw_REG (Pmode, PIC_OFFSET_TABLE_REGNUM);

Adding the line:
    pic_offset_table_rtx = NULL_RTX;
does indeed fix the issue; with that (or with my patch from comment #2) the jit
testsuite fully passes on i686.

I was hoping to keep the patch confined to toplev::finalize, since only the jit
calls this, reducing the scope of the proposed change, but if you think it's
better to have it in init_emit_regs, I'll defer to your judgement.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[enkovich.gnu at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #8 from Ilya Enkovich <enkovich.gnu at gmail dot com> ---
different hooks(In reply to Jakub Jelinek from comment #5)
> Can you explain it?  Usually when this function is called,
> pic_offset_table_rtx is NULL and your i386.h macro relies on that.
> When initializing default target during initialization it is NULL of course,
> and apparently even in target_reinit, when it is called with freshly cleared
> heap object for the non-default target.
> It is just when jit calls the initialization again without clearing all the
> variables...
> So I believe my proposed change is correct.
> 
> In any case, perhaps jit shouldn't reinitialize everything all the time, at
> least if the compilation options don't change.

I misunderstood places when init_emit_regs is called and my fix is incorrect.

It is still unclear to me how this initialization affects generated code.  IIRC
we let pic_offset_table_rtx be EBX only because of middle-end which calls
target hooks for code cost estimations.  In this case we needed some valid pic
reg to generate RTL for its estimation and EBX was used.  But in target code
pic_offset_table_rtx is initialized with pseudo register and value set in
init_emit_regs shouldn't matter.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #9 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
(In reply to Jakub Jelinek from comment #5)
> Can you explain it?  Usually when this function is called,
> pic_offset_table_rtx is NULL and your i386.h macro relies on that.
> When initializing default target during initialization it is NULL of course,
> and apparently even in target_reinit, when it is called with freshly cleared
> heap object for the non-default target.
> It is just when jit calls the initialization again without clearing all the
> variables...

Indeed: I believe the issue here is specifically with the JIT, which is
re-running the compiler multiple times in-process, and in particular, is
re-running init_emit_regs.

> So I believe my proposed change is correct.

Agreed, fwiw.

> In any case, perhaps jit shouldn't reinitialize everything all the time, at
> least if the compilation options don't change.

Ideally, yes.  Currently the out-of-process invocation of the driver (and thus
collect2) dominates the profile, and fixing that is probably GCC 6 material. 
I've tried hacking it out to directly invoke "as" and "ld", and after fixing
that, the per-compilation RTL reinit does show up significantly in the profile,
but that probably ought to be tracked as a separate bug.  I suspect that
avoiding re-initing RTL may have to wait for gcc 6.

In the meantime your patch from comment #3 seems a suitable workaround; thanks.
I'll attempt a bootstrap with it.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #10 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Notes to self on how I debugged this:
I added the following to harness.h:
  gcc_jit_context_set_bool_option (
    ctxt,
    GCC_JIT_BOOL_OPTION_DUMP_EVERYTHING,
    1);
  gcc_jit_context_set_bool_option (
    ctxt,
    GCC_JIT_BOOL_OPTION_KEEP_INTERMEDIATES,
    1);
thus populating each iteration's tempdir with logfiles (and turned off
optimizations).

Rename the tempdirs to iteration-1 and iteration-2 for the sake of sanity.

Diffing the logfiles:
  (for f in $(ls /tmp/iteration-1); do
      diff -up /tmp/iteration-1/$f /tmp/iteration-2/$f;
   done) | less
showed the first significant difference appears
in fake.c.234r.pro_and_epilogue:
--- /tmp/iteration-1/fake.c.234r.pro_and_epilogue       2015-01-22
03:31:55.243746546 -0500
+++ /tmp/iteration-2/fake.c.234r.pro_and_epilogue       2015-01-22
03:31:55.315744888 -0500
@@ -28,11 +26,11 @@ Dataflow summary:
 ;;  hardware regs used          7 [sp]
 ;;  regular block artificial uses       6 [bp] 7 [sp]
 ;;  eh block artificial uses    6 [bp] 7 [sp] 16 [argp] 20 [frame]
-;;  entry block defs    0 [ax] 1 [dx] 2 [cx] 3 [bx] 6 [bp] 7 [sp] 20 [frame]
+;;  entry block defs    0 [ax] 1 [dx] 2 [cx] 6 [bp] 7 [sp] 20 [frame]

i.e. "3 [bx]" no longer appears in entry block defs in iteration 2.

and it's in this dump that the RTL diverged; previous pass dumps show use a
use of bx in the call_insn, but this dump shows the save/restore
of ebx in iteration 1's code, and *not* in iteration 2's.

I put a breakpoint on emit_insn to find where the save insns are emitted:
Breakpoint 12, emit_insn (x=0xf653e900) at ../../src/gcc/emit-rtl.c:4922
4922    {
(gdb) call debug(x)
(set (mem:SI (pre_dec:SI (reg/f:SI 7 sp)) [0  S4 A8])
    (reg:SI 3 bx))
(gdb) bt
#0  emit_insn (x=0xf653e900) at ../../src/gcc/emit-rtl.c:4922
#1  0xf74d2744 in ix86_emit_save_regs () at
../../src/gcc/config/i386/i386.c:10282
#2  ix86_expand_prologue () at ../../src/gcc/config/i386/i386.c:11398
#3  0xf75bd6fa in gen_prologue () at ../../src/gcc/config/i386/i386.md:12187
#4  0xf7027836 in thread_prologue_and_epilogue_insns () at
../../src/gcc/function.c:5929
#5  0xf70280a5 in rest_of_handle_thread_prologue_and_epilogue () at
../../src/gcc/function.c:6499
#6  (anonymous namespace)::pass_thread_prologue_and_epilogue::execute
(this=0x8074610) at ../../src/gcc/function.c:6537
#7  0xf71a6ac5 in execute_one_pass (pass=0x8074610) at
../../src/gcc/passes.c:2326
#8  0xf71a6f57 in execute_pass_list_1 (pass=0x8074610) at
../../src/gcc/passes.c:2378
#9  0xf71a6f6a in execute_pass_list_1 (pass=0x8074450) at
../../src/gcc/passes.c:2379
#10 0xf71a6f6a in execute_pass_list_1 (pass=0x8073910, pass@entry=0x80717d0) at
../../src/gcc/passes.c:2379
#11 0xf71a6fc3 in execute_pass_list (fn=0xf6527064, pass=0x80717d0) at
../../src/gcc/passes.c:2389
#12 0xf6f338e0 in cgraph_node::expand (this=0xf65261c8) at
../../src/gcc/cgraphunit.c:1804
#13 0xf6f34517 in output_in_order (no_reorder=no_reorder@entry=false) at
../../src/gcc/cgraphunit.c:2042
#14 0xf6f3491d in symbol_table::compile (this=0xf6527000) at
../../src/gcc/cgraphunit.c:2287
#15 0xf6f362d9 in symbol_table::finalize_compilation_unit (this=0xf6527000) at
../../src/gcc/cgraphunit.c:2370
#16 0xf6eadcad in jit_langhook_write_globals () at
../../src/gcc/jit/dummy-frontend.c:231
#17 0xf725c098 in compile_file () at ../../src/gcc/toplev.c:606
#18 0xf6eaa4ac in do_compile () at ../../src/gcc/toplev.c:2048
#19 toplev::main (this=0xffffd4c7, argc=12, argv=0x804e918) at
../../src/gcc/toplev.c:2145
#20 0xf6ec7c43 in gcc::jit::playback::context::compile (this=0xffffd52c) at
../../src/gcc/jit/jit-playback.c:1721
#21 0xf6ec03c1 in gcc::jit::recording::context::compile (this=0x804d008) at
../../src/gcc/jit/jit-recording.c:1175
#22 0xf6eb5d7c in gcc_jit_context_compile (ctxt=0x804d008) at
../../src/gcc/jit/libgccjit.c:2201
#23 0x08049bd0 in test_jit (argv0=0xffffd7ea
"/root/gcc-git-jit/build-jit/gcc/testsuite/jit/test-hello-world.c.exe", 
    user_data=0x0) at /root/gcc-git-jit/src/gcc/testsuite/jit.dg/harness.h:352
#24 0x08049cc8 in main (argc=1, argv=0xffffd6b4) at
/root/gcc-git-jit/src/gcc/testsuite/jit.dg/harness.h:397

This revealed that ix86_emit_save_regs is only being called on
iteration 1, not on iteration 2.

Debugging ix86_expand_prologue showed differences in "frame":
                      iter 1   iter 2
nregs               = 1        0
reg_save_offset     = 8        12
sse_reg_save_offset = 12       8

In particular, nregs = 1 on iter1, but is 0 on iter2

nregs is computed in ix86_nsaved_regs (void)
  which loops over regno for real non-SSE registers calling:
    ix86_save_reg (unsigned int regno, bool maybe_eh_return)

within ix86_expand_prologue:
On iteration 1:
(gdb) p this_target_hard_regs->x_call_used_regs[3]
$42 = 0 '\000'
(gdb) p this_target_hard_regs->x_fixed_regs[3]
$43 = 0 '\000'
(gdb) p regs_ever_live[3]
$44 = true

On iteration 2:
(gdb) p this_target_hard_regs->x_call_used_regs[3]
$46 = 1 '\001'
(gdb) p this_target_hard_regs->x_fixed_regs[3]
$47 = 1 '\001'
(gdb) p regs_ever_live[3]
$48 = true

which led to investigating this code in ix86_conditional_register_usage:
4394  j = PIC_OFFSET_TABLE_REGNUM;
4395  if (j != INVALID_REGNUM)
4396    fixed_regs[j] = call_used_regs[j] = 1;
and line 4396 is bizarrely only called on the 2nd iteration, not the 1st,
which led me to investigate "PIC_OFFSET_TABLE_REGNUM", and discover what
appears to be the root cause, as described in comment #1.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[enkovich.gnu at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #11 from Ilya Enkovich <enkovich.gnu at gmail dot com> ---
(In reply to David Malcolm from comment #10)
> which led to investigating this code in ix86_conditional_register_usage:
> 4394  j = PIC_OFFSET_TABLE_REGNUM;
> 4395  if (j != INVALID_REGNUM)
> 4396    fixed_regs[j] = call_used_regs[j] = 1;
> and line 4396 is bizarrely only called on the 2nd iteration, not the 1st,
> which led me to investigate "PIC_OFFSET_TABLE_REGNUM", and discover what
> appears to be the root cause, as described in comment #1.

Now I see.  The problem also is in ix86_conditional_register_usage that relies
on pic_offset_table_rtx value.  As I said EBX value is used only to estimate
costs for middle-end.  Thus we shouldn't fix reg here if pseudo pic register is
used and correct code would be:

@@ -4388,7 +4388,7 @@ ix86_conditional_register_usage (void)

   /* The PIC register, if it exists, is fixed.  */
   j = PIC_OFFSET_TABLE_REGNUM;
-  if (j != INVALID_REGNUM)
+  if (j != INVALID_REGNUM && !ix86_use_pseudo_pic_reg ())
     fixed_regs[j] = call_used_regs[j] = 1;

   /* For 32-bit targets, squash the REX registers.  */

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #12 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Author: dmalcolm
Date: Fri Jan 23 15:57:42 2015
New Revision: 220044

URL: https://gcc.gnu.org/viewcvs?rev=220044&root=gcc&view=rev
Log:
PR jit/64722: fix corruption of %ebx on 32-bit i386 with libgccjit

gcc/ChangeLog:
    PR jit/64722
    * emit-rtl.c (init_emit_regs): Set pic_offset_table_rtx to
    NULL_RTX before testing PIC_OFFSET_TABLE_REGNUM, since the
    latter may be affected by the former (e.g. on i686).

Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/emit-rtl.c

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #13 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
(In reply to Ilya Enkovich from comment #11)
> (In reply to David Malcolm from comment #10)
> > which led to investigating this code in ix86_conditional_register_usage:
> > 4394  j = PIC_OFFSET_TABLE_REGNUM;
> > 4395  if (j != INVALID_REGNUM)
> > 4396    fixed_regs[j] = call_used_regs[j] = 1;
> > and line 4396 is bizarrely only called on the 2nd iteration, not the 1st,
> > which led me to investigate "PIC_OFFSET_TABLE_REGNUM", and discover what
> > appears to be the root cause, as described in comment #1.
> 
> Now I see.  The problem also is in ix86_conditional_register_usage that
> relies on pic_offset_table_rtx value.  As I said EBX value is used only to
> estimate costs for middle-end.  Thus we shouldn't fix reg here if pseudo pic
> register is used and correct code would be:
> 
> @@ -4388,7 +4388,7 @@ ix86_conditional_register_usage (void)
> 
>    /* The PIC register, if it exists, is fixed.  */
>    j = PIC_OFFSET_TABLE_REGNUM;
> -  if (j != INVALID_REGNUM)
> +  if (j != INVALID_REGNUM && !ix86_use_pseudo_pic_reg ())
>      fixed_regs[j] = call_used_regs[j] = 1;
> 
>    /* For 32-bit targets, squash the REX registers.  */

Ilya: I can't speak to the correctness of the above code or patch, but r220044
fixes the original issue I ran into.  Do you want me to keep this bug open, or
should we track the above in a separate PR?

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[enkovich.gnu at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

--- Comment #14 from Ilya Enkovich <enkovich.gnu at gmail dot com> ---
(In reply to David Malcolm from comment #13)
> 
> Ilya: I can't speak to the correctness of the above code or patch, but
> r220044 fixes the original issue I ran into.  Do you want me to keep this
> bug open, or should we track the above in a separate PR?

I think you may close this tracker if your issue is resolved.  Change I
mentioned is a minor fix I would like to have installed but I'll handle it
separately.

[Bug jit/64722] On 2nd time libgccjit is run in-process on i686, generated code clobbers %ebx register

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64722

David Malcolm <dmalcolm at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED

--- Comment #15 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
(In reply to Ilya Enkovich from comment #14)
> (In reply to David Malcolm from comment #13)
> > 
> > Ilya: I can't speak to the correctness of the above code or patch, but
> > r220044 fixes the original issue I ran into.  Do you want me to keep this
> > bug open, or should we track the above in a separate PR?
> 
> I think you may close this tracker if your issue is resolved.  Change I
> mentioned is a minor fix I would like to have installed but I'll handle it
> separately.
Thanks.  I'll close this out as RESOLVED FIXED then.