dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[Scott Gilbertson]

I originally posted this on the java list, but the suggestion was made that
I post it here to see if somebody can help.

 My previous related posts to the java list (I was originally thinking I had
two separate problems):
   http://gcc.gnu.org/ml/java/2005-11/msg00230.html
   http://gcc.gnu.org/ml/java/2005-11/msg00229.html

I wonder if someone reasonably familiar with the unwinder can have a look at
the hacks documented below, and tell me whether they indicate a bug, or
conversely whether they provide a clue as to why the interpreter and
static-built executables aren't working for me, even though other people
seem to have no problem.  Thanks.

I now have a simple program like the following one working as static
binaries with recent gcc (this program crashed without my hacks).
  public class Nothing
  {
    public static void main (String[] args)
    {
    }
  }

More complex programs also now work, including AWT with xlib, and an example
that throws and catches an Exception (demonstrating, I think, that the
unwinder sometimes works in my environment).  The GIJ interpreter (which is
of course NOT statically linked) still aborts, and since I don't get any
symbols in its backtrace, I'm not sure how to pursue that.  My programs work
fine when built as dynamic executables even without the hacks.  The
interpreter works fine with the SJLJ unwinder (without the hacks), but
static executables don't.

I'm using this software:
  gij (GNU libgcj) version 4.1.0 20051116 (experimental)
  (revision 107090 from SVN, checked out 2005-11-16)
  gcj (GCC) 4.1.0 20051116 (experimental)
  (same sources)
  Linux version 2.6.8.1-10mdk
  (cpu is Pentium 4)
  binutils-2.15.90.0.3-1mdk
  glibc-2.3.3-21mdk

My original gcc configuration, which works fine with gcc-4_0-branch from
July
 ../gcc/configure
   --prefix=/var/local/gcc/tip_20051115
   --mandir=/var/local/gcc/man
   --infodir=/var/local/gcc/info
   --enable-shared
   --enable-threads=posix
   --disable-checking
   --host=i386-redhat-linux
   --enable-java-awt=xlib
   --enable-libgcj
   --enable-languages=c,c++,java
   --with-system-zlib
   --enable-__cxa_atexit

For my SJLJ unwinder tests, I added --enable-sjlj-exceptions

Here are the hacks I did to get the static builds working.  Do they offer
any clues?

HACK #1:
The first hack addresses this crash:
#9020 <signal handler called>
#9021 0x080921db in _Unwind_IteratePhdrCallback (info=0xbffff040, size=32,
ptr=0xbffff094)
    at ../../gcc/gcc/unwind-dw2-fde-glibc.c:262

Index: unwind-dw2-fde-glibc.c
===================================================================
--- unwind-dw2-fde-glibc.c      (revision 107090)
+++ unwind-dw2-fde-glibc.c      (working copy)
@@ -257,7 +257,7 @@ _Unwind_IteratePhdrCallback (struct dl_p

   if (size >= sizeof (struct ext_dl_phdr_info))
     {
-      if (last_cache_entry != NULL)
+      if (prev_cache_entry != NULL && last_cache_entry != NULL)
        {
          prev_cache_entry->link = last_cache_entry->link;
          last_cache_entry->link = frame_hdr_cache_head;
[end of hack diff]

So prev_cache_entry is obviously null sometimes, presumably because this
thing in the same file "found an unused entry":
       last_cache_entry = cache_entry;
       /* Exit early if we found an unused entry.  */
       if ((cache_entry->pc_low | cache_entry->pc_high) == 0)
         break;
       if (cache_entry->link != NULL)
         prev_cache_entry = cache_entry;

Looking at the changes to unwind-dw2-fde-glibc.c, I see that the parts of
the code I've shown here were structured differently in the 4.0 branch
(which works just fine for me with static builds).  Maybe that's a clue.

HACK #2:
The first hack got the unwinder to return to _Jv_Throw rather than crashing,
so I was able to get an error code: _URC_END_OF_STACK.
That leads to an abort in _Jv_Throw.  I did another backtrace (that's about
all I can do - gdb hangs up running these executables for some reason, so
I'm stuck with core dumps).

#0  0x08325081 in kill ()
#1  0x0830d1aa in __pthread_raise ()
#2  0x08325368 in abort ()
#3  0x0809dc9d in _Jv_Throw (value=0x4d828) at
../../../gcc/libjava/exception.cc:114
#4  0x08093537 in catch_segv (_dummy=Could not find the frame base for
"catch_segv".
) at ../../../gcc/libjava/prims.cc:152
#5  <signal handler called>
#6  0x080b6a65 in _Jv_FreeMethodCache () at
../../../gcc/libjava/java/lang/natClass.cc:941
#7  0x080bd279 in java::lang::Thread::finish_ (this=0x61f18)
    at ../../../gcc/libjava/java/lang/natThread.cc:219
#8  0x080943a2 in _Jv_RunMain (vm_args=0x0, klass=0x851d660, name=0x0,
argc=1, argv=0xbffff8a4,
    is_jar=false) at ../../../gcc/libjava/prims.cc:1386
#9  0x080944f8 in _Jv_RunMain (klass=0x851d660, name=0x0, argc=1,
argv=0xbffff8a4, is_jar=false)
    at ../../../gcc/libjava/prims.cc:1397
#10 0x0809452b in JvRunMain (klass=0x851d660, argc=1, argv=0xbffff8a4)
    at ../../../gcc/libjava/prims.cc:1403
#11 0x08048235 in main ()

Strange: natClass.cc:941 is just "if (method_cache != NULL)", which
shouldn't be able to cause a segv.  I'm guessing the error is actually in
_Jv_Free.  So I just commented out all the innards of the
_Jv_FreeMethodCache function, and now my simple test cases work.  Like so:
  void
  _Jv_FreeMethodCache ()
  {/*
  #ifdef HAVE_TLS
    if (method_cache != NULL)
      {
        _Jv_Free(method_cache);
        method_cache = NULL;
      }
  #endif // HAVE_TLS
  */}

I'm sure this function is important, and removing it is a bad idea, but it
prevents the abort in the static build case.  That let me confirm that no
more segv's happen after this one - the programs now terminate normally.
Again, I'm wondering whether this is a clue as to what's going on.  Any
insights would be greatly appreciated.

Re: dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[Richard Henderson]

On Fri, Nov 18, 2005 at 06:17:49PM -0500, Scott Gilbertson wrote:
> -      if (last_cache_entry != NULL)
> +      if (prev_cache_entry != NULL && last_cache_entry != NULL)

This one looks like it may be a legitimate problem with the list 
manipulation, though this isn't the proper fix.

> Looking at the changes to unwind-dw2-fde-glibc.c, I see that the parts of
> the code I've shown here were structured differently in the 4.0 branch
> (which works just fine for me with static builds).  Maybe that's a clue.

Um, I don't see that at all.  I see some minor changes wrt abort,
and one spot where we use a local temporary instead of storing the
value directly into data->func.

> _Jv_FreeMethodCache function, and now my simple test cases work.  Like so:
>   void
>   _Jv_FreeMethodCache ()
>   {/*
>   #ifdef HAVE_TLS
>     if (method_cache != NULL)

I will say that staticly linked linuxthreads tls is known to be
broken.  Or at least known to me.  I encountered this while doing
OpenMP work on RHEL3.  The problem I saw doesn't appear with nptl.


r~

Re: dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[Daniel Jacobowitz]

On Fri, Nov 18, 2005 at 05:48:26PM -0800, Richard Henderson wrote:
> > _Jv_FreeMethodCache function, and now my simple test cases work.  Like so:
> >   void
> >   _Jv_FreeMethodCache ()
> >   {/*
> >   #ifdef HAVE_TLS
> >     if (method_cache != NULL)
> 
> I will say that staticly linked linuxthreads tls is known to be
> broken.  Or at least known to me.  I encountered this while doing
> OpenMP work on RHEL3.  The problem I saw doesn't appear with nptl.

Got a handy testcase, or is it catastrophic?  There are a small number
of static tests in the linuxthreads testsuite and I've run it on HEAD
recently.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[Andrew Haley]

Richard Henderson writes:
  > 
 > I will say that staticly linked linuxthreads tls is known to be
 > broken.  Or at least known to me.  I encountered this while doing
 > OpenMP work on RHEL3.  The problem I saw doesn't appear with nptl.

Ah, that's a useful clue.  I can confirm that this is definitely
unreproducible on my system with NPTL.

Andrew.

Re: dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[Richard Henderson]

On Fri, Nov 18, 2005 at 10:47:40PM -0500, Daniel Jacobowitz wrote:
> Got a handy testcase, or is it catastrophic?

It's all-pervasive.

__thread int x;
int main() { return x; }


> There are a small number
> of static tests in the linuxthreads testsuite and I've run it on HEAD
> recently.

It's entirely possible that it's been fixed, but the systems in 
question are old enough that they pre-date the fix.  I know, for
example, that FC2 and RHEL3 are affected, but FC4 isn't.


r~

Re: dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[Daniel Jacobowitz]

On Sat, Nov 19, 2005 at 01:26:47PM -0800, Richard Henderson wrote:
> > There are a small number
> > of static tests in the linuxthreads testsuite and I've run it on HEAD
> > recently.
> 
> It's entirely possible that it's been fixed, but the systems in 
> question are old enough that they pre-date the fix.  I know, for
> example, that FC2 and RHEL3 are affected, but FC4 isn't.

Definitely works on glibc HEAD, then.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[Scott Gilbertson]

> > Looking at the changes to unwind-dw2-fde-glibc.c, I see that the parts
of
> > the code I've shown here were structured differently in the 4.0 branch
> > (which works just fine for me with static builds).  Maybe that's a clue.
>
> Um, I don't see that at all.  I see some minor changes wrt abort,...

Neither do I, now that you mention it.  Sorry about that... I must have been
looking at the wrong files or something.

> I will say that staticly linked linuxthreads tls is known to be
> broken.  Or at least known to me.  I encountered this while doing
> OpenMP work on RHEL3.  The problem I saw doesn't appear with nptl.

# getconf GNU_LIBPTHREAD_VERSION
linuxthreads-0.10

The problem doesn't appear with branch-4.0 (same glibc).  Do you suppose the
gcc has recently started using some busted glibc feature (busted in my
old-ish glibc, that is) that wasn't being used before, or is it a matter of
a bug having appeared in the gcc sources?

Should I static link with a newer glibc (I'm assuming I can do that without
changing my system glibc) to see if that fixes the problem?

Re: dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[Richard Henderson]

On Mon, Nov 21, 2005 at 12:09:08PM -0500, Scott Gilbertson wrote:
> The problem doesn't appear with branch-4.0 (same glibc).  Do you suppose the
> gcc has recently started using some busted glibc feature (busted in my
> old-ish glibc, that is) that wasn't being used before...

Well, a new feature, yes.  Though the patch did get backported to
the 4.0 branch.

2005-09-27  Tom Tromey  <tromey@redhat.com>

        PR libgcj/23367:
        * include/jvm.h (_Jv_FreeMethodCache): Declare.
        * java/lang/natClass.cc (MCACHE_SIZE): Conditional on HAVE_TLS.
        (struct _Jv_mcache): Likewise.
        (method_cache): Likewise.
        (_Jv_FindMethodInCache): Do nothing unless TLS is available.
        (_Jv_AddMethodToCache): Likewise.
        (_Jv_FreeMethodCache): New function.
        * java/lang/natThread.cc (finish_): Call _Jv_FreeMethodCache.
        * aclocal.m4, configure, include/config.h.in: Rebuilt.
        * configure.ac: Invoke GCC_CHECK_TLS.

If you're seeing the difference between the tip of 4.0 and mainline,
perhaps something is going wrong with the 4.0 configure check, such
that this feature isn't being enabled?

> Should I static link with a newer glibc (I'm assuming I can do that without
> changing my system glibc) to see if that fixes the problem?

Yes.


r~

Re: dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[Scott Gilbertson]

> If you're seeing the difference between the tip of 4.0 and mainline,
> perhaps something is going wrong with the 4.0 configure check, such
> that this feature isn't being enabled?

My snapshot of branch-4.0 pre-dates the change you described.

> > Should I static link with a newer glibc (I'm assuming I can do that
without
> > changing my system glibc) to see if that fixes the problem?
> Yes.

OK.  I think the following commands are right to compile and link $2.java
using a specific GLIBC (built from latest release).  It seems to work for
very simple programs - no abort unless you remove the asterisked arguments
(i.e. use my system glibc).  There's still something funny going on, though,
because java Throwable.printStackTrace doesn't print the right thing and AWT
programs go into an infinite loop.  I can't get the resulting binaries to
run in gdb (say "run" and it sits there until I kill gdb), so I don't know
where the AWT programs are hanging, other than that my printf at the start
of _Jv_Throw repeatedly prints, and the constructor for java.awt.Frame
(first time it tries to load the AWT Toolkit, I guess) never returns.

export LD_LIBRARY_PATH=/var/local/gcc/tip_20051115-dwarf2/lib
/var/local/gcc/tip_20051115-dwarf2/bin/gcj -v  -g -O2 -c -o $2.o $2.java
/var/local/gcc/tip_20051115-dwarf2/bin/gcj
    --verbose
    --main=$2
    -o $2.bin
*   -L/var/local/glibc/2.3.5/lib
    -static
    -static-libgcc
    -save-temps
    $2.o
    libgcjx.o
    -lstdc++
    -L.
    -lX11
*   -Wl,--dynamic-linker=/var/local/glibc/2.3.5/lib/ld-linux.so.2
(libgcjx.o is just everything from lib-gnu-awt-xlib.a in a single
relocatable object file)

Example of a program that works fine, static or dynamic:
  import java.util.Properties;
  class props
  {
   static public void main(String args[])
   {
    Properties props;
    props = System.getProperties();
    props.list(System.out);
   }
  }

Example of one that doesn't:
    public class Throw
    {
       public static void main (String[] args)
       {
            try
            {
                throw new Exception("This is an exception");
            }
            catch (Throwable t)
            {
                t.printStackTrace ();
            }
       }
    }
Running Throw, static build, with glibc 2.4.5, I get:
java.lang.Exception: This is an exception
   at java.lang.VMThrowable.fillInStackTrace (Unknown Source)
   at java.lang.VMThrowable.fillInStackTrace (Unknown Source)

With my system glibc, static build, same thing, but it aborts on exit.

With my system glibc, dynamic build, I get:
java.lang.Exception: This is an exception
   <<No stacktrace available>>

I haven't managed to get dynamic building with glibc 2.3.5 to work.  The
following command produces an executable that says it doesn't exist when you
try to run it:
/var/local/gcc/tip_20051115-dwarf2/bin/gcj
    --verbose
    --main=$2
    -Dawt.toolkit=gnu.awt.xlib.XToolkit
    -g
    -o $2.bin
    -L/var/local/glibc/2.3.5/lib
    -Wl,--dynamic-linker=/var/localWglibc/2.3.5/lib/ld-linux.so.2
    $2.java

With my branch-4.0 snapshot and either my system glibc or 2.3.5, static
Throw.bin produces this nice output:
java.lang.Exception: This is an exception
   at Throw.main(java.lang.String[]) (/home/scott/javaTests/Throw.java:7)
   at gnu.java.lang.MainThread.call_main()
(/home/scott/javaTests/../../../gcc/libjava/gnu/java/lang/natMainThread.cc:4
7)
   at gnu.java.lang.MainThread.run()
(/home/scott/javaTests/../../../gcc/libjava/gnu/java/lang/MainThread.java:10
5)

With my branch-4.0 snapshot, dynamic link with my system glibc, Throw.bin
produces this output, which is nearly as nice:
java.lang.Exception: This is an exception
   at Throw.main(java.lang.String[]) (/home/scott/javaTests/Throw.java:7)
   at gnu.java.lang.MainThread.call_main()
(/var/local/gcc/branch-4.0/lib/libgcj.so.6.0.0)
   at gnu.java.lang.MainThread.run()
(/var/local/gcc/branch-4.0/lib/libgcj.so.6.0.0)

Re: dwarf2 unwinder hacks get my static build going: Bug, or indication of what I'm doing wrong?

[David Daney]

Two thoughts:

1) Post this on gcc@ as undoubtedly RTH will have an opinion about it.

2) Perhaps your glibc build is somehow screwed up.  As it is some of the 
startup files in glibc that add the end of eh table markers.

David Daney

Scott Gilbertson wrote:
> I wonder if someone reasonably familiar with the unwinder can have a look at
> the hacks documented below, and tell me whether they indicate a bug, or
> conversely whether they provide a clue as to why the interpreter and
> static-built executables aren't working for me, even though other people
> seem to have no problem.  Thanks.
> 
> I now have a simple program like the following one working as static
> binaries with recent gcc (this program crashed without my hacks).
>   public class Nothing
>   {
>     public static void main (String[] args)
>     {
>     }
>   }
> 
> More complex programs also now work, including AWT with xlib, and an example
> that throws and catches an Exception (demonstrating, I think, that the
> unwinder sometimes works in my environment).  The GIJ interpreter (which is
> of course NOT statically linked) still aborts, and since I don't get any
> symbols in its backtrace, I'm not sure how to pursue that.  My programs work
> fine when built as dynamic executables even without the hacks.  The
> interpreter works fine with the SJLJ unwinder (without the hacks), but
> static executables don't.
> 
> I'm using this software:
>   gij (GNU libgcj) version 4.1.0 20051116 (experimental)
>   (revision 107090 from SVN, checked out 2005-11-16)
>   gcj (GCC) 4.1.0 20051116 (experimental)
>   (same sources)
>   Linux version 2.6.8.1-10mdk
>   (cpu is Pentium 4)
>   binutils-2.15.90.0.3-1mdk
>   glibc-2.3.3-21mdk
> 
> My original gcc configuration, which works fine with gcc-4_0-branch from
> July
>  ../gcc/configure
>    --prefix=/var/local/gcc/tip_20051115
>    --mandir=/var/local/gcc/man
>    --infodir=/var/local/gcc/info
>    --enable-shared
>    --enable-threads=posix
>    --disable-checking
>    --host=i386-redhat-linux
>    --enable-java-awt=xlib
>    --enable-libgcj
>    --enable-languages=c,c++,java
>    --with-system-zlib
>    --enable-__cxa_atexit
> 
> For my SJLJ unwinder tests, I added --enable-sjlj-exceptions
> 
> Here are the hacks I did to get the static builds working.  Do they offer
> any clues?
> 
> HACK #1:
> The first hack addresses this crash:
> #9020 <signal handler called>
> #9021 0x080921db in _Unwind_IteratePhdrCallback (info=0xbffff040, size=32,
> ptr=0xbffff094)
>     at ../../gcc/gcc/unwind-dw2-fde-glibc.c:262
> 
> Index: unwind-dw2-fde-glibc.c
> ===================================================================
> --- unwind-dw2-fde-glibc.c      (revision 107090)
> +++ unwind-dw2-fde-glibc.c      (working copy)
> @@ -257,7 +257,7 @@ _Unwind_IteratePhdrCallback (struct dl_p
> 
>    if (size >= sizeof (struct ext_dl_phdr_info))
>      {
> -      if (last_cache_entry != NULL)
> +      if (prev_cache_entry != NULL && last_cache_entry != NULL)
>         {
>           prev_cache_entry->link = last_cache_entry->link;
>           last_cache_entry->link = frame_hdr_cache_head;
> [end of hack diff]
> 
> So prev_cache_entry is obviously null sometimes, presumably because this
> thing in the same file "found an unused entry":
>        last_cache_entry = cache_entry;
>        /* Exit early if we found an unused entry.  */
>        if ((cache_entry->pc_low | cache_entry->pc_high) == 0)
>          break;
>        if (cache_entry->link != NULL)
>          prev_cache_entry = cache_entry;
> 
> Looking at the changes to unwind-dw2-fde-glibc.c, I see that the parts of
> the code I've shown here were structured differently in the 4.0 branch
> (which works just fine for me with static builds).  Maybe that's a clue.
> 
> HACK #2:
> The first hack got the unwinder to return to _Jv_Throw rather than crashing,
> so I was able to get an error code: _URC_END_OF_STACK.
> That leads to an abort in _Jv_Throw.  I did another backtrace (that's about
> all I can do - gdb hangs up running these executables for some reason, so
> I'm stuck with core dumps).
> 
> #0  0x08325081 in kill ()
> #1  0x0830d1aa in __pthread_raise ()
> #2  0x08325368 in abort ()
> #3  0x0809dc9d in _Jv_Throw (value=0x4d828) at
> ../../../gcc/libjava/exception.cc:114
> #4  0x08093537 in catch_segv (_dummy=Could not find the frame base for
> "catch_segv".
> ) at ../../../gcc/libjava/prims.cc:152
> #5  <signal handler called>
> #6  0x080b6a65 in _Jv_FreeMethodCache () at
> ../../../gcc/libjava/java/lang/natClass.cc:941
> #7  0x080bd279 in java::lang::Thread::finish_ (this=0x61f18)
>     at ../../../gcc/libjava/java/lang/natThread.cc:219
> #8  0x080943a2 in _Jv_RunMain (vm_args=0x0, klass=0x851d660, name=0x0,
> argc=1, argv=0xbffff8a4,
>     is_jar=false) at ../../../gcc/libjava/prims.cc:1386
> #9  0x080944f8 in _Jv_RunMain (klass=0x851d660, name=0x0, argc=1,
> argv=0xbffff8a4, is_jar=false)
>     at ../../../gcc/libjava/prims.cc:1397
> #10 0x0809452b in JvRunMain (klass=0x851d660, argc=1, argv=0xbffff8a4)
>     at ../../../gcc/libjava/prims.cc:1403
> #11 0x08048235 in main ()
> 
> Strange: natClass.cc:941 is just "if (method_cache != NULL)", which
> shouldn't be able to cause a segv.  I'm guessing the error is actually in
> _Jv_Free.  So I just commented out all the innards of the
> _Jv_FreeMethodCache function, and now my simple test cases work.  Like so:
>   void
>   _Jv_FreeMethodCache ()
>   {/*
>   #ifdef HAVE_TLS
>     if (method_cache != NULL)
>       {
>         _Jv_Free(method_cache);
>         method_cache = NULL;
>       }
>   #endif // HAVE_TLS
>   */}
> 
> I'm sure this function is important, and removing it is a bad idea, but it
> prevents the abort in the static build case.  That let me confirm that no
> more segv's happen after this one - the programs now terminate normally.
> Again, I'm wondering whether this is a clue as to what's going on.  Any
> insights would be greatly appreciated.
>