Re: gdb stack trace problems (Addendum)

Re: gdb stack trace problems (Addendum)

[Roland Schwingel]

Hi...

Mark Kettenis wrote on 22.04.2005 19:51:14:
 >    Hi Mark...
 >
 >    Have you already had some time to look into it?
 >
 > I did, but I didn't have find the time to reply yet ;-).
Thank you very much...
 
 > Anyway, there is a serious problem here.  The code you show is
 > basically undebuggable.
 > [...]
 > The only thing we can do here is trust the frame pointer, like we used
 > to do in the old days.  The problem with that is that it's very likely
 > to (silently) skip some function calls.  In your particular example it
 > probably would appear as if your code called SleepEx directly and
 > there would be no trace of Sleep at all.  That can be quite puzzling.
 >
 > I'm thinking about adding an option to instruct gdb to "trust" the
 > frame pointer.  I'm not going to make it the default though.  I really
 > prefer seeing an abviously wrong backtrace over gdb silently skipping
 > function calls in its backtraces.
Well the present situation is quite problematic for us. In past days 
(gdb 5.3)
when an application crashed we had a quite accurate backtrace. It wasn't
wrong (for us). Finding/Fixing bugs was very easy. GDB has been
very useful here.

With gdb 6.x we are no longer able to get a backtrace in these cases (in 
about
95% of all cases). This is a serious problem and makes gdb 6.x quite 
unusable
for us (and maybe others on windows). Having at least an option for enabling
that feature would be IMHO absolutely necessary. I wonder why other users
on windows haven't complained earlier about this problem.

So will you implement such an option? I am quite unfamiliar with the 
internals
of gdb's stack unwinding, so I am not of much help here. But whenever I can
do something to help to fix this I am happily volunteering to do so.

Thanks again for your help Mark!

Roland

Re: gdb stack trace problems (Addendum)

[Roland Schwingel]

Hi...

Mark Kettenis wrote on 22.04.2005 19:51:14:
 >    Hi Mark...
 >
 >    Have you already had some time to look into it?
 >
 > I did, but I didn't have find the time to reply yet ;-).
Thank you very much...
 
 > Anyway, there is a serious problem here.  The code you show is
 > basically undebuggable.
 > [...]
 > The only thing we can do here is trust the frame pointer, like we used
 > to do in the old days.  The problem with that is that it's very likely
 > to (silently) skip some function calls.  In your particular example it
 > probably would appear as if your code called SleepEx directly and
 > there would be no trace of Sleep at all.  That can be quite puzzling.
 >
 > I'm thinking about adding an option to instruct gdb to "trust" the
 > frame pointer.  I'm not going to make it the default though.  I really
 > prefer seeing an abviously wrong backtrace over gdb silently skipping
 > function calls in its backtraces.
Well the present situation is quite problematic for us. In past days 
(gdb 5.3)
when an application crashed we had a quite accurate backtrace. It wasn't
wrong (for us). Finding/Fixing bugs was very easy. GDB has been
very useful here.

With gdb 6.x we are no longer able to get a backtrace in these cases (in 
about
95% of all cases). This is a serious problem and makes gdb 6.x quite 
unusable
for us (and maybe others on windows). Having at least an option for enabling
that feature would be IMHO absolutely necessary. I wonder why other users
on windows haven't complained earlier about this problem.

So will you implement such an option? I am quite unfamiliar with the 
internals
of gdb's stack unwinding, so I am not of much help here. But whenever I can
do something to help to fix this I am happily volunteering to do so.

Thanks again for your help Mark!

Roland

Re: gdb stack trace problems (Addendum)

[Mark Kettenis]

   Date: Mon, 25 Apr 2005 09:59:18 +0200
   From: Roland Schwingel <roland.schwingel@onevision.de>

    > Anyway, there is a serious problem here.  The code you show is
    > basically undebuggable.
    > [...]
    > The only thing we can do here is trust the frame pointer, like we used
    > to do in the old days.  The problem with that is that it's very likely
    > to (silently) skip some function calls.  In your particular example it
    > probably would appear as if your code called SleepEx directly and
    > there would be no trace of Sleep at all.  That can be quite puzzling.
    >
    > I'm thinking about adding an option to instruct gdb to "trust" the
    > frame pointer.  I'm not going to make it the default though.  I really
    > prefer seeing an abviously wrong backtrace over gdb silently skipping
    > function calls in its backtraces.
   Well the present situation is quite problematic for us. In past days 
   (gdb 5.3)
   when an application crashed we had a quite accurate backtrace. It wasn't
   wrong (for us). Finding/Fixing bugs was very easy. GDB has been
   very useful here.

You probably didn't notice it was wrong...

   With gdb 6.x we are no longer able to get a backtrace in these
   cases (in about 95% of all cases). This is a serious problem and
   makes gdb 6.x quite unusable for us (and maybe others on
   windows). Having at least an option for enabling that feature would
   be IMHO absolutely necessary. I wonder why other users on windows
   haven't complained earlier about this problem.

Oh they've complained, but never followed through when I asked for
information or tests.

   So will you implement such an option? I am quite unfamiliar with
   the internals of gdb's stack unwinding, so I am not of much help
   here. But whenever I can do something to help to fix this I am
   happily volunteering to do so.

Can you test the attached patch?  It introduces a new option named
"trust-frame-pointer".  Whenever you encounter a problem you can:

(gdb) set trust-frame-pointer 1

and try again.  You probably want to reset it to 0 before continuing
your program since I found out that bad things happen with some of the
tests in the gdb testsuite with this turned on.

Mark


Index: i386-tdep.c
===================================================================
RCS file: /cvs/src/src/gdb/i386-tdep.c,v
retrieving revision 1.209
diff -u -p -r1.209 i386-tdep.c
--- i386-tdep.c 6 Apr 2005 17:01:25 -0000 1.209
+++ i386-tdep.c 25 Apr 2005 19:24:54 -0000
@@ -858,10 +858,19 @@ i386_unwind_pc (struct gdbarch *gdbarch,
 
 /* Normal frames.  */
 
+static int trust_frame_pointer = 0;
+
+static void set_trust_frame_pointer (char *args, int from_tty,
+				     struct cmd_list_element *c)
+{
+  flush_cached_frames();
+}
+
 static struct i386_frame_cache *
 i386_frame_cache (struct frame_info *next_frame, void **this_cache)
 {
   struct i386_frame_cache *cache;
+  CORE_ADDR pc = 0;
   char buf[4];
   int i;
 
@@ -890,7 +899,8 @@ i386_frame_cache (struct frame_info *nex
 
   cache->pc = frame_func_unwind (next_frame);
   if (cache->pc != 0)
-    i386_analyze_prologue (cache->pc, frame_pc_unwind (next_frame), cache);
+    pc = i386_analyze_prologue (cache->pc, frame_pc_unwind (next_frame),
+				cache);
 
   if (cache->locals < 0)
     {
@@ -902,8 +912,11 @@ i386_frame_cache (struct frame_info *nex
 	 frame by looking at the stack pointer.  For truly "frameless"
 	 functions this might work too.  */
 
-      frame_unwind_register (next_frame, I386_ESP_REGNUM, buf);
-      cache->base = extract_unsigned_integer (buf, 4) + cache->sp_offset;
+      if (!trust_frame_pointer || pc == frame_pc_unwind (next_frame))
+	{
+	  frame_unwind_register (next_frame, I386_ESP_REGNUM, buf);
+	  cache->base = extract_unsigned_integer (buf, 4) + cache->sp_offset;
+	}
     }
 
   /* Now that we have the base address for the stack frame we can
@@ -2360,6 +2373,16 @@ is \"default\"."),
 			NULL, /* FIXME: i18n: */
 			&setlist, &showlist);
 
+  /* Add the variable that controls whether we trust the frame pointer.  */
+  add_setshow_boolean_cmd ("trust-frame-pointer", no_class,
+			   &trust_frame_pointer, _("\
+Set whether we trust the frame pointer"), _("\
+Show whether we trust the frame pointer"), _("\
+XXXX"),
+			   set_trust_frame_pointer,
+			   NULL, /* FIXME: i18n: */
+			   &setlist, &showlist);
+
   gdbarch_register_osabi_sniffer (bfd_arch_i386, bfd_target_coff_flavour,
 				  i386_coff_osabi_sniffer);
   gdbarch_register_osabi_sniffer (bfd_arch_i386, bfd_target_nlm_flavour,

Re: gdb stack trace problems (Addendum)

[Daniel Jacobowitz]

On Mon, Apr 25, 2005 at 09:26:32PM +0200, Mark Kettenis wrote:
> Can you test the attached patch?  It introduces a new option named
> "trust-frame-pointer".  Whenever you encounter a problem you can:
> 
> (gdb) set trust-frame-pointer 1

"set i386 trust-frame-pointer", please?

> and try again.  You probably want to reset it to 0 before continuing
> your program since I found out that bad things happen with some of the
> tests in the gdb testsuite with this turned on.

Hmm, what sort of bad things?


-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: gdb stack trace problems (Addendum)

[Mark Kettenis]

   Date: Mon, 25 Apr 2005 15:34:24 -0400
   From: Daniel Jacobowitz <drow@false.org>

   On Mon, Apr 25, 2005 at 09:26:32PM +0200, Mark Kettenis wrote:
   > Can you test the attached patch?  It introduces a new option named
   > "trust-frame-pointer".  Whenever you encounter a problem you can:
   > 
   > (gdb) set trust-frame-pointer 1

   "set i386 trust-frame-pointer", please?

That can be arranged, although this isn't something i386-specific per
se.

   > and try again.  You probably want to reset it to 0 before continuing
   > your program since I found out that bad things happen with some of the
   > tests in the gdb testsuite with this turned on.

   Hmm, what sort of bad things?

Mostly "next" running off into some random startup code instead of
stopping at the next line.  It adds about 75 failures to my testsuite
run on i386-unknown-openbsd3.7 (which has gcc 3.3.5).

Anyway, this leaves me not completely convinced that this is a good
idea.  If Windows DLL's are really so undebuggable as Roland showed,
it might be a good idea to add a Windows-specific DLL unwinder.  But
that requires some action from someone with a working Windows system
;-).

Mark

Re: gdb stack trace problems (Addendum)

[Roland Schwingel]

Hi...

gdb-owner@sources.redhat.com wrote on 09.05.2005 01:19:53:
 > On Sun, May 08, 2005 at 03:30:20PM +0200, Mark Kettenis wrote:
 > >   Date: Mon, 02 May 2005 09:03:08 +0200
 > >   From: Roland Schwingel <roland.schwingel@onevision.de>
 > >
 > >   Hi Mark...
 > >
 > >   Have you already had some time to look into my results with your 
patch to
 > >   the i386 stack unwinder? At basically it could work but obviously it
 > >   is not advancing to the next stack frame... Attached you will 
find my
 > >   results
 > >
 > >This isn't very encouraging.  My approach obviously isn't working very
 > >well.  My suggestion is to go for a Windows-specific solution, where
 > >one would use a special unwinder for the sort of undebuggable code
 > >that's found in the Windows system DLLs.  But I'm afraid I can't
 > >really do much since I don't have a Windows system.  Chris, is there
 > >any change you can hack something like this into i386-cygwin-tdep.c?
 >
 > What kind of windows-specific solution do you have in mind?  How would
 > you know what to unwind?  You could potentially figure out that you're
 > stuck in a system function but that doesn't mean that you know the
 > state of the stack.
 >
 > If a function doesn't set up a frame pointer and there is no debugging
 > information available, how would one derive a stack frame?  I could
 > imagine a really complicated "search the stack" technique but I can't
 > see how it would ever be foolproof.
Well... GDB 5.3 was able to dump a correct stack in this case... So it was
possible to do in the past. Marks patch for gdb 6.x was not really 
really bad.
It dumped the first stack frame now "somewhat" correctly, but failed to 
advance
to the next stack frame.

Maybe we could implement something similar to the gdb 5.3 code but with all
the benefits from the 6.x changes?

Roland

Re: gdb stack trace problems (Addendum)

[Roland Schwingel]

Hi...

Mark Kettenis wrote on 08.05.2005 15:30:20:

 >    Date: Mon, 02 May 2005 09:03:08 +0200
 >    From: Roland Schwingel <roland.schwingel@onevision.de>
 >
 >    Hi Mark...
 >
 >    Have you already had some time to look into my results with your 
patch to
 >    the i386 stack unwinder? At basically it could work but obviously it
 >    is not advancing to the next stack frame... Attached you will find my
 >    results
 >
 > This isn't very encouraging.  My approach obviously isn't working very
 > well.  My suggestion is to go for a Windows-specific solution, where
 > one would use a special unwinder for the sort of undebuggable code
 > that's found in the Windows system DLLs.  But I'm afraid I can't
 > really do much since I don't have a Windows system.  Chris, is there
 > any change you can hack something like this into i386-cygwin-tdep.c?
Uff.... Sad to hear...

Roland

Re: gdb stack trace problems (Addendum)

[Roland Schwingel]

Hi Mark...

Have you already had some time to look into my results with your patch to
the i386 stack unwinder? At basically it could work but obviously it
is not advancing to the next stack frame... Attached you will find my 
results
I posted to the list last week.

Thanks for your help,

Roland

----- Forwarded by Roland Schwingel/Onevision on 02.05.2005 08:59 -----

gdb-owner@sources.redhat.com wrote on 26.04.2005 11:16:03:

 > Hi...
 >
 > Mark Kettenis <mark.kettenis@xs4all.nl> wrote on 25.04.2005 21:26:32:
 >  > Can you test the attached patch?  It introduces a new option named
 >  > "trust-frame-pointer".  Whenever you encounter a problem you can:
 >  >
 >  > (gdb) set trust-frame-pointer 1
 >  >
 >  > and try again.  You probably want to reset it to 0 before continuing
 >  > your program since I found out that bad things happen with some of the
 >  > tests in the gdb testsuite with this turned on.
 >
 > I couldn't resist and checkin out and compiling went without problems so
 > I tried it out immediately... I did run my gdbtest app again (the one I
 > posted
 > to the list on 20050418). Here are the results. As you will see 
immediately
 > it looks now differently. There is no longer a SleepEx() in the stack.
 > Instead there is no a Sleep() but unfortunately the frame gets duplicated
 > and afterwards gdb complains about a maybe corrupt stack. See attached
 > output.
 >
 > Roland
 >
 > gdb.exe ./gdbtest.exe
 > GNU gdb 6.3.50.20050426-cvs
 > Copyright 2004 Free Software Foundation, Inc.
 > GDB is free software, covered by the GNU General Public License, and 
you are
 > welcome to change it and/or distribute copies of it under certain
 > conditions.
 > Type "show copying" to see the conditions.
 > There is absolutely no warranty for GDB.  Type "show warranty" for 
details.
 > This GDB was configured as "i686-pc-cygwin"...
 > (gdb) r
 > Starting program: /tmp/gdbtest.exe
 > test gdb stack tracing during a crash
 > Created thread 1 with ID: 2472
 > I am thread 1 and alive
 > Created thread 2 with ID: 2292
 > I am thread 2 and alive
 > Thread 1: var = 22
 > Created thread 3 with ID: 2728
 > I am thread 3 and alive
 > Thread 2: var = 21
 > Created 3 Threads....
 > Thread 1: var = 20
 > Thread 3: var = 19
 > Thread 2: var = 18
 > Thread 1: var = 17
 > Thread 3: var = 16
 > Thread 2: var = 15
 > Thread 1: var = 14
 > Thread 3: var = 13
 > Thread 1: var = 14
 > Thread 3: var = 13
 > Thread 2: var = 12
 > Thread 1: var = 11
 > Thread 3: var = 10
 > Thread 2: var = 9
 > Thread 1: var = 8
 > Thread 3: var = 7
 > Thread 2: var = 6
 > Thread 1: var = 5
 > Thread 3: var = 4
 > Thread 2: var = 3
 > Thread 1: var = 2
 > Thread 3: var = 1
 > I am thread 2 and I will crash now!
 >
 > Program received signal SIGSEGV, Segmentation fault.
 > [Switching to thread 2832.0x8f4]
 > 0x0040131d in crashIfZero (num=2) at gdbtest.c:17
 > 17                      *data=911;
 > (gdb) thr 1
 > [Switching to thread 1 (thread 2832.0x97c)]#0  0x7c91eb94 in
 > ntdll!LdrAccessResource () from ntdll.dll
 > (gdb) bt
 > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
 > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
 > #2  0x7c8023ed in SleepEx () from /vol/c/WINDOWS/system32/kernel32.dll
 > #3  0x00000000 in ?? () from
 > (gdb) set trust-frame-pointer 1
 > (gdb) bt
 > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
 > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
 > #2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
 > #3  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
 > Previous frame identical to this frame (corrupt stack?)
 > (gdb) thr 2
 > [Switching to thread 2 (thread 2832.0x9a8)]#0  0x7c91eb94 in
 > ntdll!LdrAccessResource () from ntdll.dll
 > (gdb) bt
 > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
 > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
 > #2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
 > #3  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
 > Previous frame identical to this frame (corrupt stack?)
 > (gdb) thr 3
 > [Switching to thread 3 (thread 2832.0x8f4)]#0  0x0040131d in crashIfZero
 > (num=2) at gdbtest.c:17
 > 17                      *data=911;
 > (gdb) bt
 > #0  0x0040131d in crashIfZero (num=2) at gdbtest.c:17
 > #1  0x00401363 in func4 (num=2) at gdbtest.c:26
 > #2  0x0040139b in func3 (num=2) at gdbtest.c:34
 > #3  0x004013c8 in func2 (num=2) at gdbtest.c:41
 > #4  0x004013f5 in func1 (num=2) at gdbtest.c:48
 > #5  0x0040136e in func4 (num=2) at gdbtest.c:27
 > #6  0x0040139b in func3 (num=2) at gdbtest.c:34
 > #7  0x004013c8 in func2 (num=2) at gdbtest.c:41
 > #8  0x004013f5 in func1 (num=2) at gdbtest.c:48
 > #9  0x00401436 in threadFunc (param=0x22ff58) at gdbtest.c:55
 > #10 0x7c80b50b in KERNEL32!GetModuleFileNameA () from
 > /vol/c/WINDOWS/system32/kernel32.dll
 > #11 0x00000000 in ?? () from
 > (gdb) thr 4
 > [Switching to thread 4 (thread 2832.0xaa8)]#0  0x7c91eb94 in
 > ntdll!LdrAccessResource () from ntdll.dll
 > (gdb) bt
 > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
 > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
 > #2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
 > #3  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
 > Previous frame identical to this frame (corrupt stack?)
 > (gdb)
 >

Re: gdb stack trace problems (Addendum)

[Mark Kettenis]

   Date: Mon, 02 May 2005 09:03:08 +0200
   From: Roland Schwingel <roland.schwingel@onevision.de>

   Hi Mark...

   Have you already had some time to look into my results with your patch to
   the i386 stack unwinder? At basically it could work but obviously it
   is not advancing to the next stack frame... Attached you will find my 
   results

This isn't very encouraging.  My approach obviously isn't working very
well.  My suggestion is to go for a Windows-specific solution, where
one would use a special unwinder for the sort of undebuggable code
that's found in the Windows system DLLs.  But I'm afraid I can't
really do much since I don't have a Windows system.  Chris, is there
any change you can hack something like this into i386-cygwin-tdep.c?

Mark

   ----- Forwarded by Roland Schwingel/Onevision on 02.05.2005 08:59 -----

   gdb-owner@sources.redhat.com wrote on 26.04.2005 11:16:03:

    > Hi...
    >
    > Mark Kettenis <mark.kettenis@xs4all.nl> wrote on 25.04.2005 21:26:32:
    >  > Can you test the attached patch?  It introduces a new option named
    >  > "trust-frame-pointer".  Whenever you encounter a problem you can:
    >  >
    >  > (gdb) set trust-frame-pointer 1
    >  >
    >  > and try again.  You probably want to reset it to 0 before continuing
    >  > your program since I found out that bad things happen with some of the
    >  > tests in the gdb testsuite with this turned on.
    >
    > I couldn't resist and checkin out and compiling went without problems so
    > I tried it out immediately... I did run my gdbtest app again (the one I
    > posted
    > to the list on 20050418). Here are the results. As you will see 
   immediately
    > it looks now differently. There is no longer a SleepEx() in the stack.
    > Instead there is no a Sleep() but unfortunately the frame gets duplicated
    > and afterwards gdb complains about a maybe corrupt stack. See attached
    > output.
    >
    > Roland
    >
    > gdb.exe ./gdbtest.exe
    > GNU gdb 6.3.50.20050426-cvs
    > Copyright 2004 Free Software Foundation, Inc.
    > GDB is free software, covered by the GNU General Public License, and 
   you are
    > welcome to change it and/or distribute copies of it under certain
    > conditions.
    > Type "show copying" to see the conditions.
    > There is absolutely no warranty for GDB.  Type "show warranty" for 
   details.
    > This GDB was configured as "i686-pc-cygwin"...
    > (gdb) r
    > Starting program: /tmp/gdbtest.exe
    > test gdb stack tracing during a crash
    > Created thread 1 with ID: 2472
    > I am thread 1 and alive
    > Created thread 2 with ID: 2292
    > I am thread 2 and alive
    > Thread 1: var = 22
    > Created thread 3 with ID: 2728
    > I am thread 3 and alive
    > Thread 2: var = 21
    > Created 3 Threads....
    > Thread 1: var = 20
    > Thread 3: var = 19
    > Thread 2: var = 18
    > Thread 1: var = 17
    > Thread 3: var = 16
    > Thread 2: var = 15
    > Thread 1: var = 14
    > Thread 3: var = 13
    > Thread 1: var = 14
    > Thread 3: var = 13
    > Thread 2: var = 12
    > Thread 1: var = 11
    > Thread 3: var = 10
    > Thread 2: var = 9
    > Thread 1: var = 8
    > Thread 3: var = 7
    > Thread 2: var = 6
    > Thread 1: var = 5
    > Thread 3: var = 4
    > Thread 2: var = 3
    > Thread 1: var = 2
    > Thread 3: var = 1
    > I am thread 2 and I will crash now!
    >
    > Program received signal SIGSEGV, Segmentation fault.
    > [Switching to thread 2832.0x8f4]
    > 0x0040131d in crashIfZero (num=2) at gdbtest.c:17
    > 17                      *data=911;
    > (gdb) thr 1
    > [Switching to thread 1 (thread 2832.0x97c)]#0  0x7c91eb94 in
    > ntdll!LdrAccessResource () from ntdll.dll
    > (gdb) bt
    > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
    > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
    > #2  0x7c8023ed in SleepEx () from /vol/c/WINDOWS/system32/kernel32.dll
    > #3  0x00000000 in ?? () from
    > (gdb) set trust-frame-pointer 1
    > (gdb) bt
    > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
    > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
    > #2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
    > #3  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
    > Previous frame identical to this frame (corrupt stack?)
    > (gdb) thr 2
    > [Switching to thread 2 (thread 2832.0x9a8)]#0  0x7c91eb94 in
    > ntdll!LdrAccessResource () from ntdll.dll
    > (gdb) bt
    > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
    > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
    > #2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
    > #3  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
    > Previous frame identical to this frame (corrupt stack?)
    > (gdb) thr 3
    > [Switching to thread 3 (thread 2832.0x8f4)]#0  0x0040131d in crashIfZero
    > (num=2) at gdbtest.c:17
    > 17                      *data=911;
    > (gdb) bt
    > #0  0x0040131d in crashIfZero (num=2) at gdbtest.c:17
    > #1  0x00401363 in func4 (num=2) at gdbtest.c:26
    > #2  0x0040139b in func3 (num=2) at gdbtest.c:34
    > #3  0x004013c8 in func2 (num=2) at gdbtest.c:41
    > #4  0x004013f5 in func1 (num=2) at gdbtest.c:48
    > #5  0x0040136e in func4 (num=2) at gdbtest.c:27
    > #6  0x0040139b in func3 (num=2) at gdbtest.c:34
    > #7  0x004013c8 in func2 (num=2) at gdbtest.c:41
    > #8  0x004013f5 in func1 (num=2) at gdbtest.c:48
    > #9  0x00401436 in threadFunc (param=0x22ff58) at gdbtest.c:55
    > #10 0x7c80b50b in KERNEL32!GetModuleFileNameA () from
    > /vol/c/WINDOWS/system32/kernel32.dll
    > #11 0x00000000 in ?? () from
    > (gdb) thr 4
    > [Switching to thread 4 (thread 2832.0xaa8)]#0  0x7c91eb94 in
    > ntdll!LdrAccessResource () from ntdll.dll
    > (gdb) bt
    > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
    > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
    > #2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
    > #3  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
    > Previous frame identical to this frame (corrupt stack?)
    > (gdb)
    >

Re: gdb stack trace problems (Addendum)

[Christopher Faylor]

On Sun, May 08, 2005 at 03:30:20PM +0200, Mark Kettenis wrote:
>   Date: Mon, 02 May 2005 09:03:08 +0200
>   From: Roland Schwingel <roland.schwingel@onevision.de>
>
>   Hi Mark...
>
>   Have you already had some time to look into my results with your patch to
>   the i386 stack unwinder? At basically it could work but obviously it
>   is not advancing to the next stack frame... Attached you will find my 
>   results
>
>This isn't very encouraging.  My approach obviously isn't working very
>well.  My suggestion is to go for a Windows-specific solution, where
>one would use a special unwinder for the sort of undebuggable code
>that's found in the Windows system DLLs.  But I'm afraid I can't
>really do much since I don't have a Windows system.  Chris, is there
>any change you can hack something like this into i386-cygwin-tdep.c?

What kind of windows-specific solution do you have in mind?  How would
you know what to unwind?  You could potentially figure out that you're
stuck in a system function but that doesn't mean that you know the
state of the stack.

If a function doesn't set up a frame pointer and there is no debugging
information available, how would one derive a stack frame?  I could
imagine a really complicated "search the stack" technique but I can't
see how it would ever be foolproof.

cgf

Re: gdb stack trace problems (Addendum)

[Eli Zaretskii]

> Date: Sun, 8 May 2005 19:19:53 -0400
> From: Christopher Faylor <me@cgf.cx>
> 
> What kind of windows-specific solution do you have in mind?  How would
> you know what to unwind?  You could potentially figure out that you're
> stuck in a system function but that doesn't mean that you know the
> state of the stack.
> 
> If a function doesn't set up a frame pointer and there is no debugging
> information available, how would one derive a stack frame?  I could
> imagine a really complicated "search the stack" technique but I can't
> see how it would ever be foolproof.

Does the MSVC debugger manage to display a sensible backtrace in this
case?  If it does not, we don't need to worry, I think.

If it does, then perhaps teh frameless functions in system DLLs follow
some pattern in their prologue that could help us?  Or perhaps they
save stack-related info in some Windows-specific place?

Just my $0.02.

Re: gdb stack trace problems (Addendum)

[Christopher Faylor]

On Mon, May 09, 2005 at 07:45:05AM +0300, Eli Zaretskii wrote:
>> Date: Sun, 8 May 2005 19:19:53 -0400
>> From: Christopher Faylor <me@cgf.cx>
>> 
>> What kind of windows-specific solution do you have in mind?  How would
>> you know what to unwind?  You could potentially figure out that you're
>> stuck in a system function but that doesn't mean that you know the
>> state of the stack.
>> 
>> If a function doesn't set up a frame pointer and there is no debugging
>> information available, how would one derive a stack frame?  I could
>> imagine a really complicated "search the stack" technique but I can't
>> see how it would ever be foolproof.
>
>Does the MSVC debugger manage to display a sensible backtrace in this
>case?  If it does not, we don't need to worry, I think.

Possibly.  They may have some separate debugging symbol files available
for system DLLs.  I'm not extremely interested in making gdb understand
native MSVC debugging format though.

>If it does, then perhaps teh frameless functions in system DLLs follow
>some pattern in their prologue that could help us?  Or perhaps they
>save stack-related info in some Windows-specific place?

I've never noticed anything special on the stack or in the assembly
language.  I'm not 100% sure that the system functions are written
in C.

cgf

Re: gdb stack trace problems (Addendum)

[Stan Shebs]

Christopher Faylor wrote:

>
>If a function doesn't set up a frame pointer and there is no debugging
>information available, how would one derive a stack frame?  I could
>imagine a really complicated "search the stack" technique but I can't
>see how it would ever be foolproof.
>
That's where GDB hacking just starts to get interesting. :-) Yes,
it's complicated and potentially expensive too, because you may have
to transfer a lot of bytes. The mips has something similar for
function starts, and a "heuristic-fence-post" variable to control
how hard to work at it.

Stan

Re: gdb stack trace problems (Addendum)

[Roland Schwingel]

Hi...

Mark Kettenis <mark.kettenis@xs4all.nl> wrote on 25.04.2005 21:26:32:
 > Can you test the attached patch?  It introduces a new option named
 > "trust-frame-pointer".  Whenever you encounter a problem you can:
 >
 > (gdb) set trust-frame-pointer 1
 >
 > and try again.  You probably want to reset it to 0 before continuing
 > your program since I found out that bad things happen with some of the
 > tests in the gdb testsuite with this turned on.

I couldn't resist and checkin out and compiling went without problems so
I tried it out immediately... I did run my gdbtest app again (the one I 
posted
to the list on 20050418). Here are the results. As you will see immediately
it looks now differently. There is no longer a SleepEx() in the stack.
Instead there is no a Sleep() but unfortunately the frame gets duplicated
and afterwards gdb complains about a maybe corrupt stack. See attached
output.

Roland

gdb.exe ./gdbtest.exe
GNU gdb 6.3.50.20050426-cvs
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-cygwin"...
(gdb) r
Starting program: /tmp/gdbtest.exe
test gdb stack tracing during a crash
Created thread 1 with ID: 2472
I am thread 1 and alive
Created thread 2 with ID: 2292
I am thread 2 and alive
Thread 1: var = 22
Created thread 3 with ID: 2728
I am thread 3 and alive
Thread 2: var = 21
Created 3 Threads....
Thread 1: var = 20
Thread 3: var = 19
Thread 2: var = 18
Thread 1: var = 17
Thread 3: var = 16
Thread 2: var = 15
Thread 1: var = 14
Thread 3: var = 13
Thread 1: var = 14
Thread 3: var = 13
Thread 2: var = 12
Thread 1: var = 11
Thread 3: var = 10
Thread 2: var = 9
Thread 1: var = 8
Thread 3: var = 7
Thread 2: var = 6
Thread 1: var = 5
Thread 3: var = 4
Thread 2: var = 3
Thread 1: var = 2
Thread 3: var = 1
I am thread 2 and I will crash now!

Program received signal SIGSEGV, Segmentation fault.
[Switching to thread 2832.0x8f4]
0x0040131d in crashIfZero (num=2) at gdbtest.c:17
17                      *data=911;
(gdb) thr 1
[Switching to thread 1 (thread 2832.0x97c)]#0  0x7c91eb94 in 
ntdll!LdrAccessResource () from ntdll.dll
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c8023ed in SleepEx () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x00000000 in ?? () from
(gdb) set trust-frame-pointer 1
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
Previous frame identical to this frame (corrupt stack?)
(gdb) thr 2
[Switching to thread 2 (thread 2832.0x9a8)]#0  0x7c91eb94 in 
ntdll!LdrAccessResource () from ntdll.dll
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
Previous frame identical to this frame (corrupt stack?)
(gdb) thr 3
[Switching to thread 3 (thread 2832.0x8f4)]#0  0x0040131d in crashIfZero 
(num=2) at gdbtest.c:17
17                      *data=911;
(gdb) bt
#0  0x0040131d in crashIfZero (num=2) at gdbtest.c:17
#1  0x00401363 in func4 (num=2) at gdbtest.c:26
#2  0x0040139b in func3 (num=2) at gdbtest.c:34
#3  0x004013c8 in func2 (num=2) at gdbtest.c:41
#4  0x004013f5 in func1 (num=2) at gdbtest.c:48
#5  0x0040136e in func4 (num=2) at gdbtest.c:27
#6  0x0040139b in func3 (num=2) at gdbtest.c:34
#7  0x004013c8 in func2 (num=2) at gdbtest.c:41
#8  0x004013f5 in func1 (num=2) at gdbtest.c:48
#9  0x00401436 in threadFunc (param=0x22ff58) at gdbtest.c:55
#10 0x7c80b50b in KERNEL32!GetModuleFileNameA () from 
/vol/c/WINDOWS/system32/kernel32.dll
#11 0x00000000 in ?? () from
(gdb) thr 4
[Switching to thread 4 (thread 2832.0xaa8)]#0  0x7c91eb94 in 
ntdll!LdrAccessResource () from ntdll.dll
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
Previous frame identical to this frame (corrupt stack?)
(gdb)

Re: gdb stack trace problems (Addendum)

[Roland Schwingel]

Hi....

 >    Well the present situation is quite problematic for us. In past days
 >    (gdb 5.3)
 >    when an application crashed we had a quite accurate backtrace. It 
wasn't
 >    wrong (for us). Finding/Fixing bugs was very easy. GDB has been
 >    very useful here.
 >
 > You probably didn't notice it was wrong...
Well we used it quite a lot and its results were reliable...

 > Can you test the attached patch?  It introduces a new option named
 > "trust-frame-pointer".  Whenever you encounter a problem you can:
 >
 > (gdb) set trust-frame-pointer 1
 >
 > and try again.  You probably want to reset it to 0 before continuing
 > your program since I found out that bad things happen with some of the
 > tests in the gdb testsuite with this turned on.
Thank you very much. I will do so. I just started a cvs up of my gdb sources
to get a completly current version of gdb and then will apply your patch and
test. But I might not be able to deliver results until tomorrow.

Roland

Re: gdb stack trace problems (Addendum)

[Roland Schwingel]

Hi...

"Mark Kettenis" <mark.kettenis@xs4all.nl> wrote on 19.04.2005 09:45:25:
 > Hi Roland,
 >
 > [Dropped Dan, he's busy enough with other stuff, and he can always 
read it
 > on the list]
 >
 > > Also I want to mention that it might have something to do with 
threading.
 > > In a nonthreaded app I don't have any problems with gdb 6.3.50 stack
 > > dumping.
 >
 > Only if it links in different code, or is multi-threading still 
cooperative
 > on Windblows.  Anyway, what I really need is the disassembly for 
SleepEx as
 > implemented in the DLL.

Here it comes... Including its call to Sleep() in the DLL (both localted 
in kernel32).
BTW. I run WinXP SP2.

(gdb) info dll
DLL Name                              Load Address
ntdll.dll                             7c911000
/vol/c/WINDOWS/system32/kernel32.dll  7c801000
/vol/c/WINDOWS/system32/msvcrt.dll    77be1000
gdb) disas SleepEx
Dump of assembler code for function SleepEx:
0x7c80239c <SleepEx+0>: push   $0x2c
0x7c80239e <SleepEx+2>: push   $0x7c802458
0x7c8023a3 <SleepEx+7>: call   0x7c8024cb <ReleaseMutex+36>
0x7c8023a8 <SleepEx+12>:        movl   $0x14,0xffffffc4(%ebp)
0x7c8023af <SleepEx+19>:        movl   $0x1,0xffffffc8(%ebp)
0x7c8023b6 <SleepEx+26>:        xor    %eax,%eax
0x7c8023b8 <SleepEx+28>:        lea    0xffffffcc(%ebp),%edi
0x7c8023bb <SleepEx+31>:        stos   %eax,%es:(%edi)
0x7c8023bc <SleepEx+32>:        stos   %eax,%es:(%edi)
0x7c8023bd <SleepEx+33>:        stos   %eax,%es:(%edi)
0x7c8023be <SleepEx+34>:        xor    %esi,%esi
0x7c8023c0 <SleepEx+36>:        push   %esi
0x7c8023c1 <SleepEx+37>:        lea    0xffffffc4(%ebp),%eax
0x7c8023c4 <SleepEx+40>:        push   %eax
0x7c8023c5 <SleepEx+41>:        call   *0x7c80123c
0x7c8023cb <SleepEx+47>:        mov    %esi,0xfffffffc(%ebp)
0x7c8023ce <SleepEx+50>:        pushl  0x8(%ebp)
0x7c8023d1 <SleepEx+53>:        lea    0xffffffd8(%ebp),%eax
0x7c8023d4 <SleepEx+56>:        push   %eax
0x7c8023d5 <SleepEx+57>:        call   0x7c802470 <Sleep+46>
0x7c8023da <SleepEx+62>:        mov    %eax,0xffffffe4(%ebp)
0x7c8023dd <SleepEx+65>:        cmp    %esi,%eax
0x7c8023df <SleepEx+67>:        je     0x7c802412 <SleepEx+118>
0x7c8023e1 <SleepEx+69>:        pushl  0xffffffe4(%ebp)
0x7c8023e4 <SleepEx+72>:        pushl  0xc(%ebp)
0x7c8023e7 <SleepEx+75>:        call   *0x7c8014bc
0x7c8023ed <SleepEx+81>:        mov    %eax,0xffffffe0(%ebp)
0x7c8023f0 <SleepEx+84>:        cmp    %esi,0xc(%ebp)
0x7c8023f3 <SleepEx+87>:        jne    0x7c802424 <SleepEx+136>
0x7c8023f5 <SleepEx+89>:        orl    $0xffffffff,0xfffffffc(%ebp)
0x7c8023f9 <SleepEx+93>:        call   0x7c802432 <SleepEx+150>
0x7c8023fe <SleepEx+98>:        mov    $0xc0,%eax
0x7c802403 <SleepEx+103>:       cmp    %eax,0xffffffe0(%ebp)
0x7c802406 <SleepEx+106>:       je     0x7c80240a <SleepEx+110>
0x7c802408 <SleepEx+108>:       xor    %eax,%eax
0x7c80240a <SleepEx+110>:       call   0x7c80250b <ReleaseMutex+100>
0x7c80240f <SleepEx+115>:       ret    $0x8
0x7c802412 <SleepEx+118>:       mov    %esi,0xffffffd8(%ebp)
0x7c802415 <SleepEx+121>:       movl   $0x80000000,0xffffffdc(%ebp)
0x7c80241c <SleepEx+128>:       lea    0xffffffd8(%ebp),%eax
0x7c80241f <SleepEx+131>:       mov    %eax,0xffffffe4(%ebp)
0x7c802422 <SleepEx+134>:       jmp    0x7c8023e1 <SleepEx+69>
0x7c802424 <SleepEx+136>:       cmp    $0x101,%eax
0x7c802429 <SleepEx+141>:       je     0x7c8023e1 <SleepEx+69>
0x7c80242b <SleepEx+143>:       jmp    0x7c8023f5 <SleepEx+89>
0x7c80242d <SleepEx+145>:       nop   
0x7c80242e <SleepEx+146>:       nop   
0x7c80242f <SleepEx+147>:       nop   
0x7c802430 <SleepEx+148>:       nop   
0x7c802431 <SleepEx+149>:       nop   
0x7c802432 <SleepEx+150>:       lea    0xffffffc4(%ebp),%eax
0x7c802435 <SleepEx+153>:       push   %eax
0x7c802436 <SleepEx+154>:       call   *0x7c801238
0x7c80243c <SleepEx+160>:       ret   
0x7c80243d <SleepEx+161>:       nop   
0x7c80243e <SleepEx+162>:       nop   
0x7c80243f <SleepEx+163>:       nop   
0x7c802440 <SleepEx+164>:       nop   
0x7c802441 <SleepEx+165>:       nop   
End of assembler dump.
(gdb) disas Sleep
Dump of assembler code for function Sleep:
0x7c802442 <Sleep+0>:   mov    %edi,%edi
0x7c802444 <Sleep+2>:   push   %ebp
0x7c802445 <Sleep+3>:   mov    %esp,%ebp
0x7c802447 <Sleep+5>:   push   $0x0
0x7c802449 <Sleep+7>:   pushl  0x8(%ebp)
0x7c80244c <Sleep+10>:  call   0x7c80239c <SleepEx>
0x7c802451 <Sleep+15>:  pop    %ebp
0x7c802452 <Sleep+16>:  ret    $0x4
0x7c802455 <Sleep+19>:  nop   
0x7c802456 <Sleep+20>:  nop   
0x7c802457 <Sleep+21>:  nop   
0x7c802458 <Sleep+22>:  (bad) 
0x7c802459 <Sleep+23>:  (bad) 
0x7c80245a <Sleep+24>:  (bad) 
0x7c80245b <Sleep+25>:  incl   (%eax)
0x7c80245d <Sleep+27>:  add    %al,(%eax)
0x7c80245f <Sleep+29>:  add    %dh,(%edx)
0x7c802461 <Sleep+31>:  and    $0x80,%al
0x7c802463 <Sleep+33>:  jl     0x7c8023f5 <SleepEx+89>
0x7c802465 <Sleep+35>:  nop   
0x7c802466 <Sleep+36>:  nop   
0x7c802467 <Sleep+37>:  nop   
0x7c802468 <Sleep+38>:  nop   
0x7c802469 <Sleep+39>:  nop   
0x7c80246a <Sleep+40>:  nop   
0x7c80246b <Sleep+41>:  nop   
0x7c80246c <Sleep+42>:  nop   
0x7c80246d <Sleep+43>:  nop   
0x7c80246e <Sleep+44>:  nop   
0x7c80246f <Sleep+45>:  nop   
0x7c802470 <Sleep+46>:  mov    %edi,%edi
0x7c802472 <Sleep+48>:  push   %ebp
0x7c802473 <Sleep+49>:  mov    %esp,%ebp
0x7c802475 <Sleep+51>:  mov    0xc(%ebp),%eax
0x7c802478 <Sleep+54>:  cmp    $0xffffffff,%eax
0x7c80247b <Sleep+57>:  je     0x7c8092ee <KERNEL32!GetTickCount+66>
0x7c802481 <Sleep+63>:  mov    $0x2710,%ecx
0x7c802486 <Sleep+68>:  mul    %ecx
0x7c802488 <Sleep+70>:  mov    0x8(%ebp),%ecx
0x7c80248b <Sleep+73>:  mov    %eax,(%ecx)
0x7c80248d <Sleep+75>:  neg    %eax
0x7c80248f <Sleep+77>:  mov    %edx,0x4(%ecx)
0x7c802492 <Sleep+80>:  adc    $0x0,%edx
0x7c802495 <Sleep+83>:  neg    %edx
0x7c802497 <Sleep+85>:  mov    %eax,(%ecx)
0x7c802499 <Sleep+87>:  mov    %edx,0x4(%ecx)
0x7c80249c <Sleep+90>:  mov    %ecx,%eax
0x7c80249e <Sleep+92>:  pop    %ebp
0x7c80249f <Sleep+93>:  ret    $0x8
0x7c8024a2 <Sleep+96>:  nop   
0x7c8024a3 <Sleep+97>:  nop   
0x7c8024a4 <Sleep+98>:  nop   
0x7c8024a5 <Sleep+99>:  nop   
0x7c8024a6 <Sleep+100>: nop   
End of assembler dump.

Good Luck,

Roland

Re: gdb stack trace problems (Addendum)

[Mark Kettenis]

   Date: Fri, 22 Apr 2005 10:43:46 +0200
   From: Roland Schwingel <roland.schwingel@onevision.de>

   Hi Mark...

   Have you already had some time to look into it?

I did, but I didn't have find the time to reply yet ;-).

Anyway, there is a serious problem here.  The code you show is
basically undebuggable.  SleepEx is basically what we call a frameless
function; it doesn't set up its own stack frame.  This means the
debugger has to track all changes of the stack frame in this function
in order to be able to unwind the stack and provide a useful
backtrace.  However, since this Windows uses the convention that the
caller pushes function arguments on the stack, and the callee is
repsonsible for restoring the stack. this gets basically impossible.
I'm wondering whether Microsoft provides debug versions of its DLLs
that don't have this problem.

The only thing we can do here is trust the frame pointer, like we used
to do in the old days.  The problem with that is that it's very likely
to (silently) skip some function calls.  In your particular example it
probably would appear as if your code called SleepEx directly and
there would be no trace of Sleep at all.  That can be quite puzzling.

I'm thinking about adding an option to instruct gdb to "trust" the
frame pointer.  I'm not going to make it the default though.  I really
prefer seeing an abviously wrong backtrace over gdb silently skipping
function calls in its backtraces.

Mark

Re: gdb stack trace problems (Addendum)

[Roland Schwingel]

Hi Mark, Daniel and list....

Sorry for replying my own post, but I forgot something...
 
 > Mark Kettenis wrote on 18.04.2005 19:10:57:
 >  >    Date: Mon, 18 Apr 2005 09:57:21 -0400
 >  >    From: Daniel Jacobowitz <drow@false.org>
 >  >
 >  >    > Program received signal SIGSEGV, Segmentation fault.
 >  >    > [Switching to thread 2316.0xf8c]
 >  >    > 0x0040131d in crashIfZero (num=2) at gdbtest.c:17
 >  >    > 17                      *data=911;
 >  >    > (gdb) thr 1
 >  >    > [Switching to thread 1 (thread 2316.0xf10)]#0  0x7c91eb94 in
 >  >    > ntdll!LdrAccessResource () from ntdll.dll
 >  >    > (gdb) bt
 >  >    > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
 >  >    > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
 >  >    > #2  0x7c8023ed in SleepEx () from
 > /vol/c/WINDOWS/system32/kernel32.dll
 >  >    > #3  0x00000000 in ?? () from
 >  >
 >  >    OK, here's a concrete problem.  GDB 6.x has a (usually) more 
reliable
 >  >    prologue analyzer for x86, compared to GDB 5.3.  However, the 
old one
 >  >    got lucky with SleepEx and the new one doesn't.  I'd ask you to 
post
 >  >    disassembly of the function except I'm not sure how wise that is
 > with a
 >  >    Microsoft DLL...
 >  >
 >  >    Mark, you did the work on the i386 prologue analyzer.  Any 
chance you
 >  >    could take a look at this?
 >  >
 >  > Sure if someone can provide me with the dissassembly in AT&T syntax
 >  > for that function, or even better some assembler code that exhibits
 >  > the problem.  No promises though that I'll fix this.  Last time I
 >  > looked at some windows disassembler it looked like it was hand
 >  > optimized in such a way that the entire function needed to be analyzed
 >  > to do something useful; something that doesn't quite fit in the way we
 >  > do things in GDB.
 > Here is the disassembly of one function (func1) that cannot be shown
 > correctly
 > from within gdb 6.3.50 in att flavor:
 > (gdb) disas func1
 > Dump of assembler code for function func1:
 > 0x004013ca <func1+0>:   push   %ebp
 > 0x004013cb <func1+1>:   mov    %esp,%ebp
 > 0x004013cd <func1+3>:   sub    $0x8,%esp
 > 0x004013d0 <func1+6>:   movl   $0x64,(%esp)
 > 0x004013d7 <func1+13>:  call   0x401830 <Sleep@4>
 > 0x004013dc <func1+18>:  sub    $0x4,%esp
 > 0x004013df <func1+21>:  mov    0x8(%ebp),%eax
 > 0x004013e2 <func1+24>:  mov    %eax,(%esp)
 > 0x004013e5 <func1+27>:  call   0x4012eb <crashIfZero>
 > 0x004013ea <func1+32>:  mov    0x8(%ebp),%eax
 > 0x004013ed <func1+35>:  mov    %eax,(%esp)
 > 0x004013f0 <func1+38>:  call   0x40139d <func2>
 > 0x004013f5 <func1+43>:  leave
 > 0x004013f6 <func1+44>:  ret  
 > End of assembler dump.
 >
 > I hope this is what you requested. I will aid in tracking this down,
 > wherever I can. As you see from the assembly Sleep() is
 > called not SleepEx() as shown in the gdb 6.3.50 stack
 > trace. In gdb 5.3 stack trace Sleep() is correctly shown if you
 > look at my posting from yesterday

As you might see Sleep() is not the direct call to msvcrt. Here
is the disassembly of the Sleep() linked to the exe.

(gdb) disas 0x401830
Dump of assembler code for function Sleep@4:
0x00401830 <Sleep@4+0>: jmp    *0x4040e4
0x00401836 <Sleep@4+6>: nop   
0x00401837 <Sleep@4+7>: nop   
0x00401838 <Sleep@4+8>: add    %al,(%eax)
0x0040183a <Sleep@4+10>:        add    %al,(%eax)
0x0040183c <Sleep@4+12>:        add    %al,(%eax)
0x0040183e <Sleep@4+14>:        add    %al,(%eax)
End of assembler dump.

(gdb) disas 0x4040e4
Dump of assembler code for function _imp__Sleep@4:
0x004040e4 <_imp__Sleep@4+0>:   in     $0x41,%al
0x004040e6 <_imp__Sleep@4+2>:   add    %al,(%eax)
0x004040e8 <_imp__Sleep@4+4>:   add    %al,(%eax)
0x004040ea <_imp__Sleep@4+6>:   add    %al,(%eax)
0x004040ec <_imp__Sleep@4+8>:   daa   
0x004040ed <_imp__Sleep@4+9>:   add    %bl,0x5f(%edi)
0x004040f0 <_imp__Sleep@4+12>:  addr16
0x004040f1 <_imp__Sleep@4+13>:  gs
0x004040f2 <_imp__Sleep@4+14>:  je     0x404161 <_imp__Sleep@4+125>
0x004040f4 <_imp__Sleep@4+16>:  popa  
0x004040f5 <_imp__Sleep@4+17>:  imul   $0x736772,0x61(%esi),%ebp
0x004040fc <_imp__Sleep@4+24>:  cmp    $0x0,%al
0x004040fe <_imp__Sleep@4+26>:  pop    %edi
0x004040ff <_imp__Sleep@4+27>:  pop    %edi
0x00404100 <_imp__Sleep@4+28>:  jo     0x404161 <_imp__Sleep@4+125>
0x00404102 <_imp__Sleep@4+30>:  pop    %edi
0x00404103 <_imp__Sleep@4+31>:  outsb  %gs:(%esi),(%dx)
0x00404105 <_imp__Sleep@4+33>:  jbe    0x404170 <_imp__Sleep@4+140>
0x00404107 <_imp__Sleep@4+35>:  jb     0x404178 <_imp__Sleep@4+148>
0x00404109 <_imp__Sleep@4+37>:  outsb  %ds:(%esi),(%dx)
0x0040410a <_imp__Sleep@4+38>:  add    %al,(%eax)
0x0040410c <_imp__Sleep@4+40>:  add    %bl,%ds:0x5f(%edi)
0x00404110 <_imp__Sleep@4+44>:  jo     0x404171 <_imp__Sleep@4+141>
0x00404112 <_imp__Sleep@4+46>:  pop    %edi
0x00404113 <_imp__Sleep@4+47>:  insw   (%dx),%es:(%edi)
0x00404115 <_imp__Sleep@4+49>:  outsl  %ds:(%esi),(%dx)
0x00404116 <_imp__Sleep@4+50>:  add    %al,%fs:%gs:(%eax)
0x0040411a <_imp__Sleep@4+54>:  add    %al,(%eax)
0x0040411c <_imp__Sleep@4+56>:  push   %eax
0x0040411d <_imp__Sleep@4+57>:  add    %bl,0x5f(%edi)
0x00404120 <_imp__Sleep@4+60>:  jae    0x404187 <_imp__Sleep@4+163>
0x00404122 <_imp__Sleep@4+62>:  je     0x404183 <_imp__Sleep@4+159>
0x00404124 <_imp__Sleep@4+64>:  popa  
0x00404125 <_imp__Sleep@4+65>:  jo     0x404197 <_imp__Sleep@4+179>
0x00404127 <_imp__Sleep@4+67>:  pop    %edi
0x00404128 <_imp__Sleep@4+68>:  je     0x4041a3 <_imp__Sleep@4+191>
0x0040412a <_imp__Sleep@4+70>:  jo     0x404191 <_imp__Sleep@4+173>
0x0040412c <_imp__Sleep@4+72>:  add    %al,(%eax)
0x0040412e <_imp__Sleep@4+74>:  add    %al,(%eax)
0x00404130 <_imp__Sleep@4+76>:  jns    0x404132 <_imp__Sleep@4+78>
0x00404132 <_imp__Sleep@4+78>:  pop    %edi
0x00404133 <_imp__Sleep@4+79>:  arpl   %sp,0x78(%ebp)
0x00404136 <_imp__Sleep@4+82>:  imul   $0xb90000,0x0(%eax,%eax,1),%esi
0x0040413e <_imp__Sleep@4+90>:  pop    %edi
0x0040413f <_imp__Sleep@4+91>:  insw   (%dx),%es:(%edi)
0x00404141 <_imp__Sleep@4+93>:  outsl  %ds:(%esi),(%dx)
0x00404142 <_imp__Sleep@4+94>:  add    %al,%fs:%gs:(%eax)
0x00404146 <_imp__Sleep@4+98>:  add    %al,(%eax)
0x00404148 <_imp__Sleep@4+100>: mov    $0x70665f00,%esp
0x0040414d <_imp__Sleep@4+105>: jb     0x4041b4 <_imp__Sleep@4+208>
0x0040414f <_imp__Sleep@4+107>: jae    0x4041b6 <_imp__Sleep@4+210>
0x00404151 <_imp__Sleep@4+109>: je     0x404153 <_imp__Sleep@4+111>
0x00404153 <_imp__Sleep@4+111>: add    %ch,%cl
0x00404155 <_imp__Sleep@4+113>: add    %bl,0x69(%edi)
0x00404158 <_imp__Sleep@4+116>: outsl  %ds:(%esi),(%dx)
0x00404159 <_imp__Sleep@4+117>: bound  %eax,(%eax)
0x0040415b <_imp__Sleep@4+119>: add    %bl,0x1(%esi)
0x0040415e <_imp__Sleep@4+122>: pop    %edi
0x0040415f <_imp__Sleep@4+123>: outsl  %ds:(%esi),(%dx)
0x00404160 <_imp__Sleep@4+124>: outsb  %ds:(%esi),(%dx)
0x00404161 <_imp__Sleep@4+125>: gs
0x00404162 <_imp__Sleep@4+126>: js     0x4041cd <_imp__Sleep@4+233>
0x00404164 <_imp__Sleep@4+128>: je     0x404166 <_imp__Sleep@4+130>
0x00404166 <_imp__Sleep@4+130>: add    %al,(%eax)
0x00404168 <_imp__Sleep@4+132>: test   %al,(%ecx)
0x0040416a <_imp__Sleep@4+134>: pop    %edi
0x0040416b <_imp__Sleep@4+135>: jae    0x4041d2 <_imp__Sleep@4+238>
0x0040416d <_imp__Sleep@4+137>: je     0x4041dc <_imp__Sleep@4+248>
0x0040416f <_imp__Sleep@4+139>: outsl  %ds:(%esi),(%dx)
0x00404170 <_imp__Sleep@4+140>: add    %al,%fs:%gs:(%eax)
0x00404174 <_imp__Sleep@4+144>: sbb    $0x2,%al
0x00404176 <_imp__Sleep@4+146>: popa  
0x00404177 <_imp__Sleep@4+147>: je     0x4041de <_imp__Sleep@4+250>
0x00404179 <_imp__Sleep@4+149>: js     0x4041e4 <_imp__Sleep@4+256>
0x0040417b <_imp__Sleep@4+151>: je     0x40417d <_imp__Sleep@4+153>
0x0040417d <_imp__Sleep@4+153>: add    %al,(%eax)
0x0040417f <_imp__Sleep@4+155>: add    %bh,0x2(%edi)
0x00404182 <_imp__Sleep@4+158>: jo     0x4041f6 <_imp__Sleep@4+274>
0x00404184 <_imp__Sleep@4+160>: imul   $0x66,0x74(%esi),%ebp
0x0040418b <_imp__Sleep@4+167>: add    %cl,0x74657302(%ebp)
0x00404191 <_imp__Sleep@4+173>: bound  %esi,0x66(%ebp)
0x00404194 <_imp__Sleep@4+176>: add    %al,(%eax)
0x00404196 <_imp__Sleep@4+178>: add    %al,(%eax)
0x00404198 <_imp__Sleep@4+180>: nop   
0x00404199 <_imp__Sleep@4+181>: add    0x69(%ebx),%dh
0x0040419c <_imp__Sleep@4+184>: addr16 outsb %ds:(%si),(%dx)
0x0040419e <_imp__Sleep@4+186>: popa  
0x0040419f <_imp__Sleep@4+187>: insb   (%dx),%es:(%edi)
0x004041a0 <_imp__Sleep@4+188>: add    %al,(%eax)
0x004041a2 <_imp__Sleep@4+190>: add    %al,(%eax)
0x004041a4 <_imp__Sleep@4+192>: pop    %edx
0x004041a5 <_imp__Sleep@4+193>: add    %al,0x72(%ebx)
0x004041a8 <_imp__Sleep@4+196>: gs
0x004041a9 <_imp__Sleep@4+197>: popa  
0x004041aa <_imp__Sleep@4+198>: je     0x404211 <_imp__Sleep@4+301>
0x004041ac <_imp__Sleep@4+200>: push   %esp
0x004041ad <_imp__Sleep@4+201>: push   $0x64616572
0x004041b2 <_imp__Sleep@4+206>: add    %al,(%eax)
0x004041b4 <_imp__Sleep@4+208>: fwait
0x004041b5 <_imp__Sleep@4+209>: add    %al,0x78(%ebp)
0x004041b8 <_imp__Sleep@4+212>: imul   $0x7365636f,0x72(%eax,%edx,2),%esi
0x004041c0 <_imp__Sleep@4+220>: jae    0x4041c2 <_imp__Sleep@4+222>
0x004041c2 <_imp__Sleep@4+222>: add    %al,(%eax)
0x004041c4 <_imp__Sleep@4+224>: fild   (%edx)
0x004041c6 <_imp__Sleep@4+226>: push   %ebx
0x004041c7 <_imp__Sleep@4+227>: gs
0x004041c8 <_imp__Sleep@4+228>: je     0x40421f <_imp__Sleep@4+315>
0x004041ca <_imp__Sleep@4+230>: outsb  %ds:(%esi),(%dx)
0x004041cb <_imp__Sleep@4+231>: push   $0x6c646e61
0x004041d0 <_imp__Sleep@4+236>: gs
0x004041d1 <_imp__Sleep@4+237>: fs
0x004041d2 <_imp__Sleep@4+238>: inc    %ebp
0x004041d3 <_imp__Sleep@4+239>: js     0x404238 <_libmsvcrt_a_iname+20>
0x004041d5 <_imp__Sleep@4+241>: gs
0x004041d6 <_imp__Sleep@4+242>: jo     0x40424c <_libkernel32_a_iname+12>
0x004041d8 <_imp__Sleep@4+244>: imul   $0x746c6946,0x6e(%edi),%ebp
0x004041df <_imp__Sleep@4+251>: gs
0x004041e0 <_imp__Sleep@4+252>: jb     0x4041e2 <_imp__Sleep@4+254>
0x004041e2 <_imp__Sleep@4+254>: add    %al,(%eax)
0x004041e4 <_imp__Sleep@4+256>: jmp    0x4041e8 <_imp__Sleep@4+260>
0x004041e6 <_imp__Sleep@4+258>: push   %ebx
0x004041e7 <_imp__Sleep@4+259>: insb   (%dx),%es:(%edi)
0x004041e8 <_imp__Sleep@4+260>: gs
0x004041e9 <_imp__Sleep@4+261>: gs
0x004041ea <_imp__Sleep@4+262>: jo     0x4041ec <_imp__Sleep@4+264>
0x004041ec <_imp__Sleep@4+264>: add    %al,0x0(%eax)
0x004041ef <_imp__Sleep@4+267>: add    %al,(%eax)
0x004041f1 <_imp__Sleep@4+269>: inc    %eax
0x004041f2 <_imp__Sleep@4+270>: add    %al,(%eax)
0x004041f4 <_imp__Sleep@4+272>: add    %al,0x0(%eax)
0x004041f7 <_imp__Sleep@4+275>: add    %al,(%eax)
0x004041f9 <_imp__Sleep@4+277>: inc    %eax
0x004041fa <_imp__Sleep@4+278>: add    %al,(%eax)
0x004041fc <_imp__Sleep@4+280>: add    %al,0x0(%eax)
0x004041ff <_imp__Sleep@4+283>: add    %al,(%eax)
0x00404201 <_imp__Sleep@4+285>: inc    %eax
0x00404202 <_imp__Sleep@4+286>: add    %al,(%eax)
0x00404204 <_imp__Sleep@4+288>: add    %al,0x0(%eax)
0x00404207 <_imp__Sleep@4+291>: add    %al,(%eax)
0x00404209 <_imp__Sleep@4+293>: inc    %eax
0x0040420a <_imp__Sleep@4+294>: add    %al,(%eax)
0x0040420c <_imp__Sleep@4+296>: add    %al,0x0(%eax)
0x0040420f <_imp__Sleep@4+299>: add    %al,(%eax)
0x00404211 <_imp__Sleep@4+301>: inc    %eax
0x00404212 <_imp__Sleep@4+302>: add    %al,(%eax)
0x00404214 <_imp__Sleep@4+304>: add    %al,0x0(%eax)
0x00404217 <_imp__Sleep@4+307>: add    %al,(%eax)
0x00404219 <_imp__Sleep@4+309>: inc    %eax
0x0040421a <_imp__Sleep@4+310>: add    %al,(%eax)
0x0040421c <_imp__Sleep@4+312>: add    %al,0x0(%eax)
0x0040421f <_imp__Sleep@4+315>: add    %al,(%eax)
0x00404221 <_imp__Sleep@4+317>: inc    %eax
0x00404222 <_imp__Sleep@4+318>: add    %al,(%eax)
End of assembler dump.

Also I want to mention that it might have something to do with threading.
In a nonthreaded app I don't have any problems with gdb 6.3.50 stack
dumping.

Thanks for your help,

Roland

Re: gdb stack trace problems (Addendum)

[Mark Kettenis]

Hi Roland,

[Dropped Dan, he's busy enough with other stuff, and he can always read it
on the list]

> Also I want to mention that it might have something to do with threading.
> In a nonthreaded app I don't have any problems with gdb 6.3.50 stack
> dumping.

Only if it links in different code, or is multi-threading still cooperative
on Windblows.  Anyway, what I really need is the disassembly for SleepEx as
implemented in the DLL.  Not the import stub, or dissassembly of some random
memory contents (the Sleep you showd is an import stub, and the call
*0xdeadbeef in it means it calls the contents of address 0xdeadbeef, not
address 0xdeadbeef itself).

Cheers,

Mark