Re: gdb stack trace problems

Re: gdb stack trace problems

[Roland Schwingel]

Hi Daniel and List,

Thanks for your response.

Daniel Jacobowitz wrote on 16.04.2005 16:09:50:
 > On Fri, Apr 15, 2005 at 05:06:05PM +0200, Roland Schwingel wrote:
 > > Hi....
 > >
 > > At present we are suffering from stack trace problems with recent
 > > versions of gdb.
 > > I desperately hope someone knows a reason for it and can help.
 > >
 > > But from the beginning:
 > > We are using gdb on cygwin to debug mingw programs. GDB works well but
 > > in recent
 > > version (starting with 6.x)  it cannot produce correct stack dumps
 > > anymore when something
 > > has crashed. Our applications are multithreaded.
 > >
 > > In the past we used 5.3 which worked much better in that case but had
 > > other problems.
 > > We switched a while ago to 6.2 and then to 6.2.1 and today I compiled
 > > 6.3.50 from CVS.
 > > And it is all the same... When I get a crash I just get corrupted 
stacks
 > > telling nothing useful.
 > >
 > > Does anyone know what is the reason for this and how to fix it?
 >
 > Sorry, but you haven't given us any information to answer your question
 > with.  Test case?  Compiler version?  Example?
Ok... Thought that would be already known in a broader space, as after 
asking
google for gdb stack trace problems there had been some hits but none with a
good answer...

So here are the infos....
I am running gcc-3.2.2 on cygwin. The binaries to debug are
compiled for mingw.

I have attached a small example with output from gdb 6.3.50 and 5.3
to see what I mean. It is a pretty simple example. It is a small hack
generating 3 additional threads. Each thread decrements after some
sleeping a global variable. When the variable reaches 0 the thread
who did the last decrement generates a segmentation violation
to crash. With this synthetic example I can get the resulting
stack trace of the crashing thread with both versions of gdb,
but with 6.x gdbs I can no longer get the stack dumps of the
other threads. In real life when generating threads from
threads from threads I most of the time did not get any
good stack dump of the crashing thread at all with 6.x
gdbs, which is really a kille. With 5.3 I can get good
stack dumps. I will try  to improve my example to
also show what I mean.

With this example you see the difference in output of gdb 5.3 and 6.2.
Any clues why the output of gdb 6.x is IMHO much bader
compared to the 5.3 output?

Example:
============================================================
#include <stdio.h>
#include <windows.h>

void func1(int num);

int var = 23;

void crashIfZero(int num)
{
    var--;

    if (var == 0)
    {
        int *data=0x0;
        printf ("I am thread %d and I will crash now!\n",num);
       
        *data=911;
    }
    else
        printf ("Thread %d: var = %d\n",num,var);       
}

void func4(int num)
{
    Sleep(100);
    crashIfZero(num);
    func1(num);
}

void func3(int num)
{
    Sleep(100);
    crashIfZero(num);
    func4(num);
}

void func2(int num)
{
    Sleep(100);
    crashIfZero(num);
    func3(num);
}

void func1(int num)
{
    Sleep(100);
    crashIfZero(num);
    func2(num);
}

DWORD WINAPI threadFunc(LPVOID param)
{
    int num = *(DWORD *)param;
    printf ("I am thread %d and alive\n");
    func1(num);
}

void makeThreads(int num)
{
    int i;

    for (i=1;i<=num;i++)
    {
        HANDLE threadHandle;
        DWORD  threadId, threadParam = i;
        threadHandle = 
CreateThread(NULL,0,threadFunc,&threadParam,0,&threadId);
       
        if (threadHandle == NULL)
            printf ("Couldn't create thread %d\n",i);
        else
            printf ("Created thread %d with ID: %d\n",i,threadId);
       
        Sleep(50);
    }
   
    printf ("Created %d Threads....\n",num);
    Sleep(200000);
}

int main(int argc,char **argv)
{
    setbuf(stdout,NULL);
    setbuf(stderr,NULL);

    printf ("test gdb stack tracing during a crash\n");
    makeThreads(3);
}
============================================================
Compiled with:
$ gcc -g -mno-cygwin gdbtest.c -o gdbtest

Debugging session with 6.3.50:
=====================
$ gdb.exe ./gdbtest.exe
GNU gdb 6.3.50.20050415-cvs
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-cygwin"...
(gdb) r
Starting program: /tmp/gdbtest.exe
test gdb stack tracing during a crash
Created thread 1 with ID: 2984
I am thread 1 and alive
Created thread 2 with ID: 3980
I am thread 2 and alive
Thread 1: var = 22
Created thread 3 with ID: 3764
I am thread 3 and alive
Thread 2: var = 21
Created 3 Threads....
Thread 1: var = 20
Thread 3: var = 19
Thread 2: var = 18
Thread 1: var = 17
Thread 3: var = 16
Thread 2: var = 15
Thread 1: var = 14
Thread 3: var = 13
Thread 2: var = 12
Thread 1: var = 11
Thread 3: var = 10
Thread 2: var = 9
Thread 1: var = 8
Thread 3: var = 7
Thread 2: var = 6
Thread 1: var = 5
Thread 3: var = 4
Thread 2: var = 3
Thread 1: var = 2
Thread 3: var = 1
I am thread 2 and I will crash now!

Program received signal SIGSEGV, Segmentation fault.
[Switching to thread 2316.0xf8c]
0x0040131d in crashIfZero (num=2) at gdbtest.c:17
17                      *data=911;
(gdb) thr 1
[Switching to thread 1 (thread 2316.0xf10)]#0  0x7c91eb94 in 
ntdll!LdrAccessResource () from ntdll.dll
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c8023ed in SleepEx () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x00000000 in ?? () from
(gdb) thr 2
[Switching to thread 2 (thread 2316.0xba8)]#0  0x7c91eb94 in 
ntdll!LdrAccessResource () from ntdll.dll
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c8023ed in SleepEx () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x00000000 in ?? () from
(gdb) thr 3
[Switching to thread 3 (thread 2316.0xf8c)]#0  0x0040131d in crashIfZero 
(num=2) at gdbtest.c:17
17                      *data=911;
(gdb) bt
#0  0x0040131d in crashIfZero (num=2) at gdbtest.c:17
#1  0x00401363 in func4 (num=2) at gdbtest.c:26
#2  0x0040139b in func3 (num=2) at gdbtest.c:34
#3  0x004013c8 in func2 (num=2) at gdbtest.c:41
#4  0x004013f5 in func1 (num=2) at gdbtest.c:48
#5  0x0040136e in func4 (num=2) at gdbtest.c:27
#6  0x0040139b in func3 (num=2) at gdbtest.c:34
#7  0x004013c8 in func2 (num=2) at gdbtest.c:41
#8  0x004013f5 in func1 (num=2) at gdbtest.c:48
#9  0x00401436 in threadFunc (param=0x22ff58) at gdbtest.c:55
#10 0x7c80b50b in KERNEL32!GetModuleFileNameA () from 
/vol/c/WINDOWS/system32/kernel32.dll
#11 0x0022ff58 in ?? ()
#12 0x0022fc40 in ?? ()
#13 0x77bfe524 in msvcrt!_get_osfhandle ()
#14 0x0022ff58 in ?? ()
#15 0x7ffdd000 in ?? ()
#16 0x867c0600 in ?? ()
#17 0x008bffc0 in ?? ()
#18 0x8601ee40 in ?? ()
#19 0xffffffff in ?? ()
#20 0x7c8399f3 in KERNEL32!FindAtomW () from 
/vol/c/WINDOWS/system32/kernel32.dll
#21 0x7c80b518 in KERNEL32!GetModuleFileNameA () from 
/vol/c/WINDOWS/system32/kernel32.dll
#22 0x00000000 in ?? () from
(gdb) thr 4
[Switching to thread 4 (thread 2316.0xeb4)]#0  0x7c91eb94 in 
ntdll!LdrAccessResource () from ntdll.dll
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c8023ed in SleepEx () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x00000000 in ?? () from

Debugging with gdb 5.3:
=================
$ gdb53 ./gdbtest.exe
GNU gdb 5.3 2003-03-03-cvs
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-cygwin"...
(gdb) r
(gdb) r
Starting program: /tmp/gdbtest.exe
test gdb stack tracing during a crash
Created thread 1 with ID: 3580
I am thread 1 and alive
Created thread 2 with ID: 3320
I am thread 2 and alive
Thread 1: var = 22
Created thread 3 with ID: 4068
I am thread 3 and alive
Thread 2: var = 21
Created 3 Threads....
Thread 1: var = 20
Thread 3: var = 19
Thread 2: var = 18
Thread 1: var = 17
Thread 3: var = 16
Thread 2: var = 15
Thread 1: var = 14
Thread 3: var = 13
Thread 2: var = 12
Thread 1: var = 11
Thread 3: var = 10
Thread 2: var = 9
Thread 1: var = 8
Thread 3: var = 7
Thread 2: var = 6
Thread 1: var = 5
Thread 3: var = 4
Thread 2: var = 3
Thread 1: var = 2
Thread 3: var = 1
I am thread 2 and I will crash now!

Program received signal SIGSEGV, Segmentation fault.

[Switching to thread 604.0xcf8]
0x0040131d in crashIfZero (num=2) at gdbtest.c:17
17                      *data=911;
(gdb) thr 1
[Switching to thread 1 (thread 604.0xd2c)]#0  0x7c91eb94 in 
ntdll!LdrAccessResource () from ntdll.dll
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x0040156c in makeThreads (num=3) at gdbtest.c:77
#4  0x0040161d in main (argc=1, argv=0x3d4068) at gdbtest.c:86
(gdb) thr 2
[Switching to thread 2 (thread 604.0xdfc)]#0  0x7c91eb94 in 
ntdll!LdrAccessResource () from ntdll.dll
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x004013dc in func1 (num=1) at gdbtest.c:46
#4  0x0040136e in func4 (num=1) at gdbtest.c:27
#5  0x0040139b in func3 (num=1) at gdbtest.c:34
#6  0x004013c8 in func2 (num=1) at gdbtest.c:41
#7  0x004013f5 in func1 (num=1) at gdbtest.c:48
#8  0x0040136e in func4 (num=1) at gdbtest.c:27
#9  0x0040139b in func3 (num=1) at gdbtest.c:34
#10 0x004013c8 in func2 (num=1) at gdbtest.c:41
#11 0x004013f5 in func1 (num=1) at gdbtest.c:48
#12 0x00401436 in threadFunc (param=0x22ff58) at gdbtest.c:55
#13 0x7c80b50b in KERNEL32!GetModuleFileNameA () from 
/vol/c/WINDOWS/system32/kernel32.dll
(gdb) thr 3
[Switching to thread 3 (thread 604.0xcf8)]#0  0x0040131d in crashIfZero 
(num=2) at gdbtest.c:17
17                      *data=911;
(gdb) bt
#0  0x0040131d in crashIfZero (num=2) at gdbtest.c:17
#1  0x00401363 in func4 (num=2) at gdbtest.c:26
#2  0x0040139b in func3 (num=2) at gdbtest.c:34
#3  0x004013c8 in func2 (num=2) at gdbtest.c:41
#4  0x004013f5 in func1 (num=2) at gdbtest.c:48
#5  0x0040136e in func4 (num=2) at gdbtest.c:27
#6  0x0040139b in func3 (num=2) at gdbtest.c:34
#7  0x004013c8 in func2 (num=2) at gdbtest.c:41
#8  0x004013f5 in func1 (num=2) at gdbtest.c:48
#9  0x00401436 in threadFunc (param=0x22ff58) at gdbtest.c:55
#10 0x7c80b50b in KERNEL32!GetModuleFileNameA () from 
/vol/c/WINDOWS/system32/kernel32.dll
(gdb) thr 4
[Switching to thread 4 (thread 604.0xfe4)]#0  0x7c91eb94 in 
ntdll!LdrAccessResource () from ntdll.dll
(gdb) bt
#0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
#1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
#2  0x7c802451 in Sleep () from /vol/c/WINDOWS/system32/kernel32.dll
#3  0x00401355 in func4 (num=3) at gdbtest.c:25
#4  0x0040139b in func3 (num=3) at gdbtest.c:34
#5  0x004013c8 in func2 (num=3) at gdbtest.c:41
#6  0x004013f5 in func1 (num=3) at gdbtest.c:48
#7  0x0040136e in func4 (num=3) at gdbtest.c:27
#8  0x0040139b in func3 (num=3) at gdbtest.c:34
#9  0x004013c8 in func2 (num=3) at gdbtest.c:41
#10 0x004013f5 in func1 (num=3) at gdbtest.c:48
#11 0x00401436 in threadFunc (param=0x22ff58) at gdbtest.c:55
#12 0x7c80b50b in KERNEL32!GetModuleFileNameA () from 
/vol/c/WINDOWS/system32/kernel32.dll

As you might see the stack trace for threads 1,2 and 4 are quite 
unusable in gdb 6.3.50 whereas they are
correct in 5.3. Any clues to get them ok in 6.x?

Regards,

Roland

Re: gdb stack trace problems

[Daniel Jacobowitz]

On Mon, Apr 18, 2005 at 03:47:02PM +0200, Roland Schwingel wrote:
> Ok... Thought that would be already known in a broader space, as after 
> asking
> google for gdb stack trace problems there had been some hits but none with a
> good answer...

Saying "stack trace problems" for GDB is like saying "my code doesn't
compile" for GCC.  It doesn't tell us very much about what has gone
wrong; stack backtracing is a complex problem with lots of
dependencies.

> Program received signal SIGSEGV, Segmentation fault.
> [Switching to thread 2316.0xf8c]
> 0x0040131d in crashIfZero (num=2) at gdbtest.c:17
> 17                      *data=911;
> (gdb) thr 1
> [Switching to thread 1 (thread 2316.0xf10)]#0  0x7c91eb94 in 
> ntdll!LdrAccessResource () from ntdll.dll
> (gdb) bt
> #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
> #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
> #2  0x7c8023ed in SleepEx () from /vol/c/WINDOWS/system32/kernel32.dll
> #3  0x00000000 in ?? () from

OK, here's a concrete problem.  GDB 6.x has a (usually) more reliable
prologue analyzer for x86, compared to GDB 5.3.  However, the old one
got lucky with SleepEx and the new one doesn't.  I'd ask you to post
disassembly of the function except I'm not sure how wise that is with a
Microsoft DLL...

Mark, you did the work on the i386 prologue analyzer.  Any chance you
could take a look at this?


-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: gdb stack trace problems

[Mark Kettenis]

   Date: Mon, 18 Apr 2005 09:57:21 -0400
   From: Daniel Jacobowitz <drow@false.org>

   > Program received signal SIGSEGV, Segmentation fault.
   > [Switching to thread 2316.0xf8c]
   > 0x0040131d in crashIfZero (num=2) at gdbtest.c:17
   > 17                      *data=911;
   > (gdb) thr 1
   > [Switching to thread 1 (thread 2316.0xf10)]#0  0x7c91eb94 in 
   > ntdll!LdrAccessResource () from ntdll.dll
   > (gdb) bt
   > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
   > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
   > #2  0x7c8023ed in SleepEx () from /vol/c/WINDOWS/system32/kernel32.dll
   > #3  0x00000000 in ?? () from

   OK, here's a concrete problem.  GDB 6.x has a (usually) more reliable
   prologue analyzer for x86, compared to GDB 5.3.  However, the old one
   got lucky with SleepEx and the new one doesn't.  I'd ask you to post
   disassembly of the function except I'm not sure how wise that is with a
   Microsoft DLL...

   Mark, you did the work on the i386 prologue analyzer.  Any chance you
   could take a look at this?

Sure if someone can provide me with the dissassembly in AT&T syntax
for that function, or even better some assembler code that exhibits
the problem.  No promises though that I'll fix this.  Last time I
looked at some windows disassembler it looked like it was hand
optimized in such a way that the entire function needed to be analyzed
to do something useful; something that doesn't quite fit in the way we
do things in GDB.

Mark

Re: gdb stack trace problems

[Roland Schwingel]

Hi Mark, Daniel and list....

Mark Kettenis wrote on 18.04.2005 19:10:57:
 >    Date: Mon, 18 Apr 2005 09:57:21 -0400
 >    From: Daniel Jacobowitz <drow@false.org>
 >
 >    > Program received signal SIGSEGV, Segmentation fault.
 >    > [Switching to thread 2316.0xf8c]
 >    > 0x0040131d in crashIfZero (num=2) at gdbtest.c:17
 >    > 17                      *data=911;
 >    > (gdb) thr 1
 >    > [Switching to thread 1 (thread 2316.0xf10)]#0  0x7c91eb94 in
 >    > ntdll!LdrAccessResource () from ntdll.dll
 >    > (gdb) bt
 >    > #0  0x7c91eb94 in ntdll!LdrAccessResource () from ntdll.dll
 >    > #1  0x7c91d85c in ntdll!ZwDelayExecution () from ntdll.dll
 >    > #2  0x7c8023ed in SleepEx () from 
/vol/c/WINDOWS/system32/kernel32.dll
 >    > #3  0x00000000 in ?? () from
 >
 >    OK, here's a concrete problem.  GDB 6.x has a (usually) more reliable
 >    prologue analyzer for x86, compared to GDB 5.3.  However, the old one
 >    got lucky with SleepEx and the new one doesn't.  I'd ask you to post
 >    disassembly of the function except I'm not sure how wise that is 
with a
 >    Microsoft DLL...
 >
 >    Mark, you did the work on the i386 prologue analyzer.  Any chance you
 >    could take a look at this?
 >
 > Sure if someone can provide me with the dissassembly in AT&T syntax
 > for that function, or even better some assembler code that exhibits
 > the problem.  No promises though that I'll fix this.  Last time I
 > looked at some windows disassembler it looked like it was hand
 > optimized in such a way that the entire function needed to be analyzed
 > to do something useful; something that doesn't quite fit in the way we
 > do things in GDB.
Here is the disassembly of one function (func1) that cannot be shown 
correctly
from within gdb 6.3.50 in att flavor:
(gdb) disas func1
Dump of assembler code for function func1:
0x004013ca <func1+0>:   push   %ebp
0x004013cb <func1+1>:   mov    %esp,%ebp
0x004013cd <func1+3>:   sub    $0x8,%esp
0x004013d0 <func1+6>:   movl   $0x64,(%esp)
0x004013d7 <func1+13>:  call   0x401830 <Sleep@4>
0x004013dc <func1+18>:  sub    $0x4,%esp
0x004013df <func1+21>:  mov    0x8(%ebp),%eax
0x004013e2 <func1+24>:  mov    %eax,(%esp)
0x004013e5 <func1+27>:  call   0x4012eb <crashIfZero>
0x004013ea <func1+32>:  mov    0x8(%ebp),%eax
0x004013ed <func1+35>:  mov    %eax,(%esp)
0x004013f0 <func1+38>:  call   0x40139d <func2>
0x004013f5 <func1+43>:  leave 
0x004013f6 <func1+44>:  ret   
End of assembler dump.

I hope this is what you requested. I will aid in tracking this down,
wherever I can. As you see from the assembly Sleep() is
called not SleepEx() as shown in the gdb 6.3.50 stack
trace. In gdb 5.3 stack trace Sleep() is correctly shown if you
look at my posting from yesterday

Roland

Re: gdb stack trace problems

[Daniel Jacobowitz]

On Fri, Apr 15, 2005 at 05:06:05PM +0200, Roland Schwingel wrote:
> Hi....
> 
> At present we are suffering from stack trace problems with recent 
> versions of gdb.
> I desperately hope someone knows a reason for it and can help.
> 
> But from the beginning:
> We are using gdb on cygwin to debug mingw programs. GDB works well but 
> in recent
> version (starting with 6.x)  it cannot produce correct stack dumps 
> anymore when something
> has crashed. Our applications are multithreaded.
> 
> In the past we used 5.3 which worked much better in that case but had 
> other problems.
> We switched a while ago to 6.2 and then to 6.2.1 and today I compiled 
> 6.3.50 from CVS.
> And it is all the same... When I get a crash I just get corrupted stacks 
> telling nothing useful.
> 
> Does anyone know what is the reason for this and how to fix it?

Sorry, but you haven't given us any information to answer your question
with.  Test case?  Compiler version?  Example?

-- 
Daniel Jacobowitz
CodeSourcery, LLC

gdb stack trace problems

[Roland Schwingel]

Hi....

At present we are suffering from stack trace problems with recent 
versions of gdb.
I desperately hope someone knows a reason for it and can help.

But from the beginning:
We are using gdb on cygwin to debug mingw programs. GDB works well but 
in recent
version (starting with 6.x)  it cannot produce correct stack dumps 
anymore when something
has crashed. Our applications are multithreaded.

In the past we used 5.3 which worked much better in that case but had 
other problems.
We switched a while ago to 6.2 and then to 6.2.1 and today I compiled 
6.3.50 from CVS.
And it is all the same... When I get a crash I just get corrupted stacks 
telling nothing useful.

Does anyone know what is the reason for this and how to fix it?

Many thanks for all your help,

Roland