Thread backtrace termination

Thread backtrace termination

[Jonathan Larmour]

After a move from GDB 6.1 to GDB 6.3, something new happens with 
backtraces on at least ARM and MIPS, e.g.:

(gdb) bt
#0  breakme () at 
/home/jlarmour/ecos/ecospro-common-040929-branch/packages/kernel/current/tests/thread_gdb.c:108
#1  0xffffffff80021800 in controller (id=0) at 
/home/jlarmour/ecos/ecospro-common-040929-branch/packages/kernel/current/tests/thread_gdb.c:155
#2  0xffffffff8002a550 in Cyg_HardwareThread::thread_entry 
(thread=0x8003d3b0) at 
/home/jlarmour/ecos/ecospro-common-040929-branch/packages/kernel/current/src/common/thread.cxx:110
#3  0xffffffff8002a500 in global constructors keyed to cyg_scheduler_start 
() at 
/home/jlarmour/ecos/ecospro-common-040929-branch/packages/kernel/current/src/common/kapi.cxx:1257
#4  0xffffffff8002a500 in global constructors keyed to cyg_scheduler_start 
() at 
/home/jlarmour/ecos/ecospro-common-040929-branch/packages/kernel/current/src/common/kapi.cxx:1257
Previous frame identical to this frame (corrupt stack?)
(gdb)

The two "global constructors keyed to cyg_scheduler_start" lines are bogus 
frame entries, although those also happened with GDB 6.1. The "corrupt 
stack" whinge is new, and is treated as an error, including terminating 
gdbinit scripts etc.

I tried reverting 
<http://sources.redhat.com/ml/gdb-patches/2004-01/msg00104.html>, but that 
in itself isn't the issue. I know there were a bunch of other dwarf 
unwinder changes for GDB 6.2. But rather than try and explain what I've 
already tried to do, I'd be interested if someone could clarify to me what 
the termination conditions for a backtrace actually _are_. i.e. as an OS 
author, how do I initialise a thread context to persuade GDB to stop when 
it reaches the innermost frame. I've tried looking at the glibc sources to 
see how its thread support works, but it's rather a twisty maze of 
passages, and I don't know whether it would have this problem as well anyway.

One thought I had was if it halted if the saved stack pointer is 0, but 
that presents all sorts of problems for OS implementors, unless thread 
entry functions must now be written in assembler with knowledge of how GDB 
does its unwinding. In trying to create that scenario, I couldn't provoke 
it to affect things anyway.

Jifl
-- 
eCosCentric    http://www.eCosCentric.com/    The eCos and RedBoot experts
--["No sense being pessimistic, it wouldn't work anyway"]-- Opinions==mine

Re: Thread backtrace termination

[Daniel Jacobowitz]

On Mon, Jul 11, 2005 at 05:20:55PM +0100, Jonathan Larmour wrote:
> The two "global constructors keyed to cyg_scheduler_start" lines are bogus 
> frame entries, although those also happened with GDB 6.1. The "corrupt 
> stack" whinge is new, and is treated as an error, including terminating 
> gdbinit scripts etc.

This is already changed in CVS.

> I tried reverting 
> <http://sources.redhat.com/ml/gdb-patches/2004-01/msg00104.html>, but that 
> in itself isn't the issue. I know there were a bunch of other dwarf 
> unwinder changes for GDB 6.2. But rather than try and explain what I've 
> already tried to do, I'd be interested if someone could clarify to me what 
> the termination conditions for a backtrace actually _are_. i.e. as an OS 
> author, how do I initialise a thread context to persuade GDB to stop when 
> it reaches the innermost frame. I've tried looking at the glibc sources to 
> see how its thread support works, but it's rather a twisty maze of 
> passages, and I don't know whether it would have this problem as well 
> anyway.

In general there's no defined way to do this.  If the start routine is
written in assembly, take a look at the example I posted earlier this
year of using dwarf2 unwind information to terminate a backtrace by
marking the return address column as undefined.  There's a matching GDB
patch, which was committed to HEAD after 6.3.

For compiler-generated code there's really no useful way to do this.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: Thread backtrace termination

[Jonathan Larmour]

Daniel Jacobowitz wrote:
> On Mon, Jul 11, 2005 at 05:20:55PM +0100, Jonathan Larmour wrote:
> 
>>The two "global constructors keyed to cyg_scheduler_start" lines are bogus 
>>frame entries, although those also happened with GDB 6.1. The "corrupt 
>>stack" whinge is new, and is treated as an error, including terminating 
>>gdbinit scripts etc.
> 
> 
> This is already changed in CVS.

Okay. I haven't seen that. It still looks like an error to me in frame.c.

BTW, my other web searches seem to indicate that a fair few (naive) people 
are thinking they are having stack corruption because GDB thinks there 
might be. That's unfortunate.

>>I'd be interested if someone could clarify to me what 
>>the termination conditions for a backtrace actually _are_. i.e. as an OS 
>>author, how do I initialise a thread context to persuade GDB to stop when 
>>it reaches the innermost frame. 
> 
> In general there's no defined way to do this.  If the start routine is
> written in assembly, take a look at the example I posted earlier this
> year of using dwarf2 unwind information to terminate a backtrace by
> marking the return address column as undefined.  There's a matching GDB
> patch, which was committed to HEAD after 6.3.

I've had a search for this and not found anything. I'm probably just not 
using the right terms. Do you have a pointer, time frame or some search 
terms I can use to pin this down? Thanks.

> For compiler-generated code there's really no useful way to do this.

I guess atleast now I know that, which saves me spending more time.

Wouldn't it make sense to make such a convention though, such as having a 
return address of 0?

Alternatively, how about adding a new command that allows you to define a 
set of entry point symbol names? People can then put an appropriate list 
for themselves or their OS in ~/.gdbinit. Or it can be pre-initialised by 
the OS support within GDB if there is one. e.g. nm-linux.h. Here's what 
I'm thinking of:

set entry-point-name-list main _start _entry

Although handling mangled symbols and multiple languages might be fun. I'm 
not an expert on such things.

Thanks,

Jifl
-- 
eCosCentric    http://www.eCosCentric.com/    The eCos and RedBoot experts
--["No sense being pessimistic, it wouldn't work anyway"]-- Opinions==mine

Re: Thread backtrace termination

[Daniel Jacobowitz]

On Mon, Jul 11, 2005 at 06:52:13PM +0100, Jonathan Larmour wrote:
> Daniel Jacobowitz wrote:
> >On Mon, Jul 11, 2005 at 05:20:55PM +0100, Jonathan Larmour wrote:
> >
> >>The two "global constructors keyed to cyg_scheduler_start" lines are 
> >>bogus frame entries, although those also happened with GDB 6.1. The 
> >>"corrupt stack" whinge is new, and is treated as an error, including 
> >>terminating gdbinit scripts etc.
> >
> >
> >This is already changed in CVS.
> 
> Okay. I haven't seen that. It still looks like an error to me in frame.c.

The error is caught in the top level code for the backtrace command;
that effectively downgrades it to a warning and backtrace termination.

> BTW, my other web searches seem to indicate that a fair few (naive) people 
> are thinking they are having stack corruption because GDB thinks there 
> might be. That's unfortunate.

What else would you suggest?  GDB is confused.  From its point of view,
the stack _is_ corrupt.

> I've had a search for this and not found anything. I'm probably just not 
> using the right terms. Do you have a pointer, time frame or some search 
> terms I can use to pin this down? Thanks.

Well, the patch was:

2005-04-08  Daniel Jacobowitz  <dan@codesourcery.com>

        * dwarf2-frame.c (struct dwarf2_frame_cache): New field
        undefined_retaddr.
        (dwarf2_frame_cache): Initialize undefined_retaddr.
        (dwarf2_frame_this_id): Return an invalid frame ID if
        undefined_retaddr.

You can find the discussion and sample use on gdb@ a month or two
earlier.

> >For compiler-generated code there's really no useful way to do this.
> 
> I guess atleast now I know that, which saves me spending more time.
> 
> Wouldn't it make sense to make such a convention though, such as having a 
> return address of 0?

This is basically a convention.  You could, I suppose, patch a compiler
to generate it.  I'm not sure that would be wise.

> Alternatively, how about adding a new command that allows you to define a 
> set of entry point symbol names? People can then put an appropriate list 
> for themselves or their OS in ~/.gdbinit. Or it can be pre-initialised by 
> the OS support within GDB if there is one. e.g. nm-linux.h. Here's what 
> I'm thinking of:
> 
> set entry-point-name-list main _start _entry
> 
> Although handling mangled symbols and multiple languages might be fun. I'm 
> not an expert on such things.

*shrug* maybe.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: Thread backtrace termination

[Jonathan Larmour]

Daniel Jacobowitz wrote:
> On Mon, Jul 11, 2005 at 06:52:13PM +0100, Jonathan Larmour wrote:
> 
>>Daniel Jacobowitz wrote:
>>
>>>On Mon, Jul 11, 2005 at 05:20:55PM +0100, Jonathan Larmour wrote:
>>>
>>>
>>>>The two "global constructors keyed to cyg_scheduler_start" lines are 
>>>>bogus frame entries, although those also happened with GDB 6.1. The 
>>>>"corrupt stack" whinge is new, and is treated as an error, including 
>>>>terminating gdbinit scripts etc.
[snip]
> The error is caught in the top level code for the backtrace command;
> that effectively downgrades it to a warning and backtrace termination.

Ah ok, thanks.

>>BTW, my other web searches seem to indicate that a fair few (naive) people 
>>are thinking they are having stack corruption because GDB thinks there 
>>might be. That's unfortunate.
> 
> 
> What else would you suggest?  GDB is confused.  From its point of view,
> the stack _is_ corrupt.

It's possibly alarmist, but it's no big deal.

> Well, the patch was:[snip]
> You can find the discussion and sample use on gdb@ a month or two
> earlier.

Aha, yes, at 
http://sources.redhat.com/ml/gdb-patches/2005-03/msg00047.html and friends.

That seems interesting but somewhat unwieldy, and as you said before, 
wouldn't apply to compiler generated code.

>>>For compiler-generated code there's really no useful way to do this.
>>
>>I guess atleast now I know that, which saves me spending more time.
>>
>>Wouldn't it make sense to make such a convention though, such as having a 
>>return address of 0?
> 
> 
> This is basically a convention.  You could, I suppose, patch a compiler
> to generate it.  I'm not sure that would be wise.

If someone were to come up with an __attribute__ that could be used with 
GCC to mark functions to be annotated this way, it might be possible. But 
it's beyond me (or at least, beyond what I have time to get up to speed 
with) and I doubt anyone else will be that interested. Ho hum.

>>Alternatively, how about adding a new command that allows you to define a 
>>set of entry point symbol names? People can then put an appropriate list 
>>for themselves or their OS in ~/.gdbinit. Or it can be pre-initialised by 
>>the OS support within GDB if there is one. e.g. nm-linux.h. Here's what 
>>I'm thinking of:
>>
>>set entry-point-name-list main _start _entry
>>
>>Although handling mangled symbols and multiple languages might be fun. I'm 
>>not an expert on such things.
> 
> 
> *shrug* maybe.

Well, I'm prepared to create a patch to add such a command if people here 
think something with that principle would be accepted.

Jifl
-- 
eCosCentric    http://www.eCosCentric.com/    The eCos and RedBoot experts
--["No sense being pessimistic, it wouldn't work anyway"]-- Opinions==mine

Re: Thread backtrace termination

[Steven Johnson]

Jonathan Larmour wrote:

> Daniel Jacobowitz wrote:
>
>> On Mon, Jul 11, 2005 at 06:52:13PM +0100, Jonathan Larmour wrote:
>>
>>> Daniel Jacobowitz wrote:
>>
[snip]

>>> Alternatively, how about adding a new command that allows you to 
>>> define a set of entry point symbol names? People can then put an 
>>> appropriate list for themselves or their OS in ~/.gdbinit. Or it can 
>>> be pre-initialised by the OS support within GDB if there is one. 
>>> e.g. nm-linux.h. Here's what I'm thinking of:
>>>
>>> set entry-point-name-list main _start _entry
>>>
>>> Although handling mangled symbols and multiple languages might be 
>>> fun. I'm not an expert on such things.
>>
>>
>>
>> *shrug* maybe.
>
>
> Well, I'm prepared to create a patch to add such a command if people 
> here think something with that principle would be accepted.

Instead of a "set" command, which sets an entire list, an "add" and 
"delete" command that adds to the list and deletes from the list 
respectively would be (in my opinion) more useful, because GDB could 
default to the standard entries for a target, and then extra's could be 
added to/removed from the list.

set entry-point-list add [name]
set entry-point-list delete [name]

(for example)

Alternatively there could be 2 lists, one full of default which you 
usually dont change, and one with user added entries.

set entry-point-name-list-default [list]
set entry-point-name-list-user [list]

By default user would be empty, and would be only set by the user (in a 
script or otherwise).  So the user could be assured they werent messing 
with any defaults they didnt want to change.

or the like.

supporting wildcards might also be useful, such as:

set entry-point-list add thread_*

or

set entry-point-list delete *

so if a programmer had a convention of calling all their threads 
"thread_blah" they wouldnt have to explicitly name them all, back trace 
would terminate at any function with that naming convention. 

Just some suggestions, if you are going to do this.  My opinion is such 
a feature would be worthwhile.  If the information can not be easily 
provided (or at all) from debug symbols, it would seem highly desirable 
to have a manual method of inserting the information.

Steven

GDB is stepping past main()

[Konstantin Karganov]

Hello, all.

Why doesn't gdb stop at the main() end and continues to step further?
Ok, it's not a big problem for human, but what should the frontend 
debugger do with unknown functions w/o source file positions?

Test case: gcc 3.2.2 and gdb 6.3 on RH Linux, session log here:

(gdb) br main
Breakpoint 1 at 0x8048618: file tmp.cpp, line 7.
(gdb) run
Starting program: /home/kostik/Work/a.out
                                                                                
Breakpoint 1, main (argc=1, argv=0xbffff2f4) at tmp.cpp:7
7               cout<<" Hello world! "<<endl;
(gdb) l
2
3       using namespace std;
4
5       int main(int argc,char* argv[])
6       {
7               cout<<" Hello world! "<<endl;
8       }
9
(gdb) step
 Hello world!
8       }
(gdb) step
0x42015574 in __libc_start_main () from /lib/tls/libc.so.6
(gdb) step
Single stepping until exit from function __libc_start_main,
which has no line number information.
                                                                                
Program exited normally.
(gdb)

What the hell "__libc_start_main" is doing here?
The same happens using "next" instead of "step" and when using a MI 
interface...

Best regards,
Konstantin.

Re: GDB is stepping past main()

[Daniel Jacobowitz]

On Wed, Jul 13, 2005 at 04:38:54PM +0400, Konstantin Karganov wrote:
> Hello, all.
> 
> Why doesn't gdb stop at the main() end and continues to step further?
> Ok, it's not a big problem for human, but what should the frontend 
> debugger do with unknown functions w/o source file positions?

Because the user (in your example) has asked it to step out of main.
You might want to examine global state now that the program is mostly
finished.  Or you might want to debug glibc.

If you want to handle this in a debugger, try detecting when you're in
main, step, are no longer in main, and main is no longer in the
backtrace.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: GDB is stepping past main()

[Konstantin Karganov]

> Because the user (in your example) has asked it to step out of main.
I asked to step and started waiting for the program termination message.
And get that stopped somewhere "don't know where"...

> If you want to handle this in a debugger, try detecting when you're in
> main, step, are no longer in main, and main is no longer in the
> backtrace.
Don't you think all this detecting is a debugger's task???

Or else it turns out that the "step" in gdb is broken and has no 
defined semantics and needs to be carefully controlled from outside 
(whether it made the stepping or not and where it happened to stop)!

And all this "detecting" and outside patches-workarounds are very ugly 
solution :(

Working with GDB MI itself is finding the way in the dark, and such
"surprises" like broken commands doesn't rise gdb in users eyes at all...

Best regards,
Konstantin.

Re: GDB is stepping past main()

[Nathan J. Williams]

Konstantin Karganov <kostik@ispras.ru> writes:

> > Because the user (in your example) has asked it to step out of main.
> I asked to step and started waiting for the program termination message.
> And get that stopped somewhere "don't know where"...
> 
> > If you want to handle this in a debugger, try detecting when you're in
> > main, step, are no longer in main, and main is no longer in the
> > backtrace.
> Don't you think all this detecting is a debugger's task???
> 
> Or else it turns out that the "step" in gdb is broken and has no 
> defined semantics and needs to be carefully controlled from outside 
> (whether it made the stepping or not and where it happened to stop)!

"step" is a command that's closely linked to the notion of a source
code line. When you're outside of the region where you have source
(such as the startup code around main()), you're in less-defined
territory.

You might find the "stepi" command better suits your needs, though I'm
not at all sure from your messages what, exactly, you think is
supposed to happen.

        - Nathan

Re: GDB is stepping past main()

[Daniel Jacobowitz]

On Wed, Jul 13, 2005 at 05:17:01PM +0400, Konstantin Karganov wrote:
> 
> > Because the user (in your example) has asked it to step out of main.
> I asked to step and started waiting for the program termination message.
> And get that stopped somewhere "don't know where"...
> 
> > If you want to handle this in a debugger, try detecting when you're in
> > main, step, are no longer in main, and main is no longer in the
> > backtrace.
> Don't you think all this detecting is a debugger's task???

As a user, I would prefer the debugger not silently run my program to
completion during a "step" command, when issued from a location with
source level debugging information.  You're the first person I can ever
recall complaining about this behavior.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: GDB is stepping past main()

[Konstantin Karganov]

> You might find the "stepi" command better suits your needs, though I'm
> not at all sure from your messages what, exactly, you think is
> supposed to happen.
I suppose that the program execution starts when main() starts and ends 
when main() ends (though in c++ it is not exactly so).

And thus I suppose the debugging to start at the main() beginning and stop 
when main() finishes.
The code above main do not exist for me.
Stepping past the end of main should report the program exit.

Best regards,
Konstantin.

RE: GDB is stepping past main()

[Dave Korn]

----Original Message----
>From: Konstantin Karganov
>Sent: 13 July 2005 14:32

>> You might find the "stepi" command better suits your needs, though I'm
>> not at all sure from your messages what, exactly, you think is
>> supposed to happen.
> I suppose that the program execution starts when main() starts and ends
> when main() ends 

  Wrong.  The program does stuff both before and after main().

> The code above main do not exist for me.

  Wrong.  It's often vital to debug the initiation and termination code.

> Stepping past the end of main should report the program exit.

  Wrong.  The program has not yet exited, so gdb should not report it.


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: GDB is stepping past main()

[Konstantin Karganov]

> As a user, I would prefer the debugger not silently run my program to
> completion during a "step" command, when issued from a location with
> source level debugging information.  You're the first person I can ever
> recall complaining about this behavior.
As the default stepping mode is "step-mode off" (do not enter the 
functions w/o debug info) I suppose that the current position is always in 
user source code and the FE debugger can display it.
But when stepping at the end of main() this assumption fails - no matter 
what is the current step-mode, the debugger gets to the place where it 
can't report the current position.
I suppose that the step-mode should affect this behavior also.

Best regards,
Konstantin.

MI *stopped reason

[Konstantin Karganov]

Hello.

When the execution of the debuggee stops the debugger reports the result 
as "*stopped, reason='...' ". I'd like to know if the reason should always 
be specified, or not.
Currently, there are cases when the reason is missing (for example, when 
stopped on a temporary breakpoint).

Best regards,
konstantin.

Re: MI *stopped reason

[Bob Rossi]

On Thu, Jul 14, 2005 at 06:12:55PM +0400, Konstantin Karganov wrote:
> Hello.
> 
> When the execution of the debuggee stops the debugger reports the result 
> as "*stopped, reason='...' ". I'd like to know if the reason should always 
> be specified, or not.
> Currently, there are cases when the reason is missing (for example, when 
> stopped on a temporary breakpoint).

I believe the reason should always be specified. Although there is no MI
standard that states it, right? If you know where to add the reason in
breakpoint.c simply add a new enumeration to
mi/mi-command.h:async_reply_reason. 

Without the reason, the front end is left a little confused as to what
happened.

Thanks,
Bob Rossi

Incorrect breakpoint diagnostics in MI

[Konstantin Karganov]

Hello.

Playing with MI breakpoint commands I found the following thing:
Only "-break-after" and "-break-condition" correctly report the error if 
the breakpoint doesn't exist. While other commands report "^done".

(gdb)
-break-after 123 123
&"No breakpoint number 123.\n"
^error,msg="No breakpoint number 123."
(gdb)
-break-enable 123
~"No breakpoint number 123.\n"
^done
(gdb)
-break-disable 123
~"No breakpoint number 123.\n"
^done
(gdb)
-break-delete 123
~"No breakpoint number 123.\n"
^done

The diagnostics goes to console output, that is possibly not visible to FE 
user and the result itself is "ok".

Is it a bug?

Best reagards,
Konstantin.