tracing, attaching to gdb processes

tracing, attaching to gdb processes

[Ed Peschko]

all,

I had a couple of suggestions for gdb, and was wondering if they had either
been implemented, or were on the 'wish list' to be implemented.


    1) trace mode. I'd like the ability to set a time delay between how often 
       statements are executed, and then have gdb show the statements as they 
       *are* being executed.

       dbx has this, and its a lifesaver. You set 'trace' on, and then hit
       'c' - and you see the program steps and code that get executed realtime.

       To be real useful, you should be able to start the trace, choose whether
       or not you are going to use 'step' mode (ie: show each step as its executed)
       or 'next' mode (ie: skip over subroutines). And you should be able to hit
       a button to stop execution at any given time.


    2) attach mode. I've noticed, especially with testing services through xinetd,
       that you can't always expect to have a gdb session come up visibly. 

       For example, I was testing cvs the other day through valgrind, and it has a 
       --db-command option for firing up a debugger if a memory leak occurs.
       If you are in a shell, this is no big deal. But if the service runs through
       something like valgrind, the gdb debugger gets fired up in a non-interactive
       place. I'd like to have the ability to attach to the gdb command from a window
       and be able to interact with the gdb session from there.


Anyways, #1 I think would be more generally useful, but #2 would be very nice in 
lots of situations too, IMO. Have either of these been implemented?

Ed

Re: tracing, attaching to gdb processes

[Bob Rossi]

On Sun, Mar 05, 2006 at 09:28:32PM -0800, Ed Peschko wrote:
> all,
> 
> I had a couple of suggestions for gdb, and was wondering if they had either
> been implemented, or were on the 'wish list' to be implemented.
> 
> 
>     2) attach mode. I've noticed, especially with testing services through xinetd,
>        that you can't always expect to have a gdb session come up visibly. 
> 
>        For example, I was testing cvs the other day through valgrind, and it has a 
>        --db-command option for firing up a debugger if a memory leak occurs.
>        If you are in a shell, this is no big deal. But if the service runs through
>        something like valgrind, the gdb debugger gets fired up in a non-interactive
>        place. I'd like to have the ability to attach to the gdb command from a window
>        and be able to interact with the gdb session from there.

For this problem, couldn't you simply have valgrind either start GDB in
screen so that you could attach to it when you want, or use a graphical
debugger (xterm -e gdb)?

Bob Rossi

RE: tracing, attaching to gdb processes

[Dave Korn]

On 06 March 2006 11:55, Bob Rossi wrote:

> On Sun, Mar 05, 2006 at 09:28:32PM -0800, Ed Peschko wrote:
>> all,
>> 
>> I had a couple of suggestions for gdb, and was wondering if they had either
>> been implemented, or were on the 'wish list' to be implemented.
>> 
>> 
>>     2) attach mode. I've noticed, especially with testing services through
>>        xinetd, that you can't always expect to have a gdb session come up
>> visibly. 
>> 
>>        For example, I was testing cvs the other day through valgrind, and
>>        it has a --db-command option for firing up a debugger if a memory
>>        leak occurs. If you are in a shell, this is no big deal. But if the
>>        service runs through something like valgrind, the gdb debugger gets
>>        fired up in a non-interactive place. I'd like to have the ability
>>        to attach to the gdb command from a window and be able to interact
>> with the gdb session from there. 
> 
> For this problem, couldn't you simply have valgrind either start GDB in
> screen so that you could attach to it when you want, or use a graphical
> debugger (xterm -e gdb)?
> 
> Bob Rossi

  Or have valgrind attach a gdbserver instance to the faulting process, and
then connect locally to _that_ with gdb?

    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: tracing, attaching to gdb processes

[Ed Peschko]

On Mon, Mar 06, 2006 at 12:01:00PM -0000, Dave Korn wrote:
> On 06 March 2006 11:55, Bob Rossi wrote:
> 
> > On Sun, Mar 05, 2006 at 09:28:32PM -0800, Ed Peschko wrote:
> >> all,
> >> 
> >> I had a couple of suggestions for gdb, and was wondering if they had either
> >> been implemented, or were on the 'wish list' to be implemented.
> >> 
> >> 
> >>     2) attach mode. I've noticed, especially with testing services through
> >>        xinetd, that you can't always expect to have a gdb session come up
> >> visibly. 
> >> 
> >>        For example, I was testing cvs the other day through valgrind, and
> >>        it has a --db-command option for firing up a debugger if a memory
> >>        leak occurs. If you are in a shell, this is no big deal. But if the
> >>        service runs through something like valgrind, the gdb debugger gets
> >>        fired up in a non-interactive place. I'd like to have the ability
> >>        to attach to the gdb command from a window and be able to interact
> >> with the gdb session from there. 
> > 
> > For this problem, couldn't you simply have valgrind either start GDB in
> > screen so that you could attach to it when you want, or use a graphical
> > debugger (xterm -e gdb)?
> > 
> > Bob Rossi
> 
>   Or have valgrind attach a gdbserver instance to the faulting process, and
> then connect locally to _that_ with gdb?
> 
>     cheers,
>       DaveK
> -- 

well, the X11 solution was the one I thought of first, but I don't always have a X11 
connection (I do most of my stuff over screen), and two, since valgrind is run under 
inetd, its not in an interactive terminal, so I would be very interested to know 
if I *could* redirect the gdb session to a screen, and if so, how.

WRT gdbserver, I'm not sure - I need to start the gdb with the format 

    gdb -nw %f %p,

ie: be able to write to the executable or core file, and to be able to attach to the
process. I guess I also need to stop the gdbserver each time I do this, the gdb through
valgrind is a blocking call..

Anyways, thanks for the suggestions. I'm assuming that the 'trace' mode, on the other
hand does not exist?

Ed

Re: tracing, attaching to gdb processes

[Daniel Jacobowitz]

On Sun, Mar 05, 2006 at 09:28:32PM -0800, Ed Peschko wrote:
>     1) trace mode. I'd like the ability to set a time delay between how often 
>        statements are executed, and then have gdb show the statements as they 
>        *are* being executed.
> 
>        dbx has this, and its a lifesaver. You set 'trace' on, and then hit
>        'c' - and you see the program steps and code that get executed realtime.
> 
>        To be real useful, you should be able to start the trace, choose whether
>        or not you are going to use 'step' mode (ie: show each step as its executed)
>        or 'next' mode (ie: skip over subroutines). And you should be able to hit
>        a button to stop execution at any given time.

In general, you can get this by having a loop that does "while 1; step;
shell sleep 1" and hitting Control-C to stop it.  Anything more
complicated than that involves the "hitting a button to stop" concept,
which means a GUI should be issuing the step commands - that's easy
to do when you're driving GDB.

>     2) attach mode. I've noticed, especially with testing services through xinetd,
>        that you can't always expect to have a gdb session come up visibly. 
> 
>        For example, I was testing cvs the other day through valgrind, and it has a 
>        --db-command option for firing up a debugger if a memory leak occurs.
>        If you are in a shell, this is no big deal. But if the service runs through
>        something like valgrind, the gdb debugger gets fired up in a non-interactive
>        place. I'd like to have the ability to attach to the gdb command from a window
>        and be able to interact with the gdb session from there.

Of course GDB supports attaching to processes.  It's just valgrind
that's a problem child, since the way valgrind starts up a GDB to talk
to is so strange.  For everything else, you just stop the affected
process when you want to debug it, and gdb FILE PID to attach at your
leisure.  Having valgrind create a screen session instead of a gdb
is probably your easiest solution.

-- 
Daniel Jacobowitz
CodeSourcery

Re: tracing, attaching to gdb processes

[Ed Peschko]

On Mon, Mar 13, 2006 at 09:28:09PM -0500, Daniel Jacobowitz wrote:
> On Sun, Mar 05, 2006 at 09:28:32PM -0800, Ed Peschko wrote:
> >     1) trace mode. I'd like the ability to set a time delay between how often 
> >        statements are executed, and then have gdb show the statements as they 
> >        *are* being executed.
> > 
> >        dbx has this, and its a lifesaver. You set 'trace' on, and then hit
> >        'c' - and you see the program steps and code that get executed realtime.
> > 
> >        To be real useful, you should be able to start the trace, choose whether
> >        or not you are going to use 'step' mode (ie: show each step as its executed)
> >        or 'next' mode (ie: skip over subroutines). And you should be able to hit
> >        a button to stop execution at any given time.
> 
> In general, you can get this by having a loop that does "while 1; step;
> shell sleep 1" and hitting Control-C to stop it.  Anything more
> complicated than that involves the "hitting a button to stop" concept,
> which means a GUI should be issuing the step commands - that's easy
> to do when you're driving GDB.

I don't understand..

Are you suggesting that I make an expect script to control gdb, and then have
that control script pass characters to the underlying process? Or something
more exotic?

And why can't this be built into gdb in the form of a 'set' variable?  Either that,
or a built-in high-resolution 'sleep' call that doesn't require spawning a shell?

Or best yet, a builtin non-blocking read call that waits for a person's input?


> >     2) attach mode. I've noticed, especially with testing services through xinetd,
> >        that you can't always expect to have a gdb session come up visibly. 
> > 
> >        For example, I was testing cvs the other day through valgrind, and it has a 
> >        --db-command option for firing up a debugger if a memory leak occurs.
> >        If you are in a shell, this is no big deal. But if the service runs through
> >        something like valgrind, the gdb debugger gets fired up in a non-interactive
> >        place. I'd like to have the ability to attach to the gdb command from a window
> >        and be able to interact with the gdb session from there.
> 
> Of course GDB supports attaching to processes.  It's just valgrind
> that's a problem child, since the way valgrind starts up a GDB to talk
> to is so strange.  For everything else, you just stop the affected
> process when you want to debug it, and gdb FILE PID to attach at your
> leisure.  Having valgrind create a screen session instead of a gdb
> is probably your easiest solution.

well, I was thinking of something like gdbserver (which I wasn't aware existed) 
but now I see what people were talking about when they were saying 'use screen
with your gdb processes' - I wasn't aware that screen could be started already
detached, nor that you could run an alternate shell with screen (eg: 'gdb').

Anyways, maybe this warrants an entry in the manual - its not only valgrind that 
could benefit from this workflow, but other processes, especially those run
through inetd. 

It would be very cool, for example, if you could somehow trigger gdb to run with
an instantiated call, say:

/* my code here */
spawn_gdb()
/* my code here */

and then a gdb would be spawned automatically and attached to the process at the
point right following the spawn_gdb call. 

I know that people could probably cook something up like this on their own, 
but IMO it makes a lot of sense to be distributed with gdb - you could simply 
use the above function and link with -lgdb to get the functionality wanted. And 
it really would revolutionize the way that daemon processes get debugged, 
letting them be debugged in-situ. (no more 'CVS_SERVER_SLEEP' for example, 
to cause the cvs server to sleep for x seconds so you can whip out 
'ps -ef | grep cvs', find the child process and attach to it before time runs out. )

Ed

Re: tracing, attaching to gdb processes

[Daniel Jacobowitz]

On Tue, Mar 14, 2006 at 07:04:37PM -0800, Ed Peschko wrote:
> > In general, you can get this by having a loop that does "while 1; step;
> > shell sleep 1" and hitting Control-C to stop it.  Anything more
> > complicated than that involves the "hitting a button to stop" concept,
> > which means a GUI should be issuing the step commands - that's easy
> > to do when you're driving GDB.
> 
> I don't understand..
> 
> Are you suggesting that I make an expect script to control gdb, and then have
> that control script pass characters to the underlying process? Or something
> more exotic?

The GDB CLI scripting language has "while".  See the manual.

It won't work very well though.  The C-c will stop the sleep, but not
the while.

> And why can't this be built into gdb in the form of a 'set' variable?  Either that,
> or a built-in high-resolution 'sleep' call that doesn't require spawning a shell?

You asked if GDB could do what you wanted; this is how it can.  I
don't know whether it should have a built in command to do this or not.

> Or best yet, a builtin non-blocking read call that waits for a person's input?

I fail to see how this relates to what you wanted.

> well, I was thinking of something like gdbserver (which I wasn't aware existed) 
> but now I see what people were talking about when they were saying 'use screen
> with your gdb processes' - I wasn't aware that screen could be started already
> detached, nor that you could run an alternate shell with screen (eg: 'gdb').
> 
> Anyways, maybe this warrants an entry in the manual - its not only valgrind that 
> could benefit from this workflow, but other processes, especially those run
> through inetd. 

Maybe.

> It would be very cool, for example, if you could somehow trigger gdb to run with
> an instantiated call, say:
> 
> /* my code here */
> spawn_gdb()
> /* my code here */
> 
> and then a gdb would be spawned automatically and attached to the process at the
> point right following the spawn_gdb call. 

GDK/GTK already have code to do this.  So do lots of other libraries. 
It's not something that needs to be distributed with GDB; it's just
  if (fork () == 0)
    execlp ("gdb", argv[0], getppid (), NULL);

-- 
Daniel Jacobowitz
CodeSourcery

Re: tracing, attaching to gdb processes

[Ed Peschko]

> > I don't understand..
> > 
> > Are you suggesting that I make an expect script to control gdb, and then have
> > that control script pass characters to the underlying process? Or something
> > more exotic?
> 
> The GDB CLI scripting language has "while".  See the manual.
> 
> It won't work very well though.  The C-c will stop the sleep, but not
> the while.

exactly...

> > And why can't this be built into gdb in the form of a 'set' variable?  Either that,
> > or a built-in high-resolution 'sleep' call that doesn't require spawning a shell?
> 
> You asked if GDB could do what you wanted; this is how it can.  I
> don't know whether it should have a built in command to do this or not.
> 
> > Or best yet, a builtin non-blocking read call that waits for a person's input?
> 
> I fail to see how this relates to what you wanted.

while 1;
> set end_on_input
> step;
> gdb_sleep .1

'set stop_on_input' would make it so that the user could interrupt the script with
the press of a key via a non-blocking read.. ie: exactly what I would want. 
gdb_sleep would be a high res sleep call that slept fractional amounts of a second.

Also, I note that when you do do something like this, you get a 'press <return> to
coninue, q<return> to quit'. It would be nice if there was a way to override this.


> > It would be very cool, for example, if you could somehow trigger gdb to run with
> > an instantiated call, say:
> > 
> > /* my code here */
> > spawn_gdb()
> > /* my code here */
> > 
> > and then a gdb would be spawned automatically and attached to the process at the
> > point right following the spawn_gdb call. 
> 
> GDK/GTK already have code to do this.  So do lots of other libraries. 
> It's not something that needs to be distributed with GDB; it's just
>   if (fork () == 0)
>     execlp ("gdb", argv[0], getppid (), NULL);

well, cool.. then simply add that as a hook inside the library. Which would 
you rather write (continuously), the first one, or the second? And isn't it useful
to have it just for suggesting ideas to end users of gdb?

Anyways, if this isn't the place to request gdb features, what *is* the correct
place?

Ed

Re: tracing, attaching to gdb processes

[Daniel Jacobowitz]

On Tue, Mar 14, 2006 at 08:16:18PM -0800, Ed Peschko wrote:
> Also, I note that when you do do something like this, you get a 'press <return> to
> coninue, q<return> to quit'. It would be nice if there was a way to override this.

"set height 0"

Stopping on input is a bit more complicated than you think.  Whenever
the program is running, it has control of the terminal - and input goes
to it, not to us.  I don't see an easy way to do this.

> well, cool.. then simply add that as a hook inside the library. Which would 
> you rather write (continuously), the first one, or the second? And isn't it useful
> to have it just for suggesting ideas to end users of gdb?

That says to me that it belongs in the manual.  Shipping a library is a
serious pain; why should we do it for a two-line function?

> Anyways, if this isn't the place to request gdb features, what *is* the correct
> place?

It's the right place; I'm just not sure I agree with your suggestions.
If someone else does, they're welcome to submit a patch :-)

-- 
Daniel Jacobowitz
CodeSourcery

Re: tracing, attaching to gdb processes

[Ed Peschko]

On Wed, Mar 15, 2006 at 09:22:31AM -0500, Daniel Jacobowitz wrote:
> On Tue, Mar 14, 2006 at 08:16:18PM -0800, Ed Peschko wrote:
> > Also, I note that when you do do something like this, you get a 'press <return> to
> > coninue, q<return> to quit'. It would be nice if there was a way to override this.
> 
> "set height 0"
> 
> Stopping on input is a bit more complicated than you think.  Whenever
> the program is running, it has control of the terminal - and input goes
> to it, not to us.  I don't see an easy way to do this.

The way to do it is thorugh something similar to ReadKey - For example, the 
following perl script does sort of what I'd want:

use Term::ReadKey;

ReadMode('cbreak');

while (1)
{
	# do stuff.
	if (defined($char = ReadKey(-1)))
	{
		last;	
	}
	system("ls");
	sleep(1);
}

At every point the program hits the ReadKey call, it checks to see if there is input to
process, and if so, drops out.  If there is nothing to read, it continues. So you basically
will get a bunch of 'lses' until you hit anything, and then the program will stop.

Term::ReadKey is implemented with its own handwritten c, but I can't imagine that 
readline wouldn't have something similar.


> > cool.. then simply add that as a hook inside the library. Which would 
> > you rather write (continuously), the first one, or the second? And isn't it useful
> > to have it just for suggesting ideas to end users of gdb?
> 
> That says to me that it belongs in the manual.  Shipping a library is a
> serious pain; why should we do it for a two-line function?

Right now, I look inside of the gdb folder inside of the build directory for
gdb-6.3, and I see that make creates a:

	libgdb.a

file. I'm assuming it wouldn't be that difficult to add there. 

Another thing that might be useful to add there is an 'automatic breakpoint' 
function. Perl has '$DB::single', ie: you set the value $DB::single = 1, the 
debugger automatically stops. Similarly, it would be nice to have a 

gdb_stop();

function, which you could use to pepper your code with useful breakpoints. 

As far as patching goes, I'd be willing to write some of this stuff, if I got a pointer
to say, how you do a non-blocking read with readline. And maybe an inkling if 
someone would be willing to include it in the 'standard distribution' (I'm not too
hot on needing to maintain a local patch)


Ed

Re: tracing, attaching to gdb processes

[Daniel Jacobowitz]

On Wed, Mar 15, 2006 at 03:18:46PM -0800, Ed Peschko wrote:
> On Wed, Mar 15, 2006 at 09:22:31AM -0500, Daniel Jacobowitz wrote:
> > Stopping on input is a bit more complicated than you think.  Whenever
> > the program is running, it has control of the terminal - and input goes
> > to it, not to us.  I don't see an easy way to do this.
> 
> The way to do it is thorugh something similar to ReadKey - For example, the 
> following perl script does sort of what I'd want:

No, sorry, you did not understand my explanation of the problem.  The
program being run is usually on the same terminal as GDB.  If it reads, it
will consume the input.

> Right now, I look inside of the gdb folder inside of the build directory for
> gdb-6.3, and I see that make creates a:
> 
> 	libgdb.a
> 
> file. I'm assuming it wouldn't be that difficult to add there. 

No; that library is only used in the creation of GDB, it should never
be linked to anything but GDB.  It's just a convenience during the
build process.  Once upon a time there were experiments with linking
things to GDB directly; eventually we decided that controlling it
through a pipe was a better choice.

> Another thing that might be useful to add there is an 'automatic breakpoint' 
> function. Perl has '$DB::single', ie: you set the value $DB::single = 1, the 
> debugger automatically stops. Similarly, it would be nice to have a 
> 
> gdb_stop();
> 
> function, which you could use to pepper your code with useful breakpoints. 

Feel free:

void gdb_stop (void) { }

(gdb) break gdb_stop

Or put it in a .gdbinit file that you ship with the application.

-- 
Daniel Jacobowitz
CodeSourcery

Re: tracing, attaching to gdb processes

[Ed Peschko]

On Wed, Mar 15, 2006 at 06:37:39PM -0500, Daniel Jacobowitz wrote:
> On Wed, Mar 15, 2006 at 03:18:46PM -0800, Ed Peschko wrote:
> > On Wed, Mar 15, 2006 at 09:22:31AM -0500, Daniel Jacobowitz wrote:
> > > Stopping on input is a bit more complicated than you think.  Whenever
> > > the program is running, it has control of the terminal - and input goes
> > > to it, not to us.  I don't see an easy way to do this.
> > 
> > The way to do it is thorugh something similar to ReadKey - For example, the 
> > following perl script does sort of what I'd want:
> 
> No, sorry, you did not understand my explanation of the problem.  The
> program being run is usually on the same terminal as GDB.  If it reads, it
> will consume the input.

Ok, that's fine, I get what you are saying now.

But its really punishing 99% of the programs out there for the requirements of 
the remaining 1%. Either they ignore stdin altogether, or take a bunch of 
stdin input and go into 'processing' mode, after which they don't need to touch
stdin.

In any there could be a variable (call it set no_block_read) which toggles
it so that the user sends input to gdb rather than the underlying program.

Then the user would type:

(gdb) set no_block_read=qq
(gdb) while 1; 
> step
> db_sleep .1 
> end

and now if users input the string 'qq' it would cause the label to stop.

In fact you could be really fancy about it - namely, if a process *was*
expecting stdin, that the no_block_read would continue until it got 
input from stdin, which it would then compare against whatever was 
listed in 'no_block_read'. Example:

	.... strings of input ... *user presses 'qq'.
	... strings of input  ... gets fed to program, qq stops loop, and 
                                  the input qq gets discarded.

Of course this slows down the feeding of input, but it wouldn't be the default
so that would be fine.. And I'd think it would be relatively easy to implement 
because you have to toggle between input streams anyways so people can 
enter commands to gdb.

And finally IMO the current behaviour is broken. As you yourself stated,
having an infinite loop in scripting is basically useless.

> No; that library is only used in the creation of GDB, it should never
> be linked to anything but GDB.  It's just a convenience during the
> build process.  Once upon a time there were experiments with linking
> things to GDB directly; eventually we decided that controlling it
> through a pipe was a better choice.

I'd put it the docs then.

> Feel free:
> 
> void gdb_stop (void) { }
> 
> (gdb) break gdb_stop
> 
> Or put it in a .gdbinit file that you ship with the application.

Or a hook could be put into gdb which automatically inserts a breakpoint 
on the function name 'gdb_stop' if it exists.

I mean, think of what happens if gdb_stop is defined in a place which is 
dynamically loaded? Then the users' setting of the breakpoint will fail.
Having gdb check upon loading a symbol for the gdb_stop function makes 
it much easier.

And even if the functionality for delayed breakpoints is added to gdb, it still
is forcing the user to reimplement something for each project he/she works
on..

Ed

Re: tracing, attaching to gdb processes

[Daniel Jacobowitz]

On Wed, Mar 15, 2006 at 07:00:13PM -0800, Ed Peschko wrote:
> In any there could be a variable (call it set no_block_read) which toggles
> it so that the user sends input to gdb rather than the underlying program.

You might want to play around with Unix process groups and shared
stdin; it doesn't work quite like you think it does.

I'm sure there's some way to do this, but I don't know how.

> > Feel free:
> > 
> > void gdb_stop (void) { }
> > 
> > (gdb) break gdb_stop
> > 
> > Or put it in a .gdbinit file that you ship with the application.
> 
> Or a hook could be put into gdb which automatically inserts a breakpoint 
> on the function name 'gdb_stop' if it exists.

"break gdb_stop" at the end of your .gdbinit file will do this.

> I mean, think of what happens if gdb_stop is defined in a place which is 
> dynamically loaded? Then the users' setting of the breakpoint will fail.
> Having gdb check upon loading a symbol for the gdb_stop function makes 
> it much easier.
> 
> And even if the functionality for delayed breakpoints is added to gdb, it still
> is forcing the user to reimplement something for each project he/she works
> on..

GDB already has delayed breakpoints (the term used is "pending").  And
it doesn't require any reimplementation if you put it in
$HOME/.gdbinit.

You might want to spend some time with the existing manual.  It's very
complete, and it seems like you don't realize a lot of the things GDB
is already capable of.

You've definitely suggested a couple of things that the manual doesn't
make clear, and I'm saving your messages to add to the manual as
suggestions at some point.

-- 
Daniel Jacobowitz
CodeSourcery