multi-{inferior,exec}

multi-{inferior,exec}

[Tom Tromey]

I've been looking at multi-inferior/multi-exec the last few days.  I
want to polish a few edges and make it work nicely on Linux.  I thought
I'd post about what I think I am going to implement.

First, the original spec:

    http://sourceware.org/ml/gdb/2008-08/msg00169.html

Also relevant:

    http://sourceware.org/gdb/wiki/MultiProcess


One concrete thing I plan to do is make objfiles not be relocated,
basically following the idea here:

    http://sourceware.org/ml/gdb-patches/2009-06/msg00668.html

This is mostly plumbing, but important for efficiency.


I think the next missing bit is support for I/T sets.  E.g., I wanted to
try to debug g++ but put a breakpoint in its cc1plus subprocess.  You
can do this today, but only clumsily: either by using add-inferior to
load the symbols for cc1plus, or by using "catch exec" and setting the
breakpoint at exec time.  I/T sets would let users set this sort of
breakpoint more easily.

I am not sure what is required to integrate I/T sets into inferior
control (something mentioned on the wiki page).  If anyone (Pedro?
Stan?) has state to share, I'm happy to hear it.

I think we can add an I/T set specifier before the linespec argument to
"break".  (I couldn't think of other commands that use linespecs that
would need this...)  E.g., in the g++ scenario, this would set a
breakpoint that would be resolved in libcpp:

    (gdb) break [*] collect_args

"[*]" is the I/T set representing "all current or future inferiors".

If no I/T set is specified, it would default to the current inferior.

The original spec had a "focus" command... but that is already used by
the TUI.  How did that get resolved?  It seems to me that we could just
use "select" instead.

I was thinking it would be nice to extend "thread apply" to take an I/T
set argument.  Maybe "inferior apply" is a nicer way to spell it.

I think maybe the I/T set syntax could be expanded or changed a little.
I'm at least thinking of "." as a synonym for "current inferior", and
being able to differentiate named sets and executables.


There are more tasks beyond those, but I think those will hold me for a
while.  The remaining big things I am aware of (thanks Pedro) are:

* Model follow-fork|vfork|exec in remote protocol + remote.c + gdbserver

* gdbserver needs work for multi-inferior + multi-arch
  Maybe we need a target description per inferior here.

Tom

Re: multi-{inferior,exec}

[Jan Kratochvil]

On Sat, 04 Sep 2010 00:40:53 +0200, Tom Tromey wrote:
> * gdbserver needs work for multi-inferior + multi-arch
>   Maybe we need a target description per inferior here.

Probably clear but it should be made with plans for the next support for
multiple remote hosts.  That is GDB connecting to multiple gdbservers.  Or as
was suggested (Pedro?) to have one gdb <-> multiplexing gdbserver <-> multiple
hosts' gdbservers.  Unaware target_gdbarch should be gdbserver protocol vs.
inferior arch related etc.:
	[01/03] Update target_gdbarch for current inferior
	http://sourceware.org/ml/gdb-patches/2010-01/msg00228.html


Thanks,
Jan

Re: multi-{inferior,exec}

[Nenad Vukicevic]

> I think the next missing bit is support for I/T sets.  E.g., I wanted to
> try to debug g++ but put a breakpoint in its cc1plus subprocess.  You
> can do this today, but only clumsily: either by using add-inferior to
> load the symbols for cc1plus, or by using "catch exec" and setting the
> breakpoint at exec time.  I/T sets would let users set this sort of
> breakpoint more easily.

I think that I/T sets is a very useful addition. Recently we've made 
some changes for debugging of the UPC
- http://www.gnu.org/software/gcc/projects/gupc.html - programs. 
However, as our run-time contains
a thread that is not UPC language related, I made some changes and 
created a "upcmode"
of operation where commands like breakpoint, info threads, .. apply only 
to UPC threads (a set in
your example). Also, "c -a" like commands apply only to UPC threads. 
Same with "thread apply all".
In essence you are focusing your debugging to a group of threads. 
Switching off "upcmode" makes
everything go back to normal (in your case selecting a thread set "all" 
turns off this feature).

Also, having set names is better then numbers (e.g. "upc" set of threads 
for UPC program set) and using
names or numbers should be interchangeable.

We also had to implement "collective" breakpoints where breakpoint is 
announced to user only
once all the threads reached it. In "upcmode" every breakpoint is a 
collective one, unless you
explicitly turn this feature off.

Nenad

Re: multi-{inferior,exec}

[Tom Tromey]

>>>>> "Nenad" == Nenad Vukicevic <nenad@intrepid.com> writes:

Nenad> However, as our run-time contains a thread that is not UPC
Nenad> language related, I made some changes and created a "upcmode" of
Nenad> operation where commands like breakpoint, info threads, .. apply
Nenad> only to UPC threads (a set in your example). Also, "c -a" like
Nenad> commands apply only to UPC threads. Same with "thread apply all".

It sounds like you made the needed inferior-control changes ... any
chance of a patch for this?

I think instead of a special "upcmode", though, just having generic I/T
sets would be better.  UPC could be one instance of this.

Nenad> Also, having set names is better then numbers (e.g. "upc" set of
Nenad> threads for UPC program set) and using
Nenad> names or numbers should be interchangeable.

Yes.  I'm also thinking that we should support implementing named sets
in Python.  That way a set could be very dynamic, when needed.

Nenad> We also had to implement "collective" breakpoints where
Nenad> breakpoint is announced to user only once all the threads reached
Nenad> it. In "upcmode" every breakpoint is a collective one, unless you
Nenad> explicitly turn this feature off.

This sounds like the "barrier" feature in HPD.
A patch for this would be interesting, too.

I am curious though -- what is it useful for?

Tom

Re: multi-{inferior,exec}

[Daniel Jacobowitz]

On Tue, Sep 07, 2010 at 11:03:48AM -0600, Tom Tromey wrote:
> This sounds like the "barrier" feature in HPD.
> A patch for this would be interesting, too.
> 
> I am curious though -- what is it useful for?

Think about this in terms of, for example, automatically parallelized
OpenMP code.  If you have a parallel region, it's handy to examine all
threads at that point.  I don't know if this would work better in UPC
than it would in OpenMP though; you can have code executed in only
some threads...

-- 
Daniel Jacobowitz
CodeSourcery

Re: multi-{inferior,exec}

[Nenad Vukicevic]

  On 9/7/2010 2:34 PM, Daniel Jacobowitz wrote:
> On Tue, Sep 07, 2010 at 11:03:48AM -0600, Tom Tromey wrote:
>> This sounds like the "barrier" feature in HPD.
>> A patch for this would be interesting, too.
>>
>> I am curious though -- what is it useful for?
> Think about this in terms of, for example, automatically parallelized
> OpenMP code.  If you have a parallel region, it's handy to examine all
> threads at that point.  I don't know if this would work better in UPC
> than it would in OpenMP though; you can have code executed in only
> some threads...
>

Also, think about 32 or more threads printing breakpoint announcments. 
:) As all
UPC threads execute the same code, but work on different sets of data,
you look at all of them as only one program.

UPC has a "upc_barrier" statement that all threads need to arrive before
going forward. At that point your shared space is consistent and it is a 
natural
place to stop.

Nenad

Re: multi-{inferior,exec}

[Tom Tromey]

Tom> I think the next missing bit is support for I/T sets.

Some updates on this...

Tom> I think we can add an I/T set specifier before the linespec
Tom> argument to "break".  (I couldn't think of other commands that use
Tom> linespecs that would need this...)

It seems to make sense for the other "catch" commands, though those
don't all use linespecs.  Maybe it still makes sense to have the I/T set
as an argument to the command (as opposed to a prefix or a "select"
command), maybe not.  Your comments appreciated.

Tom>     (gdb) break [*] collect_args

Pedro pointed out that this could have an MI problem.  Currently, a
breakpoint is defined as having a single source location (as opposed to
address locations) -- but this kind of breakpoint might have multiple
locations, one per inferior.

I was curious to see what we emit when a breakpoint has multiple
addresses.  Here's the CLI output from a gdb test case:

(gdb) info b
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   <MULTIPLE> 
1.1                         y     0x08048447 in foo(int) at ../../../archer/gdb/testsuite/gdb.cp/mb-inline.h:26
1.2                         y     0x080484a7 in foo(int) at ../../../archer/gdb/testsuite/gdb.cp/mb-inline.h:26



Here is the relevant MI:

bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="<MULTIPLE>",times="0",original-location="mb-inline.h:26"}

This seems a little weird to me in a couple of ways.

First, addr="<MULTIPLE>" is strange, in that it seems to me that an
attribute would be friendlier.  This text is also undocumented ... but I
presume that in practice MI clients must interpret this?  (I further
suppose the same applies to <PENDING>".)

Second, the various locations are not represented at all.

Third, compare with an ordinary breakpoint:

bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr="0x08048487",func="main()",file="../../../archer/gdb/testsuite/gdb.cp/mb-inline1.cc",fullname="/home/tromey/gnu/archer/archer/gdb/testsuite/gdb.cp/mb-inline1.cc",line="32",times="0",original-location="main"}

Here we get func, file, fullname, and line -- but with a <MULTIPLE>
breakpoint, we do not.


So, since the current situation doesn't seem all that friendly to begin
with, maybe extending it a little for this new use won't be so hard.

Also, it occurred to me that an MI client would have to explicitly
request such a breakpoint with an I/T set -- so maybe it is even ok to
emit something different automatically.


One partly-compatible idea would be to treat each location as a separate
breakpoint, for the purposes of MI output.  (This isn't totally
compatible because some options are per-breakpoint and not
per-location.)

Then, we could provide a new option (and corresponding token in the
-list-features output) to enable "real" output.  The real form would
show all the sub-locations of a master breakpoint.

This approach would also let us fix PR 11657 in a straightforward way.


Pedro and I talked a bit today about some specific use cases.

First, it seems most likely that a front end will typically request a
file:line breakpoint using an absolute path.  That is, the user sees
some source code and clicks on it.

Here, the FE would send the MI equivalent of:

  break [*] file:line

This will do the right thing: break in any inferior reaching that source
line.

Another use case would be putting a breakpoint on a particular function,
say from some kind of symbol view.  After some reflection, I think the
solution in this case lies in a linespec extension, not in I/T sets.  In
particular:

  break [*] #libsomething.so#function


I also looked into another idea we discussed -- having a kind of
meta-breakpoint for I/T sets that sets new concrete breakpoints when the
appropriate events occur.

This means associating new metadata with a new kind of breakpoint and
coming up with a new command name.  This seems like more work, for a
result that is not as nice to use.

Tom

Re: multi-{inferior,exec}

[Stan Shebs]

Tom Tromey wrote:
> I've been looking at multi-inferior/multi-exec the last few days.  I
> want to polish a few edges and make it work nicely on Linux.  I thought
> I'd post about what I think I am going to implement.
>
> [...]
>
> I think the next missing bit is support for I/T sets.  E.g., I wanted to
> try to debug g++ but put a breakpoint in its cc1plus subprocess.  You
> can do this today, but only clumsily: either by using add-inferior to
> load the symbols for cc1plus, or by using "catch exec" and setting the
> breakpoint at exec time.  I/T sets would let users set this sort of
> breakpoint more easily.
>   

I/T sets are one of the bits that didn't make it from 
multiprocess-20081120-branch to the trunk.  You can look at the branch's 
inferior.h and see the internal definition, which basically consists of 
a string that is the specification and a possibly-dynamic vector of 
inferiors computed from it.

> I am not sure what is required to integrate I/T sets into inferior
> control (something mentioned on the wiki page).  If anyone (Pedro?
> Stan?) has state to share, I'm happy to hear it.
>   

Much of it is a matter of deciding what to iterate over, vs limiting to 
a single current inferior.  For instance, using the prefix syntax I 
originally suggested, "[4-6] print x" should print three lines, add 
three elements to value history, etc.  So one could imagine a generic 
i/t set iteration meta-command, but it's not clear that iteration is 
always the right thing to do, and maybe individual commands should 
somehow whether they are sensible for iteration.

> I think we can add an I/T set specifier before the linespec argument to
> "break".  (I couldn't think of other commands that use linespecs that
> would need this...)  E.g., in the g++ scenario, this would set a
> breakpoint that would be resolved in libcpp:
>
>     (gdb) break [*] collect_args
>
> "[*]" is the I/T set representing "all current or future inferiors".
>   

In my global breakpoint project ( 
http://sourceware.org/ml/gdb/2010-05/msg00019.html ), I've been using a 
"process" keyword where "thread" and "task" are used now.  It doesn't 
win elegance points, but I think it corresponds better to how the user 
thinks - "OK, I'm breaking on collect_args, and oh yeah, I want to catch 
it in any process, not just the one I have now".  I don't think it would 
hurt to support a couple syntax variations, there is not going to be 
much prior expectation to accommodate.

> If no I/T set is specified, it would default to the current inferior.
>
> The original spec had a "focus" command... but that is already used by
> the TUI.  How did that get resolved?  It seems to me that we could just
> use "select" instead.
>   

I changed TUI to use "ffocus", on the theory that there aren't many TUI 
users.  "select" isn't great because completion is ambiguous with 
"select-frame".
> I was thinking it would be nice to extend "thread apply" to take an I/T
> set argument.  Maybe "inferior apply" is a nicer way to spell it.
>   

One advantage to prefix is that the '[' introduces the iteration desire 
succinctly.

> I think maybe the I/T set syntax could be expanded or changed a little.
> I'm at least thinking of "." as a synonym for "current inferior", and
> being able to differentiate named sets and executables.
>   

I added a few things beyond the original spec, but it's been so long I 
don't remember what. :-)

Stan

Re: multi-{inferior,exec}

[Stan Shebs]

Tom Tromey wrote:
> Another use case would be putting a breakpoint on a particular function,
> say from some kind of symbol view.  After some reflection, I think the
> solution in this case lies in a linespec extension, not in I/T sets.  In
> particular:
>
>   break [*] #libsomething.so#function
>   

The '#' syntax is in the old spec actually, you even commented about it 
in http://sourceware.org/ml/gdb/2008-08/msg00177.html :-)

>
> I also looked into another idea we discussed -- having a kind of
> meta-breakpoint for I/T sets that sets new concrete breakpoints when the
> appropriate events occur.
>
> This means associating new metadata with a new kind of breakpoint and
> coming up with a new command name.  This seems like more work, for a
> result that is not as nice to use.
>   

What I've been doing for now for global breakpoints is to add the 
quasi-i/t set as another attribute of a breakpoint, a la thread or 
cond_string.  This then is both interpreted by GDB, for processes that 
are currently under GDB's control, and passed to a kernel module that is 
monitoring all other processes for hits.

Stan

Re: multi-{inferior,exec}

[Tom Tromey]

>>>>> "Stan" == Stan Shebs <stan@codesourcery.com> writes:

Stan> The '#' syntax is in the old spec actually, you even commented about
Stan> it in http://sourceware.org/ml/gdb/2008-08/msg00177.html :-)

Yeah.  More or less I am working through rediscovering the reasons all
those things were in the spec in the first place.

From the global breakpoint message, it looks like you are planning to
tackle this in some way.  I forgot about this part of your project,
sorry about that.

Tom> This means associating new metadata with a new kind of breakpoint and
Tom> coming up with a new command name.  This seems like more work, for a
Tom> result that is not as nice to use.

Stan> What I've been doing for now for global breakpoints is to add the
Stan> quasi-i/t set as another attribute of a breakpoint, a la thread or
Stan> cond_string.  This then is both interpreted by GDB, for processes that
Stan> are currently under GDB's control, and passed to a kernel module that
Stan> is monitoring all other processes for hits.

My initial patch was to replace the pspace field in breakpoint with an
I/T set.  This works fine, the problem is that it breaks the MI
invariant of one source location per breakpoint.  I think your approach
probably won't have this problem, because (I assume) your global
breakpoints are inherently objfile-specific.

What kinds of specifications does the kernel module accept?
If the kernel accepts something more limited than gdb, we could always
just upload an approximation to an I/T set and let gdb filter out the
ones it doesn't care about.


FWIW, my current use case is very simple: start gdb on gcc, set a
breakpoint that will be resolved in cc1, run.  I think this is a pretty
reasonable thing to want to do; for me it seems ok to require a little
extra work (like adding "[*]" to the break command), but not a lot of
extra work (like looking up the full path to cc1 to give to
add-inferior).

Maybe it would be ok for "break collect_args" to mean "break on the
first instance of collect_args that is resolved anywhere, but now
convert this to a breakpoint that only resolves to this particular
function in this objfile.  I don't know... it doesn't feel completely
natural to me, but that isn't a very strong reason for anybody else to
do anything :)

Stan> I/T sets are one of the bits that didn't make it from
Stan> multiprocess-20081120-branch to the trunk.

I did look at this, but I ended up writing my own.  You can see it here:

http://sourceware.org/git/gitweb.cgi?p=archer.git;a=blob_plain;f=gdb/itset.h;hb=archer-tromey-multi-inferior
http://sourceware.org/git/gitweb.cgi?p=archer.git;a=blob_plain;f=gdb/itset.c;hb=archer-tromey-multi-inferior

Stan> In my global breakpoint project
Stan> (http://sourceware.org/ml/gdb/2010-05/msg00019.html ), I've been
Stan> using a "process" keyword where "thread" and "task" are used now.

My first reaction to these kinds of extensions is caution, because they
can introduce expression ambiguity.  (I've sometimes considered adding
";" as an expression terminator to help with this...)

Could you expand more on what you are planning?  E.g., exactly what
syntax you intend to support, how users will specify sets of processes,
etc?  I read your post and I see the PID range thing... is that the only
way?  That isn't ambiguous, but putting an I/T set here would be.
Unless we changed the I/T set syntax (using {} would work).

I guess I don't mind if we have multiple syntaxes, though it seems like
if the I/T set syntax is flexible enough and simple enough, we could
just use it everywhere.


Maybe I should just wait until you and/or the UPC guys have everything
finished :-).  It sounds like we have substantial overlap.  There's also
a lot of overlap between your work and the ugdb project... we'll have
some interesting discussions at the GCC Summit this year :-)

Tom