Toward multicore GDB - Set theory

Toward multicore GDB - Set theory

[Stan Shebs]

In a program with hundreds or even thousands of threads, the user cannot 
be effective with the two options that GDB currently provides - work 
with one thread, or all of them at once.

This was anticipated some years ago in the discussions of the HPD forum 
that Michael Snyder and I participated in.  The resulting spec 
introduced the concept of the process/thread set (see 
http://sourceware.org/frysk/documentation/hpd.html), which was 
implemented in the TotalView debugger 
(http://www.roguewave.com/support/product-documentation/totalview-family.aspx#totalview 
for the user manual).  We extend sets to include cores, about which more 
in a moment.

The basic idea is simple; a p/t set is just a set of processes and 
threads.  We introduce a notation that allows us to construct sets, 
where <process>.<thread> refers to a particular thread of a process, and 
the two components can be elided, wildcarded, etc.  So we would have 
things like [4523,16493] for a set of two processes, [4523.workerbee] 
for a particular named thread, [4523.1-7,*.8-15] for eight threads of 
one process plus all threads numbered 8 to 15 from any other process 
under our control, and so forth.

We then use set notation interesting with commands:

(gdb) [.7-80] print mytlsglob
$45 = [.8,16,32,64] 0xbadbad ; [.*] 0xfeedface

shows most threads with the same value for a variable, and a handful of 
threads with a different value.

Similarly, you could do selective stepping, in this case to step just 
four threads of a process:

(gdb) [.5-8] step
<misc output>
5: 45 foo() ...
6: 47 bar() ...
...
(gdb)

Similarly, the single-thread option for breakpoints will be generalized 
to sets.

Users will expect to be able to do all the usual set operations (union, 
intersection, negation), and to apply names to sets.  There needs to be 
a "focus" command that sets the default set of interest, applied to 
subsequent commands.

It should also be possible to define sets with predicates - [1245.*] can 
be thought of as having a predicate that is true for any thread of 
process 1245.  Additional predicates could be things like [1245.*future] 
for any thread not existing at the time the set is defined, or 
[1245.*suspended] for any thread that is not running at the moment.

Well, you say, all very interesting, but what does this have to with 
cores?  As a thread-like hardware resource, what really goes on with 
cores is assignment of processes and threads to cores.  So our idea is 
to add it as a sort of predicate-like annotation to p/t sets, yielding 
set descriptions like [4523.*@45], which designates all the threads of 
process 4523 that are currently assigned to core 45.

The astute reader will notice many holes.  In particular, I am not 
giving a formal definition of set syntax and how it fits into the CLI, 
because I want us to develop some consensus on what would work best.  
Key decisions to be made are:

1) square brackets? curly brackets? nothing at all?

2) prefix? postfix? both?

3) build in extensibility? or leave to Python somehow?

4) is there a better term than ptc set?

Set operations applied to execution control present some implementation 
challenges, and if you've been following Pedro's recent work, one of the 
goals is to remove the single-threaded all-or-nothing assumptions that 
have been in GDB's native support for a long time.  After that we expect 
to post a first version of ptc set support, and then the fun really begins!

Stan
stan@codesourcery.com

Re: Toward multicore GDB - Set theory

[Tom Tromey]

>>>>> "Stan" == Stan Shebs <stanshebs@earthlink.net> writes:

Stan> We then use set notation interesting with commands:

Stan> (gdb) [.7-80] print mytlsglob
Stan> $45 = [.8,16,32,64] 0xbadbad ; [.*] 0xfeedface

Stan> shows most threads with the same value for a variable, and a handful
Stan> of threads with a different value.

This is a tricky example since it implies pushing a lot of smarts into
'print'.  I'm in favor of it, I think it is the only way for this to
make sense and scale -- just painful.

Stan> Similarly, the single-thread option for breakpoints will be
Stan> generalized to sets.

Yes, nice.  I have actually been considering doing this myself, as a
follow-on to the ambiguous linespec patch.

Stan> The astute reader will notice many holes.  In particular, I am not
Stan> giving a formal definition of set syntax and how it fits into the CLI,
Stan> because I want us to develop some consensus on what would work best.
Stan> Key decisions to be made are:

Stan> 1) square brackets? curly brackets? nothing at all?
Stan> 2) prefix? postfix? both?
Stan> 3) build in extensibility? or leave to Python somehow?
Stan> 4) is there a better term than ptc set?

I think in the past we talked about square brackets and prefix.
How about we consider that as a straw proposal?

(gdb) [0-13] print x

I think that would suffice for all commands, even breakpoints; though it
may be slightly confusing that breakpoints "capture" the set while other
commands operate on them immediately.  If that is too confusing then
breakpoints could do another syntax, eg:

(gdb) break [0-13] function         # "semi-prefix"
(gdb) break function thread [0-13]  # "natural extension"


Allowing the definition of new kinds of ptc keywords from Python seems
natural.  I guess the question is whether it would be performant enough.

I think it would make sense to hold off on that, but carefully define
the PTC set syntax to be extensible and write the code to conservatively
reject unrecognized constructs.  Then we have leeway to make additions
in the future.

Tom

Re: Toward multicore GDB - Set theory

[Pedro Alves]

On Thursday 03 November 2011 21:01:32, Tom Tromey wrote:
> I think in the past we talked about square brackets and prefix.
> How about we consider that as a straw proposal?
> 
> (gdb) [0-13] print x

Yes, I had actually prototyped that yesterday for breakpoints, and it
was easy on top of a bunch of other infrustructure.  (I made `[' a command).

> I think that would suffice for all commands, even breakpoints; though it
> may be slightly confusing that breakpoints "capture" the set while other
> commands operate on them immediately.  If that is too confusing then
> breakpoints could do another syntax, eg:
> 
> (gdb) break [0-13] function         # "semi-prefix"
> (gdb) break function thread [0-13]  # "natural extension"

Breakpoints need two sets.  The trigger set, which is a generalization
of the "break foo thread N", meaning the set of inferiors/threads where
the breakpoint should fire, and, and suspend/stop set, which is the
set of inferiors/threads that should be suspended when the breakpoint
fires.  The HPD "standard" suggests:

[TRIGGER-SET] break {procedure | line | #file#line}
[ {-count n | -if condition} ] [-stop {stop-set | "[]"} ] 

In my prototype I chose to make that (simplified) for GDB (`()''s
means optional):

 [TRIGGER-SET] break -stop [STOP-SET] (--) LINESPEC

That is, put LINESPEC last, so that we can keep supporting
the current form, but avoid more hacks in linespecs like
the special termination for "thread/task/if" in the lexers
--- that wouldn't work for `['.

If you don't specify a [TRIGGER-SET] explicitly, the trigger
set should be the current global set.  I have yet to
specify how that exactly works, but it should be adjustable
with a "focus ITSET" command (whatever the spelling), and
supposedly the "thread" and "inferior" commands will affect
the default set too in some way.

So what I did was define the `[' command to temporarily
override the current global set.  I think that'll work for
other commands too.

Below's what I got now.

(gdb) info threads 
  Id   Target Id         Frame 
  3    Thread 0x7ffff7028700 (LWP 2296) "threads" (running)
* 2    Thread 0x7ffff7829700 (LWP 2295) "threads" thread_function0 (arg=0x0) at threads.c:63
  1    Thread 0x7ffff7fcb720 (LWP 2290) "threads" (running)
(gdb) [.2] break -stop [.3] 63
Breakpoint 4 at 0x40076d: file threads.c, line 63.

Trigger on thread 2 (equivalent to break 63 thread 2), and
when the breakpoint fires, stop thread 3.  Like so:

(gdb) c -a&
Continuing.
(gdb) 
Breakpoint 4, thread_function0 (arg=0x0) at threads.c:63
63              (*myp) ++;
info threads 
  Id   Target Id         Frame 
  3    Thread 0x7ffff7028700 (LWP 2296) "threads" 0x00007ffff78d75ad in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
* 2    Thread 0x7ffff7829700 (LWP 2295) "threads" thread_function0 (arg=0x0) at threads.c:63
  1    Thread 0x7ffff7fcb720 (LWP 2290) "threads" (running)

(gdb) info breakpoints 
Num     Type           Disp Enb Address            What
4       breakpoint     keep y   0x000000000040076d in thread_function0 at threads.c:63
        stop only in trigger-set: [.2]
        suspend all in stop-set: [.3]
        breakpoint already hit 1 time

We can make a breakpoint that stops the world with (recall I
had non-stop on):

(gdb) del 4
(gdb) [.2] break -stop [all] 63
Breakpoint 5 at 0x40076d: file threads.c, line 63.
(gdb) c -a&
Continuing.
(gdb) 
Breakpoint 5, thread_function0 (arg=0x0) at threads.c:63
63              (*myp) ++;
info threads 
  Id   Target Id         Frame 
  3    Thread 0x7ffff7028700 (LWP 2296) "threads" 0x00007ffff78d75ad in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
* 2    Thread 0x7ffff7829700 (LWP 2295) "threads" thread_function0 (arg=0x0) at threads.c:63
  1    Thread 0x7ffff7fcb720 (LWP 2290) "threads" 0x00007ffff7bc606d in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0

Since I've reimplemented all-stop on top of non-stop, when
non-stop is off, not specifying a stop set (or specifying it
as empty, like -stop []) means that all threads
stop by default, e.g., `[.2] break 63', but you can still
(with all-stop) specify a breakpoint that only suspends a
given itset, with:

(gdb) [.2] break -stop [.3] 63
Breakpoint 5 at 0x40076d: file threads.c, line 63.
(gdb) c &
Continuing.
(gdb) 
Breakpoint 5, thread_function0 (arg=0x0) at threads.c:63
63              (*myp) ++;
info threads 
  Id   Target Id         Frame 
  3    Thread 0x7ffff7028700 (LWP 2645) "threads" 0x00007ffff78d75ad in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
* 2    Thread 0x7ffff7829700 (LWP 2644) "threads" thread_function0 (arg=0x0) at threads.c:63
  1    Thread 0x7ffff7fcb720 (LWP 2610) "threads" (running)

So effectively, this ends up as a superset of all-stop/non-stop,
with the non-stop setting controlling the default stop set.

I haven't spent much time on the syntax bits yet (been focusing
on run control infrustructure), so this is obviously not a fully
cooked spec, but I'm mostly following the HPD spec anyway.  :-)

-- 
Pedro Alves

Re: Toward multicore GDB - Set theory

[Joel Brobecker]

> I think in the past we talked about square brackets and prefix.
> How about we consider that as a straw proposal?
> 
> (gdb) [0-13] print x

I like this option best.

-- 
Joel

Re: Toward multicore GDB - Set theory

[Daniel Jacobowitz]

On Mon, Nov 7, 2011 at 12:37 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>
> > I think in the past we talked about square brackets and prefix.
> > How about we consider that as a straw proposal?
> >
> > (gdb) [0-13] print x
>
> I like this option best.
>

Me, too.  I expect this to be a power user feature and plumbing for
IDEs; you won't very often want to specify this stuff by hand.  But if
we make it easy to name groups of threads, then the abbreviations may
be more commonly used.  Ideally I'd like to be able to hook this with
Python, so that a new thread can be categorized in some
application-specific way.

For example, it's common in the program I work on to have pools of
threads for a specific purpose.  You don't always want to treat the
request processing threads and the RPC threads the same way, but you
don't care which request processing thread hits your breakpoint - and
they can be created and destroyed dynamically.

--
Thanks,
Daniel

Re: Toward multicore GDB - Set theory

[Matt Rice]

On Thu, Nov 3, 2011 at 2:01 PM, Tom Tromey <tromey@redhat.com> wrote:

> (gdb) break [0-13] function         # "semi-prefix"

I personally prefer to avoid this one because the square brackets it
would seem difficult to
disambiguate between thread sets and objective-c methods,  at least
when naming threads is added.

Re: Toward multicore GDB - Set theory

[Pedro Alves]

On Tuesday 08 November 2011 05:17:28, Matt Rice wrote:
> On Thu, Nov 3, 2011 at 2:01 PM, Tom Tromey <tromey@redhat.com> wrote:
> 
> > (gdb) break [0-13] function         # "semi-prefix"
> 
> I personally prefer to avoid this one because the square brackets it
> would seem difficult to
> disambiguate between thread sets and objective-c methods,  at least
> when naming threads is added.
> 

Yeah, that was one of the reasons the current WIP prototype does:

 [TRIGGER-SET] break -stop [STOP-SET] (--) LINESPEC

That is [TRIGGER-SET] is a prefix, because it really is
the same meaning as applying the prefix in other commands.
That is, "the command applies to this set".  For breakpoints,
it naturally means "the breakpoint applies/triggers to/on
this set".  For "continue" it'd mean "continue this set",
for print, it'd mean "print this expression in the context
of this set", etc., etc..

The "-stop" option is required when specifying a
stop set (compared to not requiring "-stop" and just
accepting the [] part, and LINESPEC is last.  This gets rid
of LINESPEC ambiguity with obj-c at least, and leaves a saner
path open for other future options too.  I've left the
`--' option separator too for the same reason (like in your
info macro & friends changes).

-- 
Pedro Alves

Re: Toward multicore GDB - Set theory

[Tom Tromey]

>>>>> "Pedro" == Pedro Alves <pedro@codesourcery.com> writes:

Pedro> That is [TRIGGER-SET] is a prefix, because it really is
Pedro> the same meaning as applying the prefix in other commands.
Pedro> That is, "the command applies to this set".  For breakpoints,
Pedro> it naturally means "the breakpoint applies/triggers to/on
Pedro> this set".  For "continue" it'd mean "continue this set",
Pedro> for print, it'd mean "print this expression in the context
Pedro> of this set", etc., etc..

There is a subtle difference, though, between break and other commands.
Because breakpoints re-set, the trigger set must be captured and
re-evaluated at re-set.

I think it is all ok though.  It is slightly weird but I think we can
explain it adequately to users.

Pedro> The "-stop" option is required when specifying a
Pedro> stop set (compared to not requiring "-stop" and just
Pedro> accepting the [] part, and LINESPEC is last.  This gets rid
Pedro> of LINESPEC ambiguity with obj-c at least, and leaves a saner
Pedro> path open for other future options too.  I've left the
Pedro> `--' option separator too for the same reason (like in your
Pedro> info macro & friends changes).

Sounds very good to me.


I think this is going to interact with my ambiguous breakpoint /
linespec changes.  At various points in the new code, linespec iterates
over program spaces; this should be filtered according to the trigger
set.  I am wondering whether you are basing your work on this patch set,
and if not, how we can best manage the changes.  I would rather not be
in a race to finish, but instead talk about how we can best cooperate.

Tom

Re: Toward multicore GDB - Set theory

[Pedro Alves]

On Tuesday 08 November 2011 15:03:33, Tom Tromey wrote:

> I think this is going to interact with my ambiguous breakpoint /
> linespec changes.  At various points in the new code, linespec iterates
> over program spaces; this should be filtered according to the trigger
> set.  I am wondering whether you are basing your work on this patch set,

I'm currently working against current pristine mainline.

> and if not, how we can best manage the changes.  I would rather not be
> in a race to finish, but instead talk about how we can best cooperate.

My changes aren't as near complete as yours, so I don't think
there's much chance for me to win such a race.  :-)  But note
I haven't done anything like making:

  [2.*:3.*] break main

look for `main' in inferiors 2 and 3, and I don't plan to.  I'm leaving
the symbol search side out, as we'll get most of that from your
changes, with some kind of extra filtering.

There's a bunch left to try/do that doesn't touch linespec / symbol
filtering.  So I have absolutely no problem with seeing your changes
go in, and picking them up that way.  If I get further enough in
progress before your patch lands, I'll rebase on top of it.

I'll try to post the WIP code somewhere soon, so you (all) can
take a look.

-- 
Pedro Alves