non-blocking reads/writes and event loops

non-blocking reads/writes and event loops

[Andrew Cagney]

Hello,

[this is a cross post, reply-to has been set to gdb@sourceware]

Two architectural changes introduced in 5.0 were the the event-loop and
the targets ``remote async'' and ``remote extended-async''.  J.T.'s
recent posting about memory read/write raises a second significant issue
related to the event-loops, GUI's and async but first some background.

The event-loop provides a mechanism that allows GDB to block on more
than one input (or output) source.  For instance, GDB might be blocked
waiting for both the remote target and the user input.  As events
arrive, the event-loop dispatches them.

The async/extended-async targets exploit the event-loop so that, while
the target is running, the user can still enter various commands.  For
instance:

	(gdb) target extended-async ....
	(gdb) run&
	(gdb)
	(gdb) help ...
	....
	(gdb) 
	Breakpoint #1 reached, ...
	(gdb)

In getting this working, numerous problems were identified and several
compromizes were made (look for FIXMEs, especially in remote.c).

The main compromize is to do with when the target has halted.  While the
target is running the async target code returns control to the main
event-loop.  When the target starts ending it stopped packet, or when
the target is assumed to be halted, the code revers to polling.  Memory
reads/writes and the like are blocking operations - they do not return
control to the event loop.  For instance:

	(gdb) print foo

	gdb calls target-vector to read memory (&foo, sizeof(foo))
		target-vector ``calls'' remote to read memory from &foo
		target-vector blocks
		target-vector receives data from remote
	gdb receives data from target-vector
	
	$1 = "foo"		
	(gdb)

The rationale behind the decision to block in this case was very
pragmatic - it wasn't going to be feasible to invert (as was done to
serial.c) all of GDB (so that everything was event based) in a single
hit.

The consequence is that, during these blockages, GDB is slower than
dring normal operation - for the CLI the problem isn't too noticable. 
For the GUI, however, it is very noticable.  As a result, the built in
GUI (ex insight) is forced to use other means to avoid this slugishness
- during those blocked reads a timer is set up and that is used to
process the GUI events (see ui_loop_hook() and
gdb/gdbtk/generic/gdbtk-hooks.c:x_event()).

The thing that needs to be decided is how far GDB should be pushed to
address this problem.  Should GDB continue to be pushed to the point
where everything is event based or should, the current compromise remain
where a GUI is unable to exploit GDBs event-loop.

	Andrew

Re: non-blocking reads/writes and event loops

[Jim Ingham]

Hi, Andrew,

I think that the day has come when we need to be serious about making gdb a
really good backend for GUI's.  The percentage of programmers who are happy
with a purely command line tool is going down with time, and the advantages
to a good GUI Debugger are pretty clear.  For this to happen, we need to
remove as many of the fragilities of GDB as possible, and this blocking
behavior is a prime candidate.

One thing to remember about this is that any blocking operation that can
fail, particularly ones that have fairly long timeouts (or in some cases
like the RDI code, none at all) becomes a point of fragility for any GUI
based on top of gdb.  The result of not catching a potential hanging point
are serious, the application goes dead, and you have to kill it!

Having potential traps like this around in the code means the GUI will have
to analyze EVERY call down into gdb to see if it is going to block, and then
set the timers.  Often you can't really get "close" to where the blocking
call is, either, because there is a lot of setup work after the GUI hands
control off to core gdb, so doing this is not as easy as it seems.  And
having timers firing off when you are in the middle of other work also makes
the GUI coding hard to handle.

So my vote is to eradicate ALL the blocking behavior, and make GDB a pure
event driven application.  In the cases where you can't avoid blocking calls
(like ptrace calls, for instance) we should consider spinning a helper
thread, and doing the work there.  I am not in favor of a thread-happy
approach for gdb, but the careful use of short-lived threads to do
particular tasks is a lot more stable than forcing interrupt-driven
programming on the unhappy GUI's that attempt to use gdb...

Just my .02

Jim.


> Hello,
> 
> [this is a cross post, reply-to has been set to gdb@sourceware]
> 
> Two architectural changes introduced in 5.0 were the the event-loop and
> the targets ``remote async'' and ``remote extended-async''.  J.T.'s
> recent posting about memory read/write raises a second significant issue
> related to the event-loops, GUI's and async but first some background.
> 
> The event-loop provides a mechanism that allows GDB to block on more
> than one input (or output) source.  For instance, GDB might be blocked
> waiting for both the remote target and the user input.  As events
> arrive, the event-loop dispatches them.
> 
> The async/extended-async targets exploit the event-loop so that, while
> the target is running, the user can still enter various commands.  For
> instance:
> 
> (gdb) target extended-async ....
> (gdb) run&
> (gdb)
> (gdb) help ...
> ....
> (gdb) 
> Breakpoint #1 reached, ...
> (gdb)
> 
> In getting this working, numerous problems were identified and several
> compromizes were made (look for FIXMEs, especially in remote.c).
> 
> The main compromize is to do with when the target has halted.  While the
> target is running the async target code returns control to the main
> event-loop.  When the target starts ending it stopped packet, or when
> the target is assumed to be halted, the code revers to polling.  Memory
> reads/writes and the like are blocking operations - they do not return
> control to the event loop.  For instance:
> 
> (gdb) print foo
> 
> gdb calls target-vector to read memory (&foo, sizeof(foo))
> target-vector ``calls'' remote to read memory from &foo
> target-vector blocks
> target-vector receives data from remote
> gdb receives data from target-vector
> 
> $1 = "foo"  
> (gdb)
> 
> The rationale behind the decision to block in this case was very
> pragmatic - it wasn't going to be feasible to invert (as was done to
> serial.c) all of GDB (so that everything was event based) in a single
> hit.
> 
> The consequence is that, during these blockages, GDB is slower than
> dring normal operation - for the CLI the problem isn't too noticable.
> For the GUI, however, it is very noticable.  As a result, the built in
> GUI (ex insight) is forced to use other means to avoid this slugishness
> - during those blocked reads a timer is set up and that is used to
> process the GUI events (see ui_loop_hook() and
> gdb/gdbtk/generic/gdbtk-hooks.c:x_event()).
> 
> The thing that needs to be decided is how far GDB should be pushed to
> address this problem.  Should GDB continue to be pushed to the point
> where everything is event based or should, the current compromise remain
> where a GUI is unable to exploit GDBs event-loop.
> 
> Andrew
> 

Re: non-blocking reads/writes and event loops

[Todd Whitesel]

> The thing that needs to be decided is how far GDB should be pushed to
> address this problem.  Should GDB continue to be pushed to the point
> where everything is event based or should, the current compromise remain
> where a GUI is unable to exploit GDBs event-loop.

I am a advocate of long-time eventification. We should not create a mess
in an attempt to do it quickly, but it should remain a long-term goal.
The flexibility that we gain in doing so will pay us back later.

In particular, I think it is extremely inappropriate for GDB itself to
require threads. That would, in principle, be about as bad as allowing
parts of GCC to require a working C++ compiler...

-- 
Todd Whitesel
toddpw @ windriver.com

Re: non-blocking reads/writes and event loops

[Andrew Cagney]

Jim Ingham wrote:

> So my vote is to eradicate ALL the blocking behavior, and make GDB a pure
> event driven application.  In the cases where you can't avoid blocking calls
> (like ptrace calls, for instance) we should consider spinning a helper
> thread, and doing the work there.  I am not in favor of a thread-happy
> approach for gdb, but the careful use of short-lived threads to do
> particular tasks is a lot more stable than forcing interrupt-driven
> programming on the unhappy GUI's that attempt to use gdb...

Thing is, it is hard.

It means completly inverting the expression evaluator.  I've seen an
outline for how to do it so that just inferior calls are async.  Making
GDB fine-grain event based means that even memory and register transfers
are restartable.

However, making GDB responsive to more than one active target may force
the issue, regardless of any GUI.

	Andrew

Re: non-blocking reads/writes and event loops

[Jim Ingham]

on 6/12/00 6:47 PM, Todd Whitesel at toddpw@windriver.com wrote:

>> The thing that needs to be decided is how far GDB should be pushed to
>> address this problem.  Should GDB continue to be pushed to the point
>> where everything is event based or should, the current compromise remain
>> where a GUI is unable to exploit GDBs event-loop.
> 
> I am a advocate of long-time eventification. We should not create a mess
> in an attempt to do it quickly, but it should remain a long-term goal.
> The flexibility that we gain in doing so will pay us back later.

I heartily agree!

> 
> In particular, I think it is extremely inappropriate for GDB itself to
> require threads. That would, in principle, be about as bad as allowing
> parts of GCC to require a working C++ compiler...

That was also the conclusion we came to when Andrew, Elena et al were coming
up with the *-async targets.  I don't think that we should solve the problem
of making gdb non-blocking by putting everything in separate threads.
However, threads may need to be used in the case where you only have a
blocking API to control the inferior, but in that case all you are doing is
using threads to fix this local deficiency, not pushing it into the gdb
architecture.  I think this is managable.

Jim

Re: non-blocking reads/writes and event loops

[Todd Whitesel]

> However, threads may need to be used in the case where you only have a
> blocking API to control the inferior, but in that case all you are doing is
> using threads to fix this local deficiency, not pushing it into the gdb
> architecture.  I think this is managable.

Agreed. If the API designers are dumb enough to trust threads absolutely,
and don't give us an option, then well, that's life.

-- 
Todd Whitesel
toddpw @ windriver.com

Re: non-blocking reads/writes and event loops

[Andrew Cagney]

Todd Whitesel wrote:
> 
> > However, threads may need to be used in the case where you only have a
> > blocking API to control the inferior, but in that case all you are doing is
> > using threads to fix this local deficiency, not pushing it into the gdb
> > architecture.  I think this is managable.
> 
> Agreed. If the API designers are dumb enough to trust threads absolutely,
> and don't give us an option, then well, that's life.

For what it is worth, here is the ``It is hard'' reason number two:

In addition to the expression evaluator, every physical target (that is
the bit that knows how to read/write memory and registers) needs to be
inverted.  The basic programming model used when developing them
requiring a full rewrite.

At present a memory/register IO looks like (vaguely):

	event-loop
	   tty event: (gdb) print ...
		-> expression-evaluator
		    -> target->read()
		        -> os/remote->read()
		        <- data
		    <- data
		<- evaluate
	    print

at the target level, since we are inverting things, it would need to be
broken down into several transactions:

	event-loop
	    tty-event: (gdb) print ...
		-> expression-evaluator
		    -> target->request_read ()
		        -> os/remote->request_read ()

	event-loop
	    os/remote-event: some data ready
		-> os/remote->supply_some_of_data()

	event-loop
	    os/remote-event: some data ready
		-> os/remote->supply_rest_of_data() (all data ready)
		    -> target->supply_data()
			->expression_evaluator->supply_data()

	event-loop
	     ->expression-evaluator->display()

(please don't nit-pick on the detail :-).  The point being that you are
introducing a fundamentally different programming model into _all_
levels of GDB.  You are no longer using procedures but instead passing
around messages and writing state machines or similar.  All of which are
not mechanical.

Just making certain that your eyes are open :-)

	Andrew

Re: non-blocking reads/writes and event loops

[Todd Whitesel]

> > Agreed. If the API designers are dumb enough to trust threads absolutely,
> > and don't give us an option, then well, that's life.
> 
> For what it is worth, here is the ``It is hard'' reason number two:
...
> Just making certain that your eyes are open :-)

No doubts about that. That's why I said "long term goal"...

-- 
Todd Whitesel
toddpw @ windriver.com

Re: non-blocking reads/writes and event loops

[Nick Duffek]

On 12-Jun-2000, Andrew Cagney wrote:

>Should GDB continue to be pushed to the point
>where everything is event based or should, the current compromise remain
>where a GUI is unable to exploit GDBs event-loop.

On 12-Jun-2000, Jim Ingham wrote:

>One thing to remember about this is that any blocking operation that can
>fail, particularly ones that have fairly long timeouts (or in some cases
>like the RDI code, none at all) becomes a point of fragility for any GUI
>based on top of gdb.

[...]

>So my vote is to eradicate ALL the blocking behavior, and make GDB a pure
>event driven application.

Does that include file I/O?  Reading a core file over NFS could freeze a
GUI for quite a while.

If so, then we'd need to invert BFD as well.

I'd rather see the GUI problems solved by two processes:
  (1) the GDB core, which talks to inferior processes;
  (2) the user interface, which talks to (1) over a pipe using a
      well-defined protocol.

[Credit to Chris Faylor and Jim Blandy for suggesting this approach.]

The user interface could be a GUI, a command-line interface, Emacs, etc.
Pipes are non-blocking, so a GUI need never freeze due to blocking
debugging calls.

As far as I can tell, this provides all the benefits of fully
event-loopizing GDB without the cost of making GDB hugely more complex.

Toward implementing this approach, it might be helpful to have a library
for storing and transferring parts of GDB state like breakpoints,
watchpoints, and "set" values.

Such a library could be used by a GUI e.g. to maintain a copy of the
breakpoint list, so that it could perform the equivalent of "info
breakpoints" without blocking.

It could also be used to implement the saving of breakpoints, watchpoints,
and other state across GDB sessions.

Nick

Re: non-blocking reads/writes and event loops

[Andrew Cagney]

Nick Duffek wrote:

> >So my vote is to eradicate ALL the blocking behavior, and make GDB a pure
> >event driven application.
> 
> Does that include file I/O?  Reading a core file over NFS could freeze a
> GUI for quite a while.

Some how, I suspect not.  Some operations, such as reading/writing an
executable will continue to be blocking.  If an NFS partition freezes
then I suspect a hung GDB is the last thing on the users mind :-)

> I'd rather see the GUI problems solved by two processes:
>   (1) the GDB core, which talks to inferior processes;
>   (2) the user interface, which talks to (1) over a pipe using a
>       well-defined protocol.
> 
> [Credit to Chris Faylor and Jim Blandy for suggesting this approach.]

FYI, the ``well defined protocol'' is called MI :-)

> The user interface could be a GUI, a command-line interface, Emacs, etc.
> Pipes are non-blocking, so a GUI need never freeze due to blocking
> debugging calls.
> 
> As far as I can tell, this provides all the benefits of fully
> event-loopizing GDB without the cost of making GDB hugely more complex.

Unfortunately, it avoids rather than solves the problem.  One thing we
found from MI is that even after you separate out the GUI, GDB, in the
end, still needs to be 100% non blocking.  

Consider a GDBng (GDB 6) that is trying to talk to two (or more) active
remote targets.  The current remote.c blocks out all other activity
while it is communicating with a single target.  Just like it can
currently block out the GUI, it would also block out the processing of
those other targets.  While the user is surprisingly tolerant of a 5-30
second hiccup, the typical protocol tends to be be surprisingly
intolerant :-)

	Andrew

Re: non-blocking reads/writes and event loops

[Andrew Cagney]

Todd Whitesel wrote:
> 
> > > Agreed. If the API designers are dumb enough to trust threads absolutely,
> > > and don't give us an option, then well, that's life.
> >
> > For what it is worth, here is the ``It is hard'' reason number two:
> ...
> > Just making certain that your eyes are open :-)
> 
> No doubts about that. That's why I said "long term goal"...

Unfortunatly, unless feasible intermediate steps are identifed that show
how we can get from the eixsting GDB to this ``long term goal'' it will
just remain a long term goal :-(

To that end, I can see several strategies.  Interestingly at least one
discards the assumption that the event loop should _not_ be re-entrant.

o	invert the targets first
		
	This would be implemented
	using something like:

	(gdb) ->do-command
	  -> target_xfer_memory()
	    -> target->request_data()
	    while need more data, run inner event loop
	       <- target-supply-data()
	    <- return data

	That is, the GDB core would continue
	to assume that things are blocking
	but the targets would internally
	be implemented as non-blocking.

	Care would be needed that the internal
	event loop didn't re-call the CLI et.al.


o	invert the expression evaluator first

	In this case, legacy targets would
	be wrapped so that:

	(gdb) ->do-command
	   -> target_memory_request()
	     -> target_xfer_memory ()
	     schedule event to supply
	     returned data to expression evaluator
	-> supply_data to expression evaluator


(wonder if this makes any sense).  Of course there is option three, both
of the above.

For what its worth, for some reason I have preference for the first
option - not sure why, perhaphs it is simply that I'm more familar with
targets and back-ends.

As I noted above, one of the original design decisions for the event
loop that it not be re-entrant.  The above questions that decision. 
Another decision was that GDB's core assume no threads.  Should that too
be questioned?

enjoy,
	Andrew

Re: non-blocking reads/writes and event loops

[Daniel Berlin]

> Todd Whitesel wrote:
> > 
> > > > Agreed. If the API designers are dumb enough to trust threads absolutely,
> > > > and don't give us an option, then well, that's life.
> > >
> > > For what it is worth, here is the ``It is hard'' reason number two:
> > ...
> > > Just making certain that your eyes are open :-)
> > 
> > No doubts about that. That's why I said "long term goal"...
> 
> Unfortunatly, unless feasible intermediate steps are identifed that show
> how we can get from the eixsting GDB to this ``long term goal'' it will
> just remain a long term goal :-(

Almost as if planning something large was an integral part of getting it
done.
Strange.

> 
> To that end, I can see several strategies.  Interestingly at least one
> discards the assumption that the event loop should _not_ be re-entrant.
> 
> o	invert the targets first
> 		
> 	This would be implemented
> 	using something like:
> 
> 	(gdb) ->do-command
> 	  -> target_xfer_memory()
> 	    -> target->request_data()
> 	    while need more data, run inner event loop
> 	       <- target-supply-data()
> 	    <- return data
> 
> 	That is, the GDB core would continue
> 	to assume that things are blocking
> 	but the targets would internally
> 	be implemented as non-blocking.
> 
> 	Care would be needed that the internal
> 	event loop didn't re-call the CLI et.al.
> 
> 
> o	invert the expression evaluator first
> 
> 	In this case, legacy targets would
> 	be wrapped so that:
> 
> 	(gdb) ->do-command
> 	   -> target_memory_request()
> 	     -> target_xfer_memory ()
> 	     schedule event to supply
> 	     returned data to expression evaluator
> 	-> supply_data to expression evaluator
> 
> 
> (wonder if this makes any sense).  Of course there is option three, both
> of the above.
> 
> For what its worth, for some reason I have preference for the first
> option - not sure why, perhaphs it is simply that I'm more familar with
> targets and back-ends.
> 

> As I noted above, one of the original design decisions for the event
> loop that it not be re-entrant.  The above questions that decision. 
And rightfully questions it, IMHO.

> Another decision was that GDB's core assume no threads.  Should that too
> be questioned?

I have no problem with using threads, i do it on a daily basis. The only
issue i would have with GDB's core using threads would be that we take
care not to assume that everyone has pthreads. If we go with threads, i
would ask we add a simple abstraction, like most portable things using
threads (Just from personal experience, Python's source comes to mind as 
something that works with threads, using "easy to make work on a given
platform supporting threads", and provides all the thread functionality
people ask for. Works on BeOS, QNX, and every other python 
supported platform that has threads).

The question about whether we *should* use threads is a different one
altogether.
The real question splits into "Are there parts of gdb where we could be
doing
processing that isn't dependent on other processing, and we therefore are
possibly wasting time on MP systems by not having each done in a thread"
and "Are there parts of GDB where we could simplify code flow by using
threads".

 >
> enjoy, > Andrew > 

Re: non-blocking reads/writes and event loops

[Chris Faylor]

On Mon, Jun 19, 2000 at 03:01:08PM +1000, Andrew Cagney wrote:
>Nick Duffek wrote:
>
>> >So my vote is to eradicate ALL the blocking behavior, and make GDB a pure
>> >event driven application.
>> 
>> Does that include file I/O?  Reading a core file over NFS could freeze a
>> GUI for quite a while.
>
>Some how, I suspect not.  Some operations, such as reading/writing an
>executable will continue to be blocking.  If an NFS partition freezes
>then I suspect a hung GDB is the last thing on the users mind :-)
>
>> I'd rather see the GUI problems solved by two processes:
>>   (1) the GDB core, which talks to inferior processes;
>>   (2) the user interface, which talks to (1) over a pipe using a
>>       well-defined protocol.
>> 
>> [Credit to Chris Faylor and Jim Blandy for suggesting this approach.]
>
>FYI, the ``well defined protocol'' is called MI :-)

I think I suggested that this was the same mechanism that we had been
considering for future UI work.

(deftly dancing)

>> The user interface could be a GUI, a command-line interface, Emacs, etc.
>> Pipes are non-blocking, so a GUI need never freeze due to blocking
>> debugging calls.
>> 
>> As far as I can tell, this provides all the benefits of fully
>> event-loopizing GDB without the cost of making GDB hugely more complex.
>
>Unfortunately, it avoids rather than solves the problem.  One thing we
>found from MI is that even after you separate out the GUI, GDB, in the
>end, still needs to be 100% non blocking.  
>
>Consider a GDBng (GDB 6) that is trying to talk to two (or more) active
>remote targets.  The current remote.c blocks out all other activity
>while it is communicating with a single target.  Just like it can
>currently block out the GUI, it would also block out the processing of
>those other targets.  While the user is surprisingly tolerant of a 5-30
>second hiccup, the typical protocol tends to be be surprisingly
>intolerant :-)

I don't think that Nick was suggesting going back to 100% blocking
behavior.  In this case, if blocking I/O was the only thing available,
you'd need to fork another process.

I was thinking more of a process hive which communicated via shared
memory.  So, there would be a main "gdb" which printed a prompt
and as many other dedicated processes as needed to control targets.

As far as NFS is concerned, I think that Nick was talking about the
noticeable lags that can occur when loading something like a core file
over NFS.  The delay isn't necessarily a sign of a problem with NFS.
Maybe a user would expect a long delay in this case but I would think
that it would also be a problem for gdb.

If you assigned the "reading a core file" function to another process
then the lag could be circumvented.

cgf

Re: non-blocking reads/writes and event loops

[Nick Duffek]

On 19-Jun-2000, Chris Faylor wrote:

>On Mon, Jun 19, 2000 at 03:01:08PM +1000, Andrew Cagney wrote:

>>Consider a GDBng (GDB 6) that is trying to talk to two (or more) active
>>remote targets.

>I was thinking more of a process hive which communicated via shared
>memory.

Or via a pipe or socket, the point (for me) being that multiple processes
-- one per target plus one for the user interface -- give us all the
non-blocking parallelism we need.

At least, I haven't yet come up with a scenario where that's not the case;
has anyone else?

>As far as NFS is concerned, I think that Nick was talking about the
>noticeable lags that can occur when loading something like a core file
>over NFS.

Right.  Isn't getting rid of temporary GUI freezes one of the whole points
of event-loopification?

Nick

Re: non-blocking reads/writes and event loops

[Jim Ingham]

on 6/16/00 2:54 PM, Nick Duffek at nsd@redhat.com wrote:

> On 12-Jun-2000, Andrew Cagney wrote:
> 
>> Should GDB continue to be pushed to the point
>> where everything is event based or should, the current compromise remain
>> where a GUI is unable to exploit GDBs event-loop.
> 
> On 12-Jun-2000, Jim Ingham wrote:
> 
>> One thing to remember about this is that any blocking operation that can
>> fail, particularly ones that have fairly long timeouts (or in some cases
>> like the RDI code, none at all) becomes a point of fragility for any GUI
>> based on top of gdb.
> 
> [...]
> 
>> So my vote is to eradicate ALL the blocking behavior, and make GDB a pure
>> event driven application.
> 
> Does that include file I/O?  Reading a core file over NFS could freeze a
> GUI for quite a while.
> 
> If so, then we'd need to invert BFD as well.

Ideally, yes we should do this.  But this can be a gradual process, and I am
less worried about reading core files, since this is a single atomic
operation, and thus easier to protect against with timeouts, etc...

> 
> I'd rather see the GUI problems solved by two processes:
> (1) the GDB core, which talks to inferior processes;
> (2) the user interface, which talks to (1) over a pipe using a
> well-defined protocol.
> 
> [Credit to Chris Faylor and Jim Blandy for suggesting this approach.]

While I have a lot of respect for these two folks, this is hardly their
idea.  This is the approach that most GUI interfaces to GDB have taken over
the years.  

I am of two minds here.  In many IDE type applications, there is a desire to
host the debugger IN the IDE, and breaking out the debugger engine into a
separate process is certainly attractive.  It also does (with proviso's that
I note below) help solve the problem of gdb blocking without having to do
much work.  

On the other hand, GDB knows a lot about the application that it is
debugging, and in big programs there is a LOT to know...  So having an
in-process gui is also attractive, since you can get very quick and direct
access to all this knowledge, without having to package it into strings, and
unpackage it at the other end.

Doing things like presenting all the globals available to a given module, or
sometimes even all the local variables, can involve a lot of data flow.
Ditto for memory views, and even some crazy processors that have hundreds of
registers.  

Note also that this data has to be updated on EACH STEP, while the user
expects steps to complete quickly.  And most people are pretty unforgiving
about both performance in this area, AND the UI being out of synch with the
inferior at each stage.  If there is a delay in stepping, you interrupt the
concentration of the user as they are trying to chase the flow of control of
their program.  And if data is ever out of synch (even if you try to get
around this by coloring un-updated data or some such) you break the user's
trust in the information being presented.  Maintaining this trust is also
essential to allowing the user to settle into and concentrate on the
difficult problem of debugging...

So I still think it is important to try to make it easy to support
in-process GUI's to gdb like Insight.

> 
> The user interface could be a GUI, a command-line interface, Emacs, etc.
> Pipes are non-blocking, so a GUI need never freeze due to blocking
> debugging calls.

Note that even when the UI is in another process, it is really nice to have
a non-blocking gdb under the hood.  It makes it easier/possible to implement
things like progress bars, etc.  It also makes the question "Has gdb gone
south, has a board gone south, or is it just waiting a long time to do
something?" easier to ask & answer.  Moreover, the ^C mechanism for
interrupting gdb is pretty fragile as well, or at least it caused us LOTS of
problems with Insight...

There are also some interrupt type things that are quite useful to have, and
which would be more safely implemented in gdb.  One example of this is the
ability to insert breakpoints while the inferior is running.  If you have
ever used a debugger that supports this, I think you will agree this is
REALLY useful.  You can of course implement this by sending a ^C to gdb from
the UI, inserting the breakpoint, and then continuing the inferior.  But I
think (a) it is silly for each GUI to have to implement this separately, and
(b) gdb is better equipped to do this correctly than a UI which, after all,
has fairly limited knowledge of the inferior.  The architecture for this
becomes much cleaner if gdb is live and able to respond to the client at all
times.

> 
> As far as I can tell, this provides all the benefits of fully
> event-loopizing GDB without the cost of making GDB hugely more complex.

I don't think that the END RESULT of event loop-izing gdb would be to make
GDB hugely more complex.  In many cases, I think that it would make the
architecture much simpler and cleaner, since you would not have modal loops
hiding out all over gdb, but rather a very simple event loop, and,
hopefully, a standard mechanism for waiting in the event loop that all the
different modules of gdb could share.  Not to say that the PROCESS of
getting GDB to this point would be easy, as Andrew points out...

> 
> Toward implementing this approach, it might be helpful to have a library
> for storing and transferring parts of GDB state like breakpoints,
> watchpoints, and "set" values.
> 
> Such a library could be used by a GUI e.g. to maintain a copy of the
> breakpoint list, so that it could perform the equivalent of "info
> breakpoints" without blocking.

This sounds okay, but do we really want to go the route of prospectively
priming the GUI with ALL the bits that GDB knows about, and which the UI
might want to know, at each step, just so we don't have to make gdb
non-blocking?

> 
> It could also be used to implement the saving of breakpoints, watchpoints,
> and other state across GDB sessions.
> 

This sort of stuff is very useful, though I would want to make sure that
this library is more of a library, i.e. useable in process as well as for
client apps.  The MI interface is a step towards this, though it still
suffers from the "distributed" nature of information gathering in gdb.  But
I think this is a separate issue from the blocking behavior of gdb.

Jim

Re: non-blocking reads/writes and event loops

[Nick Duffek]

On 19-Jun-2000, Jim Ingham wrote:

>Note also that this data has to be updated on EACH STEP, while the user
>expects steps to complete quickly.  And most people are pretty unforgiving
>about both performance in this area

Shared memory could minimize the performance impact.  In fact, on a
multi-processor host, the 2-process approach conceivably could result in
snappier GDB response.

>AND the UI being out of synch with the
>inferior at each stage.

I don't understand how data display synchronization is a 1-GDB-process
vs. multi-GDB-process issue.  If data is being displayed while the
inferior is running, GDB's data display will lag behind reality regardless
of whether the display is happening in a front-end UI process.  If the
inferior is stopped, the data display will be accurate.

What am I missing?

>There are also some interrupt type things that are quite useful to have, and
>which would be more safely implemented in gdb.  One example of this is the
>ability to insert breakpoints while the inferior is running.

Isn't this already addressed by the asynchronous execution feature?

I see how asynchronous breakpoints could be useful, and I agree that they
should be implemented directly rather than via ^C.  I'm arguing against
making things like inferior memory I/O be asynchronous.

>I don't think that the END RESULT of event loop-izing gdb would be to make
>GDB hugely more complex.

I once modified a piece of software to do asynchronous magnetic tape I/O.
It was complex and messy to keep track of the state necessary to make that
one part of the software work asynchronously, but there was a big
performance payoff that made it worthwhile.

Making all parts of GDB work asynchronously seems very complex and messy,
and I still don't see the big payoff.  Of course, I'm extrapolating the
complexity from another software project, so perhaps I'm overestimating
it.

>> Toward implementing this approach, it might be helpful to have a library
>> for storing and transferring parts of GDB state like breakpoints,
>> watchpoints, and "set" values.

>This sounds okay, but do we really want to go the route of prospectively
>priming the GUI with ALL the bits that GDB knows about, and which the UI
>might want to know, at each step, just so we don't have to make gdb
>non-blocking?

Maybe yes, because I would expect a GUI be primed with all those bits
anyway, for the sake of displaying them graphically rather than textually.
E.g., to display the breakpoint list as something other than raw "info
breakpoints" output, a GUI would need to know about breakpoint numbers,
conditions, commands, threads, etc.

Thanks for the in-depth response,

Nick

Re: non-blocking reads/writes and event loops

[Jim Ingham]

on 6/19/00 4:08 PM, Nick Duffek at nsd@redhat.com wrote:

> On 19-Jun-2000, Jim Ingham wrote:
> 
>> Note also that this data has to be updated on EACH STEP, while the user
>> expects steps to complete quickly.  And most people are pretty unforgiving
>> about both performance in this area
> 
> Shared memory could minimize the performance impact.  In fact, on a
> multi-processor host, the 2-process approach conceivably could result in
> snappier GDB response.

I would hesitate to use shared memory to communicate between gdb and the UI,
unless I knew that I was only going to have to support a few platforms,
since it is pretty platform dependent...  Maybe if you wrapped it in some
abstraction like CORBA...

> 
>> AND the UI being out of synch with the
>> inferior at each stage.
> 
> I don't understand how data display synchronization is a 1-GDB-process
> vs. multi-GDB-process issue.  If data is being displayed while the
> inferior is running, GDB's data display will lag behind reality regardless
> of whether the display is happening in a front-end UI process.  If the
> inferior is stopped, the data display will be accurate.
> 
> What am I missing?
> 

The processing that goes into packing all the information you need to
present into string based data structures on the gdb end, shipping it across
the pipe, and then unpacking it on the other end.  With a lot of data, this
can be pretty noticable.  You can ameliorate this by using shared memory,
and using something like CORBA to transmit the data in a more structured
form.  But this is not a negligible concern, in my experience.

>> There are also some interrupt type things that are quite useful to have, and
>> which would be more safely implemented in gdb.  One example of this is the
>> ability to insert breakpoints while the inferior is running.
> 
> Isn't this already addressed by the asynchronous execution feature?

Yes, this example falls in the realm of execution, so its okay (except that
of course there is only one target that actually has an async equivalent.)

> 
> I see how asynchronous breakpoints could be useful, and I agree that they
> should be implemented directly rather than via ^C.  I'm arguing against
> making things like inferior memory I/O be asynchronous.
> 

The problem that I have seen is that many things can go wrong in the
communication between the debugger and the target, and if gdb is hung
waiting on something to happen, then the UI can't really give good feedback
as to what is going on.  This leads to times where the UI is alive, which is
good, but has no idea of what GDB is doing.  I can usually reset in this
case, but this is not all that satisfactory, since I don't know from the UI
what went wrong...

>> I don't think that the END RESULT of event loop-izing gdb would be to make
>> GDB hugely more complex.
> 
> I once modified a piece of software to do asynchronous magnetic tape I/O.
> It was complex and messy to keep track of the state necessary to make that
> one part of the software work asynchronously, but there was a big
> performance payoff that made it worthwhile.
> 
> Making all parts of GDB work asynchronously seems very complex and messy,
> and I still don't see the big payoff.  Of course, I'm extrapolating the
> complexity from another software project, so perhaps I'm overestimating
> it.

I don't know.  Tcl is pretty much all event driven.  It took a lot of work
to come up with a model that works well for all sorts of different input
sources, and on a wide variety of different platforms, but having done the
work, it is not that hard now to plug in new sources, and get them to play
nice.  I guess I am saying that we have some models where, once correctly
juked around, the results are quite nice.

> 
>>> Toward implementing this approach, it might be helpful to have a library
>>> for storing and transferring parts of GDB state like breakpoints,
>>> watchpoints, and "set" values.
> 
>> This sounds okay, but do we really want to go the route of prospectively
>> priming the GUI with ALL the bits that GDB knows about, and which the UI
>> might want to know, at each step, just so we don't have to make gdb
>> non-blocking?
> 
> Maybe yes, because I would expect a GUI be primed with all those bits
> anyway, for the sake of displaying them graphically rather than textually.
> E.g., to display the breakpoint list as something other than raw "info
> breakpoints" output, a GUI would need to know about breakpoint numbers,
> conditions, commands, threads, etc.
>

I think I was making the distinction between having data structures to
represent all these sorts of things, which the GUI probably will need, and
actually prospectively filling them all before certain classes of operation,
because the GUI knows the user might open a new window requiring their data,
and GDB will be unresponsive during the given operation, so it can't provide
the info...
 
> Thanks for the in-depth response,
> 

Interesting discussion, no prob...

Jim

Re: non-blocking reads/writes and event loops

[Andrew Cagney]

Jim Ingham wrote:


> > As far as I can tell, this provides all the benefits of fully
> > event-loopizing GDB without the cost of making GDB hugely more complex.
> 
> I don't think that the END RESULT of event loop-izing gdb would be to make
> GDB hugely more complex.  In many cases, I think that it would make the
> architecture much simpler and cleaner, since you would not have modal loops
> hiding out all over gdb, but rather a very simple event loop, and,
> hopefully, a standard mechanism for waiting in the event loop that all the
> different modules of gdb could share.  Not to say that the PROCESS of
> getting GDB to this point would be easy, as Andrew points out...

Yes, FWIW, an example of code that would benefit from being inverted is
remote.c.  remote.c, which is implementing a protocol, should be
implemented using a state machine.

	Andrew

Re: non-blocking reads/writes and event loops

[Todd Whitesel]

> I don't know.  Tcl is pretty much all event driven.  It took a lot of work
> to come up with a model that works well for all sorts of different input
> sources, and on a wide variety of different platforms, but having done the
> work, it is not that hard now to plug in new sources, and get them to play
> nice.  I guess I am saying that we have some models where, once correctly
> juked around, the results are quite nice.

While I agree with the theory here, I take issue with the statement that Tcl
implements it.

I was actually quite disappointed by the sources to Tcl/Tk 8.0.2 which
appear to have subtle sub-event loops creeping around whenever callbacks
are used.

-- 
Todd Whitesel
toddpw @ windriver.com

Re: non-blocking reads/writes and event loops

[Todd Whitesel]

> For what its worth, for some reason I have preference for the first
> option - not sure why, perhaphs it is simply that I'm more familar with
> targets and back-ends.

IMHO the simplest way is always to invert from the top down, so things are
just merged into the main event loop as you go.

> As I noted above, one of the original design decisions for the event
> loop that it not be re-entrant.  The above questions that decision. 

Well that depends on how much of the event loop machinery is used by the
low-level inversion. If it calls things that assume a single global event
loop, then we have a problem.

If you have to invert a low-level algorithm early on, there are a couple
ways to do it:

  1. make the event loop machinery instantiable and use a new instance of it.
	THIS IS ARGUABLY NECESSARY FOR MULTIPLE TARGET CONNECTIONS ANYWAY.

  2. do not attempt to provide full non-blocking facilities yet, just show
	that the low-level code works correctly in inverted form -- its
	sub-event loop just calls it repeatedly, so it still blocks but at
	least uses the new code structure.

IMHO either is fine; which one you use depends on whether you have gotten
the instantiable event loop machinery working yet.

> Another decision was that GDB's core assume no threads.  Should that too
> be questioned?

I don't mind specific native target support assuming threads.

However using threads as a general method of avoiding the inversion of
GDB core code is a COP OUT.

-- 
Todd Whitesel
toddpw @ windriver.com

Re: non-blocking reads/writes and event loops

[Elena Zannoni]

Todd Whitesel writes:
 > > For what its worth, for some reason I have preference for the first
 > > option - not sure why, perhaphs it is simply that I'm more familar with
 > > targets and back-ends.
 > 
 > IMHO the simplest way is always to invert from the top down, so things are
 > just merged into the main event loop as you go.
 > 
 > > As I noted above, one of the original design decisions for the event
 > > loop that it not be re-entrant.  The above questions that decision. 
 > 
 > Well that depends on how much of the event loop machinery is used by the
 > low-level inversion. If it calls things that assume a single global event
 > loop, then we have a problem.

Yes, that wasn't one of the things I worried about when I wrote the
event loop.

 > 
 > If you have to invert a low-level algorithm early on, there are a couple
 > ways to do it:
 > 
 >   1. make the event loop machinery instantiable and use a new instance of it.
 > 	THIS IS ARGUABLY NECESSARY FOR MULTIPLE TARGET CONNECTIONS ANYWAY.
 > 
 >   2. do not attempt to provide full non-blocking facilities yet, just show
 > 	that the low-level code works correctly in inverted form -- its
 > 	sub-event loop just calls it repeatedly, so it still blocks but at
 > 	least uses the new code structure.
 > 

Yes, we must do things in smaller steps. Otherwise the task is so
daunting, it will never get done. 

There were actually a few things that we were considering as 'next
steps' in the event-loopization process. One is to hook up the async
version of the remote target to the Insight gui. Another is to make a
native target asynchronous. The choice here would be Linux, I
think. Also there are interface/CLI issues (style decisions) still to
be resolved with the asyncrhonous remote target.

As Andrew pointed out, the remote target is the hairy case. We
encountered several problems when writing the asynchronous version. We
got as far as we could, w/o having to revise major assumptions, and
found that we got maybe 70/80 % of the stuff working fine. The
remaining bits were the hard ones (the ones where GDB gets stuck
and you don't know what happened), for which we really need to do
major surgery to remote.c and the layers below that.

Has anybody tried to use 'target async' and 'target extended-async' in
place of 'target remote' and 'target extended-remote'? I have heard no
feedback on them at all. 

Elena

 > IMHO either is fine; which one you use depends on whether you have gotten
 > the instantiable event loop machinery working yet.
 > 
 > > Another decision was that GDB's core assume no threads.  Should that too
 > > be questioned?
 > 
 > I don't mind specific native target support assuming threads.
 > 
 > However using threads as a general method of avoiding the inversion of
 > GDB core code is a COP OUT.
 > 
 > -- 
 > Todd Whitesel
 > toddpw @ windriver.com

Re: non-blocking reads/writes and event loops

[Jim Blandy]

> The question about whether we *should* use threads is a different
> one altogether.  The real question splits into "Are there parts of
> gdb where we could be doing processing that isn't dependent on other
> processing, and we therefore are possibly wasting time on MP systems
> by not having each done in a thread" and "Are there parts of GDB
> where we could simplify code flow by using threads".

Srikanth at HP has mentioned the idea of doing symbol table reads in
the background.  I thought that sounded pretty cool.

Re: non-blocking reads/writes and event loops

[Jim Blandy]

> I'd rather see the GUI problems solved by two processes:
>   (1) the GDB core, which talks to inferior processes;
>   (2) the user interface, which talks to (1) over a pipe using a
>       well-defined protocol.
> 
> [Credit to Chris Faylor and Jim Blandy for suggesting this
>   approach.]

I don't think that's what I meant to suggest.  I meant to suggest
something much less radical, and probably didn't explain it well.

Re: non-blocking reads/writes and event loops

[Eli Zaretskii]

Could we please step back for a moment and see the big picture before
talking about the details?

It's possible that I'm the only clueless one in town, in which case
please be gentle with me ;-).  But it seems to me that not everybody
in this thread has the same purpose in mind when discussing the
non-blocking features.  So perhaps it will help to revisit the broader
issue, even if it were discussed in the past.

I mean, non-blocking is nice, but where would a user want it in GDB?

To me, a debugger is a way of tightly-controlled program execution.
We all know how much concentration and detail it takes to debug a
non-trivial program.  You need to be sure what is the current state of
the program, before you make the next step.  Finding out the exact
state of the program might require printing lots of variables,
examining registers, calling functions, head-scratching, hair-pulling,
etc.

So what does ``non-blocking'' mean, in this context?  If I ask GDB for
the value of some expression that takes a long time to compute, what
possibly would I want to do while GDB and the inferior grind away to
bring me the result, except wait impatiently for the result to come
through?  I cannot switch to anything else, because that will distract
me, and I will lose my grip on the problem (unless the computation
takes hours, in which case I probably won't use that method of
debugging anyway).

Someone brought up the issue of several targets being debugged
simultaneously.  One possible situation when this will be useful is
when the targets being debugged are parts of a complex system,
i.e. they talk to each other through various means, or act on the same
data.  (Are there any other situations where debugging several targets
simultaneously makes sense?)

If that is the case, would you want to let one target run while the
other is blocked because GDB is interrogating it?  I think not: if you
do, you lose control of what's going on in the larger _system_ you are
debugging.  So, again, there's nothing you can do but wait for the
lengthy operation to complete and show you the results.

Only *after* you have seen these results, can you decide what to do
next.  You could let the program run some more, or add some variables
to the displayed ones and single-step it, or maybe look at some more
variables before you know what to do.  It will not help you to be able
to run the inferior while you are examining variables and thinking
about what you see, because it's quite possible that what the inferior
does while you think will have to be discarded, or even destroys some
data and covers up the traces of the bug you are hunting.  Since
there's no easy way to run the inferior backwards, it's better not to
let it run until you know how do you want to proceed with debugging.

It does make sense to have some degree of non-blocking behavior when
the info I requested is so voluminous that it is larger than I can
grasp in a single glance.  In that case, it is useful to have GDB work
asynchronously while I'm reading the first screenful.  But this kind
of functionality is already supported by the event loop, so if GDB
talks to targets through `select' or `poll', we are done.  This is
nowhere near the problem of GDB and the UI falling out of sync, data
coming to the user that lags behind the inferior program's state,
etc., which, to me, are in sharp contradiction to the very purpose of
debugging: a controlled execution of a program.

In other words, I submit that debugging is a single-thread activity,
as far as we humans are concerned.  And if that is the case, what
exactly do we want the totally-non-blocking GDB for, except the
voluminous output case?