RE: (hardware) watchpoints and actions

RE: (hardware) watchpoints and actions

[duane]

david> Consider a remote target that you are communicating with via the
remote protocol.
david> If the target's GDB stub does not support breakpoint commands or
you wish to run
david> a command that is not convertible to a string of agent expression
opcodes, then to
david> run the command there is a lot of back and forth between GDB and
the target.

david> Further, GDB must remain connected to the target.

david> By contrast, when a tracepoint is hit, there is NO back and forth
with GDB.
david> GDB might not even be connected to the target. The target
executes a series of
david> commands that were expressed as strings of agent expression
opcodes and arguments.
david> A command might be to collect registers, or collect a particular
variable or collect
david> some stack. After the commands execute the program continues --
there is no
david> explicit 'continue' command.

Two problems you need to address:

a) Most "gdb-server" type things are rather "dumb" they do not know what
symbols are.
    GDB's protocol speaks "address" as an integer like value, not
symbolic.
    GDB's protocol has no idea what base+offset might be.
    GDB's protocol has no concept of "stack" or "register" based
variables.
    The STUB or GDB-SERVER has no access to the dwarf (or whatever)
debug information

Example: A linked list, you want to trace some data element, i.e.: 
trace( listPtr->Data )

At the breakpoint:

The variable: "listPtr" might be on the stack or a register
The Element "DATA" is at some base + offset

Thus you can only trace global variables
Secondary problem:  Has the compiler optimization updated the global? Or
is it still in the register?

How do you plan to solve this? I am also ignoring ENDIAN issues here.

b) Something in the target must 'interpret' the series of commands to
execute when the watch point hits.

Your options are: Install a small script interpreter into he
gdb-stub/gdb-server 
This might work for a GDB-SERVER on linux... you have a rich
environment.
Again - same problems as  above (access to debug information)
It does not work well for JTAG type GDB-Servers.

The other option something like GDB-STUBS (rom-monitor based) solution,
when the breakpoint/watchpoint exception occurs
For example - you could do sometime of JIT compilation, and download
code to the target that executes when the exception occurs.

A simple example might be you insert a "JSR my_bp_handler" at the
breakpoint location. 
That JSR/CALL does something special.

Questions (only a few)
  Where do you locate the code it jumps to?
  Can you modify the code? (is it in FLASH memory, or ReadOnly memory?)
  Can you modify the memory region attributes (i.e.: MMU bits) that
allow execution in that area?

------

Don't get me wrong, I am very aware of the problem you want to solve,
and the challenges
A good example might be a PID loop motor control, 100 times a second the
CPU calculates 
new motor control variables and must re-write hardware registers
otherwise the motor suddenly stops.

Other examples might be a network protocol with data flowing... you have
overrun and lost packets if you do not respond in time

What actually works better is a hardware based trace scheme, and that is
*VERY* chip/silicon/vendor specific.  One example is the ARM STM (system
trace macrocell)

But the code that supports this type of stuff often works better if it
is compiled into the actual application, which to some degree violates
the "do not ship debug code in production code" rule.

-Duane.

RE: (hardware) watchpoints and actions

[taylor, david]

    From: duane@duaneellis.com [mailto:duane@duaneellis.com] 

    Two problems you need to address:

    a) Most "gdb-server" type things are rather "dumb" they do not know what symbols are.
        GDB's protocol speaks "address" as an integer like value, not symbolic.
        GDB's protocol has no idea what base+offset might be.
        GDB's protocol has no concept of "stack" or "register" based variables.
        The STUB or GDB-SERVER has no access to the dwarf (or whatever) debug information

Our remote stub does not know the symbols, does not have access to the debug information,
and does not need the symbols nor the debug information.

    Example: A linked list, you want to trace some data element, i.e.: 
    trace( listPtr->Data )

    At the breakpoint:

    The variable: "listPtr" might be on the stack or a register The Element "DATA" is at some base + offset

    Thus you can only trace global variables Secondary problem:  Has the compiler optimization updated the global? Or is it still in the register?

Not true.  The debug information, read by GDB, says where to find the variable.
GDB knows, it's in this register or it's at this offset from this register, or it's at this
memory location.

If you say, "collect listPtr->Data", then GDB knows where listPtr is.  And it knows the
offset of Data relative to the start of the object.  And it knows the size of Data.  It knows
what it needs to know to generate an agent expression.  Look at the command

  maint agent

Similarly, for evaluating without collecting, examine "maint agent-eval".

    How do you plan to solve this? I am also ignoring ENDIAN issues here.

Not an issue and already solved.  Our stubs (both old and new) contain substantial
tracepoint support.  And the tracepoint handling engine contains an agent expression
evaluator.

    b) Something in the target must 'interpret' the series of commands to execute when the watch point hits.

The tracepoint support already contains an agent expression 'interpreter'.

    Your options are: Install a small script interpreter into he gdb-stub/gdb-server This might work for a GDB-SERVER on linux... you have a rich environment.

We already have such an 'interpreter'.  It's been in our stub for more than a decade.
There is also such an 'interpreter' in gdb-server.

    Again - same problems as  above (access to debug information) It does not work well for JTAG type GDB-Servers.

It does not need and does not have access to debug information.  The expressions
that come across contain things like register numbers, constants (offsets, memory
addresses, sizes, whatever), and operators.  It works beautifully.  Look at the agent
expression section (I think it's one of the appendixes)  in the GDB manual.

Our machine has an operating system.  We communicate with our GDB stub via TCP/IP.

    The other option something like GDB-STUBS (rom-monitor based) solution, when the breakpoint/watchpoint exception occurs For example - you could do sometime of JIT compilation, and download code to the target that executes when the exception occurs.

    A simple example might be you insert a "JSR my_bp_handler" at the breakpoint location. 
    That JSR/CALL does something special.

It's an X86-64 processor.  A breakpoint or tracepoint causes an int3 instruction to be inserted.
The operating system invokes a handler for the int3 giving it information such as the address.

    ------

    Don't get me wrong, I am very aware of the problem you want to solve, and the challenges A good example might be a PID loop motor control, 100 times a second the CPU calculates new motor control variables and must re-write hardware registers otherwise the motor suddenly stops.

    Other examples might be a network protocol with data flowing... you have overrun and lost packets if you do not respond in time

    What actually works better is a hardware based trace scheme, and that is
    *VERY* chip/silicon/vendor specific.  One example is the ARM STM (system trace macrocell)

The hardware supports trapping on accesses to stub specified addresses.
It only supports a trapping on a small number of data addresses.

The data tracepoint stuff I described works.  It works around GDB rather than
working with GDB.  I'd like it to work with GDB.

    But the code that supports this type of stuff often works better if it is compiled into the actual application, which to some degree violates the "do not ship debug code in production code" rule.

The code *IS* compiled into the application.  The GDB stub is always compiled in.

The stub is NOT debug code.  It enables the developer to examine the system, set
breakpoints, tracepoints, watchpoints, or whatever.

Another rule is to test what you ship.  You don't want to debug with the stub removed. 
And if a problem is found in testing, it's much easier to track it down if the stub is
compiled in.  The stub is always compiled in.  It is part of the shipped product.

    -Duane.

David

Re: (hardware) watchpoints and actions

[Paul_Koning]

> On Nov 12, 2015, at 3:44 PM, taylor, david <david.taylor@emc.com> wrote:
> ...
>    But the code that supports this type of stuff often works better if it is compiled into the actual application, which to some degree violates the "do not ship debug code in production code" rule.
> 
> The code *IS* compiled into the application.  The GDB stub is always compiled in.
> 
> The stub is NOT debug code.  It enables the developer to examine the system, set
> breakpoints, tracepoints, watchpoints, or whatever.
> 
> Another rule is to test what you ship.  You don't want to debug with the stub removed. 
> And if a problem is found in testing, it's much easier to track it down if the stub is
> compiled in.  The stub is always compiled in.  It is part of the shipped product.

That's fine if your security mechanisms are good enough to prevent unauthorized access to the stub in a production setting.  That's the issue that the "don't ship debug code" rule is concerned with.

	paul

RE: (hardware) watchpoints and actions

[duane]

duane> a) Most "gdb-server" type things are rather "dumb" they do not
know what symbols are.

david> [snip; see] maint agent [snip]

david> Our stubs (both old and new) contain substantial tracepoint
support. 

Hmm, you learn something every day. 

I am very familiar with several gdbserver JTAG type debug environments
(mostly bare metal), where this feature is *NOT* supported. I have spent
very little time in the linux based gdb-server, hence my ignorance of
this feature.

david> The code *IS* compiled into the application. The GDB stub is
always compiled in. 

Yes, that is quite a different environment then my background. 

Paul @ Dell - brings up another point - security/access control to the
GDB stub feature. 

In your case "compiled-in-gdb-stub" is a product requirement, and/or
product feature. 

In the space constrained world (ie: micro controller, and/or bare-metal)

(1) you often don't have code/ram space for items like this, 
(2) for performance reasons, this code must be removed
(3) There is no support for this in the underling operating system (if
there is one)
(4) Or the GDB server exists external to the target (ie: JTAG)

Each of us come from different backgrounds.

-Duane.

RE: (hardware) watchpoints and actions

[duane]

>> The goal is to run at full speed until a watched location is 
>> accessed in some prescribed manner. At that time, like a 
>> tracepoint, execute some actions to collect information 
>> and then continue at full speed.

This sounds like a macro to execute on a (data|code) breakpoint.

    https://sourceware.org/gdb/onlinedocs/gdb/Break-Commands.html

Doesn't this already exist? If not, what part of this existing feature
needs to be fixed?

Or are you saying that the watchpoints do not support macros like as
described above?

If so, it seems the UI solution if needed would be the same as the
existing breakpoint-commands.

-Duane.

RE: (hardware) watchpoints and actions

[taylor, david]

    From: duane@duaneellis.com [mailto:duane@duaneellis.com] 
    Sent: Wednesday, November 11, 2015 5:02 PM
    To: taylor, david; gdb@sourceware.org
    Subject: RE: (hardware) watchpoints and actions

    >> The goal is to run at full speed until a watched location is accessed 
    >> in some prescribed manner. At that time, like a tracepoint, execute 
    >> some actions to collect information and then continue at full speed.

    This sounds like a macro to execute on a (data|code) breakpoint.

        https://sourceware.org/gdb/onlinedocs/gdb/Break-Commands.html

    Doesn't this already exist? If not, what part of this existing feature needs to be fixed?

    Or are you saying that the watchpoints do not support macros like as described above?

    If so, it seems the UI solution if needed would be the same as the existing breakpoint-commands.

    -Duane.

The beauty of tracepoints, and part of their reason for existing, is to have low
impact on the system being debugged.

Consider a remote target that you are communicating with via the remote protocol.
If the target's GDB stub does not support breakpoint commands or you wish to run
a command that is not convertible to a string of agent expression opcodes, then to
run the command there is a lot of back and forth between GDB and the target.

Further, GDB must remain connected to the target.

By contrast, when a tracepoint is hit, there is NO back and forth with GDB.
GDB might not even be connected to the target.  The target executes a series of
commands that were expressed as strings of agent expression opcodes and arguments.
A command might be to collect registers, or collect a particular variable or collect
some stack.  After the commands execute the program continues -- there is no
explicit 'continue' command.

The information collected from a particular hit of a tracepoint is called a trace frame.
Collectively, the trace frames are stored in a trace frame buffer.

Later GDB can reconnect to the target and examine the collected trace frames.

The idea is to support this for data, not just instructions.

Now, data is harder than instructions.  With instructions, you insert a breakpoint
Instruction, and when it is hit, you collect the information and then either do a
displaced step or replace the breakpoint with the instruction, single step, and put
the breakpoint back.  

With data, to do it at reasonable speed you need hardware support.  And our
hardware supports a small number of 'watched' locations.

If you try to trace a data location, GDB will report an error.  So, currently, our GUI
will take the expression given by the user, determine (by asking GDB) whether
the address is a data or instruction address.  If it is instruction, it acts as it has
for some time.  If it is data, it tells GDB to trace *address (which works).   If the user
did not specify a size to watch, then it uses GDB to get the size of the data item
(think: sizeof); if GDB doesn't know the size, it uses a default value for the size.
It currently uses the tracepoint step count [read: kludge] to pass the size to the
GDB stub.

The stub knows the memory layout, so it can easily determine whether the address
Is text or data and interpret the rest of the QTDV packet accordingly.

The above is somewhat tolerable when using the GUI.  But, I'd like to do it without
kludges.  I'd like data tracepoints to be first class citizens, not kludges.  Some people
[read: me, but there are others, too] like to use either the bare GDB or to use GDB
from within Emacs.  [I don't need no stinking GUI for GDB, I have Emacs.]

My preference would be for a new command -- dtrace, maybe?, for data tracepoints.
I'm less clear on what the interface should be for specifying the size.    I think passcount
makes sense; stepcount is, to me, less clear.  For the stub, something like
DataTracepoints[+-] as a qSupported response to determine whether the stub
supports data tracepoints, and I am ambivalent on whether there should be a new
remote protocol message or continue to use QTDV.

Hopefully that clarifies a few things.

(hardware) watchpoints and actions

[taylor, david]

For instructions, GDB has breakpoints and tracepoints.  For data, GDB has
watchpoints.  It would be nice if there was a data equivalent of tracepoints.
There is processor support for a limited number of small locations to be watched.
The goal is to run at full speed until a watched location is accessed in some
prescribed manner.  At that time, like a tracepoint, execute some actions to
collect information and then continue at full speed.

It's sort of like a hardware watchpoint with actions attached.  For lack of a better
name, we call them 'data tracepoints'.

The current implementation is somewhat of a kludge and done without any GDB
modifications.  The GUI checks whether the address given for the tracepoint is a
code address or a data address.  For instructions, it does what it has always done,
for data, it does some kludges to avoid getting an error from GDB and also 'overloads'
some tracepoint fields.

I'd like for it to be a part of GDB and also to avoid the kludges.

Two questions immediately come to mind:

. what should the GDB user interface be?

. what should the remote protocol messages be?

Opinions?  Thoughts?

Re: (hardware) watchpoints and actions

[Yao Qi]

"taylor, david" <david.taylor@emc.com> writes:

> For instructions, GDB has breakpoints and tracepoints.  For data, GDB has
> watchpoints.  It would be nice if there was a data equivalent of tracepoints.
> There is processor support for a limited number of small locations to
> be watched.
> The goal is to run at full speed until a watched location is accessed in some
> prescribed manner.  At that time, like a tracepoint, execute some actions to
> collect information and then continue at full speed.

This is a useful feature IMO.  However, do we deal with the local
variable?  Like watchpoint, GDB should stop watching the address when
the program goes out of the scope of the local variable.

If GDB can only trace global variable, that is acceptable to me.

>
> I'd like for it to be a part of GDB and also to avoid the kludges.
>
> Two questions immediately come to mind:
>
> . what should the GDB user interface be?

In CLI, we've already had "trace", "ftrace" and "strace", so the command
can be "dtrace" (or "mtrace" if "dtrace" is misleading).  In MI,
-break-insert is to create breakpoint/tracepoint, probably we can add
"-m" option for creating "data tracepoint".

Beside tracepoint creation, we should think about finding traceframes
via command 'tfind'.  Nowadays, we have 'tfind pc', 'tfind outside' and
'tfind range'.  How do these commands handle trace frames collected by
"data tracepoint"? or what do 'tfind outside' and 'tfind range' mean
regarding "data tracepoint"?  We need to update the semantics "QTFrame"
packet accordingly.

>
> . what should the remote protocol messages be?
>

We can reuse and extend QTDP to download "data tracepoint".

> Opinions?  Thoughts?

-- 
Yao (齐尧)