[PATCH v2] tracepoint: add new trace command "printf" and agent expression "printf" [0]

[PATCH v2] tracepoint: add new trace command "printf" and agent expression "printf" [0]

[Hui Zhu]

Hi guys,

The prev version patches were reverted because I misunderstood the
Tom's comments and checked in the patch.

So I send a new mail for review.

I add a new patch add new trace command "printf" and agent expression
"printf" in gdb, gdbserver and doc.

This printf is same with the simple gdb command, it can do formatted
output.  But it will happen in gdbserver part when it break by a
tracepoint.
Then the user can get the format value that he want in tracepint.  It
will be more easy and clear to handle the bug sometimes.

About why I add printf to the tracepoint, I have 2 reasons:
1. The gdb and gdbserver connect through a low speed net.  Sometimes,
the debug target that I use is in the other side of the earth.
The breakpoint commands "printf" is too slow for that issue, because
each time the inferior is break by the breakpoint, gdbserver need send
the rsp package to gdb, and gdb will get the data that "printf" need
though low speed net from gdbsever.  And sometime, it will disconnect.
But if through tracepoint, I will not have this trouble.  I can "set
disconnected-tracing on" to handle the network disconnect issue.  I
still need to get the value from inferior through tfind and other
commands.  It is still be affect by the low speed network.  So I make
the tracepoint "printf" to handle it.

2.  KGTP(https://code.google.com/p/kgtp/) just support the gdb
tracepoint rsp commands.  For not stop the Linux the Kernel.  It
doesn't support the breakpoint.
So if it want directly show the Kernel val value, it need "printf".
This printf will be very powerful that can set most part of Kernel and
we can set condition for it.
And in https://code.google.com/p/kgtp/wiki/HOWTO#Offline_debug,  we
can dump the gdbrsp package to a file and send to Kernel.  Then kernel
can be debug without a local gdb or a remote connect.   But user still
need copy the trace file to pc that have GDB.  But if support
tracepoint "printf", we will not need do that.
BTW, the kgtp have supported the agent expression "printf".

About the command part, I use the "printf" instead add a new commands
because the behavior of  this command is same with printf. They will
use the same function string_printf(update from ui_printf) to parse
the command arg.

To support the printf command, I add a new agent expression 0x31
printf, the format for it is:
0x31(op_printf) + arg(1/0) + format string with end by 0x0.
The arg is the number of argument of printf.  It will only be 1 (one
argument) or 0 (no argument).  I make it cannot have more than one
argument because I cannot found a good way to handle va_list that send
arguments to vprintf.
The format string with end by 0x0 is the simple format string.  It end
by 0x0 then the gdbserver can use it directly.

Example:
(gdb) trace 16
During symbol reading, DW_AT_name missing from DW_TAG_base_type.
Tracepoint 1 at 0x4004c3: file 1.c, line 16.
(gdb) tvariable $c
Trace state variable $c created, with initial value 0.
(gdb) actions
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>printf "%d 0x%lx %d\n",$c=$c+1,$rax,argc
>end
(gdb) target remote localhost:1234
(gdb) tstart
(gdb) c

gdbserver/gdbserver  localhost:1234 ./a.out
Process ./a.out created; pid = 25804
Listening on port 1234
Remote debugging from host 127.0.0.1
1 0x7f2cb8711ec8 1
2 0x7f2cb8711ec8 2
3 0x7f2cb8711ec8 3
4 0x7f2cb8711ec8 4
5 0x7f2cb8711ec8 5
6 0x7f2cb8711ec8 6
7 0x7f2cb8711ec8 7

Please help me review the patches.

Thanks,
Hui

Re: [PATCH v2] tracepoint: add new trace command "printf" and agent expression "printf" [0]

[Hui Zhu]

Ping. :)

Thanks,
Hui

On Thu, Feb 24, 2011 at 16:36, Hui Zhu <teawater@gmail.com> wrote:
> Hi guys,
>
> The prev version patches were reverted because I misunderstood the
> Tom's comments and checked in the patch.
>
> So I send a new mail for review.
>
> I add a new patch add new trace command "printf" and agent expression
> "printf" in gdb, gdbserver and doc.
>
> This printf is same with the simple gdb command, it can do formatted
> output.  But it will happen in gdbserver part when it break by a
> tracepoint.
> Then the user can get the format value that he want in tracepint.  It
> will be more easy and clear to handle the bug sometimes.
>
> About why I add printf to the tracepoint, I have 2 reasons:
> 1. The gdb and gdbserver connect through a low speed net.  Sometimes,
> the debug target that I use is in the other side of the earth.
> The breakpoint commands "printf" is too slow for that issue, because
> each time the inferior is break by the breakpoint, gdbserver need send
> the rsp package to gdb, and gdb will get the data that "printf" need
> though low speed net from gdbsever.  And sometime, it will disconnect.
> But if through tracepoint, I will not have this trouble.  I can "set
> disconnected-tracing on" to handle the network disconnect issue.  I
> still need to get the value from inferior through tfind and other
> commands.  It is still be affect by the low speed network.  So I make
> the tracepoint "printf" to handle it.
>
> 2.  KGTP(https://code.google.com/p/kgtp/) just support the gdb
> tracepoint rsp commands.  For not stop the Linux the Kernel.  It
> doesn't support the breakpoint.
> So if it want directly show the Kernel val value, it need "printf".
> This printf will be very powerful that can set most part of Kernel and
> we can set condition for it.
> And in https://code.google.com/p/kgtp/wiki/HOWTO#Offline_debug,  we
> can dump the gdbrsp package to a file and send to Kernel.  Then kernel
> can be debug without a local gdb or a remote connect.   But user still
> need copy the trace file to pc that have GDB.  But if support
> tracepoint "printf", we will not need do that.
> BTW, the kgtp have supported the agent expression "printf".
>
> About the command part, I use the "printf" instead add a new commands
> because the behavior of  this command is same with printf. They will
> use the same function string_printf(update from ui_printf) to parse
> the command arg.
>
> To support the printf command, I add a new agent expression 0x31
> printf, the format for it is:
> 0x31(op_printf) + arg(1/0) + format string with end by 0x0.
> The arg is the number of argument of printf.  It will only be 1 (one
> argument) or 0 (no argument).  I make it cannot have more than one
> argument because I cannot found a good way to handle va_list that send
> arguments to vprintf.
> The format string with end by 0x0 is the simple format string.  It end
> by 0x0 then the gdbserver can use it directly.
>
> Example:
> (gdb) trace 16
> During symbol reading, DW_AT_name missing from DW_TAG_base_type.
> Tracepoint 1 at 0x4004c3: file 1.c, line 16.
> (gdb) tvariable $c
> Trace state variable $c created, with initial value 0.
> (gdb) actions
> Enter actions for tracepoint 1, one per line.
> End with a line saying just "end".
>>printf "%d 0x%lx %d\n",$c=$c+1,$rax,argc
>>end
> (gdb) target remote localhost:1234
> (gdb) tstart
> (gdb) c
>
> gdbserver/gdbserver  localhost:1234 ./a.out
> Process ./a.out created; pid = 25804
> Listening on port 1234
> Remote debugging from host 127.0.0.1
> 1 0x7f2cb8711ec8 1
> 2 0x7f2cb8711ec8 2
> 3 0x7f2cb8711ec8 3
> 4 0x7f2cb8711ec8 4
> 5 0x7f2cb8711ec8 5
> 6 0x7f2cb8711ec8 6
> 7 0x7f2cb8711ec8 7
>
> Please help me review the patches.
>
> Thanks,
> Hui
>

Re: [PATCH v2] tracepoint: add new trace command "printf" and agent expression "printf" [0]

[Hui Zhu]

Hi guys,

Does anyone want review or reviewing this patches?

If not, I will begin to make KGTP support printk with itself.

Thanks,
Hui

On Thu, Feb 24, 2011 at 16:36, Hui Zhu <teawater@gmail.com> wrote:
> Hi guys,
>
> The prev version patches were reverted because I misunderstood the
> Tom's comments and checked in the patch.
>
> So I send a new mail for review.
>
> I add a new patch add new trace command "printf" and agent expression
> "printf" in gdb, gdbserver and doc.
>
> This printf is same with the simple gdb command, it can do formatted
> output.  But it will happen in gdbserver part when it break by a
> tracepoint.
> Then the user can get the format value that he want in tracepint.  It
> will be more easy and clear to handle the bug sometimes.
>
> About why I add printf to the tracepoint, I have 2 reasons:
> 1. The gdb and gdbserver connect through a low speed net.  Sometimes,
> the debug target that I use is in the other side of the earth.
> The breakpoint commands "printf" is too slow for that issue, because
> each time the inferior is break by the breakpoint, gdbserver need send
> the rsp package to gdb, and gdb will get the data that "printf" need
> though low speed net from gdbsever.  And sometime, it will disconnect.
> But if through tracepoint, I will not have this trouble.  I can "set
> disconnected-tracing on" to handle the network disconnect issue.  I
> still need to get the value from inferior through tfind and other
> commands.  It is still be affect by the low speed network.  So I make
> the tracepoint "printf" to handle it.
>
> 2.  KGTP(https://code.google.com/p/kgtp/) just support the gdb
> tracepoint rsp commands.  For not stop the Linux the Kernel.  It
> doesn't support the breakpoint.
> So if it want directly show the Kernel val value, it need "printf".
> This printf will be very powerful that can set most part of Kernel and
> we can set condition for it.
> And in https://code.google.com/p/kgtp/wiki/HOWTO#Offline_debug,  we
> can dump the gdbrsp package to a file and send to Kernel.  Then kernel
> can be debug without a local gdb or a remote connect.   But user still
> need copy the trace file to pc that have GDB.  But if support
> tracepoint "printf", we will not need do that.
> BTW, the kgtp have supported the agent expression "printf".
>
> About the command part, I use the "printf" instead add a new commands
> because the behavior of  this command is same with printf. They will
> use the same function string_printf(update from ui_printf) to parse
> the command arg.
>
> To support the printf command, I add a new agent expression 0x31
> printf, the format for it is:
> 0x31(op_printf) + arg(1/0) + format string with end by 0x0.
> The arg is the number of argument of printf.  It will only be 1 (one
> argument) or 0 (no argument).  I make it cannot have more than one
> argument because I cannot found a good way to handle va_list that send
> arguments to vprintf.
> The format string with end by 0x0 is the simple format string.  It end
> by 0x0 then the gdbserver can use it directly.
>
> Example:
> (gdb) trace 16
> During symbol reading, DW_AT_name missing from DW_TAG_base_type.
> Tracepoint 1 at 0x4004c3: file 1.c, line 16.
> (gdb) tvariable $c
> Trace state variable $c created, with initial value 0.
> (gdb) actions
> Enter actions for tracepoint 1, one per line.
> End with a line saying just "end".
>>printf "%d 0x%lx %d\n",$c=$c+1,$rax,argc
>>end
> (gdb) target remote localhost:1234
> (gdb) tstart
> (gdb) c
>
> gdbserver/gdbserver  localhost:1234 ./a.out
> Process ./a.out created; pid = 25804
> Listening on port 1234
> Remote debugging from host 127.0.0.1
> 1 0x7f2cb8711ec8 1
> 2 0x7f2cb8711ec8 2
> 3 0x7f2cb8711ec8 3
> 4 0x7f2cb8711ec8 4
> 5 0x7f2cb8711ec8 5
> 6 0x7f2cb8711ec8 6
> 7 0x7f2cb8711ec8 7
>
> Please help me review the patches.
>
> Thanks,
> Hui
>

Re: [PATCH v2] tracepoint: add new trace command "printf" and agent expression "printf" [0]

[Stan Shebs]

On 2/24/11 12:36 AM, Hui Zhu wrote:
> Hi guys,
>
> The prev version patches were reverted because I misunderstood the
> Tom's comments and checked in the patch.
>
> So I send a new mail for review.
>
> I add a new patch add new trace command "printf" and agent expression
> "printf" in gdb, gdbserver and doc.

This one is a tough choice.  As Doug said a while back, it doesn't 
really fit into the overall tracepoint structure - the basic idea of 
tracepoints is to accumulate data into a buffer, while this change adds 
a tracepoint action that prints back to the host, and doesn't save 
anything at all.  This can actually lose massively, because in the 
absence of GDB roundtrips, a tracepoint can easily rack up a million 
hits in a second (I've done it) - but a million printfs coming back to 
the host is going to choke both the network and GDB.  In fact it reduces 
debugging performance considerably; while still faster than a breakpoint 
with a command list, most of the tracepoint performance win is lost to 
the I/O back-and-forth.

The flip side is that it does seem like a useful pragmatic addition.  
When a tracepoint experiment runs, there is a bit of guessing at how 
it's going.  You can do tstatus commands, and maybe tfind will work, but 
neither has the live update advantage of the familiar printf.  So I like 
the ability for users to have a sort of live monitoring of their trace 
run, even at the risk of massive spew back up to GDB.

Now if these are the only considerations, I would say to put this in, 
give it a spin, see how it works for users.  But this is not an isolated 
addition.

The first other consideration is that new bytecodes amount to permanent 
commitments.  We now have a handful of targets that think they know what 
the valid bytecodes are, but if we're adding new bytecodes to some but 
not others, then we need to start doing some new infrastructure for the 
target to tell the host about its supported bytecodes.

The second consideration is that we have proposals for target-side 
breakpoint handling on the table.  Mentor now has a contract to do some 
work on this, and we're going to be kicking off public design discussion 
within the next few weeks.  So if we can do random breakpoint actions on 
the target, with no GDB involvement, I think it removes a large part of 
the motivation for this patch.

So overall I think we should put this on the back burner for now, and 
focus on target-side breakpoint handling instead.  There are some 
specific issues that would have to addressed to make this particular 
patch acceptable (the pervasive 100-byte limitation is hard to justify 
for instance), but I'd rather put our brain power into achieving the 
real goal, rather than piggybacking on tracepoints.

Stan

> This printf is same with the simple gdb command, it can do formatted
> output.  But it will happen in gdbserver part when it break by a
> tracepoint.
> Then the user can get the format value that he want in tracepint.  It
> will be more easy and clear to handle the bug sometimes.
>
> About why I add printf to the tracepoint, I have 2 reasons:
> 1. The gdb and gdbserver connect through a low speed net.  Sometimes,
> the debug target that I use is in the other side of the earth.
> The breakpoint commands "printf" is too slow for that issue, because
> each time the inferior is break by the breakpoint, gdbserver need send
> the rsp package to gdb, and gdb will get the data that "printf" need
> though low speed net from gdbsever.  And sometime, it will disconnect.
> But if through tracepoint, I will not have this trouble.  I can "set
> disconnected-tracing on" to handle the network disconnect issue.  I
> still need to get the value from inferior through tfind and other
> commands.  It is still be affect by the low speed network.  So I make
> the tracepoint "printf" to handle it.
>
> 2.  KGTP(https://code.google.com/p/kgtp/) just support the gdb
> tracepoint rsp commands.  For not stop the Linux the Kernel.  It
> doesn't support the breakpoint.
> So if it want directly show the Kernel val value, it need "printf".
> This printf will be very powerful that can set most part of Kernel and
> we can set condition for it.
> And in https://code.google.com/p/kgtp/wiki/HOWTO#Offline_debug,  we
> can dump the gdbrsp package to a file and send to Kernel.  Then kernel
> can be debug without a local gdb or a remote connect.   But user still
> need copy the trace file to pc that have GDB.  But if support
> tracepoint "printf", we will not need do that.
> BTW, the kgtp have supported the agent expression "printf".
>
> About the command part, I use the "printf" instead add a new commands
> because the behavior of  this command is same with printf. They will
> use the same function string_printf(update from ui_printf) to parse
> the command arg.
>
> To support the printf command, I add a new agent expression 0x31
> printf, the format for it is:
> 0x31(op_printf) + arg(1/0) + format string with end by 0x0.
> The arg is the number of argument of printf.  It will only be 1 (one
> argument) or 0 (no argument).  I make it cannot have more than one
> argument because I cannot found a good way to handle va_list that send
> arguments to vprintf.
> The format string with end by 0x0 is the simple format string.  It end
> by 0x0 then the gdbserver can use it directly.
>
> Example:
> (gdb) trace 16
> During symbol reading, DW_AT_name missing from DW_TAG_base_type.
> Tracepoint 1 at 0x4004c3: file 1.c, line 16.
> (gdb) tvariable $c
> Trace state variable $c created, with initial value 0.
> (gdb) actions
> Enter actions for tracepoint 1, one per line.
> End with a line saying just "end".
>> printf "%d 0x%lx %d\n",$c=$c+1,$rax,argc
>> end
> (gdb) target remote localhost:1234
> (gdb) tstart
> (gdb) c
>
> gdbserver/gdbserver  localhost:1234 ./a.out
> Process ./a.out created; pid = 25804
> Listening on port 1234
> Remote debugging from host 127.0.0.1
> 1 0x7f2cb8711ec8 1
> 2 0x7f2cb8711ec8 2
> 3 0x7f2cb8711ec8 3
> 4 0x7f2cb8711ec8 4
> 5 0x7f2cb8711ec8 5
> 6 0x7f2cb8711ec8 6
> 7 0x7f2cb8711ec8 7
>
> Please help me review the patches.
>
> Thanks,
> Hui
>

Re: [PATCH v2] tracepoint: add new trace command "printf" and agent expression "printf" [0]

[Hui Zhu]

Hi Stan,

Thanks for your reply.

On Wed, Mar 16, 2011 at 12:48, Stan Shebs <stan@codesourcery.com> wrote:
> On 2/24/11 12:36 AM, Hui Zhu wrote:
>>
>> Hi guys,
>>
>> The prev version patches were reverted because I misunderstood the
>> Tom's comments and checked in the patch.
>>
>> So I send a new mail for review.
>>
>> I add a new patch add new trace command "printf" and agent expression
>> "printf" in gdb, gdbserver and doc.
>
> This one is a tough choice.  As Doug said a while back, it doesn't really
> fit into the overall tracepoint structure - the basic idea of tracepoints is
> to accumulate data into a buffer, while this change adds a tracepoint action
> that prints back to the host, and doesn't save anything at all.  This can
> actually lose massively, because in the absence of GDB roundtrips, a
> tracepoint can easily rack up a million hits in a second (I've done it) -
> but a million printfs coming back to the host is going to choke both the
> network and GDB.  In fact it reduces debugging performance considerably;
> while still faster than a breakpoint with a command list, most of the
> tracepoint performance win is lost to the I/O back-and-forth.
>
"but a million printfs coming back to the host is going to choke both the
> network and GDB."

The printf will not send back to GDB.  the server (gdbserver or kgtp)
will handle it with itself.

So it will not have the IO trouble or something else.  And it will
very fast and low cost.

Thanks,
Hui

> The flip side is that it does seem like a useful pragmatic addition.  When a
> tracepoint experiment runs, there is a bit of guessing at how it's going.
>  You can do tstatus commands, and maybe tfind will work, but neither has the
> live update advantage of the familiar printf.  So I like the ability for
> users to have a sort of live monitoring of their trace run, even at the risk
> of massive spew back up to GDB.
>
> Now if these are the only considerations, I would say to put this in, give
> it a spin, see how it works for users.  But this is not an isolated
> addition.
>
> The first other consideration is that new bytecodes amount to permanent
> commitments.  We now have a handful of targets that think they know what the
> valid bytecodes are, but if we're adding new bytecodes to some but not
> others, then we need to start doing some new infrastructure for the target
> to tell the host about its supported bytecodes.
>
> The second consideration is that we have proposals for target-side
> breakpoint handling on the table.  Mentor now has a contract to do some work
> on this, and we're going to be kicking off public design discussion within
> the next few weeks.  So if we can do random breakpoint actions on the
> target, with no GDB involvement, I think it removes a large part of the
> motivation for this patch.
>
> So overall I think we should put this on the back burner for now, and focus
> on target-side breakpoint handling instead.  There are some specific issues
> that would have to addressed to make this particular patch acceptable (the
> pervasive 100-byte limitation is hard to justify for instance), but I'd
> rather put our brain power into achieving the real goal, rather than
> piggybacking on tracepoints.
>
> Stan
>
>> This printf is same with the simple gdb command, it can do formatted
>> output.  But it will happen in gdbserver part when it break by a
>> tracepoint.
>> Then the user can get the format value that he want in tracepint.  It
>> will be more easy and clear to handle the bug sometimes.
>>
>> About why I add printf to the tracepoint, I have 2 reasons:
>> 1. The gdb and gdbserver connect through a low speed net.  Sometimes,
>> the debug target that I use is in the other side of the earth.
>> The breakpoint commands "printf" is too slow for that issue, because
>> each time the inferior is break by the breakpoint, gdbserver need send
>> the rsp package to gdb, and gdb will get the data that "printf" need
>> though low speed net from gdbsever.  And sometime, it will disconnect.
>> But if through tracepoint, I will not have this trouble.  I can "set
>> disconnected-tracing on" to handle the network disconnect issue.  I
>> still need to get the value from inferior through tfind and other
>> commands.  It is still be affect by the low speed network.  So I make
>> the tracepoint "printf" to handle it.
>>
>> 2.  KGTP(https://code.google.com/p/kgtp/) just support the gdb
>> tracepoint rsp commands.  For not stop the Linux the Kernel.  It
>> doesn't support the breakpoint.
>> So if it want directly show the Kernel val value, it need "printf".
>> This printf will be very powerful that can set most part of Kernel and
>> we can set condition for it.
>> And in https://code.google.com/p/kgtp/wiki/HOWTO#Offline_debug,  we
>> can dump the gdbrsp package to a file and send to Kernel.  Then kernel
>> can be debug without a local gdb or a remote connect.   But user still
>> need copy the trace file to pc that have GDB.  But if support
>> tracepoint "printf", we will not need do that.
>> BTW, the kgtp have supported the agent expression "printf".
>>
>> About the command part, I use the "printf" instead add a new commands
>> because the behavior of  this command is same with printf. They will
>> use the same function string_printf(update from ui_printf) to parse
>> the command arg.
>>
>> To support the printf command, I add a new agent expression 0x31
>> printf, the format for it is:
>> 0x31(op_printf) + arg(1/0) + format string with end by 0x0.
>> The arg is the number of argument of printf.  It will only be 1 (one
>> argument) or 0 (no argument).  I make it cannot have more than one
>> argument because I cannot found a good way to handle va_list that send
>> arguments to vprintf.
>> The format string with end by 0x0 is the simple format string.  It end
>> by 0x0 then the gdbserver can use it directly.
>>
>> Example:
>> (gdb) trace 16
>> During symbol reading, DW_AT_name missing from DW_TAG_base_type.
>> Tracepoint 1 at 0x4004c3: file 1.c, line 16.
>> (gdb) tvariable $c
>> Trace state variable $c created, with initial value 0.
>> (gdb) actions
>> Enter actions for tracepoint 1, one per line.
>> End with a line saying just "end".
>>>
>>> printf "%d 0x%lx %d\n",$c=$c+1,$rax,argc
>>> end
>>
>> (gdb) target remote localhost:1234
>> (gdb) tstart
>> (gdb) c
>>
>> gdbserver/gdbserver  localhost:1234 ./a.out
>> Process ./a.out created; pid = 25804
>> Listening on port 1234
>> Remote debugging from host 127.0.0.1
>> 1 0x7f2cb8711ec8 1
>> 2 0x7f2cb8711ec8 2
>> 3 0x7f2cb8711ec8 3
>> 4 0x7f2cb8711ec8 4
>> 5 0x7f2cb8711ec8 5
>> 6 0x7f2cb8711ec8 6
>> 7 0x7f2cb8711ec8 7
>>
>> Please help me review the patches.
>>
>> Thanks,
>> Hui
>>
>
>