[RFC] Make target_read_string faster over high-latency links.

[RFC] Make target_read_string faster over high-latency links.

[Paul Pluzhnikov]

Greetings,

Here at Google we sometimes debug binaries running in production data
centers via remote gdbserver.

When GDB is running in Mountain View, California and the gdbserver is running
in Taiwan, the speed-of-light fiber round trip delay is approximately 110ms,
and the actual ping time I see is slightly better than 200ms.

Unfortunately (for us), GDB is not tuned for such high-latency links.

In particular, I see the following timestamped packet traces:

...
00:00:25 Sending packet: $m2b6b7958a7b8,28#3e...Packet received: 00c078796b2b000090a758796b2b0000f85d99796b2b000000509c796b2b0000d8b678796b2b0000
00:00:25 Sending packet: $m2b6b7958a790,4#d7...Packet received: 2f757372
00:00:25 Sending packet: $m2b6b7958a794,4#db...Packet received: 2f677274
00:00:26 Sending packet: $m2b6b7958a798,4#df...Packet received: 652f7632
00:00:26 Sending packet: $m2b6b7958a79c,4#0a...Packet received: 2f6c6962
00:00:26 Sending packet: $m2b6b7958a7a0,4#ff...Packet received: 36342f6c
00:00:26 Sending packet: $m2b6b7958a7a4,4#03...Packet received: 69626372
00:00:26 Sending packet: $m2b6b7958a7a8,4#07...Packet received: 7970742e
00:00:26 Sending packet: $m2b6b7958a7ac,4#32...Packet received: 736f2e31
00:00:27 Sending packet: $m2b6b7958a7b0,4#00...Packet received: 00000000
00:00:27 Sending packet: $m2b6b799c5000,28#00...Packet received: 00609c796b2b000078ac58796b2b0000a08dbc796b2b0000e0549c796b2b0000b8a758796b2b0000
00:00:27 Sending packet: $m2b6b7958ac78,4#09...Packet received: 2f757372
00:00:27 Sending packet: $m2b6b7958ac7c,4#34...Packet received: 2f677274
00:00:27 Sending packet: $m2b6b7958ac80,4#02...Packet received: 652f7632
00:00:27 Sending packet: $m2b6b7958ac84,4#06...Packet received: 2f6c6962
00:00:28 Sending packet: $m2b6b7958ac88,4#0a...Packet received: 36342f6c
00:00:28 Sending packet: $m2b6b7958ac8c,4#35...Packet received: 6962646c
00:00:28 Sending packet: $m2b6b7958ac90,4#03...Packet received: 2e736f2e
00:00:28 Sending packet: $m2b6b7958ac94,4#07...Packet received: 32000000
...

This is GDB walking through the shared library list, and reading shared
library names from gdbserver via target_read_string 4 bytes at a time ;-(

It takes approximately 15s to read 10 (relatively short) pathnames, and
this is repeated twice, for a total of 30s, which is about 1/3 of the
total time it takes to attach the remote process.

I have a patch that changes minimum size to read in target_read_string()
to MIN(len, 128), and with that patch I can save quite a few round trips:

...
00:00:29 Sending packet: $m2b6b7958a7b8,28#3e...Packet received: 00c078796b2b000090a758796b2b0000f85d99796b2b000000509c796b2b0000d8b678796b2b0000
00:00:29 Sending packet: $m2b6b7958a790,80#0b...Packet received: 2f7573722f677274652f76322f6c696236342f6c696263727970742e736f2e31000000000000000000c078796b2b000090a758796b2b0000f85d99796b2b000000509c796b2b0000d8b678796b2b0000b8a758796b2b0000000000000000000028ac58796b2b00000000000000000000f85d99796b2b0000a85e99796b2b0000
00:00:29 Sending packet: $m2b6b799c5000,28#00...Packet received: 00609c796b2b000078ac58796b2b0000a08dbc796b2b0000e0549c796b2b0000b8a758796b2b0000
00:00:29 Sending packet: $m2b6b7958ac78,80#3d...Packet received: 2f7573722f677274652f76322f6c696236342f6c6962646c2e736f2e32000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00:00:30 Sending packet: $m2b6b799c54e0,28#39...Packet received: 00a0bc796b2b0000b8549c796b2b0000881dde796b2b0000c0599c796b2b000000509c796b2b0000
00:00:30 Sending packet: $m2b6b799c54b8,80#3c...Packet received: 2f7573722f677274652f76322f6c696236342f6c6962707468726561642e736f2e3000000000000000a0bc796b2b0000b8549c796b2b0000881dde796b2b0000c0599c796b2b000000509c796b2b0000e0549c796b2b0000000000000000000050599c796b2b00000000000000000000981dde796b2b0000481ede796b2b0000
...

Questions:

1. is it ok to read strings 128 bytes at a time, or are there scenarios
   (JTAG?) where the latency is low but throughput is also low (and so reading
   "unnecessary" data is expensive) ?

1a. If yes, would it be acceptable to make this size a runtime-configurable
    parameter?

2. Current target_read_string aligns reads on a 4-byte boundary. Code
   could be simplified quite a bit if it didn't do that.
   Are there targets where such alignment is required?

Thanks,
--
Paul Pluzhnikov

Re: [RFC] Make target_read_string faster over high-latency links.

[Tom Tromey]

>>>>> "Paul" == Paul Pluzhnikov <ppluzhnikov@google.com> writes:

Paul> 1. is it ok to read strings 128 bytes at a time, or are there
Paul> scenarios (JTAG?) where the latency is low but throughput is also
Paul> low (and so reading "unnecessary" data is expensive) ?

I don't know, but I assume so.

Paul> 1a. If yes, would it be acceptable to make this size a
Paul> runtime-configurable parameter?

Yes.

Paul> 2. Current target_read_string aligns reads on a 4-byte boundary. Code
Paul>    could be simplified quite a bit if it didn't do that.
Paul>    Are there targets where such alignment is required?

I don't know this either, sorry.

What about making it possible for gdbserver to do the string-reading
itself, with a fallback to the existing code for older versions?  Then
you don't need a parameter or any tuning.

Tom

Re: [RFC] Make target_read_string faster over high-latency links.

[Paul Pluzhnikov]

On Mon, Jul 18, 2011 at 10:09 AM, Tom Tromey <tromey@redhat.com> wrote:

> Paul> 2. Current target_read_string aligns reads on a 4-byte boundary. Code
> Paul>    could be simplified quite a bit if it didn't do that.
> Paul>    Are there targets where such alignment is required?
>
> I don't know this either, sorry.

In gdb.texinfo, Node Packets:

  @item m @var{addr},@var{length}
  @cindex @samp{m} packet
  Read @var{length} bytes of memory starting at address @var{addr}.
  Note that @var{addr} may not be aligned to any particular boundary.

I *think* this suggests that alignment is not really necessary here.

> What about making it possible for gdbserver to do the string-reading
> itself, with a fallback to the existing code for older versions?  Then
> you don't need a parameter or any tuning.

Just to clarify, you are suggesting a new packet, e.g.
  qStr {addr},{maxlen}
in response to which gdbserver will read up to either terminating NUL,
or maxlen.

That sounds like a good idea, except I would also add {terminator} byte, so
you could read "memory from addr to {terminator} but no more than {maxlen}".

Terminator could be optional.

Don't know if it's really necessary, but it could come in handy if you
want to read up to '\n' or some such.

Thanks,
-- 
Paul Pluzhnikov

Re: [RFC] Make target_read_string faster over high-latency links.

[Paul Pluzhnikov]

On Mon, Jul 18, 2011 at 11:06 AM, Paul Pluzhnikov
<ppluzhnikov@google.com> wrote:
> On Mon, Jul 18, 2011 at 10:09 AM, Tom Tromey <tromey@redhat.com> wrote:
>
> Just to clarify, you are suggesting a new packet, e.g.
>  qStr {addr},{maxlen}
> in response to which gdbserver will read up to either terminating NUL,
> or maxlen.

Ping?

Tom, is that what you had in mind?

Thanks,
-- 
Paul Pluzhnikov

Re: [RFC] Make target_read_string faster over high-latency links.

[Tom Tromey]

>>>>> "Paul" == Paul Pluzhnikov <ppluzhnikov@google.com> writes:

>> Just to clarify, you are suggesting a new packet, e.g.
>>  qStr {addr},{maxlen}
>> in response to which gdbserver will read up to either terminating NUL,
>> or maxlen.

Paul> Ping?
Paul> Tom, is that what you had in mind?

Sorry, I didn't think that you were expecting a response.

I am not all familiar with the gdb remote protocol, but yes, that is
what I was thinking of.

Tom

Re: [RFC] Make target_read_string faster over high-latency links.

[Joel Brobecker]

> Paul> 1. is it ok to read strings 128 bytes at a time, or are there
> Paul> scenarios (JTAG?) where the latency is low but throughput is also
> Paul> low (and so reading "unnecessary" data is expensive) ?
> 
> I don't know, but I assume so.

I think that the risk is to attempt a read that gets past the
readable memory region. I don't remember if GDB is protecting
itself against this sort of thing (by checking the range against
known regions), but I remember having problems on LynxOS for instance,
where we tried to read some data on the stack, tried to read too much
in one go, and ended up getting a failure and no data as a result.
That was a while ago, so may be OBE...

> What about making it possible for gdbserver to do the string-reading
> itself, with a fallback to the existing code for older versions?  Then
> you don't need a parameter or any tuning.

Seems like a good idea to me.

-- 
Joel

Re: [RFC] Make target_read_string faster over high-latency links.

[Paul Pluzhnikov]

On Fri, Jul 22, 2011 at 9:43 AM, Joel Brobecker <brobecker@adacore.com> wrote:

>> Paul> 1. is it ok to read strings 128 bytes at a time, or are there
>> Paul> scenarios (JTAG?) where the latency is low but throughput is also
>> Paul> low (and so reading "unnecessary" data is expensive) ?
>>
>> I don't know, but I assume so.
>
> I think that the risk is to attempt a read that gets past the
> readable memory region.

That shouldn't be a risk (I think): target_read will (is supposed to)
return partial results.

>> What about making it possible for gdbserver to do the string-reading
>> itself, with a fallback to the existing code for older versions?  Then
>> you don't need a parameter or any tuning.
>
> Seems like a good idea to me.

FWIW, additional discussion here:
http://sourceware.org/ml/gdb-patches/2011-07/msg00546.html

-- 
Paul Pluzhnikov

Re: [RFC] Make target_read_string faster over high-latency links.

[Daniel Jacobowitz]

On Fri, Jul 22, 2011 at 12:52 PM, Paul Pluzhnikov
<ppluzhnikov@google.com> wrote:
> On Fri, Jul 22, 2011 at 9:43 AM, Joel Brobecker <brobecker@adacore.com> wrote:
>
>>> Paul> 1. is it ok to read strings 128 bytes at a time, or are there
>>> Paul> scenarios (JTAG?) where the latency is low but throughput is also
>>> Paul> low (and so reading "unnecessary" data is expensive) ?
>>>
>>> I don't know, but I assume so.
>>
>> I think that the risk is to attempt a read that gets past the
>> readable memory region.
>
> That shouldn't be a risk (I think): target_read will (is supposed to)
> return partial results.

The problem's the other way around; you can't always rely on the
target to know what areas of memory not to read.  If you read a big
chunk of a string off of the stack on uClinux (e.g. auxv or
environment), you'll wander up into unmapped memory - and who knows
what will happen then!  (More recent versions of uClinux manage to
catch this in ptrace, thankfully).

-- 
Thanks,
Daniel

Re: [RFC] Make target_read_string faster over high-latency links.

[Jan Kratochvil]

On Fri, 15 Jul 2011 20:07:48 +0200, Paul Pluzhnikov wrote:
> This is GDB walking through the shared library list, and reading shared
> library names from gdbserver via target_read_string 4 bytes at a time ;-(

Just that you do not mention supporting qXfer:libraries:read also on GNU/Linux
platforms (currently used only on MS-Windows), that is to move solib-svr4.c to
gdb/common/ etc.

It is sure a larger change but it could bring even a better performance.


Thanks,
Jan