why I dislike qXfer

why I dislike qXfer

[David Taylor]

There are a number of remote protocol packets of the form

    qXfer:object:read:annex:offset,length

For some object or object/annex combinations, this is a reasonable
interface -- for example, when it is referring to data that does not
change (and often it will reasonably fit within a single remote protocol
reply packet).

An example would be qXfer:features.  My expectation is that if we
implement this that it will be compiled in strings.  Unless you connect
to a different target or load new code on the target, it is not going to
change.

A board has a set of features and it doesn't typically change.

But, there are some objects for which it is, in my opinion, a poor
interface.

For concreteness, consider the 'threads' object -- as it is the one I'm
currently wrestling with.

Threads come and go.  And the amount of information I want to return per
currently existing thread means that number of threads * bytes of info
per thread exceeds the maximum packet size.  So, multiple requests have
to be issued to get everything.

The system native format for the information is, of course, not XML.
The per-thread data is variable length (e.g., the name field and, in our
case, the 'extra information').  I want to always return the most
current information.  I want to return information on as many threads as
possible.  And I want to minimize the bookkeeping required.

So, while I can avoid having to store the current position, if I am
going to minimize bookkeeping and return the latest information, I have
to pad each entry to the same size and cannot skip empty (read: dead
thread) entries in the system process table.  If the goal is to speed
things up by, in part, having fewer network round trips, having to pad
is not a win.

With the qT{f,s}{STM,P,V} q{f,s}ThreadInfo (and possibly others)
interfaces, nothing needs to be precomputed, and I either start at the
beginning (f -- first) or where the previous request left off (s --
subsequent).

I have to store, per connection, my location.  But, there is no random
reading.  The next request of that flavor will either start at the
beginning (f) or where the last one left off (s).  Reads are sequential.

With the offset,length interface I don't know that reads will be
sequential so I need to pad and leave gaps.

What do people do?

Re: why I dislike qXfer

[Pedro Alves]

On 06/13/2016 07:15 PM, David Taylor wrote:

> With the qT{f,s}{STM,P,V} q{f,s}ThreadInfo (and possibly others)
> interfaces, nothing needs to be precomputed, and I either start at the
> beginning (f -- first) or where the previous request left off (s --
> subsequent).

> I have to store, per connection, my location.  But, there is no random
> reading.  The next request of that flavor will either start at the
> beginning (f) or where the last one left off (s).  Reads are sequential.

If you support non-stop mode, the target is running and list of threads
changes as gdb is iterating.  The "location" thread can exit and you're
left not knowing where to continue from, for example.  To get around that,
generate a stable snapshot when you get the f requests, and serve gdb
requests from that snapshot.

> With the offset,length interface I don't know that reads will be
> sequential so I need to pad and leave gaps.
> 

> What do people do?

Generate a snapshot when gdb requests offset 0.  Then serve requests
from that snapshot.

See handle_qxfer_threads in gdbserver's sources.

Thanks,
Pedro Alves

RE: why I dislike qXfer

[taylor, david]

> From: Pedro Alves [mailto:palves@redhat.com]
> Sent: Monday, June 13, 2016 2:36 PM
> To: taylor, david; gdb@sourceware.org
> Subject: Re: why I dislike qXfer
> 
> On 06/13/2016 07:15 PM, David Taylor wrote:
> 
> > With the qT{f,s}{STM,P,V} q{f,s}ThreadInfo (and possibly others)
> > interfaces, nothing needs to be precomputed, and I either start at the
> > beginning (f -- first) or where the previous request left off (s --
> > subsequent).
> 
> > I have to store, per connection, my location.  But, there is no random
> > reading.  The next request of that flavor will either start at the
> > beginning (f) or where the last one left off (s).  Reads are sequential.
> 
> If you support non-stop mode, the target is running and list of threads
> changes as gdb is iterating.  The "location" thread can exit and you're left not
> knowing where to continue from, for example.  To get around that, generate
> a stable snapshot when you get the f requests, and serve gdb requests from
> that snapshot.

We are non-stop.  The "location" thread exiting would not be a problem.

Each request, whether first or subsequent would send one or more complete
thread entries.  When sending a reply, you know where in the process
table to start, you skip dead threads, and you fill entries until
after doing the XML escaping and the GDB escaping an additional complete
entry will not fit.  You record where you stopped -- where to resume.

We allow an arbitrary number of GDBs to connect to the GDB stub running
in the OS kernel -- each connection gets a dedicated thread.

Currently, we support 320 threads.  This might well increase in the
future.  With thread name and everything else I want to send back at the
maximum (because that reflects how much space I might need under the
offset & length scheme), I calculate 113 bytes per thread (this counts
<thread> and </thread>) to send back -- before escaping.

So, if I 'snapshot' everything every time I get a packet with an offset of 0,
the buffer would need to be over 32K bytes in size.  I don't want to
increase the GDB stub stack size by this much.  So, that mens either
limiting the number of connections (fixed, pre-allocated buffers) or
using kernel equivalents of malloc and free (which is discouraged) or
coming up with a different approach -- e.g., avoiding the need for the
buffer...

So, in terms of saved state, with the snapshot it is 35-36K bytes, with the
process table index it is 2-8 bytes.

It's too late now, but I would much prefer interfaces something like:

either
    qfXfer:object:read:annex:length
    qsXfer:object:read:annex:length
or
    qfXfer:object:read:annex
    qsXfer:object:read:annex

[If the :length wasn't part of the spec, then send as much
as you want so long as you stay within the maximum packet size.  My
preference would be to leave off the length, but I'd be happy either way.]

David

Re: why I dislike qXfer

[Pedro Alves]

On 06/16/2016 06:42 PM, taylor, david wrote:
> 
>> From: Pedro Alves [mailto:palves@redhat.com]

> 
> We allow an arbitrary number of GDBs to connect to the GDB stub running
> in the OS kernel -- each connection gets a dedicated thread.
> 
> Currently, we support 320 threads.  This might well increase in the
> future.  With thread name and everything else I want to send back at the
> maximum (because that reflects how much space I might need under the
> offset & length scheme), I calculate 113 bytes per thread (this counts
> <thread> and </thread>) to send back -- before escaping.
> 
> So, if I 'snapshot' everything every time I get a packet with an offset of 0,
> the buffer would need to be over 32K bytes in size.  I don't want to
> increase the GDB stub stack size by this much.  So, that mens either
> limiting the number of connections (fixed, pre-allocated buffers) or
> using kernel equivalents of malloc and free (which is discouraged) or
> coming up with a different approach -- e.g., avoiding the need for the
> buffer...

So a workaround that probably will never break is to adjust your stub to
only remember the xml fragment for only one (or a few) threads at a time, and
serve off of that.  That would only be a problem if gdb "goes backwards"
I.e., if gdb requests a lower offset (other than 0) than the previous
requested offset.

The issue is that qXfer was originally invented for (binary) target objects
for which gdb wants random access.  However, "threads", and few other target
objects are xml based.  And for those, it must always be that gdb reads
the whole object, or at least reads it sequentially starting from the
beginning.  I can well imagine optimizations where gdb processes the xml
as it is reading it and stops reading before reaching EOF.  But that
wouldn't break the workaround.

Starting a read somewhere in the middle of the file could be possible
too, but it's require understanding how to skip until some xml element
starts and ignore the fact that the file wouldn't validate.  Plus gdb
doesn't know the size of the file until it reads it fully, so we'd either
some other way to determine that, or make gdb take guesses.
So I'm not seeing this happening anytime soon.

> 
> So, in terms of saved state, with the snapshot it is 35-36K bytes, with the
> process table index it is 2-8 bytes.
> 
> It's too late now, but I would much prefer interfaces something like:
> 
> either
>     qfXfer:object:read:annex:length
>     qsXfer:object:read:annex:length
> or
>     qfXfer:object:read:annex
>     qsXfer:object:read:annex
> 
> [If the :length wasn't part of the spec, then send as much
> as you want so long as you stay within the maximum packet size.  My
> preference would be to leave off the length, but I'd be happy either way.]

What would you do if the object to retrieve is larger than
the maximum packet size?

Thanks,
Pedro Alves

RE: why I dislike qXfer

[taylor, david]

> From: Pedro Alves [mailto:palves@redhat.com]
> Sent: Thursday, June 16, 2016 2:25 PM
> To: taylor, david; gdb@sourceware.org
> Subject: Re: why I dislike qXfer
> 
> On 06/16/2016 06:42 PM, taylor, david wrote:
> >
> >> From: Pedro Alves [mailto:palves@redhat.com]
> 
> >
> > We allow an arbitrary number of GDBs to connect to the GDB stub
> > running in the OS kernel -- each connection gets a dedicated thread.
> >
> > Currently, we support 320 threads.  This might well increase in the
> > future.  With thread name and everything else I want to send back at
> > the maximum (because that reflects how much space I might need under
> > the offset & length scheme), I calculate 113 bytes per thread (this
> > counts <thread> and </thread>) to send back -- before escaping.
> >
> > So, if I 'snapshot' everything every time I get a packet with an
> > offset of 0, the buffer would need to be over 32K bytes in size.  I
> > don't want to increase the GDB stub stack size by this much.  So, that
> > mens either limiting the number of connections (fixed, pre-allocated
> > buffers) or using kernel equivalents of malloc and free (which is
> > discouraged) or coming up with a different approach -- e.g., avoiding
> > the need for the buffer...
> 
> So a workaround that probably will never break is to adjust your stub to only
> remember the xml fragment for only one (or a few) threads at a time, and
> serve off of that.  That would only be a problem if gdb "goes backwards"
> I.e., if gdb requests a lower offset (other than 0) than the previous
> requested offset.

What I was thinking of doing was having no saved entries or, depending on
GDB details yet to be discovered, one saved entry.

Talk to the core OS people about prohibiting characters that require quoting
from occurring in the thread name.

Compute the maximum potential size of an entry with no padding.

Do arithmetic on the offset to figure out which process table entry to start with.

Do arithmetic on the length to figure out how many entries to process

Pad each entry at the end with spaces to bring it up to the maximum

For dead threads, fill the entry with spaces.

Report done ('l') when there are no more live threads between the current
position and the end of the process table.

> The issue is that qXfer was originally invented for (binary) target objects for
> which gdb wants random access.  However, "threads", and few other target
> objects are xml based.  And for those, it must always be that gdb reads the
> whole object, or at least reads it sequentially starting from the beginning.  I
> can well imagine optimizations where gdb processes the xml as it is reading it
> and stops reading before reaching EOF.  But that wouldn't break the
> workaround.

The qXfer objects for which I am thinking of implementing stub support, fall into
two categories:

. small enough that I would expect GDB to read it in toto in one chunk.
  For example, auxv.  Initially, I will likely have two entries (AT_ENTRY, AT_NULL);
  6 or 7 others might get added later.  Worst case, it all easily fits in one packet.

. larger with structure and possibly variable length elements  -- where I would
  expect multiple sequential reads starting at the beginning and continuing
  until everything is read.  For example, threads with no padding and skipping
  dead threads.

> Starting a read somewhere in the middle of the file could be possible too, but
> it's require understanding how to skip until some xml element starts and
> ignore the fact that the file wouldn't validate.  Plus gdb doesn't know the size
> of the file until it reads it fully, so we'd either some other way to determine
> that, or make gdb take guesses.
> So I'm not seeing this happening anytime soon.

But, alas, the community won't commit to it.

> > So, in terms of saved state, with the snapshot it is 35-36K bytes,
> > with the process table index it is 2-8 bytes.
> >
> > It's too late now, but I would much prefer interfaces something like:
> >
> > either
> >     qfXfer:object:read:annex:length
> >     qsXfer:object:read:annex:length
> > or
> >     qfXfer:object:read:annex
> >     qsXfer:object:read:annex
> >
> > [If the :length wasn't part of the spec, then send as much as you want
> > so long as you stay within the maximum packet size.  My preference
> > would be to leave off the length, but I'd be happy either way.]
> 
> What would you do if the object to retrieve is larger than the maximum
> packet size?

Huh?  qfXfer would read the first part, each subsequent qsXfer would read
the next chunk.  If you wanted to think of it in offset/length terms, the offset
for qfXfer would be zero; for qsXfer it would be the sum of the sizes (ignoring
GDB escaping modifications) of the qfXfer packet and any qsXfer that occurred
after the qfXfer and before this qsXfer.

As now, sub-elements (e.g. <thread> within <threads>) could be contained within
one packet or split between multiple packets.  Put the packets together in the order
received with no white space or anything else between them and pass the result off
to GDB's XML processing.

Or do I not understand your question?

> Thanks,
> Pedro Alves

Re: why I dislike qXfer

[Pedro Alves]

On 06/16/2016 08:59 PM, taylor, david wrote:
> 
>> From: Pedro Alves [mailto:palves@redhat.com]

>> So a workaround that probably will never break is to adjust your stub to only
>> remember the xml fragment for only one (or a few) threads at a time, and
>> serve off of that.  That would only be a problem if gdb "goes backwards"
>> I.e., if gdb requests a lower offset (other than 0) than the previous
>> requested offset.
> 
> What I was thinking of doing was having no saved entries or, depending on
> GDB details yet to be discovered, one saved entry.
> 
> Talk to the core OS people about prohibiting characters that require quoting
> from occurring in the thread name.
> 
> Compute the maximum potential size of an entry with no padding.
> 
> Do arithmetic on the offset to figure out which process table entry to start with.
> 
> Do arithmetic on the length to figure out how many entries to process
> 
> Pad each entry at the end with spaces to bring it up to the maximum
> 
> For dead threads, fill the entry with spaces.
> 
> Report done ('l') when there are no more live threads between the current
> position and the end of the process table.

That sounds over complicated, but, up to you.

I think "no saved entries" would be problematic, unless you assume
that gdb never requests a chunk smaller than the size of one entry.
Because if it does, and you return half of a thread element,
when gdb fetches the rest of the element, the thread might have
changed state already.  So e.g., you end up returning an impossible
extended info, or thread name, with a Frankenstein-like mix of before/after
state change  (extended info goes "AAAA" -> "BBBB", and you report
back "AA" + "BB").  And if you're going to save one entry, might as well
keep it simple, as in my original suggestion.

> 
>> The issue is that qXfer was originally invented for (binary) target objects for
>> which gdb wants random access.  However, "threads", and few other target
>> objects are xml based.  And for those, it must always be that gdb reads the
>> whole object, or at least reads it sequentially starting from the beginning.  I
>> can well imagine optimizations where gdb processes the xml as it is reading it
>> and stops reading before reaching EOF.  But that wouldn't break the
>> workaround.
> 
> The qXfer objects for which I am thinking of implementing stub support, fall into
> two categories:
> 
> . small enough that I would expect GDB to read it in toto in one chunk.
>   For example, auxv.  Initially, I will likely have two entries (AT_ENTRY, AT_NULL);
>   6 or 7 others might get added later.  Worst case, it all easily fits in one packet.

GDB does cache some objects like that, but others doesn't.  E.g., auvx 
is cached nowadays, but that wasn't always the case, and most others
objects are not cached.

>>> It's too late now, but I would much prefer interfaces something like:
>>>
>>> either
>>>     qfXfer:object:read:annex:length
>>>     qsXfer:object:read:annex:length
>>> or
>>>     qfXfer:object:read:annex
>>>     qsXfer:object:read:annex
>>>
>>> [If the :length wasn't part of the spec, then send as much as you want
>>> so long as you stay within the maximum packet size.  My preference
>>> would be to leave off the length, but I'd be happy either way.]
>>
>> What would you do if the object to retrieve is larger than the maximum
>> packet size?
> 
> Huh?  qfXfer would read the first part, each subsequent qsXfer would read
> the next chunk.  If you wanted to think of it in offset/length terms, the offset
> for qfXfer would be zero; for qsXfer it would be the sum of the sizes (ignoring
> GDB escaping modifications) of the qfXfer packet and any qsXfer that occurred
> after the qfXfer and before this qsXfer.
> 
> As now, sub-elements (e.g. <thread> within <threads>) could be contained within
> one packet or split between multiple packets.  Put the packets together in the order
> received with no white space or anything else between them and pass the result off
> to GDB's XML processing.
> 
> Or do I not understand your question?

If you're still going to need to handle sub-elements split between packets,
then other than that making explicit the assumption that gdb reads the
object sequentially, what's the real difference between this and gdb
fetching using the existing qXfer, but requesting larger chunks, e.g.,
the size of the stub's reported max packet length?

On the "leave off the length", I don't think it'd be a good idea for the
target to be in complete control of the transfer chunk size, without
having a way to interrupt the transfer.  I mean, there's no real limit
on the incoming packet size (gdb grows the buffer dynamically), and
if gdb requests qfXfer:object:read:annex and the stub decides to send the
whole multi-megabyte object back in one go, that's going to hog
the RSP channel until the packet is fully transferred.

Thanks,
Pedro Alves