Using XML in GDB?

Using XML in GDB?

[Daniel Jacobowitz]

Hi folks,

I've spent the last week and a half working on the "target available
feature" interface that I described on this list last May.  A big chunk
of the last two days has been spent trying to nail down a useful format
to store (in files) and transfer (over the remote protocol) descriptions of
remote "features", especially their register sets.

At first I was primarily focused on compactness.  But I've got a pretty good
handle on that problem now; a well-defined naming scheme and some caching,
and the size of the data is no longer a major concern.  That leaves
expressibility, parsability, and extensibility.  At which point I eventually
asked myself why I was reinventing the wheel.

And we get all sorts of things for free; for instance, UTF-8, which will be
handy if someone ever wants to include internationalized descriptions in the
target description.

Does anyone have a good reason why GDB should not make use of this
well-standardized format instead of inventing additional ad-hoc formats?

Where appropriate, of course.  People sometimes use XML for the silliest
things.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Andrew STUBBS]

I don't have anything to add to the XML debate yet. The idea of the 
"target available feature" interface sounds interesting and may be of 
use to us here, but I am having a little difficulty tracking down 
details. I have found the beginning of this conversation in the May 
archives, but it is always harder to find the definitive conclusion.

Is there anywhere you can point me to find the final specification of 
what this interface will include - what features will it describe, what 
aspects of those features are covered? Is it just limited to registers?

Thanks

Andrew Stubbs

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Thu, Jan 26, 2006 at 12:07:37PM +0000, Andrew STUBBS wrote:
> I don't have anything to add to the XML debate yet. The idea of the 
> "target available feature" interface sounds interesting and may be of 
> use to us here, but I am having a little difficulty tracking down 
> details. I have found the beginning of this conversation in the May 
> archives, but it is always harder to find the definitive conclusion.
> 
> Is there anywhere you can point me to find the final specification of 
> what this interface will include - what features will it describe, what 
> aspects of those features are covered? Is it just limited to registers?

There isn't a definitive conclusion because it's evolved substantially
once I began (re-) implementing it.  This is what I had two weeks ago:

  http://sourceware.org/ml/gdb/2005-05/msg00171.html

But it doesn't look entirely like that any more.

Registers can be generically described without hard-coded GDB
knowledge.  Most other features can't, except for their presence or
absence.  But the point of suggesting XML is to keep this extensible
should someone have a bright idea :-)

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Andrew STUBBS]

Daniel Jacobowitz wrote:
> There isn't a definitive conclusion because it's evolved substantially
> once I began (re-) implementing it.  This is what I had two weeks ago:
> 
>   http://sourceware.org/ml/gdb/2005-05/msg00171.html
> 
> But it doesn't look entirely like that any more.
> 
> Registers can be generically described without hard-coded GDB
> knowledge.  Most other features can't, except for their presence or
> absence.  But the point of suggesting XML is to keep this extensible
> should someone have a bright idea :-)

That sounds good. We have been considering doing something with memory 
mapped registers (devices, exception/interrupt reason codes, etc.), and 
this might be the answer.

More generally we have been looking at ways to pass back information 
about the target to replace the GDB scripts we currently use that just 
trust that the target is as they describe - there is no check that the 
user hasn't connected to the wrong type.

You may remember that my proposal concerning a plugin interface provided 
means for the target plugin to run GDB commands. This was one way I 
planned to solve this problem.

Specifically, the data we need/want GDB to know about the target are as 
follows:
  - architecture variant (i.e. what registers and instructions are valid);
  - endian;
  - memory map (not that GDB seems to do much with this at present);
  - non-core control/information registers;
  - other features.

'Other features' includes simulator specific trace features, silicon 
specific performance measurement devices, low level jtag commands etc. 
These things are controlled through custom GDB commands that may or may 
not be available on any given target.

In general, there is a selection of GDB 'set' commands which are of 
interest to targets and for which an interface might be nice.

None of this directly addresses the question of XML, but it does 
represent the sort of thing (some) users are looking to do, and 
therefore what 'extensibility' might entail. Of course, I'm still not 
sure exactly what you envisage as the limits of the interface.

Speaking of XML, are you aware of the SPIRIT SOC definition standard 
(http://www.spiritconsortium.com) which contains some things that the 
debugger might be interested in (and much in which it is not). I've no 
idea how compatible they are, but it might be nice if GDB could pull 
relevant information out of one of these.

Thanks

Andrew Stubbs

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Thu, Jan 26, 2006 at 03:06:27PM +0000, Andrew STUBBS wrote:
> That sounds good. We have been considering doing something with memory 
> mapped registers (devices, exception/interrupt reason codes, etc.), and 
> this might be the answer.

Would you expose them as registers, or as memory mapped I/O, from the
stub?  If the latter, what sort of information would you want GDB to
have about them, and how would you present them to the user or
frontend?

> Specifically, the data we need/want GDB to know about the target are as 
> follows:
>  - architecture variant (i.e. what registers and instructions are valid);
>  - endian;

While I'm not covering these at the moment, I do plan to.  Right now
I'm focusing on...

>  - non-core control/information registers;

... registers otherwise unknown to GDB.  Completely target-specific
features work too.

> None of this directly addresses the question of XML, but it does 
> represent the sort of thing (some) users are looking to do, and 
> therefore what 'extensibility' might entail. Of course, I'm still not 
> sure exactly what you envisage as the limits of the interface.

I'll be posting more about this real soon, I hope.  It's coming
together; I just wanted to have it a little more baked before I posted
it.

> Speaking of XML, are you aware of the SPIRIT SOC definition standard 
> (http://www.spiritconsortium.com) which contains some things that the 
> debugger might be interested in (and much in which it is not). I've no 
> idea how compatible they are, but it might be nice if GDB could pull 
> relevant information out of one of these.

I've vaguely heard of SPIRIT, but never any details.  Does it contain a
useful amount of information that GDB might care about?

I'm not going to pursue this for now because of IP issues; the terms on
the SPIRIT documents make me leery of using them for an open source
program, at least without talking to a lawyer about it first.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Paul Koning]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

 Daniel> On Thu, Jan 26, 2006 at 03:06:27PM +0000, Andrew STUBBS
 Daniel> wrote:
 >> That sounds good. We have been considering doing something with
 >> memory mapped registers (devices, exception/interrupt reason
 >> codes, etc.), and this might be the answer.

 Daniel> Would you expose them as registers, or as memory mapped I/O,
 Daniel> from the stub?  

I would very much like to have access to device registers.  But I
doubt that treating them as memory or as CPU GP registers would work.

Device registers tend to have side effects.  Neither memory nor GPU
general registers do.  So machinery like the register cache can't be
involved if you are playing with device registers; you want to apply
"volatile" rules to them.  

So each UI action has to correspond with exactly one operation on a
device register, not more, not less.  And you want to make sure that
there are ways to do just a write, or just a read -- a store can't be
turned into a load/store pair.

     paul

Re: Using XML in GDB?

[Andrew STUBBS]

Daniel Jacobowitz wrote:
> On Thu, Jan 26, 2006 at 03:06:27PM +0000, Andrew STUBBS wrote:
>> That sounds good. We have been considering doing something with memory 
>> mapped registers (devices, exception/interrupt reason codes, etc.), and 
>> this might be the answer.
> 
> Would you expose them as registers, or as memory mapped I/O, from the
> stub?  If the latter, what sort of information would you want GDB to
> have about them, and how would you present them to the user or
> frontend?

Well, right now we have a script that sets up a lot of convenience 
variables (part of the reason for 'keep-variable') which contain the 
address of the register. This works, but is more than a little clunky, 
what with the dereferencing syntax.

If they could look like regular registers, but in another reggroup, that 
would be good.

>> Specifically, the data we need/want GDB to know about the target are as 
>> follows:
>>  - architecture variant (i.e. what registers and instructions are valid);
>>  - endian;
> 
> While I'm not covering these at the moment, I do plan to.  Right now
> I'm focusing on...
> 
>>  - non-core control/information registers;
> 
> ... registers otherwise unknown to GDB.  Completely target-specific
> features work too.

Good, as long as they are in the road map.

You have skipped memory maps.  Is there no plan to make an automatic 
'mem' command? I would prefer that the memory maps were honoured by a 
little more of GDB - attempts to access bad addresses can crash the 
target, but that's a separate project.

> I've vaguely heard of SPIRIT, but never any details.  Does it contain a
> useful amount of information that GDB might care about?

It lists all the IP blocks on an SoC, but more importantly the location, 
format (down to bit level in some cases) and content of all the memory 
mapped registers in the device. It also describes the memory regions in 
the various memory interface devices.

Even if it isn't of direct interest to the debugger, it may be of 
interest to the user and worth presenting.

This is why I suggest that you don't go out of your way to be 
incompatible. I'm not sure what it takes to be compatible, but since 
it's all XML, it might be enough to just not clash with any of its tags, 
or conversely use the same names where they happen to be the same format.

> I'm not going to pursue this for now because of IP issues; the terms on
> the SPIRIT documents make me leery of using them for an open source
> program, at least without talking to a lawyer about it first.

Really? I would have thought they would be OK (the point of SPIRIT is 
that everybody uses it), but IANAL and anyway I haven't read the terms.

Thanks

Andrew Stubbs

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Thu, Jan 26, 2006 at 05:31:23PM +0000, Andrew STUBBS wrote:
> You have skipped memory maps.  Is there no plan to make an automatic 
> 'mem' command? I would prefer that the memory maps were honoured by a 
> little more of GDB - attempts to access bad addresses can crash the 
> target, but that's a separate project.

There's no plan for this, certainly; but the bits I'm implementing
would allow you to convey this information from target (or stub)
to GDB, and if someone else was motivated to make GDB do this...
well, that could be very nice indeed.

> It lists all the IP blocks on an SoC, but more importantly the location, 
> format (down to bit level in some cases) and content of all the memory 
> mapped registers in the device. It also describes the memory regions in 
> the various memory interface devices.
> 
> Even if it isn't of direct interest to the debugger, it may be of 
> interest to the user and worth presenting.
> 
> This is why I suggest that you don't go out of your way to be 
> incompatible. I'm not sure what it takes to be compatible, but since 
> it's all XML, it might be enough to just not clash with any of its tags, 
> or conversely use the same names where they happen to be the same format.

The XML bits I'm proposing would definitely have a simpler and
GDB-local definition.  That doesn't stop us from supporting the SPIRIT
schemas later on, as an alternative.

> >I'm not going to pursue this for now because of IP issues; the terms on
> >the SPIRIT documents make me leery of using them for an open source
> >program, at least without talking to a lawyer about it first.
> 
> Really? I would have thought they would be OK (the point of SPIRIT is 
> that everybody uses it), but IANAL and anyway I haven't read the terms.

For instance, it has explicit "may contain patented IP and we disclaim
all liability if you use it" language.  You also need to be a corporate
member to get access to some bits of it, I think - but I didn't spend
much time poking around at it.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Mark Kettenis]

> Date: Thu, 26 Jan 2006 11:38:32 -0500
> From: Daniel Jacobowitz <drow@false.org>
> 
> I'm not going to pursue this for now because of IP issues; the terms on
> the SPIRIT documents make me leery of using them for an open source
> program, at least without talking to a lawyer about it first.

Are these documents public?  If not we should stay away from this
stuff.  But if they are public, I can't see why we cannot implement an
interface specification.

Mark

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Thu, Jan 26, 2006 at 10:03:19PM +0100, Mark Kettenis wrote:
> > Date: Thu, 26 Jan 2006 11:38:32 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > 
> > I'm not going to pursue this for now because of IP issues; the terms on
> > the SPIRIT documents make me leery of using them for an open source
> > program, at least without talking to a lawyer about it first.
> 
> Are these documents public?  If not we should stay away from this
> stuff.  But if they are public, I can't see why we cannot implement an
> interface specification.

You have to register and agree to their terms to get a copy:
www.spiritconsortium.com.  However anyone can register, which
wasn't obvious to me the first time I looked.

Implementing an interface specification is not necessarily OK depending
on one's jurisdiction and what patents are involved; that's one of the
reasons there's a lot of people who object to software patents :-)  It
is entirely possible to write an interface such that any use of it
would require a patent license, or more than one.

I'm not saying that that's the case here, but the disclaimer certainly
suggests that it may be:

b) COMPLIANCE WITH THE SPIRIT SPECIFICATION MAY REQUIRE USE OF ONE OR
MORE FEATURES COVERED BY PATENT RIGHTS OR OTHER INTELLECTUAL PROPERTY
RIGHTS OF A CONSORTIUM MEMBER OR THIRD PARTY. THE MEMBERS TO THE
CONSORTIUM HEREBY EXPRESSLY DISCLAIM ANY LIABILITY FOR INFRINGEMENT OF
INTELLECTUAL PROPERTY RIGHTS OF OTHERS BY VIRTUE OF THE CONSORTIUM
SPECIFICATIONS, NOR DO THE MEMBERS TO THE CONSORTIUM UNDERTAKE A DUTY
TO ADVISE USERS OR POTENTIAL USERS OF THE CONSORTIUM SPECIFICATIONS OF
SUCH NOTICES OR ALLEGATIONS.

Anyway, my curiousity has been effectively engaged.  I'll take a look
at it.  If necessary I can find some US legal advice about the terms.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Mark Kettenis]

> Date: Thu, 26 Jan 2006 16:12:52 -0500
> From: Daniel Jacobowitz <drow@false.org>
>
> Implementing an interface specification is not necessarily OK depending
> on one's jurisdiction and what patents are involved; that's one of the
> reasons there's a lot of people who object to software patents :-)  It
> is entirely possible to write an interface such that any use of it
> would require a patent license, or more than one.

No software patents here yet ;-).  And some laws that explicitly allow
for interoperability.

> I'm not saying that that's the case here, but the disclaimer certainly
> suggests that it may be:
> 
> b) COMPLIANCE WITH THE SPIRIT SPECIFICATION MAY REQUIRE USE OF ONE OR
> MORE FEATURES COVERED BY PATENT RIGHTS OR OTHER INTELLECTUAL PROPERTY
> RIGHTS OF A CONSORTIUM MEMBER OR THIRD PARTY. THE MEMBERS TO THE
> CONSORTIUM HEREBY EXPRESSLY DISCLAIM ANY LIABILITY FOR INFRINGEMENT OF
> INTELLECTUAL PROPERTY RIGHTS OF OTHERS BY VIRTUE OF THE CONSORTIUM
> SPECIFICATIONS, NOR DO THE MEMBERS TO THE CONSORTIUM UNDERTAKE A DUTY
> TO ADVISE USERS OR POTENTIAL USERS OF THE CONSORTIUM SPECIFICATIONS OF
> SUCH NOTICES OR ALLEGATIONS.

This doesn't really say anything.  They're just covering their ass and
you're completely responsible for your own actions.

> Anyway, my curiousity has been effectively engaged.  I'll take a look
> at it.  If necessary I can find some US legal advice about the terms.

Those lawyers really got you folks by the balls isn't it?  It's almost
as if you're back to the days where you need yo ask the
priest/rabbi/imam for permission for everything you do.  Well, I
suppose the FSF could provide the necessary advice.

Mark

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Thu, Jan 26, 2006 at 10:40:54PM +0100, Mark Kettenis wrote:
> No software patents here yet ;-).  And some laws that explicitly allow
> for interoperability.

You're a lucky man in a smart country.

In any case, to change the subject.  To people whose targets don't
already come with SPIRIT, I don't think there's any benefit in using
it; while it can accomodate GDB's current needs, it's a ridiculous
amount of overkill.  Like swatting a fly with a well-targeted asteroid.

To people whose targets _do_ come with SPIRIT descriptions, i.e. mostly
SoC designers, I suspect that GDB support for those descriptions could
be incredibly handy.  But we're never going to pull this kind of data
over the wire, so I think it represents a more or less orthoganal
feature to my current plans.  I'm not going to pursue it.

If it builds automatically in a GDB source tree, do you have any
concerns about using a third-party XML parser?

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Bob Rossi]

> If it builds automatically in a GDB source tree, do you have any
> concerns about using a third-party XML parser?

None. On top of that, when I'm down with my MI parser, it could be very
beneficial to some folks, if it would translate to XML.

Honestly, I really think the nonsense of parsing these adhoc protocols
has to go away. I'm excited about what you are doing. People like me
spend a significant amount of time working on adhoc protocols. The time 
could be spent better on other things.

Bob Rossi

Re: Using XML in GDB?

[Mark Kettenis]

> Date: Thu, 26 Jan 2006 00:57:44 -0500
> From: Daniel Jacobowitz <drow@false.org>
> 
> Hi folks,
> 
> I've spent the last week and a half working on the "target available
> feature" interface that I described on this list last May.  A big chunk
> of the last two days has been spent trying to nail down a useful format
> to store (in files) and transfer (over the remote protocol) descriptions of
> remote "features", especially their register sets.
> 
> At first I was primarily focused on compactness.  But I've got a pretty good
> handle on that problem now; a well-defined naming scheme and some caching,
> and the size of the data is no longer a major concern.

Does that hold for slow serial links too?

> And we get all sorts of things for free; for instance, UTF-8, which will be
> handy if someone ever wants to include internationalized descriptions in the
> target description.
> 
> Does anyone have a good reason why GDB should not make use of this
> well-standardized format instead of inventing additional ad-hoc formats?

No objection to using a well-standardized format, but that probably
only makes sense if you're not going to write your own parser.  And I
really would like to avoid making GDB dependent on a library that
isn't standardly available on all systems we support.

Mark

Re: Using XML in GDB?

[Bob Rossi]

> > And we get all sorts of things for free; for instance, UTF-8, which will be
> > handy if someone ever wants to include internationalized descriptions in the
> > target description.
> > 
> > Does anyone have a good reason why GDB should not make use of this
> > well-standardized format instead of inventing additional ad-hoc formats?
> 
> No objection to using a well-standardized format, but that probably
> only makes sense if you're not going to write your own parser.  And I
> really would like to avoid making GDB dependent on a library that
> isn't standardly available on all systems we support.

What's wrong with libxml? http://xmlsoft.org/

Bob Rossi

Re: Using XML in GDB?

[Mark Kettenis]

> Date: Thu, 26 Jan 2006 15:39:30 -0500
> From: Bob Rossi <bob@brasko.net>
> 
> > No objection to using a well-standardized format, but that probably
> > only makes sense if you're not going to write your own parser.  And I
> > really would like to avoid making GDB dependent on a library that
> > isn't standardly available on all systems we support.
> 
> What's wrong with libxml? http://xmlsoft.org/

It doesn't come by default with any operating system (although most
Linux distributions will probably have it installed).

Mark

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Thu, Jan 26, 2006 at 09:36:33PM +0100, Mark Kettenis wrote:
> > Date: Thu, 26 Jan 2006 00:57:44 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > 
> > Hi folks,
> > 
> > I've spent the last week and a half working on the "target available
> > feature" interface that I described on this list last May.  A big chunk
> > of the last two days has been spent trying to nail down a useful format
> > to store (in files) and transfer (over the remote protocol) descriptions of
> > remote "features", especially their register sets.
> > 
> > At first I was primarily focused on compactness.  But I've got a pretty good
> > handle on that problem now; a well-defined naming scheme and some caching,
> > and the size of the data is no longer a major concern.
> 
> Does that hold for slow serial links too?

Yes.  Here's how it works in my current design:

- GDB connects to the target.
- GDB queries the target for its features.
- The target replies briefly (basically just name and a base integer
  for register numbering).
- GDB looks up the features by name in the set of features it
  recognizes.
- GDB queries the target for any features it does not recognize.
- GDB caches the queried features (with some exceptions).  The cache
  could be either in RAM, or on disk if desired and configured.

Suppose a verbose, well-documented register description is 50K - my
current (non-XML, but also non-verbose) descriptions are only about 2K. 
Over a 9600 baud serial link, which is about the slowest we're likely
to encounter, this could take about a minute to transfer - the first
time.  That would be a good motivation to use the on-disk caching
and maybe a progress indicator.

Most of my current debug targets have higher bandwidth links than that,
although often with a very high latency; I've got some tricks up my
sleeve for reducing the impact of latency, too (coming soon to a gdb@
near you).

> > And we get all sorts of things for free; for instance, UTF-8, which will be
> > handy if someone ever wants to include internationalized descriptions in the
> > target description.
> > 
> > Does anyone have a good reason why GDB should not make use of this
> > well-standardized format instead of inventing additional ad-hoc formats?
> 
> No objection to using a well-standardized format, but that probably
> only makes sense if you're not going to write your own parser.  And I
> really would like to avoid making GDB dependent on a library that
> isn't standardly available on all systems we support.

I definitely have no intention of either writing my own XML parser or
introducing an external dependence.  I am tentatively planning to
import one of {expat, libxml2} into src/ and arranging for it to be
built as necessary.

Sadly there does not seem to be a GNU implementation of an XML parser
with C bindings; I only found Java bindings.  If anyone knows to the
contrary, please holler.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Bob Rossi]

> > > And we get all sorts of things for free; for instance, UTF-8, which will be
> > > handy if someone ever wants to include internationalized descriptions in the
> > > target description.
> > > 
> > > Does anyone have a good reason why GDB should not make use of this
> > > well-standardized format instead of inventing additional ad-hoc formats?
> > 
> > No objection to using a well-standardized format, but that probably
> > only makes sense if you're not going to write your own parser.  And I
> > really would like to avoid making GDB dependent on a library that
> > isn't standardly available on all systems we support.
> 
> I definitely have no intention of either writing my own XML parser or
> introducing an external dependence.  I am tentatively planning to
> import one of {expat, libxml2} into src/ and arranging for it to be
> built as necessary.

Daniel,

One limitation I have found using libxml is that it doesn't appreciate
you putting certain char's in the file. The NUL char has given me
trouble, and has forced me to use conversion to hex or something.

Bob Rossi

Re: Using XML in GDB?

[Bob Rossi]

> Does anyone have a good reason why GDB should not make use of this
> well-standardized format instead of inventing additional ad-hoc formats?
> 
> Where appropriate, of course.  People sometimes use XML for the silliest
> things.

Hi Daniel,

The only other possibility I can think of is to use the MI protocol.
I've worked on the parsing side of the MI output command. However, I've
often wondered why there is no "factory" for creating MI output commands
in GDB.

Anyways, MI could be the official protocol used by GDB, if XML turns out
not to be such a hit. We already have a parser for input commands. We
would need an additional parser for output commands. We would also need
to units that would allow us to create mi output and input commands from
data.

Bob Rossi

Re: Using XML in GDB?

[Eli Zaretskii]

> Date: Thu, 26 Jan 2006 00:57:44 -0500
> From: Daniel Jacobowitz <drow@false.org>
> 
> Does anyone have a good reason why GDB should not make use of this
> well-standardized format instead of inventing additional ad-hoc formats?

I'd like to see some spec, if you please.  It's hard to make up one's
mind without at least that, since XML is not the ideal format for all
and every type of communications.  It has disadvantages, some of them
were already mentioned in this thread: it's very wordy, and existing
implementations are not guaranteed to exist on every supported
platform (telling users to download and install additional packages as
a prerequisite to building GDB is a mild nuisance).

So I'd like to weigh these demerits against the advantages, and that
is only possible if some kind of specification for this specific
interface which we are discussing is available.

I've read your description from May (and just re-read it now), and it
sounds like the number of different entities we need to communicate is
quite small and their structure is simple.  So why did you come to the
conclusion that you needed something like XML to express that
interface?

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Fri, Jan 27, 2006 at 07:41:30PM +0200, Eli Zaretskii wrote:
> > Date: Thu, 26 Jan 2006 00:57:44 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > 
> > Does anyone have a good reason why GDB should not make use of this
> > well-standardized format instead of inventing additional ad-hoc formats?
> 
> I'd like to see some spec, if you please.  It's hard to make up one's
> mind without at least that, since XML is not the ideal format for all
> and every type of communications.  It has disadvantages, some of them
> were already mentioned in this thread: it's very wordy, and existing
> implementations are not guaranteed to exist on every supported
> platform (telling users to download and install additional packages as
> a prerequisite to building GDB is a mild nuisance).

[As mentioned earlier, I do not intend to add any requirements for
building GDB; I would include it in-tree, unless someone had a good
reason not to do so.  I'm one of the folks who would get annoyed at
additional build requirements :-)]

> So I'd like to weigh these demerits against the advantages, and that
> is only possible if some kind of specification for this specific
> interface which we are discussing is available.
> 
> I've read your description from May (and just re-read it now), and it
> sounds like the number of different entities we need to communicate is
> quite small and their structure is simple.  So why did you come to the
> conclusion that you needed something like XML to express that
> interface?

At the moment I don't have a new specification; it's in too much flux
as the implementation continues to reveal issues I hadn't considered
adequately.  However, I can definitely explain what brought me to this
conclusion!

The proposal I sketched out in May is indeed simple.  The bits with
"architecture-specific data" and "registers tags" are the only hints
that it may need to evolve in the future.  However, there's a lot of
other things which GDB would like to know about targets, in an ideal
world.  A great example that GDB wouldn't make use of today, but
could, is Andrew Stubbs' mention of memory maps paired with the recent
report of backtrace blowing up on uclinux.  If we can receive the
memory map from the remote stub, we can absolutely guarantee that we
never send reads to outside of RAM when we wanted RAM access.

My original sketch did allow for this; the feature: and reg: prefixes
existed so that later prefixes like memory: could be added.  But then
for every new data type we don't just have to define the contents but
also the encoding.  With XML, we only have to define the contents.

The thing that pushed me over the edge this week was a combination of
two additions: files containing the same data, for GDB to parse in
advance without querying the target, and a "description" field.

The existing format is extremely file unfriendly because it uses colon
as a field separator and semicolon as a list separator - no newlines.
So the file becomes a single, awkwardly long, line.  Otherwise the
parser has to tolerate newlines, and that complicates the set of
characters which have to be handled specially.

And it would be nice if we could offer a user-friendly description of
target provided features - especially if we can pass it through to an
IDE.  Instead of an opaque "gdb.arm.vfpv2_user", you ought to be able
to view details about "ARM VFP unit (version 2, user mode)" and have
your debugger explain to you that the enabled interrupts are stored in
particular bits of FPSCR.

If we're going to do that, it would be a real shame not to consider
localization; most ARM system programmers can probably manage the
English names of the registers, but if we want to offer help text,
being able to provide it in Japanese is a big win.  So that means
character encodings, and in turn that means we need to be somewhat
careful with the contents of descriptions.

Combining this with files provided the third straw: now a single
register description might contain embedded newlines!  The question
of field separators has gotten even more complicated.

The biggest win of XML, for me, is that there are standard answers to
all of these problems and standard tools for editing and
checking XML files.  I get to rip out a half-usable parser that I spent
several days writing, and integrate a new XML parser in probably about
the amount of time it would take me to finish the one I've got.  And
the documentation for this feature (which is not yet (re)written but I
intend to be fairly substantial) can focus on its semantics, and just
reference the relevant bits of XML.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Eli Zaretskii]

> Date: Fri, 27 Jan 2006 13:04:29 -0500
> From: Daniel Jacobowitz <drow@false.org>
> 
> The proposal I sketched out in May is indeed simple.  The bits with
> "architecture-specific data" and "registers tags" are the only hints
> that it may need to evolve in the future.  However, there's a lot of
> other things which GDB would like to know about targets, in an ideal
> world.  A great example that GDB wouldn't make use of today, but
> could, is Andrew Stubbs' mention of memory maps paired with the recent
> report of backtrace blowing up on uclinux.  If we can receive the
> memory map from the remote stub, we can absolutely guarantee that we
> never send reads to outside of RAM when we wanted RAM access.

Can we at least have a pipe-dream list of things we think GDB would
ideally like to know about targets, and how structured each one of
them is?

> If we're going to do that, it would be a real shame not to consider
> localization; most ARM system programmers can probably manage the
> English names of the registers, but if we want to offer help text,
> being able to provide it in Japanese is a big win.  So that means
> character encodings, and in turn that means we need to be somewhat
> careful with the contents of descriptions.

That part is something I never understood in your reasoning: XML does
not do anything special to allow UTF-8, nor help you deal with the
resulting non-ASCII text on the GDB side.  If the underlying libc
supports UTF-8, you have that now; if it doesn't, you won't be better
off even if the target speaks XML.

> The biggest win of XML, for me, is that there are standard answers to
> all of these problems and standard tools for editing and
> checking XML files.

Is XML the only widely used standard that supports what we want?

Re: Using XML in GDB?

[Paul Koning]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

 >> Date: Fri, 27 Jan 2006 13:04:29 -0500 From: Daniel Jacobowitz
 >> <drow@false.org>

 >> ...The biggest win of XML, for me, is that there are standard answers
 >> to all of these problems and standard tools for editing and
 >> checking XML files.

 Eli> Is XML the only widely used standard that supports what we want?

No, it isn't.

You're essentially asking for an easily extensible, flexible
protocol.  XML is one of those; it is the extremely verbose end of the
"make it all ASCII" approach used in the Internet for decades.  Then
again, the same is true for the existing remote stub protocol.

Another extensible protocol that comes with definition tools and
checkers, but is much more compact, is SNMP.  Or, more precisely,
ASN.1 (the definition language) and BER/DER (the encoding rules).

If one of the goals is bit efficiency of the encoding, rather than
maximum verbosity, then ASN.1/DER is far better than XML.  Chances are
the implementation would be smaller, too.

Just a thought about a rather different approach to the problem...

     paul

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Fri, Jan 27, 2006 at 01:39:49PM -0500, Paul Koning wrote:
>  Eli> Is XML the only widely used standard that supports what we want?
> 
> No, it isn't.
> 
> You're essentially asking for an easily extensible, flexible
> protocol.  XML is one of those; it is the extremely verbose end of the
> "make it all ASCII" approach used in the Internet for decades.  Then
> again, the same is true for the existing remote stub protocol.
> 
> Another extensible protocol that comes with definition tools and
> checkers, but is much more compact, is SNMP.  Or, more precisely,
> ASN.1 (the definition language) and BER/DER (the encoding rules).
> 
> If one of the goals is bit efficiency of the encoding, rather than
> maximum verbosity, then ASN.1/DER is far better than XML.  Chances are
> the implementation would be smaller, too.

I'm curious; are there "authoring" tools for ASN.1?  I need something
that will work as both a file format and a wire protocol.

Ah, it looks like there are.  Debian has an emacs mode, a specification
compiler, an object dumper, perl bindings, and two C libraries.  A bit
sparse compared to the XML tools available, but should be adequate if
there's a compelling advantage for it.  Beyond that, I'm not at all
familiar with it.

Of course, there's also some rumors of "an ASN.1 variant of XML"...
and "gzip layered over XML", which is a pretty good one too.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Jim Blandy]

I think if you're going to have structured data, something like XML or
ASN.1 is the way to go.  (I'd actually prefer to say "lisp
s-expressions", but then nobody would speak to me any more. :) ) 
There's no reason to re-invent all that.

I believe XML has better support in Python and Perl and Ruby and Java
and that lot than ASN.1.

Bob Rossi mentioned issues with null characters in XML: it really is a
text format; you're not supposed to put binary blobs in there at all,
except as CDATA (which makes the data much bigger).  If you're trying
to put binary data in XML, you're on the wrong train.

Re: Using XML in GDB?

[Eli Zaretskii]

> Date: Fri, 27 Jan 2006 22:33:07 -0800
> From: Jim Blandy <jimb@red-bean.com>
> 
> I think if you're going to have structured data, something like XML or
> ASN.1 is the way to go.  (I'd actually prefer to say "lisp
> s-expressions", but then nobody would speak to me any more. :) ) 

Don't worry, I'd still speak to you ;-)

Re: Using XML in GDB?

[Paul Koning]

>>>>> "Jim" == Jim Blandy <jimb@red-bean.com> writes:

 Jim> I believe XML has better support in Python and Perl and Ruby and
 Jim> Java and that lot than ASN.1.

Maybe.  I've done some battle with DOM::XML:: in Perl.  It may be ok
if you want to parse XML-encoded documents; that's what it appears to
be for.  But the documentation is very feeble, and I found it to be a
nightmare when I tried to use it to parse something that wasn't a
document.   So I wouldn't necessarily say that XML Is "supported" in
Perl.  

Maybe the Python XML package is better -- gotta try that one of these
days. 

BER/DER is that it's very simple, so even if the support isn't there
yet that may not be a big issue.

 Jim> Bob Rossi mentioned issues with null characters in XML: it
 Jim> really is a text format; you're not supposed to put binary blobs
 Jim> in there at all, except as CDATA (which makes the data much
 Jim> bigger).  If you're trying to put binary data in XML, you're on
 Jim> the wrong train.

Exactly.  Conversely, ASN.1 has good data type support, including
binary types of all kinds.  

       paul

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Fri, Jan 27, 2006 at 08:25:42PM +0200, Eli Zaretskii wrote:
> Can we at least have a pipe-dream list of things we think GDB would
> ideally like to know about targets, and how structured each one of
> them is?

Well, I don't think I can.  I haven't a clue; every time I think I've
got a handle on the set, people come up with creative new ones.  For
instance I hadn't considered that we might want memory-mapped I/O
devices to be explicitly explained to GDB.

> > If we're going to do that, it would be a real shame not to consider
> > localization; most ARM system programmers can probably manage the
> > English names of the registers, but if we want to offer help text,
> > being able to provide it in Japanese is a big win.  So that means
> > character encodings, and in turn that means we need to be somewhat
> > careful with the contents of descriptions.
> 
> That part is something I never understood in your reasoning: XML does
> not do anything special to allow UTF-8, nor help you deal with the
> resulting non-ASCII text on the GDB side.  If the underlying libc
> supports UTF-8, you have that now; if it doesn't, you won't be better
> off even if the target speaks XML.

The mere existance of character encodings isn't the issue; the
issue is encoding free-form text, possibly containing strange
"characters", within a structured element.  In particular, within a
structured element that a client may not recognize and support.

We've got field separators - colon and semicolon in my working copy,
and the status of newlines is fuzzy.  If they may validly occur within
free-form text we need to have an alternate way to escape them. In
ASCII how to do this is quite clear-cut.  In UTF-8 it's a little less
clear-cut although still pretty simple - but it does require knowing
something about the contents of UTF-8 when defining the encoding, if
you want the encoded result to still be valid UTF-8.  And I do, because
otherwise it will become awkward to edit the descriptions in a text
editor.

If you want to optionally support other encodings rather than UTF-8 it
becomes even trickier.  You have to know, eventually, how fields are
encoded.  For us I don't think that's necessary; we can define all
encoded text as UTF-8.  But there's a similar problem if someone wants
to add a descriptive element transfered as a binary blob for some
reason - I don't have an example for this, but I can certainly accept
that someone will come up with one someday.  Maybe bytecode!

XML's already considered this and solved it.  There are defined ways
to express a document's encoding, and to escape characters that
would otherwise serve as syntax elements.  You can store arbitrary text
or byte sequences in an element (e.g OpenDocument).

> > The biggest win of XML, for me, is that there are standard answers to
> > all of these problems and standard tools for editing and
> > checking XML files.
> 
> Is XML the only widely used standard that supports what we want?

I'm sure it isn't, but I think it's the most standardized.  You could
do something similar in an RFC-822 style format, for instance (Header:
value as in email, in case any of the list readers aren't familiar with
RFC-822; it also does handle multiline values, but I'm not sure how it
is on encoded text).

I'm not a die-hard XML advocate.  In fact I've never used it before
for a new project, although I'm fairly familiar with it.  If someone
has an alternate representation that they believe is superior, I'm
listening.  What I want to do, however, is draw the line past which
we should use standardized representations instead of ad-hoc.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Paul Schlie]

> From: Jim Blandy <jimb@red-bean.com>
> I think if you're going to have structured data, something like
> XML or ASN.1 is the way to go.  (I'd actually prefer to say "lisp
> s-expressions", but then nobody would speak to me any more. :) )

- Actually I'd have thought s-expression structured data would be a
  vastly superior choice relative either perverted (IMHO) alternative.

Re: Using XML in GDB?

[Jim Blandy]

On 1/28/06, Paul Schlie <schlie@comcast.net> wrote:
> > From: Jim Blandy <jimb@red-bean.com>
> > I think if you're going to have structured data, something like
> > XML or ASN.1 is the way to go.  (I'd actually prefer to say "lisp
> > s-expressions", but then nobody would speak to me any more. :) )
>
> - Actually I'd have thought s-expression structured data would be a
>   vastly superior choice relative either perverted (IMHO) alternative.

S-expressions are great if you've already got a lisp interpreter
around.  If GDB had Guile integrated into it, making all this work
would be so easy we'd barely bother to have a conversation about it. 
But our audience is more comfortable with Python, Ruby, and Perl; I
think a syntax that is well-supported by those tools is a better
choice.

It was a toss-off joke; I didn't mean to distract attention from
Daniel's proposal.  We can have a GDB extension language discussion on
a separate thread, if people are interested.

Re: Using XML in GDB?

[Daniel Jacobowitz]

On Sat, Jan 28, 2006 at 08:27:42PM -0800, Jim Blandy wrote:
> On 1/28/06, Paul Schlie <schlie@comcast.net> wrote:
> > > From: Jim Blandy <jimb@red-bean.com>
> > > I think if you're going to have structured data, something like
> > > XML or ASN.1 is the way to go.  (I'd actually prefer to say "lisp
> > > s-expressions", but then nobody would speak to me any more. :) )
> >
> > - Actually I'd have thought s-expression structured data would be a
> >   vastly superior choice relative either perverted (IMHO) alternative.
> 
> S-expressions are great if you've already got a lisp interpreter
> around.  If GDB had Guile integrated into it, making all this work
> would be so easy we'd barely bother to have a conversation about it. 
> But our audience is more comfortable with Python, Ruby, and Perl; I
> think a syntax that is well-supported by those tools is a better
> choice.
> 
> It was a toss-off joke; I didn't mean to distract attention from
> Daniel's proposal.  We can have a GDB extension language discussion on
> a separate thread, if people are interested.

For the record, I've implemented most of Guile integration on a lark
once; I wouldn't say it would make anything quite that easy, though :-)

We were still lacking the bits to make extension languages useful, at
the time, and somewhat still today.  Once that's fixed, I anticipate
sprouting Guile, Python, and updated Perl bindings all at about the
same time.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Using XML in GDB?

[Paul Schlie]

> From: Jim Blandy <jimb@red-bean.com>
>> On 1/28/06, Paul Schlie <schlie@comcast.net> wrote:
>>> From: Jim Blandy <jimb@red-bean.com>
>>> I think if you're going to have structured data, something like
>>> XML or ASN.1 is the way to go.  (I'd actually prefer to say "lisp
>>> s-expressions", but then nobody would speak to me any more. :) )
>> 
>> - Actually I'd have thought s-expression structured data would be a
>>   vastly superior choice relative either perverted (IMHO) alternative.
> 
> S-expressions are great if you've already got a lisp interpreter
> around.  If GDB had Guile integrated into it, making all this work
> would be so easy we'd barely bother to have a conversation about it.
> But our audience is more comfortable with Python, Ruby, and Perl; I
> think a syntax that is well-supported by those tools is a better
> choice.

- sorry, thought the objective only likely necessitated a static parse of
  a hierarchical definition composed of likely just strings and integers.

> It was a toss-off joke; I didn't mean to distract attention from
> Daniel's proposal.  We can have a GDB extension language discussion
> on a separate thread, if people are interested.

- I'm admittedly partial to scheme.