cp tree question

cp tree question

[Stefan Strasser]

Good morning,
I'm wondering if it is a bug or desired behaviour that, in the current
release version of gcc, TREE_CHAIN()ing the cp_namespace_decls() result
doesn't bring up type_decl nodes of c++ enum's, but gcc version 3.3.4 does?
have they been moved into another list?

you can easily check this with -fdump-translation-unit on the code "enum
blah{};". there is no blah in the .tu file with version 3.4.3.

(don't put any enumerators in there because so the type_decl node gets
queued by the DECL_TYPE() of the enumerator node. but it's not in
cp_namespace_decls())

thanks for your help,

-- 
Stefan Strasser

-fdump-translation-unit considered harmful (was Re: cp tree question)

[Zack Weinberg]

We should get rid of -fdump-translation-unit.  It has never done what
people think it does, the output format has never been documented, no
one cares to improve either of those things, and we now have
tree-pretty-print.c which is language-independent and far easier to
read.  No, it's not machine parseable, but neither is
-fdump-translation-unit's format, if you really sit down and think
about it.

zw

Re: -fdump-translation-unit considered harmful (was Re: cp tree question)

[Nathan Sidwell]

Zack Weinberg wrote:
> We should get rid of -fdump-translation-unit.  It has never done what
> people think it does, the output format has never been documented, no
> one cares to improve either of those things, and we now have
> tree-pretty-print.c which is language-independent and far easier to
> read.  No, it's not machine parseable, but neither is
> -fdump-translation-unit's format, if you really sit down and think
> about it.

fine by me.

nathan

-- 
Nathan Sidwell    ::   http://www.codesourcery.com   ::     CodeSourcery LLC
nathan@codesourcery.com    ::     http://www.planetfall.pwp.blueyonder.co.uk

Re: -fdump-translation-unit considered harmful (was Re: cp tree question)

[Stefan Strasser]

no matter if you should get rid of dump-translation-unit - I was talking 
about the different behaviour of the cp tree interface between versions.
I'm not trying to read dump-translation-unit output, but you can see 
what I mean using it because it also uses the tree interface.
so have there been any major changes to this interface lately or should 
it be reported as a bug when a function which is documented to return 
all decls of a namespace leaves out some?



Zack Weinberg schrieb:
> We should get rid of -fdump-translation-unit.  It has never done what
> people think it does, the output format has never been documented, no
> one cares to improve either of those things, and we now have
> tree-pretty-print.c which is language-independent and far easier to
> read.  No, it's not machine parseable, but neither is
> -fdump-translation-unit's format, if you really sit down and think
> about it.
> 
> zw
> 
> 



-- 
Stefan Strasser

Re: -fdump-translation-unit considered harmful

[Zack Weinberg]

Stefan Strasser <sstrasser@systemhaus-gruppe.de> writes:

> no matter if you should get rid of dump-translation-unit - I was
> talking about the different behaviour of the cp tree interface between
> versions.
> I'm not trying to read dump-translation-unit output, but you can see
> what I mean using it because it also uses the tree interface.
> so have there been any major changes to this interface lately or
> should it be reported as a bug when a function which is documented to
> return all decls of a namespace leaves out some?

I don't know what changed, but the internal interfaces *are* subject
to change without notice, and the C++ front end's data structures
*have* been revised extensively in the current development cycle.  You
should report bugs if the code and the comments describing the code
are inconsistent, but you should not expect functions that do one
thing in version X to do the same thing in version X+1.

Me ranting about -fdump-translation-unit had nothing to do with your
problem, really, it's just that your message reminded me that I was
going to bring that up.

zw

Re: cp tree question

[Gabriel Dos Reis]

Stefan Strasser <sstrasser@systemhaus-gruppe.de> writes:

| Good morning,
| I'm wondering if it is a bug or desired behaviour that, in the current
| release version of gcc, TREE_CHAIN()ing the cp_namespace_decls() result
| doesn't bring up type_decl nodes of c++ enum's, but gcc version 3.3.4 does?
| have they been moved into another list?
| 
| you can easily check this with -fdump-translation-unit on the code "enum
| blah{};". there is no blah in the .tu file with version 3.4.3.

And you don't get the body of namespaces other than the global one
neither.  But nobody seems to body since most them of the time, it is
used to dump C programs ;-) 
Well, except academic papers that claim that they get everything,
which of course is not true but that is another story.

| 
| (don't put any enumerators in there because so the type_decl node gets
| queued by the DECL_TYPE() of the enumerator node. but it's not in
| cp_namespace_decls())
| 
| thanks for your help,
| 
| -- 
| Stefan Strasser

-- 
                                                       Gabriel Dos Reis 
                                           gdr@integrable-solutions.net

Re: -fdump-translation-unit considered harmful (was Re: cp tree question)

[Gabriel Dos Reis]

Zack Weinberg <zack@codesourcery.com> writes:

| We should get rid of -fdump-translation-unit.  It has never done what
| people think it does, the output format has never been documented, no
| one cares to improve either of those things,

Actually people do care about -fdump-translation-unit -- except that
nodbody want to check in the improvements because otherwise they would
loss an "advantage" over competitors.  some maintainers do get request
to improve it though :-/

| and we now have
| tree-pretty-print.c which is language-independent and far easier to
| read.  No, it's not machine parseable, but neither is
| -fdump-translation-unit's format, if you really sit down and think
| about it.

Well, -fdump-translation-unit is more machine readale than
tree-pretty-print.c which is awful at the moment.  I do know of
several distinct parsers of -fdump-translation-unit.

I intented to improve it -- and sent a note in that respect to Mark --
but we got into stage 3, after that.  I could have pretented that it
was a "clean up" patch, as many other patches that got in despite
being in stage 3.

| 
| zw

-- 
                                                       Gabriel Dos Reis 
                                           gdr@integrable-solutions.net

Re: -fdump-translation-unit considered harmful (was Re: cp tree question)

[Gabriel Dos Reis]

Nathan Sidwell <nathan@codesourcery.com> writes:

| Zack Weinberg wrote:
| > We should get rid of -fdump-translation-unit.  It has never done what
| > people think it does, the output format has never been documented, no
| > one cares to improve either of those things, and we now have
| > tree-pretty-print.c which is language-independent and far easier to
| > read.  No, it's not machine parseable, but neither is
| > -fdump-translation-unit's format, if you really sit down and think
| > about it.
| 
| fine by me.

It would be a real mistake.  If you don't care about it, and other
people are using it, that you should leave it useful to those people.

Re: -fdump-translation-unit considered harmful

[Gabriel Dos Reis]

Zack Weinberg <zack@codesourcery.com> writes:

[...]

| Me ranting about -fdump-translation-unit had nothing to do with your
| problem, really, it's just that your message reminded me that I was
| going to bring that up.

I think -fdump-translation-unit should remain.  I thereby volunteer to
take any question about it.  

-- Gaby

Re: -fdump-translation-unit considered harmful

[Paul Brook]

> | Me ranting about -fdump-translation-unit had nothing to do with your
> | problem, really, it's just that your message reminded me that I was
> | going to bring that up.
>
> I think -fdump-translation-unit should remain.  I thereby volunteer to
> take any question about it.

Ok: Where can I find the documentation for it?

Given this is intended for third party utilities it should be a stable and 
well documented interface, right?

Paul

Re: -fdump-translation-unit considered harmful

[Gabriel Dos Reis]

Paul Brook <paul@codesourcery.com> writes:

| > | Me ranting about -fdump-translation-unit had nothing to do with your
| > | problem, really, it's just that your message reminded me that I was
| > | going to bring that up.
| >
| > I think -fdump-translation-unit should remain.  I thereby volunteer to
| > take any question about it.
| 
| Ok: Where can I find the documentation for it?

[ Ask your sourcerer in chief ;-p ]
It is undocumented, as is the output of tree-pretty-print.c.
Would you want to remove tree-pretty-print.c too?

| Given this is intended for third party utilities 

It is intented for debugging.  

| it should be a stable and  well documented interface, right?

You see, I would not mind having a well defined interface for parsing;
but since it is for debugging purpose, I would expect some variations
in the output as needs arise and we gain better understanding of its
uses. 

-- Gaby

Re: -fdump-translation-unit considered harmful

[Stefan Strasser]

> It is intented for debugging.  
> 
> | it should be a stable and  well documented interface, right?
> 


I think the reason why a debugging tool is misused that often so you 
want to remove it is that "the world" is missing a decent free 
standalone c++ parser, e.g. for code analyzing tools. and since c++ 
parsing is non-trivial and gcc is the only advanced one people try get 
information out of that.
at least this is what I'm trying to do(using the tree structures, not 
dumps). I'd be happy to contribute it when I'm done but I don't think 
it'll fit your needs because it is a c++ model.
but I think there should be a stable interface or a stable xml output 
for c++ code. (I know there is gccxml patch, but it is nothing else than 
a dump-translation-unit written to a flat xml file, no hierarchy, no 
statements)


-- 
Stefan Strasser

Re: -fdump-translation-unit considered harmful

[Robert Dewar]

Stefan Strasser wrote:

> but I think there should be a stable interface or a stable xml output 
> for c++ code. (I know there is gccxml patch, but it is nothing else than 
> a dump-translation-unit written to a flat xml file, no hierarchy, no 
> statements)

I assume you are aware of the non-technical difficulties with such an
approach, but just to continue with the technical discussions, I think
that an output of this kind is only a first step. What is really needed
is something analogous to the ASIS interface in Ada, i.e. a well defined
API that let's programs make structured queries on the semantics of a
parsed program. This approach has several advantages:

1. You do not have to try to stablilize the intermdiate output used as
an input to this tool (e.g. the GNAT tree changes all the time), but
instead you just keep the API stable.

2. The API is at a far more convenient level for writing typical tools

3. The API is not in general planned to be convenient for, or efficient
enough, to write an entire compiler (thus addressing one of the critical
non-technical issues).

Robert Dewar

Re: -fdump-translation-unit considered harmful

[Pjotr Kourzanov]

Guys,

   please. let the  -fdump-translation-unit work as it did, this saves
us quite some time custom ad-hoc parsing C/C++. I understand there are 
political issues with it generating full tree dump (from a previous 
thread on this list), however, I don't see why we the OSS developers can 
not benefit from the great work you guys have been doing in impoving 
C(++) compiler. Another problem is of course the quantity of generated 
data which is not convenient to parse with home-grown (Awk) scripts I 
have, see the following paragraph...

   Although I am not familiar with ASIS, I do believe we need a way to
flexibly generate data/code from the tree (dump), be it with an API or 
an external tool. I don't see why such a tool can not be built into GCC,
as long as it has to be supplied externally in form of a script. Such an 
Awk (or for that matter, Icon) script would get direct access to the 
tree via standard Awk (Icon) data-structures or functions, and would 
necessarily have to be supplied in source form, thereby prohibiting 
vendor lock-in.

   Don't cut the trees!

Pjotr Kourzanov

Robert Dewar wrote:
> Stefan Strasser wrote:
> 
>> but I think there should be a stable interface or a stable xml output 
>> for c++ code. (I know there is gccxml patch, but it is nothing else 
>> than a dump-translation-unit written to a flat xml file, no hierarchy, 
>> no statements)
> 
> 
> I assume you are aware of the non-technical difficulties with such an
> approach, but just to continue with the technical discussions, I think
> that an output of this kind is only a first step. What is really needed
> is something analogous to the ASIS interface in Ada, i.e. a well defined
> API that let's programs make structured queries on the semantics of a
> parsed program. This approach has several advantages:
> 
> 1. You do not have to try to stablilize the intermdiate output used as
> an input to this tool (e.g. the GNAT tree changes all the time), but
> instead you just keep the API stable.
> 
> 2. The API is at a far more convenient level for writing typical tools
> 
> 3. The API is not in general planned to be convenient for, or efficient
> enough, to write an entire compiler (thus addressing one of the critical
> non-technical issues).
> 
> Robert Dewar
> 

Re: -fdump-translation-unit considered harmful

[Aaron W. LaFramboise]

Robert Dewar wrote:

> I assume you are aware of the non-technical difficulties with such an
> approach, but just to continue with the technical discussions, I think
> that an output of this kind is only a first step. What is really needed
> is something analogous to the ASIS interface in Ada, i.e. a well defined
> API that let's programs make structured queries on the semantics of a
> parsed program. This approach has several advantages:

<snip advantages>

4. A binary API that interfaced with GCC would require the client party
to the interface to adhere to the GPL, and thereby require the client to
be /free software/.  Would this address Richard Stallman's concerns
about the danger of making it easy for proprietary non-free programs to
freeload off GCC?

In terms of interface design, if a file format could be created that
would insulate a user from casual changes to GCC internals, then so
could a binary interface.

I understand the "GCC should be a compiler only." arguments, but it
seems there is a genuine demand in the free software community for a
high-quality source analyzer, particularly for languages like C++ where
this is not trivial.  GCC seems to be very nearly where it needs to be
to fill this need.


Aaron W. LaFramboise

Re: cp tree question

[Stefan Strasser]

this is additional information on this subject I sent here two days ago 
which caused the dump-translation-unit discussion.
answers to bug 19073 please, not here.

---
ok, I took a closer look at this.
we all agree that type_decls(even c++ implicit ones) should be in 
cp_binding_level::names, don't we?

decls get added to this list by name-lookup.c add_decl_to_level(), which 
gets called by name-lookup.c:pushdecl()
pushdecl() checks if there is already a binding for the decl(via 
namespace_binding()).
if there is(which is the case for the implicit type decls in current 
version), it only calls add_decl_to_level() under certain circumstances, 
one of these is: not being a TYPE_DECL.

in gcc version 3.3.4(which doesn't have this problem) 
namespace_binding() returns 0, there is no binding, in current CVS there is.
so, why is this?

because of these lines in 
name-lookup.c:set_identifier_type_value_with_scope:
	  if (binding->value)
	    supplement_binding (binding, decl);
	  else
	    binding->value = decl;

I understand what they're doing but I don't understand why they are at 
this point and were introduced in decl.c version 1.1114.
the cvs log message doesn't contain anything that's useful to me either.
why are these lines there? before this there was no assignment to 
binding->value at all.

Thanks for your help

---


Stefan Strasser schrieb:
> Good morning,
> I'm wondering if it is a bug or desired behaviour that, in the current
> release version of gcc, TREE_CHAIN()ing the cp_namespace_decls() result
> doesn't bring up type_decl nodes of c++ enum's, but gcc version 3.3.4 does?
> have they been moved into another list?
> 
> you can easily check this with -fdump-translation-unit on the code "enum
> blah{};". there is no blah in the .tu file with version 3.4.3.
> 
> (don't put any enumerators in there because so the type_decl node gets
> queued by the DECL_TYPE() of the enumerator node. but it's not in
> cp_namespace_decls())
> 
> thanks for your help,
> 



-- 
Stefan Strasser

Re: -fdump-translation-unit considered harmful

[Joe Buck]

On Sat, Dec 18, 2004 at 03:13:50PM +0000, Paul Brook wrote:
> > | Me ranting about -fdump-translation-unit had nothing to do with your
> > | problem, really, it's just that your message reminded me that I was
> > | going to bring that up.
> >
> > I think -fdump-translation-unit should remain.  I thereby volunteer to
> > take any question about it.
> 
> Ok: Where can I find the documentation for it?
> 
> Given this is intended for third party utilities it should be a stable and 
> well documented interface, right?

It is most emphatically *not* intended for third-party utilities.  It is
not intended to be a path to interface other software to GCC; it's a debugging
aid.

Re: -fdump-translation-unit considered harmful

[Joe Buck]

On Sun, Dec 19, 2004 at 02:35:06PM -0600, Aaron W. LaFramboise wrote:
> 4. A binary API that interfaced with GCC would require the client party
> to the interface to adhere to the GPL, and thereby require the client to
> be /free software/.  Would this address Richard Stallman's concerns
> about the danger of making it easy for proprietary non-free programs to
> freeload off GCC?

For anyone interested in interfacing proprietary programs to GCC, that's
not even much of a speed bump.

As soon as that binary API is written, someone will write a small calling
program that will use the API to dump the translation unit into a nicely
documented XML format.  The dumper will, of course, be released under the
GPL.  Then the proprietary backend will simply read in the XML data.

That said, in the long term the free software world needs good code
analysis tools, so I suspect that at some point the FSF will have to
revisit this policy.

> I understand the "GCC should be a compiler only." arguments, but it
> seems there is a genuine demand in the free software community for a
> high-quality source analyzer, particularly for languages like C++ where
> this is not trivial.  GCC seems to be very nearly where it needs to be
> to fill this need.

Agreed; in any case the proprietary software world just licenses a very good
C++ front end from [some company, since we're not supposed to plug proprietary
software on this list, but most of us know who], for quite reasonable cost,
so the fact that they can't use the g++ front end doesn't slow them down
much.

But this is an argument that you'll have to make with RMS.

Re: -fdump-translation-unit considered harmful (was Re: cp tree question)

[Joe Buck]

On Sat, Dec 18, 2004 at 02:31:06PM +0100, Gabriel Dos Reis wrote:
> Nathan Sidwell <nathan@codesourcery.com> writes:
> 
> | Zack Weinberg wrote:
> | > We should get rid of -fdump-translation-unit.  It has never done what
> | > people think it does, the output format has never been documented, no
> | > one cares to improve either of those things, and we now have
> | > tree-pretty-print.c which is language-independent and far easier to
> | > read.  No, it's not machine parseable, but neither is
> | > -fdump-translation-unit's format, if you really sit down and think
> | > about it.
> | 
> | fine by me.
> 
> It would be a real mistake.  If you don't care about it, and other
> people are using it, that you should leave it useful to those people.

On any sensitive issue like this, dumping the feature would need an SC
decision.  If Gaby's willing to maintain it, let it be.

Re: -fdump-translation-unit considered harmful

[Tom Tromey]

>>>>> "Joe" == Joe Buck <Joe.Buck@synopsys.COM> writes:

Joe> That said, in the long term the free software world needs good code
Joe> analysis tools, so I suspect that at some point the FSF will have to
Joe> revisit this policy.

These seem to be under development outside of GCC -- there's both
sparse and the C/C++ parsing stuff in Eclipse.  Maybe GCC already
missed the boat here.  For all we know things will go the other way
and in a few years we'll be looking at how to use these external front
ends instead of ours :-)

Tom

Re: -fdump-translation-unit considered harmful

[Gabriel Dos Reis]

Tom Tromey <tromey@redhat.com> writes:

| >>>>> "Joe" == Joe Buck <Joe.Buck@synopsys.COM> writes:
| 
| Joe> That said, in the long term the free software world needs good code
| Joe> analysis tools, so I suspect that at some point the FSF will have to
| Joe> revisit this policy.
| 
| These seem to be under development outside of GCC -- there's both
| sparse and the C/C++ parsing stuff in Eclipse.  Maybe GCC already
| missed the boat here.  For all we know things will go the other way
| and in a few years we'll be looking at how to use these external front
| ends instead of ours :-)

And that is regretable.  While I believe competition is good, I don't
think it is worthwhile spending resources re-inventing the same wheel
over and over again, just because some purely non-technical reasons
very hard to explain and justify before facts and reality.

To approach correctness, one needs to exceed the critical mass of user
base.  That is hard to achieve when the community is fractured between
incomplete and competing tools.  Many projects  in this area had
failed precisely because they did not get the critical mass. 

-- Gaby

Re: -fdump-translation-unit considered harmful

[Pjotr Kourzanov]

Joe Buck wrote:
> On Sun, Dec 19, 2004 at 02:35:06PM -0600, Aaron W. LaFramboise wrote:
> 
>>4. A binary API that interfaced with GCC would require the client party
>>to the interface to adhere to the GPL, and thereby require the client to
>>be /free software/.  Would this address Richard Stallman's concerns
>>about the danger of making it easy for proprietary non-free programs to
>>freeload off GCC?
> 
> 
> For anyone interested in interfacing proprietary programs to GCC, that's
> not even much of a speed bump.
> 
> As soon as that binary API is written, someone will write a small calling
> program that will use the API to dump the translation unit into a nicely
> documented XML format.  The dumper will, of course, be released under the
> GPL.  Then the proprietary backend will simply read in the XML data.

   I suspect the XML output thus generated will be comparable is size to 
what -fdump-translation-unit generates now. As it is, it takes *ages* to 
re-parse those megabytes of ASCII data (even for simple C source files 
including lots of headers). The really problematic case is only when a 
binary API at the tree level is exposed.

> 
> That said, in the long term the free software world needs good code
> analysis tools, so I suspect that at some point the FSF will have to
> revisit this policy.
> 
> 
>>I understand the "GCC should be a compiler only." arguments, but it
>>seems there is a genuine demand in the free software community for a
>>high-quality source analyzer, particularly for languages like C++ where
>>this is not trivial.  GCC seems to be very nearly where it needs to be
>>to fill this need.
> 
> 
> Agreed; in any case the proprietary software world just licenses a very good
> C++ front end from [some company, since we're not supposed to plug proprietary
> software on this list, but most of us know who], for quite reasonable cost,
> so the fact that they can't use the g++ front end doesn't slow them down
> much.
> 
> But this is an argument that you'll have to make with RMS.
> 

   Will that still be a problem if such a back-end would be required to 
be supplied in source-form (read: a script)? The front-end would then 
first compile the script and check that it's compliant to whatever 
rules, before compiling actual C(++) code and applying the script's 
actions on the tree.

Re: -fdump-translation-unit considered harmful

[Stefan Strasser]

Pjotr Kourzanov schrieb:
>> For anyone interested in interfacing proprietary programs to GCC, that's
>> not even much of a speed bump.
>>
>> As soon as that binary API is written, someone will write a small calling
>> program that will use the API to dump the translation unit into a nicely
>> documented XML format.  The dumper will, of course, be released under the
>> GPL.  Then the proprietary backend will simply read in the XML data.
> 
> 
>   I suspect the XML output thus generated will be comparable is size to 
> what -fdump-translation-unit generates now. As it is, it takes *ages* to 
> re-parse those megabytes of ASCII data (even for simple C source files 
> including lots of headers). The really problematic case is only when a 
> binary API at the tree level is exposed.

I really don't understand why this is so much of a political problem. I 
understand the intentions, but isn't compiling proprietary code with gcc 
the same? and that is explicitely allowed.
then, the API is already there, cp tree. it's just not very stable so 
another level of abstraction would be nice to make it stable. but I 
don't think this will prevent anyone to export data. (at least, not me)
to make things clear, I have no proprietary interest in this, but I am 
following the XML approach stated above for other reasons. (with the 
exception of "nicely documented" ;)


-- 
Stefan Strasser

Re: -fdump-translation-unit considered harmful

[Joe Buck]

On Tue, Dec 21, 2004 at 01:44:56AM +0100, Stefan Strasser wrote:
> I really don't understand why this is so much of a political problem. I 
> understand the intentions, but isn't compiling proprietary code with gcc 
> the same? and that is explicitely allowed.

RMS would tell you that we only have a GNU C++ compiler because Mike
Tiemann's employer could not make it proprietary, and we only have a
GNU Objective-C compiler because Steve Jobs could not make it proprietary.
Had the equivalent of dump formats existed at the time, we'd only have
C.  (Of course, that's the inverse issue: proprietary front ends connecting
to GNU back ends).  By making just building onto GCC and GPLing the code
the path of least resistance, RMS hopes to motivate more people to produce
free software.

Re: -fdump-translation-unit considered harmful

[peter.kourzanov]

Stefan,

  OK, great. Would you release the code itself when its stable?

  I've just tried optimising my AWK script a bit. Here are the
results on tcp_input.c from Linux 2.4.26 (a rather complex piece of
networking code of 120806 bytes):

gcc-3.3 -fdump-translation-unit -c tcp_input.c -D__KERNEL__ -I/usr/src/linux-2.4.26/include
-Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing
-fno-common -fomit-frame-pointer -pipe -mpreferred-stack-boundary=2
-march=i686 -nostdinc -iwithprefix include

(options required by Linux kernel)

generated a .tu file of 8504511 bytes which took about 23 seconds
(real-time) to re-parse (a C representation is re-generated). 

  I am really curious what mileage will be for a binary gcc tree
interface (e.g., c++ model) for this code.

Pjotr Kourzanov

On Wed, Dec 22, 2004 at 08:39:07AM +0100, Stefan Strasser wrote:
> I've just parsed the model.h I sent you itself into the Model defined in 
> model.h. now THAT'S cool ;-)
> 
> worked. including its headers. (still ignoring templates).
> the .tu file(you know, the output of -fdump-translation-unit), for this 
> is 6 MB. memory consumption isn't really noticeable. might be different 
> for large function bodies but we'll see.
> 
> I'm downloading libxml :)
>