Re: [gnat] reuse of ASTs already constructed

Re: [gnat] reuse of ASTs already constructed

[oliver.kellogg]

[-- Attachment #1: Type: text/plain, Size: 1571 bytes --]

I am finally finding some some time to continue this work,
having left off with
  http://gcc.gnu.org/ml/gcc/2009-08/msg00475.html :

> [...]
> There are two problems right now:
> 
> 1) Mixing of Ada.Text_IO and Text_IO as described at
>    http://gcc.gnu.org/ml/gcc-help/2009-08/msg00113.html

The solution to this one turned out to be quite simple, see attached patch.
Sem_Ch12.Analyze_Package_Instantiation calls Rtsfind.Text_IO_Kludge,
and the latter contains:

          if Chrs in Text_IO_Package_Name then
             for U in Main_Unit .. Last_Unit loop
                Get_Name_String (Unit_File_Name (U));
                ...
On compiling the second file, Main_Unit is not 0 but some larger value.
The existing node for Text_IO was not being found because it is at an
index smaller than the current Main_Unit. The solution was to start the
loop at index 0.

I will now start looking into the second problem,

> 2) The 'X' lines in the ALI files are not what they should be.
> This is due to the fact that Lib.Xref.Generate_(Definition|Reference) is
> called during semantic analysis. However, when I discover that a
> tree was already built for a main unit by a previous compilation,
> Sem is not redone for that tree. Depending on whether
> Lib.Xref.Initialize is called per-job or per-file, this leads to either
> too much or too little cross ref info.

By the way, I am currently using trunk r152433 (4.5.0 pre-release) and
moving forward to the 4.5.0 release.
When that is done: Should I continue on trunk or switch to the
gcc-4_5-branch ?

Thanks,

Oliver

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch; name="rtsfind.adb.diff", Size: 472 bytes --]

Index: rtsfind.adb
===================================================================
--- rtsfind.adb	(revision 152433)
+++ rtsfind.adb	(working copy)
@@ -1406,7 +1406,7 @@
       --  or [Wide_]Wide_Text_IO already loaded, then load the proper child.
 
       if Chrs in Text_IO_Package_Name then
-         for U in Main_Unit .. Last_Unit loop
+         for U in 0 .. Last_Unit loop
             Get_Name_String (Unit_File_Name (U));
 
             if Name_Len = 12 then

Re: [gnat] reuse of ASTs already constructed

[Arnaud Charlet]

> I will now start looking into the second problem,
> 
> > 2) The 'X' lines in the ALI files are not what they should be.
> > This is due to the fact that Lib.Xref.Generate_(Definition|Reference)
> > is
> > called during semantic analysis. However, when I discover that a
> > tree was already built for a main unit by a previous compilation,
> > Sem is not redone for that tree. Depending on whether
> > Lib.Xref.Initialize is called per-job or per-file, this leads to either
> > too much or too little cross ref info.
> 
> By the way, I am currently using trunk r152433 (4.5.0 pre-release) and
> moving forward to the 4.5.0 release.
> When that is done: Should I continue on trunk or switch to the
> gcc-4_5-branch ?

GCC 4.5 is quite old at this stage, so you should definitely update
to the latest trunk revision.

Arno

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

[-- Attachment #1: Type: text/plain, Size: 1560 bytes --]

At http://gcc.gnu.org/ml/gcc/2012-06/msg00332.html,
Arnaud Charlet wrote:

> > In multi source compilations, main units may get assigned different
> > unit numbers than in single source compilations; this is due to the
> > reuse of entries in the Lib.Units table.
> > 
> > Is that tolerable?
> 
> You'd need to check whether the IDEs (in particular GPS) will not have
> troubles with this.

It turns out that in this case, my analysis was incorrect.
The reason for the different unit numbers in 'X' lines is that a few
dependencies ('D' lines) are missing. I will look into that soon.

Meanwhile, I have made some comparisons between single and multi
file compilations. For a package hierarchy consisting of 262 specs
totalling 82164 SLOC and 77 bodies totalling 135771 SLOC, compiling
the bodies yields the following results.

1. many gnat1 invocations without Ada syntax tree sharing

time gcc -S *.adb
real    9m12.159s
user    4m59.391s
sys     0m7.061s

time gcc -S -g *.adb
real    9m59.429s
user    8m57.776s
sys     0m5.750s

2. single gnat1 invocation with Ada syntax tree sharing

time gcc -multi -S *.adb
real    4m22.915s
user    3m34.466s
sys     0m2.900s

time gcc -multi -S -g *.adb
real    7m16.770s
user    6m29.554s
sys     0m2.711s

I have not been able to compare optimizing compilation due
to a crash in multi mode when using the -O switch.
(Right now I'm still concentrating on the front end, and I will
look into that later.)

The comparison was done using the attached patch which is
against gcc/trunk r177548 of 2011-08-07.

-- Oliver

[-- Attachment #2: Type: application/x-gzip, Size: 42427 bytes --]

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

Hello,

The patch attached at http://gcc.gnu.org/ml/gcc/2012-06/msg00261.html
was just barely enough to get gnat1 compiled but failed on compiling the
rts files.  Please replace it by the patch here.  (This one is against
gcc/trunk r162542 of 2010-07-26 but I'm moving forward these days.)

One occasional ALI difference wrt single compile mode is the unit
number appearing in 'X' lines, for example:

--- single/blanks_to_underscores.ali  2012-06-23 18:15:09
+++ multi/blanks_to_underscores.ali   2012-06-23 19:07:39
@@ -7,20 +7,17 @@
 .....
-X 3 blanks_to_underscores.adb
+X 1 blanks_to_underscores.adb

In multi source compilations, main units may get assigned different
unit numbers than in single source compilations; this is due to the
reuse of entries in the Lib.Units table.

Is that tolerable?

Thanks,

Oliver


On 2012-06-17 17:12, Oliver Kellogg wrote:
> 
> Continuing on from http://gcc.gnu.org/ml/gcc/2012-04/msg00654.html,
> [...]
> Here is my current patch (against trunk r158428), plus a test case
> that demonstrates the fix for the problem described, and also shows
> remaining problems with ALI generation in the multi source compile
> mode.
> 

[-- Attachment #2: Type: application/x-gzip, Size: 38052 bytes --]

Re: [gnat] reuse of ASTs already constructed

[Arnaud Charlet]

> --- single/blanks_to_underscores.ali  2012-06-23 18:15:09
> +++ multi/blanks_to_underscores.ali   2012-06-23 19:07:39
> @@ -7,20 +7,17 @@
>  .....
> -X 3 blanks_to_underscores.adb
> +X 1 blanks_to_underscores.adb
> 
> In multi source compilations, main units may get assigned different
> unit numbers than in single source compilations; this is due to the
> reuse of entries in the Lib.Units table.
> 
> Is that tolerable?

You'd need to check whether the IDEs (in particular GPS) will not have
troubles with this.

Arno

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]

Continuing on from http://gcc.gnu.org/ml/gcc/2012-04/msg00654.html,

> I looked into the second problem from
> http://gcc.gnu.org/ml/gcc/2009-08/msg00475.html ,
> 
> > 2) The 'X' lines in the ALI files are not what they should be.
> > This is due to the fact that
> > Lib.Xref.Generate_(Definition|Reference) is called during semantic
> > analysis. However, when I discover that a tree was already built for
> > a main unit by a previous compilation, Sem is not redone for that
> > tree. [...]
> > 
> 
> and I see two possible solutions:
> 
> 1) Extend Lib.Xref.Generate_Reference, Sem_Util.Process_End_Label and
> others for the case "not In_Extended_Main_Source_Unit" in multi source
> compilation mode to buffer the generated references for possible later
> consumption by the main unit for which they are intended.
> If the main unit is not part of the main units given in the multi
> source compile job then the buffered data can be discarded.
> 
> 2) Determine an ordering of the main units such that main units with
> little or no dependencies precede main units that depend on them.
> Submit units to semantic analysis in the determined order.

The second approach would have entailed earth shaking changes to the
GNAT code base so I chose the first approach.
Here is my current patch (against trunk r158428), plus a test case that
demonstrates the fix for the problem described, and also shows remaining
problems with ALI generation in the multi source compile mode.

Oliver



[-- Attachment #2: Type: application/x-gzip, Size: 40540 bytes --]

[-- Attachment #3: Type: application/x-gzip, Size: 1365 bytes --]

Re: [gnat] reuse of ASTs already constructed

[oliver.kellogg]

I looked into the second problem from
  http://gcc.gnu.org/ml/gcc/2009-08/msg00475.html ,

> 2) The 'X' lines in the ALI files are not what they should be.
> This is due to the fact that Lib.Xref.Generate_(Definition|Reference) is
> called during semantic analysis. However, when I discover that a
> tree was already built for a main unit by a previous compilation,
> Sem is not redone for that tree. [...]

and I see two possible solutions:

1) Extend Lib.Xref.Generate_Reference, Sem_Util.Process_End_Label
and others for the case "not In_Extended_Main_Source_Unit" in multi
source compilation mode to buffer the generated references for possible
later consumption by the main unit for which they are intended.
If the main unit is not part of the main units given in the multi source
compile job then the buffered data can be discarded.

2) Determine an ordering of the main units such that main units with
little or no dependencies precede main units that depend on them.
Submit units to semantic analysis in the determined order.

Both approaches seem quite heavy.
Solution 2 looks more elegant to me. OTOH this would require a
separate analysis of the with clauses prior to invoking Sem proper.

Oliver

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

I've been making progress on the Ada side, the basic usage pattern for
gnat1 and gnatmake is working.

There are two problems right now:

1) Mixing of Ada.Text_IO and Text_IO as described at
   http://gcc.gnu.org/ml/gcc-help/2009-08/msg00113.html

2) The 'X' lines in the ALI files are not what they should be.
This is due to the fact that Lib.Xref.Generate_(Definition|Reference) is
called during semantic analysis. However, when I discover that a
tree was already built for a main unit by a previous compilation,
Sem is not redone for that tree. Depending on whether
Lib.Xref.Initialize is called per-job or per-file, this leads to either
too much or too little cross ref info.

Ideas?

Thanks,

 Oliver


On Thu, 2009-07-02 at 00:23 +0200, Oliver Kellogg wrote:
> I am approaching the point where the basic multi-source mechanism
> in the gnat1/gnatmake Ada code is working but I am getting
> lots of crashes in the gigi/gnat_to_gnu triggered core gcc code.
> I will have to learn much more about the GCC internals than
> I know right now so progress will inevitably be slow.
> 
> For your information I am appending the current patch that
> I am using, as well as a log of a valgrind session that shows
> some of the current problems.
> 
> On Wed, 2009-04-22 at 18:22 -0400, Robert Dewar wrote:
> > Oliver Kellogg wrote:
> > > [...]
> > > One more thing, procedure Initialize crashes with a libc report of
> > > double free() when invoked more than once. (Found this while
> > > working on the gnat1 multi-source compile feature, see
> > > http://gcc.gnu.org/ml/gcc/2009-04/msg00380.html)
> > > FYI I'm attaching the patch that works for me.
> > 
> > gnat1 does not have the capability of compiling more than one
> > source file without being loaded, yes there is some preparation
> > for this, but it would require major work to get this working,
> > there are many known problems.
> 
> For interest, what are the known problems?
> 
> Thanks,
> 
> Oliver
> 

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

Consider compilation of the following code:

   function Exception_Id (Self : in Object) return CORBA.String is
   begin
      return To_Unbounded_String
               (Interfaces.C.Strings.Value
                   (Self.C_Str));
   end Exception_Id;

The interesting part is the call to Interfaces.C.Strings.Value which
returns String.

If this is in the the first file compiled, there is no problem.
However, if this is the second file of a multi file compile job,
a GNAT node may have already been built for the entity "value"
(i.e. the same function call already appeared in a previous file.)

The problem is in gnat_to_gnu_entity (ada/gcc-interface/decl.c:3996ff.)

    /* If the Mechanism is By_Reference, ensure the return type uses
       the machine's by-reference mechanism, which may not the same
       as above (e.g., it might be by passing a fake parameter).  */
    else if (kind == E_Function
             && Mechanism (gnat_entity) == By_Reference)
      {
        TREE_ADDRESSABLE (gnu_return_type) = 1;
        ....
      }

Since the entity's Mechanism attribute was changed from Default to
By_Reference by a previous compilation (see Set_Mechanism calls in
decl.c), the "if" condition here becomes true, and TREE_ADDRESSABLE
is set true on the gnu_return_type.
But that seems to wreak havoc with the further processing.

I'm not sure about how best to address this problem.
Perhaps the easy way out would be to buffer the original value of 
the Mechanism of all entities, and restore those values before
compiling the next file?

Thanks,

Oliver



On Wed, 2009-08-26 at 19:45 +0200, Oliver Kellogg wrote:
> I've been making progress on the Ada side, the basic usage pattern for
> gnat1 and gnatmake is working.
> 
> There are two problems right now:
> 
> 1) Mixing of Ada.Text_IO and Text_IO as described at
>    http://gcc.gnu.org/ml/gcc-help/2009-08/msg00113.html
> 
> 2) The 'X' lines in the ALI files are not what they should be.
> This is due to the fact that Lib.Xref.Generate_(Definition|Reference) is
> called during semantic analysis. However, when I discover that a
> tree was already built for a main unit by a previous compilation,
> Sem is not redone for that tree. Depending on whether
> Lib.Xref.Initialize is called per-job or per-file, this leads to either
> too much or too little cross ref info.
> 
> Ideas?
> 
> Thanks,
> 
>  Oliver
> 
> 
> On Thu, 2009-07-02 at 00:23 +0200, Oliver Kellogg wrote:
> > I am approaching the point where the basic multi-source mechanism
> > in the gnat1/gnatmake Ada code is working but I am getting
> > lots of crashes in the gigi/gnat_to_gnu triggered core gcc code.
> > I will have to learn much more about the GCC internals than
> > I know right now so progress will inevitably be slow.
> > 
> > For your information I am appending the current patch that
> > I am using, as well as a log of a valgrind session that shows
> > some of the current problems.
> > 
> > On Wed, 2009-04-22 at 18:22 -0400, Robert Dewar wrote:
> > > Oliver Kellogg wrote:
> > > > [...]
> > > > One more thing, procedure Initialize crashes with a libc report of
> > > > double free() when invoked more than once. (Found this while
> > > > working on the gnat1 multi-source compile feature, see
> > > > http://gcc.gnu.org/ml/gcc/2009-04/msg00380.html)
> > > > FYI I'm attaching the patch that works for me.
> > > 
> > > gnat1 does not have the capability of compiling more than one
> > > source file without being loaded, yes there is some preparation
> > > for this, but it would require major work to get this working,
> > > there are many known problems.
> > 
> > For interest, what are the known problems?
> > 
> > Thanks,
> > 
> > Oliver
> > 

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]

Picking up an old thread,
http://gcc.gnu.org/ml/gcc/2003-03/msg00281.html


On Tue, 4 Mar 2003, Geert Bosch <bosch at gnat dot com> wrote:
> [...]
> Best would be to first post a design overview,
> before doing a lot of work in order to prevent spending time
> on implementing something that may turn out to have fundamental
> problems.

I've done a little experimenting to get a feel for this.

I've looked at the work done toward the GCC compile server but
decided that I want to concentrate on GNAT trees (whereas the
compile server targets the GNU trees.)

Also I am aiming somewhat lower - not making a separate compile
server process but rather extending gnat1 to handle multiple
files in a single invocation.

The current GNAT code makes a strong assumption that there be
only one main unit, and this Main_Unit be located at index 0 of
Lib.Units.Table (see procedure Lib.Load.Load_Main_Source).

I am currently looking at having each main unit supplied on
the gnat1 command line overwrite the Main_Unit in the Units table.

What do you think of this approach?

The attached patch sets the stage for passing multiple source
files to a single gnat1 invocation. (Beware, this is a rough cut.
Best use "svn diff --diff-cmd diff -x -uw" after applying as
there are many changes that only affect indentation.)

Thanks,

Oliver



[-- Attachment #2: gnat1_multi_source_compile-0.diff.gz --]
[-- Type: application/x-gzip, Size: 9248 bytes --]

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

Interim note:

Apparently, calling compile_file() more than once has not been done
before?
I am seeing many global variables in cgraphunit.c that need to
be reinitialized in this case.
Also, some static variables are defined locally in functions, e.g.
'first_analyzed' and 'first_analyzed_var' in cgraph_analyze_functions().
Those need to be pulled outside for reinitialization.

I am making the necessary changes and extending init_cgraph() as needed.

I hope my changes stand a chance for being integrated ;)

Oliver


On 2009-04-12 at 19:29 +0200, Oliver Kellogg wrote:
> Picking up an old thread,
> http://gcc.gnu.org/ml/gcc/2003-03/msg00281.html
> 
> 
> On Tue, 4 Mar 2003, Geert Bosch <bosch at gnat dot com> wrote:
> > [...]
> > Best would be to first post a design overview,
> > before doing a lot of work in order to prevent spending time
> > on implementing something that may turn out to have fundamental
> > problems.
> 
> I've done a little experimenting to get a feel for this.
> 
> I've looked at the work done toward the GCC compile server but
> decided that I want to concentrate on GNAT trees (whereas the
> compile server targets the GNU trees.)
> 
> Also I am aiming somewhat lower - not making a separate compile
> server process but rather extending gnat1 to handle multiple
> files in a single invocation.
> 
> The current GNAT code makes a strong assumption that there be
> only one main unit, and this Main_Unit be located at index 0 of
> Lib.Units.Table (see procedure Lib.Load.Load_Main_Source).
> 
> I am currently looking at having each main unit supplied on
> the gnat1 command line overwrite the Main_Unit in the Units table.
> 
> What do you think of this approach?
> 
> The attached patch sets the stage for passing multiple source
> files to a single gnat1 invocation. (Beware, this is a rough cut.
> Best use "svn diff --diff-cmd diff -x -uw" after applying as
> there are many changes that only affect indentation.)
> 
> Thanks,
> 
> Oliver
> 
> 

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

I've run up against a problem with reusing the GNAT trees:
Apparently during semantic analysis entity names are expanded
to their fully qualified names in the tree.

For example, when compiling the following file with the node
for System already installed by a previous compilation,

-- file: ext.ads
with System;
procedure Ext (Addr : System.Address);

I get the following error from sem_ch8.adb Find_Expanded_Name
during semantic analysis of the above System.Address,

ext.ads:3:29: "Address" is not a visible entity of "System"

This is because in sem_ch8.adb:4482,

  if No (Id) or else Chars (Id) /= Chars (Selector) then ...

the second part of the condition becomes true as Chars(Id) is
"system__address" [and Chars(Selector) is "address"].
When compiling the same file afresh without preinstalled nodes,
Chars(Id) is just "address" and all is fine.

Oliver


On 2009-04-12 at 19:29 +0200, Oliver Kellogg wrote:
> Picking up an old thread,
> http://gcc.gnu.org/ml/gcc/2003-03/msg00281.html
> 
> 
> On Tue, 4 Mar 2003, Geert Bosch <bosch at gnat dot com> wrote:
> > [...]
> > Best would be to first post a design overview,
> > before doing a lot of work in order to prevent spending time
> > on implementing something that may turn out to have fundamental
> > problems.
> 
> I've done a little experimenting to get a feel for this.
> 
> I've looked at the work done toward the GCC compile server but
> decided that I want to concentrate on GNAT trees (whereas the
> compile server targets the GNU trees.)
> 
> Also I am aiming somewhat lower - not making a separate compile
> server process but rather extending gnat1 to handle multiple
> files in a single invocation.
> 
> The current GNAT code makes a strong assumption that there be
> only one main unit, and this Main_Unit be located at index 0 of
> Lib.Units.Table (see procedure Lib.Load.Load_Main_Source).
> 
> I am currently looking at having each main unit supplied on
> the gnat1 command line overwrite the Main_Unit in the Units table.
> 
> What do you think of this approach?
> 
> The attached patch sets the stage for passing multiple source
> files to a single gnat1 invocation. (Beware, this is a rough cut.
> Best use "svn diff --diff-cmd diff -x -uw" after applying as
> there are many changes that only affect indentation.)
> 
> Thanks,
> 
> Oliver
> 
> 

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

<PING> Anybody out there?

How about not doing the name expansion in-place but rather
storing the expanded name in an extra node field?

--Oliver


On 2009-04-18 at 21:35 +0200, Oliver Kellogg wrote:
> I've run up against a problem with reusing the GNAT trees:
> Apparently during semantic analysis entity names are expanded
> to their fully qualified names in the tree.
> 
> For example, when compiling the following file with the node
> for System already installed by a previous compilation,
> 
> -- file: ext.ads
> with System;
> procedure Ext (Addr : System.Address);
> 
> I get the following error from sem_ch8.adb Find_Expanded_Name
> during semantic analysis of the above System.Address,
> 
> ext.ads:3:29: "Address" is not a visible entity of "System"
> 
> This is because in sem_ch8.adb:4482,
> 
>   if No (Id) or else Chars (Id) /= Chars (Selector) then ...
> 
> the second part of the condition becomes true as Chars(Id) is
> "system__address" [and Chars(Selector) is "address"].
> When compiling the same file afresh without preinstalled nodes,
> Chars(Id) is just "address" and all is fine.
> 
> Oliver
> 
> 
> On 2009-04-12 at 19:29 +0200, Oliver Kellogg wrote:
> > Picking up an old thread,
> > http://gcc.gnu.org/ml/gcc/2003-03/msg00281.html
> > 
> > 
> > On Tue, 4 Mar 2003, Geert Bosch <bosch at gnat dot com> wrote:
> > > [...]
> > > Best would be to first post a design overview,
> > > before doing a lot of work in order to prevent spending time
> > > on implementing something that may turn out to have fundamental
> > > problems.
> > 
> > I've done a little experimenting to get a feel for this.
> > 
> > I've looked at the work done toward the GCC compile server but
> > decided that I want to concentrate on GNAT trees (whereas the
> > compile server targets the GNU trees.)
> > 
> > Also I am aiming somewhat lower - not making a separate compile
> > server process but rather extending gnat1 to handle multiple
> > files in a single invocation.
> > 
> > The current GNAT code makes a strong assumption that there be
> > only one main unit, and this Main_Unit be located at index 0 of
> > Lib.Units.Table (see procedure Lib.Load.Load_Main_Source).
> > 
> > I am currently looking at having each main unit supplied on
> > the gnat1 command line overwrite the Main_Unit in the Units table.
> > 
> > What do you think of this approach?
> > 
> > The attached patch sets the stage for passing multiple source
> > files to a single gnat1 invocation. (Beware, this is a rough cut.
> > Best use "svn diff --diff-cmd diff -x -uw" after applying as
> > there are many changes that only affect indentation.)
> > 
> > Thanks,
> > 
> > Oliver
> > 
> > 

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

On 2009-04-19, at 23:19 +0200, Oliver Kellogg wrote:
> [...]
> 
> How about not doing the name expansion in-place but rather
> storing the expanded name in an extra node field?

I haven't received any reaction on this question yet.
Perhaps I could reuse the Str3 field instead? (I haven't looked
into the depths of its usage yet.)
Please advise.

BTW, I added the invocation of 'as' for assembly files 2 to N
in gcc.c, this means that the multi-source mechanism can now
be used in a normal "gcc -c" invocation using the -multi switch.

Up next would be gnatmake:
I would like to add a switch to gnatmake that enables multi-
source mode. What should I use for the switch name?

Also, I'm thinking that in multi-source mode, the switch "-o"
can perhaps continue to be used - not by giving a filename
but instead by giving a directory. All object files would then
be placed in the given directory. What do you think?

Thanks,

Oliver


> On 2009-04-18 at 21:35 +0200, Oliver Kellogg wrote:
> > I've run up against a problem with reusing the GNAT trees:
> > Apparently during semantic analysis entity names are expanded
> > to their fully qualified names in the tree.
> > 
> > For example, when compiling the following file with the node
> > for System already installed by a previous compilation,
> > 
> > -- file: ext.ads
> > with System;
> > procedure Ext (Addr : System.Address);
> > 
> > I get the following error from sem_ch8.adb Find_Expanded_Name
> > during semantic analysis of the above System.Address,
> > 
> > ext.ads:3:29: "Address" is not a visible entity of "System"
> > 
> > This is because in sem_ch8.adb:4482,
> > 
> >   if No (Id) or else Chars (Id) /= Chars (Selector) then ...
> > 
> > the second part of the condition becomes true as Chars(Id) is
> > "system__address" [and Chars(Selector) is "address"].
> > When compiling the same file afresh without preinstalled nodes,
> > Chars(Id) is just "address" and all is fine.
> > 
> > Oliver
> > 

Re: [gnat] reuse of ASTs already constructed

[Robert Dewar]

Oliver Kellogg wrote:
> On 2009-04-19, at 23:19 +0200, Oliver Kellogg wrote:
>> [...]
>>
>> How about not doing the name expansion in-place but rather
>> storing the expanded name in an extra node field?
> 
> I haven't received any reaction on this question yet.
> Perhaps I could reuse the Str3 field instead? (I haven't looked
> into the depths of its usage yet.)

I don't see any room in the node for this extra field ...

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

Sorry if I'm slow in understanding:
Are you saying that introducing an extra field would
cause problems (memory or other) ?
Do you think it would be okay to use the Str3 field, then?

Thanks.

Robert Dewar wrote:
> Oliver Kellogg wrote:
> > On 2009-04-19, at 23:19 +0200, Oliver Kellogg wrote:
> >> [...]
> >>
> >> How about not doing the name expansion in-place but rather
> >> storing the expanded name in an extra node field?
> > 
> > I haven't received any reaction on this question yet.
> > Perhaps I could reuse the Str3 field instead? (I haven't looked
> > into the depths of its usage yet.)
> 
> I don't see any room in the node for this extra field ...
> 

Re: [gnat] reuse of ASTs already constructed

[Robert Dewar]

Oliver Kellogg wrote:
> Sorry if I'm slow in understanding:
> Are you saying that introducing an extra field would
> cause problems (memory or other) ?
> Do you think it would be okay to use the Str3 field, then?

I don't see that Str3 is free in any of these nodes, what
nodes are you talking about (remember that Str3 overlaps
Node3, List3 etc).
> 
> Thanks.
> 
> Robert Dewar wrote:
>> Oliver Kellogg wrote:
>>> On 2009-04-19, at 23:19 +0200, Oliver Kellogg wrote:
>>>> [...]
>>>>
>>>> How about not doing the name expansion in-place but rather
>>>> storing the expanded name in an extra node field?

You could have a separate vector for expanded names I suppose ...

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

Robert Dewar wrote:
> 
> I don't see that Str3 is free in any of these nodes, what
> nodes are you talking about (remember that Str3 overlaps
> Node3, List3 etc).

Okay.

> >>>> How about not doing the name expansion in-place but rather
> >>>> storing the expanded name in an extra node field?
> 
> You could have a separate vector for expanded names I suppose ...

So be it. I will change the code to not overwrite the Name field
with the expanded name but rather store the expanded name in a
data structure separate from the Node.

Any thoughts on my questions related to gnatmake?

Further question, what is the process for integrating my changes
into the GCC trunk?
I would assume that I need to
1) Make my modifications complete and functioning
2) Make sure bootstrap works for all languages
3) Run the gcc testsuite and make sure I do not introduce regressions
4) What about testing on different platforms? I'm only testing on x86
 SuSE Linux and setting up other platforms could be a problem.
 (Coming to think, I could actually test on x86_64 SuSE Linux with
 small effort, though.)
5) Copyright assignment should be on file at FSF, please check
6) Anything else?

Thanks,

Oliver

Re: [gnat] reuse of ASTs already constructed

[Dave Korn]

Oliver Kellogg wrote:

> Further question, what is the process for integrating my changes
> into the GCC trunk?

  The generic procedure is documented at the gcc web site:

	http://gcc.gnu.org/contribute.html
	http://gcc.gnu.org/svnwrite.html

> I would assume that I need to
> 1) Make my modifications complete and functioning
> 2) Make sure bootstrap works for all languages
> 3) Run the gcc testsuite and make sure I do not introduce regressions

  Basically, that's what it says on those pages, yep!  Probably still worth
having a quick browse of the above links, to fill out the details.

> 4) What about testing on different platforms? I'm only testing on x86
>  SuSE Linux and setting up other platforms could be a problem.
>  (Coming to think, I could actually test on x86_64 SuSE Linux with
>  small effort, though.)

  Knowing how much testing to do can be tricky, it depends on the nature of
the changes.  Target-specific changes need testing on the targets they affect;
with more generic changes (IIUC you're contemplating FE changes in this
thread) it's usually sufficient to show bootstrap + no testsuite regressions
on any of the primary or secondary supported platforms.  I'd recommend doing
that before submitting the patch, and the relevant maintainers can ask you to
do further testing if they feel it's needed.

  Testing on both 32 and 64 bit platforms is an obvious good way to spot minor
type-size-assumption bugs.  One thing that does have a tendency to go a bit
undertested and break from time to time is SjLj EH, which GNAT uses on at
least some platforms (non-ZCX targets), you might want to try testing one each
way for extra brownie points :)

    cheers,
      DaveK

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

On 2009-05-04, at 07:36 +0200, Oliver Kellogg wrote:
> Robert Dewar wrote:
> > 
> > >>>> How about not doing the name expansion in-place but rather
> > >>>> storing the expanded name in an extra node field?
> > 
> > You could have a separate vector for expanded names I suppose ...
> 
> So be it. I will change the code to not overwrite the Name field
> with the expanded name but rather store the expanded name in a
> data structure separate from the Node.
> 

Luckily, this change was limited to the Exp_Dbug package body.
FYI I am appending the diff.
The new Q_E_Name array holds the Name_Id for the expanded name
of a given entity. The implementation as a fixed array is
preliminary and I would appreciate suggestions on a better
data structure to use.

Oliver



[-- Attachment #2: exp_dbug.adb.diff --]
[-- Type: text/x-patch, Size: 6807 bytes --]

Index: exp_dbug.adb
===================================================================
--- exp_dbug.adb	(revision 147319)
+++ exp_dbug.adb	(working copy)
@@ -55,6 +55,31 @@
      Table_Increment      => Alloc.Name_Qualify_Units_Increment,
      Table_Name           => "Name_Qualify_Units");
 
+   --  Qualified Entity Name array is indexed by Entity_Id and
+   --  returns the Name_Id of the qualified name.
+
+   subtype Q_E_Name_Id is Entity_Id
+      range Entity_Id'First .. Entity_Id'First + 399999;
+
+   Q_E_Name : array (Q_E_Name_Id) of Name_Id := (others => No_Name);
+
+   function Get_Qualified_Name (E : Entity_Id) return Name_Id;
+
+   function Get_Qualified_Name (E : Entity_Id) return Name_Id is
+   begin
+      if Original_Operating_Mode /= Generate_Code then
+         return Chars (E);
+      else
+         pragma Assert (E in Q_E_Name_Id);
+         if Q_E_Name (E) = No_Name then
+            --  Write_Str ("Exp_Dbug.Get_Qualified_Name: returning Chars");
+            --  Write_Eol;
+            return Chars (E);
+         end if;
+      end if;
+      return Q_E_Name (E);
+   end Get_Qualified_Name;
+
    --------------------------------
    -- Use of Qualification Flags --
    --------------------------------
@@ -82,7 +107,7 @@
    --       B : Ddd.Ttt;
    --       procedure Y is ..
 
-   --  Here B is a procedure local variable, so it does not need fully
+   --  Here B is a procedure local variable, so it does not need full
    --  qualification. The flag Has_Qualified_Name will be set on the
    --  first attempt to qualify B, to indicate that the job is done
    --  and need not be redone.
@@ -143,8 +168,8 @@
    --  Prepend image of universal integer to Name_Buffer, updating Name_Len
 
    procedure Qualify_Entity_Name (Ent : Entity_Id);
-   --  If not already done, replaces the Chars field of the given entity
-   --  with the appropriate fully qualified name.
+   --  If not already done, puts the appropriate fully qualified name in the
+   --  Q_E_Name array element of the given entity.
 
    procedure Strip_Suffixes (BNPE_Suffix_Found : in out Boolean);
    --  Given an qualified entity name in Name_Buffer, remove any plain X or
@@ -512,7 +537,7 @@
          return;
       end if;
 
-      Get_Name_String (Chars (E));
+      Get_Name_String (Get_Qualified_Name (E));
 
       --  Nothing to do if we do not have a type
 
@@ -619,7 +644,8 @@
                if Lo_Con then
                   Add_Uint_To_Buffer (Expr_Rep_Value (Lo));
                elsif Lo_Discr then
-                  Get_Name_String_And_Append (Chars (Entity (Lo)));
+                  Get_Name_String_And_Append
+                    (Get_Qualified_Name (Entity (Lo)));
                end if;
 
                if Lo_Encode and Hi_Encode then
@@ -629,7 +655,8 @@
                if Hi_Con then
                   Add_Uint_To_Buffer (Expr_Rep_Value (Hi));
                elsif Hi_Discr then
-                  Get_Name_String_And_Append (Chars (Entity (Hi)));
+                  Get_Name_String_And_Append
+                    (Get_Qualified_Name (Entity (Hi)));
                end if;
             end if;
          end;
@@ -670,7 +697,7 @@
       procedure Get_Qualified_Name_And_Append (Entity : Entity_Id) is
       begin
          --  If the entity is a compilation unit, its scope is Standard,
-         --  there is no outer scope, and the no further qualification
+         --  there is no outer scope, and no further qualification
          --  is required.
 
          --  If the front end has already computed a fully qualified name,
@@ -682,11 +709,11 @@
          then
             Get_Qualified_Name_And_Append (Scope (Entity));
             Add_Str_To_Name_Buffer ("__");
-            Get_Name_String_And_Append (Chars (Entity));
+            Get_Name_String_And_Append (Get_Qualified_Name (Entity));
             Append_Homonym_Number (Entity);
 
          else
-            Get_Name_String_And_Append (Chars (Entity));
+            Get_Name_String_And_Append (Get_Qualified_Name (Entity));
          end if;
       end Get_Qualified_Name_And_Append;
 
@@ -935,7 +962,7 @@
       return  Name_Id
    is
    begin
-      Get_Name_String (Chars (Typ));
+      Get_Name_String (Get_Qualified_Name (Typ));
       Add_Str_To_Name_Buffer ("___XP");
       Add_Uint_To_Buffer (Csize);
       return Name_Find;
@@ -1077,9 +1104,6 @@
       BNPE_Suffix_Needed : Boolean := False;
       --  Set true if a body-nested package entity suffix is required
 
-      Save_Chars : constant Name_Id := Chars (Ent);
-      --  Save original name
-
       ------------------------
       -- Fully_Qualify_Name --
       ------------------------
@@ -1110,7 +1134,7 @@
          --  If we reached fully qualified name, then just copy it
 
          if Has_Fully_Qualified_Name (E) then
-            Get_Name_String (Chars (E));
+            Get_Name_String (Get_Qualified_Name (E));
             Strip_Suffixes (Discard);
             Full_Qualify_Name (1 .. Name_Len) := Name_Buffer (1 .. Name_Len);
             Full_Qualify_Len := Name_Len;
@@ -1221,7 +1245,7 @@
          --  will be separately put on later.
 
          if Has_Qualified_Name (E) then
-            Get_Name_String_And_Append (Chars (E));
+            Get_Name_String_And_Append (Get_Qualified_Name (E));
             Strip_Suffixes (BNPE_Suffix_Needed);
 
             --  If the top level name we are adding is itself fully
@@ -1306,13 +1330,13 @@
             Name_Len := Name_Len + Insert_Len;
          end;
 
-         --  Reset the name of the variable to the new name that includes the
+         --  Store in Q_E_Name the new name that includes the
          --  name of the renamed entity.
 
-         Set_Chars (Ent, Name_Enter);
+         Q_E_Name (Ent) := Name_Enter;
 
          --  If the entity needs qualification by its scope then develop it
-         --  here, add the variable's name, and again reset the entity name.
+         --  here, add the variable's name, and store it in Q_E_Name.
 
          if Qualify_Needed (Scope (Ent)) then
             Name_Len := 0;
@@ -1321,7 +1345,7 @@
 
             Get_Name_String_And_Append (Chars (Ent));
 
-            Set_Chars (Ent, Name_Enter);
+            Q_E_Name (Ent) := Name_Enter;
          end if;
 
          Set_Has_Qualified_Name (Ent);
@@ -1365,14 +1389,14 @@
          Name_Len := Name_Len - 1;
       end if;
 
-      Set_Chars (Ent, Name_Enter);
+      Q_E_Name (Ent) := Name_Enter;
       Set_Has_Qualified_Name (Ent);
 
       if Debug_Flag_BB then
          Write_Str ("*** ");
-         Write_Name (Save_Chars);
+         Write_Name (Chars (Ent));
          Write_Str (" qualified as ");
-         Write_Name (Chars (Ent));
+         Write_Name (Q_E_Name (Ent));
          Write_Eol;
       end if;
    end Qualify_Entity_Name;

Re: [gnat] reuse of ASTs already constructed

[Robert Dewar]

Oliver Kellogg wrote:
> On 2009-05-04, at 07:36 +0200, Oliver Kellogg wrote:
>> Robert Dewar wrote:
>>>>>>> How about not doing the name expansion in-place but rather
>>>>>>> storing the expanded name in an extra node field?
>>> You could have a separate vector for expanded names I suppose ...
>> So be it. I will change the code to not overwrite the Name field
>> with the expanded name but rather store the expanded name in a
>> data structure separate from the Node.
>>
> 
> Luckily, this change was limited to the Exp_Dbug package body.
> FYI I am appending the diff.
> The new Q_E_Name array holds the Name_Id for the expanded name
> of a given entity. The implementation as a fixed array is
> preliminary and I would appreciate suggestions on a better
> data structure to use.

use a table .. we never use fixed length arrays in the compiler
> 
> Oliver
> 
> 
> 

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

A further problem that I'm seeing is that
Lib.Writ.Write_Unit_Information assumes the current unit to depend on
all entries in Lib.Units. However, when letting Lib.Units accumulate
units across several compilations this dependency does not
necessarily hold true. The generated ALI files for compilation 2
to N contain more entries than needed.

--Oliver


On Sat, 2009-05-09 at 16:08 -0400, Robert Dewar wrote:
> Oliver Kellogg wrote:
> > On 2009-05-04, at 07:36 +0200, Oliver Kellogg wrote:
> >> Robert Dewar wrote:
> >>>>>>> How about not doing the name expansion in-place but rather
> >>>>>>> storing the expanded name in an extra node field?
> >>> You could have a separate vector for expanded names I suppose ...
> >> So be it. I will change the code to not overwrite the Name field
> >> with the expanded name but rather store the expanded name in a
> >> data structure separate from the Node.
> >>
> > 
> > Luckily, this change was limited to the Exp_Dbug package body.
> > FYI I am appending the diff.
> > The new Q_E_Name array holds the Name_Id for the expanded name
> > of a given entity. The implementation as a fixed array is
> > preliminary and I would appreciate suggestions on a better
> > data structure to use.
> 
> use a table .. we never use fixed length arrays in the compiler

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

[-- Attachment #1: Type: text/plain, Size: 209 bytes --]

I got my first main program built in multi-source mode running.
The ALI file and debug-info generation isn't perfect yet; also I'm
using raw gnat1 as gnatmake is not yet adapted to multi-source mode.

Oliver


[-- Attachment #2: gnat1_multi_demo.tgz --]
[-- Type: application/x-compressed-tar, Size: 10677 bytes --]

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

[-- Attachment #1: Type: text/plain, Size: 1270 bytes --]

I'm now using a variable length data structure for Q_E_Name
as suggested by Robert Dewar.

Also I've added a -multi option to gnatmake to switch on
the multi source compile mode. This allows test driving the
new feature.

I plan to send my next patch to gcc-patches and appreciate
any comments you may have.

Thanks,

Oliver


On Sat, 2009-05-09 at 16:08 -0400, Robert Dewar wrote:
> Oliver Kellogg wrote:
> > On 2009-05-04, at 07:36 +0200, Oliver Kellogg wrote:
> >> Robert Dewar wrote:
> >>>>>>> How about not doing the name expansion in-place but rather
> >>>>>>> storing the expanded name in an extra node field?
> >>> You could have a separate vector for expanded names I suppose ...
> >> So be it. I will change the code to not overwrite the Name field
> >> with the expanded name but rather store the expanded name in a
> >> data structure separate from the Node.
> >>
> > 
> > Luckily, this change was limited to the Exp_Dbug package body.
> > FYI I am appending the diff.
> > The new Q_E_Name array holds the Name_Id for the expanded name
> > of a given entity. The implementation as a fixed array is
> > preliminary and I would appreciate suggestions on a better
> > data structure to use.
> 
> use a table .. we never use fixed length arrays in the compiler


[-- Attachment #2: gnat1_multi_source_compile-2.diff.gz --]
[-- Type: application/x-gzip, Size: 27186 bytes --]

Re: [gnat] reuse of ASTs already constructed

[Tom Tromey]

>>>>> "Oliver" == Oliver Kellogg <oliver.kellogg@t-online.de> writes:

Oliver> Also, I'm thinking that in multi-source mode, the switch "-o"
Oliver> can perhaps continue to be used - not by giving a filename
Oliver> but instead by giving a directory. All object files would then
Oliver> be placed in the given directory. What do you think?

I think a new option would be preferable to overloading the meaning of
an existing one.

Tom

Re: [gnat] reuse of ASTs already constructed

[Geert Bosch]

On Apr 12, 2009, at 13:29, Oliver Kellogg wrote:

> On Tue, 4 Mar 2003, Geert Bosch <bosch at gnat dot com> wrote:
>> [...]
>> Best would be to first post a design overview,
>> before doing a lot of work in order to prevent spending time
>> on implementing something that may turn out to have fundamental
>> problems.
>
> I've done a little experimenting to get a feel for this.
>
> I've looked at the work done toward the GCC compile server but
> decided that I want to concentrate on GNAT trees (whereas the
> compile server targets the GNU trees.)
>
> Also I am aiming somewhat lower - not making a separate compile
> server process but rather extending gnat1 to handle multiple
> files in a single invocation.

While this may be an interesting idea, there are some fundamental  
assumptions
in the compiler that each compilation indeed processes a single  
compilation unit,
resulting in a single object and .ali file. It would be best to first  
contemplate
what output a single invocation of the compiler, with multiple  
compilation units
as arguments, should produce.

How would you decide if a unit needs recompilation if there was no 1:1
correspondence between compilation units and object/.ali files?
Note that unlike many other languages, Ada requires checks to avoid
including out-of-date compilation results in a program.

   -Geert

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

Hi Geert,

Thanks for your answer - I was starting to feel I'm entertaining a
monologue here ;)

On Mon, 2009-04-20 at 13:05 -0400, Geert Bosch wrote:
>
> While this may be an interesting idea, there are some fundamental  
> assumptions in the compiler that each compilation indeed processes
> a single compilation unit, resulting in a single object and .ali
> file. It would be best to first contemplate what output a single
> invocation of the compiler, with multiple compilation units
> as arguments, should produce.

For an invocation
 gnat1 a.adb b.adb c.adb
, the files a.{s,ali} b.{s,ali} c.{s,ali} are produced.

> How would you decide if a unit needs recompilation if there was no 1:1
> correspondence between compilation units and object/.ali files?

The correspondence is still 1:1, see above.

> Note that unlike many other languages, Ada requires checks to avoid
> including out-of-date compilation results in a program.

Certainly - but that's the job of gnatmake. I'm at the level of gnat1.
In the end, gnatmake would not generate a sequence of individual calls
to gcc but rather supply all files to be recompiled in a single call.

Oliver

Re: [gnat] reuse of ASTs already constructed

[Geert Bosch]

On Apr 20, 2009, at 14:45, Oliver Kellogg wrote:
>> It would be best to first contemplate what output a single
>> invocation of the compiler, with multiple compilation units
>> as arguments, should produce.
>
> For an invocation
> gnat1 a.adb b.adb c.adb
> , the files a.{s,ali} b.{s,ali} c.{s,ali} are produced.

The back end is not prepared to produce multiple assembly files.
The "gcc" driver program also assumes each invocation produces a
single .s file.

So, if this is what you want to do, you'd have to address all these
underlying limitations first.

   -Geert

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

On Mon, 2009-04-20 at 14:55 -0400, Geert Bosch wrote:
> > For an invocation
> > gnat1 a.adb b.adb c.adb
> > , the files a.{s,ali} b.{s,ali} c.{s,ali} are produced.
> 
> The back end is not prepared to produce multiple assembly files.
> The "gcc" driver program also assumes each invocation produces a
> single .s file.
> 
> So, if this is what you want to do, you'd have to address all these
> underlying limitations first.
> 

Sure, that's what I'm doing. 
See also the first rough patch which I attached to
http://gcc.gnu.org/ml/gcc/2009-04/msg00380.html , namely
http://gcc.gnu.org/ml/gcc/2009-04/msg00380/gnat1_multi_source_compile-0.diff.gz
(which meanwhile is outdated.)

Oliver

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

[-- Attachment #1: Type: text/plain, Size: 4634 bytes --]

Here's the current status of my work (see appended patch.)

gnat1 can now handle multiple input files as previously
described (produce one assembly and ALI file per input file.)

First tests look very promising. I'm getting noticeable speedups
of when supplying N interrelated bodies in a single call as opposed
to N individual calls to gnat1. Precise measurements will follow.

gcc has a new command line switch, "-multi".
This tells gcc to forward all given input files to a single execution
of the compiler proper, in a similar way as done for "-combine".
For now, the switch is not forwarded to the compilers, i.e. it is
purely a gcc driver switch.
It's not totally functioning yet:
1) invocation of "as" (assembler) is only done for the first file
2) gcc is passing too many switches to the compiler, for example
"-o" (output file) should be suppressed when in multi-source mode.

The Ada backend still has a problem with calls to instantiated
generic subprograms, there is a crash in gigi(), deep in gimplify(),
which only happens in multi-source mode. I can provide the details.

Here are more details:

* gcc.c and lang-specs.h of all languages:
Added a new flag to struct compiler, "multisource".
If nonzero, compiler can deal with multiple source files at once.
For each input file, a corresponding output file is generated.
This is different from "combine", where a single output file is
generated for all input files.
This flag is set to true only for Ada.

* gcc.c:
Added a new command line switch, "-multi".
This tells gcc to forward all given input files to a single execution
of the compiler proper, in a similar way as done for "-combine".
For now, this switch is not forwarded to the backends, i.e. it is
purely a gcc driver switch.

* toplev.c:
1) Had to make realloc_for_line_map() global because when repeatedly
calling linemap_init(), the member "reallocator" is NULL and I
couldn't find a way of setting it to the proper value again.
2) lang_dependent_init(name): When in multi_source mode, changed the
main_input_filename to the given "name" argument.  This change is
responsible for getting a separate assembly file per each input file.
3) do_compile(): When in multi_source mode, loop over in_fnames[],
calling lang_dependent_init(), compile_file(), and finalize() for
each name.

* cgraph.{h,c}, cgraphunit.c:
Added many missing reinitializations that are necessary when calling
compile_file() repeatedly.

* ada/frontend.ad{s,b}:
Changed this procedure into a package with two procedures, Initialize
and Call_Frontend. This was necessary because Initialize is only
once, but Call_Frontend is called per each input file.
ATTENTION: In order for the whole multi-source mechanism to work, atm
it is necessary to comment out line 389 as follows,
 if VM_Target = No_VM then
    null;  --  ATTENTION - THIS IS A TEMPORARY HACK - DO NOT COMMIT !
    --  Exp_Dbug.Qualify_All_Entity_Names;
 end if;
In other words, the qualification of all entity names currently
disturbs the name lookup for repeated runs of semantic analysis.

* ada/lib-load.ad{s,b}:
Added new flag Main_Unit_Previously_Loaded. This is needed because
Units.Increment_Last shall only be called for the first main unit
(further main units are stored in Units.Table(0) therefore no
incrementing is required.)

* ada/{inline,sem,fname-uf}.ad[sb]:
Add procedure Unlock, required for processing of subsequent main
units.

* ada/sem.adb, procedure Initialize: Fix glibc report of double
free() on reinvocation.

* ada/gcc-interface/misc.c:
gnat_parse_file(): Move one-time-only initializations to gnat_init().
gnat_init(): Added call to linemap_init().  Added a new flag,
first_call_to_gnat_init, in order to ensure that one-time-only
initializations are really done only once, even when gnat_init()
it called multiple times.

* ada/gnat1drv.adb:
Ensure that frontend initializations are done only once.
Reformat comments to obey GNAT style checks.
Unlock tables after processing the main unit.

Oliver


On 2009-04-20 at 14:55, Geert Bosch wrote:
> On Apr 20, 2009, at 14:45, Oliver Kellogg wrote:
> >> It would be best to first contemplate what output a single
> >> invocation of the compiler, with multiple compilation units
> >> as arguments, should produce.
> >
> > For an invocation
> > gnat1 a.adb b.adb c.adb
> > , the files a.{s,ali} b.{s,ali} c.{s,ali} are produced.
> 
> The back end is not prepared to produce multiple assembly files.
> The "gcc" driver program also assumes each invocation produces a
> single .s file.
> 
> So, if this is what you want to do, you'd have to address all these
> underlying limitations first.
> 
>    -Geert

[-- Attachment #2: gnat1_multi_source_compile-1.diff.gz --]
[-- Type: application/x-gzip, Size: 15072 bytes --]

Re: [gnat] reuse of ASTs already constructed

[Robert Dewar]

Oliver Kellogg wrote:
> Here's the current status of my work (see appended patch.)

This is great work, something we always thought about (as you
can see from some of the existing code), but seemed like a
very big task!

> gnat1 can now handle multiple input files as previously
> described (produce one assembly and ALI file per input file.)

> First tests look very promising. I'm getting noticeable speedups
> of when supplying N interrelated bodies in a single call as opposed
> to N individual calls to gnat1. Precise measurements will follow.

Definitely makes sense, the time it will really help is when you have
a big cluster of commonly used specs, especially if you have inlining
active. Of course there will be big memory usage, but these days that's
seeming less of a problem (the PC sitting next to me has 16 gigs of
memory, which is accessible in 64-bit mode).

Re: [gnat] reuse of ASTs already constructed

[Ian Lance Taylor]

oliver.kellogg@t-online.de (Oliver Kellogg) writes:

> gcc has a new command line switch, "-multi".
> This tells gcc to forward all given input files to a single execution
> of the compiler proper, in a similar way as done for "-combine".

This is good work, but why not just reuse -combine?  What is the
difference btween -multi and -combine?

Ian

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

On 2009-04-27 at 10:27, Ian Lance Taylor wrote:
> 
> > gcc has a new command line switch, "-multi".
> > This tells gcc to forward all given input files to a single execution
> > of the compiler proper, in a similar way as done for "-combine".
> 
> This is good work, but why not just reuse -combine?  What is the
> difference btween -multi and -combine?

-combine produces just one assembler file for all input files
while -multi produces one assembler file per each input file.

Actually, I started out reusing -combine for Ada (with the changed
semantics of N output files) until I realized that someday other
languages might also become multi-output capable.

Oliver

Re: [gnat] reuse of ASTs already constructed

[Tom Tromey]

>>>>> "Oliver" == Oliver Kellogg <oliver.kellogg@t-online.de> writes:

Oliver> First tests look very promising. I'm getting noticeable speedups
Oliver> of when supplying N interrelated bodies in a single call as opposed
Oliver> to N individual calls to gnat1. Precise measurements will follow.

This sounds like it has a lot of overlap with the incremental compiler
project I was working on.  I think you could probably rearrange things
to use the compile server infrastructure from the branch without
trouble.

There are some differences in invocation.  I designed the incremental
compiler so that no changes to user Makefiles would be needed; I don't
know whether that is a consideration with Ada.

I think the compile server infrastructure is in decent shape, though I
haven't worked on it in a while, so it is a bit hard to remember
exactly.  The difficult bit is converting a front end to be
server-ready; but from your note it sounds like you have already done
that.

Oliver> It's not totally functioning yet:
Oliver> 1) invocation of "as" (assembler) is only done for the first file

This is solved on the branch.  There, the server starts the assembler
from compile_file.

Oliver> 2) gcc is passing too many switches to the compiler, for example
Oliver> "-o" (output file) should be suppressed when in multi-source mode.

With the compile server you don't have to worry about this kind of
thing.  I changed the options processing code to be re-initializable;
each compiler job in the server runs with its own set of options.

Oliver> * toplev.c:
Oliver> 1) Had to make realloc_for_line_map() global because when repeatedly
Oliver> calling linemap_init(), the member "reallocator" is NULL and I
Oliver> couldn't find a way of setting it to the proper value again.

I suggest a new public function in toplev.c that creates a line table.
I did something similar to this on the incremental branch; except
there I did not have to make it public, since server re-initialization
is done in toplev.c.

I am curious how you handle locations on shared bits of the AST.
I needed some disturbing hacks to make this work well for C.

Oliver> * cgraph.{h,c}, cgraphunit.c:
Oliver> Added many missing reinitializations that are necessary when calling
Oliver> compile_file() repeatedly.

I did this on the branch too.  I needed similar changes in dwarf2out
that you might be interested in.  I also changed things so that debug
output starts "late", but offhand I don't recall whether that is only
relevant to the server case.

Tom

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

On 2009-04-27, at 12:19, Tom Tromey wrote:
>
> This sounds like it has a lot of overlap with the incremental compiler
> project I was working on.  I think you could probably rearrange things
> to use the compile server infrastructure from the branch without
> trouble.
> 
> There are some differences in invocation.  I designed the incremental
> compiler so that no changes to user Makefiles would be needed; I don't
> know whether that is a consideration with Ada.

The idea is that gnatmake determines the total set of files to be
recompiled and then makes a single call to gcc with all those files.
The gain with that scheme is that the compiler proper can assume
the compilation of those files to be "atomic" in the sense that the
trees are guaranteed to be consistent within one compile job.
(Does the compile server also work this way?)

One other big plus with Ada is the lack of a preprocessor - no headaches
with different #define and #ifdef settings among files of a compile job.

> I think the compile server infrastructure is in decent shape, though I
> haven't worked on it in a while, so it is a bit hard to remember
> exactly.  The difficult bit is converting a front end to be
> server-ready; but from your note it sounds like you have already done
> that.
> 
> Oliver> It's not totally functioning yet:
> Oliver> 1) invocation of "as" (assembler) is only done for the first file
> 
> This is solved on the branch.  There, the server starts the assembler
> from compile_file.

That sounds interesting. I'll take a look.

> 
> Oliver> 2) gcc is passing too many switches to the compiler, for example
> Oliver> "-o" (output file) should be suppressed when in multi-source mode.
> 
> With the compile server you don't have to worry about this kind of
> thing.  I changed the options processing code to be re-initializable;
> each compiler job in the server runs with its own set of options.

Hmm. I'll have to look at that.

> Oliver> * toplev.c:
> Oliver> 1) Had to make realloc_for_line_map() global because when repeatedly
> Oliver> calling linemap_init(), the member "reallocator" is NULL and I
> Oliver> couldn't find a way of setting it to the proper value again.
> 
> I suggest a new public function in toplev.c that creates a line table.
> I did something similar to this on the incremental branch; except
> there I did not have to make it public, since server re-initialization
> is done in toplev.c.
> 
> I am curious how you handle locations on shared bits of the AST.
> I needed some disturbing hacks to make this work well for C.

I'm not sure what you mean; could you explain?

Oliver

Re: [gnat] reuse of ASTs already constructed

[Tom Tromey]

>>>>> "Oliver" == Oliver Kellogg <oliver.kellogg@t-online.de> writes:

Tom> There are some differences in invocation.  I designed the incremental
Tom> compiler so that no changes to user Makefiles would be needed; I don't
Tom> know whether that is a consideration with Ada.

Oliver> The idea is that gnatmake determines the total set of files to be
Oliver> recompiled and then makes a single call to gcc with all those files.
Oliver> The gain with that scheme is that the compiler proper can assume
Oliver> the compilation of those files to be "atomic" in the sense that the
Oliver> trees are guaranteed to be consistent within one compile job.
Oliver> (Does the compile server also work this way?)

Yes, more or less.  The idea behind the incremental compiler is to
share declarations across compile jobs.  The details of this sharing
are left to the front ends.

There is no concept of atomicity here; the front end has to ensure
that whatever it is doing makes sense.  I don't know Ada so I can't
really comment on what this would look like in gnat.

I suppose that gnatmake could start and stop the server, thus
effectively providing the same guarantee as in your scenario.  That
is, it would "gcc --server" at the start, invoke gcc once for each
input, and then and "gcc --kill-server" at the end.

Tom> I am curious how you handle locations on shared bits of the AST.
Tom> I needed some disturbing hacks to make this work well for C.

Oliver> I'm not sure what you mean; could you explain?

In the server, the global line map is reset for each compile job (and
it sounds like this is true for your patch as well).  So, suppose a
compilation reuses a declaration that was parsed during an earlier
job.  The location_t values in this declaration will no longer be
valid; if the compiler emits a diagnostic using this declaration, then
the location will be incorrect.

In the C compiler I worked around this by resetting locations on
shared decls.  The details of this are tied to how the C compiler
implements sharing.

I was curious to know how you handled this situation.  If you re-lower
AST bits to trees for each job, or something like that, then it
probably is not as big a problem for you.

Tom

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

On 2009-04-27, at 14:28, Tom Tromey wrote:
>
> Tom> I am curious how you handle locations on shared bits of the AST.
> Tom> I needed some disturbing hacks to make this work well for C.
> 
> Oliver> I'm not sure what you mean; could you explain?
> 
> In the server, the global line map is reset for each compile job (and
> it sounds like this is true for your patch as well).  So, suppose a
> compilation reuses a declaration that was parsed during an earlier
> job.  The location_t values in this declaration will no longer be
> valid; if the compiler emits a diagnostic using this declaration, then
> the location will be incorrect.
> 
> In the C compiler I worked around this by resetting locations on
> shared decls.  The details of this are tied to how the C compiler
> implements sharing.
> 
> I was curious to know how you handled this situation.  If you re-lower
> AST bits to trees for each job, or something like that, then it
> probably is not as big a problem for you.
> 

Ah, yes. There's a GNAT to GNU tree transformation pass (gigi) which
is called afresh each time thus I haven't needed to worry about that
(AFAIK the line info is filled in from the GNAT tree.)
IMHO it would be much more demanding to try to optimize away (parts of)
this pass. Certainly beyond me :)

Oliver

Re: [gnat] reuse of ASTs already constructed

[Robert Dewar]

> Ah, could you point me to those discussions?
> Are they here on this list?
> (I found searching for "gnat" difficult because there are
> too many hits on "gnats".)

No I am talking about internal discussions at ACT, dating back several years. We never
pursued them because as I said, the gains seemed small compared to the effort, and
other things have had higher priority.

> Would you consider inclusion into GNAT if someone implemented this?

Sure, but I would suggest that what is needed first is a very careful design which
gets discussed before even one line of code is written (that's the procedure we always
follow internally). The reason that this is particularly important is that there are
lots of delicate interactions here.

Re: [gnat] reuse of ASTs already constructed

[Robert Dewar]

> I think it's a terrible waste that gnat1 can only process
> one single file.
> If gnat1 could be invoked with (for example) all the files
> that gnatmake finds to need recompilation, that could
> give tremendous compilation time savings - by keeping the
> syntax trees of units compiled for reuse by further units.
> 
> Many Ada projects have large package specs that tend
> to be pervasively used. (The GNAT compiler is itself
> an example of such a design.) These applications
> could benefit a lot from in-memory AST reuse.
> 
> What do you think about this?

It's definitely techncially feasible (probably with some compilation server
approach, and we have discussed more than once what might be involved), but
right now it is not on our radar screen, since front end compilation time (which
is what this would save) is not a critical issue for us in the total picture.

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

Robert Dewar wrote:
>
> It's definitely techncially feasible (probably with some compilation server
> approach, and we have discussed more than once what might be involved),

Ah, could you point me to those discussions?
Are they here on this list?
(I found searching for "gnat" difficult because there are
too many hits on "gnats".)

>  but
> right now it is not on our radar screen, since front end compilation time
>    is not a critical issue for us in the total picture.

Would you consider inclusion into GNAT if someone implemented this?

Re: [gnat] reuse of ASTs already constructed

[Geert Bosch]

On Tuesday, Mar 4, 2003, at 15:50 America/New_York, Oliver Kellogg 
wrote:

> Would you consider inclusion into GNAT if someone implemented this?

Yes, in general, if the benefits (in measurable speedup) outweigh
the costs (speed degradation in single-unit case, maintainability
of the compiler, ...) of an optimization, it will be considered
for inclusion. Best would be to first post a design overview,
before doing a lot of work in order to prevent spending time
on implementing something that may turn out to have fundamental
problems.

The discussions Robert mentioned were before Ada was part
of GCC, and I am not sure I have anything in writing on
these ideas.

   -Geert

[gnat] reuse of ASTs already constructed

[Oliver Kellogg]

Hello developers of GNAT,

I think it's a terrible waste that gnat1 can only process
one single file.
If gnat1 could be invoked with (for example) all the files
that gnatmake finds to need recompilation, that could
give tremendous compilation time savings - by keeping the
syntax trees of units compiled for reuse by further units.

Many Ada projects have large package specs that tend
to be pervasively used. (The GNAT compiler is itself
an example of such a design.) These applications
could benefit a lot from in-memory AST reuse.

What do you think about this?

Thanks,

Oliver Kellogg

Re: [gnat] reuse of ASTs already constructed

[Geert Bosch]

On Tuesday, Mar 4, 2003, at 11:57 America/New_York, Oliver Kellogg 
wrote:

> I think it's a terrible waste that gnat1 can only process
> one single file.
> If gnat1 could be invoked with (for example) all the files
> that gnatmake finds to need recompilation, that could
> give tremendous compilation time savings - by keeping the
> syntax trees of units compiled for reuse by further units.
>
> Many Ada projects have large package specs that tend
> to be pervasively used. (The GNAT compiler is itself
> an example of such a design.) These applications
> could benefit a lot from in-memory AST reuse.
>
> What do you think about this?

This is not a new idea, and indeed there have been
thoughts for implementing some sort of scheme like
this. However, even though this may seem relatively
simple at first, the problems are in the details
and this is a bigger project than you might think.

Also, in most typical development schemes, only a
few units are recompiled at a time, and the back end
takes significantly more time than the front end anyway.
This means potential speedup is not "tremendous".

For example, for the run time library, about 30% of the
time is spent in the front end. The time building AST's
is a fraction of that, say 70% at most. Even in the
hypothetical case we could speed this up by a factor of
three, this still would gain us just 14% in overall speed
of the compiler. Significant sure, but not tremendous.

Re: [gnat] reuse of ASTs already constructed

[Oliver Kellogg]

Geert Bosch wrote:
>
> Also, in most typical development schemes, only a
> few units are recompiled at a time, and the back end
> takes significantly more time than the front end anyway.

I have seen several Ada systems with some package specs
on the order of thousands of LOC.

Changing just one line in such a spec results in
massive recompiles - although the only thing changed
is perhaps just a minute detail in the AST of that
large package spec, for example an enum literal added.

Here, the time spent in the front end is quite significant -
if you have to recompile dozens of (not so large) bodies that
depend on the large spec.