[patch 0/3] Header file reduction.

[patch 0/3] Header file reduction.

[Andrew MacLeod]

OK, newly regenerated patches to remove header files from the latest 
version of the tools.

The patches are generated by a pair of tools.
* gcc-order-includes goes through the headers and canonically reorders 
some of our more common/troublesome headers and removes any duplicates.  
This includes headers which are included by other headers. (ie, 
obstack.h can be removed as a duplicate if bitmap.h is included already.)
* remove-includes is the tool which tries to remove each non-conditional 
header file and does the real work.

I'll have a patch shortly to add these and some other useful tools to a 
header-tools directory in contrib.

There are 3 patches which follow:  backend.a files, fe files, and config 
files.. Effecting 547 files total

These were generated from a  trunk snapshot taken about 2 weeks ago. The 
tools ran, and once I finished some minor tweaking, I reapplied them to 
a 9/28 branch. Anything which didn't apply cleanly due to intervening 
changes, was simply re-reduced.  I then reapplied them to a snapshot 
from this morning for these patches.

The tool also monitor what macros are defined and conditionally 
consumed, and wont remove a header which still compiles if it may define 
a macro which is used in conditional compilation (tm.h is frequently 
affected by this). The tool spits out messages like these:

* Passed host and target builds, but must keep target.h because it 
provides ASM_OUTPUT_DEF Possibly required by ipa-icf.c
* Passed host and target builds, but must keep insn-attr.h because it 
provides DELAY_SLOTS Possibly required by toplev.c

note my host arch doesn't define DELAY_SLOTS in insn-attr-common.h 
(included by insn-attr.h) , but some other target archs do, and this is 
caught this during some of the target builds. (toplev.c has the 
following lines:)

    #ifndef DELAY_SLOTS
       if (flag_delayed_branch)
         warning (0, "this target machine does not have delayed branches");
    #endif

So as far as I can tell I'm catching all those conditional compilation 
cases.  The only ones I might have missed would be macros which some 
host build defines on the command line and which doesn't show up in a 
config-list.mk target build.. I guess.

Everything bootstraps on x86_64-pc-linux-gnu and 
powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still 
build. Regressions tests also came up clean.

OK for trunk?

I will make minor tweaks as needed when applying to trunk for 
check-in..  I know that the new requirement for cgraph.h in builtins.c 
is no longer required in trunk. 
(https://gcc.gnu.org/ml/gcc-patches/2015-10/msg00102.html)   Going 
forward, we'll know when we are adding new dependencies, as this thread 
so timely shows :-).

Andrew

PS.  Then keep an eye for anything funny. I'm not expecting much, but if 
a file causes an issue, simple reverting the change for that one file 
should be sufficient until we figure out why it is an issue,.  No change 
in any of these files is dependent on any other..  it simple include 
reduction, and there should be no functional change in code.

[patch 1/3] Header file reduction - backend files.

[Andrew MacLeod]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

these are all in the main gcc directory.  297 files total.

Everything bootstraps on x86_64-pc-linux-gnu and 
powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still 
build. Regressions tests also came up clean.

OK for trunk?

Andrew

[-- Attachment #2: backend.patch.bz2 --]
[-- Type: application/x-bzip, Size: 23694 bytes --]

Re: [patch 1/3] Header file reduction - backend files.

[Jeff Law]

On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
> these are all in the main gcc directory.  297 files total.
>
> Everything bootstraps on x86_64-pc-linux-gnu and
> powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still
> build. Regressions tests also came up clean.
>
> OK for trunk?
So as I look at this and make various spot checks, what really stands 
out is how often something like alias.h gets included, often in places 
that have absolutely no business/need to be looking at that file. 
Cut-n-paste at its worst.  It happens to many others, but alias.h seems 
to have gotten its grubby self into just about everywhere for reasons 
unkonwn.

I find myself also wondering if a two step approach would make this 
easier.  Step #1 being ordering the headers, step #2 being removal of 
the duplicates.  As you note, the downside is two checkins that would 
affect most files in the tree.  I guess I'll keep slogging through the 
patch as is...

jeff

Re: [patch 1/3] Header file reduction - backend files.

[Andrew MacLeod]

On 10/07/2015 06:02 PM, Jeff Law wrote:
> On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
>> these are all in the main gcc directory. 297 files total.
>>
>> Everything bootstraps on x86_64-pc-linux-gnu and
>> powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still
>> build. Regressions tests also came up clean.
>>
>> OK for trunk?
> So as I look at this and make various spot checks, what really stands 
> out is how often something like alias.h gets included, often in places 
> that have absolutely no business/need to be looking at that file. 
> Cut-n-paste at its worst.  It happens to many others, but alias.h 
> seems to have gotten its grubby self into just about everywhere for 
> reasons unkonwn.
>
> I find myself also wondering if a two step approach would make this 
> easier.  Step #1 being ordering the headers, step #2 being removal of 
> the duplicates.  As you note, the downside is two checkins that would 
> affect most files in the tree.  I guess I'll keep slogging through the 
> patch as is...
>
> jeff

No problem...   I can generate the header reordering patch for you to 
look at.  gotta run right now, but either later tonight or first thingin 
the morning.

alias.h is particularly bad because there were some headers which had 
stupid dependencies. I broke those dependencies a couple of months ago, 
now it doesn't need to be everywhere anymore.  I had also noticed it was 
the one which was removed the most frequently now :-)

Andrew

[patch] header file re-ordering.

[Andrew MacLeod]

[-- Attachment #1: Type: text/plain, Size: 2194 bytes --]

On 10/07/2015 06:02 PM, Jeff Law wrote:
> On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
>> these are all in the main gcc directory. 297 files total.
>>
>> Everything bootstraps on x86_64-pc-linux-gnu and
>> powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still
>> build. Regressions tests also came up clean.
>>
>> OK for trunk?
> So as I look at this and make various spot checks, what really stands 
> out is how often something like alias.h gets included, often in places 
> that have absolutely no business/need to be looking at that file. 
> Cut-n-paste at its worst.  It happens to many others, but alias.h 
> seems to have gotten its grubby self into just about everywhere for 
> reasons unkonwn.
>
> I find myself also wondering if a two step approach would make this 
> easier.  Step #1 being ordering the headers, step #2 being removal of 
> the duplicates.  As you note, the downside is two checkins that would 
> affect most files in the tree.  I guess I'll keep slogging through the 
> patch as is...
>
> jeff
Heres the patch for reordered headers.  Building as we speak.  Hard to 
fully verify since Ada doesn't seem to bootstrap on trunk at the moment:

+===========================GNAT BUG DETECTED==============================+
| 6.0.0 20151008 (experimental) (x86_64-pc-linux-gnu) GCC error:           |
| in gen_lowpart_common, at emit-rtl.c:1399                                |
| Error detected around s-regpat.adb:1029:22                               |

<...>
raised TYPES.UNRECOVERABLE_ERROR : comperr.adb:423
../gcc-interface/Makefile:311: recipe for target 's-regpat.o' failed


However, the tool has been run, and I've made the minor adjustments 
required to the source files to make it work.  (ie, a few multi-line 
comments and the fact that mul-tables.c is generated on the tile* targets.

So this is what it should look like.  I used -cp.    Other languages are 
bootstrapping, and I have yet to build all the targets... that'll just 
take a day.   Be nice if ada worked tho.

I can run the reduction tool over the weekend (its a long weekend here 
:-) on this if you want...  the other patch is a couple of weeks out of 
date anyway now.

Andrew


[-- Attachment #2: backend-order.patch.bz2 --]
[-- Type: application/x-bzip, Size: 21703 bytes --]

[-- Attachment #3: FE-order.patch.bz2 --]
[-- Type: application/x-bzip, Size: 10166 bytes --]

[-- Attachment #4: config-order.patch.bz2 --]
[-- Type: application/x-bzip, Size: 8283 bytes --]

Re: [patch] header file re-ordering.

[Jeff Law]

On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
> On 10/07/2015 06:02 PM, Jeff Law wrote:
>> On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
>>> these are all in the main gcc directory. 297 files total.
>>>
>>> Everything bootstraps on x86_64-pc-linux-gnu and
>>> powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still
>>> build. Regressions tests also came up clean.
>>>
>>> OK for trunk?
>> So as I look at this and make various spot checks, what really stands
>> out is how often something like alias.h gets included, often in places
>> that have absolutely no business/need to be looking at that file.
>> Cut-n-paste at its worst.  It happens to many others, but alias.h
>> seems to have gotten its grubby self into just about everywhere for
>> reasons unkonwn.
>>
>> I find myself also wondering if a two step approach would make this
>> easier.  Step #1 being ordering the headers, step #2 being removal of
>> the duplicates.  As you note, the downside is two checkins that would
>> affect most files in the tree.  I guess I'll keep slogging through the
>> patch as is...
>>
>> jeff
> Heres the patch for reordered headers.  Building as we speak.  Hard to
> fully verify since Ada doesn't seem to bootstrap on trunk at the moment:
Saw in IRC it was Jan's patch that broke Ada bootstrap.  So you might 
consider reverting that bit locally to restore bootstrapping for Ada.


>
> However, the tool has been run, and I've made the minor adjustments
> required to the source files to make it work.  (ie, a few multi-line
> comments and the fact that mul-tables.c is generated on the tile* targets.
>
> So this is what it should look like.  I used -cp.    Other languages are
> bootstrapping, and I have yet to build all the targets... that'll just
> take a day.   Be nice if ada worked tho.
OK.  I'll take a look at this version and I think running the reducer 
over the weekend sounds good.

Jeff

[BUILDROBOT] Bootstrap broken in Ada (was: [patch] header file re-ordering.)

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

On Thu, 2015-10-08 09:37:03 -0400, Andrew MacLeod <amacleod@redhat.com> wrote:
[...]
> Heres the patch for reordered headers.  Building as we speak.  Hard to fully
> verify since Ada doesn't seem to bootstrap on trunk at the moment:
> 
> +===========================GNAT BUG DETECTED==============================+
> | 6.0.0 20151008 (experimental) (x86_64-pc-linux-gnu) GCC error:           |
> | in gen_lowpart_common, at emit-rtl.c:1399                                |
> | Error detected around s-regpat.adb:1029:22                               |
> 
> <...>
> raised TYPES.UNRECOVERABLE_ERROR : comperr.adb:423
> ../gcc-interface/Makefile:311: recipe for target 's-regpat.o' failed

Bulid log (native build on gcc76.fsffrance.org) can be found at
http://toolchain.lug-owl.de/buildbot/deliver_artifact.php?mode=view&id=4292383
for build
http://toolchain.lug-owl.de/buildbot/show_build_details.php?id=472655 .

So it's probably one of these two commits:

	r228586 = 54ac7405ce75c141dae33532d491d5793fb583e3
		Jan Hubicka  <hubicka@ucw.cz>
		Do not use TYPE_CANONICAL in useless_type_conversion

	r228585 = 5b4ada2a11ab19842d77296fc4b75971ddb07434
		Jeff Law <law@redhat.com>
		[PATCH] Improve DOM's optimization of control statements

Haven't looked at the patches or what they're doing, but maybe you two
instantly recognize the issue?

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
  Signature of:                           Wenn ich wach bin, träume ich.
  the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [BUILDROBOT] Bootstrap broken in Ada

[Jeff Law]

On 10/11/2015 02:58 PM, Jan-Benedict Glaw wrote:
> On Thu, 2015-10-08 09:37:03 -0400, Andrew MacLeod <amacleod@redhat.com> wrote:
> [...]
>> Heres the patch for reordered headers.  Building as we speak.  Hard to fully
>> verify since Ada doesn't seem to bootstrap on trunk at the moment:
>>
>> +===========================GNAT BUG DETECTED==============================+
>> | 6.0.0 20151008 (experimental) (x86_64-pc-linux-gnu) GCC error:           |
>> | in gen_lowpart_common, at emit-rtl.c:1399                                |
>> | Error detected around s-regpat.adb:1029:22                               |
>>
>> <...>
>> raised TYPES.UNRECOVERABLE_ERROR : comperr.adb:423
>> ../gcc-interface/Makefile:311: recipe for target 's-regpat.o' failed
>
> Bulid log (native build on gcc76.fsffrance.org) can be found at
> http://toolchain.lug-owl.de/buildbot/deliver_artifact.php?mode=view&id=4292383
> for build
> http://toolchain.lug-owl.de/buildbot/show_build_details.php?id=472655 .
>
> So it's probably one of these two commits:
>
> 	r228586 = 54ac7405ce75c141dae33532d491d5793fb583e3
> 		Jan Hubicka  <hubicka@ucw.cz>
> 		Do not use TYPE_CANONICAL in useless_type_conversion
It's one of Jan's patches.  I've got them reverted locally and Ada 
builds fine.

jeff

Re: [BUILDROBOT] Bootstrap broken in Ada

[Jan Hubicka]

> On 10/11/2015 02:58 PM, Jan-Benedict Glaw wrote:
> >On Thu, 2015-10-08 09:37:03 -0400, Andrew MacLeod <amacleod@redhat.com> wrote:
> >[...]
> >>Heres the patch for reordered headers.  Building as we speak.  Hard to fully
> >>verify since Ada doesn't seem to bootstrap on trunk at the moment:
> >>
> >>+===========================GNAT BUG DETECTED==============================+
> >>| 6.0.0 20151008 (experimental) (x86_64-pc-linux-gnu) GCC error:           |
> >>| in gen_lowpart_common, at emit-rtl.c:1399                                |
> >>| Error detected around s-regpat.adb:1029:22                               |
> >>
> >><...>
> >>raised TYPES.UNRECOVERABLE_ERROR : comperr.adb:423
> >>../gcc-interface/Makefile:311: recipe for target 's-regpat.o' failed
> >
> >Bulid log (native build on gcc76.fsffrance.org) can be found at
> >http://toolchain.lug-owl.de/buildbot/deliver_artifact.php?mode=view&id=4292383
> >for build
> >http://toolchain.lug-owl.de/buildbot/show_build_details.php?id=472655 .
> >
> >So it's probably one of these two commits:
> >
> >	r228586 = 54ac7405ce75c141dae33532d491d5793fb583e3
> >		Jan Hubicka  <hubicka@ucw.cz>
> >		Do not use TYPE_CANONICAL in useless_type_conversion
> It's one of Jan's patches.  I've got them reverted locally and Ada
> builds fine.

A proposed patch is https://gcc.gnu.org/ml/gcc-patches/2015-10/msg01011.html

Honza
> 
> jeff

Re: [patch] header file re-ordering.

[Jeff Law]

On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
> On 10/07/2015 06:02 PM, Jeff Law wrote:
>> On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
>>> these are all in the main gcc directory. 297 files total.
>>>
>>> Everything bootstraps on x86_64-pc-linux-gnu and
>>> powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still
>>> build. Regressions tests also came up clean.
>>>
>>> OK for trunk?
>> So as I look at this and make various spot checks, what really stands
>> out is how often something like alias.h gets included, often in places
>> that have absolutely no business/need to be looking at that file.
>> Cut-n-paste at its worst.  It happens to many others, but alias.h
>> seems to have gotten its grubby self into just about everywhere for
>> reasons unkonwn.
>>
>> I find myself also wondering if a two step approach would make this
>> easier.  Step #1 being ordering the headers, step #2 being removal of
>> the duplicates.  As you note, the downside is two checkins that would
>> affect most files in the tree.  I guess I'll keep slogging through the
>> patch as is...
>>
>> jeff
> Heres the patch for reordered headers.  Building as we speak.  Hard to
> fully verify since Ada doesn't seem to bootstrap on trunk at the moment:
>
> +===========================GNAT BUG
> DETECTED==============================+
> | 6.0.0 20151008 (experimental) (x86_64-pc-linux-gnu) GCC
> error:           |
> | in gen_lowpart_common, at
> emit-rtl.c:1399                                |
> | Error detected around
> s-regpat.adb:1029:22                               |
>
> <...>
> raised TYPES.UNRECOVERABLE_ERROR : comperr.adb:423
> ../gcc-interface/Makefile:311: recipe for target 's-regpat.o' failed
>
>
> However, the tool has been run, and I've made the minor adjustments
> required to the source files to make it work.  (ie, a few multi-line
> comments and the fact that mul-tables.c is generated on the tile* targets.
>
> So this is what it should look like.  I used -cp.    Other languages are
> bootstrapping, and I have yet to build all the targets... that'll just
> take a day.   Be nice if ada worked tho.
>
> I can run the reduction tool over the weekend (its a long weekend here
> :-) on this if you want...  the other patch is a couple of weeks out of
> date anyway now.
I find myself looking at the objc stuff and wondering if it was built. 
For example objc-act.c calls functions prototyped in fold-const.h, but 
that header is no longer included after your patch.

Similarly in objcp we remove tree.h from objcp-decl.c, but it uses TREE 
macros and I don't immediately see where those macros would be coming 
from if tree.h is no longer included.

In general, I'm worried about the objc/objcp stuff.  That in turn makes 
me wonder about the other stuff in a more general sense.  Regardless, I 
think I can take a pretty good stab at the config/ changes.


A pattern that seems to play out a lot in the target files is they liked 
to include insn-config.h, insn-codes.h, & timevar.h.  I can see how 
those typically won't be needed.  The first two are amazingly common.  A 
comment in the nds32 port indicates that insn-config.h may have been 
needed by recog.h in the past.  nds32 actually included insn-config 
twice :-)


Interestingly enough m32r, mcore & pdp11 still need insn-config....

The strangest thing I saw was rs6000 dropping an include of emit-rtl.h. 
  But presumably various powerpc targets were built, so I guess it's 
really not needed.

I'm slightly concerned about the darwin, windows and solaris bits.  The 
former primarily because Darwin has been a general source of pain, and 
in the others because I'm not sure the cross testing will exercise that 
code terribly much.

I'll go ahead and approve all the config/ bits.  Please be on the 
lookout for any fallout.

I'll try and get into more of the other patches tomorrow.

jeff

Re: [patch] header file re-ordering.

[Andrew MacLeod]

On 10/12/2015 04:04 AM, Jeff Law wrote:
>
>> <...>
>> raised TYPES.UNRECOVERABLE_ERROR : comperr.adb:423
>> ../gcc-interface/Makefile:311: recipe for target 's-regpat.o' failed
>>
>>
>> However, the tool has been run, and I've made the minor adjustments
>> required to the source files to make it work.  (ie, a few multi-line
>> comments and the fact that mul-tables.c is generated on the tile* 
>> targets.
>>
>> So this is what it should look like.  I used -cp.    Other languages are
>> bootstrapping, and I have yet to build all the targets... that'll just
>> take a day.   Be nice if ada worked tho.
>>
>> I can run the reduction tool over the weekend (its a long weekend here
>> :-) on this if you want...  the other patch is a couple of weeks out of
>> date anyway now.
> I find myself looking at the objc stuff and wondering if it was built. 
> For example objc-act.c calls functions prototyped in fold-const.h, but 
> that header is no longer included after your patch.

wait, what?   I don't see any differences to objc-act.c in the 
reordering patches....

Oh, you must be looking at the original combined patch?

fold-const.h is indirectly included by cp-tree.h, which gets it from 
including c-common.h.   some of the output from show-headers on 
objc-act.c  (indentation represents levels of including.  The number in 
parenthesis is the number of times that include has been seen so far in 
the files include list.   As you can see, we include ansidecl.h a lot 
:-)  Most of the time there isn't much we can do about those sorts of 
things. :

cp-tree.h
     tm.h  (2)
     hard-reg-set.h
     function.h  (1)
     c-common.h
       splay-tree.h
         ansidecl.h  (4)
       cpplib.h
         symtab.h  (2)
         line-map.h  (2)
       alias.h
       tree.h  (2)
       fold-const.h
       diagnostic-core.h  (1)
         bversion.h

I guess It could be a useful addition to show-headers to specify a 
header file you are looking for and show you where it comes from if its 
included...

I any case, there is some indirection here because none of the front end 
files were flattened that much

incidentally, you may notice this is the second time tree.h is 
included.  The first occurrence of tree.h is included directly by 
objc-act.c, but it needs to be left because something between that and 
cp-tree.h needs tree.h to compile.    This sort of thing is resolved by 
using the re-order tool, but I did not run that tool on most of the objc 
and objcp files as they have some complex conditionals in their include 
list:
#include "tree.h"
#include "stringpool.h"
#include "stor-layout.h"
#include "attribs.h"

#ifdef OBJCPLUS
#include "cp/cp-tree.h"
#else
#include "c/c-tree.h"
#include "c/c-lang.h"
#endif

#include "c-family/c-objc.h"
#include "langhooks.h"

Its beyond the scope of the reorder tool to deal with re-positioning 
this automatically... and happens so rarely I didn't even look into it.
So they are not optimal as far as ordering goes.

>
> Similarly in objcp we remove tree.h from objcp-decl.c, but it uses 
> TREE macros and I don't immediately see where those macros would be 
> coming from if tree.h is no longer included.
>
Again, thanks to no flattening of the front end files :-)  It also comes 
from cp-tree.h.  The objcp source files don't specify the full patch of 
cp/cp-tree.h like objc does, so the simplistic show-headers tool doesn't 
know where to look for cp-tree.h to show you what it included like in 
the above example.    Maybe I'll tweak the tool to look in common header 
directories.

> In general, I'm worried about the objc/objcp stuff.  That in turn 
> makes me wonder about the other stuff in a more general sense. 
> Regardless, I think I can take a pretty good stab at the config/ changes.
>

So you can not worry about that.  It builds fine.
>
> A pattern that seems to play out a lot in the target files is they 
> liked to include insn-config.h, insn-codes.h, & timevar.h.  I can see 
> how those typically won't be needed.  The first two are amazingly 
> common.  A comment in the nds32 port indicates that ii may have been 
> needed by recog.h in the past.  nds32 actually included insn-config 
> twice :-)
>
>
> Interestingly enough m32r, mcore & pdp11 still need insn-config....

most ports get insn-config.h from optabs.h:
   optabs.h
     optabs-query.h
       insn-opinit.h  (1)
     optabs-libfuncs.h
       insn-opinit.h  (2)
     insn-config.h

I think those ports that still include it do not include optabs.h


>
> The strangest thing I saw was rs6000 dropping an include of 
> emit-rtl.h.  But presumably various powerpc targets were built, so I 
> guess it's really not needed.

It gets emit-rtl.h from ira.h:
   regs.h
   ira.h
     emit-rtl.h
   recog.h
     insn-codes.h  (2)


>
> I'm slightly concerned about the darwin, windows and solaris bits.  
> The former primarily because Darwin has been a general source of pain, 
> and in the others because I'm not sure the cross testing will exercise 
> that code terribly much.
>
Its easy enough to NOT do this for any of those files if were too 
worried about them.   Its also easy to revert a single file if it 
appears to be an issue.     Thats why I wanted to run as many of these 
on the compile farm natively as I could... but alas, powerPC was the 
only thing the farm really offered me.


> I'll go ahead and approve all the config/ bits.  Please be on the 
> lookout for any fallout.

even darwin, windows and solaris? :-)

Im going to tweak the show-headers tool to look in a few common places, 
and look for requested headers, and then repost the tools change with 
most of the documentation changes Bernd pointed out.  I didn't do  lot 
to the code itself, but I did comment a few more things.

Perhaps using the tool when you have the above questions would show its 
usefulness :-)    Its particularly useful when combined with 
included-by, so you can see how many files include specific ones, and it 
does point out some silly things, but we cant fix all of them without 
flattening EVERYTHING... I tried to just do the main parts in the 
backend we were likely to care about.    Of course, the tool is useless 
unless the patch is applied :-P   However, I can continue to answer 
questions easily enough :-)

Andrew

Re: [patch] header file re-ordering.

[Jeff Law]

On 10/14/2015 08:05 AM, Andrew MacLeod wrote:
> On 10/12/2015 04:04 AM, Jeff Law wrote:
>
> Oh, you must be looking at the original combined patch?
Possibly :-)

>
> fold-const.h is indirectly included by cp-tree.h, which gets it from
> including c-common.h.   some of the output from show-headers on
> objc-act.c  (indentation represents levels of including.  The number in
> parenthesis is the number of times that include has been seen so far in
> the files include list.   As you can see, we include ansidecl.h a lot
> :-)  Most of the time there isn't much we can do about those sorts of
> things. :
>
> cp-tree.h
>      tm.h  (2)
>      hard-reg-set.h
>      function.h  (1)
>      c-common.h
>        splay-tree.h
>          ansidecl.h  (4)
>        cpplib.h
>          symtab.h  (2)
>          line-map.h  (2)
>        alias.h
>        tree.h  (2)
>        fold-const.h
>        diagnostic-core.h  (1)
>          bversion.h
>
> I guess It could be a useful addition to show-headers to specify a
> header file you are looking for and show you where it comes from if its
> included...
Yea.  Though I think it's probably easy enough to get it from the 
current output.

>
> I any case, there is some indirection here because none of the front end
> files were flattened that much
And I think that's probably some source of the confusion on my part.  I 
thought we'd flattened the front-end .h files too.  So I didn't look 
deeply into the .h files to see if they were doing something undesirable 
behind my back.

>
> incidentally, you may notice this is the second time tree.h is
> included.  The first occurrence of tree.h is included directly by
> objc-act.c, but it needs to be left because something between that and
> cp-tree.h needs tree.h to compile.    This sort of thing is resolved by
> using the re-order tool, but I did not run that tool on most of the objc
> and objcp files as they have some complex conditionals in their include
> list:
> #include "tree.h"
> #include "stringpool.h"
> #include "stor-layout.h"
> #include "attribs.h"
>
> #ifdef OBJCPLUS
> #include "cp/cp-tree.h"
> #else
> #include "c/c-tree.h"
> #include "c/c-lang.h"
> #endif
>
> #include "c-family/c-objc.h"
> #include "langhooks.h"
>
> Its beyond the scope of the reorder tool to deal with re-positioning
> this automatically... and happens so rarely I didn't even look into it.
> So they are not optimal as far as ordering goes.
Understood.  This unholy sharing had me concerned as well.

> So you can not worry about that.  It builds fine.
OK.  I think the major source of confusion was the lack of flattening 
for the front-ends.  I'll go back to it with that in mind and probably 
start using the tools when I get a WTF moment.


>
>>
>> I'm slightly concerned about the darwin, windows and solaris bits. The
>> former primarily because Darwin has been a general source of pain, and
>> in the others because I'm not sure the cross testing will exercise
>> that code terribly much.
>>
> Its easy enough to NOT do this for any of those files if were too
> worried about them.   Its also easy to revert a single file if it
> appears to be an issue.     Thats why I wanted to run as many of these
> on the compile farm natively as I could... but alas, powerPC was the
> only thing the farm really offered me.
>
>
>> I'll go ahead and approve all the config/ bits.  Please be on the
>> lookout for any fallout.
>
> even darwin, windows and solaris? :-)
Yup.  The changes are straighforward enough that if there's fallout (and 
to some degree I expect minor fallout from native builds) it can be 
easily fixed.

Jeff

config header file reduction patch checked in.

[Andrew MacLeod]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

On 10/12/2015 04:04 AM, Jeff Law wrote:
> On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
>> On 10/07/2015 06:02 PM, Jeff Law wrote:
>
> I'm slightly concerned about the darwin, windows and solaris bits.  
> The former primarily because Darwin has been a general source of pain, 
> and in the others because I'm not sure the cross testing will exercise 
> that code terribly much.
>
> I'll go ahead and approve all the config/ bits.  Please be on the 
> lookout for any fallout.
>
> I'll try and get into more of the other patches tomorrow.
>
>

OK, I've checked in the config changes.  I rebuilt all the cross 
compilers for the 200+ targets, and they still build.. as well as 
bootstrapping on x86_64-pc-linux-gnu with no regressions.

So. If any one runs into a native build issue you can either add the 
required header back in, or back out the file for your port, and I'll 
look into why something happened.   The only thing I can imagine is 
files that have conditional compilation based on a  macro that is only 
ever defined on a native build command line or headers.  Its unlikely... 
but possible.


I've attached the latest version of the patch for the record.

Andrew



[-- Attachment #2: config2-final.patch.bz2 --]
[-- Type: application/x-bzip, Size: 8957 bytes --]

Re: config header file reduction patch checked in.

[Andrew MacLeod]

On 10/16/2015 03:49 PM, Andrew MacLeod wrote:
> On 10/12/2015 04:04 AM, Jeff Law wrote:
>> On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
>>> On 10/07/2015 06:02 PM, Jeff Law wrote:
>>
>> I'm slightly concerned about the darwin, windows and solaris bits.  
>> The former primarily because Darwin has been a general source of 
>> pain, and in the others because I'm not sure the cross testing will 
>> exercise that code terribly much.
>>
>> I'll go ahead and approve all the config/ bits.  Please be on the 
>> lookout for any fallout.
>>
>> I'll try and get into more of the other patches tomorrow.
>>
>>
>
> OK, I've checked in the config changes.  I rebuilt all the cross 
> compilers for the 200+ targets, and they still build.. as well as 
> bootstrapping on x86_64-pc-linux-gnu with no regressions.
>
> So. If any one runs into a native build issue you can either add the 
> required header back in, or back out the file for your port, and I'll 
> look into why something happened.   The only thing I can imagine is 
> files that have conditional compilation based on a macro that is only 
> ever defined on a native build command line or headers.  Its 
> unlikely... but possible.
>
btw, out of all the targets, the only one which didn't build before my 
patch was i686-interix3OPT-enable-obsolete...

so that one isn't my fault :-)

Andrew

Re: config header file reduction patch checked in.

[Iain Sandoe]

Hi Andrew,

On 16 Oct 2015, at 20:49, Andrew MacLeod wrote:

> On 10/12/2015 04:04 AM, Jeff Law wrote:
>> On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
>>> On 10/07/2015 06:02 PM, Jeff Law wrote:
>> 
>> I'm slightly concerned about the darwin, windows and solaris bits.  The former primarily because Darwin has been a general source of pain, and in the others because I'm not sure the cross testing will exercise that code terribly much.
>> 
>> I'll go ahead and approve all the config/ bits.  Please be on the lookout for any fallout.
>> 
>> I'll try and get into more of the other patches tomorrow.
>> 
>> 
> 
> OK, I've checked in the config changes.  I rebuilt all the cross compilers for the 200+ targets, and they still build.. as well as bootstrapping on x86_64-pc-linux-gnu with no regressions.
> 
> So. If any one runs into a native build issue you can either add the required header back in, or back out the file for your port, and I'll look into why something happened.   The only thing I can imagine is files that have conditional compilation based on a  macro that is only ever defined on a native build command line or headers.  Its unlikely... but possible.

I've applied the following to fix Darwin native bootstrap.
AFAICT (from reading the other thread on the re-ordering tools) putting the diagnostics header at the end of the list is the right thing to do.

FWIW, 
a) of course, Darwin exercises ObjC/ObjC++ in *both* NeXT and GNU mode - so those are pretty well covered by this too.

b) darwin folks will usually do their best to test any patch that you think is specifically risky - but you need to ask, because we have (very) limited resources in time and hardware ;-) ...

thanks for tidying things up!
(I, for one, think that improving the separation of things is worth a small amount of pain along the way).

cheers,
Iain

gcc/

+2015-10-18  Iain Sandoe  <iain@codesourcery.com>
+
+	* config/darwin-driver.h: Adjust includes to add diagnostic-core.
+
 2015-10-16  Trevor Saunders  <tbsaunde+gcc@tbsaunde.org>
 
 	* lra-constraints.c (add_next_usage_insn): Change argument type
Index: gcc/config/darwin-driver.c
===================================================================
--- gcc/config/darwin-driver.c	(revision 228938)
+++ gcc/config/darwin-driver.c	(working copy)
@@ -23,6 +23,7 @@
 #include "coretypes.h"
 #include "tm.h"
 #include "opts.h"
+#include "diagnostic-core.h"
 
 #ifndef CROSS_DIRECTORY_STRUCTURE
 #include <sys/sysctl.h>

Re: config header file reduction patch checked in.

[Andrew MacLeod]

On 10/18/2015 05:31 AM, Iain Sandoe wrote:
> Hi Andrew,
>
> On 16 Oct 2015, at 20:49, Andrew MacLeod wrote:
>
>> On 10/12/2015 04:04 AM, Jeff Law wrote:
>>> On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
>>>> On 10/07/2015 06:02 PM, Jeff Law wrote:
>>> I'm slightly concerned about the darwin, windows and solaris bits.  The former primarily because Darwin has been a general source of pain, and in the others because I'm not sure the cross testing will exercise that code terribly much.
>>>
>>> I'll go ahead and approve all the config/ bits.  Please be on the lookout for any fallout.
>>>
>>> I'll try and get into more of the other patches tomorrow.
>>>
>>>
>> OK, I've checked in the config changes.  I rebuilt all the cross compilers for the 200+ targets, and they still build.. as well as bootstrapping on x86_64-pc-linux-gnu with no regressions.
>>
>> So. If any one runs into a native build issue you can either add the required header back in, or back out the file for your port, and I'll look into why something happened.   The only thing I can imagine is files that have conditional compilation based on a  macro that is only ever defined on a native build command line or headers.  Its unlikely... but possible.
> I've applied the following to fix Darwin native bootstrap.
> AFAICT (from reading the other thread on the re-ordering tools) putting the diagnostics header at the end of the list is the right thing to do.
>
> FWIW,
> a) of course, Darwin exercises ObjC/ObjC++ in *both* NeXT and GNU mode - so those are pretty well covered by this too.
>
> b) darwin folks will usually do their best to test any patch that you think is specifically risky - but you need to ask, because we have (very) limited resources in time and hardware ;-) ...
>
> thanks for tidying things up!
> (I, for one, think that improving the separation of things is worth a small amount of pain along the way).
>
> cheers,
> Iain
>
> gcc/
>
> +2015-10-18  Iain Sandoe  <iain@codesourcery.com>
> +
> +	* config/darwin-driver.h: Adjust includes to add diagnostic-core.
> +


interesting that none of the cross builds need diagnostics-core.h. I see 
it used in 7 different targets.  Must be something on the native build 
command line that is defined which causes it to be needed.

Anyway, Thanks for fixing it.

btw, that should be darwin-driver.c not .h  in the changelog right?

Andrew
>   2015-10-16  Trevor Saunders  <tbsaunde+gcc@tbsaunde.org>
>   
>   	* lra-constraints.c (add_next_usage_insn): Change argument type
> Index: gcc/config/darwin-driver.c
> ===================================================================
> --- gcc/config/darwin-driver.c	(revision 228938)
> +++ gcc/config/darwin-driver.c	(working copy)
> @@ -23,6 +23,7 @@
>   #include "coretypes.h"
>   #include "tm.h"
>   #include "opts.h"
> +#include "diagnostic-core.h"
>   
>   #ifndef CROSS_DIRECTORY_STRUCTURE
>   #include <sys/sysctl.h>
>

Re: config header file reduction patch checked in.

[Bernd Schmidt]

On 10/19/2015 05:53 PM, Andrew MacLeod wrote:
> interesting that none of the cross builds need diagnostics-core.h. I see
> it used in 7 different targets.  Must be something on the native build
> command line that is defined which causes it to be needed.

I'm guessing it's the CROSS_DIRECTORY_STRUCTURE macro which is used by 
darwin targets. It's also used for several other targets, so you may 
want to double check those.


Bernd

Re: config header file reduction patch checked in.

[Mike Stump]

On Oct 23, 2015, at 9:57 AM, Bernd Schmidt <bschmidt@redhat.com> wrote:
> 
> I'm guessing it's the CROSS_DIRECTORY_STRUCTURE macro which is used by darwin targets. It's also used for several other targets, so you may want to double check those.

No, only darwin is special, as presently only darwin has the requisite support in the object file format to do what needs doing.

Re: config header file reduction patch checked in.

[Bernd Schmidt]

On 10/23/2015 07:15 PM, Mike Stump wrote:
> On Oct 23, 2015, at 9:57 AM, Bernd Schmidt <bschmidt@redhat.com>
> wrote:
>>
>> I'm guessing it's the CROSS_DIRECTORY_STRUCTURE macro which is used
>> by darwin targets. It's also used for several other targets, so you
>> may want to double check those.
>
> No, only darwin is special, as presently only darwin has the
> requisite support in the object file format to do what needs doing.

Not sure what you mean by "what needs doing", but grep shows a number of 
uses of CROSS_DIRECTORY_STRUCTURE. Anything that uses it would 
presumably cause a difference between a cross and host build which could 
lead to an issue like the one Iain found.


Bernd

Re: config header file reduction patch checked in.

[Andrew MacLeod]

On 10/23/2015 01:24 PM, Bernd Schmidt wrote:
> On 10/23/2015 07:15 PM, Mike Stump wrote:
>> On Oct 23, 2015, at 9:57 AM, Bernd Schmidt <bschmidt@redhat.com>
>> wrote:
>>>
>>> I'm guessing it's the CROSS_DIRECTORY_STRUCTURE macro which is used
>>> by darwin targets. It's also used for several other targets, so you
>>> may want to double check those.
>>
>> No, only darwin is special, as presently only darwin has the
>> requisite support in the object file format to do what needs doing.
>
> Not sure what you mean by "what needs doing", but grep shows a number 
> of uses of CROSS_DIRECTORY_STRUCTURE. Anything that uses it would 
> presumably cause a difference between a cross and host build which 
> could lead to an issue like the one Iain found.
>
well, in the config directories, darwin-driver.c is the only source file 
which uses the definition... the other places are all .h files.

A quick glance at those and virtually all the uses of the macro are to 
change the definition of a macro...  which is harmless as far as this 
exercise goes.

  darwin-driver.c had some code that depended on one of the include 
files, bit no other part of the file needed it, so that was the issue 
there. The only other place it seems could be an issue is with 
collect2.c...  so I'll monitor that one closely before checking anything 
in.. and get the darwin guys to test it for me before committing.

Andrew

Re: config header file reduction patch checked in.

[Mike Stump]

On Oct 23, 2015, at 10:36 AM, Andrew MacLeod <amacleod@redhat.com> wrote:
> 
> darwin-driver.c had some code that depended on one of the include files, bit no other part of the file needed it, so that was the issue there. The only other place it seems could be an issue is with collect2.c...  so I'll monitor that one closely before checking anything in.. and get the darwin guys to test it for me before committing.

I’m fine with checking it in on the assumption darwin won’t break here.  If it does, I think it will be trivial to clean it up.  Just watch out for any darwin doesn’t build due to missing decls or some such and let them know what header needs to be included in darwin.h based upon the missing decls.  I faced one of these recently with my port as the headers flexed.  Annoying, but not too hard to recover from.

Re: config header file reduction patch checked in.

[Mike Stump]

On Oct 23, 2015, at 10:24 AM, Bernd Schmidt <bschmidt@redhat.com> wrote:
> On 10/23/2015 07:15 PM, Mike Stump wrote:
>> On Oct 23, 2015, at 9:57 AM, Bernd Schmidt <bschmidt@redhat.com>
>> wrote:
>>> 
>>> I'm guessing it's the CROSS_DIRECTORY_STRUCTURE macro which is used
>>> by darwin targets. It's also used for several other targets, so you
>>> may want to double check those.
>> 
>> No, only darwin is special, as presently only darwin has the
>> requisite support in the object file format to do what needs doing.
> 
> Not sure what you mean by "what needs doing", but grep shows a number of uses of CROSS_DIRECTORY_STRUCTURE. Anything that uses it would presumably cause a difference between a cross and host build which could lead to an issue like the one Iain found.

What needs doing, means the ability to pack two different architectures into one file.  If you look at all the uses, you discover two things.  All non-darwin ports use it in trivial ways.  Only darwin uses it (doesn’t use it), in non-trivial ways that may impact headers.  For the non-darwin targets, the use of it is universal, not related to any target.  So, testing any cross (the condition under which things are different) and any non-cross will test most things.  It is this notion of several other targets in your email that just doesn’t apply.  Any target which is a cross, is the wording that would apply.

Re: [patch] header file re-ordering.

[Jeff Law]

On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
> On 10/07/2015 06:02 PM, Jeff Law wrote:
>
> However, the tool has been run, and I've made the minor adjustments
> required to the source files to make it work.  (ie, a few multi-line
> comments and the fact that mul-tables.c is generated on the tile* targets.
>
> So this is what it should look like.  I used -cp.    Other languages are
> bootstrapping, and I have yet to build all the targets... that'll just
> take a day.   Be nice if ada worked tho.
>
> I can run the reduction tool over the weekend (its a long weekend here
> :-) on this if you want...  the other patch is a couple of weeks out of
> date anyway now.
So I'm playing with this stuff a little.  I was surprised to see that 
the reordering script also removes duplicates.

For some dumb reason I thought that functionality was part of the header 
file reducer, but that's only concerned with removing stuff that's 
unnecessary.

Anyway, just surprised me.  Not sure if it's worth splitting that 
functionality out or making it conditional on a flag is worth it.

It certainly helps in that I won't look at the changes and expect that 
headers are just reordered :-)

jeff

Re: [patch] header file re-ordering.

[Andrew MacLeod]

On 10/22/2015 04:55 PM, Jeff Law wrote:
> On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
>> On 10/07/2015 06:02 PM, Jeff Law wrote:
>>
>> However, the tool has been run, and I've made the minor adjustments
>> required to the source files to make it work.  (ie, a few multi-line
>> comments and the fact that mul-tables.c is generated on the tile* 
>> targets.
>>
>> So this is what it should look like.  I used -cp.    Other languages are
>> bootstrapping, and I have yet to build all the targets... that'll just
>> take a day.   Be nice if ada worked tho.
>>
>> I can run the reduction tool over the weekend (its a long weekend here
>> :-) on this if you want...  the other patch is a couple of weeks out of
>> date anyway now.
> So I'm playing with this stuff a little.  I was surprised to see that 
> the reordering script also removes duplicates.
>
> For some dumb reason I thought that functionality was part of the 
> header file reducer, but that's only concerned with removing stuff 
> that's unnecessary.
>
> Anyway, just surprised me.  Not sure if it's worth splitting that 
> functionality out or making it conditional on a flag is worth it.
>
> It certainly helps in that I won't look at the changes and expect that 
> headers are just reordered :-)
>
> jeff
Yeah, the reordering removes anything which is a duplicate.  The way the 
processing works, it was very natural to do it there, and trivial.  And 
it seemed silly to put 2copies in a  row when it reordered something.

The reducer also gets an extra order of complexity when it has to deal 
with duplicate header files.. ie, no longer does a #include become a 
unique thing that I can hash and build a dictionary on... , it has to 
remember whether it was the first or second or nth instance, and it was 
just much much simpler to make it only have to deal with removing 
#Include "header.h".   The original version dealt with multiples OK, but 
I eventually removed it on one of the iterations as being superfluous 
with the addition of the ordering tool.

It was actually only at the 11th hour I decided to keep the ordering 
tool and reducer as seperate tools.. they were going to be combined, but 
it seemed better to leave them separate.

Andrew

Re: [patch] header file re-ordering.

[Jeff Law]

On 10/22/2015 03:07 PM, Andrew MacLeod wrote:
> On 10/22/2015 04:55 PM, Jeff Law wrote:
>> On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
>>> On 10/07/2015 06:02 PM, Jeff Law wrote:
>>>
>>> However, the tool has been run, and I've made the minor adjustments
>>> required to the source files to make it work.  (ie, a few multi-line
>>> comments and the fact that mul-tables.c is generated on the tile*
>>> targets.
>>>
>>> So this is what it should look like.  I used -cp.    Other languages are
>>> bootstrapping, and I have yet to build all the targets... that'll just
>>> take a day.   Be nice if ada worked tho.
>>>
>>> I can run the reduction tool over the weekend (its a long weekend here
>>> :-) on this if you want...  the other patch is a couple of weeks out of
>>> date anyway now.
>> So I'm playing with this stuff a little.  I was surprised to see that
>> the reordering script also removes duplicates.
>>
>> For some dumb reason I thought that functionality was part of the
>> header file reducer, but that's only concerned with removing stuff
>> that's unnecessary.
>>
>> Anyway, just surprised me.  Not sure if it's worth splitting that
>> functionality out or making it conditional on a flag is worth it.
>>
>> It certainly helps in that I won't look at the changes and expect that
>> headers are just reordered :-)
>>
>> jeff
> Yeah, the reordering removes anything which is a duplicate.  The way the
> processing works, it was very natural to do it there, and trivial.  And
> it seemed silly to put 2copies in a  row when it reordered something.
Yup.  Understood.  I'll probably approve a goodly amount of the backend 
files today...  Roughly half are really easy.   Making my 2nd pass 
through right now.

Jeff

Re: [patch] header file re-ordering.

[Jeff Law]

On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
> On 10/07/2015 06:02 PM, Jeff Law wrote:
>> On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
>>> these are all in the main gcc directory. 297 files total.
>>>
>>> Everything bootstraps on x86_64-pc-linux-gnu and
>>> powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still
>>> build. Regressions tests also came up clean.
>>>
>>> OK for trunk?
>> So as I look at this and make various spot checks, what really stands
>> out is how often something like alias.h gets included, often in places
>> that have absolutely no business/need to be looking at that file.
>> Cut-n-paste at its worst.  It happens to many others, but alias.h
>> seems to have gotten its grubby self into just about everywhere for
>> reasons unkonwn.
>>
>> I find myself also wondering if a two step approach would make this
>> easier.  Step #1 being ordering the headers, step #2 being removal of
>> the duplicates.  As you note, the downside is two checkins that would
>> affect most files in the tree.  I guess I'll keep slogging through the
>> patch as is...
>>
>> jeff
> Heres the patch for reordered headers.  Building as we speak.  Hard to
> fully verify since Ada doesn't seem to bootstrap on trunk at the moment:
>
> +===========================GNAT BUG
> DETECTED==============================+
> | 6.0.0 20151008 (experimental) (x86_64-pc-linux-gnu) GCC
> error:           |
> | in gen_lowpart_common, at
> emit-rtl.c:1399                                |
> | Error detected around
> s-regpat.adb:1029:22                               |
>
> <...>
> raised TYPES.UNRECOVERABLE_ERROR : comperr.adb:423
> ../gcc-interface/Makefile:311: recipe for target 's-regpat.o' failed
>
>
> However, the tool has been run, and I've made the minor adjustments
> required to the source files to make it work.  (ie, a few multi-line
> comments and the fact that mul-tables.c is generated on the tile* targets.
>
> So this is what it should look like.  I used -cp.    Other languages are
> bootstrapping, and I have yet to build all the targets... that'll just
> take a day.   Be nice if ada worked tho.
>
> I can run the reduction tool over the weekend (its a long weekend here
> :-) on this if you want...  the other patch is a couple of weeks out of
> date anyway now.
So I'll approve the reordering and duplicate removal for the front-ends 
as well.

Now it's just removal of unnecessary crud and the scripts, right?

jeff

Re: [patch] header file re-ordering.

[Andrew MacLeod]

On 10/23/2015 03:09 PM, Jeff Law wrote:
> On 10/08/2015 07:37 AM, Andrew MacLeod wrote:
>> On 10/07/2015 06:02 PM, Jeff Law wrote:
>>> On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
>>>> these are all in the main gcc directory. 297 files total.
>>>>
>>>> Everything bootstraps on x86_64-pc-linux-gnu and
>>>> powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still
>>>> build. Regressions tests also came up clean.
>>>>
>>>> OK for trunk?
>>> So as I look at this and make various spot checks, what really stands
>>> out is how often something like alias.h gets included, often in places
>>> that have absolutely no business/need to be looking at that file.
>>> Cut-n-paste at its worst.  It happens to many others, but alias.h
>>> seems to have gotten its grubby self into just about everywhere for
>>> reasons unkonwn.
>>>
>>> I find myself also wondering if a two step approach would make this
>>> easier.  Step #1 being ordering the headers, step #2 being removal of
>>> the duplicates.  As you note, the downside is two checkins that would
>>> affect most files in the tree.  I guess I'll keep slogging through the
>>> patch as is...
>>>
>>> jeff
>> Heres the patch for reordered headers.  Building as we speak. Hard to
>> fully verify since Ada doesn't seem to bootstrap on trunk at the moment:
>>
>> +===========================GNAT BUG
>> DETECTED==============================+
>> | 6.0.0 20151008 (experimental) (x86_64-pc-linux-gnu) GCC
>> error:           |
>> | in gen_lowpart_common, at
>> emit-rtl.c:1399                                |
>> | Error detected around
>> s-regpat.adb:1029:22                               |
>>
>> <...>
>> raised TYPES.UNRECOVERABLE_ERROR : comperr.adb:423
>> ../gcc-interface/Makefile:311: recipe for target 's-regpat.o' failed
>>
>>
>> However, the tool has been run, and I've made the minor adjustments
>> required to the source files to make it work.  (ie, a few multi-line
>> comments and the fact that mul-tables.c is generated on the tile* 
>> targets.
>>
>> So this is what it should look like.  I used -cp.    Other languages are
>> bootstrapping, and I have yet to build all the targets... that'll just
>> take a day.   Be nice if ada worked tho.
>>
>> I can run the reduction tool over the weekend (its a long weekend here
>> :-) on this if you want...  the other patch is a couple of weeks out of
>> date anyway now.
> So I'll approve the reordering and duplicate removal for the 
> front-ends as well.
>
> Now it's just removal of unnecessary crud and the scripts, right?
>
> jeff

Correct.  I gave you a new patch for the backend removal stuff today.   
the config stuff is all done. and I'll give you another patch  the 
beginning of next week for just the removals of headers in the front 
ends..  I ran in into some build hiccups which delayed getting it ready 
today as well.

Andrew

Re: [patch 1/3] Header file reduction - backend files.

[Jeff Law]

On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
> these are all in the main gcc directory.  297 files total.
>
> Everything bootstraps on x86_64-pc-linux-gnu and
> powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still
> build. Regressions tests also came up clean.
>
> OK for trunk?
So just to be clear, I'm looking at the backend-reorder patch.  Now that 
I know it's going to be reordering & removing duplicates, I know better 
what to expect.

Can you please look at data-streamer-{in,out}.c, data-streamer.c, 
gimplify-me.c.   Maybe its changed recently, but options.h is removed as 
a duplicate by your patch, but on the trunk, I only see it showing up 
once (using your tools)



I realize it's gotten a little stale, but IMHO once approved, if you 
need to make adjustments due to changes since you originally produced 
the patch those adjustments are pre-approved.

Given all this stuff should be independent, I'm going to start by 
carving out those which I'm most confident are correct.  Please go ahead 
and commit the reorder/remove duplicate changes for the following:



auto-inc-dec.c
bt-load.c
caller-save.c
cfganal.c
cfgbuild.c
cfghooks.c
cfgloop.c
cfgloopanal.c
cfgloopmanip.c
cfgrtl.c
combine-stack-adj.c
compare-elim.c
cse.c
ddg.c
debug.c
dfp.c
dominance.c
double-int.c
dumpfile.c
emit-rtl.c
fixed-value.c
fwprop.c
ggc-common.c
gimple-expr.c
gimple-iterator.c
gimple-pretty-print.c
gimple-walk.c
godump.c
graph.c
graphite-poly.c
hw-dooploop.c
init-regs.c
ipa-comdats.c
ipa-inline-analysis.c
ipa-inline-transform.c
ipa-polyporphic-call.c
ipa-visibility.c
jump.c
lcm.c
lists.c
loop-doloop.c
loop-init.c
loop-iv.c
lower-subreg.c
lto-cgraph.c
lto-section-in.c
lto-section-out.c
lto-streamer.c
mode-switching.c
plugin.c
postreload-gcse.c
predict.c
realmpfr.c
reg-stack.c
regcprop.c
regrename.c
reorg.c
resource.c
rtl-chkp.c
rtl-error.c
rtlanal.c
rtlhooks.c
sched-ebb.c
sched-rgn.c
stack-ptr-mod.c
statistics.c
store-motion.c
stringpool.c
tracer.c
tree-call-cdce.c
tree-cfgcleanup.c
tree-chkp.c
tree-complex.c
tree-dfa.c
tree-diagnostic.c
tree-eh.c
tree-inline.c
tree-into-ssa.c
tree-iterator.c
tree-nested.c
tree-object-size.c
tree-outof-ssa.c
tree-phinodes.c
tree-sra.c
tree-ssa-address.c
tree-ssa-coalesce.c
tree-ssa-copy.c
tree-ssa-dce.c
tree-ssa-dom.c
tree-ssa-dse.c
tree-ssa-ifcombine.c
tree-ssa-live.c
tree-ssa-loop-im.c
tree-ssa-loop-ivopts.c
tree-ssa-loop-manip.c
tree-ssa-loop-unswitch.c
tree-ssa-loop.c
tree-ssa-phiopt.c
tree-ssa-phiprop.c
tree-ssa-pre.c
tree-ssa-propagate.c
tree-ssa-sccvn.c
tree-ssa-sink.c
tree-ssa-structalias.c
tree-ssa-ter.c
tree-ssa-threadedge.c
tree-ssa-uninit.c
tree-ssanames.c
tree-switch-conversion.c
tree-tailcall.c
tree-vect-loop-manip.c
tree-vectorizer.c
tree-vrp.c
var-tracking.c
web.c




While I was going through those, I'm pretty sure there's another sizable 
number which I can get through on a second pass.  So the following are 
approved as well:

alias.c
attribs.c
auto-profile.c
cfg.c
cfgcleanup.c
cgraph.c
cgraphbuild.c
cgraphclones.c
convert.c
cprop.c
cselib.c
dbxout.c
dce.c
df-core.c
df-problems.c
df-scan.c
dojump.c
dse.c
dwarf2asm.c
dwarf2cfi.c
dwarf2out.c
fold-const.c
function.c
incpath.c
profile.c

Out of time for now.  More to follow...

If you want to commit the stuff approved so far, be my guest.


Jeff

Re: [patch 1/3] Header file reduction - backend files.

[Andrew MacLeod]

On 10/22/2015 06:25 PM, Jeff Law wrote:
> On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
>> these are all in the main gcc directory. 297 files total.
>>
>> Everything bootstraps on x86_64-pc-linux-gnu and
>> powerpc64le-unknown-linux-gnu.  All targets in config-list.mk still
>> build. Regressions tests also came up clean.
>>
>> OK for trunk?
> So just to be clear, I'm looking at the backend-reorder patch. Now 
> that I know it's going to be reordering & removing duplicates, I know 
> better what to expect.
>
> Can you please look at data-streamer-{in,out}.c, data-streamer.c, 
> gimplify-me.c.   Maybe its changed recently, but options.h is removed 
> as a duplicate by your patch, but on the trunk, I only see it showing 
> up once (using your tools)
>
show-headers data-streamer.c  -soptions.h
on trunk shows me:
   alias.h
   backend.h
     tm.h
       options.h  (1)               <<-------
         flag-types.h
         i386-opts.h
<...>
   hard-reg-set.h
   options.h  (2)                <<-------
   fold-const.h

so the first occurrence is loaded via tm.h from backend.h


>
>
> I realize it's gotten a little stale, but IMHO once approved, if you 
> need to make adjustments due to changes since you originally produced 
> the patch those adjustments are pre-approved.

yeah. what I've been doing is apply the patch... and anything that fails 
to apply, simply run the tool again on that file...  its easier than 
trying to fix things up :-)

Andrew

Re: [patch 1/3] Header file reduction - backend files.

[Jeff Law]

On 10/22/2015 04:25 PM, Jeff Law wrote:
> On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
>> these are all in the main gcc directory.  297 files total.
[ ... ]
>
> Out of time for now.  More to follow...
So a quirk of show-headers.   Let's look at cgraphunit.c on the trunk:

[law@tor gcc]$ ../contrib/show-headers cgraphunit.c -sgimplify.h
[ ... ]
   gimplify.h                <<-------
[ ... ]

That's all it spits out.  But...
#include "gimplify.h"
#include "gimplify.h"

I'm not sure if that's intentional, but it does make it harder to use 
show-headers to help understand the changes being made by your scripts.


As for the rest of the backend files.  They're OK for the trunk.

jeff

Re: [patch 1/3] Header file reduction - backend files.

[Andrew MacLeod]

On 10/23/2015 01:43 AM, Jeff Law wrote:
> On 10/22/2015 04:25 PM, Jeff Law wrote:
>> On 10/01/2015 08:33 PM, Andrew MacLeod wrote:
>>> these are all in the main gcc directory.  297 files total.
> [ ... ]
>>
>> Out of time for now.  More to follow...
> So a quirk of show-headers.   Let's look at cgraphunit.c on the trunk:
>
> [law@tor gcc]$ ../contrib/show-headers cgraphunit.c -sgimplify.h
> [ ... ]
>   gimplify.h                <<-------
> [ ... ]
>
> That's all it spits out.  But...
> #include "gimplify.h"
> #include "gimplify.h"
>
> I'm not sure if that's intentional, but it does make it harder to use 
> show-headers to help understand the changes being made by your scripts.
>
>
> As for the rest of the backend files.  They're OK for the trunk.
>

Hmm. yeah,   It was mostly intended to show you what structure you have 
and are getting indirectly, not so much indicate that you actually 
include things directly twice.    It utilizes a common function which 
returns unique header files (find_unique_include_list) and just 
processes those.  A quick tweak should resolve that to just linearly 
parse and process.

It also occurs to me the reason you only found one options.h in the 
previous note... show-headers probably didn't know where to look for 
your build directory to delve into tm.h.   It defaults to ../../build  
which is what I use.  I will also tweak that to attempt to automatically 
find a build directory below ../..  and failing that, have you specify one.

Andrew

Re: [patch 1/3] Header file reduction - backend files.

[Jeff Law]

On 10/23/2015 06:24 AM, Andrew MacLeod wrote:
>
> Hmm. yeah,   It was mostly intended to show you what structure you have
> and are getting indirectly, not so much indicate that you actually
> include things directly twice.    It utilizes a common function which
> returns unique header files (find_unique_include_list) and just
> processes those.  A quick tweak should resolve that to just linearly
> parse and process.
It may not be worth the trouble.

There were maybe a half-dozen files with repeated direct includes and 
they stood out as not being as mechanical as the others (with respect to 
reviewing the changes).  I suspect that in the future it won't happen 
much, if at all, and we won't be looking at 15k line patches to clean 
things up.

>
> It also occurs to me the reason you only found one options.h in the
> previous note... show-headers probably didn't know where to look for
> your build directory to delve into tm.h.   It defaults to ../../build
> which is what I use.  I will also tweak that to attempt to automatically
> find a build directory below ../..  and failing that, have you specify one.
Yea, I almost asked about procedures for the tool to find tm.h as it was 
relatively common for tm.h to include something indirectly and make 
later stuff redundant.  But after a while, I got pretty good at knowing 
what was in tm.h and how that would effect what headers became redundant 
as tm.h got included earlier (via target.h).   Similarly for optabs.h 
moving around and making other stuff redundant.

Jeff

Re: [patch 1/3] Header file reduction - backend files.

[Andrew MacLeod]

[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]

On 10/23/2015 11:11 AM, Jeff Law wrote:
> On 10/23/2015 06:24 AM, Andrew MacLeod wrote:
>>
>> Hmm. yeah,   It was mostly intended to show you what structure you have
>> and are getting indirectly, not so much indicate that you actually
>> include things directly twice.    It utilizes a common function which
>> returns unique header files (find_unique_include_list) and just
>> processes those.  A quick tweak should resolve that to just linearly
>> parse and process.
> It may not be worth the trouble.
>
> There were maybe a half-dozen files with repeated direct includes and 
> they stood out as not being as mechanical as the others (with respect 
> to reviewing the changes).  I suspect that in the future it won't 
> happen much, if at all, and we won't be looking at 15k line patches to 
> clean things up.
>
>>
>> It also occurs to me the reason you only found one options.h in the
>> previous note... show-headers probably didn't know where to look for
>> your build directory to delve into tm.h.   It defaults to ../../build
>> which is what I use.  I will also tweak that to attempt to automatically
>> find a build directory below ../..  and failing that, have you 
>> specify one.
> Yea, I almost asked about procedures for the tool to find tm.h as it 
> was relatively common for tm.h to include something indirectly and 
> make later stuff redundant.  But after a while, I got pretty good at 
> knowing what was in tm.h and how that would effect what headers became 
> redundant as tm.h got included earlier (via target.h).   Similarly for 
> optabs.h moving around and making other stuff redundant.
>

Apply this diff to the contents of contrib/headers

Duplicates are now treated as they are seen instead of ignored.
Rather than hardcoding ../../build/gcc, the tool now makes an attempt to 
find a "gcc/tm.h" from ../..   It will print a message if it fails to 
find one.  In which case you can use -i to specify a directory to look in

Andrew



[-- Attachment #2: H --]
[-- Type: text/plain, Size: 6412 bytes --]

*** headerutils.py	2015-10-23 11:55:46.652867427 -0400
--- /home/amacleod/headers/headerutils.py	2015-10-23 11:57:13.247023166 -0400
*************** def ii_write (fname, obj):
*** 216,221 ****
--- 216,258 ----
    pickle.dump (obj[5], f)
    f.close ()
  
+ # execute a system command which returns file names
+ def execute_command (command):
+   files = list()
+   f = os.popen (command)
+   for x in f:
+     if x[0:2] == "./":
+       fn = x.rstrip()[2:]
+     else:
+       fn = x.rstrip()
+     files.append(fn)
+   return files
+ 
+ # Try to locate a build directory from PATH
+ def find_gcc_bld_dir (path):
+   blddir = ""
+   # Look for blddir/gcc/tm.h
+   command = "find " + path + " -mindepth 2 -maxdepth 3 -name tm.h"
+   files = execute_command (command)
+   for y in files:
+     p = os.path.dirname (y)
+     if os.path.basename (p) == "gcc":
+       blddir = p
+       break
+   # If not found, try looking a bit deeper
+   # Dont look this deep initially because a lot of cross target builds may show
+   # up in the list before a native build... but those are better than nothing.
+   if not blddir:
+     command = "find " + path + " -mindepth 3 -maxdepth 5 -name tm.h"
+     files = execute_command (command)
+     for y in files:
+       p = os.path.dirname (y)
+       if os.path.basename (p) == "gcc":
+ 	blddir = p
+ 	break
+ 
+   return blddir
+ 
  
  # Find files matching pattern NAME, return in a list.
  # CURRENT is True if you want to include the current directory
*************** def find_gcc_files (name, current, deepe
*** 235,247 ****
        command = "find -maxdepth 4 -mindepth 2 -name " + name + " -not -path \"./testsuite/*\""
  
    if command != "":
!     f = os.popen (command)
!     for x in f:
!       if x[0] == ".":
!         fn = x.rstrip()[2:]
!       else:
!         fn = x
!       files.append(fn)
  
    return files
  
--- 272,278 ----
        command = "find -maxdepth 4 -mindepth 2 -name " + name + " -not -path \"./testsuite/*\""
  
    if command != "":
!     files = execute_command (command)
  
    return files
  
*** show-headers	2015-10-23 11:55:46.655867328 -0400
--- /home/amacleod/headers/show-headers	2015-10-23 11:53:43.668906942 -0400
*************** sawcore = False
*** 17,26 ****
  # list of headers to emphasize
  highlight = list ()
  
  # search path for headers
! incl_dirs = [".", "../include", "../../build/gcc", "../libcpp/include" ]
  # extra search paths to look in *after* the directory the source file is in. 
- extra_dirs = [ "common", "c-family", "c", "cp", "config" ]
  
  # append (1) to the end of the first line which includes INC in list INC.
  def append_1 (output, inc):
--- 17,26 ----
  # list of headers to emphasize
  highlight = list ()
  
+ bld_dir = ""
  # search path for headers
! incl_dirs = ["../include", "../libcpp/include", "common", "c-family", "c", "cp", "config" ]
  # extra search paths to look in *after* the directory the source file is in. 
  
  # append (1) to the end of the first line which includes INC in list INC.
  def append_1 (output, inc):
*************** def process_include (inc, indent):
*** 75,89 ****
  
      
  
! blddir = [ "." ]
  usage = False
  src = list()
  
  for x in sys.argv[1:]:
    if x[0:2] == "-i":
      bld = x[2:]
!     print "Build dir : " + bld
!     blddir.append (bld)
    elif x[0:2] == "-s":
      highlight.append (os.path.basename (x[2:]))
    elif x[0:2] == "-h":
--- 75,88 ----
  
      
  
! extradir = list()
  usage = False
  src = list()
  
  for x in sys.argv[1:]:
    if x[0:2] == "-i":
      bld = x[2:]
!     extradir.append (bld)
    elif x[0:2] == "-s":
      highlight.append (os.path.basename (x[2:]))
    elif x[0:2] == "-h":
*************** if usage:
*** 104,129 ****
    print " is included in a source file.  Should be run from the source directory"
    print " files from find-include-depends"
    print "      -s : search for a header, and point it out."
!   print "      -i : Specifies 1 or more directories to search for includes."
    sys.exit(0)
  
  
- if len(blddir) > 1:
-   incl_dirs = blddir
  
  x = src[0]
- # if source is in a subdirectory, add the subdirectory to the search list
  srcpath = os.path.dirname(x)
  if srcpath:
!   incl_dirs.append (srcpath)
! for yy in extra_dirs:
!   incl_dirs.append (yy)
  
  output = list()
  sawcore = False
! incl = find_unique_include_list (x)
! for inc in incl:
!   process_include (inc, 1)
  print "\n" + x
  for line in output:
    print line
--- 103,142 ----
    print " is included in a source file.  Should be run from the source directory"
    print " files from find-include-depends"
    print "      -s : search for a header, and point it out."
!   print "      -i : Specifies additonal directories to search for includes."
    sys.exit(0)
  
  
  
+ if extradir:
+   incl_dirs = extradir + incl_dirs;
+ 
+ blddir = find_gcc_bld_dir ("../..")
+ 
+ if blddir:
+   print "Using build directory: " + blddir
+   incl_dirs.insert (0, blddir)
+ else:
+   print "Could not find a build directory, better results if you specify one with -i"
+ 
+ # search path is now ".", blddir, extradirs_from_-i, built_in_incl_dirs
+ incl_dirs.insert (0, ".")
+ 
+ # if source is in a subdirectory, prepend the subdirectory to the search list
  x = src[0]
  srcpath = os.path.dirname(x)
  if srcpath:
!   incl_dirs.insert (0, srcpath)
  
  output = list()
  sawcore = False
! 
! data = open (x).read().splitlines()
! for line in data:
!   d = find_pound_include (line, True, True)
!   if d and d[-2:] == ".h":
!     process_include (d, 1)
! 
  print "\n" + x
  for line in output:
    print line
*** README	2015-10-23 11:55:46.655867328 -0400
--- /home/amacleod/headers/README	2015-10-23 11:58:50.100841918 -0400
*************** show-headers
*** 62,68 ****
    is indented, and when any duplicate headers are seen, they have their
    duplicate number shown
  
!   -i may be used to specify alternate search directories for headers to parse.
    -s specifies headers to look for and emphasize in the output.
  
    This tool must be run in the core gcc source directory.
--- 62,68 ----
    is indented, and when any duplicate headers are seen, they have their
    duplicate number shown
  
!   -i may be used to specify additional search directories for headers to parse.
    -s specifies headers to look for and emphasize in the output.
  
    This tool must be run in the core gcc source directory.

[patch 3/3] Header file reduction - FE files.

[Andrew MacLeod]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

141 front end files.. all in subdirectories of gcc.

Everything bootstraps an a x86_64-pc-linux-gnu and 
powerpc64le-unknown-linux-gnu with 
--enable-languages=all,ada,go,obj-c++,jit --enable-host-shared.  All 
targets in config-list.mk still build. Regressions tests also came up 
clean.

OK for trunk?

Andrew

[-- Attachment #2: FE.patch.bz2 --]
[-- Type: application/x-bzip, Size: 10885 bytes --]

[patch 2/3] Header file reduction - config files.

[Andrew MacLeod]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

there are most of the config files...   109 files total.

  I built every target in config-list.mk.   The reduction tool reduced 
each source file by examining *every* target object directory for the 
resulting object, and testing the reduction each of those targets.   
This took a long time to run :-)

all targets in config-list.mk still build, and x86 was bootstrapped 
regression tested on x86_64-pc-linux-gnu.  OK for trunk?

Andrew

[-- Attachment #2: config.patch.bz2 --]
[-- Type: application/x-bzip, Size: 8919 bytes --]

Re: [patch 0/3] Header file reduction.

[Bernd Schmidt]

On 10/02/2015 04:22 AM, Andrew MacLeod wrote:
> The patches are generated by a pair of tools.
> * gcc-order-includes goes through the headers and canonically reorders some of our more common/troublesome headers and removes any duplicates.  This includes headers which are included by other headers. (ie, obstack.h can be removed as a duplicate if bitmap.h is included already.)
> * remove-includes is the tool which tries to remove each non-conditional header file and does the real work.

Is the bitmap/obstack example really one of a change that is desirable? 
I think if a file uses obstacks then an include of obstack.h is 
perfectly fine, giving us freedom to e.g. change bitmaps not to use 
obstacks. Given that multiple headers include obstack.h, and pretty much 
everything seems to indirectly include bitmap.h anyway, maybe a better 
change would be to just include it always in system.h.

> I'll have a patch shortly to add these and some other useful tools to a
> header-tools directory in contrib.

How soon? It's difficult to meaningfully comment on these patches 
without looking at how they were generated. Two points:
  * diff -c is somewhat unusual and I find diff -u much more readable.
  * Maybe the patches for reordering and removing should be split, also
    for readability and for easier future identification of problems.


Bernd

Re: [patch 0/3] Header file reduction.

[Richard Biener]

On Mon, Oct 5, 2015 at 3:27 PM, Bernd Schmidt <bschmidt@redhat.com> wrote:
> On 10/02/2015 04:22 AM, Andrew MacLeod wrote:
>>
>> The patches are generated by a pair of tools.
>> * gcc-order-includes goes through the headers and canonically reorders
>> some of our more common/troublesome headers and removes any duplicates.
>> This includes headers which are included by other headers. (ie, obstack.h
>> can be removed as a duplicate if bitmap.h is included already.)
>> * remove-includes is the tool which tries to remove each non-conditional
>> header file and does the real work.
>
>
> Is the bitmap/obstack example really one of a change that is desirable? I
> think if a file uses obstacks then an include of obstack.h is perfectly
> fine, giving us freedom to e.g. change bitmaps not to use obstacks. Given
> that multiple headers include obstack.h, and pretty much everything seems to
> indirectly include bitmap.h anyway, maybe a better change would be to just
> include it always in system.h.

Not system.h please - use coretypes.h if really necessary.

Richard.

>> I'll have a patch shortly to add these and some other useful tools to a
>> header-tools directory in contrib.
>
>
> How soon? It's difficult to meaningfully comment on these patches without
> looking at how they were generated. Two points:
>  * diff -c is somewhat unusual and I find diff -u much more readable.
>  * Maybe the patches for reordering and removing should be split, also
>    for readability and for easier future identification of problems.
>
>
> Bernd
>

Re: [patch 0/3] Header file reduction.

[Andrew MacLeod]

On 10/05/2015 09:27 AM, Bernd Schmidt wrote:
> On 10/02/2015 04:22 AM, Andrew MacLeod wrote:
>> The patches are generated by a pair of tools.
>> * gcc-order-includes goes through the headers and canonically 
>> reorders some of our more common/troublesome headers and removes any 
>> duplicates.  This includes headers which are included by other 
>> headers. (ie, obstack.h can be removed as a duplicate if bitmap.h is 
>> included already.)
>> * remove-includes is the tool which tries to remove each 
>> non-conditional header file and does the real work.
>
> Is the bitmap/obstack example really one of a change that is 
> desirable? I think if a file uses obstacks then an include of 
> obstack.h is perfectly fine, giving us freedom to e.g. change bitmaps 
> not to use obstacks. Given that multiple headers include obstack.h, 
> and pretty much everything seems to indirectly include bitmap.h 
> anyway, maybe a better change would be to just include it always in 
> system.h.

Its just an example of the sort of redundant includes the tool removes.  
And your assertion turns out to be incorrect... bitmap.h is barely used 
outside the backend, thus it is included in the backend.h aggregator 
(This is the only header now which includes bitmap.h... Most of this 
many-month effort was to untangle all those indirect includes.)     
There are only 6  remaining uses of bitmap.h in all the front end 
files.   ( Most files can get obstack.h from tree.h. (it comes from 
symtab.h in tree-core.)  If they don't get it there, it often comes from 
diagnostics-core.h.)

   I don't see the point in leaving redundant #includes in the source 
code because of direct uses from that header in the source.  I'm not 
even sure how I could automate detecting that accurately.. Going 
forward,  If anyone ever makes a change which removes a header from an 
include file, they just have to correct the fallout. heh. Thats kinda 
all I've done for 4 months :-)   At least we'll have grasp of the 
ramifications..


>> I'll have a patch shortly to add these and some other useful tools to a
>> header-tools directory in contrib.
>
> How soon? It's difficult to meaningfully comment on these patches 
> without looking at how they were generated. Two points:

within the next day, im just cobbling together some minimal 
documentation.  Dunno how much that will help reviewing the patches 
tho.  the include reduction process was described in more detail earlier 
in this project.

>  * diff -c is somewhat unusual and I find diff -u much more readable.

unsual? I've been using -cp for the past 2 decades and no one has ever 
mentioned it before...   poking around  the wiki I see it mentions you 
can use either -up or -cp.

I guess I could repackage things using -up...  I don't even know where 
my script is to change it :-).   is -u what everyone uses now? no one 
has mentioned it before that I am aware of.


>  * Maybe the patches for reordering and removing should be split, also
>    for readability and for easier future identification of problems.
>
I was trying to avoid too much churn on 550ish files...  I didn't think 
each one needed 2 sets of check-ins.    It could be done, but it will 
take a while.  The reordering patch can be quickly generated, but the 
reduction on all those files will take the better part of a week.

My theory is it perfectly safe to back out any single file from the 
patch set if we discover it has an issue and then examine what the root 
of the problem is..

tool patch coming shortly... probably tomorrow now.

Andrew

Re: [patch 0/3] Header file reduction.

[Bernd Schmidt]

On 10/05/2015 10:10 PM, Andrew MacLeod wrote:
> Its just an example of the sort of redundant includes the tool removes.
> And your assertion turns out to be incorrect... bitmap.h is barely used
> outside the backend, thus it is included in the backend.h aggregator
> (This is the only header now which includes bitmap.h... Most of this
> many-month effort was to untangle all those indirect includes.)

I said a few headers include obstack.h, not bitmap.h, and that's true in 
my (maybe a week old) checkout. My suggestion was to move the include of 
the former to (as Richi corrected) coretypes.h.

And it's one example, but it does point out a problem with this sort of 
automated approach: realistically no one is going to check the whole 
patch, and it may contain changes that could be done better.

>>  * diff -c is somewhat unusual and I find diff -u much more readable.
>
> unsual? I've been using -cp for the past 2 decades and no one has ever
> mentioned it before...   poking around  the wiki I see it mentions you
> can use either -up or -cp.
>
> I guess I could repackage things using -up...  I don't even know where
> my script is to change it :-).   is -u what everyone uses now? no one
> has mentioned it before that I am aware of.

I'm pretty much used to seeing diff -u, whenever I get a -c diff things 
become harder to work out, because the region in the diff you're looking 
at never tells you the full story. In this case in particular, the 
existence of both reordering and removing changes makes it very hard to 
mentally keep track of what's going on.

Let's take this example:

Index: attribs.c
===================================================================
*** attribs.c   (revision 228331)
--- attribs.c   (working copy)
*************** along with GCC; see the file COPYING3.
*** 20,36 ****
   #include "config.h"
   #include "system.h"
   #include "coretypes.h"
! #include "tm.h"
   #include "tree.h"
- #include "alias.h"
   #include "stringpool.h"
   #include "attribs.h"
   #include "stor-layout.h"
- #include "flags.h"
- #include "diagnostic-core.h"
- #include "tm_p.h"
- #include "cpplib.h"
- #include "target.h"
   #include "langhooks.h"
   #include "plugin.h"

--- 20,31 ----
   #include "config.h"
   #include "system.h"
   #include "coretypes.h"
! #include "target.h"
   #include "tree.h"
   #include "stringpool.h"
+ #include "diagnostic-core.h"
   #include "attribs.h"
   #include "stor-layout.h"
   #include "langhooks.h"
   #include "plugin.h"

You could be misled into thinking that diagnostic-core.h and target.h 
are removed, and the algorithm confuses the issue by showing that lines 
"changed" rather than getting removed or added. With a unified diff, the 
size is cut in half which makes things more readable to begin with:

   Index: attribs.c
===================================================================
--- attribs.c   (revision 228331)
+++ attribs.c   (working copy)
@@ -20,17 +20,12 @@
  #include "config.h"
  #include "system.h"
  #include "coretypes.h"
-#include "tm.h"
+#include "target.h"
  #include "tree.h"
-#include "alias.h"
  #include "stringpool.h"
+#include "diagnostic-core.h"
  #include "attribs.h"
  #include "stor-layout.h"
-#include "flags.h"
-#include "diagnostic-core.h"
-#include "tm_p.h"
-#include "cpplib.h"
-#include "target.h"
  #include "langhooks.h"
  #include "plugin.h"

and with things closer together it's easier to follow what's actually 
going on. This is a smaller example, many instances in your patch are 
actually about a page long and you have to scroll back and forth to work 
things out, getting confused because everything in the 410k of text 
looks the same.

This was actually one of the reasons I proposed splitting the patch into 
reordering and removal phases, it would alleviate the diff -c disadvantages.

> the reduction on all those files will take the better part of a week.

That's a little concerning due to the possibility of intervening 
commits. I'd like to make one requirement for checkin, that you take the 
revision at which you're committing and then run the script again, 
verifying that the process produces the same changes as the patch you 
committed. (Or do things in smaller chunks.).


Bernd

Re: [patch 0/3] Header file reduction.

[Andrew MacLeod]

On 10/05/2015 04:37 PM, Bernd Schmidt wrote:
> On 10/05/2015 10:10 PM, Andrew MacLeod wrote:
>> Its just an example of the sort of redundant includes the tool removes.
>> And your assertion turns out to be incorrect... bitmap.h is barely used
>> outside the backend, thus it is included in the backend.h aggregator
>> (This is the only header now which includes bitmap.h... Most of this
>> many-month effort was to untangle all those indirect includes.)
>
> I said a few headers include obstack.h, not bitmap.h, and that's true 
> in my (maybe a week old) checkout. My suggestion was to move the 
> include of the former to (as Richi corrected) coretypes.h.
>

Ah, sorry the parsing was non-deterministic and I parsed it the other 
way.   My comments refer to the "true" dependencies in the code after 
all the un-needed headers have been trimmed out... i've little doubt 
mainline shows obstack.h and bitmap.h being included everywhere.

In any case, a direct include of obstack.h in coretypes.h was considered 
earlier in the aggregation process and it didn't show up as something 
that would be a win.  It is included a couple of common places that we 
have no control over..  in particular libcpp/include/symtab.h includes 
obstack.h and  is included by tree-core.h.  A very significant number of 
files bring that in.  If we included obstack.h in coretypes.h then those 
files would be including it again for a second time for no particularly 
good reason.  So I made the judgement call to not put it in coretypes.h.

> And it's one example, but it does point out a problem with this sort 
> of automated approach: realistically no one is going to check the 
> whole patch, and it may contain changes that could be done better.

The point being that the aggregation *wasn't* automated... and has 
nothing to do with this patch set.    I analyzed and preformed all that 
sort of thing  earlier. Sure judgment calls were made, but it wasn't 
automated in the slightest.   There are certainly further aggregation 
improvements that could be made... and either I or someone else could do 
more down the road.,  The heavy lifting has all been done now.

So  the *only* thing that is automated is removing include files which 
are not needed so that we can get an idea of what the true dependencies 
in the source base are.




>>>  * diff -c is somewhat unusual and I find diff -u much more readable.
>>
>> unsual? I've been using -cp for the past 2 decades and no one has ever
>> mentioned it before...   poking around  the wiki I see it mentions you
>> can use either -up or -cp.
>>
>> I guess I could repackage things using -up...  I don't even know where
>> my script is to change it :-).   is -u what everyone uses now? no one
>> has mentioned it before that I am aware of.
>
> I'm pretty much used to seeing diff -u, whenever I get a -c diff 
> things become harder to work out, because the region in the diff 
> you're looking at never tells you the full story. In this case in 
> particular, the existence of both reordering and removing changes 
> makes it very hard to mentally keep track of what's going on.
>

I can switch to -u.. I've just never seen the request before.

I can regenerate the patches with -u if you want.
>
>> the reduction on all those files will take the better part of a week.
>
> That's a little concerning due to the possibility of intervening 
> commits. I'd like to make one requirement for checkin, that you take 
> the revision at which you're committing and then run the script again, 
> verifying that the process produces the same changes as the patch you 
> committed. (Or do things in smaller chunks.).
>

Well, sure there are intervening commits.. the only ones that actually 
matter are the ones which fail to compile because someone made a code 
change which now requires a header that wasn't needed before.  which is 
really a state we're looking for I think.   I fix those up before 
committing.   Its *possible* a conditional compilation issue could creep 
in, but highly unlikely.

I can rerun everything on that revision from just before I committed and 
see if everything is the same.  It'll take a week to find out :-) but 
that seems like a reasonable sanity check.

Andrew

Re: [patch 0/3] Header file reduction. - unified patches

[Andrew MacLeod]

[-- Attachment #1: Type: text/plain, Size: 306 bytes --]

On 10/05/2015 05:11 PM, Andrew MacLeod wrote:
>
> I can switch to -u.. I've just never seen the request before.
>
> I can regenerate the patches with -u if you want.

You are right, the patches are significantly easier to read with -u..  
I've changed my svn diff script.    here's all 3 patches:

Andrew


[-- Attachment #2: backend.patch.bz2 --]
[-- Type: application/x-bzip, Size: 19751 bytes --]

[-- Attachment #3: FE.patch.bz2 --]
[-- Type: application/x-bzip, Size: 9513 bytes --]

[-- Attachment #4: config.patch.bz2 --]
[-- Type: application/x-bzip, Size: 7709 bytes --]

Re: [patch 0/3] Header file reduction.

[Jeff Law]

On 10/05/2015 03:11 PM, Andrew MacLeod wrote:
>
> In any case, a direct include of obstack.h in coretypes.h was considered
> earlier in the aggregation process and it didn't show up as something
> that would be a win.  It is included a couple of common places that we
> have no control over..  in particular libcpp/include/symtab.h includes
> obstack.h and  is included by tree-core.h.  A very significant number of
> files bring that in.  If we included obstack.h in coretypes.h then those
> files would be including it again for a second time for no particularly
> good reason.  So I made the judgement call to not put it in coretypes.h.
And just as important, we can revisit the aggregators and when we do so, 
we ought to be able to answer the question, "if obstack.h is put into 
coretypes.h" does that clean things up elsewhere and re-run the tools to 
clean things up.


>
>> And it's one example, but it does point out a problem with this sort
>> of automated approach: realistically no one is going to check the
>> whole patch, and it may contain changes that could be done better.
>
> The point being that the aggregation *wasn't* automated... and has
> nothing to do with this patch set.    I analyzed and preformed all that
> sort of thing  earlier. Sure judgment calls were made, but it wasn't
> automated in the slightest.   There are certainly further aggregation
> improvements that could be made... and either I or someone else could do
> more down the road.,  The heavy lifting has all been done now.
Agreed.

>
> So  the *only* thing that is automated is removing include files which
> are not needed so that we can get an idea of what the true dependencies
> in the source base are.
Also agreed.

>>> the reduction on all those files will take the better part of a week.
>>
>> That's a little concerning due to the possibility of intervening
>> commits. I'd like to make one requirement for checkin, that you take
>> the revision at which you're committing and then run the script again,
>> verifying that the process produces the same changes as the patch you
>> committed. (Or do things in smaller chunks.).
>>
>
> Well, sure there are intervening commits.. the only ones that actually
> matter are the ones which fail to compile because someone made a code
> change which now requires a header that wasn't needed before.  which is
> really a state we're looking for I think.   I fix those up before
> committing.   Its *possible* a conditional compilation issue could creep
> in, but highly unlikely.
More likely is conditional compilation will be removed :-)  We're trying 
to get away from conditional compilation as a general direction.

Intervening commits are always a problem with this kind of large patch 
that hits many places.   But IMHO, they're an easily managed problem.

jeff

Re: [patch 0/3] Header file reduction.

[Jeff Law]

On 10/05/2015 02:10 PM, Andrew MacLeod wrote:
>>
>> Is the bitmap/obstack example really one of a change that is
>> desirable? I think if a file uses obstacks then an include of
>> obstack.h is perfectly fine, giving us freedom to e.g. change bitmaps
>> not to use obstacks. Given that multiple headers include obstack.h,
>> and pretty much everything seems to indirectly include bitmap.h
>> anyway, maybe a better change would be to just include it always in
>> system.h.
>
> Its just an example of the sort of redundant includes the tool removes.
It may not be the best example.  The tools don't treat obstack specially 
(nor should they IMHO).  So let's pretend it's not obstack.h which has 
been arguably a core part of GCC for a long time.

>
>    I don't see the point in leaving redundant #includes in the source
> code because of direct uses from that header in the source.  I'm not
> even sure how I could automate detecting that accurately.. Going
> forward,  If anyone ever makes a change which removes a header from an
> include file, they just have to correct the fallout. heh. Thats kinda
> all I've done for 4 months :-)   At least we'll have grasp of the
> ramifications..
And the last sentence is the key here.  We're trying to get to a point 
where we can make certain kinds of changes, then have the compiler spit 
out errors, fix the errors and have a high degree of confidence that the 
final change is correct and that we've found all the places that need to 
change.

The change could be as simple as moving a function declaration to its 
natural place, collecting interfaces & data into classes, or something 
more ambitious like removing trees from the backend.  Folks will note 
that these are all refactorings that we don't want to change any 
observable behaviour.





>>  * diff -c is somewhat unusual and I find diff -u much more readable.
>
> unsual? I've been using -cp for the past 2 decades and no one has ever
> mentioned it before...   poking around  the wiki I see it mentions you
> can use either -up or -cp.
>
> I guess I could repackage things using -up...  I don't even know where
> my script is to change it :-).   is -u what everyone uses now? no one
> has mentioned it before that I am aware of.
I'm probably the last person in the world that still generally prefers 
-cp :-)  I'm getting to the point where I can tolerate -u.


>
>
>>  * Maybe the patches for reordering and removing should be split, also
>>    for readability and for easier future identification of problems.
>>
> I was trying to avoid too much churn on 550ish files...  I didn't think
> each one needed 2 sets of check-ins.    It could be done, but it will
> take a while.  The reordering patch can be quickly generated, but the
> reduction on all those files will take the better part of a week.
>
> My theory is it perfectly safe to back out any single file from the
> patch set if we discover it has an issue and then examine what the root
> of the problem is..
>
> tool patch coming shortly... probably tomorrow now.
I haven't looked at the 3 patches in detail yet.  Given my familiarity 
with the overall process/goal, I can probably handle them as-is. 
They're just big mechanical changes.

jeff

Re: [patch 0/3] Header file reduction.

[Richard Biener]

On Tue, Oct 6, 2015 at 11:43 PM, Jeff Law <law@redhat.com> wrote:
> On 10/05/2015 02:10 PM, Andrew MacLeod wrote:
>>>
>>>
>>> Is the bitmap/obstack example really one of a change that is
>>> desirable? I think if a file uses obstacks then an include of
>>> obstack.h is perfectly fine, giving us freedom to e.g. change bitmaps
>>> not to use obstacks. Given that multiple headers include obstack.h,
>>> and pretty much everything seems to indirectly include bitmap.h
>>> anyway, maybe a better change would be to just include it always in
>>> system.h.
>>
>>
>> Its just an example of the sort of redundant includes the tool removes.
>
> It may not be the best example.  The tools don't treat obstack specially
> (nor should they IMHO).  So let's pretend it's not obstack.h which has been
> arguably a core part of GCC for a long time.
>
>>
>>    I don't see the point in leaving redundant #includes in the source
>> code because of direct uses from that header in the source.  I'm not
>> even sure how I could automate detecting that accurately.. Going
>> forward,  If anyone ever makes a change which removes a header from an
>> include file, they just have to correct the fallout. heh. Thats kinda
>> all I've done for 4 months :-)   At least we'll have grasp of the
>> ramifications..
>
> And the last sentence is the key here.  We're trying to get to a point where
> we can make certain kinds of changes, then have the compiler spit out
> errors, fix the errors and have a high degree of confidence that the final
> change is correct and that we've found all the places that need to change.
>
> The change could be as simple as moving a function declaration to its
> natural place, collecting interfaces & data into classes, or something more
> ambitious like removing trees from the backend.  Folks will note that these
> are all refactorings that we don't want to change any observable behaviour.
>
>
>
>
>
>>>  * diff -c is somewhat unusual and I find diff -u much more readable.
>>
>>
>> unsual? I've been using -cp for the past 2 decades and no one has ever
>> mentioned it before...   poking around  the wiki I see it mentions you
>> can use either -up or -cp.
>>
>> I guess I could repackage things using -up...  I don't even know where
>> my script is to change it :-).   is -u what everyone uses now? no one
>> has mentioned it before that I am aware of.
>
> I'm probably the last person in the world that still generally prefers -cp
> :-)  I'm getting to the point where I can tolerate -u.

No, I prefer -cp too - diff just too easily makes a mess out of diffs with -u,
esp. if you have re-indenting going on as well.

Richard.

>
>>
>>
>>>  * Maybe the patches for reordering and removing should be split, also
>>>    for readability and for easier future identification of problems.
>>>
>> I was trying to avoid too much churn on 550ish files...  I didn't think
>> each one needed 2 sets of check-ins.    It could be done, but it will
>> take a while.  The reordering patch can be quickly generated, but the
>> reduction on all those files will take the better part of a week.
>>
>> My theory is it perfectly safe to back out any single file from the
>> patch set if we discover it has an issue and then examine what the root
>> of the problem is..
>>
>> tool patch coming shortly... probably tomorrow now.
>
> I haven't looked at the 3 patches in detail yet.  Given my familiarity with
> the overall process/goal, I can probably handle them as-is. They're just big
> mechanical changes.
>
> jeff

Re: [patch 0/3] Header file reduction.

[Michael Matz]

Hi,

On Wed, 7 Oct 2015, Richard Biener wrote:

> > I'm probably the last person in the world that still generally prefers 
> > -cp :-)  I'm getting to the point where I can tolerate -u.
> 
> No, I prefer -cp too - diff just too easily makes a mess out of diffs 
> with -u, esp. if you have re-indenting going on as well.

Actually -c was the recommended form of sending patches for many years 
even in our own guidelines.  It only got changed to -up or -cp when moving 
instructions from the texinfo files to the website in 2001.  From gcc 3.0 
(https://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_10.html):

  Use `diff -c' to make your diffs. Diffs without context are hard for us 
  to install reliably. More than that, they make it hard for us to study 
  the diffs to decide whether we want to install them. Unidiff format is 
  better than contextless diffs, but not as easy to read as `-c' format.
  If you have GNU diff, use `diff -cp', which shows the name of the 
  function that each change occurs in.


;-)  (IMHO it depends on what the patch does if -c or -u is better, if 
the _change_ is important -u might be better, if the new state is the 
more interesting thing, -c is)


Ciao,
Michael.

[patch 4/3] Header file reduction - Tools for contrib

[Andrew MacLeod]

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

Here's the patch to add all the tools to contrib/headers.

There are 9 tools I used over the run of the project.  They were 
developed in various stages and iterations, but I tried to at least have 
some common interface things, and I tried some cleaning up and 
documentation.  No commenting on the quality of python code... :-) I was 
learning python on the fly.    Im sure some things are QUITE awful.,

There is a readme file which gives a common use cases for each tool

Some of the tools are for analysis, aggregation, or flattening, some for 
visualization, and some are for the include reduction. I would have just 
filed them away somewhere, but  Jeff suggested I contribute them in case 
someone wants to do something with them down the road... which 
presumably also includes me :-)   Less chance of losing them this way.

They need more polishing, but I'm tired of looking at them. I will 
return to them down the road and see about cleaning them up a bit more.  
They still aren't perfect by any means, but should do their job safely. 
when used properly.   Comments in the code vary from good to absent, 
depending on how irritable I was at the time I was working on itl

I will soon also provide a modified config-list.mk which still works 
like the current one, but allows for easy overrides of certain things 
the include reducer requires..  until now I've just made a copy of 
config-list.mk and modified it for my own means.

The 2 tools for include reduction are  gcc-order-headers   and 
reduce-headers

what the process/conditions for checking things into contrib?  I've 
never had to do it before :-)

Andrew


[-- Attachment #2: contrib.patch.bz2 --]
[-- Type: application/x-bzip, Size: 18444 bytes --]

Re: [patch 4/3] Header file reduction - Tools for contrib

[Bernd Schmidt]

[-- Attachment #1: Type: text/plain, Size: 1928 bytes --]

On 10/05/2015 11:18 PM, Andrew MacLeod wrote:
> Here's the patch to add all the tools to contrib/headers.

Small patches should not be sent in compressed form, it makes reading 
and quoting them harder. This message is only intended to contain the 
patch in plain text so that I can quote it in further replies.

> There are 9 tools I used over the run of the project.  They were
> developed in various stages and iterations, but I tried to at least have
> some common interface things, and I tried some cleaning up and
> documentation.  No commenting on the quality of python code... :-) I was
> learning python on the fly.    Im sure some things are QUITE awful.,
>
> There is a readme file which gives a common use cases for each tool
>
> Some of the tools are for analysis, aggregation, or flattening, some for
> visualization, and some are for the include reduction. I would have just
> filed them away somewhere, but  Jeff suggested I contribute them in case
> someone wants to do something with them down the road... which
> presumably also includes me :-)   Less chance of losing them this way.
>
> They need more polishing, but I'm tired of looking at them. I will
> return to them down the road and see about cleaning them up a bit more.
> They still aren't perfect by any means, but should do their job safely.
> when used properly.   Comments in the code vary from good to absent,
> depending on how irritable I was at the time I was working on itl
>
> I will soon also provide a modified config-list.mk which still works
> like the current one, but allows for easy overrides of certain things
> the include reducer requires..  until now I've just made a copy of
> config-list.mk and modified it for my own means.
>
> The 2 tools for include reduction are  gcc-order-headers   and
> reduce-headers
>
> what the process/conditions for checking things into contrib?  I've
> never had to do it before :-)
>
> Andrew
>


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: contrib.patch --]
[-- Type: text/x-patch; name="contrib.patch", Size: 77046 bytes --]

Index: contrib/headers/ChangeLog
===================================================================
*** contrib/headers/ChangeLog	(revision 0)
--- contrib/headers/ChangeLog	(working copy)
***************
*** 0 ****
--- 1,12 ----
+ 2015-10-06  Andrew MacLeod  <amacleod@redhat.com>
+ 
+ 	* README : New File.
+ 	* count-headers : New File.
+ 	* gcc-order-headers : New File.
+ 	* graph-header-logs : New File.
+ 	* graph-include-web : New File.
+ 	* headerutils.py : New File.
+ 	* included-by : New File.
+ 	* reduce-headers : New File.
+ 	* replace-header : New File.
+ 	* show-headers : New File.
Index: contrib/headers/README
===================================================================
*** contrib/headers/README	(revision 0)
--- contrib/headers/README	(working copy)
***************
*** 0 ****
--- 1,282 ----
+ Quick start documentation for the header file utilities.  
+ 
+ This isn't a full breakdown of the tools, just they typical use scenarios.
+ 
+ - Each tool accepts -h to show its usage. usually no parameters will also
+ trigger the help message.  Help may specify additonal functionality to what is
+ listed here.
+ 
+ - For *all* tools, option format for specifying filenames must have no spaces
+ between the option and filename.
+ ie.:     tool -lfilename.h  target.h
+ 
+ - Many of the tools are required to be run from the core gcc source directory
+ containing coretypes.h  typically that is  in gcc/gcc from a source checkout.
+ For these tools to work on files not in this directory, their path needs to be
+ specified on the command line, 
+ ie.:     tool c/c-decl.c  lto/lto.c
+ 
+ - options can be intermixed with filenames anywhere on the command line
+ ie.   tool ssa.h rtl.h -a   is equivalent to 
+       tool ssa.h -a rtl.h
+ 
+ 
+ 
+ 
+ 
+ gcc-order-headers
+ -----------------
+   This will reorder any primary backend headers files into a canonical order
+   which will resolve any hidden dependencies they may have.  Any unknown
+   headers will simply be occur after the recognized core files, and retain the
+   same relative ordering they had.
+  
+   Must be run in the core gcc source directory
+ 
+   simply execute the command listing any files you wish to process on the
+   command line.
+ 
+   Any files which are changed are output, and the original is saved with a
+   .bak extention.
+ 
+   ex.:     gcc-order-headers tree-ssa.c c/c-decl.c
+ 
+   -s will list all of the known headers in their canonical order. It does not
+   show which of those headers include other headers, just the final canonical
+   ordering.
+ 
+   if any header files are included within a conditional code block, the tool
+   will issue a message and not change the file.  When this happens, you can
+   manually inspect the file, and if reorder it will be fine, rerun the command
+   with -i on the files.  This will ignore the conditional error condition
+   and perform the re-ordering anyway.
+   
+   If any #include line has the beginning of a multi-line comment, it will also
+   refuse to process the file until that is resolved. 
+  
+ 
+ 
+ 
+ show-headers
+ ------------
+   This will show the include structure for any given file. Each level of nesting
+   is indented, and when any duplicate headers are seen, they have their
+   duplicate number shown
+ 
+   -i may be used to specify alternate search directories for headers to parse.
+ 
+   Must be run in the core gcc source directory
+ 
+   ex.: show-headers -i../../build/gcc -i../libcpp tree-ssa.c
+ 	tree-ssa.c
+ 	  config.h
+ 	    auto-host.h
+ 	    ansidecl.h  (1)
+ 	  system.h
+ 	    safe-ctype.h
+ 	    filenames.h
+ 	      hashtab.h  (1)
+ 		ansidecl.h  (2)
+ 	    libiberty.h
+ 	      ansidecl.h  (3)
+ 	    hwint.h
+ 	  coretypes.h
+ 	    machmode.h  (1)
+ 	      insn-modes.h  (1)
+ 	    signop.h
+ 	  <...>
+ 
+ 
+ 
+ 
+ count-headers
+ -------------
+   simply count all the headers found in the specified files. A summary is 
+   printed showing occurrences from high to low.
+ 
+   ex.:    count-headers  tree*.c
+ 	    86 : coretypes.h
+ 	    86 : config.h
+ 	    86 : system.h
+ 	    86 : tree.h
+ 	    82 : backend.h
+ 	    80 : gimple.h
+ 	    72 : gimple-iterator.h
+ 	    70 : ssa.h
+ 	    68 : fold-const.h
+             <...>
+ 
+ 
+ 
+ included-by
+ -----------
+   This tool will search all the .c,.cc and .h files and output a list of files
+   which include the specified header(s).
+ 
+   It does a 4 level deep find of all source files from the current directory
+   and look in each of those for a #include of the specified headers.  So expect
+   a little bit of slowness.
+ 
+   -i limits the search to only other header files.
+   -c limits the search to .c and .cc files.
+   -a shows only source files which include *all* specified headers.
+   -f allows you to specify a file which contains a list of source files to
+      check rather than performing the much slower find command.
+ 
+   ex: included-by tree-vectorizer.h
+ 	config/aarch64/aarch64.c
+ 	config/i386/i386.c
+ 	config/rs6000/rs6000.c
+ 	tree-loop-distribution.c
+ 	tree-parloops.c
+ 	tree-ssa-loop-ivopts.c
+ 	tree-ssa-loop.c
+ 
+ 
+ 
+ 
+ replace-header
+ --------------
+   This tool simply replaces a single header file with one or more other headers.
+   -r specifies the include to replace, and one or more -f options specify the
+   replacement headers, in the order they occur.
+   
+   This is commonly used in conjunction with 'included-by' to change all 
+   occurrences of a header file to something else, or to insert new headers 
+   before or after.  
+ 
+   ex:  to insert #include "before.h" before every occurence of tree.h in all
+   .c and .cc source files:
+ 
+   replace-header -rtree.h -fbefore.h -ftree.h `included-by -c tree.h`
+ 
+ 
+ 
+ 
+ reduce-headers
+ --------------
+ 
+   This tool removes any header files which are not needed from a source file.
+ 
+   This tool must be run for the core gcc source directory, and requires either
+   a native build and sometimes target builds, depending on what you are trying
+   to reduce.
+ 
+   it is good practive to run 'gcc-order-headers' on a source file before trying
+   to reduce it.  This removes duplicates and performs some simplifications 
+   which reduce the chances of the reduction tool missing things.
+   
+   start with a completely bootstrapped native compiler.
+ 
+   Any desired target builds should be built in one directory using a modified
+   config-list.mk file which doesnt delete the build directory when its done.
+   any target directories which do not successfully complete a 'make all-gcc'
+   may cause the tool to not reduce anything.
+   (todo - provide a config-list.mk that leaves successful target builds, but
+           deletes ones which do not compile)
+ 
+   The tool will examine all the target builds to determine which targets build
+   the file, and include those targets in the testing.
+   
+ 
+ 
+   The tool will analyze a source file and attempt to remove each non-conditional
+   header from last to first in the file.:
+     It will first attempt to build the native all-gcc target.
+     If that succeeds, it will attempt to build any target build .o files
+     If that suceeds, it will check to see if there are any conditional
+        compilation dependencies between this header file and the source file or
+        any header whihch have already been determined as non-removable.
+     If all these tests are passed, the header file is determined to be removable
+        and is removed from the source file.
+     This continues until all headers have been checked.
+   At this point, the a bootstrap is attempted in the native build, and if that
+      passes the file is considered reduced.
+ 
+   Any files from the config subdirectory require target builds to be present
+   in order to proceed.
+ 
+   A small subset of targets has been determined to provide excellent coverage,
+   at least as of Aug 31/15 .  A fullset of targets reduced all of the files
+   making up libbackend.a.  All of the features which requires target testing 
+   were found to be triggered by one or more of these targets.  They are
+   actually known to the tool, and when checkiong target, it will check those
+   targets first, then the rest.  It is mostly safe to do a reduction with just
+   these targets, at least until some new whacky target comes along.
+   building config-list.mk with :
+   LIST="aarch64-linux-gnu arm-netbsdelf avr-rtems c6x-elf epiphany-elf hppa2.0-hpux10.1 i686-mingw32crt i686-pc-msdosdjgpp mipsel-elf powerpc-eabisimaltivec rs6000-ibm-aix5.1.0 sh-superh-elf sparc64-elf spu-elf"
+ 
+   -b specifies the native bootstrapped build root directory
+   -t specifies a target build root directory that config-list.mk was run from
+   -f is used to limit the headers for consideration.
+ 
+   example:
+ 
+   mkdir gcc          // checkout gcc in subdir gcc
+   mdsir build        // boostrap gcc in subdir build
+   mkdir target       // create target directory and run config-list.mk
+   cd gcc/gcc
+ 
+   reduce-headers -b../../build -t../../targets -falias.h -fexpr.h tree*.c  (1)
+        #  This will attempt to remove only alias.h and expr.h from tree*.c
+ 
+   reduce-headers -b../../build -t../../targets tree-ssa-live.c
+        #  This will attempt to remove all header files from tree-ssa-live.c
+   
+ 
+   the tool will generate a number of log files:
+ 
+     reduce-headers.log :  All the compilation failure output that tool tried.
+     reduce-headers.sum : One line summary of what happened to each source file.
+ 
+   (All the remaining logs are appended to, so if you run the tool multiple times
+   these files are just added to. You must physically remove them yourself.)
+ 
+     reduce-headers-kept.log: List of all the successful compiles that were
+                              ignored because of conditional macro dependencies
+ 			     and why it thinks taht is the case
+     $src.c.log  : for each failed header removal, the compilation
+ 		  messages as to why it failed.
+     $header.h.log: The same log is put into the relevent header log as well.
+ 
+ 
+ a sample output from ira.c.log:
+ 
+ Compilation failed:
+  for shrink-wrap.h:
+ 
+  ============================================
+  /gcc/2015-09-09/gcc/gcc/ira.c: In function ‘bool split_live_ranges_for_shrink_wrap()’:
+  /gcc/2015-09-09/gcc/gcc/ira.c:4839:8: error: ‘SHRINK_WRAPPING_ENABLED’ was not declared in this scope
+     if (!SHRINK_WRAPPING_ENABLED)
+             ^
+ 	    make: *** [ira.o] Error 1
+ 
+ 
+ the same message would be put into shrink-wrap.h.log.
+ 
+ 
+ 
+ graph-header-logs
+ -----------------
+   This tool will parse all the messages from the .C files, looking for failures
+   that show up in other headers...  meaning there is a compilation dependency
+   between the 2 header files. 
+ 
+   The tool will aggregate all these and generate a graph of the dependencies
+   exposed during compilation.  red lines indicate dependecies that are
+   presednt because a head file physically includes another header. Black lines
+   represent data dependencies causing compilation if the header isnt present.
+ 
+   ex.: graph-header-logs *.c.log
+ 
+ 
+ 
+ graph-include-web
+ -----------------
+   This tool can be used to visualize the include structure in files.  It is
+   rapidly turned useless if you specify too many things, but it can be 
+   useful for finding cycles and redundancies, or simply to see what a single
+   file looks like.
+ 
+   ex.: graph-include-web tree.c
Index: contrib/headers/count-headers
===================================================================
*** contrib/headers/count-headers	(revision 0)
--- contrib/headers/count-headers	(working copy)
***************
*** 0 ****
--- 1,63 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ 
+ 
+ usage = False
+ src = list()
+ flist = { }
+ process_h = True
+ process_c = True
+ verbose = False
+ all_inc = True
+ level = 0
+ 
+ only_use_list = list()
+ 
+ 
+ 
+ for x in sys.argv[1:]:
+   if x[0:2] == "-h":
+     usage = True
+   else:
+     src.append (x)
+ 
+ 
+ if not usage and len(src) > 0:
+ 
+   incl = { }
+   for fn in src:
+     src = readwholefile (fn)
+     dup = { }
+     for line in src:
+       d = find_pound_include (line, True, True)
+       if d != "" and d[-2:] ==".h":
+         if dup.get(d) == None:
+ 	  if incl.get(d) == None:
+ 	    incl[d] = 1
+ 	  else:
+ 	    incl[d] = incl[d]+ 1
+ 	  dup[d] = 1
+ 
+   l = list()
+   for i in incl:
+     l.append ((incl[i], i))
+   l.sort(key=lambda tup:tup[0], reverse=True)
+ 
+   for f in l:
+     print str(f[0]) + " : " + f[1]
+ 
+ else:
+   print "count-headers file1 [filen]"
+   print "Count the number of occurrences of all includes across all listed files"
+ 
+  
+ 
+ 
+ 
+ 

Property changes on: contrib/headers/count-headers
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: contrib/headers/gcc-order-headers
===================================================================
*** contrib/headers/gcc-order-headers	(revision 0)
--- contrib/headers/gcc-order-headers	(working copy)
***************
*** 0 ****
--- 1,366 ----
+ #! /usr/bin/python2
+ import os
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ import Queue
+ 
+ file_list = list ()
+ usage = False
+ 
+ ignore_conditional = False
+ 
+ order = [
+   "system.h",
+   "coretypes.h",
+   "backend.h",
+   "target.h",
+   "rtl.h",
+   "c-family/c-target.h",
+   "c-family/c-target-def.h",
+   "tree.h",
+   "cp/cp-tree.h",
+   "c-family/c-common.h",  # these must come before diagnostic.h
+   "c/c-tree.h",
+   "fortran/gfortran.h",
+   "gimple.h",
+   "cfghooks.h",
+   "df.h",
+   "tm_p.h",
+   "gimple-iterators.h",
+   "ssa.h",
+   "expmed.h",
+   "optabs.h",
+   "regs.h",
+   "ira.h",
+   "ira-int.h",
+   "gimple-streamer.h"
+ 
+ ]
+ 
+ exclude_special = [  "bversion.h", "obstack.h", "insn-codes.h", "hooks.h" ]
+ includes = { }
+ dups = { }
+ exclude_processing = [ "tree-vectorizer.h" , "c-target.h", "c-target-def.h", "cp-tree.h", "c-common.h", "c-tree.h", "gfortran.h" ]
+ 
+ master_list = list()
+ # where include file comes from in src
+ h_from = { }
+ 
+ # create the master ordering list... this is the desired order of headers
+ def create_master_list (fn, verbose):
+   if fn not in exclude_processing:
+     for x in includes[fn][1]:
+       create_master_list (x, verbose)
+   if not fn in master_list:
+     # Don't put diagnostic*.h into the ordering list, its special since
+     # various front ends have to set GCC_DIAG_STYLE before including it
+     # for each file, we'll tailor where it belongs by looking at the dup
+     # list and seeing which file is included, and position it appropriately.
+     if fn != "diagnostic.h" and fn != "diagnostic-core.h":
+       master_list.append (fn)
+       if (verbose):
+ 	print fn + "      included by: " + includes[fn][0]
+ 
+ 
+ 
+ def print_dups ():
+   if dups:
+     print "\nduplicated includes"
+   for i in dups:
+     string =  "dup : " + i + " : "
+     string += includes[i][0] 
+     for i2 in dups[i]:
+       string += ", "+i2
+     print string
+ 
+ 
+ def process_known_dups ():
+   # rtl.h gets tagged as a duplicate includer for all of coretypes, but thats
+   # really for only generator files
+   rtl_remove = includes["coretypes.h"][1] + ["statistics.h", "vec.h"]
+   for i in rtl_remove:
+     if dups[i] and "rtl.h" in dups[i]:
+       dups[i].remove("rtl.h")
+     if not dups[i]:
+       dups.pop (i, None)
+ 
+   # make sure diagnostic.h is the owner of diagnostic-core.h
+   if includes["diagnostic-core.h"][0] != "diagnostic.h":
+     dups["diagnostic-core.h"].append (includes["diagnostic-core.h"][0])
+     includes["diagnostic-core.h"] = ("diagnostic.h", includes["diagnostic-core.h"][1])
+ 
+ def indirectly_included (header, header_list):
+   nm = os.path.basename (header)
+   while nm and includes.get(nm):
+     if includes[nm][0] in header_list:
+       return includes[nm][0]
+     nm = includes[nm][0]
+ 
+   if header == "diagnostic-core.h":
+     if dups.get("diagnostic-core.h"):
+       for f in dups["diagnostic-core.h"]:
+ 	if f in header_list:
+ 	  return f
+     else:
+       if header in header_list:
+ 	return header
+     # Now check if diagnostics is included indirectly anywhere
+     header = "diagnostic.h"
+ 
+   if header == "diagnostic.h":
+     if dups.get("diagnostic.h"):
+       for f in dups["diagnostic.h"]:
+ 	if f in header_list:
+ 	  return f
+     else:
+       if header in header_list:
+ 	return header 
+ 
+   return ""
+ 
+ 
+ def get_new_order (src_h, desired_order):
+   new_order = list()
+   for h in desired_order:
+     if h in master_list:
+       # find what included this
+       iclist = list()
+       ib = includes[h][0]
+       while ib:
+         iclist.insert(0, ib)
+ 	ib = includes[ib][0]
+       if iclist:
+ 	for x in iclist:
+ 	  if x in src_h and x not in exclude_processing:
+ 	    if x not in new_order and x[:10] != "diagnostic" and h not in exclude_special:
+ 	      new_order.append (x)
+ 	      break;
+       else:
+ 	if h not in new_order:
+ 	  new_order.append (h)
+ 
+   f = ""
+   if "diagnostic.h" in src_h:
+     f = "diagnostic.h"
+   elif "diagnostic-core.h" in src_h:
+     f = "diagnostic-core.h"
+ 
+  
+   if f:
+     ii = indirectly_included (f, src_h)
+     if not ii or ii == f:
+       new_order.append (f)
+ 
+   return new_order
+         
+     
+ 
+ # stack of files to process
+ process_stack = list()
+ 
+ def process_one (info):
+   i = info[0]
+   owner = info[1]
+   name = os.path.basename(i)
+   if os.path.exists (i):
+     if includes.get(name) == None:
+       l = find_unique_include_list (i)
+       # create a list which has just basenames in it
+       new_list = list()
+       for x in l:
+ 	new_list.append (os.path.basename (x))
+ 	process_stack.append((x, name))
+       includes[name] = (owner, new_list)
+     elif owner:
+       if dups.get(name) == None:
+         dups[name] = [ owner ]
+       else:
+         dups[name].append (owner)
+   else:
+     # seed tm.h with options.h since its a build file and won't be seen. 
+     if not includes.get(name):
+       if name == "tm.h":
+ 	includes[name] = (owner, [ "options.h" ])
+ 	includes["options.h"] = ("tm.h", list())
+       else:
+ 	includes[name] = (owner, list())
+ 
+ 
+ show_master = False
+ 
+ for arg in sys.argv[1:]:
+   if arg[0:1] == "-":
+     if arg[0:2] == "-h":
+       usage = True
+     elif arg[0:2] == "-i":
+       ignore_conditional = True
+     elif arg[0:2] == "-s":
+       show_master = True
+     else:
+       print "Error: unrecognized option " + arg
+   elif os.path.exists(arg):
+     file_list.append (arg)
+   else:
+     print "Error: file " + arg + " Does not exist."
+     usage = True
+ 
+ if not file_list and not show_master:
+   usage = True
+ 
+ if not usage and not os.path.exists ("coretypes.h"):
+   usage = True
+   print "Error: Must run command in main gcc source directory containing coretypes.h\n"
+ 
+ # process diagnostic.h first.. it's special since GCC_DIAG_STYLE can be
+ # overridden by languages, but must be done so by a file included BEFORE it.
+ # so make sure it isn't seen as inclujded by one of those files by making it 
+ # appear to be included by the src file.
+ process_stack.insert (0, ("diagnostic.h", ""))
+ 
+ # Add the list of files in reverse order since it is processed as a stack later
+ for i in order:
+   process_stack.insert (0, (i, "") )
+ 
+ # build up the library of what header files include what other files.
+ while process_stack:
+   info = process_stack.pop ()
+   process_one (info)
+ 
+ # Now crate the master ordering list
+ for i in order:
+   create_master_list (os.path.basename (i), False)
+ 
+ # handle warts in the duplicate list
+ process_known_dups ()
+ desired_order = master_list
+ 
+ if show_master:
+   print " Canonical order of gcc include files: "
+   for x in master_list:
+     print x
+   print " "
+ 
+ if usage:
+   print "gcc-order-headers [-i] [-s] file1 [filen]"
+   print "    Ensures gcc's headers files are included in a normalized form with"
+   print "    redundant headers removed.  The original files are saved in filename.bak"
+   print "    Outputs a list of files which changed."
+   print " -i ignore conditional compilation."
+   print "    Use after examining the file to be sure includes within #ifs are safe"
+   print "    Any headers within conditional sections will be ignored."
+   print " -s Show the cananoical order of known includes"
+   sys.exit(0)
+ 
+ 
+ didnt_do = list ()
+ 
+ for fn in file_list:
+   nest = 0
+   src_h = list()
+   src_line = { }
+ 
+   master_list = list()
+   includes = { }
+   dups = { }
+ 
+   iinfo = process_ii_src (fn)
+   src = ii_src (iinfo)
+   include_list = ii_include_list (iinfo)
+ 
+   if ii_include_list_cond (iinfo):
+     if not ignore_conditional:
+       print fn + ": Cannot process due to conditional compilation of includes"
+       didnt_do.append (fn)
+       src = list ()
+ 
+   if not src:
+     continue
+ 
+   process_stack = list()
+   # prime the stack with headers in the main ordering list so we get them in
+   # this order.
+   for d in order:
+     if d in include_list:
+       process_stack.insert (0, (d, ""))
+ 
+   for d in include_list:
+       nm = os.path.basename(d)
+       src_h.append (nm)
+       iname = d
+       iname2 = os.path.dirname (fn) + "/" + d
+       if not os.path.exists (d) and os.path.exists (iname2):
+         iname = iname2
+       if iname not in process_stack:
+ 	process_stack.insert (0, (iname, ""))
+       src_line[nm] = ii_src_line(iinfo)[d]
+       if src_line[nm].find("/*") != -1 and src_line[nm].find("*/") == -1:
+         # this means we have a multi line comment, abort!'
+ 	print fn + ": Cannot process due to a multi-line comment :"
+ 	print "        " + src_line[nm]
+ 	if fn not in didnt_do:
+ 	  didnt_do.append (fn)
+ 	src = list ()
+ 
+   if not src:
+     continue
+ 
+   # Now create the list of includes as seen by the source file.
+   while process_stack:
+     info = process_stack.pop ()
+     process_one (info)
+  
+   for i in include_list:
+     create_master_list (os.path.basename (i), False)
+ 
+   new_src = list()
+   header_added = list()
+   new_order = list()
+   for line in src:
+     d = find_pound_include (line, True, True)
+     if not d or d[-2:] != ".h":
+       new_src.append (line)
+     else:
+       if d == order[0] and not new_order:
+         new_order = get_new_order (src_h, desired_order)
+ 	for i in new_order:
+ 	  new_src.append (src_line[i])
+ 	  # if not seen, add it.
+ 	  if i not in header_added:
+ 	    header_added.append (i)
+       else:
+ 	nm = os.path.basename(d)
+ 	if nm not in header_added:
+ 	  iby = indirectly_included (nm, src_h)
+ 	  if not iby:
+ 	    new_src.append (line)
+ 	    header_added.append (nm)
+ 
+   if src != new_src:
+     os.rename (fn, fn + ".bak")
+     fl = open(fn,"w")
+     for line in new_src:
+       fl.write (line)
+     fl.close ()
+     print fn 
+ 
+  
+ if didnt_do:
+   print "\n\n Did not process the following files due to conditional dependencies:"
+   str = ""
+   for x in didnt_do:
+     str += x + " "
+   print str
+   print "\n"
+   print "Please examine to see if they are safe to process, and re-try with -i. "
+   print "Safeness is determined by checking whether any of the reordered headers are"
+   print "within a conditional and could be hauled out of the conditional, thus changing"
+   print "what the compiler will see."
+   print "Multi-line comments after a #include can also cause failuer, they must be turned"
+   print "into single line comments or removed."
+ 
+ 
+ 
+ 

Property changes on: contrib/headers/gcc-order-headers
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: contrib/headers/graph-header-logs
===================================================================
*** contrib/headers/graph-header-logs	(revision 0)
--- contrib/headers/graph-header-logs	(working copy)
***************
*** 0 ****
--- 1,226 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ header_roots = { }
+ extra_edges = list()
+ verbose = False
+ verbosity = 0
+ nodes = list()
+ 
+ def unpretty (name):
+   if name[-2:] == "_h":
+     name = name[:-2] + ".h"
+   return name.replace("_", "-")
+ 
+ def pretty_name (name):
+   name = os.path.basename (name)
+   return name.replace(".","_").replace("-","_").replace("/","_").replace("+","_");
+ 
+ depstring = ("In file included from", "                 from")
+ 
+ ignore = [ "coretypes_h",
+ 	     "machmode_h",
+ 	     "signop_h",
+ 	     "wide_int_h",
+ 	     "double_int_h",
+ 	     "real_h",
+ 	     "fixed_value_h",
+ 	     "hash_table_h",
+ 	       "statistics_h",
+ 	       "ggc_h",
+ 	       "vec_h",
+ 	       "hashtab_h",
+ 	       "inchash_h",
+ 	       "mem_stats_traits_h",
+ 	       "hash_map_traits_h",
+ 	       "mem_stats_h",
+ 	       "hash_map_h",
+ 	     "hash_set_h",
+ 	     "input_h",
+ 	       "line_map_h",
+ 	     "is_a_h",
+ 	   "system_h",
+ 	   "config_h" ]
+ 
+ def process_log_file (header, logfile):
+   if header_roots.get (header) != None:
+     print "Error: already processed log file: " + header + ".log"
+     return
+   hname = pretty_name (header)
+   header_roots[hname] = { }
+   
+   sline = list();
+   incfrom = list()
+   newinc = True
+   for line in logfile:
+     if len (line) > 21 and line[:21] in depstring:
+       if newinc:
+         incfrom = list()
+ 	newinc = False
+       fn = re.findall(ur".*/(.*?):", line)
+       if len(fn) != 1:
+         continue
+       if fn[0][-2:] != ".h":
+         continue
+       n = pretty_name (fn[0])
+       if n not in ignore:
+ 	incfrom.append (n)
+       continue
+     newinc = True
+     note = re.findall (ur"^.*note: (.*)", line)
+     if len(note) > 0:
+       sline.append (("note", note[0]))
+     else:
+       err_msg = re.findall (ur"^.*: error: (.*)", line)
+       if len(err_msg) == 1:
+ 	msg = err_msg[0]
+ 	if (len (re.findall("error: forward declaration", line))) != 0:
+ 	  continue
+ 	path = re.findall (ur"^(.*?):.*error: ", line)
+ 	if len(path) != 1:
+ 	  continue
+ 	if path[0][-2:] != ".h":
+ 	  continue
+ 	fname = pretty_name (path[0])
+ 	if fname in ignore or fname[0:3] == "gt_":
+ 	  continue
+ 	sline.append (("error", msg, fname, incfrom))
+ 
+   print str(len(sline)) + " lines to process"
+   lastline = "note"
+   for line in sline:
+     if line[0] != "note" and lastline[0] == "error":
+       fname = lastline[2]
+       msg = lastline[1]
+       incfrom = lastline[3]
+       string = ""
+       ofname = fname
+       if len(incfrom) != 0:
+ 	for t in incfrom:
+ 	  string = string + t + " : "
+ 	  ee = (fname, t)
+ 	  if ee not in extra_edges:
+ 	    extra_edges.append (ee)
+ 	  fname = t
+ 	  print string
+ 
+       if hname not in nodes:
+ 	nodes.append(hname)
+       if fname not in nodes:
+ 	nodes.append (ofname)
+       for y in incfrom:
+ 	if y not in nodes:
+ 	  nodes.append (y)
+ 
+ 
+       if header_roots[hname].get(fname) == None:
+ 	header_roots[hname][fname] = list()
+       if msg not in header_roots[hname][fname]:
+ 	print string + ofname + " : " +msg
+ 	header_roots[hname][fname].append (msg)
+     lastline = line;
+ 
+ 
+ dotname = "graph.dot"
+ graphname = "graph.png"
+ 
+ 
+ def build_dot_file (file_list):
+   output = open(dotname, "w")
+   output.write ("digraph incweb {\n");
+   for x in file_list:
+     if os.path.exists (x) and x[-4:] == ".log":
+       header =  x[:-4]
+       logfile = open(x).read().splitlines()
+       process_log_file (header, logfile)
+     elif os.path.exists (x + ".log"):
+       logfile = open(x + ".log").read().splitlines()
+       process_log_file (x, logfile)
+ 
+   for n in nodes:
+     fn = unpretty(n)
+     label = n + " [ label = \"" + fn  + "\" ];"
+     output.write (label + "\n")
+     if os.path.exists (fn):
+       h = open(fn).read().splitlines()
+       for l in h:
+         t = find_pound_include (l, True, False)
+ 	if t != "":
+ 	  t = pretty_name (t)
+ 	  if t in ignore or t[-2:] != "_h":
+ 	    continue
+ 	  if t not in nodes:
+ 	    nodes.append (t)
+ 	  ee = (t, n)
+ 	  if ee not in extra_edges:
+ 	    extra_edges.append (ee)
+ 
+   depcount = list()
+   for h in header_roots:
+     for dep in header_roots[h]:
+       label = " [ label = "+ str(len(header_roots[h][dep])) + " ];"
+       string = h + " -> " + dep + label
+       output.write (string + "\n");
+       if verbose:
+         depcount.append ((h, dep, len(header_roots[h][dep])))
+ 
+   for ee in extra_edges:
+     string = ee[0] + " -> " + ee[1] + "[ color=red ];"
+     output.write (string + "\n");
+ 
+   
+   if verbose:
+     depcount.sort(key=lambda tup:tup[2])
+     for x in depcount:
+       print " ("+str(x[2])+ ") : " + x[0] + " -> " + x[1]
+       if (x[2] <= verbosity):
+         for l in header_roots[x[0]][x[1]]:
+ 	  print "            " + l
+ 
+   output.write ("}\n");
+ 
+ 
+ files = list()
+ dohelp = False
+ edge_thresh = 0
+ for arg in sys.argv[1:]:
+   if arg[0:2] == "-o":
+     dotname = arg[2:]+".dot"
+     graphname = arg[2:]+".png"
+   elif arg[0:2] == "-h":
+     dohelp = True
+   elif arg[0:2] == "-v":
+     verbose = True
+     if len(arg) > 2:
+       verbosity = int (arg[2:])
+       if (verbosity == 9):
+         verbosity = 9999
+   elif arg[0:1] == "-":
+     print "Unrecognized option " + arg
+     dohelp = True
+   else:
+     files.append (arg)
+     
+ if len(sys.argv) == 1:
+   dohelp = True
+ 
+ if dohelp:
+   print "Parses the log files from remove-include processes to generate"
+   print " dependency graphs for the include web for specified files."
+   print "Usage:  [-nnum] [-h] [-v[n]] [-ooutput] file1 [[file2] ... [filen]]"
+   print "	-ooutput : Specifies output to output.dot and output.png"
+   print "                   Defaults to 'graph.dot and graph.png"
+   print "       -vn : verbose mode, shows the number of connections, and if n"
+   print "             is specifies, show the messages if # < n. 9 is infinity"
+   print "	-h : help"
+ else:
+   print files
+   build_dot_file (files)
+   os.system ("dot -Tpng " + dotname + " -o" + graphname)
+ 
+ 

Property changes on: contrib/headers/graph-header-logs
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: contrib/headers/graph-include-web
===================================================================
*** contrib/headers/graph-include-web	(revision 0)
--- contrib/headers/graph-include-web	(working copy)
***************
*** 0 ****
--- 1,122 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ def pretty_name (name):
+   return name.replace(".","_").replace("-","_").replace("/","_").replace("+","_");
+ 
+ 
+ include_files = list()
+ edges = 0
+ one_c = False
+ clink = list()
+ noterm = False
+ 
+ def build_inclist (output, filen):
+   global edges
+   global one_c
+   global clink
+   global noterm
+   inc = build_include_list (filen)
+   if one_c and filen[-2:] == ".c":
+     pn = "all_c"
+   else:
+     pn = pretty_name(filen)
+   for nm in inc:
+     if pn == "all_c":
+       if nm not in clink:
+         if len(build_include_list(nm)) != 0 or not noterm:
+ 	  output.write (pretty_name(nm) + " -> " + pn + ";\n")
+ 	  edges = edges + 1
+ 	  if nm not in include_files:
+ 	    include_files.append(nm)
+ 	clink.append (nm)
+     else:
+       output.write (pretty_name(nm) + " -> " + pn + ";\n")
+       edges = edges + 1
+       if nm not in include_files:
+ 	include_files.append(nm)
+   return len(inc) == 0
+ 
+ dotname = "graph.dot"
+ graphname = "graph.png"
+ 
+ def build_dot_file (file_list):
+   global one_c
+   output = open(dotname, "w")
+   output.write ("digraph incweb {\n");
+   if one_c:
+     output.write ("all_c [shape=box];\n");
+   for x in file_list:
+     if x[-2:] == ".h":
+       include_files.append (x)
+     elif os.path.exists (x):
+       build_inclist (output, x)
+       if not one_c:
+ 	output.write (pretty_name (x) + "[shape=box];\n")
+ 
+   for x in include_files:
+     term = build_inclist (output, x)
+     if term:
+       output.write (pretty_name(x) + " [style=filled];\n")
+ 
+   output.write ("}\n");
+ 
+ 
+ files = list()
+ dohelp = False
+ edge_thresh = 0
+ for arg in sys.argv[1:]:
+   if arg[0:2] == "-o":
+     dotname = arg[2:]+".dot"
+     graphname = arg[2:]+".png"
+   elif arg[0:2] == "-h":
+     dohelp = True
+   elif arg[0:2] == "-a":
+     one_c = True
+     if arg[0:3] == "-at":
+       noterm = True
+   elif arg[0:2] == "-f":
+     if not os.path.exists (arg[2:]):
+       print "Option " + arg +" doesn't specify a proper file"
+       dohelp = True
+     else:
+       sfile = open (arg[2:], "r")
+       srcdata = sfile.readlines()
+       sfile.close()
+       for x in srcdata:
+ 	files.append(x.rstrip())
+   elif arg[0:2] == "-n":
+     edge_thresh = int (arg[2:])
+   elif arg[0:1] == "-":
+     print "Unrecognized option " + arg
+     dohelp = True
+   else:
+     files.append (arg)
+     
+ if len(sys.argv) == 1:
+   dohelp = True
+ 
+ if dohelp:
+   print "Generates a graph of the include web for specified files."
+   print "Usage:  [-finput_file] [-h] [-ooutput] [file1 ... [filen]]"
+   print "	-finput_file : Input file is file containing a list of files"
+   print "	-ooutput : Specifies output to output.dot and output.png"
+   print "                  defaults to graph.dot and graph.png"
+   print "	-nnum : specifies the # of edges beyond which sfdp is invoked. def=0"
+   print "       -a : Aggregate all .c files to 1 file.  Shows only include web."
+   print "       -at :  Aggregate, but don't include terminal.h to .c links. "
+   print "	-h : help"
+ else:
+   print files
+   build_dot_file (files)
+   if edges > edge_thresh:
+     os.system ("sfdp -Tpng " + dotname + " -o" + graphname)
+   else:
+     os.system ("dot -Tpng " + dotname + " -o" + graphname)
+ 
+ 

Property changes on: contrib/headers/graph-include-web
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: contrib/headers/headerutils.py
===================================================================
*** contrib/headers/headerutils.py	(revision 0)
--- contrib/headers/headerutils.py	(working copy)
***************
*** 0 ****
--- 1,500 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ import subprocess
+ import shutil
+ import pickle
+ 
+ import multiprocessing 
+ 
+ def find_pound_include (line, use_outside, use_slash):
+   inc = re.findall (ur"^\s*#\s*include\s*\"(.+?)\"", line)
+   if len(inc) == 1:
+     nm = inc[0]
+     if use_outside or os.path.exists (nm):
+       if use_slash or '/' not in nm:
+ 	return nm
+   return ""
+ 
+ def find_system_include (line):
+   inc = re.findall (ur"^\s*#\s*include\s*<(.+?)>", line)
+   if len(inc) == 1:
+     return inc[0]
+   return ""
+   
+ def find_pound_define (line):
+   inc = re.findall (ur"^\s*#\s*define ([A-Za-z0-9_]+)", line)
+   if len(inc) != 0:
+     if len(inc) > 1:
+       print "What? more than 1 match in #define??"
+       print inc
+       sys.exit(5)
+     return inc[0];
+   return ""
+ 
+ def is_pound_if (line):
+   inc = re.findall ("^\s*#\s*if\s", line)
+   if not inc:
+     inc = re.findall ("^\s*#\s*if[n]?def\s", line)
+   if inc:
+     return True
+   return False
+ 
+ def is_pound_endif (line):
+   inc = re.findall ("^\s*#\s*endif", line)
+   if inc:
+     return True
+   return False
+ 
+ def find_pound_if (line):
+   inc = re.findall (ur"^\s*#\s*if\s+(.*)", line)
+   if len(inc) == 0:
+     inc = re.findall (ur"^\s*#\s*elif\s+(.*)", line)
+   if len(inc) > 0:
+ #    inc2 = re.findall (ur"defined *\((.+?)\)", inc[0])
+     inc2 = re.findall (ur"defined\s*\((.+?)\)", inc[0])
+     inc3 = re.findall (ur"defined\s+([a-zA-Z0-9_]+)", inc[0])
+     for yy in inc3:
+       inc2.append (yy)
+     return inc2
+   else:
+     inc = re.findall (ur"^\s*#\s*ifdef\s(.*)", line)
+     if len(inc) == 0:
+       inc = re.findall (ur"^\s*#\s*ifndef\s(.*)", line)
+     if len(inc) > 0:
+       inc2 = re.findall ("[A-Za-z_][A-Za-z_0-9]*", inc[0])
+       return inc2
+   if len(inc) == 0:
+     return list ()
+   print "WTF. more than one line returned for find_pound_if"
+   print inc
+   sys.exit(5)
+ 
+ empty_iinfo =  ("", "", list(), list(), list(), list(), list())
+ 
+ # find all relevant include data. 
+ def process_include_info (filen, do_macros, keep_src):
+   header = False
+   if not os.path.exists (filen):
+     return empty_iinfo
+ 
+   sfile = open (filen, "r");
+   data = sfile.readlines()
+   sfile.close()
+ 
+   # Ignore the initial #ifdef HEADER_H in header files
+   if filen[-2:] == ".h":
+     nest = -1
+     header = True
+   else:
+     nest = 0
+ 
+   macout = list ()
+   macin = list()
+   incl = list()
+   cond_incl = list()
+   src_line = { }
+   guard = ""
+ 
+   for line in (data):
+     if is_pound_if (line):
+       nest += 1
+     elif is_pound_endif (line):
+       nest -= 1
+ 
+     nm = find_pound_include (line, True, True)
+     if nm != "" and nm not in incl and nm[-2:] == ".h":
+       incl.append (nm)
+       if nest > 0:
+         cond_incl.append (nm)
+       if keep_src:
+         src_line[nm] = line
+       continue
+ 
+     if do_macros:
+       d = find_pound_define (line)
+       if d:
+         if d not in macout:
+ 	  macout.append (d);
+ 	  continue
+ 
+       d = find_pound_if (line)
+       if d:
+         # The first #if in a header file should be the guard
+         if header and len (d) == 1 and guard == "":
+ 	  if d[0][-2:] == "_H":
+ 	    guard = d
+ 	  else:
+ 	    guard = "Guess there was no guard..."
+ 	else:
+ 	  for mac in d:
+ 	    if mac != "defined" and mac not in macin:
+ 	      macin.append (mac);
+ 
+   if not keep_src:
+     data = list()
+ 
+   return (os.path.basename (filen), os.path.dirname (filen), incl, cond_incl,
+ 	  macin, macout, data, src_line)
+ 
+ def process_ii (filen):
+   return process_include_info (filen, False, False)
+ 
+ def process_ii_macro (filen):
+   return process_include_info (filen, True, False)
+ 
+ def process_ii_src (filen):
+   return process_include_info (filen, False, True)
+ 
+ def process_ii_macro_src (filen):
+   return process_include_info (filen, True, True)
+ 
+ def ii_base (iinfo):
+   return iinfo[0]
+ 
+ def ii_path (iinfo):
+   return iinfo[1]
+ 
+ def ii_include_list (iinfo):
+   return iinfo[2]
+ 
+ def ii_include_list_cond (iinfo):
+   return iinfo[3]
+ 
+ def ii_include_list_non_cond (iinfo):
+   l = ii_include_list (iinfo)
+   for n in ii_include_list_cond (iinfo):
+     l.remove (n)
+   return l
+ 
+ def ii_macro_consume (iinfo):
+   return iinfo[4]
+   
+ def ii_macro_define (iinfo):
+   return iinfo[5]
+ 
+ def ii_src (iinfo):
+   return iinfo[6]
+ 
+ def ii_src_line (iinfo):
+   return iinfo[7]
+ 
+ def ii_read (fname):
+   f = open (fname, 'rb')
+   incl = pickle.load (f)
+   consumes = pickle.load (f)
+   defines = pickle.load (f)
+   obj = (fname,fname,incl,list(), list(), consumes, defines, list(), list())
+   return obj
+ 
+ def ii_write (fname, obj):
+   f = open (fname, 'wb')
+   pickle.dump (obj[2], f)
+   pickle.dump (obj[4], f)
+   pickle.dump (obj[5], f)
+   f.close ()
+ 
+ 
+ # Find files matching pattern NAME, return in a list.
+ # CURRENT is True if you want to include the current directory
+ # DEEPER is True if you want to search 3 levels below the current directory
+ # any files with testsuite diurectories are ignored
+ 
+ def find_gcc_files (name, current, deeper):
+   files = list()
+   command = ""
+   if current:
+     if not deeper:
+       command = "find -maxdepth 1 -name " + name + " -not -path \"./testsuite/*\""
+     else:
+       command = "find -maxdepth 4 -name " + name + " -not -path \"./testsuite/*\""
+   else:
+     if deeper:
+       command = "find -maxdepth 4 -mindepth 2 -name " + name + " -not -path \"./testsuite/*\""
+ 
+   if command != "":
+     f = os.popen (command)
+     for x in f:
+       if x[0] == ".":
+         fn = x.rstrip()[2:]
+       else:
+ 	fn = x
+       files.append(fn)
+ 
+   return files
+ 
+ # find the list of unique include names found in a file.
+ def find_unique_include_list_src (data):
+   found = list ()
+   for line in data:
+     d = find_pound_include (line, True, True)
+     if d and d not in found and d[-2:] == ".h":
+       found.append (d)
+   return found
+ 
+ # find the list of unique include names found in a file.
+ def find_unique_include_list (filen):
+   data = open (filen).read().splitlines()
+   return find_unique_include_list_src (data)
+ 
+ 
+ # Create the macin, macout, and incl vectors for a file FILEN.
+ # macin are the macros that are used in #if* conditional expressions
+ # macout are the macros which are #defined
+ # incl is the list of incluide files encountered
+ # returned as a tuple of the filename followed by the triplet of lists
+ # (filen, macin, macout, incl)
+ 
+ def create_macro_in_out (filen):
+   sfile = open (filen, "r");
+   data = sfile.readlines()
+   sfile.close()
+ 
+   macout = list ()
+   macin = list()
+   incl = list()
+ 
+   for line in (data):
+     d = find_pound_define (line)
+     if d != "":
+       if d not in macout:
+ 	macout.append (d);
+       continue
+ 
+     d = find_pound_if (line)
+     if len(d) != 0:
+       for mac in d:
+ 	if mac != "defined" and mac not in macin:
+ 	  macin.append (mac);
+       continue
+ 
+     nm = find_pound_include (line, True, True)
+     if nm != "" and nm not in incl:
+       incl.append (nm)
+ 
+   return (filen, macin, macout, incl)
+ 
+ # create the macro information for filen, and create .macin, .macout, and .incl
+ # files.  Return the created macro tuple.
+ def create_include_data_files (filen):
+ 
+   macros = create_macro_in_out (filen)
+   depends = macros[1]
+   defines = macros[2]
+   incls = macros[3]
+   
+   disp_message = filen
+   if len (defines) > 0:
+     disp_message = disp_message + " " + str(len (defines)) + " #defines"
+   dfile = open (filen + ".macout", "w")
+   for x in defines:
+     dfile.write (x + "\n")
+   dfile.close ()
+ 
+   if len (depends) > 0:
+     disp_message = disp_message + " " + str(len (depends)) + " #if dependencies"
+   dfile = open (filen + ".macin", "w")
+   for x in depends:
+     dfile.write (x + "\n")
+   dfile.close ()
+ 
+   if len (incls) > 0:
+     disp_message = disp_message + " " + str(len (incls)) + " #includes"
+   dfile = open (filen + ".incl", "w")
+   for x in incls:
+     dfile.write (x + "\n")
+   dfile.close ()
+ 
+   return macros
+ 
+ 
+ 
+ # extract data for include file name_h and enter it into the dictionary.
+ # this doesnt change once read in.  use_requies is True if you want to 
+ # prime the values with already created .requires and .provides files.
+ def get_include_data (name_h, use_requires):
+   macin = list()
+   macout = list()
+   incl = list ()
+   if use_requires and os.path.exists (name_h + ".requires"):
+     macin = open (name_h + ".requires").read().splitlines()
+   elif os.path.exists (name_h + ".macin"):
+     macin = open (name_h + ".macin").read().splitlines()
+ 
+   if use_requires and os.path.exists (name_h + ".provides"):
+     macout  = open (name_h + ".provides").read().splitlines()
+   elif os.path.exists (name_h + ".macout"):
+     macout  = open (name_h + ".macout").read().splitlines()
+ 
+   if os.path.exists (name_h + ".incl"):
+     incl = open (name_h + ".incl").read().splitlines()
+ 
+   if len(macin) == 0 and len(macout) == 0 and len(incl) == 0:
+     return ()
+   data = ( name_h, macin, macout, incl )
+   return data
+   
+ # find FIND in src, and replace it with the list of includes in REPLACE
+ # remove any duplicates of find or replace, and if some of hte replace
+ # includes occur earlier in the inlude chain, leave them.
+ # return the new SRC only if anything changed.
+ def find_replace_include (find, replace, src):
+   res = list()
+   seen = { }
+   anything = False
+   for line in src:
+     inc = find_pound_include (line, True, True)
+     if inc == find:
+       for y in replace:
+         if seen.get(y) == None:
+ 	  res.append("#include \""+y+"\"\n")
+ 	  seen[y] = True
+ 	  if y != find:
+ 	    anything = True
+ # if find isnt in the replacement list, then we are deleting FIND, so changes.
+       if find not in replace:
+         anything = True
+     else:
+       if inc in replace:
+         if seen.get(inc) == None:
+ 	  res.append (line)
+ 	  seen[inc] = True
+       else:
+ 	res.append (line)
+ 
+   if (anything):
+     return res
+   else:
+     return list()
+       
+ 
+ # pass in a require and provide dictionary to be read in.
+ def read_require_provides (require, provide):
+   if not os.path.exists ("require-provide.master"):
+     print "require-provide.master file is not available. please run data collection."
+     sys.exit(1)
+   incl_list = open("require-provide.master").read().splitlines()
+   for f in incl_list:
+     if os.path.exists (f+".requires"):
+       require[os.path.basename (f)] = open (f + ".requires").read().splitlines()
+     else:
+       require[os.path.basename (f)] = list ()
+     if os.path.exists (f+".provides"):
+       provide[os.path.basename (f)] = open (f + ".provides").read().splitlines()
+     else:
+       provide [os.path.basename (f)] = list ()
+ 
+    
+ def build_include_list (filen):
+   include_files = list()
+   sfile = open (filen, "r")
+   data = sfile.readlines()
+   sfile.close()
+   for line in data:
+     nm = find_pound_include (line, False, False)
+     if nm != "" and nm[-2:] == ".h":
+       if nm not in include_files:
+ 	include_files.append(nm)
+   return include_files
+  
+ def build_reverse_include_list (filen):
+   include_files = list()
+   sfile = open (filen, "r")
+   data = sfile.readlines()
+   sfile.close()
+   for line in reversed(data):
+     nm = find_pound_include (line, False, False)
+     if nm != "":
+       if nm not in include_files:
+ 	include_files.append(nm)
+   return include_files
+      
+ # compensate for this stupid warning that should be an error for 
+ # inlined templates
+ def get_make_rc (rc, output):
+   rc = rc % 1280
+   if rc == 0:
+     # This is not considered a fatal error for a build!  /me rolls eyes
+     h = re.findall ("warning: inline function.*used but never defined", output)
+     if len(h) != 0:
+       rc = 1
+   return rc;
+ 
+ def get_make_output (build_dir, make_opt):
+   devnull = open('/dev/null', 'w')
+   at_a_time = multiprocessing.cpu_count() * 2
+   make = "make -j"+str(at_a_time)+ " "
+   if build_dir != "":
+     command = "cd " + build_dir +"; " + make + make_opt
+   else:
+     command = make + make_opt
+   process = subprocess.Popen(command, stdout=devnull, stderr=subprocess.PIPE, shell=True)
+   output = process.communicate();
+   rc = get_make_rc (process.returncode, output[1])
+   return (rc , output[1])
+ 
+ def spawn_makes (command_list):
+   devnull = open('/dev/null', 'w')
+   rc = (0,"", "")
+   proc_res = list()
+   text = "  Trying target builds : "
+   for command_pair in command_list:
+     tname = command_pair[0]
+     command = command_pair[1]
+     text += tname + ", "
+     c = subprocess.Popen(command, bufsize=-1, stdout=devnull, stderr=subprocess.PIPE, shell=True)
+     proc_res.append ((c, tname))
+ 
+   print text[:-2]
+ 
+   for p in proc_res:
+     output = p[0].communicate()
+     ret = (get_make_rc (p[0].returncode, output[1]), output[1], p[1])
+     if (ret[0] != 0):
+       # Just record the first one.
+       if rc[0] == 0:
+ 	rc = ret;
+   return rc
+ 
+ def get_make_output_parallel (targ_list, make_opt, at_a_time):
+   command = list()
+   targname = list()
+   if at_a_time == 0:
+     at_a_time = multiprocessing.cpu_count() * 2
+   proc_res = [0] * at_a_time
+   for x in targ_list:
+     if make_opt[-2:] == ".o":
+       s = "cd " + x[1] + "/gcc/; make " + make_opt
+     else:
+       s = "cd " + x[1] +"; make " + make_opt
+     command.append ((x[0],s))
+ 
+   num = len(command) 
+   rc = (0,"", "")
+   loops = num // at_a_time
+   
+   if (loops > 0):
+     for idx in range (loops):
+       ret = spawn_makes (command[idx*at_a_time:(idx+1)*at_a_time])
+       if ret[0] != 0:
+         rc = ret
+         break
+ 
+   if (rc[0] == 0):
+     leftover = num % at_a_time
+     if (leftover > 0):
+       ret = spawn_makes (command[-leftover:])
+       if ret[0] != 0:
+         rc = ret
+ 
+   return rc
+ 
+ 
+ def readwholefile (src_file):
+   sfile = open (src_file, "r")
+   src_data = sfile.readlines()
+   sfile.close()
+   return src_data
+ 

Property changes on: contrib/headers/headerutils.py
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: contrib/headers/included-by
===================================================================
*** contrib/headers/included-by	(revision 0)
--- contrib/headers/included-by	(working copy)
***************
*** 0 ****
--- 1,112 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ 
+ 
+ usage = False
+ src = list()
+ flist = { }
+ process_h = False
+ process_c = False
+ verbose = False
+ level = 0
+ match_all = False
+ num_match = 1
+ 
+ file_list = list()
+ current = True
+ deeper = True
+ scanfiles = True
+ for x in sys.argv[1:]:
+   if x[0:2] == "-h":
+     usage = True
+   elif x[0:2] == "-i":
+     process_h = True
+   elif x[0:2] == "-s" or x[0:2] == "-c":
+     process_c = True
+   elif x[0:2] == "-v":
+     verbose = True
+   elif x[0:2] == "-a":
+     match_all = True
+   elif x[0:2] == "-n":
+     num_match = int(x[2:])
+   elif x[0:2] == "-1":
+     deeper = False
+   elif x[0:2] == "-2":
+     current = False
+   elif x[0:2] == "-f":
+     file_list = open (x[2:]).read().splitlines()
+     scanfiles = False
+   elif x[0] == "-":
+     print "Error: Unknown option " + x
+     usage = True
+   else:
+     src.append (x)
+ 
+ if match_all:
+   num_match = len (src)
+ 
+ if not process_h and not process_c:
+   process_h = True
+   process_c = True
+ 
+ if len(src) == 0:
+   usage = True
+ 
+ if not usage:
+   if scanfiles:
+     if process_h:
+       file_list = find_gcc_files ("\*.h", current, deeper)
+     if process_c:
+       file_list = file_list + find_gcc_files ("\*.c", current, deeper)
+       file_list = file_list + find_gcc_files ("\*.cc", current, deeper)
+   else:
+     newlist = list()
+     for x in file_list:
+       if process_h and x[-2:] == ".h":
+ 	newlist.append (x)
+       elif process_c and (x[-2:] == ".c" or x[-3:] == ".cc"):
+ 	newlist.append (x)
+     file_list = newlist;
+      
+   file_list.sort()
+   for fn in file_list:
+     found = find_unique_include_list (fn)
+     careabout = list()
+     output = ""
+     for inc in found:
+       if inc in src:
+         careabout.append (inc)
+         if output == "":
+ 	  output = fn
+         if verbose:
+ 	  output = output + " [" + inc +"]"
+     if len (careabout) < num_match:
+         output = ""
+     if output != "":
+       print output
+ else:
+   print "included-by [-h] [-i] [-c] [-v] [-a] [-nx] file1 [file2] ... [filen]"
+   print "find the list of all files in subdirectories that include any of "
+   print "the listed files. processed to a depth of 3 subdirs"
+   print " -h  : Show this message"
+   print " -i  : process only header files (*.h) for #include"
+   print " -c  : process only source files (*.c *.cc) for #include"
+   print "       If nothing is specified, defaults to -i -c"
+   print " -s  : Same as -c."
+   print " -v  : Show which include(s) were found"
+   print " -nx : Only list files which have at least x different matches. Default = 1"
+   print " -a  : Show only files which *all* listed files are included"
+   print "       This is equivilent to -nT where T == # of items in list"
+   print " -flistfile  : Show only files contained in the list of files"
+ 
+  
+ 
+ 
+ 
+ 

Property changes on: contrib/headers/included-by
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: contrib/headers/reduce-headers
===================================================================
*** contrib/headers/reduce-headers	(revision 0)
--- contrib/headers/reduce-headers	(working copy)
***************
*** 0 ****
--- 1,596 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ import tempfile
+ import copy
+ 
+ from headerutils import *
+ 
+ requires = { }
+ provides = { }
+ 
+ no_remove = [ "system.h", "coretypes.h", "config.h" , "bconfig.h", "backend.h" ]
+ 
+ # These targets are the ones which provide "coverage".  Typically, if any
+ # target is going to fail compilation, it's one of these.  This was determined
+ # during the initial runs of reduce-headers... On a full set of target builds,
+ # every failure which occured was triggered by one of these.  
+ # This list is used during target-list construction simply to put any of these
+ # *first* in the candidate list, increasing the probability that a failure is 
+ # found quickly.
+ target_priority = [
+     "aarch64-linux-gnu",
+     "arm-netbsdelf",
+     "avr-rtems",
+     "c6x-elf",
+     "epiphany-elf",
+     "hppa2.0-hpux10.1",
+     "i686-mingw32crt",
+     "i686-pc-msdosdjgpp",
+     "mipsel-elf",
+     "powerpc-eabisimaltivec",
+     "rs6000-ibm-aix5.1.0",
+     "sh-superh-elf",
+     "sparc64-elf",
+     "spu-elf"
+ ]
+ 
+ 
+ target_dir = ""
+ build_dir = ""
+ ignore_list = list()
+ target_builds = list()
+ 
+ target_dict = { }
+ header_dict = { }
+ search_path = [ ".", "../include", "../libcpp/include" ]
+ 
+ remove_count = { }
+ 
+ 
+ # given a header name, normalize it.  ie  cp/cp-tree.h could be in gcc, while
+ # the same header could be referenecd from within the cp subdirectory as
+ # just cp-tree.h
+ # for now, just assume basenames are unique
+ 
+ def normalize_header (header):
+   return os.path.basename (header)
+ 
+ 
+ # Adds a header file and it's sub includes to the global dictionary if they
+ # aren't already there.  SPecify s_path since different build directories may
+ # append themselves on demand to the global list.
+ # return entry for the specified header, knowing all sub entries are completed
+ 
+ def get_header_info (header, s_path):
+   global header_dict
+   global empty_iinfo
+   process_list = list ()
+   location = ""
+   bname = ""
+   bname_iinfo = empty_iinfo
+   for path in s_path:
+     if os.path.exists (path + "/" + header):
+       location = path + "/" + header
+       break
+ 
+   if location:
+     bname = normalize_header (location)
+     if header_dict.get (bname):
+       bname_iinfo = header_dict[bname]
+       loc2 = ii_path (bname_iinfo)+ "/" + bname
+       if loc2[:2] == "./":
+         loc2 = loc2[2:]
+       if location[:2] == "./":
+         location = location[2:]
+       if loc2 != location:
+         # Don't use the cache if it isnt the right one.
+         bname_iinfo = process_ii_macro (location)
+       return bname_iinfo
+ 
+     bname_iinfo = process_ii_macro (location)
+     header_dict[bname] = bname_iinfo
+     # now decend into the include tree
+     for i in ii_include_list (bname_iinfo):
+       get_header_info (i, s_path)
+   else:
+     # if the file isnt in the source directories, look in the build and target
+     # directories. If its here, then aggregate all the versions.
+     location = build_dir + "/gcc/" + header
+     build_inc = target_inc = False
+     if os.path.exists (location):
+       build_inc = True
+     for x in target_dict:
+       location = target_dict[x] + "/gcc/" + header
+       if os.path.exists (location):
+ 	target_inc = True
+ 	break
+ 
+     if (build_inc or target_inc):
+       bname = normalize_header(header)
+       defines = set()
+       consumes = set()
+       incl = set()
+       if build_inc:
+ 	iinfo = process_ii_macro (build_dir + "/gcc/" + header)
+ 	defines = set (ii_macro_define (iinfo))
+ 	consumes = set (ii_macro_consume (iinfo))
+ 	incl = set (ii_include_list (iinfo))
+ 
+       if (target_inc):
+ 	for x in target_dict:
+ 	  location = target_dict[x] + "/gcc/" + header
+ 	  if os.path.exists (location):
+ 	    iinfo = process_ii_macro (location)
+ 	    defines.update (ii_macro_define (iinfo))
+ 	    consumes.update (ii_macro_consume (iinfo))
+ 	    incl.update (ii_include_list (iinfo))
+ 
+       bname_iinfo = (header, "build", list(incl), list(), list(consumes), list(defines), list(), list())
+ 
+       header_dict[bname] = bname_iinfo
+       for i in incl:
+ 	get_header_info (i, s_path)
+ 
+   return bname_iinfo
+ 
+ 
+ # return a list of all headers brought in by this header
+ def all_headers (fname):
+   global header_dict
+   headers_stack = list()
+   headers_list = list()
+   if header_dict.get (fname) == None:
+     return list ()
+   for y in ii_include_list (header_dict[fname]):
+     headers_stack.append (y)
+ 
+   while headers_stack:
+     h = headers_stack.pop ()
+     hn = normalize_header (h)
+     if hn not in headers_list:
+       headers_list.append (hn)
+       if header_dict.get(hn):
+ 	for y in ii_include_list (header_dict[hn]):
+ 	  if normalize_header (y) not in headers_list:
+ 	    headers_stack.append (y)
+ 
+   return headers_list
+ 
+ 
+ 
+ 
+ # Search bld_dir for all target tuples, confirm that they have a build path with
+ # bld_dir/target-tuple/gcc, and build a dictionary of build paths indexed by
+ # target tuple..
+ 
+ def build_target_dict (bld_dir, just_these):
+   global target_dict
+   target_doct = { }
+   error = False
+   if os.path.exists (bld_dir):
+     if just_these:
+       ls = just_these
+     else:
+       ls = os.listdir(bld_dir)
+     for t in ls:
+       if t.find("-") != -1:
+ 	target = t.strip()
+ 	tpath = bld_dir + "/" + target
+ 	if not os.path.exists (tpath + "/gcc"):
+ 	  print "Error: gcc build directory for target " + t + " Does not exist: " + tpath + "/gcc"
+ 	  error = True
+ 	else:
+ 	  target_dict[target] = tpath
+ 
+   if error:
+     target_dict = { }
+ 
+ def get_obj_name (src_file):
+   if src_file[-2:] == ".c":
+     return src_file.replace (".c", ".o")
+   elif src_file[-3:] == ".cc":
+     return src_file.replace (".cc", ".o")
+   return ""
+ 
+ def target_obj_exists (target, obj_name):
+   global target_dict
+   # look in a subdir if src has a subdir, then check gcc base directory.
+   if target_dict.get(target):
+     obj = target_dict[target] + "/gcc/" + obj_name
+     if not os.path.exists (obj):
+       obj = target_dict[target] + "/gcc/" + os.path.basename(obj_name)
+     if os.path.exists (obj):
+       return True
+   return False
+  
+ # Given a src file, return a list of targets which may build this file.
+ def find_targets (src_file):
+   global target_dict
+   targ_list = list()
+   obj_name = get_obj_name (src_file)
+   if not obj_name:
+     print "Error: " + src_file + " - Cannot determine object name."
+     return list()
+ 
+   # Put the high priority targets which tend to trigger failures first
+   for target in target_priority:
+     if target_obj_exists (target, obj_name):
+       targ_list.append ((target, target_dict[target]))
+ 
+   for target in target_dict:
+     if target not in target_priority and target_obj_exists (target, obj_name):
+       targ_list.append ((target, target_dict[target]))
+         
+   return targ_list
+ 
+ 
+ def try_to_remove (src_file, h_list, verbose):
+   global target_dict
+   global header_dict
+   global build_dir
+ 
+   # build from scratch each time
+   header_dict = { }
+   summary = ""
+   rmcount = 0
+ 
+   because = { }
+   src_info = process_ii_macro_src (src_file)
+   src_data = ii_src (src_info)
+   if src_data:
+     inclist = ii_include_list_non_cond (src_info)
+     # work is done if there are no includes to check
+     if not inclist:
+       return src_file + ": No include files to attempt to remove"
+ 
+     # work on the include list in reverse.
+     inclist.reverse()
+ 
+     # Get the target list 
+     targ_list = list()
+     targ_list = find_targets (src_file)
+ 
+     spath = search_path
+     if os.path.dirname (src_file):
+       spath.append (os.path.dirname (src_file))
+ 
+     hostbuild = True
+     if src_file.find("config/") != -1:
+       # config files dont usually build on the host
+       hostbuild = False
+       obn = get_obj_name (os.path.basename (src_file))
+       if obn and os.path.exists (build_dir + "/gcc/" + obn):
+         hostbuild = True
+       if not target_dict:
+         summary = src_file + ": Target builds are required for config files.  None found."
+ 	print summary
+ 	return summary
+       if not targ_list:
+         summary =src_file + ": Cannot find any targets which build this file."
+ 	print summary
+ 	return summary
+ 
+     if hostbuild:
+       # confirm it actually builds before we do anything
+       print "Confirming source file builds"
+       res = get_make_output (build_dir + "/gcc", "all")
+       if res[0] != 0:
+         message = "Error: " + src_file + " does not build currently."
+ 	summary = src_file + " does not build on host."
+ 	print message
+ 	print res[1]
+ 	if verbose:
+ 	  verbose.write (message + "\n")
+ 	  verbose.write (res[1]+ "\n")
+ 	return summary
+ 
+     src_requires = set (ii_macro_consume (src_info))
+     for macro in src_requires:
+       because[macro] = src_file
+     header_seen = list ()
+ 
+     os.rename (src_file, src_file + ".bak")
+     src_orig = copy.deepcopy (src_data)
+     src_tmp = copy.deepcopy (src_data)
+ 
+     try:
+       # process the includes from bottom to top.  This is because we know that
+       # later includes have are known to be needed, so any dependency from this 
+       # header is a true dependency
+       for inc_file in inclist:
+ 	inc_file_norm = normalize_header (inc_file)
+ 	
+ 	if inc_file in no_remove:
+ 	  continue
+ 	if len (h_list) != 0 and inc_file_norm not in h_list:
+ 	  continue
+ 	if inc_file_norm[0:3] == "gt-":
+ 	  continue
+ 	if inc_file_norm[0:6] == "gtype-":
+ 	  continue
+ 	if inc_file_norm.replace(".h",".c") == os.path.basename(src_file):
+ 	  continue
+ 	     
+ 	lookfor = ii_src_line(src_info)[inc_file]
+ 	src_tmp.remove (lookfor)
+ 	message = "Trying " + src_file + " without " + inc_file
+ 	print message
+ 	if verbose:
+ 	  verbose.write (message + "\n")
+ 	out = open(src_file, "w")
+ 	for line in src_tmp:
+ 	  out.write (line)
+ 	out.close()
+ 	  
+ 	keep = False
+ 	if hostbuild:
+ 	  res = get_make_output (build_dir + "/gcc", "all")
+ 	else:
+ 	  res = (0, "")
+ 
+ 	rc = res[0]
+ 	message = "Passed Host build"
+ 	if (rc != 0):
+ 	  # host build failed
+ 	  message  = "Compilation failed:\n";
+ 	  keep = True
+ 	else:
+ 	  if targ_list:
+ 	    objfile = get_obj_name (src_file)
+ 	    t1 = targ_list[0]
+ 	    if objfile and os.path.exists(t1[1] +"/gcc/"+objfile):
+ 	      res = get_make_output_parallel (targ_list, objfile, 0)
+ 	    else:
+ 	      res = get_make_output_parallel (targ_list, "all-gcc", 0)
+ 	    rc = res[0]
+ 	    if rc != 0:
+ 	      message = "Compilation failed on TARGET : " + res[2]
+ 	      keep = True
+ 	    else:
+ 	      message = "Passed host and target builds"
+ 
+         if keep:
+ 	  print message + "\n"
+ 
+ 	if (rc != 0):
+ 	  if verbose:
+ 	    verbose.write (message + "\n");
+ 	    verbose.write (res[1])
+ 	    verbose.write ("\n");
+ 	    if os.path.exists (inc_file):
+ 	      ilog = open(inc_file+".log","a")
+ 	      ilog.write (message + " for " + src_file + ":\n\n");
+ 	      ilog.write ("============================================\n");
+ 	      ilog.write (res[1])
+ 	      ilog.write ("\n");
+ 	      ilog.close()
+ 	    if os.path.exists (src_file):
+ 	      ilog = open(src_file+".log","a")
+ 	      ilog.write (message + " for " +inc_file + ":\n\n");
+ 	      ilog.write ("============================================\n");
+ 	      ilog.write (res[1])
+ 	      ilog.write ("\n");
+ 	      ilog.close()
+ 
+ 	# Given a sequence where :
+ 	# #include "tm.h"
+ 	# #include "target.h"  // includes tm.h
+ 
+ 	# target.h was required, and when attempting to remove tm.h we'd see that
+ 	# all the macro defintions are "required" since they all look like:
+ 	# #ifndef HAVE_blah
+ 	# #define HAVE_blah
+ 	# endif
+ 
+ 	# when target.h was found to be required, tm.h will be tagged as included.
+ 	# so when we get this far, we know we dont have to check the macros for
+ 	# tm.h since we know its already been included.
+ 
+         if inc_file_norm not in header_seen:
+ 	  iinfo = get_header_info (inc_file, spath)
+ 	  newlist = all_headers (inc_file_norm)
+ 	  if ii_path(iinfo) == "build" and not target_dict:
+ 	    keep = True
+ 	    text = message + " : Will not remove a build file without some targets."
+ 	    print text
+ 	    ilog = open(src_file+".log","a")
+ 	    ilog.write (text +"\n")
+ 	    ilog.write ("============================================\n");
+ 	    ilog.close()
+ 	    ilog = open("reduce-headers-kept.log","a")
+ 	    ilog.write (src_file + " " + text +"\n")
+ 	    ilog.close()
+ 	else:
+ 	  newlist = list()
+ 	if not keep and inc_file_norm not in header_seen:
+ 	  # now look for any macro requirements.
+ 	  for h in newlist:
+ 	    if not h in header_seen:
+ 	      if header_dict.get(h):
+ 		defined = ii_macro_define (header_dict[h])
+ 		for dep in defined:
+ 		  if dep in src_requires and dep not in ignore_list:
+ 		    keep = True;
+ 		    text = message + ", but must keep " + inc_file + " because it provides " + dep 
+ 		    if because.get(dep) != None:
+ 		      text = text + " Possibly required by " + because[dep]
+ 		    print text
+ 		    ilog = open(inc_file+".log","a")
+ 		    ilog.write (because[dep]+": Requires [dep] in "+src_file+"\n")
+ 		    ilog.write ("============================================\n");
+ 		    ilog.close()
+ 		    ilog = open(src_file+".log","a")
+ 		    ilog.write (text +"\n")
+ 		    ilog.write ("============================================\n");
+ 		    ilog.close()
+ 		    ilog = open("reduce-headers-kept.log","a")
+ 		    ilog.write (src_file + " " + text +"\n")
+ 		    ilog.close()
+ 		    if verbose:
+ 		      verbose.write (text + "\n")
+ 
+ 	if keep:
+ 	  # add all headers 'consumes' to src_requires list, and mark as seen
+ 	  for h in newlist:
+ 	    if not h in header_seen:
+ 	      header_seen.append (h)
+ 	      if header_dict.get(h):
+ 		consume = ii_macro_consume (header_dict[h])
+ 		for dep in consume:
+ 		  if dep not in src_requires:
+ 		    src_requires.add (dep)
+ 		    if because.get(dep) == None:
+ 		      because[dep] = inc_file
+ 
+ 	  src_tmp = copy.deepcopy (src_data)
+ 	else:
+ 	  print message + "  --> removing " + inc_file + "\n"
+ 	  rmcount += 1
+ 	  if verbose:
+ 	    verbose.write (message + "  --> removing " + inc_file + "\n")
+ 	  if remove_count.get(inc_file) == None:
+ 	    remove_count[inc_file] = 1
+ 	  else:
+ 	    remove_count[inc_file] += 1
+ 	  src_data = copy.deepcopy (src_tmp)
+     except:
+       print "Interuption: restoring original file"
+       out = open(src_file, "w")
+       for line in src_orig:
+ 	out.write (line)
+       out.close()
+       raise
+ 
+     # copy current version, since its the "right" one now.
+     out = open(src_file, "w")
+     for line in src_data:
+       out.write (line)
+     out.close()
+     
+     # Try a final host bootstrap build to make sure everything is kosher.
+     if hostbuild:
+       res = get_make_output (build_dir, "all")
+       rc = res[0]
+       if (rc != 0):
+ 	# host build failed! return to original version
+ 	print "Error: " + src_file + " Failed to bootstrap at end!!! restoring."
+ 	print "        Bad version at " + src_file + ".bad"
+ 	os.rename (src_file, src_file + ".bad")
+ 	out = open(src_file, "w")
+ 	for line in src_orig:
+ 	  out.write (line)
+ 	out.close()
+ 	return src_file + ": failed to build after reduction.  Restored original"
+ 
+     if src_data == src_orig:
+       summary = src_file + ": No change."
+     else:
+       summary = src_file + ": Reduction performed, "+str(rmcount)+" includes removed."
+   print summary
+   return summary
+ 
+ only_h = list ()
+ ignore_cond = False
+ 
+ usage = False
+ src = list()
+ only_targs = list ()
+ for x in sys.argv[1:]:
+   if x[0:2] == "-b":
+     build_dir = x[2:]
+   elif x[0:2] == "-f":
+     fn = normalize_header (x[2:])
+     if fn not in only_h:
+       only_h.append (fn)
+   elif x[0:2] == "-h":
+     usage = True
+   elif x[0:2] == "-d":
+     ignore_cond = True
+   elif x[0:2] == "-D":
+     ignore_list.append(x[2:])
+   elif x[0:2] == "-T":
+     only_targs.append(x[2:])
+   elif x[0:2] == "-t":
+     target_dir = x[2:]
+   elif x[0] == "-":
+     print "Error:  Unrecognized option " + x
+     usgae = True
+   else:
+     if not os.path.exists (x):
+       print "Error: specified file " + x + " does not exist."
+       usage = True
+     else:
+       src.append (x)
+ 
+ if target_dir:
+   build_target_dict (target_dir, only_targs)
+ 
+ if build_dir == "" and target_dir == "":
+   print "Error: Must specify a build directory, and/or a target directory."
+   usage = True
+ 
+ if build_dir and not os.path.exists (build_dir):
+     print "Error: specified build directory does not exist : " + build_dir
+     usage = True
+ 
+ if target_dir and not os.path.exists (target_dir):
+     print "Error: specified target directory does not exist : " + target_dir
+     usage = True
+ 
+ if usage:
+   print "Attempts to remove extraneous include files from source files. "
+   print " "
+   print "Should be run from the main gcc source directory, and works on a target"
+   print "directory, as we attempt to make the 'all' target."
+   print " "
+   print "By default, gcc-reorder-includes is run on each file before attempting"
+   print "to remove includes. this removes duplicates and puts some headers in a"
+   print "canonical ordering"
+   print " "
+   print "The build directory should be ready to compile via make. Time is saved "
+   print "if the build is already complete, so that only changes need to be built."
+   print " "
+   print "Usage: [options] file1.c [file2.c] ... [filen.c]"
+   print "      -bdir    : the root build directory to attempt buiding .o files."
+   print "      -tdir    : the target build directory"
+   print "      -d       : Ignore conditional macro dependencies."
+   print " "
+   print "      -Dmacro  : Ignore a specific macro for dependencies"
+   print "      -Ttarget : Only consider target in target directory."
+   print "      -fheader : Specifies a specific .h file to be considered."
+   print " "
+   print "      -D, -T, and -f can be specified mulitple times and are aggregated."
+   print " "
+   print "  The original file will be in filen.bak"
+   print " "
+   sys.exit (0)
+  
+ if only_h:
+   print "Attempting to remove only these files:"
+   for x in only_h:
+     print x
+   print " "
+ 
+ logfile = open("reduce-headers.log","w")
+ 
+ for x in src:
+   msg = try_to_remove (x, only_h, logfile)
+   ilog = open("reduce-headers.sum","a")
+   ilog.write (msg + "\n")
+   ilog.close()
+ 
+ ilog = open("reduce-headers.sum","a")
+ ilog.write ("===============================================================\n")
+ for x in remove_count:
+   msg = x + ": Removed " + str(remove_count[x]) + " times."
+   print msg
+   logfile.write (msg + "\n")
+   ilog.write (msg + "\n")
+ 
+ 
+ 
+ 
+ 

Property changes on: contrib/headers/reduce-headers
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: contrib/headers/replace-header
===================================================================
*** contrib/headers/replace-header	(revision 0)
--- contrib/headers/replace-header	(working copy)
***************
*** 0 ****
--- 1,53 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ 
+ files = list()
+ replace = list()
+ find = ""
+ usage = False
+ 
+ for x in sys.argv[1:]:
+   if x[0:2] == "-h":
+     usage = True
+   elif x[0:2] == "-f" and find == "":
+     find = x[2:]
+   elif x[0:2] == "-r":
+     replace.append (x[2:])
+   elif x[0:1] == "-":
+     print "Error: unrecognized option " + x
+     usage = True
+   else:
+     files.append (x)
+ 
+ if find == "":
+   usage = True
+ 
+ if usage:
+   print "replace-header -fheader -rheader [-rheader] file1 [filen.]"
+   sys.exit(0)
+ 
+ string = ""
+ for x in replace:
+   string = string + " '"+x+"'"
+ print "Replacing '"+find+"'  with"+string
+ 
+ for x in files:
+   src = readwholefile (x)
+   src = find_replace_include (find, replace, src)
+   if (len(src) > 0):
+     print x + ": Changed"
+     out = open(x, "w")
+     for line in src:
+       out.write (line);
+     out.close ()
+   else:
+     print x
+ 
+ 
+ 

Property changes on: contrib/headers/replace-header
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: contrib/headers/show-headers
===================================================================
*** contrib/headers/show-headers	(revision 0)
--- contrib/headers/show-headers	(working copy)
***************
*** 0 ****
--- 1,108 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ 
+ tabstop = 2
+ padding = "                                                                  "
+ seen = { }
+ output = list()
+ sawcore = False
+ 
+ incl_dirs = [".", "../include", "../../build/gcc", "../libcpp/include" ]
+ 
+ def append_1 (output, inc):
+   for n,t in enumerate (output):
+     if t.find(inc) != -1:
+       t += "  (1)"
+       output[n] = t
+       return
+ 
+ rtl_core = [ "machmode.h" , "signop.h" , "wide-int.h" , "double-int.h" , "real.h" , "fixed-value.h" , "statistics.h" , "vec.h" , "hash-table.h" , "hash-set.h" , "input.h" , "is-a.h" ]
+ 
+ def find_include_data (inc):
+   global sawcore
+   for x in incl_dirs:
+     nm = x+"/"+inc
+     if os.path.exists (nm):
+       info = find_unique_include_list (nm)
+       # rtl.h mimics coretypes for GENERATOR FILES, remove if coretypes.h seen.
+       if inc == "coretypes.h":
+         sawcore = True
+       elif inc  == "rtl.h" and sawcore:
+         for i in rtl_core:
+ 	  if i in info:
+ 	    info.remove (i)
+       return info
+   return list()
+ 
+ def process_include (inc, indent):
+   if inc[-2:] != ".h":
+     return
+   if seen.get(inc) == None:
+     seen[inc] = 1
+     output.append (padding[:indent*tabstop] + os.path.basename (inc))
+     info = find_include_data (inc)
+     for y in info:
+       process_include (y, indent+1)
+   else:
+     seen[inc] += 1
+     if (seen[inc] == 2):
+       append_1(output, inc)
+     output.append (padding[:indent*tabstop] + os.path.basename (inc) + "  ("+str(seen[inc])+")")
+ 
+     
+ 
+ blddir = [ "." ]
+ usage = False
+ src = list()
+ 
+ for x in sys.argv[1:]:
+   if x[0:2] == "-i":
+     bld = x[2:]
+     print "Build dir : " + bld
+     blddir.append (bld)
+   elif x[0:2] == "-h":
+     usage = True
+   else:
+     src.append (x)
+ 
+ if len(src) != 1:
+   usage = True
+ 
+ if usage:
+   print "show-headers [-idir]  file1 "
+   print " "
+   print " show in a hierarchical visual format how many times each header file"
+   print " is included ina source file.  Should be run from the source directory"
+   print "  files from find-include-depends"
+   print "      -i : Specifies 1 or more directories to search for includes."
+   print "           defaults to looking in :"
+   print "           . , ../include, ../libcpp/include, and ../../build/gcc"
+   print "      specifying anything else creates a new list starting with '.'"
+   sys.exit(0)
+ 
+ 
+ if len(blddir) > 1:
+   incl_dirs = blddir
+ 
+ x = src[0]
+ # if source is in a subdirectory, add the subdirectory to the search list
+ srcpath = os.path.dirname(x)
+ if srcpath:
+   incl_dirs.append (srcpath)
+ 
+ output = list()
+ sawcore = False
+ incl = find_unique_include_list (x)
+ for inc in incl:
+   process_include (inc, 1)
+ print "\n" + x
+ for line in output:
+   print line
+ 
+ 

Property changes on: contrib/headers/show-headers
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property

Re: [patch 4/3] Header file reduction - Tools for contrib

[Bernd Schmidt]

>> There are 9 tools I used over the run of the project.  They were
>> developed in various stages and iterations, but I tried to at least have
>> some common interface things, and I tried some cleaning up and
>> documentation.

I'll probably have to make multiple passes over this. A disclaimer 
first, I have done enough Python programming to develop a dislike for 
the language, but not enough to call myself an expert.

General comments first. Where applicable, I think we should apply the 
same coding standards to Python as we do for C/C++. That means things 
like function comments documenting parameters. They are absent for the 
most part in this patch, and I won't point out individual instances.
Also, I think the documentation should follow our usual rules. There are 
spelling and grammar problems. I will point out what I find (only the 
first instance for recurring problems), but please proofread the whole 
thing for the next submission. The Thunderbird spellchecker actually is 
pointing out a lot of these. Capitalize starts of sentences, write full 
sentences and terminate with punctuation.

> No commenting on the quality of python code... :-) I was
> learning python on the fly.    Im sure some things are QUITE awful.,

Yeah, the general impression is of fairly ad-hoc code. Not sure how much 
can be done about this.

> + trigger the help message.  Help may specify additonal functionality to what is

"additional"

> + - For*all*  tools, option format for specifying filenames must have no spaces

Space after "For", remove double space. This pattern occurs very often - 
something your editor does maybe?

> + - Many of the tools are required to be run from the core gcc source directory
> + containing coretypes.h  typically that is  in gcc/gcc from a source checkout.

Odd whitespace, and probably lack of punctuation before "typically".

> + gcc-order-headers
> + -----------------
> +   This will reorder any primary backend headers files into a canonical order
> +   which will resolve any hidden dependencies they may have.  Any unknown
> +   headers will simply be occur after the recognized core files, and retain the
> +   same relative ordering they had.

Grammar ("be occur").

This sounds like the intention is to move recognized core files (I 
assume these are the ones in the "order" array in the tool) to the 
start, and leaving everything alone? I was a bit confused about this at 
first; I see for example "timevar.h" moving around without being present 
in the list, but it looks like it gets added implicitly through being 
included by df.h. (Incidentally, this looks like another case like the 
obstack one - a file using timevars should include timevar.h IMO, even 
if it also includes df.h).

> +
> +   Must be run in the core gcc source directory

"This tool must be run in the core gcc source directory." (Punctuation 
and grammar).

> +   Any files which are changed are output, and the original is saved with a
> +   .bak extention.
> +
> +   ex.:     gcc-order-headers tree-ssa.c c/c-decl.c

It might be more useful to produce a diff rather than modify files in-place.

> +   if any header files are included within a conditional code block, the tool
> +   will issue a message and not change the file.  When this happens, you can
> +   manually inspect the file, and if reorder it will be fine, rerun the command

"if reorder it will be fine"?

> +   It does a 4 level deep find of all source files from the current directory
> +   and look in each of those for a #include of the specified headers.  So expect
> +   a little bit of slowness.

"looks"?

> +
> +   -i limits the search to only other header files.
> +   -c limits the search to .c and .cc files.
> +   -a shows only source files which include*all*  specified headers.

Whitespace.

> +   it is good practive to run 'gcc-order-headers' on a source file before trying

"practice"

> +   Any desired target builds should be built in one directory using a modified
> +   config-list.mk file which doesnt delete the build directory when its done.

"doesn't", or more probably "does not" in documentation.

> +   The tool will analyze a source file and attempt to remove each non-conditional
> +   header from last to first in the file.:
> +     It will first attempt to build the native all-gcc target.
> +     If that succeeds, it will attempt to build any target build .o files
> +     If that suceeds, it will check to see if there are any conditional

"succeeds"
> +        compilation dependencies between this header file and the source file or
> +        any header whihch have already been determined as non-removable.

"whihch"

> +     If all these tests are passed, the header file is determined to be removable
> +        and is removed from the source file.
> +     This continues until all headers have been checked.

One thing I've wondered about - have you tried checking for object file 
differences?
As far as I can tell the dependency checking does not check for undefs. 
Is that correct? I think that needs to be added.

> +   At this point, the a bootstrap is attempted in the native build, and if that

"the a"

> +   A small subset of targets has been determined to provide excellent coverage,
> +   at least as of Aug 31/15 .  A fullset of targets reduced all of the files

"fullset", and whitespace. Determined how?

> +   making up libbackend.a.  All of the features which requires target testing
> +   were found to be triggered by one or more of these targets.  They are
> +   actually known to the tool, and when checkiong target, it will check those

"checkiong".

> +   targets first, then the rest.  It is mostly safe to do a reduction with just
> +   these targets, at least until some new whacky target comes along.
> +   building config-list.mk with :
> +   LIST="aarch64-linux-gnu arm-netbsdelf avr-rtems c6x-elf epiphany-elf hppa2.0-hpux10.1 i686-mingw32crt i686-pc-msdosdjgpp mipsel-elf powerpc-eabisimaltivec rs6000-ibm-aix5.1.0 sh-superh-elf sparc64-elf spu-elf"

I think I get what you're trying to say, but the whole paragraph could 
be rewritten for clarity.

> +     reduce-headers.log :  All the compilation failure output that tool tried.

"the tool"?

> + 			     and why it thinks taht is the case

"taht".

> +     $src.c.log  : for each failed header removal, the compilation
> + 		  messages as to why it failed.
> +     $header.h.log: The same log is put into the relevent header log as well.

"relevant"

> +   The tool will aggregate all these and generate a graph of the dependencies
> +   exposed during compilation.  red lines indicate dependecies that are

"depndecies"

> +   presednt because a head file physically includes another header. Black lines

"presednt"

> +   represent data dependencies causing compilation if the header isnt present.

"is not"

> + for x in sys.argv[1:]:
> +   if x[0:2] == "-h":
> +     usage = True
> +   else:
> +     src.append (x)

There are getopt and argparse libraries available for python. I seem to 
recall fighting them at some point because they didn't quite do what I 
expected from a C getopt, so it may not be worth it trying to use them.

> + if not usage and len(src) > 0:
> +
> +   incl = { }

Watch the extra blank lines.

> +         if dup.get(d) == None:

I think we want to be consistent with our C style? I.e., extra space 
before parentheses.

> +   l.sort(key=lambda tup:tup[0], reverse=True)

And spaces around things like = operators.

> +     # Don't put diagnostic*.h into the ordering list, its special since

"it is". Many instances, please grep for "its" and fix them all.

> +     # various front ends have to set GCC_DIAG_STYLE before including it
> +     # for each file, we'll tailor where it belongs by looking at the dup
> +     # list and seeing which file is included, and position it appropriately.

 From that comment it's not entirely clear how they are handled. Please 
expand documentation of this mechanism.

> +   # rtl.h gets tagged as a duplicate includer for all of coretypes, but thats

"that's"

> + # process diagnostic.h first.. it's special since GCC_DIAG_STYLE can be
> + # overridden by languages, but must be done so by a file included BEFORE it.
> + # so make sure it isn't seen as inclujded by one of those files by making it

"inclujded"

> + # Now crate the master ordering list

"create".

> + for i in order:
> +   create_master_list (os.path.basename (i), False)

I found myself wanting to pass True. The tool could use a "-v" flag.

> +   print " -s Show the cananoical order of known includes"

"canonical"

> +   print "Multi-line comments after a #include can also cause failuer, they must be turned"

"failuer"

> + ignore = [ "coretypes_h",
> + 	     "machmode_h",
> + 	     "signop_h",
> + 	     "wide_int_h",
> + 	     "double_int_h",
> + 	     "real_h",
> + 	     "fixed_value_h",
> + 	     "hash_table_h",
> + 	       "statistics_h",
> + 	       "ggc_h",
> + 	       "vec_h",
> + 	       "hashtab_h",
> + 	       "inchash_h",
> + 	       "mem_stats_traits_h",
> + 	       "hash_map_traits_h",
> + 	       "mem_stats_h",
> + 	       "hash_map_h",
> + 	     "hash_set_h",
> + 	     "input_h",
> + 	       "line_map_h",
> + 	     "is_a_h",
> + 	   "system_h",
> + 	   "config_h" ]

Is the random indentation indicating some kind of nesting? If not, 
please fix.

> +   for line in logfile:
> +     if len (line) > 21 and line[:21] in depstring:
> +       if newinc:
> +         incfrom = list()
> + 	newinc = False

It looks like you are mixing tab and space indentation. For a language 
like Python, that is absolutely scary. Please fix throughout (I think 
only spaces is probably best).

> + if dohelp:
> +   print "Generates a graph of the include web for specified files."
> +   print "Usage:  [-finput_file] [-h] [-ooutput] [file1 ... [filen]]"
> +   print "	-finput_file : Input file is file containing a list of files"
> +   print "	-ooutput : Specifies output to output.dot and output.png"
> +   print "                  defaults to graph.dot and graph.png"
> +   print "	-nnum : specifies the # of edges beyond which sfdp is invoked. def=0"
> +   print "       -a : Aggregate all .c files to 1 file.  Shows only include web."
> +   print "       -at :  Aggregate, but don't include terminal.h to .c links. "
> +   print "	-h : help"

The formatting of the help output seems somewhat random. Also "is a file"?

> +   if len(inc) > 0:
> + #    inc2 = re.findall (ur"defined *\((.+?)\)", inc[0])
> +     inc2 = re.findall (ur"defined\s*\((.+?)\)", inc[0])

Intentionally commented out?

> +
> + def process_ii (filen):
> +   return process_include_info (filen, False, False)
> +
> + def process_ii_macro (filen):
> +   return process_include_info (filen, True, False)
> +
> + def process_ii_src (filen):
> +   return process_include_info (filen, False, True)
> +
> + def process_ii_macro_src (filen):
> +   return process_include_info (filen, True, True)
> +
> + def ii_base (iinfo):
> +   return iinfo[0]
> +
> + def ii_path (iinfo):
> +   return iinfo[1]
> +
> + def ii_include_list (iinfo):
> +   return iinfo[2]
> +
> + def ii_include_list_cond (iinfo):
> +   return iinfo[3]
> +
> + def ii_include_list_non_cond (iinfo):
> +   l = ii_include_list (iinfo)
> +   for n in ii_include_list_cond (iinfo):
> +     l.remove (n)
> +   return l
> +
> + def ii_macro_consume (iinfo):
> +   return iinfo[4]
> +
> + def ii_macro_define (iinfo):
> +   return iinfo[5]
> +
> + def ii_src (iinfo):
> +   return iinfo[6]
> +
> + def ii_src_line (iinfo):
> +   return iinfo[7]

That's a lot of little functions with pretty much no clue for the reader 
what's going on. It looks like maybe there's an array where a struct 
should have been used?

> + # extract data for include file name_h and enter it into the dictionary.
> + # this doesnt change once read in.  use_requies is True if you want to

"does not", "use_requies"

> + # find FIND in src, and replace it with the list of includes in REPLACE
> + # remove any duplicates of find or replace, and if some of hte replace

"hte"

> + # includes occur earlier in the inlude chain, leave them.

"inlude"

> + # compensate for this stupid warning that should be an error for
> + # inlined templates
> + def get_make_rc (rc, output):
> +   rc = rc % 1280
> +   if rc == 0:
> +     # This is not considered a fatal error for a build!  /me rolls eyes
> +     h = re.findall ("warning: inline function.*used but never defined", output)
> +     if len(h) != 0:
> +       rc = 1
> +   return rc;

What's this about?

> +   print " -a  : Show only files which*all*  listed files are included"

Whitespace around *all*. Seems to happen quite often.

> + # given a header name, normalize it.  ie  cp/cp-tree.h could be in gcc, while

Formatting, capitalization.

> + # the same header could be referenecd from within the cp subdirectory as

"referenced"

> + # Adds a header file and it's sub includes to the global dictionary if they

This time, "its".

> + # aren't already there.  SPecify s_path since different build directories may

"SPecify"

> + if usage:
> +   print "Attempts to remove extraneous include files from source files. "
> +   print " "
> +   print "Should be run from the main gcc source directory, and works on a target"
> +   print "directory, as we attempt to make the 'all' target."
> +   print " "
> +   print "By default, gcc-reorder-includes is run on each file before attempting"
> +   print "to remove includes. this removes duplicates and puts some headers in a"
> +   print "canonical ordering"
> +   print " "
> +   print "The build directory should be ready to compile via make. Time is saved "

Space at the end of the line (two cases in this block).

> +   print " "
> +   print " show in a hierarchical visual format how many times each header file"
> +   print " is included ina source file.  Should be run from the source directory"

"ina".


Bernd

Re: [patch 4/3] Header file reduction - Tools for contrib

[Andrew MacLeod]

On 10/06/2015 08:02 AM, Bernd Schmidt wrote:
>
>>> There are 9 tools I used over the run of the project.  They were
>>> developed in various stages and iterations, but I tried to at least 
>>> have
>>> some common interface things, and I tried some cleaning up and
>>> documentation.
>
> I'll probably have to make multiple passes over this. A disclaimer 
> first, I have done enough Python programming to develop a dislike for 
> the language, but not enough to call myself an expert.
>
> General comments first. Where applicable, I think we should apply the 
> same coding standards to Python as we do for C/C++. That means things 
> like function comments documenting parameters. They are absent for the 
> most part in this patch, and I won't point out individual instances.
> Also, I think the documentation should follow our usual rules. There 
> are spelling and grammar problems. I will point out what I find (only 
> the first instance for recurring problems), but please proofread the 
> whole thing for the next submission. The Thunderbird spellchecker 
> actually is pointing out a lot of these. Capitalize starts of 
> sentences, write full sentences and terminate with punctuation.
>

I primarily submitted it early because you wanted to look at the tools  
before the code patch, which is the one I care about since the longer it 
goes, the more effort it is to update the patch to mainline.   I 
apologize for not proofreading it as much as I usually do.  My longer 
term intention was to polish the readme stuff and put it into each tool 
as well.

however, none of the other tools or scripts in contrib subscribe to 
commenting every function the same as we do for c/c++.  I did put 
comments in many places where it wasn't obvious what was going on to 
help with readability, but other cases it seemed obvious enough not to 
bother.    I don't mind adding missing ones that are important, but I do 
not see why every function needs to have the full c/c++ coding standard 
applied to it when no other tool does.  These certainly appear as good 
to me if not better than the existing scripts...


>> No commenting on the quality of python code... :-) I was
>> learning python on the fly.    Im sure some things are QUITE awful.,
>
> Yeah, the general impression is of fairly ad-hoc code. Not sure how 
> much can be done about this.
they were never intended as general purpose tools, they were developed 
over multiple iterations and bugfixing and never properly designed.. 
they were never originally intended for public submission, so they 
suffer...  and I'm not interested in rewriting them yet again

Andrew

Re: [patch 4/3] Header file reduction - Tools for contrib

[Bernd Schmidt]

On 10/06/2015 04:04 PM, Andrew MacLeod wrote:

> I primarily submitted it early because you wanted to look at the tools
> before the code patch, which is the one I care about since the longer it
> goes, the more effort it is to update the patch to mainline.

The problem is that the generated patch is impossible to review on its 
own. It's just a half a megabyte dump of changes that can't 
realistically be verified for correctness. Reading it can throw up some 
interesting questions which can then (hopefully) be answered by 
reference to the tools, such as "why does timevar.h move?" For that to 
work, the tools need at least to have a minimum level of readability. 
They are the important part here, not the generated patch. (Unless you 
find a reviewer who's less risk-averse than me and is willing to approve 
the whole set and hope for the best.)

I suspect you'll have to regenerate the includes patch anyway, because 
of the missing #undef tracking I mentioned.

Let's consider the timevar.h example a bit more. Does the include have 
to move? I don't see anything in that file that looks like a dependency, 
and include files that need it are already including it. Is the fact 
that df.h includes it in any way material for generating an order of 
headers? IMO, no, it's an unnecessary change indicating a bug in the 
script, and any kind of unnecessary change in a patch like this makes it 
so much harder to verify. I think the canonical order that's produced 
should probably ignore files included from other headers so that these 
are left alone in their original order.

I'd still like more explanations of special cases in the tools like the 
diagnostic.h area as well as
     # seed tm.h with options.h since its a build file and won't be seen.
and I think we need to understand what makes them special in a way that 
makes the rest of the algorithm not handle them correctly (so that we 
don't overlook any other such cases).


Bernd

Re: [patch 4/3] Header file reduction - Tools for contrib

[Andrew MacLeod]

On 10/06/2015 10:56 AM, Bernd Schmidt wrote:
> On 10/06/2015 04:04 PM, Andrew MacLeod wrote:
>
>> I primarily submitted it early because you wanted to look at the tools
>> before the code patch, which is the one I care about since the longer it
>> goes, the more effort it is to update the patch to mainline.
>
> The problem is that the generated patch is impossible to review on its 
> own. It's just a half a megabyte dump of changes that can't 
> realistically be verified for correctness. Reading it can throw up 
> some interesting questions which can then (hopefully) be answered by 
> reference to the tools, such as "why does timevar.h move?" For that to 
> work, the tools need at least to have a minimum level of readability. 
> They are the important part here, not the generated patch. (Unless you 
> find a reviewer who's less risk-averse than me and is willing to 
> approve the whole set and hope for the best.)

I dont get your fear.  I could have created that patch by hand, it would 
just take a long time, and would likely be less complete, but just as large.

I'm not  changing functionality.  ALL the tool is doing is removing 
header files which aren't needed to compile.  It goes to great pains to 
make sure it doesn't remove a silent dependency that conditional 
compilation might introduce.  Other than that, the sanity check is that 
everything compiles on every target and regression tests show nothing.   
Since we're doing this with just include files, and not changing 
functionality, Im not sure what your primary concern is? You are 
unlikely to ever be able to read the patch and decide for yourself 
whether removing expr.h from the header list is correct or not.  Much 
like if I proposed the same thing by hand.

Yes, I added the other tool in which reorders the headers and removes 
duplicates, and perhaps that is what is causing you the angst.  The 
canonical ordering was developed by taking current practice and adding 
in other core files which had ordering issues that showed up during the 
reduction process.   Reorderiing all files to this order should actually 
resolve more issues than it causes.   I can generate and provide that as 
a patch if you want to look at it separately...  I dont know what that 
buys you.  you could match the includes to the master list to make sure 
the tool did its job by itself I guess.

The tools are unlikely to ever be used again... Jeff suggested I provide 
them to contrib just in case someone decided to do something with them 
someday, they wouldn't be lost,or at least they wouldn't have to track 
me down to get them.

IF we discover that one or more of the tools does continue to have some 
life, well then maybe at that point its worth putting some time into 
refining it a bit better.


> I suspect you'll have to regenerate the includes patch anyway, because 
> of the missing #undef tracking I mentioned.

I dont see that #undef is relevant at all.  All the conditional 
dependencies care about is "MAY DEFINE"  Its conservative in that if 
something could be defined, we'll assume it is and not remove any file 
which may depend on it.  to undefine something in a MAY DEFINE world 
doesnt mean anything.



>
> Let's consider the timevar.h example a bit more. Does the include have 
> to move? I don't see anything in that file that looks like a 
> dependency, and include files that need it are already including it. 
> Is the fact that df.h includes it in any way material for generating 
> an order of headers? IMO, no, it's an unnecessary change indicating a 
> bug in the script, and any kind of unnecessary change in a patch like 
> this makes it so much harder to verify. I think the canonical order 
> that's produced should probably ignore files included from other 
> headers so that these are left alone in their original order.
>
I covered  this in the last note.  Pretty much every file is going to 
have a "core" of up to 95 files reordered into the canonical form, which 
taken as a snapshot of any given file, may look arbitrary but is in fact 
a specific subset of the canonical ordering.   You cant only order some 
parts of it because there are subtle dependencies between the files 
which force you to look at them all.  Trust me, I didnt start by 
reordering all of them this way... it developed over time.


> I'd still like more explanations of special cases in the tools like 
> the diagnostic.h area as well as
>     # seed tm.h with options.h since its a build file and won't be seen.
> and I think we need to understand what makes them special in a way 
> that makes the rest of the algorithm not handle them correctly (so 
> that we don't overlook any other such cases).
>
See the other note, its because of the front end files/diagnostic 
dependencies  or irreconcilable cycles because of what a header 
includes.     Any other case would have shown up the way those did 
during development.

Andrew

Re: [patch 4/3] Header file reduction - Tools for contrib

[Bernd Schmidt]

On 10/06/2015 09:19 PM, Andrew MacLeod wrote:
> I dont get your fear.  I could have created that patch by hand, it would
> just take a long time, and would likely be less complete, but just as
> large.
>
> I'm not  changing functionality.  ALL the tool is doing is removing
> header files which aren't needed to compile.  It goes to great pains to
> make sure it doesn't remove a silent dependency that conditional
> compilation might introduce.  Other than that, the sanity check is that
> everything compiles on every target and regression tests show nothing.
> Since we're doing this with just include files, and not changing
> functionality, Im not sure what your primary concern is?

My concern is that I've seen occasions in the past where "harmless 
cleanups" that were not intended to alter functionality introduced 
severe and subtle bugs that went unnoticed for a significant amount of 
time. If a change does not alter functionality, then there is a valid 
question of "why apply it then?", and the question of correctness 
becomes very important (to me anyway). The patch was produced by a 
fairly complex process, and I'd want to at least be able to convince 
myself that the process is correct.

Anyhow, I'll step back from this, you're probably better served by 
someone else reviewing the patch.


Bernd

Re: [patch 4/3] Header file reduction - Tools for contrib

[Jeff Law]

On 10/06/2015 02:37 PM, Bernd Schmidt wrote:
> On 10/06/2015 09:19 PM, Andrew MacLeod wrote:
>> I dont get your fear.  I could have created that patch by hand, it would
>> just take a long time, and would likely be less complete, but just as
>> large.
>>
>> I'm not  changing functionality.  ALL the tool is doing is removing
>> header files which aren't needed to compile.  It goes to great pains to
>> make sure it doesn't remove a silent dependency that conditional
>> compilation might introduce.  Other than that, the sanity check is that
>> everything compiles on every target and regression tests show nothing.
>> Since we're doing this with just include files, and not changing
>> functionality, Im not sure what your primary concern is?
>
> My concern is that I've seen occasions in the past where "harmless
> cleanups" that were not intended to alter functionality introduced
> severe and subtle bugs that went unnoticed for a significant amount of
> time. If a change does not alter functionality, then there is a valid
> question of "why apply it then?", and the question of correctness
> becomes very important (to me anyway). The patch was produced by a
> fairly complex process, and I'd want to at least be able to convince
> myself that the process is correct.
A very valid concern.  In fact, one could argue that one of the long 
term problems we're likely to face as a project is the inability to do 
this kind of refactoring with high degrees of confidence that we're not 
breaking things.



>
> Anyhow, I'll step back from this, you're probably better served by
> someone else reviewing the patch.
That's fine.  I don't mind covering this.

jeff

Re: [patch 4/3] Header file reduction - Tools for contrib

[Andrew MacLeod]

On 10/06/2015 04:37 PM, Bernd Schmidt wrote:
> On 10/06/2015 09:19 PM, Andrew MacLeod wrote:
>> I dont get your fear.  I could have created that patch by hand, it would
>> just take a long time, and would likely be less complete, but just as
>> large.
>>
>> I'm not  changing functionality.  ALL the tool is doing is removing
>> header files which aren't needed to compile.  It goes to great pains to
>> make sure it doesn't remove a silent dependency that conditional
>> compilation might introduce.  Other than that, the sanity check is that
>> everything compiles on every target and regression tests show nothing.
>> Since we're doing this with just include files, and not changing
>> functionality, Im not sure what your primary concern is?
>
> My concern is that I've seen occasions in the past where "harmless 
> cleanups" that were not intended to alter functionality introduced 
> severe and subtle bugs that went unnoticed for a significant amount of 
> time. If a change does not alter functionality, then there is a valid 
> question of "why apply it then?", and the question of correctness 
> becomes very important (to me anyway). The patch was produced by a 
> fairly complex process, and I'd want to at least be able to convince 
> myself that the process is correct.
>
> Anyhow, I'll step back from this, you're probably better served by 
> someone else reviewing the patch.
>
>
> Bernd
I do get it.  And I have spent a lot of time trying to make sure none of 
those sort of bugs come in, and ultimately have tried to be 
conservative.. after all, its better to have the tool leave an include 
than remove one that may be required.

Ultimately, these changes are unlikely to introduce an issue, but there 
is a very slight possibility.  Any issues that do surface should be of 
the "not using a pattern" kind because a conditional compilation code 
case was somehow missed. I'm hoping for none of those obviously.  
Anyway, the tool does seem to work on all the tests I have looked at.  
If any bugs are uncovered by this, then they are also latent issues we 
didn't know about that should be exposed and fixed anyway.

I am fine if we'd like to separate the patches into the reordering, and 
the deleting.   Its not a lot of effort on my part, just a lot of time 
compiling for the reducer in the background.. and we can do them as 2 
commits if that is helpful.

What I don't want to do is spend a lot more time massaging the tools for 
contrib because I am sick of looking at them right now, and no one is in 
a hurry to use them anyway...  if anyone ever does.:-) The documentation 
grammer should certainly be fixed up and I will add some comments around 
the questions you had.

we could also do a small scale submission on half a dozen files, provide 
the reorder patch, and then  the reduction patch with the logs  if that 
helps whoever is reviewing get comfortable with what the tool is doing, 
then its easier to simply acknowledge the mechanical nature of the large 
commit.

Perhaps it would be educational anyway.

I'll do it however you guys want...  i just want to get it done :-)

Andrew

Re: [patch 4/3] Header file reduction - Tools for contrib

[Jeff Law]

On 10/06/2015 08:04 AM, Andrew MacLeod wrote:
>>> No commenting on the quality of python code... :-) I was
>>> learning python on the fly.    Im sure some things are QUITE awful.,
>>
>> Yeah, the general impression is of fairly ad-hoc code. Not sure how
>> much can be done about this.
> they were never intended as general purpose tools, they were developed
> over multiple iterations and bugfixing and never properly designed..
> they were never originally intended for public submission, so they
> suffer...  and I'm not interested in rewriting them yet again
So a little background for Bernd.

The tangled mess that our header files has been makes it extremely 
difficult to do something introduce a new classes/interfaces to improve 
the separation of various parts of GCC.    Consider the case if we 
wanted to drop trees from gimple onward by initially wrapping trees in a 
trivially compatible class then converting files one by one to use the 
new representation.

We'd want to be able to do the conversion, then ensure ourselves that 
the old interfaces couldn't sneak in.  Getting there required some 
significant header file deconstruction, then reconstruction.

So Andrew set forth to try and untangle the mess of dependencies, remove 
unnecessary includes, etc etc.  He had the good sense to write some 
scripts to help :-0

A few months ago as this stage of refactoring header files as nearing 
completion, I asked Andrew how we were going to prevent things from 
getting into the sorry shape we were in last year.  From that discussion 
the suggestion that he should polish up his scripts and submit them for 
inclusion into the contrib/ subdirectory for future reference/use.

Ideally we'd occasionally run those scripts to ensure that we don't muck 
things up too badly again in the future.

Anyway, that's how we got here.  The scripts are just helper tools, but 
I wouldn't consider them a core part of GCC.  Obviously the cleaner and 
easier to run, the better.

It's interesting that a lot of work done by Andrew has ended up 
mirroring stuff I'm reading these days in Feathers' book.


Jeff

Re: [patch 4/3] Header file reduction - Tools for contrib

[Joseph Myers]

On Tue, 6 Oct 2015, Bernd Schmidt wrote:

> General comments first. Where applicable, I think we should apply the same
> coding standards to Python as we do for C/C++. That means things like function

FWIW, glibc's rule is to follow PEP 8 formatting for Python code.

https://sourceware.org/glibc/wiki/Style_and_Conventions#Code_formatting_in_python_sources

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [patch 4/3] Header file reduction - Tools for contrib

[Andrew MacLeod]

On 10/06/2015 08:02 AM, Bernd Schmidt wrote:
>
>
> This sounds like the intention is to move recognized core files (I 
> assume these are the ones in the "order" array in the tool) to the 
> start, and leaving everything alone? I was a bit confused about this 
> at first; I see for example "timevar.h" moving around without being 
> present in the list, but it looks like it gets added implicitly 
> through being included by df.h. (Incidentally, this looks like another 
> case like the obstack one - a file using timevars should include 
> timevar.h IMO, even if it also includes df.h).
>

Ordering the includes is perhaps more complex than you realize. It more 
complex than I realized when I first started it. it took a long and very 
frustrating period to get it working properly.

There are implicit dependencies between some include files.  The primary 
ordering list is to provide a canonical order for key files so that 
those dependencies are automatically taken care of.  Until now we've 
managed it by hand.    The problem is that the dependencies are not 
necessary always from the main header file.. they may come from one of 
the headers that were included in it. There are lots of dependencies on 
symtab.h for instance, which comes from tree.h   Some other source files 
don't need tree.h, but they do need symtab.h.   If symtab.h isn't in the 
ordering list and the header which uses it is (like cgraph.h) , the tool 
would move cgraph.h above symtab.h and the result doesn't work.

The solution is to take that initial canonical list, and fully expand it 
to include everything that those headers include. This gives a linear 
canonical list of close to 100 files.    It means things like timevar.h 
(which is included by df.h)  are in this "ordering":
<...>
regset.h
alloc-pool.h
timevar.h
df.h
tm_p.h
gimple-iterator
<...>

A source file which does not include df.h  but includes timevar.h muist 
keep it in this same relative ordering, or some other header from the 
ordering list which uses timevar.h may no longer compile. (timevar.h 
would end up after everything in the canonical list instead of in fromt 
of the other file)

This means the any of those 100 headers files which occur in a source 
file should occur in this order.  The original version of the tool tried 
to spell out this exact order, but I realized that was not maintainable 
as headers change, and it was actually far simply to specify the core 
ones In the tool, and let it do the expansion based on what is in the 
current tree.

This also means that taken as a snapshot, you are going to see things 
like timevar.h move around in apparently random fashion... but it is not 
random.  It will be in front of any and all headers listed after it in 
the ordering.  Any headers which don't appear in the canonical list will 
simply retain their current order in the source file, but AFTER all the 
ones in the canonical list.

This also made it fairly easy to remove redundant includes that have 
been seen already by including some other header... I just build the 
list of headers that have been seen already

There are a couple of specialty cases that are handled..
The 'exclude processing' list are headers which shouldn't be expanded 
like above. They can cause irreconcilable problems when expanded , 
especially the front end file files.  They do need to be ordered since 
diagnostics require them to be included first in order to satisfy the 
requirement that   GCC_DIAG_STYLE  be defined before diagnostic.h is 
included.  Plus most of them include tree.h and/or diagnostic.h 
themselves, but we don't want them to impact  the ordering for the 
backend files.

That list puts those core files in an appropriate place canoncailly, but 
doesn't expand into the file because the order we get for the different 
front ends would be different .  Finally diagnostic*.h and friends are 
removed from the list and put at the end to ensure eveything that might 
be needed by them is available.  Again, the front end files would have 
made it much earlier than we wanted for the backend files.

I also disagree with the assertion that " a file using timevars should 
include timevar.h IMO, even if it also includes df.h"  It could, but I 
don't see the value, and I doubt anyone really cares much.    If someone 
ever removes the only thing that does bring timevar.h, you simply add it 
then. That is just part of updating headers.   I'm sure before I run 
this patch not every file which uses timevar.h actually physically 
includes it.  This process will set us to a somewhat consistent state.

  Its simple enough to remove the ones that are redundant in an 
automated way, and very difficult to determine whether they not 
required, but contain content that is used.

The fully expanded canonical list looks something like this:

safe-ctype.h
filenames.h
libiberty.h
hwint.h
system.h
insn-modes.h
machmode.h
signop.h
wide-int.h
double-int.h
real.h
fixed-value.h
statistics.h
gtype-desc.h
ggc.h
vec.h
hashtab.h
inchash.h
mem-stats-traits.h
hash-traits.h
hash-map-traits.h
mem-stats.h
hash-map.h
hash-table.h
hash-set.h
line-map.h
input.h
is-a.h
memory-block.h
coretypes.h
options.h
tm.h
function.h
obstack.h
bitmap.h
sbitmap.h
basic-block.h
dominance.h
cfg.h
backend.h
insn-codes.h
hard-reg-set.h
target.h
genrtl.h
rtl.h
c-target.h
c-target-def.h
symtab.h
tree-core.h
tree-check.h
tree.h
cp-tree.h
c-common.h
c-tree.h
gfortran.h
tree-ssa-alias.h
gimple-expr.h
gimple.h
predict.h
cfghooks.h
regset.h
alloc-pool.h
timevar.hdf.h
tm_p.h
gimple-iterators.h
stringpool.h
tree-ssa-operands.h
gimple-ssa.h
tree-ssanames.h
tree-phinodes.h
ssa-iterators.h
ssa.h
expmed.h
insn-opinit.h
optabs-query.h
optabs-libfuncs.h
insn-config.h
optabs.h
regs.h
emit-rtl.h
ira.h
recog.h
ira-int.h
streamer-hooks.h
plugin-api.h
gcov-iov.h
gcov-io.h
wide-int-print.h
pretty-print.h
bversion.h
lto-streamer.h
data-streamer.h
tree-streamer.h
gimple-streamer.h




>
> Intentionally commented out?
>
>> +
>> + def process_ii (filen):
>> +   return process_include_info (filen, False, False)
>> +
>> + def process_ii_macro (filen):
>> +   return process_include_info (filen, True, False)
>> +
>> + def process_ii_src (filen):
>> +   return process_include_info (filen, False, True)
>> +
>> + def process_ii_macro_src (filen):
>> +   return process_include_info (filen, True, True)
>> +
>> + def ii_base (iinfo):
>> +   return iinfo[0]
>> +
>> + def ii_path (iinfo):
>> +   return iinfo[1]
>> +
>> + def ii_include_list (iinfo):
>> +   return iinfo[2]
>> +
>> + def ii_include_list_cond (iinfo):
>> +   return iinfo[3]
>> +
>> + def ii_include_list_non_cond (iinfo):
>> +   l = ii_include_list (iinfo)
>> +   for n in ii_include_list_cond (iinfo):
>> +     l.remove (n)
>> +   return l
>> +
>> + def ii_macro_consume (iinfo):
>> +   return iinfo[4]
>> +
>> + def ii_macro_define (iinfo):
>> +   return iinfo[5]
>> +
>> + def ii_src (iinfo):
>> +   return iinfo[6]
>> +
>> + def ii_src_line (iinfo):
>> +   return iinfo[7]
>
> That's a lot of little functions with pretty much no clue for the 
> reader what's going on. It looks like maybe there's an array where a 
> struct should have been used?
>

there once was a large comment at the start of process_include_info 
describing the return value vactor... they simply access it.  Im not 
sure where it went.  I will find and put the big comment back in.

Andrew

Re: [patch 4/3] Header file reduction - Tools for contrib

[Andrew MacLeod]

I went through and addressed the comments..  Just for info, a few replies:

>> +     # various front ends have to set GCC_DIAG_STYLE before 
>> including it
>> +     # for each file, we'll tailor where it belongs by looking at 
>> the dup
>> +     # list and seeing which file is included, and position it 
>> appropriately.
>
> From that comment it's not entirely clear how they are handled. Please 
> expand documentation of this mechanism.

I modified the comments in a couple of places to hopefully make it clearer.
>> + for i in order:
>> +   create_master_list (os.path.basename (i), False)
>
> I found myself wanting to pass True. The tool could use a "-v" flag.
>
I changed the existing -s flag to -v, and simply passed the value 
here...  Now you see the final list, as well as the list of where each 
one came from.
>> +   for line in logfile:
>> +     if len (line) > 21 and line[:21] in depstring:
>> +       if newinc:
>> +         incfrom = list()
>> +     newinc = False
>
> It looks like you are mixing tab and space indentation. For a language 
> like Python, that is absolutely scary. Please fix throughout (I think 
> only spaces is probably best).
>

vi is doing that automatically for me.. I will expandtabs everything.
>> + # compensate for this stupid warning that should be an error for
>> + # inlined templates
>> + def get_make_rc (rc, output):
>> +   rc = rc % 1280
>> +   if rc == 0:
>> +     # This is not considered a fatal error for a build!  /me rolls 
>> eyes
>> +     h = re.findall ("warning: inline function.*used but never 
>> defined", output)
>> +     if len(h) != 0:
>> +       rc = 1
>> +   return rc;
>
> What's this about?
I've updated the comment to be clearer.  Apparently its only a warning 
to use a template inline function with no definition. I suspect this is 
some oddball C++ thing :-).   Maybe it can be resolved at link time 
somehow?   Anyway,  what I found is that the return code from this is 0 
since its just a warning. SO the tool would remove the header file, and 
when I later try to link and build and object, it becomes a fatal link 
error with the function used but undefined.

It shows up when checking target builds since I only try to build the .o 
file there rather than build and link.   So the tool checks the output 
from the compilation, and if it sees this error, decided to be 
conservative and report it as a build error, and thus it will leave the 
header file in the source.

>
>> +   print " -a  : Show only files which*all*  listed files are included"
>
> Whitespace around *all*. Seems to happen quite often.

Yeah, that is very odd.   In the code here, there is a space in front of 
every single one of those. I simply changed all these to 'all' instead 
of '*all*'.

I'm also going to add a few more comments to functions in 
gcc-order-headers and reduce-headers, as well as utils.py

Andrew

Re: [patch 4/3] Header file reduction - Tools for contrib - second cut

[Andrew MacLeod]

[-- Attachment #1: Type: text/plain, Size: 932 bytes --]

Here's the latest version of the tools for a sub directory in contrib.   
I've handled all the feedback, except I have not fully commented the 
python code in the tools, nor followed any particular coding 
convention...   Documentation has been handled, and I've added some 
additional comments to the places which were noted as being unclear. Ive 
also removed all tabs from the source files.

Ive also updated show-headers slightly to be a little more 
error-resistant and to put some emphasis on any header files specified 
on the command as being of interest . (when there are 140 shown, it can 
be hard to find the one you are looking for sometimes)

Do we wish to impose anything in particular on the source for  tools 
going into this sub-directory of contrib? The other tools in contrib 
don't seem to have much in the way of coding standards.     I also 
wonder if anyone other than me will look at them much :-)

Andrew



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: header3.patch --]
[-- Type: text/x-patch; name="header3.patch", Size: 83193 bytes --]



headers/

	* README : New File.
	* count-headers : New File.
	* gcc-order-headers : New File.
	* graph-header-logs : New File.
	* graph-include-web : New File.
	* headerutils.py : New File.
	* included-by : New File.
	* reduce-headers : New File.
	* replace-header : New File.
	* show-headers : New File.

Index: headers/README
===================================================================
*** headers/README	(revision 0)
--- headers/README	(working copy)
***************
*** 0 ****
--- 1,283 ----
+ Quick start documentation for the header file utilities.  
+ 
+ This isn't a full breakdown of the tools, just they typical use scenarios.
+ 
+ - Each tool accepts -h to show it's usage.  Usually no parameters will also
+ trigger the help message.  Help may specify additional functionality to what is
+ listed here.
+ 
+ - For all tools, option format for specifying filenames must have no spaces
+ between the option and filename.
+ ie.:     tool -lfilename.h  target.h
+ 
+ - Many of the tools are required to be run from the core gcc source directory
+ containing coretypes.h.  Typically that is in gcc/gcc from a source checkout.
+ For these tools to work on files not in this directory, their path needs to be
+ specified on the command line.
+ ie.:     tool c/c-decl.c  lto/lto.c
+ 
+ - options can be intermixed with filenames anywhere on the command line
+ ie.   tool ssa.h rtl.h -a   is equivalent to 
+       tool ssa.h -a rtl.h
+ 
+ 
+ 
+ 
+ 
+ gcc-order-headers
+ -----------------
+   This will reorder any primary backend headers files known to the tool into a
+   canonical order which will resolve any hidden dependencies they may have.
+   Any unknown headers will simply be placed after the recognized files, and
+   retain the same relative ordering they had.
+  
+   This tool must be run in the core gcc source directory.
+ 
+   Simply execute the command listing any files you wish to process on the
+   command line.
+ 
+   Any files which are changed are output, and the original is saved with a
+   .bak extention.
+ 
+   ex.:     gcc-order-headers tree-ssa.c c/c-decl.c
+ 
+   -s will list all of the known headers in their canonical order. It does not
+   show which of those headers include other headers, just the final canonical
+   ordering.
+ 
+   if any header files are included within a conditional code block, the tool
+   will issue a message and not change the file.  When this happens, you can
+   manually inspect the file to determine if reordering it is actually OK.  Then
+   rerun the command with the -i option.  This will ignore the conditional error
+   condition and perform the re-ordering anyway.
+   
+   If any #include line has the beginning of a multi-line comment, it will also
+   refuse to process the file until that is resolved by terminating the comment
+   on the same line, or removing it.
+ 
+ 
+ show-headers
+ ------------
+   This will show the include structure for any given file. Each level of nesting
+   is indented, and when any duplicate headers are seen, they have their
+   duplicate number shown
+ 
+   -i may be used to specify alternate search directories for headers to parse.
+   -s specifies headers to look for and emphasize in the output.
+ 
+   This tool must be run in the core gcc source directory.
+ 
+   ex.: show-headers -sansidecl.h tree-ssa.c
+ 	tree-ssa.c
+ 	  config.h
+ 	    auto-host.h
+ 	    ansidecl.h  (1)               <<-------
+ 	  system.h
+ 	    safe-ctype.h
+ 	    filenames.h
+ 	      hashtab.h  (1)
+ 		ansidecl.h  (2)                <<-------
+ 	    libiberty.h
+ 	      ansidecl.h  (3)                <<-------
+ 	    hwint.h
+ 	  coretypes.h
+ 	    machmode.h  (1)
+ 	      insn-modes.h  (1)
+ 	    signop.h
+ 	  <...>
+ 
+ 
+ 
+ 
+ count-headers
+ -------------
+   simply count all the headers found in the specified files. A summary is 
+   printed showing occurrences from high to low.
+ 
+   ex.:    count-headers  tree*.c
+ 	    86 : coretypes.h
+ 	    86 : config.h
+ 	    86 : system.h
+ 	    86 : tree.h
+ 	    82 : backend.h
+ 	    80 : gimple.h
+ 	    72 : gimple-iterator.h
+ 	    70 : ssa.h
+ 	    68 : fold-const.h
+             <...>
+ 
+ 
+ 
+ included-by
+ -----------
+   This tool will search all the .c,.cc and .h files and output a list of files
+   which include the specified header(s).
+ 
+   A 4 level deep 'find' of all source files is performed from the current
+   directory and each of those is inspected for a #include of the specified
+   headers.  So expect a little bit of slowness.
+ 
+   -i limits the search to only other header files.
+   -c limits the search to .c and .cc files.
+   -a shows only source files which include all specified headers.
+   -f allows you to specify a file which contains a list of source files to
+      check rather than performing the much slower find command.
+ 
+   ex: included-by tree-vectorizer.h
+ 	config/aarch64/aarch64.c
+ 	config/i386/i386.c
+ 	config/rs6000/rs6000.c
+ 	tree-loop-distribution.c
+ 	tree-parloops.c
+ 	tree-ssa-loop-ivopts.c
+ 	tree-ssa-loop.c
+ 
+ 
+ 
+ 
+ replace-header
+ --------------
+   This tool simply replaces a single header file with one or more other headers.
+   -r specifies the include to replace, and one or more -f options specify the
+   replacement headers, in the order they occur.
+   
+   This is commonly used in conjunction with 'included-by' to change all 
+   occurrences of a header file to something else, or to insert new headers 
+   before or after.  
+ 
+   ex:  to insert #include "before.h" before every occurence of tree.h in all
+   .c and .cc source files:
+ 
+   replace-header -rtree.h -fbefore.h -ftree.h `included-by -c tree.h`
+ 
+ 
+ 
+ 
+ reduce-headers
+ --------------
+ 
+   This tool removes any header files which are not needed from a source file.
+ 
+   This tool must be run for the core gcc source directory, and requires either
+   a native build and sometimes target builds, depending on what you are trying
+   to reduce.
+ 
+   it is good practice to run 'gcc-order-headers' on a source file before trying
+   to reduce it.  This removes duplicates and performs some simplifications 
+   which reduce the chances of the reduction tool missing things.
+   
+   start with a completely bootstrapped native compiler.
+ 
+   Any desired target builds should be built in one directory using a modified
+   config-list.mk file which does not delete the build directory when it is done.
+   any target directories which do not successfully complete a 'make all-gcc'
+   may cause the tool to not reduce anything.
+   (todo - provide a config-list.mk that leaves successful target builds, but
+           deletes ones which do not compile)
+ 
+   The tool will examine all the target builds to determine which targets build
+   the file, and include those targets in the testing.
+   
+ 
+ 
+   The tool will analyze a source file and attempt to remove each non-conditional
+   header from last to first in the file.:
+     It will first attempt to build the native all-gcc target.
+     If that succeeds, it will attempt to build any target build .o files
+     If that succeeds, it will check to see if there are any conditional
+        compilation dependencies between this header file and the source file or
+        any header which have already been determined as non-removable.
+     If all these tests are passed, the header file is determined to be removable
+        and is removed from the source file.
+     This continues until all headers have been checked.
+   At this point, a bootstrap is attempted in the native build, and if that
+      passes the file is considered reduced.
+ 
+   Any files from the config subdirectory require target builds to be present
+   in order to proceed.
+ 
+   A small subset of targets has been determined to provide excellent coverage,
+   at least as of Aug 31/15 .  They were found by reducing all the files
+   contained in libbackend.a oer a full set of targets(207).  All conditions
+   which disallowed removal of a header file were triggered by one or more of
+   these targets.  They are also known to the tool.  When building targets it
+   will check those targets before the rest.  
+   This coverage can be achieved by building config-list.mk with :
+   LIST="aarch64-linux-gnu arm-netbsdelf avr-rtems c6x-elf epiphany-elf hppa2.0-hpux10.1 i686-mingw32crt i686-pc-msdosdjgpp mipsel-elf powerpc-eabisimaltivec rs6000-ibm-aix5.1.0 sh-superh-elf sparc64-elf spu-elf"
+ 
+   -b specifies the native bootstrapped build root directory
+   -t specifies a target build root directory that config-list.mk was run from
+   -f is used to limit the headers for consideration.
+ 
+   example:
+ 
+   mkdir gcc          // checkout gcc in subdir gcc
+   mdsir build        // boostrap gcc in subdir build
+   mkdir target       // create target directory and run config-list.mk
+   cd gcc/gcc
+ 
+   reduce-headers -b../../build -t../../targets -falias.h -fexpr.h tree*.c  (1)
+        #  This will attempt to remove only alias.h and expr.h from tree*.c
+ 
+   reduce-headers -b../../build -t../../targets tree-ssa-live.c
+        #  This will attempt to remove all header files from tree-ssa-live.c
+   
+ 
+   the tool will generate a number of log files:
+ 
+     reduce-headers.log : All compilation failures from attempted reductions.
+     reduce-headers.sum : One line summary of what happened to each source file.
+ 
+   (All the remaining logs are appended to, so if the tool is run multiple times
+   these files are just added to. You must physically remove them yourself in
+   order to reset the logs.)
+ 
+     reduce-headers-kept.log: List of all the successful compiles that were
+                              ignored because of conditional macro dependencies
+ 			     and why it thinks that is the case
+     $src.c.log  : for each failed header removal, the compilation
+ 		  messages as to why it failed.
+     $header.h.log: The same log is put into the relevant header log as well.
+ 
+ 
+ a sample output from ira.c.log:
+ 
+ Compilation failed:
+  for shrink-wrap.h:
+ 
+  ============================================
+  /gcc/2015-09-09/gcc/gcc/ira.c: In function ‘bool split_live_ranges_for_shrink_wrap()’:
+  /gcc/2015-09-09/gcc/gcc/ira.c:4839:8: error: ‘SHRINK_WRAPPING_ENABLED’ was not declared in this scope
+     if (!SHRINK_WRAPPING_ENABLED)
+             ^
+ 	    make: *** [ira.o] Error 1
+ 
+ 
+ the same message would be put into shrink-wrap.h.log.
+ 
+ 
+ 
+ graph-header-logs
+ -----------------
+   This tool will parse all the messages from the .C files, looking for failures
+   that show up in other headers...  meaning there is a compilation dependency
+   between the 2 header files. 
+ 
+   The tool will aggregate all these and generate a graph of the dependencies
+   exposed during compilation.  Red lines indicate dependencies that are
+   present because a header file physically includes another file. Black lines
+   represent data dependencies causing compilation failures if the header is
+   not present.
+ 
+   ex.: graph-header-logs *.c.log
+ 
+ 
+ 
+ graph-include-web
+ -----------------
+   This tool can be used to visualize the include structure in files.  It is
+   rapidly turned useless if you specify too many things, but it can be 
+   useful for finding cycles and redundancies, or simply to see what a single
+   file looks like.
+ 
+   ex.: graph-include-web tree.c
Index: headers/count-headers
===================================================================
*** headers/count-headers	(revision 0)
--- headers/count-headers	(working copy)
***************
*** 0 ****
--- 1,58 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ usage = False
+ src = list ()
+ flist = { }
+ process_h = True
+ process_c = True
+ verbose = False
+ all_inc = True
+ level = 0
+ 
+ only_use_list = list ()
+ 
+ for x in sys.argv[1:]:
+   if x[0:2] == "-h":
+     usage = True
+   else:
+     src.append (x)
+ 
+ 
+ if not usage and len (src) > 0:
+   incl = { }
+   for fn in src:
+     src = readwholefile (fn)
+     dup = { }
+     for line in src:
+       d = find_pound_include (line, True, True)
+       if d != "" and d[-2:] ==".h":
+         if dup.get (d) == None:
+           if incl.get (d) == None:
+             incl[d] = 1
+           else:
+             incl[d] = incl[d]+ 1
+           dup[d] = 1
+ 
+   l = list ()
+   for i in incl:
+     l.append ((incl[i], i))
+   l.sort (key=lambda tup:tup[0], reverse=True)
+ 
+   for f in l:
+     print str (f[0]) + " : " + f[1]
+ 
+ else:
+   print "count-headers file1 [filen]"
+   print "Count the number of occurrences of all includes across all listed files"
+ 
+  
+ 
+ 
+ 
+ 

Property changes on: headers/count-headers
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: headers/gcc-order-headers
===================================================================
*** headers/gcc-order-headers	(revision 0)
--- headers/gcc-order-headers	(working copy)
***************
*** 0 ****
--- 1,397 ----
+ #! /usr/bin/python2
+ import os
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ import Queue
+ 
+ file_list = list ()
+ usage = False
+ 
+ ignore_conditional = False
+ 
+ order = [
+   "system.h",
+   "coretypes.h",
+   "backend.h",
+   "target.h",
+   "rtl.h",
+   "c-family/c-target.h",
+   "c-family/c-target-def.h",
+   "tree.h",
+   "cp/cp-tree.h",
+   "c-family/c-common.h",  # these must come before diagnostic.h
+   "c/c-tree.h",
+   "fortran/gfortran.h",
+   "gimple.h",
+   "cfghooks.h",
+   "df.h",
+   "tm_p.h",
+   "gimple-iterators.h",
+   "ssa.h",
+   "expmed.h",
+   "optabs.h",
+   "regs.h",
+   "ira.h",
+   "ira-int.h",
+   "gimple-streamer.h"
+ 
+ ]
+ 
+ exclude_special = [  "bversion.h", "obstack.h", "insn-codes.h", "hooks.h" ]
+ 
+ # includes is a dictionary indexed by a header files basename.
+ # it consists of a 2 element tuple:
+ # [0] - Name of header file which included this header.
+ # [1] - vector of header file names included by this file.
+ 
+ includes = { }
+ 
+ # when a header is included multiple times, indexing this dictionary will
+ # return a vector of all the headers which included it.
+ dups = { }
+ 
+ # When creating the master list, do not descend into these files for what 
+ # they include. Simply put the file itself in the list.  This is primarily
+ # required because the front end files inlcude orders tend to be at odds with
+ # the order of middle end files, and its impossible to synchronize them.\
+ # They are ordered such that everything resolves properly.
+ exclude_processing = [ "tree-vectorizer.h" , "c-target.h", "c-target-def.h", "cp-tree.h", "c-common.h", "c-tree.h", "gfortran.h" ]
+ 
+ master_list = list ()
+ # where include file comes from in src
+ h_from = { }
+ 
+ # create the master ordering list... this is the desired order of headers
+ def create_master_list (fn, verbose):
+   if fn not in exclude_processing:
+     for x in includes[fn][1]:
+       create_master_list (x, verbose)
+   if not fn in master_list:
+     # Don't put diagnostic*.h into the ordering list. It is special since
+     # various front ends have to set GCC_DIAG_STYLE before including it.
+     # for each file, we'll tailor where it belongs by looking at the include
+     # list and determine its position appropriately.
+     if fn != "diagnostic.h" and fn != "diagnostic-core.h":
+       master_list.append (fn)
+       if (verbose):
+         print fn + "      included by: " + includes[fn][0]
+ 
+ 
+ 
+ def print_dups ():
+   if dups:
+     print "\nduplicated includes"
+   for i in dups:
+     string =  "dup : " + i + " : "
+     string += includes[i][0] 
+     for i2 in dups[i]:
+       string += ", "+i2
+     print string
+ 
+ 
+ def process_known_dups ():
+   # rtl.h gets tagged as a duplicate includer for all of coretypes.h, but that
+   # is really for only generator files
+   rtl_remove = includes["coretypes.h"][1] + ["statistics.h", "vec.h"]
+   if dups:
+     for i in rtl_remove:
+       if dups[i] and "rtl.h" in dups[i]:
+         dups[i].remove("rtl.h")
+       if not dups[i]:
+         dups.pop (i, None)
+ 
+   # make sure diagnostic.h is the owner of diagnostic-core.h
+   if includes["diagnostic-core.h"][0] != "diagnostic.h":
+     dups["diagnostic-core.h"].append (includes["diagnostic-core.h"][0])
+     includes["diagnostic-core.h"] = ("diagnostic.h", includes["diagnostic-core.h"][1])
+ 
+ # This function scans back thorugh the list of headers which included other
+ # headers to determine what file in HEADER_LIST brought 'HEADER' in.
+ def indirectly_included (header, header_list):
+   nm = os.path.basename (header)
+   while nm and includes.get(nm):
+     if includes[nm][0] in header_list:
+       return includes[nm][0]
+     nm = includes[nm][0]
+ 
+   # diagnostic.h and diagnostic-core.h may not show up because we removed them
+   # from the header list to manually position in an appropriate place. They have
+   # specific requirements that they need to occur after certain FE files which
+   # may overide the definition of GCC_DIAG_STYLE.
+   # Check the dup list for whete they may have been included from and return
+   # that header.
+   if header == "diagnostic-core.h":
+     if dups.get("diagnostic-core.h"):
+       for f in dups["diagnostic-core.h"]:
+         if f in header_list:
+           return f
+     else:
+       if header in header_list:
+         return header
+     # Now check if diagnostics is included indirectly anywhere
+     header = "diagnostic.h"
+ 
+   if header == "diagnostic.h":
+     if dups.get("diagnostic.h"):
+       for f in dups["diagnostic.h"]:
+         if f in header_list:
+           return f
+     else:
+       if header in header_list:
+         return header 
+ 
+   return ""
+ 
+ 
+ # This function will take a list of headers from a source file and return 
+ # the desired new new order of the canonical headers in DESIRED_ORDER. 
+ def get_new_order (src_h, desired_order):
+   new_order = list ()
+   for h in desired_order:
+     if h in master_list:
+       # Create the list of nested headers which included this file.
+       iclist = list ()
+       ib = includes[h][0]
+       while ib:
+         iclist.insert(0, ib)
+         ib = includes[ib][0]
+       if iclist:
+         for x in iclist:
+           # If header is in the source code, and we are allowed to look inside
+           if x in src_h and x not in exclude_processing:
+             if x not in new_order and x[:10] != "diagnostic" and h not in exclude_special:
+               new_order.append (x)
+               break;
+       else:
+         if h not in new_order:
+           new_order.append (h)
+ 
+   f = ""
+   if "diagnostic.h" in src_h:
+     f = "diagnostic.h"
+   elif "diagnostic-core.h" in src_h:
+     f = "diagnostic-core.h"
+ 
+  
+   # If either diagnostic header was directly included in the main file, check to
+   # see if its already included indirectly, or whether we need to add it to the
+   # end of the canonically orders headers.
+   if f:
+     ii = indirectly_included (f, src_h)
+     if not ii or ii == f:
+       new_order.append (f)
+ 
+   return new_order
+         
+     
+ 
+ # stack of files to process
+ process_stack = list ()
+ 
+ def process_one (info):
+   i = info[0]
+   owner = info[1]
+   name = os.path.basename(i)
+   if os.path.exists (i):
+     if includes.get(name) == None:
+       l = find_unique_include_list (i)
+       # create a list which has just basenames in it
+       new_list = list ()
+       for x in l:
+         new_list.append (os.path.basename (x))
+         process_stack.append((x, name))
+       includes[name] = (owner, new_list)
+     elif owner:
+       if dups.get(name) == None:
+         dups[name] = [ owner ]
+       else:
+         dups[name].append (owner)
+   else:
+     # seed tm.h with options.h since it is a build file and won't be seen. 
+     if not includes.get(name):
+       if name == "tm.h":
+         includes[name] = (owner, [ "options.h" ])
+         includes["options.h"] = ("tm.h", list ())
+       else:
+         includes[name] = (owner, list ())
+ 
+ 
+ show_master = False
+ 
+ for arg in sys.argv[1:]:
+   if arg[0:1] == "-":
+     if arg[0:2] == "-h":
+       usage = True
+     elif arg[0:2] == "-i":
+       ignore_conditional = True
+     elif arg[0:2] == "-v":
+       show_master = True
+     else:
+       print "Error: unrecognized option " + arg
+   elif os.path.exists(arg):
+     file_list.append (arg)
+   else:
+     print "Error: file " + arg + " Does not exist."
+     usage = True
+ 
+ if not file_list and not show_master:
+   usage = True
+ 
+ if not usage and not os.path.exists ("coretypes.h"):
+   usage = True
+   print "Error: Must run command in main gcc source directory containing coretypes.h\n"
+ 
+ # process diagnostic.h first.. it's special since GCC_DIAG_STYLE can be
+ # overridden by languages, but must be done so by a file included BEFORE it.
+ # so make sure it isn't seen as included by one of those files by making it 
+ # appear to be included by the src file.
+ process_stack.insert (0, ("diagnostic.h", ""))
+ 
+ # Add the list of files in reverse order since it is processed as a stack later
+ for i in order:
+   process_stack.insert (0, (i, "") )
+ 
+ # build up the library of what header files include what other files.
+ while process_stack:
+   info = process_stack.pop ()
+   process_one (info)
+ 
+ # Now create the master ordering list
+ for i in order:
+   create_master_list (os.path.basename (i), show_master)
+ 
+ # handle warts in the duplicate list
+ process_known_dups ()
+ desired_order = master_list
+ 
+ if show_master:
+   print " Canonical order of gcc include files: "
+   for x in master_list:
+     print x
+   print " "
+ 
+ if usage:
+   print "gcc-order-headers [-i] [-v] file1 [filen]"
+   print "    Ensures gcc's headers files are included in a normalized form with"
+   print "    redundant headers removed.  The original files are saved in filename.bak"
+   print "    Outputs a list of files which changed."
+   print " -i ignore conditional compilation."
+   print "    Use after examining the file to be sure includes within #ifs are safe"
+   print "    Any headers within conditional sections will be ignored."
+   print " -v Show the canonical order of known headers"
+   sys.exit(0)
+ 
+ 
+ didnt_do = list ()
+ 
+ for fn in file_list:
+   nest = 0
+   src_h = list ()
+   src_line = { }
+ 
+   master_list = list ()
+ 
+   includes = { }
+   dups = { }
+ 
+   iinfo = process_ii_src (fn)
+   src = ii_src (iinfo)
+   include_list = ii_include_list (iinfo)
+ 
+   if ii_include_list_cond (iinfo):
+     if not ignore_conditional:
+       print fn + ": Cannot process due to conditional compilation of includes"
+       didnt_do.append (fn)
+       src = list ()
+ 
+   if not src:
+     continue
+ 
+   process_stack = list ()
+   # prime the stack with headers in the main ordering list so we get them in
+   # this order.
+   for d in order:
+     if d in include_list:
+       process_stack.insert (0, (d, ""))
+ 
+   for d in include_list:
+       nm = os.path.basename(d)
+       src_h.append (nm)
+       iname = d
+       iname2 = os.path.dirname (fn) + "/" + d
+       if not os.path.exists (d) and os.path.exists (iname2):
+         iname = iname2
+       if iname not in process_stack:
+         process_stack.insert (0, (iname, ""))
+       src_line[nm] = ii_src_line(iinfo)[d]
+       if src_line[nm].find("/*") != -1 and src_line[nm].find("*/") == -1:
+         # this means we have a multi line comment, abort!'
+         print fn + ": Cannot process due to a multi-line comment :"
+         print "        " + src_line[nm]
+         if fn not in didnt_do:
+           didnt_do.append (fn)
+         src = list ()
+ 
+   if not src:
+     continue
+ 
+   # Now create the list of includes as seen by the source file.
+   while process_stack:
+     info = process_stack.pop ()
+     process_one (info)
+  
+   for i in include_list:
+     create_master_list (os.path.basename (i), False)
+ 
+   new_src = list ()
+   header_added = list ()
+   new_order = list ()
+   for line in src:
+     d = find_pound_include (line, True, True)
+     if not d or d[-2:] != ".h":
+       new_src.append (line)
+     else:
+       if d == order[0] and not new_order:
+         new_order = get_new_order (src_h, desired_order)
+         for i in new_order:
+           new_src.append (src_line[i])
+           # if not seen, add it.
+           if i not in header_added:
+             header_added.append (i)
+       else:
+         nm = os.path.basename(d)
+         if nm not in header_added:
+           iby = indirectly_included (nm, src_h)
+           if not iby:
+             new_src.append (line)
+             header_added.append (nm)
+ 
+   if src != new_src:
+     os.rename (fn, fn + ".bak")
+     fl = open(fn,"w")
+     for line in new_src:
+       fl.write (line)
+     fl.close ()
+     print fn 
+ 
+  
+ if didnt_do:
+   print "\n\n Did not process the following files due to conditional dependencies:"
+   str = ""
+   for x in didnt_do:
+     str += x + " "
+   print str
+   print "\n"
+   print "Please examine to see if they are safe to process, and re-try with -i. "
+   print "Safeness is determined by checking whether any of the reordered headers are"
+   print "within a conditional and could be hauled out of the conditional, thus changing"
+   print "what the compiler will see."
+   print "Multi-line comments after a #include can also cause failuer, they must be turned"
+   print "into single line comments or removed."
+ 
+ 
+ 
+ 

Property changes on: headers/gcc-order-headers
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: headers/graph-header-logs
===================================================================
*** headers/graph-header-logs	(revision 0)
--- headers/graph-header-logs	(working copy)
***************
*** 0 ****
--- 1,227 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ header_roots = { }
+ extra_edges = list()
+ verbose = False
+ verbosity = 0
+ nodes = list()
+ 
+ def unpretty (name):
+   if name[-2:] == "_h":
+     name = name[:-2] + ".h"
+   return name.replace("_", "-")
+ 
+ def pretty_name (name):
+   name = os.path.basename (name)
+   return name.replace(".","_").replace("-","_").replace("/","_").replace("+","_");
+ 
+ depstring = ("In file included from", "                 from")
+ 
+ # indentation indicates nesting levels of included files
+ ignore = [ "coretypes_h",
+              "machmode_h",
+              "signop_h",
+              "wide_int_h",
+              "double_int_h",
+              "real_h",
+              "fixed_value_h",
+              "hash_table_h",
+                "statistics_h",
+                "ggc_h",
+                "vec_h",
+                "hashtab_h",
+                "inchash_h",
+                "mem_stats_traits_h",
+                "hash_map_traits_h",
+                "mem_stats_h",
+                "hash_map_h",
+              "hash_set_h",
+              "input_h",
+                "line_map_h",
+              "is_a_h",
+            "system_h",
+            "config_h" ]
+ 
+ def process_log_file (header, logfile):
+   if header_roots.get (header) != None:
+     print "Error: already processed log file: " + header + ".log"
+     return
+   hname = pretty_name (header)
+   header_roots[hname] = { }
+   
+   sline = list();
+   incfrom = list()
+   newinc = True
+   for line in logfile:
+     if len (line) > 21 and line[:21] in depstring:
+       if newinc:
+         incfrom = list()
+         newinc = False
+       fn = re.findall(ur".*/(.*?):", line)
+       if len(fn) != 1:
+         continue
+       if fn[0][-2:] != ".h":
+         continue
+       n = pretty_name (fn[0])
+       if n not in ignore:
+         incfrom.append (n)
+       continue
+     newinc = True
+     note = re.findall (ur"^.*note: (.*)", line)
+     if len(note) > 0:
+       sline.append (("note", note[0]))
+     else:
+       err_msg = re.findall (ur"^.*: error: (.*)", line)
+       if len(err_msg) == 1:
+         msg = err_msg[0]
+         if (len (re.findall("error: forward declaration", line))) != 0:
+           continue
+         path = re.findall (ur"^(.*?):.*error: ", line)
+         if len(path) != 1:
+           continue
+         if path[0][-2:] != ".h":
+           continue
+         fname = pretty_name (path[0])
+         if fname in ignore or fname[0:3] == "gt_":
+           continue
+         sline.append (("error", msg, fname, incfrom))
+ 
+   print str(len(sline)) + " lines to process"
+   lastline = "note"
+   for line in sline:
+     if line[0] != "note" and lastline[0] == "error":
+       fname = lastline[2]
+       msg = lastline[1]
+       incfrom = lastline[3]
+       string = ""
+       ofname = fname
+       if len(incfrom) != 0:
+         for t in incfrom:
+           string = string + t + " : "
+           ee = (fname, t)
+           if ee not in extra_edges:
+             extra_edges.append (ee)
+           fname = t
+           print string
+ 
+       if hname not in nodes:
+         nodes.append(hname)
+       if fname not in nodes:
+         nodes.append (ofname)
+       for y in incfrom:
+         if y not in nodes:
+           nodes.append (y)
+ 
+ 
+       if header_roots[hname].get(fname) == None:
+         header_roots[hname][fname] = list()
+       if msg not in header_roots[hname][fname]:
+         print string + ofname + " : " +msg
+         header_roots[hname][fname].append (msg)
+     lastline = line;
+ 
+ 
+ dotname = "graph.dot"
+ graphname = "graph.png"
+ 
+ 
+ def build_dot_file (file_list):
+   output = open(dotname, "w")
+   output.write ("digraph incweb {\n");
+   for x in file_list:
+     if os.path.exists (x) and x[-4:] == ".log":
+       header =  x[:-4]
+       logfile = open(x).read().splitlines()
+       process_log_file (header, logfile)
+     elif os.path.exists (x + ".log"):
+       logfile = open(x + ".log").read().splitlines()
+       process_log_file (x, logfile)
+ 
+   for n in nodes:
+     fn = unpretty(n)
+     label = n + " [ label = \"" + fn  + "\" ];"
+     output.write (label + "\n")
+     if os.path.exists (fn):
+       h = open(fn).read().splitlines()
+       for l in h:
+         t = find_pound_include (l, True, False)
+         if t != "":
+           t = pretty_name (t)
+           if t in ignore or t[-2:] != "_h":
+             continue
+           if t not in nodes:
+             nodes.append (t)
+           ee = (t, n)
+           if ee not in extra_edges:
+             extra_edges.append (ee)
+ 
+   depcount = list()
+   for h in header_roots:
+     for dep in header_roots[h]:
+       label = " [ label = "+ str(len(header_roots[h][dep])) + " ];"
+       string = h + " -> " + dep + label
+       output.write (string + "\n");
+       if verbose:
+         depcount.append ((h, dep, len(header_roots[h][dep])))
+ 
+   for ee in extra_edges:
+     string = ee[0] + " -> " + ee[1] + "[ color=red ];"
+     output.write (string + "\n");
+ 
+   
+   if verbose:
+     depcount.sort(key=lambda tup:tup[2])
+     for x in depcount:
+       print " ("+str(x[2])+ ") : " + x[0] + " -> " + x[1]
+       if (x[2] <= verbosity):
+         for l in header_roots[x[0]][x[1]]:
+           print "            " + l
+ 
+   output.write ("}\n");
+ 
+ 
+ files = list()
+ dohelp = False
+ edge_thresh = 0
+ for arg in sys.argv[1:]:
+   if arg[0:2] == "-o":
+     dotname = arg[2:]+".dot"
+     graphname = arg[2:]+".png"
+   elif arg[0:2] == "-h":
+     dohelp = True
+   elif arg[0:2] == "-v":
+     verbose = True
+     if len(arg) > 2:
+       verbosity = int (arg[2:])
+       if (verbosity == 9):
+         verbosity = 9999
+   elif arg[0:1] == "-":
+     print "Unrecognized option " + arg
+     dohelp = True
+   else:
+     files.append (arg)
+     
+ if len(sys.argv) == 1:
+   dohelp = True
+ 
+ if dohelp:
+   print "Parses the log files from the reduce-headers tool to generate"
+   print "dependency graphs for the include web for specified files."
+   print "Usage:  [-nnum] [-h] [-v[n]] [-ooutput] file1 [[file2] ... [filen]]"
+   print "       -ooutput : Specifies output to output.dot and output.png"
+   print "                  Defaults to 'graph.dot and graph.png"
+   print "       -vn : verbose mode, shows the number of connections, and if n"
+   print "             is specified, show the messages if # < n. 9 is infinity"
+   print "       -h : help"
+ else:
+   print files
+   build_dot_file (files)
+   os.system ("dot -Tpng " + dotname + " -o" + graphname)
+ 
+ 

Property changes on: headers/graph-header-logs
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: headers/graph-include-web
===================================================================
*** headers/graph-include-web	(revision 0)
--- headers/graph-include-web	(working copy)
***************
*** 0 ****
--- 1,122 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ def pretty_name (name):
+   return name.replace(".","_").replace("-","_").replace("/","_").replace("+","_");
+ 
+ 
+ include_files = list()
+ edges = 0
+ one_c = False
+ clink = list()
+ noterm = False
+ 
+ def build_inclist (output, filen):
+   global edges
+   global one_c
+   global clink
+   global noterm
+   inc = build_include_list (filen)
+   if one_c and filen[-2:] == ".c":
+     pn = "all_c"
+   else:
+     pn = pretty_name(filen)
+   for nm in inc:
+     if pn == "all_c":
+       if nm not in clink:
+         if len(build_include_list(nm)) != 0 or not noterm:
+           output.write (pretty_name(nm) + " -> " + pn + ";\n")
+           edges = edges + 1
+           if nm not in include_files:
+             include_files.append(nm)
+         clink.append (nm)
+     else:
+       output.write (pretty_name(nm) + " -> " + pn + ";\n")
+       edges = edges + 1
+       if nm not in include_files:
+         include_files.append(nm)
+   return len(inc) == 0
+ 
+ dotname = "graph.dot"
+ graphname = "graph.png"
+ 
+ def build_dot_file (file_list):
+   global one_c
+   output = open(dotname, "w")
+   output.write ("digraph incweb {\n");
+   if one_c:
+     output.write ("all_c [shape=box];\n");
+   for x in file_list:
+     if x[-2:] == ".h":
+       include_files.append (x)
+     elif os.path.exists (x):
+       build_inclist (output, x)
+       if not one_c:
+         output.write (pretty_name (x) + "[shape=box];\n")
+ 
+   for x in include_files:
+     term = build_inclist (output, x)
+     if term:
+       output.write (pretty_name(x) + " [style=filled];\n")
+ 
+   output.write ("}\n");
+ 
+ 
+ files = list()
+ dohelp = False
+ edge_thresh = 0
+ for arg in sys.argv[1:]:
+   if arg[0:2] == "-o":
+     dotname = arg[2:]+".dot"
+     graphname = arg[2:]+".png"
+   elif arg[0:2] == "-h":
+     dohelp = True
+   elif arg[0:2] == "-a":
+     one_c = True
+     if arg[0:3] == "-at":
+       noterm = True
+   elif arg[0:2] == "-f":
+     if not os.path.exists (arg[2:]):
+       print "Option " + arg +" doesn't specify a proper file"
+       dohelp = True
+     else:
+       sfile = open (arg[2:], "r")
+       srcdata = sfile.readlines()
+       sfile.close()
+       for x in srcdata:
+         files.append(x.rstrip())
+   elif arg[0:2] == "-n":
+     edge_thresh = int (arg[2:])
+   elif arg[0:1] == "-":
+     print "Unrecognized option " + arg
+     dohelp = True
+   else:
+     files.append (arg)
+     
+ if len(sys.argv) == 1:
+   dohelp = True
+ 
+ if dohelp:
+   print "Generates a graph of the include web for specified files."
+   print "Usage:  [-finput_file] [-h] [-ooutput] [file1 ... [filen]]"
+   print "  -finput_file : Input file containing a list of files to process."
+   print "  -ooutput : Specifies output to output.dot and output.png."
+   print "             defaults to graph.dot and graph.png."
+   print "  -nnum : Specifies the # of edges beyond which sfdp is invoked. def=0."
+   print "  -a : Aggregate all .c files to 1 file.  Shows only include web."
+   print "  -at : Aggregate, but don't include terminal.h to .c links."
+   print "  -h : Print this help."
+ else:
+   print files
+   build_dot_file (files)
+   if edges > edge_thresh:
+     os.system ("sfdp -Tpng " + dotname + " -o" + graphname)
+   else:
+     os.system ("dot -Tpng " + dotname + " -o" + graphname)
+ 
+ 

Property changes on: headers/graph-include-web
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: headers/headerutils.py
===================================================================
*** headers/headerutils.py	(revision 0)
--- headers/headerutils.py	(working copy)
***************
*** 0 ****
--- 1,523 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ import subprocess
+ import shutil
+ import pickle
+ 
+ import multiprocessing 
+ 
+ def find_pound_include (line, use_outside, use_slash):
+   inc = re.findall (ur"^\s*#\s*include\s*\"(.+?)\"", line)
+   if len(inc) == 1:
+     nm = inc[0]
+     if use_outside or os.path.exists (nm):
+       if use_slash or '/' not in nm:
+         return nm
+   return ""
+ 
+ def find_system_include (line):
+   inc = re.findall (ur"^\s*#\s*include\s*<(.+?)>", line)
+   if len(inc) == 1:
+     return inc[0]
+   return ""
+   
+ def find_pound_define (line):
+   inc = re.findall (ur"^\s*#\s*define ([A-Za-z0-9_]+)", line)
+   if len(inc) != 0:
+     if len(inc) > 1:
+       print "What? more than 1 match in #define??"
+       print inc
+       sys.exit(5)
+     return inc[0];
+   return ""
+ 
+ def is_pound_if (line):
+   inc = re.findall ("^\s*#\s*if\s", line)
+   if not inc:
+     inc = re.findall ("^\s*#\s*if[n]?def\s", line)
+   if inc:
+     return True
+   return False
+ 
+ def is_pound_endif (line):
+   inc = re.findall ("^\s*#\s*endif", line)
+   if inc:
+     return True
+   return False
+ 
+ def find_pound_if (line):
+   inc = re.findall (ur"^\s*#\s*if\s+(.*)", line)
+   if len(inc) == 0:
+     inc = re.findall (ur"^\s*#\s*elif\s+(.*)", line)
+   if len(inc) > 0:
+     inc2 = re.findall (ur"defined\s*\((.+?)\)", inc[0])
+     inc3 = re.findall (ur"defined\s+([a-zA-Z0-9_]+)", inc[0])
+     for yy in inc3:
+       inc2.append (yy)
+     return inc2
+   else:
+     inc = re.findall (ur"^\s*#\s*ifdef\s(.*)", line)
+     if len(inc) == 0:
+       inc = re.findall (ur"^\s*#\s*ifndef\s(.*)", line)
+     if len(inc) > 0:
+       inc2 = re.findall ("[A-Za-z_][A-Za-z_0-9]*", inc[0])
+       return inc2
+   if len(inc) == 0:
+     return list ()
+   print "WTF. more than one line returned for find_pound_if"
+   print inc
+   sys.exit(5)
+ 
+ 
+ # IINFO - this is a vector of include information. It consists of 7 elements.
+ # [0] - base name of the file
+ # [1] - path leading to this file.
+ # [2] - orderd list of all headers directly included by this file.
+ # [3] - Ordered list of any headers included within condionally compiled code.
+ #       headers files are expected to have all includes one level deep due to
+ #       the omnipresent guards at the top of the file.  
+ # [4] - List of all macros which are consumed (used) within this file.
+ # [5] - list of all macros which may be defined in this file.
+ # [6] - The source code for this file, if cached.
+ # [7] - line number info for any headers in the source file.  Indexed by base
+ #       name, returning the line the include is on.
+ 
+ empty_iinfo =  ("", "", list(), list(), list(), list(), list())
+ 
+ # This function will process a file and extract interesting information.
+ # DO_MACROS indicates whether macros defined and used should be recorded.
+ # KEEP_SRC indicates the source for the file should be cached.
+ def process_include_info (filen, do_macros, keep_src):
+   header = False
+   if not os.path.exists (filen):
+     return empty_iinfo
+ 
+   sfile = open (filen, "r");
+   data = sfile.readlines()
+   sfile.close()
+ 
+   # Ignore the initial #ifdef HEADER_H in header files
+   if filen[-2:] == ".h":
+     nest = -1
+     header = True
+   else:
+     nest = 0
+ 
+   macout = list ()
+   macin = list()
+   incl = list()
+   cond_incl = list()
+   src_line = { }
+   guard = ""
+ 
+   for line in (data):
+     if is_pound_if (line):
+       nest += 1
+     elif is_pound_endif (line):
+       nest -= 1
+ 
+     nm = find_pound_include (line, True, True)
+     if nm != "" and nm not in incl and nm[-2:] == ".h":
+       incl.append (nm)
+       if nest > 0:
+         cond_incl.append (nm)
+       if keep_src:
+         src_line[nm] = line
+       continue
+ 
+     if do_macros:
+       d = find_pound_define (line)
+       if d:
+         if d not in macout:
+           macout.append (d);
+           continue
+ 
+       d = find_pound_if (line)
+       if d:
+         # The first #if in a header file should be the guard
+         if header and len (d) == 1 and guard == "":
+           if d[0][-2:] == "_H":
+             guard = d
+           else:
+             guard = "Guess there was no guard..."
+         else:
+           for mac in d:
+             if mac != "defined" and mac not in macin:
+               macin.append (mac);
+ 
+   if not keep_src:
+     data = list()
+ 
+   return (os.path.basename (filen), os.path.dirname (filen), incl, cond_incl,
+           macin, macout, data, src_line)
+ 
+ # Extract header info, but no macros or source code.
+ def process_ii (filen):
+   return process_include_info (filen, False, False)
+ 
+ # Extract header information, and collect macro information.
+ def process_ii_macro (filen):
+   return process_include_info (filen, True, False)
+ 
+ # Extract header information, cache the source lines.
+ def process_ii_src (filen):
+   return process_include_info (filen, False, True)
+ 
+ # Extract header information, coolewc macro info and cache the source lines.
+ def process_ii_macro_src (filen):
+   return process_include_info (filen, True, True)
+ 
+ 
+ def ii_base (iinfo):
+   return iinfo[0]
+ 
+ def ii_path (iinfo):
+   return iinfo[1]
+ 
+ def ii_include_list (iinfo):
+   return iinfo[2]
+ 
+ def ii_include_list_cond (iinfo):
+   return iinfo[3]
+ 
+ def ii_include_list_non_cond (iinfo):
+   l = ii_include_list (iinfo)
+   for n in ii_include_list_cond (iinfo):
+     l.remove (n)
+   return l
+ 
+ def ii_macro_consume (iinfo):
+   return iinfo[4]
+   
+ def ii_macro_define (iinfo):
+   return iinfo[5]
+ 
+ def ii_src (iinfo):
+   return iinfo[6]
+ 
+ def ii_src_line (iinfo):
+   return iinfo[7]
+ 
+ def ii_read (fname):
+   f = open (fname, 'rb')
+   incl = pickle.load (f)
+   consumes = pickle.load (f)
+   defines = pickle.load (f)
+   obj = (fname,fname,incl,list(), list(), consumes, defines, list(), list())
+   return obj
+ 
+ def ii_write (fname, obj):
+   f = open (fname, 'wb')
+   pickle.dump (obj[2], f)
+   pickle.dump (obj[4], f)
+   pickle.dump (obj[5], f)
+   f.close ()
+ 
+ 
+ # Find files matching pattern NAME, return in a list.
+ # CURRENT is True if you want to include the current directory
+ # DEEPER is True if you want to search 3 levels below the current directory
+ # any files with testsuite diurectories are ignored
+ 
+ def find_gcc_files (name, current, deeper):
+   files = list()
+   command = ""
+   if current:
+     if not deeper:
+       command = "find -maxdepth 1 -name " + name + " -not -path \"./testsuite/*\""
+     else:
+       command = "find -maxdepth 4 -name " + name + " -not -path \"./testsuite/*\""
+   else:
+     if deeper:
+       command = "find -maxdepth 4 -mindepth 2 -name " + name + " -not -path \"./testsuite/*\""
+ 
+   if command != "":
+     f = os.popen (command)
+     for x in f:
+       if x[0] == ".":
+         fn = x.rstrip()[2:]
+       else:
+         fn = x
+       files.append(fn)
+ 
+   return files
+ 
+ # find the list of unique include names found in a file.
+ def find_unique_include_list_src (data):
+   found = list ()
+   for line in data:
+     d = find_pound_include (line, True, True)
+     if d and d not in found and d[-2:] == ".h":
+       found.append (d)
+   return found
+ 
+ # find the list of unique include names found in a file.
+ def find_unique_include_list (filen):
+   data = open (filen).read().splitlines()
+   return find_unique_include_list_src (data)
+ 
+ 
+ # Create the macin, macout, and incl vectors for a file FILEN.
+ # macin are the macros that are used in #if* conditional expressions
+ # macout are the macros which are #defined
+ # incl is the list of incluide files encountered
+ # returned as a tuple of the filename followed by the triplet of lists
+ # (filen, macin, macout, incl)
+ 
+ def create_macro_in_out (filen):
+   sfile = open (filen, "r");
+   data = sfile.readlines()
+   sfile.close()
+ 
+   macout = list ()
+   macin = list()
+   incl = list()
+ 
+   for line in (data):
+     d = find_pound_define (line)
+     if d != "":
+       if d not in macout:
+         macout.append (d);
+       continue
+ 
+     d = find_pound_if (line)
+     if len(d) != 0:
+       for mac in d:
+         if mac != "defined" and mac not in macin:
+           macin.append (mac);
+       continue
+ 
+     nm = find_pound_include (line, True, True)
+     if nm != "" and nm not in incl:
+       incl.append (nm)
+ 
+   return (filen, macin, macout, incl)
+ 
+ # create the macro information for filen, and create .macin, .macout, and .incl
+ # files.  Return the created macro tuple.
+ def create_include_data_files (filen):
+ 
+   macros = create_macro_in_out (filen)
+   depends = macros[1]
+   defines = macros[2]
+   incls = macros[3]
+   
+   disp_message = filen
+   if len (defines) > 0:
+     disp_message = disp_message + " " + str(len (defines)) + " #defines"
+   dfile = open (filen + ".macout", "w")
+   for x in defines:
+     dfile.write (x + "\n")
+   dfile.close ()
+ 
+   if len (depends) > 0:
+     disp_message = disp_message + " " + str(len (depends)) + " #if dependencies"
+   dfile = open (filen + ".macin", "w")
+   for x in depends:
+     dfile.write (x + "\n")
+   dfile.close ()
+ 
+   if len (incls) > 0:
+     disp_message = disp_message + " " + str(len (incls)) + " #includes"
+   dfile = open (filen + ".incl", "w")
+   for x in incls:
+     dfile.write (x + "\n")
+   dfile.close ()
+ 
+   return macros
+ 
+ 
+ 
+ # extract data for include file name_h and enter it into the dictionary.
+ # this does not change once read in.  use_requires is True if you want to 
+ # prime the values with already created .requires and .provides files.
+ def get_include_data (name_h, use_requires):
+   macin = list()
+   macout = list()
+   incl = list ()
+   if use_requires and os.path.exists (name_h + ".requires"):
+     macin = open (name_h + ".requires").read().splitlines()
+   elif os.path.exists (name_h + ".macin"):
+     macin = open (name_h + ".macin").read().splitlines()
+ 
+   if use_requires and os.path.exists (name_h + ".provides"):
+     macout  = open (name_h + ".provides").read().splitlines()
+   elif os.path.exists (name_h + ".macout"):
+     macout  = open (name_h + ".macout").read().splitlines()
+ 
+   if os.path.exists (name_h + ".incl"):
+     incl = open (name_h + ".incl").read().splitlines()
+ 
+   if len(macin) == 0 and len(macout) == 0 and len(incl) == 0:
+     return ()
+   data = ( name_h, macin, macout, incl )
+   return data
+   
+ # find FIND in src, and replace it with the list of headers in REPLACE.
+ # Remove any duplicates of FIND in REPLACE, and if some of the REPLACE
+ # headers occur earlier in the include chain, leave them.
+ # Return the new SRC only if anything changed.
+ def find_replace_include (find, replace, src):
+   res = list()
+   seen = { }
+   anything = False
+   for line in src:
+     inc = find_pound_include (line, True, True)
+     if inc == find:
+       for y in replace:
+         if seen.get(y) == None:
+           res.append("#include \""+y+"\"\n")
+           seen[y] = True
+           if y != find:
+             anything = True
+ # if find isnt in the replacement list, then we are deleting FIND, so changes.
+       if find not in replace:
+         anything = True
+     else:
+       if inc in replace:
+         if seen.get(inc) == None:
+           res.append (line)
+           seen[inc] = True
+       else:
+         res.append (line)
+ 
+   if (anything):
+     return res
+   else:
+     return list()
+       
+ 
+ # pass in a require and provide dictionary to be read in.
+ def read_require_provides (require, provide):
+   if not os.path.exists ("require-provide.master"):
+     print "require-provide.master file is not available. please run data collection."
+     sys.exit(1)
+   incl_list = open("require-provide.master").read().splitlines()
+   for f in incl_list:
+     if os.path.exists (f+".requires"):
+       require[os.path.basename (f)] = open (f + ".requires").read().splitlines()
+     else:
+       require[os.path.basename (f)] = list ()
+     if os.path.exists (f+".provides"):
+       provide[os.path.basename (f)] = open (f + ".provides").read().splitlines()
+     else:
+       provide [os.path.basename (f)] = list ()
+ 
+    
+ def build_include_list (filen):
+   include_files = list()
+   sfile = open (filen, "r")
+   data = sfile.readlines()
+   sfile.close()
+   for line in data:
+     nm = find_pound_include (line, False, False)
+     if nm != "" and nm[-2:] == ".h":
+       if nm not in include_files:
+         include_files.append(nm)
+   return include_files
+  
+ def build_reverse_include_list (filen):
+   include_files = list()
+   sfile = open (filen, "r")
+   data = sfile.readlines()
+   sfile.close()
+   for line in reversed(data):
+     nm = find_pound_include (line, False, False)
+     if nm != "":
+       if nm not in include_files:
+         include_files.append(nm)
+   return include_files
+      
+ # Get compilation return code, and compensate for a warning that we want to 
+ # consider an error when it comes to inlined templates.
+ def get_make_rc (rc, output):
+   rc = rc % 1280
+   if rc == 0:
+     # This is not considered an error during compilation of an individual file,
+     # but it will cause an error during link if it isn't defined.  If this
+     # warning is seen during compiling a file, make it a build error so we 
+     # don't remove the header.
+     h = re.findall ("warning: inline function.*used but never defined", output)
+     if len(h) != 0:
+       rc = 1
+   return rc;
+ 
+ def get_make_output (build_dir, make_opt):
+   devnull = open('/dev/null', 'w')
+   at_a_time = multiprocessing.cpu_count() * 2
+   make = "make -j"+str(at_a_time)+ " "
+   if build_dir != "":
+     command = "cd " + build_dir +"; " + make + make_opt
+   else:
+     command = make + make_opt
+   process = subprocess.Popen(command, stdout=devnull, stderr=subprocess.PIPE, shell=True)
+   output = process.communicate();
+   rc = get_make_rc (process.returncode, output[1])
+   return (rc , output[1])
+ 
+ def spawn_makes (command_list):
+   devnull = open('/dev/null', 'w')
+   rc = (0,"", "")
+   proc_res = list()
+   text = "  Trying target builds : "
+   for command_pair in command_list:
+     tname = command_pair[0]
+     command = command_pair[1]
+     text += tname + ", "
+     c = subprocess.Popen(command, bufsize=-1, stdout=devnull, stderr=subprocess.PIPE, shell=True)
+     proc_res.append ((c, tname))
+ 
+   print text[:-2]
+ 
+   for p in proc_res:
+     output = p[0].communicate()
+     ret = (get_make_rc (p[0].returncode, output[1]), output[1], p[1])
+     if (ret[0] != 0):
+       # Just record the first one.
+       if rc[0] == 0:
+         rc = ret;
+   return rc
+ 
+ def get_make_output_parallel (targ_list, make_opt, at_a_time):
+   command = list()
+   targname = list()
+   if at_a_time == 0:
+     at_a_time = multiprocessing.cpu_count() * 2
+   proc_res = [0] * at_a_time
+   for x in targ_list:
+     if make_opt[-2:] == ".o":
+       s = "cd " + x[1] + "/gcc/; make " + make_opt
+     else:
+       s = "cd " + x[1] +"; make " + make_opt
+     command.append ((x[0],s))
+ 
+   num = len(command) 
+   rc = (0,"", "")
+   loops = num // at_a_time
+   
+   if (loops > 0):
+     for idx in range (loops):
+       ret = spawn_makes (command[idx*at_a_time:(idx+1)*at_a_time])
+       if ret[0] != 0:
+         rc = ret
+         break
+ 
+   if (rc[0] == 0):
+     leftover = num % at_a_time
+     if (leftover > 0):
+       ret = spawn_makes (command[-leftover:])
+       if ret[0] != 0:
+         rc = ret
+ 
+   return rc
+ 
+ 
+ def readwholefile (src_file):
+   sfile = open (src_file, "r")
+   src_data = sfile.readlines()
+   sfile.close()
+   return src_data
+ 
Index: headers/included-by
===================================================================
*** headers/included-by	(revision 0)
--- headers/included-by	(working copy)
***************
*** 0 ****
--- 1,112 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ 
+ 
+ usage = False
+ src = list()
+ flist = { }
+ process_h = False
+ process_c = False
+ verbose = False
+ level = 0
+ match_all = False
+ num_match = 1
+ 
+ file_list = list()
+ current = True
+ deeper = True
+ scanfiles = True
+ for x in sys.argv[1:]:
+   if x[0:2] == "-h":
+     usage = True
+   elif x[0:2] == "-i":
+     process_h = True
+   elif x[0:2] == "-s" or x[0:2] == "-c":
+     process_c = True
+   elif x[0:2] == "-v":
+     verbose = True
+   elif x[0:2] == "-a":
+     match_all = True
+   elif x[0:2] == "-n":
+     num_match = int(x[2:])
+   elif x[0:2] == "-1":
+     deeper = False
+   elif x[0:2] == "-2":
+     current = False
+   elif x[0:2] == "-f":
+     file_list = open (x[2:]).read().splitlines()
+     scanfiles = False
+   elif x[0] == "-":
+     print "Error: Unknown option " + x
+     usage = True
+   else:
+     src.append (x)
+ 
+ if match_all:
+   num_match = len (src)
+ 
+ if not process_h and not process_c:
+   process_h = True
+   process_c = True
+ 
+ if len(src) == 0:
+   usage = True
+ 
+ if not usage:
+   if scanfiles:
+     if process_h:
+       file_list = find_gcc_files ("\*.h", current, deeper)
+     if process_c:
+       file_list = file_list + find_gcc_files ("\*.c", current, deeper)
+       file_list = file_list + find_gcc_files ("\*.cc", current, deeper)
+   else:
+     newlist = list()
+     for x in file_list:
+       if process_h and x[-2:] == ".h":
+         newlist.append (x)
+       elif process_c and (x[-2:] == ".c" or x[-3:] == ".cc"):
+         newlist.append (x)
+     file_list = newlist;
+      
+   file_list.sort()
+   for fn in file_list:
+     found = find_unique_include_list (fn)
+     careabout = list()
+     output = ""
+     for inc in found:
+       if inc in src:
+         careabout.append (inc)
+         if output == "":
+           output = fn
+         if verbose:
+           output = output + " [" + inc +"]"
+     if len (careabout) < num_match:
+         output = ""
+     if output != "":
+       print output
+ else:
+   print "included-by [-h] [-i] [-c] [-v] [-a] [-nx] file1 [file2] ... [filen]"
+   print "find the list of all files in subdirectories that include any of "
+   print "the listed files. processed to a depth of 3 subdirs"
+   print " -h  : Show this message"
+   print " -i  : process only header files (*.h) for #include"
+   print " -c  : process only source files (*.c *.cc) for #include"
+   print "       If nothing is specified, defaults to -i -c"
+   print " -s  : Same as -c."
+   print " -v  : Show which include(s) were found"
+   print " -nx : Only list files which have at least x different matches. Default = 1"
+   print " -a  : Show only files which all listed files are included"
+   print "       This is equivilent to -nT where T == # of items in list"
+   print " -flistfile  : Show only files contained in the list of files"
+ 
+  
+ 
+ 
+ 
+ 

Property changes on: headers/included-by
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: headers/reduce-headers
===================================================================
*** headers/reduce-headers	(revision 0)
--- headers/reduce-headers	(working copy)
***************
*** 0 ****
--- 1,596 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ import tempfile
+ import copy
+ 
+ from headerutils import *
+ 
+ requires = { }
+ provides = { }
+ 
+ no_remove = [ "system.h", "coretypes.h", "config.h" , "bconfig.h", "backend.h" ]
+ 
+ # These targets are the ones which provide "coverage".  Typically, if any
+ # target is going to fail compilation, it's one of these.  This was determined
+ # during the initial runs of reduce-headers... On a full set of target builds,
+ # every failure which occured was triggered by one of these.  
+ # This list is used during target-list construction simply to put any of these
+ # *first* in the candidate list, increasing the probability that a failure is 
+ # found quickly.
+ target_priority = [
+     "aarch64-linux-gnu",
+     "arm-netbsdelf",
+     "avr-rtems",
+     "c6x-elf",
+     "epiphany-elf",
+     "hppa2.0-hpux10.1",
+     "i686-mingw32crt",
+     "i686-pc-msdosdjgpp",
+     "mipsel-elf",
+     "powerpc-eabisimaltivec",
+     "rs6000-ibm-aix5.1.0",
+     "sh-superh-elf",
+     "sparc64-elf",
+     "spu-elf"
+ ]
+ 
+ 
+ target_dir = ""
+ build_dir = ""
+ ignore_list = list()
+ target_builds = list()
+ 
+ target_dict = { }
+ header_dict = { }
+ search_path = [ ".", "../include", "../libcpp/include" ]
+ 
+ remove_count = { }
+ 
+ 
+ # Given a header name, normalize it.  ie.  cp/cp-tree.h could be in gcc, while
+ # the same header could be referenced from within the cp subdirectory as
+ # just cp-tree.h
+ # for now, just assume basenames are unique
+ 
+ def normalize_header (header):
+   return os.path.basename (header)
+ 
+ 
+ # Adds a header file and its sub-includes to the global dictionary if they
+ # aren't already there.  Specify s_path since different build directories may
+ # append themselves on demand to the global list.
+ # return entry for the specified header, knowing all sub entries are completed
+ 
+ def get_header_info (header, s_path):
+   global header_dict
+   global empty_iinfo
+   process_list = list ()
+   location = ""
+   bname = ""
+   bname_iinfo = empty_iinfo
+   for path in s_path:
+     if os.path.exists (path + "/" + header):
+       location = path + "/" + header
+       break
+ 
+   if location:
+     bname = normalize_header (location)
+     if header_dict.get (bname):
+       bname_iinfo = header_dict[bname]
+       loc2 = ii_path (bname_iinfo)+ "/" + bname
+       if loc2[:2] == "./":
+         loc2 = loc2[2:]
+       if location[:2] == "./":
+         location = location[2:]
+       if loc2 != location:
+         # Don't use the cache if it isnt the right one.
+         bname_iinfo = process_ii_macro (location)
+       return bname_iinfo
+ 
+     bname_iinfo = process_ii_macro (location)
+     header_dict[bname] = bname_iinfo
+     # now decend into the include tree
+     for i in ii_include_list (bname_iinfo):
+       get_header_info (i, s_path)
+   else:
+     # if the file isnt in the source directories, look in the build and target
+     # directories. If it is here, then aggregate all the versions.
+     location = build_dir + "/gcc/" + header
+     build_inc = target_inc = False
+     if os.path.exists (location):
+       build_inc = True
+     for x in target_dict:
+       location = target_dict[x] + "/gcc/" + header
+       if os.path.exists (location):
+         target_inc = True
+         break
+ 
+     if (build_inc or target_inc):
+       bname = normalize_header(header)
+       defines = set()
+       consumes = set()
+       incl = set()
+       if build_inc:
+         iinfo = process_ii_macro (build_dir + "/gcc/" + header)
+         defines = set (ii_macro_define (iinfo))
+         consumes = set (ii_macro_consume (iinfo))
+         incl = set (ii_include_list (iinfo))
+ 
+       if (target_inc):
+         for x in target_dict:
+           location = target_dict[x] + "/gcc/" + header
+           if os.path.exists (location):
+             iinfo = process_ii_macro (location)
+             defines.update (ii_macro_define (iinfo))
+             consumes.update (ii_macro_consume (iinfo))
+             incl.update (ii_include_list (iinfo))
+ 
+       bname_iinfo = (header, "build", list(incl), list(), list(consumes), list(defines), list(), list())
+ 
+       header_dict[bname] = bname_iinfo
+       for i in incl:
+         get_header_info (i, s_path)
+ 
+   return bname_iinfo
+ 
+ 
+ # return a list of all headers brought in by this header
+ def all_headers (fname):
+   global header_dict
+   headers_stack = list()
+   headers_list = list()
+   if header_dict.get (fname) == None:
+     return list ()
+   for y in ii_include_list (header_dict[fname]):
+     headers_stack.append (y)
+ 
+   while headers_stack:
+     h = headers_stack.pop ()
+     hn = normalize_header (h)
+     if hn not in headers_list:
+       headers_list.append (hn)
+       if header_dict.get(hn):
+         for y in ii_include_list (header_dict[hn]):
+           if normalize_header (y) not in headers_list:
+             headers_stack.append (y)
+ 
+   return headers_list
+ 
+ 
+ 
+ 
+ # Search bld_dir for all target tuples, confirm that they have a build path with
+ # bld_dir/target-tuple/gcc, and build a dictionary of build paths indexed by
+ # target tuple..
+ 
+ def build_target_dict (bld_dir, just_these):
+   global target_dict
+   target_doct = { }
+   error = False
+   if os.path.exists (bld_dir):
+     if just_these:
+       ls = just_these
+     else:
+       ls = os.listdir(bld_dir)
+     for t in ls:
+       if t.find("-") != -1:
+         target = t.strip()
+         tpath = bld_dir + "/" + target
+         if not os.path.exists (tpath + "/gcc"):
+           print "Error: gcc build directory for target " + t + " Does not exist: " + tpath + "/gcc"
+           error = True
+         else:
+           target_dict[target] = tpath
+ 
+   if error:
+     target_dict = { }
+ 
+ def get_obj_name (src_file):
+   if src_file[-2:] == ".c":
+     return src_file.replace (".c", ".o")
+   elif src_file[-3:] == ".cc":
+     return src_file.replace (".cc", ".o")
+   return ""
+ 
+ def target_obj_exists (target, obj_name):
+   global target_dict
+   # look in a subdir if src has a subdir, then check gcc base directory.
+   if target_dict.get(target):
+     obj = target_dict[target] + "/gcc/" + obj_name
+     if not os.path.exists (obj):
+       obj = target_dict[target] + "/gcc/" + os.path.basename(obj_name)
+     if os.path.exists (obj):
+       return True
+   return False
+  
+ # Given a src file, return a list of targets which may build this file.
+ def find_targets (src_file):
+   global target_dict
+   targ_list = list()
+   obj_name = get_obj_name (src_file)
+   if not obj_name:
+     print "Error: " + src_file + " - Cannot determine object name."
+     return list()
+ 
+   # Put the high priority targets which tend to trigger failures first
+   for target in target_priority:
+     if target_obj_exists (target, obj_name):
+       targ_list.append ((target, target_dict[target]))
+ 
+   for target in target_dict:
+     if target not in target_priority and target_obj_exists (target, obj_name):
+       targ_list.append ((target, target_dict[target]))
+         
+   return targ_list
+ 
+ 
+ def try_to_remove (src_file, h_list, verbose):
+   global target_dict
+   global header_dict
+   global build_dir
+ 
+   # build from scratch each time
+   header_dict = { }
+   summary = ""
+   rmcount = 0
+ 
+   because = { }
+   src_info = process_ii_macro_src (src_file)
+   src_data = ii_src (src_info)
+   if src_data:
+     inclist = ii_include_list_non_cond (src_info)
+     # work is done if there are no includes to check
+     if not inclist:
+       return src_file + ": No include files to attempt to remove"
+ 
+     # work on the include list in reverse.
+     inclist.reverse()
+ 
+     # Get the target list 
+     targ_list = list()
+     targ_list = find_targets (src_file)
+ 
+     spath = search_path
+     if os.path.dirname (src_file):
+       spath.append (os.path.dirname (src_file))
+ 
+     hostbuild = True
+     if src_file.find("config/") != -1:
+       # config files dont usually build on the host
+       hostbuild = False
+       obn = get_obj_name (os.path.basename (src_file))
+       if obn and os.path.exists (build_dir + "/gcc/" + obn):
+         hostbuild = True
+       if not target_dict:
+         summary = src_file + ": Target builds are required for config files.  None found."
+         print summary
+         return summary
+       if not targ_list:
+         summary =src_file + ": Cannot find any targets which build this file."
+         print summary
+         return summary
+ 
+     if hostbuild:
+       # confirm it actually builds before we do anything
+       print "Confirming source file builds"
+       res = get_make_output (build_dir + "/gcc", "all")
+       if res[0] != 0:
+         message = "Error: " + src_file + " does not build currently."
+         summary = src_file + " does not build on host."
+         print message
+         print res[1]
+         if verbose:
+           verbose.write (message + "\n")
+           verbose.write (res[1]+ "\n")
+         return summary
+ 
+     src_requires = set (ii_macro_consume (src_info))
+     for macro in src_requires:
+       because[macro] = src_file
+     header_seen = list ()
+ 
+     os.rename (src_file, src_file + ".bak")
+     src_orig = copy.deepcopy (src_data)
+     src_tmp = copy.deepcopy (src_data)
+ 
+     try:
+       # process the includes from bottom to top.  This is because we know that
+       # later includes have are known to be needed, so any dependency from this 
+       # header is a true dependency
+       for inc_file in inclist:
+         inc_file_norm = normalize_header (inc_file)
+         
+         if inc_file in no_remove:
+           continue
+         if len (h_list) != 0 and inc_file_norm not in h_list:
+           continue
+         if inc_file_norm[0:3] == "gt-":
+           continue
+         if inc_file_norm[0:6] == "gtype-":
+           continue
+         if inc_file_norm.replace(".h",".c") == os.path.basename(src_file):
+           continue
+              
+         lookfor = ii_src_line(src_info)[inc_file]
+         src_tmp.remove (lookfor)
+         message = "Trying " + src_file + " without " + inc_file
+         print message
+         if verbose:
+           verbose.write (message + "\n")
+         out = open(src_file, "w")
+         for line in src_tmp:
+           out.write (line)
+         out.close()
+           
+         keep = False
+         if hostbuild:
+           res = get_make_output (build_dir + "/gcc", "all")
+         else:
+           res = (0, "")
+ 
+         rc = res[0]
+         message = "Passed Host build"
+         if (rc != 0):
+           # host build failed
+           message  = "Compilation failed:\n";
+           keep = True
+         else:
+           if targ_list:
+             objfile = get_obj_name (src_file)
+             t1 = targ_list[0]
+             if objfile and os.path.exists(t1[1] +"/gcc/"+objfile):
+               res = get_make_output_parallel (targ_list, objfile, 0)
+             else:
+               res = get_make_output_parallel (targ_list, "all-gcc", 0)
+             rc = res[0]
+             if rc != 0:
+               message = "Compilation failed on TARGET : " + res[2]
+               keep = True
+             else:
+               message = "Passed host and target builds"
+ 
+         if keep:
+           print message + "\n"
+ 
+         if (rc != 0):
+           if verbose:
+             verbose.write (message + "\n");
+             verbose.write (res[1])
+             verbose.write ("\n");
+             if os.path.exists (inc_file):
+               ilog = open(inc_file+".log","a")
+               ilog.write (message + " for " + src_file + ":\n\n");
+               ilog.write ("============================================\n");
+               ilog.write (res[1])
+               ilog.write ("\n");
+               ilog.close()
+             if os.path.exists (src_file):
+               ilog = open(src_file+".log","a")
+               ilog.write (message + " for " +inc_file + ":\n\n");
+               ilog.write ("============================================\n");
+               ilog.write (res[1])
+               ilog.write ("\n");
+               ilog.close()
+ 
+         # Given a sequence where :
+         # #include "tm.h"
+         # #include "target.h"  // includes tm.h
+ 
+         # target.h was required, and when attempting to remove tm.h we'd see that
+         # all the macro defintions are "required" since they all look like:
+         # #ifndef HAVE_blah
+         # #define HAVE_blah
+         # endif
+ 
+         # when target.h was found to be required, tm.h will be tagged as included.
+         # so when we get this far, we know we dont have to check the macros for
+         # tm.h since we know it is already been included.
+ 
+         if inc_file_norm not in header_seen:
+           iinfo = get_header_info (inc_file, spath)
+           newlist = all_headers (inc_file_norm)
+           if ii_path(iinfo) == "build" and not target_dict:
+             keep = True
+             text = message + " : Will not remove a build file without some targets."
+             print text
+             ilog = open(src_file+".log","a")
+             ilog.write (text +"\n")
+             ilog.write ("============================================\n");
+             ilog.close()
+             ilog = open("reduce-headers-kept.log","a")
+             ilog.write (src_file + " " + text +"\n")
+             ilog.close()
+         else:
+           newlist = list()
+         if not keep and inc_file_norm not in header_seen:
+           # now look for any macro requirements.
+           for h in newlist:
+             if not h in header_seen:
+               if header_dict.get(h):
+                 defined = ii_macro_define (header_dict[h])
+                 for dep in defined:
+                   if dep in src_requires and dep not in ignore_list:
+                     keep = True;
+                     text = message + ", but must keep " + inc_file + " because it provides " + dep 
+                     if because.get(dep) != None:
+                       text = text + " Possibly required by " + because[dep]
+                     print text
+                     ilog = open(inc_file+".log","a")
+                     ilog.write (because[dep]+": Requires [dep] in "+src_file+"\n")
+                     ilog.write ("============================================\n");
+                     ilog.close()
+                     ilog = open(src_file+".log","a")
+                     ilog.write (text +"\n")
+                     ilog.write ("============================================\n");
+                     ilog.close()
+                     ilog = open("reduce-headers-kept.log","a")
+                     ilog.write (src_file + " " + text +"\n")
+                     ilog.close()
+                     if verbose:
+                       verbose.write (text + "\n")
+ 
+         if keep:
+           # add all headers 'consumes' to src_requires list, and mark as seen
+           for h in newlist:
+             if not h in header_seen:
+               header_seen.append (h)
+               if header_dict.get(h):
+                 consume = ii_macro_consume (header_dict[h])
+                 for dep in consume:
+                   if dep not in src_requires:
+                     src_requires.add (dep)
+                     if because.get(dep) == None:
+                       because[dep] = inc_file
+ 
+           src_tmp = copy.deepcopy (src_data)
+         else:
+           print message + "  --> removing " + inc_file + "\n"
+           rmcount += 1
+           if verbose:
+             verbose.write (message + "  --> removing " + inc_file + "\n")
+           if remove_count.get(inc_file) == None:
+             remove_count[inc_file] = 1
+           else:
+             remove_count[inc_file] += 1
+           src_data = copy.deepcopy (src_tmp)
+     except:
+       print "Interuption: restoring original file"
+       out = open(src_file, "w")
+       for line in src_orig:
+         out.write (line)
+       out.close()
+       raise
+ 
+     # copy current version, since it is the "right" one now.
+     out = open(src_file, "w")
+     for line in src_data:
+       out.write (line)
+     out.close()
+     
+     # Try a final host bootstrap build to make sure everything is kosher.
+     if hostbuild:
+       res = get_make_output (build_dir, "all")
+       rc = res[0]
+       if (rc != 0):
+         # host build failed! return to original version
+         print "Error: " + src_file + " Failed to bootstrap at end!!! restoring."
+         print "        Bad version at " + src_file + ".bad"
+         os.rename (src_file, src_file + ".bad")
+         out = open(src_file, "w")
+         for line in src_orig:
+           out.write (line)
+         out.close()
+         return src_file + ": failed to build after reduction.  Restored original"
+ 
+     if src_data == src_orig:
+       summary = src_file + ": No change."
+     else:
+       summary = src_file + ": Reduction performed, "+str(rmcount)+" includes removed."
+   print summary
+   return summary
+ 
+ only_h = list ()
+ ignore_cond = False
+ 
+ usage = False
+ src = list()
+ only_targs = list ()
+ for x in sys.argv[1:]:
+   if x[0:2] == "-b":
+     build_dir = x[2:]
+   elif x[0:2] == "-f":
+     fn = normalize_header (x[2:])
+     if fn not in only_h:
+       only_h.append (fn)
+   elif x[0:2] == "-h":
+     usage = True
+   elif x[0:2] == "-d":
+     ignore_cond = True
+   elif x[0:2] == "-D":
+     ignore_list.append(x[2:])
+   elif x[0:2] == "-T":
+     only_targs.append(x[2:])
+   elif x[0:2] == "-t":
+     target_dir = x[2:]
+   elif x[0] == "-":
+     print "Error:  Unrecognized option " + x
+     usgae = True
+   else:
+     if not os.path.exists (x):
+       print "Error: specified file " + x + " does not exist."
+       usage = True
+     else:
+       src.append (x)
+ 
+ if target_dir:
+   build_target_dict (target_dir, only_targs)
+ 
+ if build_dir == "" and target_dir == "":
+   print "Error: Must specify a build directory, and/or a target directory."
+   usage = True
+ 
+ if build_dir and not os.path.exists (build_dir):
+     print "Error: specified build directory does not exist : " + build_dir
+     usage = True
+ 
+ if target_dir and not os.path.exists (target_dir):
+     print "Error: specified target directory does not exist : " + target_dir
+     usage = True
+ 
+ if usage:
+   print "Attempts to remove extraneous include files from source files."
+   print " "
+   print "Should be run from the main gcc source directory, and works on a target"
+   print "directory, as we attempt to make the 'all' target."
+   print " "
+   print "By default, gcc-reorder-includes is run on each file before attempting"
+   print "to remove includes. this removes duplicates and puts some headers in a"
+   print "canonical ordering"
+   print " "
+   print "The build directory should be ready to compile via make. Time is saved"
+   print "if the build is already complete, so that only changes need to be built."
+   print " "
+   print "Usage: [options] file1.c [file2.c] ... [filen.c]"
+   print "      -bdir    : the root build directory to attempt buiding .o files."
+   print "      -tdir    : the target build directory"
+   print "      -d       : Ignore conditional macro dependencies."
+   print " "
+   print "      -Dmacro  : Ignore a specific macro for dependencies"
+   print "      -Ttarget : Only consider target in target directory."
+   print "      -fheader : Specifies a specific .h file to be considered."
+   print " "
+   print "      -D, -T, and -f can be specified mulitple times and are aggregated."
+   print " "
+   print "  The original file will be in filen.bak"
+   print " "
+   sys.exit (0)
+  
+ if only_h:
+   print "Attempting to remove only these files:"
+   for x in only_h:
+     print x
+   print " "
+ 
+ logfile = open("reduce-headers.log","w")
+ 
+ for x in src:
+   msg = try_to_remove (x, only_h, logfile)
+   ilog = open("reduce-headers.sum","a")
+   ilog.write (msg + "\n")
+   ilog.close()
+ 
+ ilog = open("reduce-headers.sum","a")
+ ilog.write ("===============================================================\n")
+ for x in remove_count:
+   msg = x + ": Removed " + str(remove_count[x]) + " times."
+   print msg
+   logfile.write (msg + "\n")
+   ilog.write (msg + "\n")
+ 
+ 
+ 
+ 
+ 

Property changes on: headers/reduce-headers
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: headers/replace-header
===================================================================
*** headers/replace-header	(revision 0)
--- headers/replace-header	(working copy)
***************
*** 0 ****
--- 1,53 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ 
+ files = list()
+ replace = list()
+ find = ""
+ usage = False
+ 
+ for x in sys.argv[1:]:
+   if x[0:2] == "-h":
+     usage = True
+   elif x[0:2] == "-f" and find == "":
+     find = x[2:]
+   elif x[0:2] == "-r":
+     replace.append (x[2:])
+   elif x[0:1] == "-":
+     print "Error: unrecognized option " + x
+     usage = True
+   else:
+     files.append (x)
+ 
+ if find == "":
+   usage = True
+ 
+ if usage:
+   print "replace-header -fheader -rheader [-rheader] file1 [filen.]"
+   sys.exit(0)
+ 
+ string = ""
+ for x in replace:
+   string = string + " '"+x+"'"
+ print "Replacing '"+find+"'  with"+string
+ 
+ for x in files:
+   src = readwholefile (x)
+   src = find_replace_include (find, replace, src)
+   if (len(src) > 0):
+     print x + ": Changed"
+     out = open(x, "w")
+     for line in src:
+       out.write (line);
+     out.close ()
+   else:
+     print x
+ 
+ 
+ 

Property changes on: headers/replace-header
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Index: headers/show-headers
===================================================================
*** headers/show-headers	(revision 0)
--- headers/show-headers	(working copy)
***************
*** 0 ****
--- 1,138 ----
+ #! /usr/bin/python2
+ import os.path
+ import sys
+ import shlex
+ import re
+ 
+ from headerutils import *
+ 
+ 
+ tabstop = 2
+ padding = "                                                                  "
+ seen = { }
+ output = list()
+ summary = list()
+ sawcore = False
+ 
+ # list of headers to emphasize
+ highlight = list ()
+ 
+ # search path for headers
+ incl_dirs = [".", "../include", "../../build/gcc", "../libcpp/include" ]
+ # extra search paths to look in *after* the directory the source file is in. 
+ extra_dirs = [ "common", "c-family", "c", "cp", "config" ]
+ 
+ # append (1) to the end of the first line which includes INC in list INC.
+ def append_1 (output, inc):
+   for n,t in enumerate (output):
+     idx = t.find(inc)
+     if idx != -1:
+       eos = idx + len (inc)
+       t = t[:eos] + "  (1)" + t[eos+1:]
+       output[n] = t
+       return
+ 
+ # These headers show up as duplicates in rtl.h due to conditional code arund the includes
+ rtl_core = [ "machmode.h" , "signop.h" , "wide-int.h" , "double-int.h" , "real.h" , "fixed-value.h" , "statistics.h" , "vec.h" , "hash-table.h" , "hash-set.h" , "input.h" , "is-a.h" ]
+ 
+ def find_include_data (inc):
+   global sawcore
+   for x in incl_dirs:
+     nm = x+"/"+inc
+     if os.path.exists (nm):
+       info = find_unique_include_list (nm)
+       # rtl.h mimics coretypes for GENERATOR FILES, remove if coretypes.h seen.
+       if inc == "coretypes.h":
+         sawcore = True
+       elif inc  == "rtl.h" and sawcore:
+         for i in rtl_core:
+           if i in info:
+             info.remove (i)
+       return info
+   return list()
+ 
+ def process_include (inc, indent):
+   if inc[-2:] != ".h":
+     return
+   bname  = os.path.basename (inc)
+   if bname in highlight:
+     arrow = "                <<-------"
+     if bname not in summary:
+       summary.append (bname)
+   else:
+     arrow = ""
+   if seen.get(inc) == None:
+     seen[inc] = 1
+     output.append (padding[:indent*tabstop] + bname + arrow)
+     info = find_include_data (inc)
+     for y in info:
+       process_include (y, indent+1)
+   else:
+     seen[inc] += 1
+     if (seen[inc] == 2):
+       append_1(output, inc)
+     output.append (padding[:indent*tabstop] + bname + "  ("+str(seen[inc])+")" + arrow)
+ 
+     
+ 
+ blddir = [ "." ]
+ usage = False
+ src = list()
+ 
+ for x in sys.argv[1:]:
+   if x[0:2] == "-i":
+     bld = x[2:]
+     print "Build dir : " + bld
+     blddir.append (bld)
+   elif x[0:2] == "-s":
+     highlight.append (os.path.basename (x[2:]))
+   elif x[0:2] == "-h":
+     usage = True
+   else:
+     src.append (x)
+ 
+ if len(src) != 1:
+   usage = True
+ elif not os.path.exists (src[0]):
+   print src[0] + ": Requested source file does not exist.\n"
+   usage = True
+ 
+ if usage:
+   print "show-headers [-idir] [-sfilen] file1 "
+   print " "
+   print " Show a hierarchical visual format how many times each header file"
+   print " is included in a source file.  Should be run from the source directory"
+   print " files from find-include-depends"
+   print "      -s : search for a header, and point it out."
+   print "      -i : Specifies 1 or more directories to search for includes."
+   sys.exit(0)
+ 
+ 
+ if len(blddir) > 1:
+   incl_dirs = blddir
+ 
+ x = src[0]
+ # if source is in a subdirectory, add the subdirectory to the search list
+ srcpath = os.path.dirname(x)
+ if srcpath:
+   incl_dirs.append (srcpath)
+ for yy in extra_dirs:
+   incl_dirs.append (yy)
+ 
+ output = list()
+ sawcore = False
+ incl = find_unique_include_list (x)
+ for inc in incl:
+   process_include (inc, 1)
+ print "\n" + x
+ for line in output:
+   print line
+ 
+ if highlight:
+   print " "
+   for h in summary:
+     print h + " is included by source file."
+   for h in highlight:
+     if h not in summary:
+       print h + " is not included by source file."
+ 

Property changes on: headers/show-headers
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property

Re: [patch 4/3] Header file reduction - Tools for contrib - second cut

[Jeff Law]

On 10/14/2015 09:14 AM, Andrew MacLeod wrote:
> Here's the latest version of the tools for a sub directory in contrib.
> I've handled all the feedback, except I have not fully commented the
> python code in the tools, nor followed any particular coding
> convention...   Documentation has been handled, and I've added some
> additional comments to the places which were noted as being unclear. Ive
> also removed all tabs from the source files.
>
> Ive also updated show-headers slightly to be a little more
> error-resistant and to put some emphasis on any header files specified
> on the command as being of interest . (when there are 140 shown, it can
> be hard to find the one you are looking for sometimes)
>
> Do we wish to impose anything in particular on the source for  tools
> going into this sub-directory of contrib? The other tools in contrib
> don't seem to have much in the way of coding standards.     I also
> wonder if anyone other than me will look at them much :-)
I'm certainly interested in them.

Do you have any sense of whether or not coverage of the tools has 
improved over short time since we started squashing out conditional 
compilation?  I was running the header file reordering bits on the trunk 
and was a bit surprised of how many things they're still changing.  But 
that would make sense if some files are now being processed that weren't 
before because we've squashed out the conditional compilation.

It certainly is true that the total result is smaller than any of the 
backend, config/ or languages changes that you posted, and I'm running 
it across the entire source tree, but I'm still surprised at how much 
churn I'm seeing.

If it weren't for the level of churn, I'd probably be suggesting we just 
have this stuff run regularly (weekly, monthly, whatever) and commit the 
result after a sanity looksie.  I've yet to see this tool botch anything 
and if we're not unnecessarily churning the sources, keeping us clean 
WRT canononical ordering and duplicate removal automatically seems like 
a good place to be.

Maybe do another commit of the reordering output and evaluate again in a 
month?

I don't think we're quite there on the reducer and it obviously requires 
more infrastructure in place to test.  But it'd be nice to get to a 
similar state on that tool.

Which reminds me, you ought to add a VMS target to your tests.  The 
reducer botched vmsdbgout.c.


alpha64-dec-vms
alpha-dec-vms
ia64-hp-vms

Covering any one of those ought to do the trick.

Thoughts?

jeff

Re: [patch 4/3] Header file reduction - Tools for contrib - second cut

[Andrew MacLeod]

On 11/03/2015 01:06 AM, Jeff Law wrote:
> On 10/14/2015 09:14 AM, Andrew MacLeod wrote:
>> Here's the latest version of the tools for a sub directory in contrib.
>> I've handled all the feedback, except I have not fully commented the
>> python code in the tools, nor followed any particular coding
>> convention...   Documentation has been handled, and I've added some
>> additional comments to the places which were noted as being unclear. Ive
>> also removed all tabs from the source files.
>>
>> Ive also updated show-headers slightly to be a little more
>> error-resistant and to put some emphasis on any header files specified
>> on the command as being of interest . (when there are 140 shown, it can
>> be hard to find the one you are looking for sometimes)
>>
>> Do we wish to impose anything in particular on the source for tools
>> going into this sub-directory of contrib? The other tools in contrib
>> don't seem to have much in the way of coding standards.     I also
>> wonder if anyone other than me will look at them much :-)
> I'm certainly interested in them.
>
> Do you have any sense of whether or not coverage of the tools has 
> improved over short time since we started squashing out conditional 
> compilation?  I was running the header file reordering bits on the 
> trunk and was a bit surprised of how many things they're still 
> changing.  But that would make sense if some files are now being 
> processed that weren't before because we've squashed out the 
> conditional compilation.

hmm. no, i dont have a feel for that.  Anl to be fair, I didn't run the 
tools on every file in trunk.  I limited it to the ones in backend.h, 
and took out even a few of those that were troublesome in some way or 
other at some point.   I wouldnt expect the conditional stuff to affect 
reordering much.  reducing...  we might start to see things like tm.h or 
target.h included less.

A further enhancement in line with that would be to teach the reducer 
about a couple of special files.. like the relationship between 
options.h, tm.h and target.h.   sometimes target.h was included when in 
fact options.h was the only thing actually needed..  During the 
flayttening process I manually handled this by flattening tm.h out of 
target.h and options.h into anything that included tm.h...  so every 
file had options.h, tm.h and target.h explicitly included, and then the 
reducer would just pick the "minimum".  of course, the reorder tool 
works against this by combining them again :-)

however, the tool could be taught when it see target.h for instance, if 
it can't be removed, it could try replacing it with options.h and if 
that fails, tm.h..    That sort of thing could automatically remove 
headers that arent needed because target macros have been turned into 
hooks or something.  I suppose that could even be generalized to trying 
to replace  each header that included other headers...  I wonder how 
safe that would be. hum.

>
> It certainly is true that the total result is smaller than any of the 
> backend, config/ or languages changes that you posted, and I'm running 
> it across the entire source tree, but I'm still surprised at how much 
> churn I'm seeing.
>
> If it weren't for the level of churn, I'd probably be suggesting we 
> just have this stuff run regularly (weekly, monthly, whatever) and 
> commit the result after a sanity looksie.  I've yet to see this tool 
> botch anything and if we're not unnecessarily churning the sources, 
> keeping us clean WRT canononical ordering and duplicate removal 
> automatically seems like a good place to be.
>
it can botch one of the go files.. go has a backend.h of it's own... 
which buggers things up quite nicely since it doesnt include a bunch of 
the headers gcc's backend.h does :-)

The reordering tool is likely safer to run across the board.. especially 
if we can determine the very small subset it shouldn't be run on.

Right now it triggers off the presence of system.h... if system.h is not 
present, it wont do anything to the file. I haven't tried running it 
against *.c to see if there are any other failures, perhaps thats not a 
bad idea.   That will also provide us with a list of files which have 
headers included within conditional compliation... there are a few of 
those :-P  and maybe they could be fixed.  by default it wou=nt do 
anything to those either.

Anyway, if we run it against everything and check it in, then in theory 
there isn't any reason you couldnt spot run it at some interval.. there 
shouldn't be much churn then.

> Maybe do another commit of the reordering output and evaluate again in 
> a month?
>
> I don't think we're quite there on the reducer and it obviously 
> requires more infrastructure in place to test.  But it'd be nice to 
> get to a similar state on that tool.
>
yeah, the reducer still needs some tweaks to be generally runnable I 
think.   IN particular, how to deal with externally supplied macros it 
cant really see.  Im still thinking about that one.

> Which reminds me, you ought to add a VMS target to your tests.  The 
> reducer botched vmsdbgout.c.

Thats one of the reasons vmsdbgout.c wasn't in the list of things I 
reduced :-)

#include "config.h"
#include "system.h"
#include "coretypes.h"

#ifdef VMS_DEBUGGING_INFO
#include "alias.h"
#include "tree.h"
#include "varasm.h"
#include "version.h"
#include "flags.h"
#include "rtl.h"
#include "output.h"
#include "vmsdbg.h"
#include "debug.h"
#include "langhooks.h"
#include "function.h"
#include "target.h"

You have to override it with -i in order to reorder them too.. There are 
headers included within  conditional compilation, and this is one of 
those that is simply not safe...  And yes, I guess adding a vms target 
would cover that for the reducer at least...

I almost suggested moving all the includes out of the conditional...  
which is not unreasonable... and would resolve the issue as well.   It 
would be good to remove all the conditionals in all the source files 
like that.. some are easier than others tho.

back to reordering...  the gen files are a bit of a pain too because of 
the rtl.h conditional inclusions.. which I never really found a good 
solution for...   maybe we should have a brtl.h which is used in concert 
with any source which uses bconfig.h.. brtl.h could verifies bconfig.h 
has been included and then includes those headers it needs, followed by 
rtl.h itself.. and the tool could confirm the right pairing of 
config.h/rtl.h  bconfig.h/brtl.h   is used.   hmm.

Andrew

Re: [patch 4/3] Header file reduction - Tools for contrib - second cut

[Jeff Law]

On 11/03/2015 06:24 AM, Andrew MacLeod wrote:
>>
>> Do you have any sense of whether or not coverage of the tools has
>> improved over short time since we started squashing out conditional
>> compilation?  I was running the header file reordering bits on the
>> trunk and was a bit surprised of how many things they're still
>> changing.  But that would make sense if some files are now being
>> processed that weren't before because we've squashed out the
>> conditional compilation.
>
> hmm. no, i dont have a feel for that.  Anl to be fair, I didn't run the
> tools on every file in trunk.  I limited it to the ones in backend.h,
> and took out even a few of those that were troublesome in some way or
> other at some point.   I wouldnt expect the conditional stuff to affect
> reordering much.  reducing...  we might start to see things like tm.h or
> target.h included less.
Well, the reorder tool will punt if it sees conditional compilation in 
the headers, so I was kind hoping that some of the churn would be 
explainable by the ongoing removal of conditional compilation causing 
files to be processed now that weren't before.   But it appears its 
other factors.

>
> A further enhancement in line with that would be to teach the reducer
> about a couple of special files.. like the relationship between
> options.h, tm.h and target.h.   sometimes target.h was included when in
> fact options.h was the only thing actually needed..  During the
> flayttening process I manually handled this by flattening tm.h out of
> target.h and options.h into anything that included tm.h...  so every
> file had options.h, tm.h and target.h explicitly included, and then the
> reducer would just pick the "minimum".  of course, the reorder tool
> works against this by combining them again :-)
A fair amount of the churn was options.h related.  I'll run it again and 
look closer to see how much exactly.

>>
>> If it weren't for the level of churn, I'd probably be suggesting we
>> just have this stuff run regularly (weekly, monthly, whatever) and
>> commit the result after a sanity looksie.  I've yet to see this tool
>> botch anything and if we're not unnecessarily churning the sources,
>> keeping us clean WRT canononical ordering and duplicate removal
>> automatically seems like a good place to be.
>>
> it can botch one of the go files.. go has a backend.h of it's own...
> which buggers things up quite nicely since it doesnt include a bunch of
> the headers gcc's backend.h does :-)
Cute.

>
> The reordering tool is likely safer to run across the board.. especially
> if we can determine the very small subset it shouldn't be run on.
go, the gen* files perhaps a few others.  Blacklisting and running 
regularly is probably the way to go then.

>
> Right now it triggers off the presence of system.h... if system.h is not
> present, it wont do anything to the file. I haven't tried running it
> against *.c to see if there are any other failures, perhaps thats not a
> bad idea.   That will also provide us with a list of files which have
> headers included within conditional compliation... there are a few of
> those :-P  and maybe they could be fixed.  by default it wou=nt do
> anything to those either.
I didn't know it keyed on system.h.  I'd manually blacklisted testsuite/ 
but otherwise let it run wild just for giggles.  Knowing it keys on 
system.h is helpful in that we don't have to blacklist nearly as much stuff.

And yes, there's a few files with conditional headers.  It wasn't 
terrible and makes a nice todo list for someone new to tackle.

>
> Anyway, if we run it against everything and check it in, then in theory
> there isn't any reason you couldnt spot run it at some interval.. there
> shouldn't be much churn then.
That's the idea and obviously the more automated the better.

> yeah, the reducer still needs some tweaks to be generally runnable I
> think.   IN particular, how to deal with externally supplied macros it
> cant really see.  Im still thinking about that one.
Well, the solution is obvious, we continue the move away from 
conditionally compiled code so that those macros don't matter in the end :-)


>
>> Which reminds me, you ought to add a VMS target to your tests.  The
>> reducer botched vmsdbgout.c.
>
> Thats one of the reasons vmsdbgout.c wasn't in the list of things I
> reduced :-)
Ahem, but vmsdbgout.c was part of the commit on Friday...

>
> back to reordering...  the gen files are a bit of a pain too because of
> the rtl.h conditional inclusions.. which I never really found a good
> solution for...   maybe we should have a brtl.h which is used in concert
> with any source which uses bconfig.h.. brtl.h could verifies bconfig.h
> has been included and then includes those headers it needs, followed by
> rtl.h itself.. and the tool could confirm the right pairing of
> config.h/rtl.h  bconfig.h/brtl.h   is used.   hmm.
I think initially we could blacklist the gen* files.  I'm less concerned 
about the generators than I am the compiler proper.

jeff

Re: [patch 4/3] Header file reduction - Tools for contrib - second cut

[Andrew MacLeod]

On 11/03/2015 09:00 AM, Jeff Law wrote:
>
>> yeah, the reducer still needs some tweaks to be generally runnable I
>> think.   IN particular, how to deal with externally supplied macros it
>> cant really see.  Im still thinking about that one.
> Well, the solution is obvious, we continue the move away from 
> conditionally compiled code so that those macros don't matter in the 
> end :-)
>
yeah but in the meantime its an issue.   I *think* I can simply provide 
to tool with a set of macros to define on the build command whenever it 
tries building a file..    we'll see.

  It should also be possible to extract, after reduction, a list of 
macros that were used in the source file in conditional compilation, but 
which never saw a definition in any of the files.     THat could also be 
useful information.  IN fact, that could be a stand alone analysis 
pretty easily I think...
>
>>
>>> Which reminds me, you ought to add a VMS target to your tests.  The
>>> reducer botched vmsdbgout.c.
>>
>> Thats one of the reasons vmsdbgout.c wasn't in the list of things I
>> reduced :-)
> Ahem, but vmsdbgout.c was part of the commit on Friday...

ahh opps. it snuck back in over time :-P  sorry.

>
>>
>> back to reordering...  the gen files are a bit of a pain too because of
>> the rtl.h conditional inclusions.. which I never really found a good
>> solution for...   maybe we should have a brtl.h which is used in concert
>> with any source which uses bconfig.h.. brtl.h could verifies bconfig.h
>> has been included and then includes those headers it needs, followed by
>> rtl.h itself.. and the tool could confirm the right pairing of
>> config.h/rtl.h  bconfig.h/brtl.h   is used.   hmm.
> I think initially we could blacklist the gen* files.  I'm less 
> concerned about the generators than I am the compiler proper.
>
yeah, its just annoying from a more abstract level (and results from a 
few of the tools)  for rtl.h to have to conditionally include a bunch of 
stuff provided by coretypes.h.

> jeff

Re: [patch 4/3] Header file reduction - Tools for contrib - second cut

[Jeff Law]

Andrew, can you go ahead and commit those changes into contrib?  I think 
in a subdirectory would be best so that you can include the README.

Make sure the permissions are set correctly.  Applying them as a patch 
kept mucking them up.

header-tools or somesuch should be a fine directory name to use.

Generally we haven't required the same level of rigor on the contrib/ 
bits that's required elsewhere.  And I really don't want to lose these 
tools and see them bitrot.

Jeff

Re: [patch 4/3] Header file reduction - Tools for contrib

[David Malcolm]

On Tue, 2015-10-06 at 14:02 +0200, Bernd Schmidt wrote:
[...]
> > No commenting on the quality of python code... :-) I was
> > learning python on the fly.    Im sure some things are QUITE awful.,

[...]

> > + def ii_base (iinfo):
> > +   return iinfo[0]
> > +
> > + def ii_path (iinfo):
> > +   return iinfo[1]
> > +
> > + def ii_include_list (iinfo):
> > +   return iinfo[2]
> > +
> > + def ii_include_list_cond (iinfo):
> > +   return iinfo[3]
> > +
> > + def ii_include_list_non_cond (iinfo):
> > +   l = ii_include_list (iinfo)
> > +   for n in ii_include_list_cond (iinfo):
> > +     l.remove (n)
> > +   return l
> > +
> > + def ii_macro_consume (iinfo):
> > +   return iinfo[4]
> > +
> > + def ii_macro_define (iinfo):
> > +   return iinfo[5]
> > +
> > + def ii_src (iinfo):
> > +   return iinfo[6]
> > +
> > + def ii_src_line (iinfo):
> > +   return iinfo[7]
> 
> That's a lot of little functions with pretty much no clue for the reader 
> what's going on. It looks like maybe there's an array where a struct 
> should have been used?

FWIW, this kind of thing is often made a lot neater and easier to debug
by using "namedtuple" from within the "collections" module in the
standard library:

https://docs.python.org/2/library/collections.html#collections.namedtuple

which lets you refer e.g. to field 5 of the tuple as a "define"
attribute.

  iinfo.define

and avoid all these accessor functions (and you can add methods and
properties, giving e.g. a "list_non_cond").

Not that I'm asking you to rewrite it; merely that namedtuple is one of
many gems in the python stdlib that are worth knowing about.

[...]

Hope this is constructive
Dave