On the toplevel configure and build system

On the toplevel configure and build system

[Joseph S. Myers]

Having been cleaning up the toplevel configure.ac in various ways 
following removing deprecated targets for GCC 4.7, I would like to propose 
some principles relating to the toplevel configure and build system.  
These are intended as principles to guide future development and indicate 
the directions in which this system should develop - not something I have 
any immediate plans to implement most of.

1. The toplevel configure and build system exists to build the components 
in the GCC and src repositories.  These include GCC, binutils, GDB, newlib 
and some other components such as cgen and winsup, as well as libraries 
used by these components.

Additional tools for the build (not host) system may be built (not 
installed) when present in the source tree, if of direct use in building 
and testing the components in those repositories, and likewise additional 
libraries used by build or host tools or target libraries in those 
repositories may also be built.  But support for tools and libraries that 
do not meet that criterion should be considered obsolete and removed.

Specifically, I propose removal of all support for building: ash autoconf 
automake bash byacc bzip2 diff dosutils fileutils findutils find gawk 
gettext gnuserv gzip hello indent libiconv libtool make mmalloc patch perl 
prms rcs release recode sed send-pr shellutils tar textutils time uudecode 
wdiff zip expect guile target-gperf target-examples target-qthreads.  See 
<http://gcc.gnu.org/ml/gcc-patches/2011-03/msg01674.html> and followups 
regarding reasons to keep a few packages in my original list that aren't 
in the list above.

Furthermore, the system is for the *current code* in those repositories.  
The existence of some out-of-tree version with additional or different 
target support is irrelevant; such trees should include their own local 
toplevel patches as needed.

2. If you put directories from the GCC repository into your build, you 
should expect GCC and its libraries to be built; toplevel should not 
disable GCC on the grounds that GCC does not support a given target.  
Similarly, if you put the src repository or part thereof into your build, 
you should expect at least something from the src repository to be built, 
so if nothing there has support for a target then toplevel should not try 
to disable everything from src.  However, since you may well have a 
checkout with both supported and unsupported parts of src, it is useful 
(for example) to disable gdb if a target has a binutils port but not a gdb 
port.

If in future the src repository is split up into separate repositories for 
different projects, this principle will apply in a more fine-grained way - 
so if there is a "binutils+gdb" repository then toplevel should not try to 
disable the whole of binutils+gdb for targets supporting none of it, only 
subsets for targets supporting a nonempty subset.

3. If it is known that GCC's libraries for a given language do not build 
for a given target, it is generally appropriate to disable that language 
(via unsupported_languages) instead of disabling the libraries directly, 
unless there is some reason the front end is considered useful without the 
libraries.  If it is not known, then the language and libraries should not 
be disabled (subject to some languages not being built by default anyway).  
Various reasons were advanced in reply to 
<http://gcc.gnu.org/ml/gcc/2011-03/msg00326.html> for the default 
disabling of libgcj on many targets, but I think we can reasonably 
conclude that a more carefully considered set of disablings is 
appropriate.

4. It would be desirable to get the information about supported 
directories out of the toplevel configure.ac altogether, and into 
subdirectories.  Subdirectories with known code with portability issues 
should provide configure fragments that can be sourced both at toplevel 
and within that subdirectory, and that set variables in an appropriate 
namespace that indicate the state of support for that directory.  Toplevel 
should then use these fragments to disable directories or GCC languages as 
appropriate.  For example, libffi might provide a fragment defining 
libffi_target and libffi_targetdir (where libffi/configure.ac presently 
defines TARGET and TARGETDIR), and toplevel would source it, and, if 
libffi_targetdir ended up as "unknown", disable languages (java and go) 
that use target-libffi.  Since libffi is the most obvious source of 
portability issues for libgcj, perhaps this could be an initial 
replacement for the ${libgcj} disabling.  Such things as libgloss_dir 
should also come from subdirectories.

5. Similarly, it is unfortunate that a lot of knowledge about how to 
configure GCC with various host libraries such as PPL is embedded in the 
toplevel configure script, and it would be better for this to be in the 
gcc/ subdirectory if possible.  Both this and the previous principle would 
reduce the frequency with which logically GCC-specific changes need 
toplevel changes synced to the src repository.

6. Splitting up the src repository into multiple separate repositories 
would be desirable as part of moving components away from CVS; different 
projects should not be tied so closely to each other.  I propose the 
following as a natural division (moving anything out is complicated, with 
various scripts, documentation etc. to update, but as noted here it should 
be possible to move cgen and rda without any dependencies on other 
projects):

(a) toplevel.  Master repository for the shared toplevel files, config/, 
include/, intl/, libdecnumber/, libiberty/, texinfo/.  Changes merged from 
here to other repositories in whatever way is convenient (depending on the 
version control systems etc. for each repository).

(b) binutils+gdb.  Includes the cpu, etc, libgui, readline and sim 
subdirectories of the src repository (could consider putting libgui and 
other gdbtk code only in a separate repository or branch rather than the 
main binutils+gdb one).  I think the parts of include/ in the src 
repository that aren't shared by GCC can also go here, not in the master 
toplevel repository (although a separate directory for those would be 
better than the present system of having different sets of files in 
include/ in different places).

(c) newlib.  Includes newlib and libgloss.

(d) winsup/Cygwin.  Includes the winsup directory; may need to be based on 
the newlib repository.

(e) cgen.  Does not appear to use libiberty or other toplevel code so need 
not have any relation to the shared toplevel, and if moved out then 
toplevel need no longer support it except maybe as a build tool.  (Where 
cgen descriptions are used to generate code in binutils, those 
descriptions should go in cpu/ in the binutils+gdb repository, not 
cgen/cpu/ in the cgen repository.)

(f) rda.  Again, appears completely independent and toplevel support could 
be removed if it moves out.

(g) sid.

(h) utils - I don't know what to do with this directory or where it best 
goes.

(i) I specifically propose not moving tcl, tk or itcl to any new 
repositories; they would better be removed from HEAD with the expectation 
being that people use preinstalled copies (though the option of building 
the libraries in-tree could be kept if necessary).  It would be good to 
get readline to this state as well (support building sources dropped in 
but don't include a local copy).


Any comments on these principles?

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: On the toplevel configure and build system

[DJ Delorie]

> 2. If you put directories from the GCC repository into your build, you 
> should expect GCC and its libraries to be built; toplevel should not 
> disable GCC on the grounds that GCC does not support a given target.  

I disagree.  We have a single combined gcc+binutils+etc internal tree
that's used for many targets; some support gcc, some do not.  I added
another "don't build gcc for this" just last week.

> If in future the src repository is split up into separate repositories for 
> different projects, this principle will apply in a more fine-grained way - 

I see no reason to stop people from building in a combined source tree
for multiple targets, and expecting it to work.

Re: On the toplevel configure and build system

[Geoffrey Keating]

"Joseph S. Myers" <joseph@codesourcery.com> writes:

> 2. If you put directories from the GCC repository into your build, you 
> should expect GCC and its libraries to be built; toplevel should not 
> disable GCC on the grounds that GCC does not support a given target.  

I'd appreciate it if creating a combined tree and building it still
worked on/for every target, even those which one or more of GDB, GCC,
newlib, sid, etc., do not support.

Re: On the toplevel configure and build system

[Joseph S. Myers]

On Tue, 29 Mar 2011, DJ Delorie wrote:

> > 2. If you put directories from the GCC repository into your build, you 
> > should expect GCC and its libraries to be built; toplevel should not 
> > disable GCC on the grounds that GCC does not support a given target.  
> 
> I disagree.  We have a single combined gcc+binutils+etc internal tree
> that's used for many targets; some support gcc, some do not.  I added
> another "don't build gcc for this" just last week.

Well, disabling gcc for targets where it is unsupported is hardly 
consistent existing practice in the FSF tree; before my patch 
<http://gcc.gnu.org/ml/gcc-patches/2011-03/msg02027.html> there were just 
two targets (c54x*-*-* | tic54x-*-*, v810-*-*) disabling the gcc 
directory, so that patch makes things consistent with the normal practice 
for non-gcc-supporting targets by removing those special cases.

I don't however object much to toplevel disabling gcc on targets without 
support (if you have a GCC-only tree, it *does* seem unfortunate for 
configuring it for an unsupported target to quietly build nothing and exit 
with status 0, instead of exiting with error status), as long as it is 
done consistently in accordance with principle 4: that is, through 
config.gcc being adapted so toplevel configure.ac can use it to get 
positive information from the gcc/ directory about supported targets, 
rather than through negative declarations at toplevel for unsupported 
targets.  And much the same applies to other components in a unified tree: 
negative declarations that something is unsupported are much worse than 
positive information from a subdirectory about what is supported there, 
and it would be good for one-component-only trees to be able to give 
meaningful errors if nothing useful gets built - maybe some check is 
needed to give errors if no installed host tool or target library would be 
built.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: On the toplevel configure and build system

[Ian Lance Taylor]

"Joseph S. Myers" <joseph@codesourcery.com> writes:

> Specifically, I propose removal of all support for building: ash autoconf 
> automake bash byacc bzip2 diff dosutils fileutils findutils find gawk 
> gettext gnuserv gzip hello indent libiconv libtool make mmalloc patch perl 
> prms rcs release recode sed send-pr shellutils tar textutils time uudecode 
> wdiff zip expect guile target-gperf target-examples target-qthreads.  See 
> <http://gcc.gnu.org/ml/gcc-patches/2011-03/msg01674.html> and followups 
> regarding reasons to keep a few packages in my original list that aren't 
> in the list above.

The expect program is used by dejagnu, and I believe it will be used
when running tests if both dejagnu and expect are in the source tree.
At least, it used to work that way.  Everything else looks OK to me.

Ian

Re: On the toplevel configure and build system

[Frank Ch. Eigler]

dj wrote:

>> [...]
> I see no reason to stop people from building in a combined source tree
> for multiple targets, and expecting it to work.

Perhaps if we do move to git for all the /src stuff, we can have a
/toplevel git repository with different branches suitable for each of
your tastes of such policy.

- FChE

Re: On the toplevel configure and build system

[Joseph S. Myers]

On Tue, 29 Mar 2011, Ian Lance Taylor wrote:

> "Joseph S. Myers" <joseph@codesourcery.com> writes:
> 
> > Specifically, I propose removal of all support for building: ash autoconf 
> > automake bash byacc bzip2 diff dosutils fileutils findutils find gawk 
> > gettext gnuserv gzip hello indent libiconv libtool make mmalloc patch perl 
> > prms rcs release recode sed send-pr shellutils tar textutils time uudecode 
> > wdiff zip expect guile target-gperf target-examples target-qthreads.  See 
> > <http://gcc.gnu.org/ml/gcc-patches/2011-03/msg01674.html> and followups 
> > regarding reasons to keep a few packages in my original list that aren't 
> > in the list above.
> 
> The expect program is used by dejagnu, and I believe it will be used
> when running tests if both dejagnu and expect are in the source tree.
> At least, it used to work that way.  Everything else looks OK to me.

I included expect in the list on the basis of what Paolo said in 
<http://gcc.gnu.org/ml/gcc-patches/2011-03/msg01709.html> and 
<http://gcc.gnu.org/ml/gcc-patches/2011-03/msg01726.html>.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: On the toplevel configure and build system

[DJ Delorie]

> Perhaps if we do move to git for all the /src stuff, we can have a
> /toplevel git repository with different branches suitable for each
> of your tastes of such policy.

Perhaps I misunderstand how this would work (/me hates git) but I'm
not looking forward to adding support to umpteen million branches each
time I do a base port...

Re: On the toplevel configure and build system

[Joseph S. Myers]

On Tue, 29 Mar 2011, Frank Ch. Eigler wrote:

> Perhaps if we do move to git for all the /src stuff, we can have a

It is very strongly in my principle 6 that there should be no "we" moving 
"for all the /src stuff" - that it should be made possible for each 
project (a) through (i) to make its own decisions about version control 
systems and repository hosting, and to have its own set of people with 
write access, because the interdependencies associated with the shared 
repository are one of the greatest flaws in the present system.  I believe 
it is already possible for cgen and rda to move our of src on their own 
because they appear to have no dependence on the toplevel infrastructure.  
(Other components would complicate merge issues if they move without the 
toplevel repository (a) being created first.)

> /toplevel git repository with different branches suitable for each of
> your tastes of such policy.

I am hoping that the toplevel repository, (a) in my list, should not have 
branches at all.  All branching and tagging should take place in 
repositories for individual projects (which could also choose when to 
merge changes from the toplevel repository if merges at a particular time 
are considered risky in the light of planned branches or releases).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: On the toplevel configure and build system

[Tom Tromey]

>>>>> "Joseph" == Joseph S Myers <joseph@codesourcery.com> writes:

Joseph> Additional tools for the build (not host) system may be built
Joseph> (not installed) when present in the source tree, if of direct
Joseph> use in building and testing the components in those
Joseph> repositories, and likewise additional libraries used by build or
Joseph> host tools or target libraries in those repositories may also be
Joseph> built.  But support for tools and libraries that do not meet
Joseph> that criterion should be considered obsolete and removed.

Joseph> Specifically, I propose removal of all support for building:
Joseph> ... libiconv ...

libiconv is used by some people building gdb, to work around limitations
in the host platform's iconv.

Tom

Re: On the toplevel configure and build system

[Joseph S. Myers]

On Tue, 29 Mar 2011, Joseph S. Myers wrote:

> Specifically, I propose removal of all support for building: ash autoconf 
> automake bash byacc bzip2 diff dosutils fileutils findutils find gawk 
> gettext gnuserv gzip hello indent libiconv libtool make mmalloc patch perl 
> prms rcs release recode sed send-pr shellutils tar textutils time uudecode 
> wdiff zip expect guile target-gperf target-examples target-qthreads.  See 
> <http://gcc.gnu.org/ml/gcc-patches/2011-03/msg01674.html> and followups 
> regarding reasons to keep a few packages in my original list that aren't 
> in the list above.

There's been push-back on libiconv and expect.

Having looked more at tools in src, I see that cgen has code for using 
in-tree guile - but there are no associated dependencies at toplevel.  I 
continue to think that the right model for cgen is a separate repository 
not containing any of the toplevel machinery or makefile rules referring 
outside the source directory; the rules for converting files in the 
toplevel cpu/ directory to C code used in opcodes etc. should go in the 
toplevel cpu/ directory (or elsewhere in binutils+gdb) and should be 
written to use an installed cgen binary.

I've seen no push-back on the other packages here, and some of those not 
proposed for removal may nevertheness not be of current use.

> 2. If you put directories from the GCC repository into your build, you 
> should expect GCC and its libraries to be built; toplevel should not 
> disable GCC on the grounds that GCC does not support a given target.  

There have been objections to this principle, so I propose a weaker:

2'. If the combination of directories present in the source tree would not 
build and install anything for the host or target (other than host 
libraries such as libiberty) then there should be an error at configure or 
build time.  For example, a GCC source tree should not quietly build 
nothing because GCC isn't supported for the target; a binutils+gdb tree 
should not quietly build nothing for lack of BFD support, although it 
might build only a subset of binutils if ld and gdb aren't supported.

> 6. Splitting up the src repository into multiple separate repositories 
> would be desirable as part of moving components away from CVS; different 
> projects should not be tied so closely to each other.  I propose the 
> following as a natural division (moving anything out is complicated, with 
> various scripts, documentation etc. to update, but as noted here it should 
> be possible to move cgen and rda without any dependencies on other 
> projects):

I'd still like comments on how

> (h) utils - I don't know what to do with this directory or where it best 
> goes.

relates to the other bits of src and where it would best go if src is 
split up.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: On the toplevel configure and build system

[DJ Delorie]

> > (h) utils - I don't know what to do with this directory or where it best 
> > goes.
> 
> relates to the other bits of src and where it would best go if src is 
> split up.

For example, the MeP utils is used to reconfigure gcc, binutils, cgen,
sid, and a few other places (libgloss) according to the templates
provided by the chip fab (MeP is a synthetic chip) so as to match the
specific cpu configurations in your chip (how many cores, which
options each core has, memory layout, etc).

(for the lawyers: we were careful to make the resulting files just as
editable as one would expect, you do not *need* these utils to
build/hack/use the gnu tools, only if you want an easier way to
customize them for specific hardware)

Re: On the toplevel configure and build system

[Joseph S. Myers]

On Wed, 30 Mar 2011, DJ Delorie wrote:

> > > (h) utils - I don't know what to do with this directory or where it best 
> > > goes.
> > 
> > relates to the other bits of src and where it would best go if src is 
> > split up.
> 
> For example, the MeP utils is used to reconfigure gcc, binutils, cgen,
> sid, and a few other places (libgloss) according to the templates
> provided by the chip fab (MeP is a synthetic chip) so as to match the
> specific cpu configurations in your chip (how many cores, which
> options each core has, memory layout, etc).

Thanks.  My inclination is to say that this should be considered an 
independent tool in its own repository, as something not required in the 
build of any of the other tools.  More specifically, utils/mep and 
utils/wince look like independent tools each of which would better go in 
its own toplevel directory (mep-integrator, cesetup) (and would each go in 
an independent repository based on the shared toplevel, since they use 
libiberty), while utils/spu appears to have no toplevel dependencies and 
so should be completely independent, possibly without toplevel support for 
building it.  Since utils/spu and utils/wince have no non-build-system 
changes since 2000, I'd be inclined to say we should declare those two 
subdirectories dead and run "cvs rm" on them - people wanting to resurrect 
them can always extract the data from CVS later.  (And I still think 
utils/mep should move to its own toplevel directory.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: On the toplevel configure and build system

[Paolo Bonzini]

On 03/30/2011 05:54 PM, Joseph S. Myers wrote:
> Thanks.  My inclination is to say that this should be considered an
> independent tool in its own repository, as something not required in the
> build of any of the other tools.  More specifically, utils/mep and
> utils/wince look like independent tools each of which would better go in
> its own toplevel directory (mep-integrator, cesetup) (and would each go in
> an independent repository based on the shared toplevel, since they use
> libiberty), while utils/spu appears to have no toplevel dependencies and
> so should be completely independent, possibly without toplevel support for
> building it.  Since utils/spu and utils/wince have no non-build-system
> changes since 2000, I'd be inclined to say we should declare those two
> subdirectories dead and run "cvs rm" on them - people wanting to resurrect
> them can always extract the data from CVS later.  (And I still think
> utils/mep should move to its own toplevel directory.)

No, these tools _are_ built after all.

However, moving them to a new toplevel directory and getting rid of 
utils would be a good thing.

Paolo

Re: On the toplevel configure and build system

[Paolo Bonzini]

On 03/30/2011 05:54 PM, Joseph S. Myers wrote:
> Thanks.  My inclination is to say that this should be considered an
> independent tool in its own repository, as something not required in the
> build of any of the other tools.  More specifically, utils/mep and
> utils/wince look like independent tools each of which would better go in
> its own toplevel directory (mep-integrator, cesetup) (and would each go in
> an independent repository based on the shared toplevel, since they use
> libiberty), while utils/spu appears to have no toplevel dependencies and
> so should be completely independent, possibly without toplevel support for
> building it.  Since utils/spu and utils/wince have no non-build-system
> changes since 2000, I'd be inclined to say we should declare those two
> subdirectories dead and run "cvs rm" on them - people wanting to resurrect
> them can always extract the data from CVS later.  (And I still think
> utils/mep should move to its own toplevel directory.)

No, these tools _are_ built after all.

However, moving them to a new toplevel directory and getting rid of 
utils would be a good thing.

Paolo

Re: On the toplevel configure and build system

[Paolo Bonzini]

On 03/30/2011 05:15 PM, Joseph S. Myers wrote:
> On Tue, 29 Mar 2011, Joseph S. Myers wrote:
>
>> Specifically, I propose removal of all support for building: ash autoconf
>> automake bash byacc bzip2 diff dosutils fileutils findutils find gawk
>> gettext gnuserv gzip hello indent libiconv libtool make mmalloc patch perl
>> prms rcs release recode sed send-pr shellutils tar textutils time uudecode
>> wdiff zip expect guile target-gperf target-examples target-qthreads.  See
>> <http://gcc.gnu.org/ml/gcc-patches/2011-03/msg01674.html>  and followups
>> regarding reasons to keep a few packages in my original list that aren't
>> in the list above.
>
> There's been push-back on libiconv and expect.

I still think expect has no reason to stay in the toplevel (not any more 
than tcl or tk).

dejagnu is another story, and there's good reasons for leaving libiconv 
as well.

> 2'. If the combination of directories present in the source tree would not
> build and install anything for the host or target (other than host
> libraries such as libiberty) then there should be an error at configure or
> build time.  For example, a GCC source tree should not quietly build
> nothing because GCC isn't supported for the target; a binutils+gdb tree
> should not quietly build nothing for lack of BFD support, although it
> might build only a subset of binutils if ld and gdb aren't supported.

Agreed.

Paolo

Re: On the toplevel configure and build system

[Joseph S. Myers]

On Thu, 31 Mar 2011, Paolo Bonzini wrote:

> On 03/30/2011 05:54 PM, Joseph S. Myers wrote:
> > Thanks.  My inclination is to say that this should be considered an
> > independent tool in its own repository, as something not required in the
> > build of any of the other tools.  More specifically, utils/mep and
> > utils/wince look like independent tools each of which would better go in
> > its own toplevel directory (mep-integrator, cesetup) (and would each go in
> > an independent repository based on the shared toplevel, since they use
> > libiberty), while utils/spu appears to have no toplevel dependencies and
> > so should be completely independent, possibly without toplevel support for
> > building it.  Since utils/spu and utils/wince have no non-build-system
> > changes since 2000, I'd be inclined to say we should declare those two
> > subdirectories dead and run "cvs rm" on them - people wanting to resurrect
> > them can always extract the data from CVS later.  (And I still think
> > utils/mep should move to its own toplevel directory.)
> 
> No, these tools _are_ built after all.

My claim is that while they *are* built, the lack of substantive changes 
in the past decade is evidence that utils/spu and utils/wince are probably 
no longer being used.

Stan, you appear to be the only person with non-build-system changes in 
the utils/spu ChangeLog (from the years 1994 and 2000).  Do you have any 
reason to believe it is still being used, or is it OK to run "cvs rm" on 
that directory's contents in the src repository?

DJ, the same questions apply for utils/wince; you appear to have the only 
non-build-system changes, in 1999 and 2000.

In the modules file the only place utils appears is in old-gdb (that is, 
it used to be considered part of the GDB sources but no longer is).

> However, moving them to a new toplevel directory and getting rid of utils
> would be a good thing.

Moving utils/mep to its own toplevel directory, that is, if the other two 
aren't being used.

-- 
Joseph S. Myers
joseph@codesourcery.com