Possible contradiction in INSTALL and manual/install.texi

Possible contradiction in INSTALL and manual/install.texi

[Andrea Monaco]

Hello.


Both the INSTALL file and manual/install.texi claim that a separate
build directory "allows removing the whole build directory in case an
error occurs, which is the safest way to get a fresh start and should
always be done".

But soon later they also claim that "even though you're building in a
separate build directory, the compilation may need to create or modify
files and directories in the source directory".

I think that this is close to a contradiction and can be clarified; at
least I don't find it very clear as a beginner to glibc development.


Maybe the paragraph could summarise what changes the build process does
on the source tree, whether they are reversible with simple operations
like removing some files, and whether such changes make a new build with
a different configuration possible.



Thanks,

Andrea Monaco

Re: Possible contradiction in INSTALL and manual/install.texi

[Joseph Myers]

On Fri, 17 Dec 2021, Andrea Monaco via Libc-alpha wrote:

> But soon later they also claim that "even though you're building in a
> separate build directory, the compilation may need to create or modify
> files and directories in the source directory".

See fix_glibc_timestamps in scripts/build-many-glibcs.py for the cases 
where a file in the source directory might get regenerated if the 
timestamps are out of order.  If you touch files as in that function 
before a build, nothing should get regenerated in the source directory.

There were previously other cases both of files in the source directory 
that might get regenerated as part of the build, depending on timestamps, 
and generated files that went in the source directory rather than the 
build directory.  See, for example, commit 
f2da2fd81f1d3f43678de9cf39b12692c6fa449b ("Do not build .mo files in 
source directory (bug 14121).", 15 Dec 2017) for a fix of one such case, 
with a commit message listing various other such cases that have since 
been fixed.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Possible contradiction in INSTALL and manual/install.texi

[Zack Weinberg]

On Mon, Dec 20, 2021, at 4:10 PM, Joseph Myers wrote:
> On Fri, 17 Dec 2021, Andrea Monaco via Libc-alpha wrote:
>
>> But soon later they also claim that "even though you're building in a
>> separate build directory, the compilation may need to create or modify
>> files and directories in the source directory".
>
> See fix_glibc_timestamps in scripts/build-many-glibcs.py for the cases 
> where a file in the source directory might get regenerated if the 
> timestamps are out of order.  If you touch files as in that function 
> before a build, nothing should get regenerated in the source directory.

Personally, I consider it a bug if:

- any file that is generated is checked into git; no exceptions
- any file that is generated is written to the source directory;
  with the exception of files generated or copied in by autoreconf -i

However, some such bugs have been tolerated for a very long time because they are difficult to fix and rarely cause grief in practice; for instance, sysdeps/mach/hurd/bits/errno.h falls into this category (it's generated from the manual and *shouldn't* need to be checked in, but because it's needed very early in the build, because it changes very rarely, because glibc's Makefiles are such a steaming pile of spaghetti, and because most of the active developers don't have convenient access to a Hurd system to validate fixes with, we've chosen to leave it alone).

Patches bringing the build system closer to the ideal of 'nothing written into the source directory at build time' would be appreciated.

zw

Re: Possible contradiction in INSTALL and manual/install.texi

[Joseph Myers]

On Tue, 21 Dec 2021, Zack Weinberg via Libc-alpha wrote:

> On Mon, Dec 20, 2021, at 4:10 PM, Joseph Myers wrote:
> > On Fri, 17 Dec 2021, Andrea Monaco via Libc-alpha wrote:
> >
> >> But soon later they also claim that "even though you're building in a
> >> separate build directory, the compilation may need to create or modify
> >> files and directories in the source directory".
> >
> > See fix_glibc_timestamps in scripts/build-many-glibcs.py for the cases 
> > where a file in the source directory might get regenerated if the 
> > timestamps are out of order.  If you touch files as in that function 
> > before a build, nothing should get regenerated in the source directory.
> 
> Personally, I consider it a bug if:
> 
> - any file that is generated is checked into git; no exceptions
> - any file that is generated is written to the source directory;
>   with the exception of files generated or copied in by autoreconf -i

Note that there are at least three classes of generated files in the 
source directory, with regard to how they might be regenerated by 
Makefiles: (a) files that might be regenerated during building and testing 
glibc because of Makefile dependencies (the cases touched in 
build-many-glibcs.py); (b) files where there are Makefile rules to rebuild 
them but nothing should ever regenerate them during a build because 
nothing in a build depends on them (e.g. INSTALL - a generated file not 
used in the build); (c) files without Makefile rules to regenerate them, 
that are only regenerated manually.  One can also argue for considering 
generated files together with a related category: (d) source files that 
are maintained elsewhere and only copied into glibc verbatim.

There are also various different reasons for having generated files 
included in the repository, including allowing release tarballs to be 
built with "git archive", and reducing dependencies (or dependencies on 
particular versions of tools) for building and testing glibc.

By way of example, having auto-libm-test-out-* checked in avoids people 
building and testing glibc needing to have headers and .so symlinks for 
GMP / MPFR / MPC, avoids people needing to have new-enough versions of 
those libraries (sometimes more recent than widely available in 
distributions) to avoid any bugs or missing features that show up in 
generating that test data, and with older MPC versions also served to 
avoid a very slow regeneration process (it used to take about an hour, 
newer MPC versions have made it much faster) unless you changed the input 
data for a few functions where slowness was an issue.  It also means that 
gen-auto-libm-tests, as a tool only needed when modifying 
auto-libm-test-in rather than for normal glibc build and test, can freely 
make use of glibc features itself, without needing to be concerned about 
portability to any non-GNU systems people might use to cross-compile glibc 
(if anyone actually does that).

-- 
Joseph S. Myers
joseph@codesourcery.com