GSoC Blog Post 0 - GCCprefab build system

GSoC Blog Post 0 - GCCprefab build system

[Wileam Yonatan Phan]

Hi everyone,

This weekend marks the end of the community bonding period for GSoC '22, and
here’s my progress so far with the GSoC project on Fortran DO CONCURRENT.

I've initialized a GCC mirror on GitHub where I plan to track all patches that
I will create during the GSoC:
<https://github.com/wyphan/gfortran-do-concurrent>

I met with Tobias over a MS Teams call on May 30, 2022. Together, we picked GCC
PR# 102003 as a good starter issue to start delving into the Fortran parser in
GCC. He also guided me through how to debug the compiler using gdb. Tobias,
thanks a lot for spending some of your (technically) vacation time with me!

In the meantime, I’ve implemented a simple build script system for GCC that I
christen "GCCprefab". Before this build system existed, there are only two
relatively easy ways to build GCC painlessly:

1. Using Spack package manager: `spack install gcc'
2. Using the install script for OpenCoarrays
<https://github.com/sourceryinstitute/OpenCoarrays>

The name pays homage to _prefab_ricated buildings such as sheds/barns (even
sections of houses) that are commonly sold in the US at hardware stores such as
Home Depot or Lowe’s. It’s arguably an overengineered solution to my laziness
having to memorize all the different configure flags when building GCC from
sources.

Right now, GCCprefab has the following features:

- One single script written in Bash
- One single argument: a config file written in a custom format inspired by
  Spack spec syntax and the Windows INI / TOML format for configuration files
- Clones the main GCC Git repo, or a custom mirror of your choice
- Upon execution, logs standard output for each phase of the build process into
  a timestamped log file, which is xz-compressed after each phase completes
  successfully
- Licensed under the FSF-compatible Apache 2.0 license

To try it out, you can head over to my GitHub repo:
<https://github.com/wyphan/gccprefab>

Please feel free to open an issue there if you found a bug or to suggest new
features. Pull requests are welcome too!

Also, please feel free to jump in anytime to the discussion thread at the
Fortran-lang Discourse forums:
<
https://fortran-lang.discourse.group/t/gsoc-2022-accelerating-fortran-do-concurrent-in-gcc/3269
>

Thanks,
Wil

Re: GSoC Blog Post 0 - GCCprefab build system

[Jonathan Wakely]

On Mon, 13 Jun 2022 at 02:51, Wileam Yonatan Phan via Gcc
<gcc@gcc.gnu.org> wrote:
>
> Hi everyone,
>
> This weekend marks the end of the community bonding period for GSoC '22, and
> here’s my progress so far with the GSoC project on Fortran DO CONCURRENT.
>
> I've initialized a GCC mirror on GitHub where I plan to track all patches that
> I will create during the GSoC:
> <https://github.com/wyphan/gfortran-do-concurrent>
>
> I met with Tobias over a MS Teams call on May 30, 2022. Together, we picked GCC
> PR# 102003 as a good starter issue to start delving into the Fortran parser in
> GCC. He also guided me through how to debug the compiler using gdb. Tobias,
> thanks a lot for spending some of your (technically) vacation time with me!
>
> In the meantime, I’ve implemented a simple build script system for GCC that I
> christen "GCCprefab". Before this build system existed, there are only two
> relatively easy ways to build GCC painlessly:
>
> 1. Using Spack package manager: `spack install gcc'
> 2. Using the install script for OpenCoarrays
> <https://github.com/sourceryinstitute/OpenCoarrays>

I disagree there are "only two" ways.

https://gcc.gnu.org/wiki/InstallingGCC describes an arguably much simpler way.

Re: GSoC Blog Post 0 - GCCprefab build system

[Wileam Yonatan Phan]

Hi Jonathan,

Thanks for the feedback. Regarding the linked page <
https://gcc.gnu.org/wiki/InstallingGCC>
if you're referring to the part that tells you to use your distro's package
manager, yes that's indeed the simplest way to install GCC, but from pre-built
binaries, not building directly from sources. But if you're referring to the
example at the bottom of the page, this script does exactly that, but the build
process is automated based on the build configuration file.

Thanks again,
Wil

On Mon, 2022-06-13 at 12:59 +0100, Jonathan Wakely wrote:
> On Mon, 13 Jun 2022 at 02:51, Wileam Yonatan Phan via Gcc
> <
> gcc@gcc.gnu.org
> > wrote:
> > Hi everyone,
> > 
> > This weekend marks the end of the community bonding period for GSoC '22,
> > and
> > here’s my progress so far with the GSoC project on Fortran DO CONCURRENT.
> > 
> > I've initialized a GCC mirror on GitHub where I plan to track all patches
> > that
> > I will create during the GSoC:
> > <
> > https://github.com/wyphan/gfortran-do-concurrent
> > >
> > 
> > I met with Tobias over a MS Teams call on May 30, 2022. Together, we picked
> > GCC
> > PR# 102003 as a good starter issue to start delving into the Fortran parser
> > in
> > GCC. He also guided me through how to debug the compiler using gdb. Tobias,
> > thanks a lot for spending some of your (technically) vacation time with me!
> > 
> > In the meantime, I’ve implemented a simple build script system for GCC that
> > I
> > christen "GCCprefab". Before this build system existed, there are only two
> > relatively easy ways to build GCC painlessly:
> > 
> > 1. Using Spack package manager: `spack install gcc'
> > 2. Using the install script for OpenCoarrays
> > <
> > https://github.com/sourceryinstitute/OpenCoarrays
> > >
> 
> I disagree there are "only two" ways.
> 
> https://gcc.gnu.org/wiki/InstallingGCC
>  describes an arguably much simpler way.
> 

Re: GSoC Blog Post 0 - GCCprefab build system

[Jonathan Wakely]

On Mon, 13 Jun 2022 at 14:34, Wileam Yonatan Phan wrote:
>
> Hi Jonathan,
>
> Thanks for the feedback. Regarding the linked page <
> https://gcc.gnu.org/wiki/InstallingGCC>
> if you're referring to the part that tells you to use your distro's package
> manager, yes that's indeed the simplest way to install GCC, but from pre-built
> binaries, not building directly from sources. But if you're referring to the
> example at the bottom of the page,

Yes, that's what I mean.

> this script does exactly that, but the build
> process is automated based on the build configuration file.

Yes, it does that, but takes 400 lines of shell script to do so.

If you want "relatively easy ways to build GCC painlessly" then you
can do it with nine lines of shell commands.

Or in about 80, for any non-prehistoric version, with no config file
needed (just a single option to the script, the release number or
snapshot name to build):
https://gist.github.com/jwakely/95b3a790157f55d75e18f577e12b50d7#file-build_gcc_versions-sh

My point is just that there are already easier ways to do it, not only
by using spack or the OpenCoarrays build script.

N.B. your https://github.com/wyphan/gccprefab/blob/main/gcc12.cfg
config file says "Latest GCC 12 Release" but actually builds from the
tip of the branch, not a release.

Re: GSoC Blog Post 0 - GCCprefab build system

[Wileam Yonatan Phan]

Hi Jonathan,

Your build script is really impressive! I'm definitely not at that level yet
when it comes to Bash shell scripting. I'll start deciphering it and improve my
script accordingly.

By "relatively easy", I refer to the end-user of the script being able to
customize the build as s/he sees fit through the config file, and simply
running

$ ./build.sh config.cfg

substituting `config.cfg` with said build config file. So, one single line of
shell command instead of nine lines.

I also pepper my code with plenty of comments and strategically-placed
linebreaks for readability, which adds to the script length.

To be fair, most of the ~300 lines of the script belongs to the config file
interpreter function readcfg() because it uses if..elif..fi blocks instead of
case..esac blocks, and the bulk of the function is mostly for adding configure
flags, languages to compile, etc. (I plan to extend the script even further
once I start building GCC with GPU offloading support.)

Actually, the very first version of the script takes in a version string (like
your script) instead of a config file. You can access it here as a GitHub gist:
<
https://gist.github.com/wyphan/e2672336b80253a813abb795a5db567c/86eee347a8d26881a91356d9007790c5cff6654f
>
I later realized that relying on just the version string is insufficient for my
needs, specifically for adding configure flags like `--disable-bootstrap`,
which is hard-coded in that early version of the script.

Yes, you are correct that the script currently uses the tip of the
`releases/gcc-[version]` branches. Is that not the same as using the published
tarballs on the FTP server? I _do_ notice that these branches only get pushed
whenever a minor version or a patchlevel is released.

Thanks,
Wil

On Mon, 2022-06-13 at 16:26 +0100, Jonathan Wakely wrote:
> On Mon, 13 Jun 2022 at 14:34, Wileam Yonatan Phan wrote:
> > Hi Jonathan,
> > 
> > Thanks for the feedback. Regarding the linked page <
> > https://gcc.gnu.org/wiki/InstallingGCC
> > >
> > if you're referring to the part that tells you to use your distro's package
> > manager, yes that's indeed the simplest way to install GCC, but from pre-
> > built
> > binaries, not building directly from sources. But if you're referring to
> > the
> > example at the bottom of the page,
> 
> Yes, that's what I mean.
> 
> > this script does exactly that, but the build
> > process is automated based on the build configuration file.
> 
> Yes, it does that, but takes 400 lines of shell script to do so.
> 
> If you want "relatively easy ways to build GCC painlessly" then you
> can do it with nine lines of shell commands.
> 
> Or in about 80, for any non-prehistoric version, with no config file
> needed (just a single option to the script, the release number or
> snapshot name to build):
> https://gist.github.com/jwakely/95b3a790157f55d75e18f577e12b50d7#file-build_gcc_versions-sh
> 
> 
> My point is just that there are already easier ways to do it, not only
> by using spack or the OpenCoarrays build script.
> 
> N.B. your 
> https://github.com/wyphan/gccprefab/blob/main/gcc12.cfg
> 
> config file says "Latest GCC 12 Release" but actually builds from the
> tip of the branch, not a release.
> 

Re: GSoC Blog Post 0 - GCCprefab build system

[Jonathan Wakely]

On Mon, 13 Jun 2022, 18:12 Wileam Yonatan Phan, <wileamyp@outlook.com>
wrote:

>
>
> Yes, you are correct that the script currently uses the tip of the
> `releases/gcc-[version]` branches. Is that not the same as using the
> published
> tarballs on the FTP server?



No.

I _do_ notice that these branches only get pushed
> whenever a minor version or a patchlevel is released.
>

For the gcc-12 branch, it currently changes several times a week:
https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/releases/gcc-12

Releases only happen every few months, and even snapshots from the branch
happen only once a week.

Your config file uses the tip of the branch, which is not the "latest GCC
12 release" as it claims. It's something different depending when you
happen to run the script, not a release.

The latest GCC 12 release would be the releases/gcc-12.1.0 tag in Git.

>

Re: GSoC Blog Post 0 - GCCprefab build system

[Damian Rouson]

On Mon, Jun 13, 2022 at 8:27 AM Jonathan Wakely via Fortran <
fortran@gcc.gnu.org> wrote:

>
> Yes, it does that, but takes 400 lines of shell script to do so.
>
> If you want "relatively easy ways to build GCC painlessly" then you
> can do it with nine lines of shell commands.
>
> Or in about 80, for any non-prehistoric version, with no config file
> needed (just a single option to the script, the release number or
> snapshot name to build):
>
> https://gist.github.com/jwakely/95b3a790157f55d75e18f577e12b50d7#file-build_gcc_versions-sh


Do the above 9 lines or 80 lines include the entire prerequisite software
stack or just the ones that the download_prerequisites script downloads?
If I recall correctly, building gfortran also requires flex and building
flex requires bison and building bison requires m4 and the
download_prerequisites script didn't download any of those the last time I
checked.

I realize that building gfortran from source comes very easily to GCC
developers, especially if they do it regularly.  I've encountered a lot of
people who found it challenging, including myself initially, which was the
reason for incorporating the capability into the OpenCoarrays installer.

Damian

Re: GSoC Blog Post 0 - GCCprefab build system

[Jonathan Wakely]

On Tue, 14 Jun 2022 at 01:37, Damian Rouson wrote:
>
>
>
> On Mon, Jun 13, 2022 at 8:27 AM Jonathan Wakely via Fortran <fortran@gcc.gnu.org> wrote:
>>
>>
>> Yes, it does that, but takes 400 lines of shell script to do so.
>>
>> If you want "relatively easy ways to build GCC painlessly" then you
>> can do it with nine lines of shell commands.
>>
>> Or in about 80, for any non-prehistoric version, with no config file
>> needed (just a single option to the script, the release number or
>> snapshot name to build):
>> https://gist.github.com/jwakely/95b3a790157f55d75e18f577e12b50d7#file-build_gcc_versions-sh
>
>
> Do the above 9 lines or 80 lines include the entire prerequisite software stack or just the ones that the download_prerequisites script downloads?  If I recall correctly, building gfortran also requires flex and building flex requires bison and building bison requires m4 and the download_prerequisites script didn't download any of those the last time I checked.

It doesn't include them, but they are standard system packages that
everybody can install without downloading the sources and building
them from scratch. There are no run-time dependencies on flex and
bison, just build-time. GMP, MPFR and MPC regularly cause problems for
people doing it the wrong way and ending up with run-time dependencies
on shared libs in non-standard locations (as described in the
InstallingGCC wiki page).

You still need to have the other prerequisites listed at
https://gcc.gnu.org/install/prerequisites.html

Re: GSoC Blog Post 0 - GCCprefab build system

[Damian Rouson]

On Mon, Jun 13, 2022 at 8:27 AM Jonathan Wakely via Fortran <
fortran@gcc.gnu.org> wrote:

>
> It doesn't include them, but they are standard system packages that
> everybody can install without downloading the sources and building
> them from scratch.


unless the person is on a system on which they are not preinstalled and a
system for which the person doesn’t have the sudo privileges that package
managers often require.  What I’m describing is the norm for a lot of
government employees and even many people at private corporations with
strict security policies.  For what it’s worth, I’ve been assisting someone
who contacted me with this very issue over the past few days.  Building the
entire stack from source is the least painful option for this person.

You still need to have the other prerequisites listed at
> https://gcc.gnu.org/install/prerequisites.html


That is a long and daunting list for a newcomer.  I’ve listened to gfortran
developers describe building gfortran as “easy” for more than a decade
now.  Simply
saying it’s easy doesn’t make it so. I don’t know that I’ve ever met
someone who described the process as easy unless that person was a gfortran
developer.

Damian

<https://gcc.gnu.org/install/prerequisites.html>
>

Re: GSoC Blog Post 0 - GCCprefab build system

[Wileam Yonatan Phan]

Hi Jonathan,

I just pushed a commit to update the build script and config files to use the
release tags instead of the tip of the branch. Thanks again for pointing this
out!

Thanks,
Wil

---

Hi Damian,

That's indeed a tricky issue to implement with the build script if the user
doesn't have sudo rights to install software on the system using the package
manager. Maybe I can make the script download the tarballs and build them from
sources.

Full disclosure: the script currently assumes all prerequisites have been
successfully installed, but based on the discussion here, I can add several
lines to check for their existence using `command -v`.

I'll start working on this later tonight.

Thanks,
Wil

On Tue, 2022-06-14 at 04:16 -0700, Damian Rouson wrote:
> On Mon, Jun 13, 2022 at 8:27 AM Jonathan Wakely via Fortran <
> fortran@gcc.gnu.org> wrote:
> > It doesn't include them, but they are standard system packages that
> > everybody can install without downloading the sources and building
> > them from scratch. 
> 
> unless the person is on a system on which they are not preinstalled and a
> system for which the person doesn’t have the sudo privileges that package
> managers often require.  What I’m describing is the norm for a lot of
> government employees and even many people at private corporations with strict
> security policies.  For what it’s worth, I’ve been assisting someone who
> contacted me with this very issue over the past few days.  Building the
> entire stack from source is the least painful option for this person. 
> 
> > You still need to have the other prerequisites listed at
> > https://gcc.gnu.org/install/prerequisites.html
> 
> That is a long and daunting list for a newcomer.  I’ve listened to gfortran
> developers describe building gfortran as “easy” for more than a decade
> now.  Simply saying it’s easy doesn’t make it so. I don’t know that I’ve ever
> met someone who described the process as easy unless that person was a
> gfortran developer. 
> 
> Damian
> 
> 
> 

Re: GSoC Blog Post 0 - GCCprefab build system

[Bernhard Reutner-Fischer]

On 13 June 2022 17:26:59 CEST, Jonathan Wakely via Fortran <fortran@gcc.gnu.org> wrote:

>https://gist.github.com/jwakely/95b3a790157f55d75e18f577e12b50d7#file-build_gcc_versions-sh

s/[[/[/
s/==/=/

The former are deprecated or obsolescent notations of test(1) syntax, fwiw.

PS: we should rm https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=contrib/gcc_build
It was not updated since the switch to git so, apparently, nobody uses it anymore (sadly, seeing who authored it).

Re: GSoC Blog Post 0 - GCCprefab build system

[Jakub Jelinek]

On Fri, Jun 17, 2022 at 08:45:04PM +0200, Bernhard Reutner-Fischer via Gcc wrote:
> PS: we should rm https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=contrib/gcc_build

No.  gcc_build is used by maintainer-scripts/gcc_release, so by killing it
you'd make gcc unreleasable.

> It was not updated since the switch to git so, apparently, nobody uses it anymore (sadly, seeing who authored it).

	Jakub

Re: GSoC Blog Post 0 - GCCprefab build system

[Bernhard Reutner-Fischer]

On 17 June 2022 21:55:47 CEST, Jakub Jelinek <jakub@redhat.com> wrote:
>On Fri, Jun 17, 2022 at 08:45:04PM +0200, Bernhard Reutner-Fischer via Gcc wrote:
>> PS: we should rm https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=contrib/gcc_build
>
>No.  gcc_build is used by maintainer-scripts/gcc_release, so by killing it
>you'd make gcc unreleasable.

Doh, didn't grep in maintainer-scripts.
So, should the SVN parts be ripped out?

For non-maintainers, switch it to git instead?

>> It was not updated since the switch to git so, apparently, nobody uses it anymore (sadly, seeing who authored it).

The hunks talking about subversion do seem to be misleading nowadays, aren't they?

thanks,

Re: GSoC Blog Post 0 - GCCprefab build system

[Jonathan Wakely]

On Fri, 17 Jun 2022 at 19:45, Bernhard Reutner-Fischer wrote:
>
> On 13 June 2022 17:26:59 CEST, Jonathan Wakely via Fortran <fortran@gcc.gnu.org> wrote:
>
> >https://gist.github.com/jwakely/95b3a790157f55d75e18f577e12b50d7#file-build_gcc_versions-sh
>
> s/[[/[/
> s/==/=/
>
> The former are deprecated or obsolescent notations of test(1) syntax, fwiw.

The former are defined by bash. If I made your change, [ $ver = ?.?.?
] would never be true, because ?.?.? is not a GCC version.

With Bash [[ $ver == ?.?.? ]] doesn't do word splitting etc. and
matches the RHS as a glob patttern, which is exactly what I want it to
do.