Re: Updated Sourceware infrastructure plans

Re: Updated Sourceware infrastructure plans

[Thomas Koenig]

Am 18.04.24 um 01:27 schrieb Mark Wielaard:
> We also should make sure that all generated files (either in git or in
> the release/snapshot tar balls) can be reliably and reproducibly
> regenerated. This also helps the (pre-commit) CI buildbots. We already
> have the autoregen bots for gcc and binutils-gdb. And Christoph has
> been working on extending the scripts to regenerate more kinds of
> files.

I regenerate auto* files from time to time for libgfortran. Regenerating
them has always been very fragile (using --enable-maintainer-mode),
and difficult to get right.

If there is a better process available to do it the right way is
that is documented and easy to use, this will make work easier.

If not, it has the potential to stop the work I am planning to
contribute in a project that is about a month from starting
(and maybe stop the project altogether).

Can anybody point me towards the tools that will be the
gold standard in the future, and the reproducible way
of regenerating them?

Best regards

	Thomas

Re: Updated Sourceware infrastructure plans

[FX Coudert]

> I regenerate auto* files from time to time for libgfortran. Regenerating
> them has always been very fragile (using --enable-maintainer-mode),
> and difficult to get right.

I have never found them difficult to regenerate, but if you have only a non maintainer build, it is a pain to have to make a new maintainer build for a minor change.

Moreover, our m4 code is particularly painful to use and unreadable. I have been wondering for some time: should we switch to simpler Python scripts? It would also mean that we would have fewer files in the generated/ folder: right now, every time we add new combinations of types, we have a combinatorial explosion of files.

$ ls generated/sum_*
generated/sum_c10.c generated/sum_c17.c generated/sum_c8.c  generated/sum_i16.c generated/sum_i4.c  generated/sum_r10.c generated/sum_r17.c generated/sum_r8.c
generated/sum_c16.c generated/sum_c4.c  generated/sum_i1.c  generated/sum_i2.c  generated/sum_i8.c  generated/sum_r16.c generated/sum_r4.c

We could imagine having a single file for all sum intrinsics.

How do Fortran maintainers feel about that?

FX

Re: Updated Sourceware infrastructure plans

[Christophe Lyon]

Hi,

On Thu, 18 Apr 2024 at 10:15, FX Coudert <fxcoudert@gmail.com> wrote:
>
> > I regenerate auto* files from time to time for libgfortran. Regenerating
> > them has always been very fragile (using --enable-maintainer-mode),
> > and difficult to get right.
>
> I have never found them difficult to regenerate, but if you have only a non maintainer build, it is a pain to have to make a new maintainer build for a minor change.
>

FWIW, we have noticed lots of warnings from autoreconf in libgfortran.
I didn't try to investigate, since the regenerated files are identical
to what is currently in the repo.

For instance, you can download the "stdio" output from the
autoregen.py step in
https://builder.sourceware.org/buildbot/#/builders/269/builds/4373

Thanks,

Christophe


> Moreover, our m4 code is particularly painful to use and unreadable. I have been wondering for some time: should we switch to simpler Python scripts? It would also mean that we would have fewer files in the generated/ folder: right now, every time we add new combinations of types, we have a combinatorial explosion of files.
>
> $ ls generated/sum_*
> generated/sum_c10.c generated/sum_c17.c generated/sum_c8.c  generated/sum_i16.c generated/sum_i4.c  generated/sum_r10.c generated/sum_r17.c generated/sum_r8.c
> generated/sum_c16.c generated/sum_c4.c  generated/sum_i1.c  generated/sum_i2.c  generated/sum_i8.c  generated/sum_r16.c generated/sum_r4.c
>
> We could imagine having a single file for all sum intrinsics.
>
> How do Fortran maintainers feel about that?
>
> FX

Re: Updated Sourceware infrastructure plans

[Janne Blomqvist]

On Thu, Apr 18, 2024 at 11:15 AM FX Coudert <fxcoudert@gmail.com> wrote:
>
> > I regenerate auto* files from time to time for libgfortran. Regenerating
> > them has always been very fragile (using --enable-maintainer-mode),
> > and difficult to get right.
>
> I have never found them difficult to regenerate, but if you have only a non maintainer build, it is a pain to have to make a new maintainer build for a minor change.
>
> Moreover, our m4 code is particularly painful to use and unreadable. I have been wondering for some time: should we switch to simpler Python scripts? It would also mean that we would have fewer files in the generated/ folder: right now, every time we add new combinations of types, we have a combinatorial explosion of files.
>
> $ ls generated/sum_*
> generated/sum_c10.c generated/sum_c17.c generated/sum_c8.c  generated/sum_i16.c generated/sum_i4.c  generated/sum_r10.c generated/sum_r17.c generated/sum_r8.c
> generated/sum_c16.c generated/sum_c4.c  generated/sum_i1.c  generated/sum_i2.c  generated/sum_i8.c  generated/sum_r16.c generated/sum_r4.c
>
> We could imagine having a single file for all sum intrinsics.
>
> How do Fortran maintainers feel about that?

For the time being I'm not an active maintainer, so my opinion doesn't
per se have weight, but back when I was active I did think about this
issue. IMHO the best of my ideas was to convert these into C++
templates. What we're essentially doing with the M4 stuff and the
proposed in-house Python reimplementation is to make up for lack of
monomorphization in plain old C. Rather than doing some DIY templates,
switch the implementation language to something which has that feature
built-in, in this case C++.  No need to convert the entire libgfortran
to C++ if you don't want to, just those objects that are generated
from the M4 templates. Something like

template<typename T>
void matmul(T* a, T* b, T* c, ...)
{
   // actual matmul code here
}

extern "C" {
  // Instantiate template for every type and export the symbol
  void matmul_r4(gfc_array_r4* a, gfc_array_r4* b, gfc_array_r4* c, ...)
  {
    matmul(a, b, c, ...);
  }
  // And so on for other types
}


-- 
Janne Blomqvist

Generated files in libgfortran for Fortran intrinsic procedures (was: Updated Sourceware infrastructure plans)

[Tobias Burnus]

Hi Janne,

Janne Blomqvist wrote:
> back when I was active I did think about this
> issue. IMHO the best of my ideas was to convert these into C++
> templates.

I think this will work – but we have to be super careful:

With C++, there is the problem that we definitely do not want to add 
dependency on libstdc++ nor to use some features which require special 
hardware support (like exceptions [always bad], symbol aliases, ...). — 
On some systems, a full C++ support might be not available, like 
embedded systems (including some odd embedded OS) or offloading devices.

The libstdc++ dependency would be detected by linking as we currently 
do. For in-language features, we have to ensure the appropriate flags 
-fno-exceptions (and probably a few more). And it should be clear what 
language features to use.

If we do, I think that would surely be an option.

> What we're essentially doing with the M4 stuff and the
> proposed in-house Python reimplementation is to make up for lack of
> monomorphization in plain old C. Rather than doing some DIY templates,
> switch the implementation language to something which has that feature
> built-in, in this case C++.  No need to convert the entire libgfortran
> to C++ if you don't want to, just those objects that are generated
> from the M4 templates. Something like
>
> template<typename T>
> void matmul(T* a, T* b, T* c, ...)
> {
>     // actual matmul code here
> }
>
> extern "C" {
>    // Instantiate template for every type and export the symbol
>    void matmul_r4(gfc_array_r4* a, gfc_array_r4* b, gfc_array_r4* c, ...)
>    {
>      matmul(a, b, c, ...);
>    }
>    // And so on for other types
> }

Cheers,

Tobias

Re: Generated files in libgfortran for Fortran intrinsic procedures (was: Updated Sourceware infrastructure plans)

[Martin Uecker]

Am Donnerstag, dem 18.04.2024 um 14:01 +0200 schrieb Tobias Burnus:
> Hi Janne,
> 
> Janne Blomqvist wrote:
> > back when I was active I did think about this
> > issue. IMHO the best of my ideas was to convert these into C++
> > templates.

I haven't looked at libgfortran but I didn't find it problematic
at all to use C in similar numerical code and this helps
with portability. 

Either I use macros, which I keep short and then do not find
inferior to templates (having used C++ for years previously) or 
- if there is really a lot of code that needs to be specialized 
for a type - simply by using includes:

#define matmul_type double
#include "matmul_impl.c"

Martin


> 
> I think this will work – but we have to be super careful:
> 
> With C++, there is the problem that we definitely do not want to add 
> dependency on libstdc++ nor to use some features which require special 
> hardware support (like exceptions [always bad], symbol aliases, ...). — 
> On some systems, a full C++ support might be not available, like 
> embedded systems (including some odd embedded OS) or offloading devices.
> 
> The libstdc++ dependency would be detected by linking as we currently 
> do. For in-language features, we have to ensure the appropriate flags 
> -fno-exceptions (and probably a few more). And it should be clear what 
> language features to use.
> 
> If we do, I think that would surely be an option.
> 
> > What we're essentially doing with the M4 stuff and the
> > proposed in-house Python reimplementation is to make up for lack of
> > monomorphization in plain old C. Rather than doing some DIY templates,
> > switch the implementation language to something which has that feature
> > built-in, in this case C++.  No need to convert the entire libgfortran
> > to C++ if you don't want to, just those objects that are generated
> > from the M4 templates. Something like
> > 
> > template<typename T>
> > void matmul(T* a, T* b, T* c, ...)
> > {
> >     // actual matmul code here
> > }
> > 
> > extern "C" {
> >    // Instantiate template for every type and export the symbol
> >    void matmul_r4(gfc_array_r4* a, gfc_array_r4* b, gfc_array_r4* c, ...)
> >    {
> >      matmul(a, b, c, ...);
> >    }
> >    // And so on for other types
> > }
> 
> Cheers,
> 
> Tobias

Re: Updated Sourceware infrastructure plans

[Jonathan Wakely]

On Thu, 18 Apr 2024 at 07:06, Thomas Koenig via Gcc <gcc@gcc.gnu.org> wrote:
>
> Am 18.04.24 um 01:27 schrieb Mark Wielaard:
> > We also should make sure that all generated files (either in git or in
> > the release/snapshot tar balls) can be reliably and reproducibly
> > regenerated. This also helps the (pre-commit) CI buildbots. We already
> > have the autoregen bots for gcc and binutils-gdb. And Christoph has
> > been working on extending the scripts to regenerate more kinds of
> > files.
>
> I regenerate auto* files from time to time for libgfortran. Regenerating
> them has always been very fragile (using --enable-maintainer-mode),
> and difficult to get right.

I've been curious for ages why gfortran requires using maintainer mode
for that. Nobody else uses maintainer mode for GCC development, so it
causes friction when somebody has to do changes across the whole of
gcc including gfortran parts.

If it doesn't work with a simple autoreconf then that should be fixed.