Clarification on newlib version for building AMDGCN offloading backend

Clarification on newlib version for building AMDGCN offloading backend

[Wileam Yonatan Phan]

Hi,

I'm working on adding a build recipe for GCC with AMDGCN offloading backend in Spack. Can anyone clarify the following sentence listed on the wiki?

>The Newlib version needs to be contemporaeous with GCC, at least until the ABI is finalized.
<https://gcc.gnu.org/wiki/Offloading#How_to_build_an_offloading-enabled_GCC>

What are the correct contemporaneous versions for each version of GCC >= 10?

Thanks,
Wil

Re: Clarification on newlib version for building AMDGCN offloading backend

[Andrew Stubbs]

On 06/03/2023 19:23, Wileam Yonatan Phan via Gcc wrote:
> Hi,
> 
> I'm working on adding a build recipe for GCC with AMDGCN offloading backend in Spack. Can anyone clarify the following sentence listed on the wiki?
> 
>> The Newlib version needs to be contemporaeous with GCC, at least until the ABI is finalized.
> <https://gcc.gnu.org/wiki/Offloading#How_to_build_an_offloading-enabled_GCC>
> 
> What are the correct contemporaneous versions for each version of GCC >= 10?

Just match the dates and you'll probably be fine. We've mostly 
synchronised the ABI changes across the GCC mainline and the development 
branch precisely because the Newlib dependency is shared.

Right now the required version of Newlib is 4.3.0.20230120. Prior to the 
ABI change a month or so ago you would have to use a Newlib snapshot.

I wouldn't recommend spending very much of your valuable time on 
enabling old versions of these toolchains.

Andrew

Re: Clarification on newlib version for building AMDGCN offloading backend

[Wileam Yonatan Phan]

Hi Andrew,

Thanks! I've created the first draft as a GitHub PR here:
<https://github.com/spack/spack/pull/35919>

I think it still needs some work on the part where it builds LLVM utilities with CMake.
As much as I want supporting only the newest GCC version, Spack currently supports building all stable versions of GCC from the tarballs, as well as latest develop branch, so I've enabled all versions >= 10.

Thanks,
Wil

From: Andrew Stubbs <ams@codesourcery.com>
Sent: Tuesday, March 7, 2023 10:38
To: Wileam Yonatan Phan <wileamyp@outlook.com>; gcc@gcc.gnu.org <gcc@gcc.gnu.org>
Subject: Re: Clarification on newlib version for building AMDGCN offloading backend 
 
On 06/03/2023 19:23, Wileam Yonatan Phan via Gcc wrote:
> Hi,
> 
> I'm working on adding a build recipe for GCC with AMDGCN offloading backend in Spack. Can anyone clarify the following sentence listed on the wiki?
> 
>> The Newlib version needs to be contemporaeous with GCC, at least until the ABI is finalized.
> <https://gcc.gnu.org/wiki/Offloading#How_to_build_an_offloading-enabled_GCC>
> 
> What are the correct contemporaneous versions for each version of GCC >= 10?

Just match the dates and you'll probably be fine. We've mostly 
synchronised the ABI changes across the GCC mainline and the development 
branch precisely because the Newlib dependency is shared.

Right now the required version of Newlib is 4.3.0.20230120. Prior to the 
ABI change a month or so ago you would have to use a Newlib snapshot.

I wouldn't recommend spending very much of your valuable time on 
enabling old versions of these toolchains.

Andrew

Re: Clarification on newlib version for building AMDGCN offloading backend

[Wileam Yonatan Phan]

Hi Andrew,

I just built GCC 12.2.0 with AMDGCN offloading successfully with Spack!
However, when I tried to test it with an OpenACC test code that I have, I encountered the following error message:

wyp@basecamp:~/work/testcodes/f90-acc-ddot$ gfortran -fopenacc -foffload=amdgcn-unknown-amdhsa="-march=gfx900" ddot.f90 
as: unrecognized option '-triple=amdgcn--amdhsa'
mkoffload: fatal error: x86_64-pc-linux-gnu-accel-amdgcn-unknown-amdhsa-gcc returned 1 exit status
compilation terminated.
lto-wrapper: fatal error: /home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-12.2.0/gcc-12.2.0-w7lclfarefmge3uegn2a5vw37bnwhwto/libexec/gcc/x86_64-pc-linux-gnu/12.2.0//accel/amdgcn-unknown-amdhsa/mkoffload returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

For reference, the target options that are recognized by llvm-mc are the following:

yp@basecamp:~/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-12.2.0/llvm-13.0.1-6zvbqlbev3wiqihvz2god7pzcptnoxu3/bin$ llvm-mc -version
LLVM (http://llvm.org/):
  LLVM version 13.0.1
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver2

  Registered Targets:
    amdgcn - AMD GCN GPUs
    r600   - AMD GPUs HD2XXX-HD6XXX
    x86    - 32-bit X86: Pentium-Pro and above
    x86-64 - 64-bit X86: EM64T and AMD64

Please advise,
Wil

From: Wileam Yonatan Phan <wileamyp@outlook.com>
Sent: Tuesday, March 7, 2023 13:42
To: Andrew Stubbs <ams@codesourcery.com>; gcc@gcc.gnu.org <gcc@gcc.gnu.org>
Subject: Re: Clarification on newlib version for building AMDGCN offloading backend 
 
Hi Andrew,

Thanks! I've created the first draft as a GitHub PR here:
<https://github.com/spack/spack/pull/35919>

I think it still needs some work on the part where it builds LLVM utilities with CMake.
As much as I want supporting only the newest GCC version, Spack currently supports building all stable versions of GCC from the tarballs, as well as latest develop branch, so I've enabled all versions >= 10.

Thanks,
Wil

From: Andrew Stubbs <ams@codesourcery.com>
Sent: Tuesday, March 7, 2023 10:38
To: Wileam Yonatan Phan <wileamyp@outlook.com>; gcc@gcc.gnu.org <gcc@gcc.gnu.org>
Subject: Re: Clarification on newlib version for building AMDGCN offloading backend 
 
On 06/03/2023 19:23, Wileam Yonatan Phan via Gcc wrote:
> Hi,
> 
> I'm working on adding a build recipe for GCC with AMDGCN offloading backend in Spack. Can anyone clarify the following sentence listed on the wiki?
> 
>> The Newlib version needs to be contemporaeous with GCC, at least until the ABI is finalized.
> <https://gcc.gnu.org/wiki/Offloading#How_to_build_an_offloading-enabled_GCC>
> 
> What are the correct contemporaneous versions for each version of GCC >= 10?

Just match the dates and you'll probably be fine. We've mostly 
synchronised the ABI changes across the GCC mainline and the development 
branch precisely because the Newlib dependency is shared.

Right now the required version of Newlib is 4.3.0.20230120. Prior to the 
ABI change a month or so ago you would have to use a Newlib snapshot.

I wouldn't recommend spending very much of your valuable time on 
enabling old versions of these toolchains.

Andrew

Re: Clarification on newlib version for building AMDGCN offloading backend

[Andrew Stubbs]

On 29/03/2023 19:18, Wileam Yonatan Phan wrote:
> Hi Andrew,
> 
> I just built GCC 12.2.0 with AMDGCN offloading successfully with Spack!
> However, when I tried to test it with an OpenACC test code that I have, I encountered the following error message:
> 
> wyp@basecamp:~/work/testcodes/f90-acc-ddot$ gfortran -fopenacc -foffload=amdgcn-unknown-amdhsa="-march=gfx900" ddot.f90
> as: unrecognized option '-triple=amdgcn--amdhsa'
> mkoffload: fatal error: x86_64-pc-linux-gnu-accel-amdgcn-unknown-amdhsa-gcc returned 1 exit status
> compilation terminated.
> lto-wrapper: fatal error: /home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-12.2.0/gcc-12.2.0-w7lclfarefmge3uegn2a5vw37bnwhwto/libexec/gcc/x86_64-pc-linux-gnu/12.2.0//accel/amdgcn-unknown-amdhsa/mkoffload returned 1 exit status
> compilation terminated.
> /usr/bin/ld: error: lto-wrapper failed
> collect2: error: ld returned 1 exit status

My guess is that it's trying to use the wrong assembler. Usually this 
means there is a problem with your installation procedure and/or your 
PATH. I think you should be able to investigate further using -v and/or 
strace. The correct one should be named 
$DESTDIR/usr/local/amdgcn-amdhsa/bin/as, but this will be different if 
you configured GCC with a custom --prefix location. If you have 
relocated the toolchain since installation then the toolchain will 
attempt to locate libraries and tools relative to the gcc binary. If it 
does not find them there then it looks in the "usual places", and those 
usually contain an "as" suitable only for the host system.

If you find an error on the Wiki instructions please let me know and I 
will correct them.

Andrew

Re: Clarification on newlib version for building AMDGCN offloading backend

[Wileam Yonatan Phan]

Hi Andrew,

Just wanna follow up on the progress of this endeavor of enabling GCC with AMDGCN offloading in Spack. So far I think I've got everything working, except for the part where libgomp is pulled from the wrong place at runtime, because Spack prefers using RPATH to LD_LIBRARY_PATH. As outlined in the Spack PR comments, the proposed fix is modifying the *link_gomp field inside <install-prefix>/lib/gcc/amdgcn-amdhsa/<gcc-version>/specs to add the rpath flags for libgomp. But I'm honestly unsure if this should be done at configure time, build time, or install time.

The Spack PR can be accessed here: <https://github.com/spack/spack/pull/35919>

Please advise,
Wil

From: Andrew Stubbs <ams@codesourcery.com>
Sent: Thursday, March 30, 2023 04:45
To: Wileam Yonatan Phan <wileamyp@outlook.com>; gcc@gcc.gnu.org <gcc@gcc.gnu.org>
Subject: Re: Clarification on newlib version for building AMDGCN offloading backend 
 
On 29/03/2023 19:18, Wileam Yonatan Phan wrote:
> Hi Andrew,
> 
> I just built GCC 12.2.0 with AMDGCN offloading successfully with Spack!
> However, when I tried to test it with an OpenACC test code that I have, I encountered the following error message:
> 
> wyp@basecamp:~/work/testcodes/f90-acc-ddot$ gfortran -fopenacc -foffload=amdgcn-unknown-amdhsa="-march=gfx900" ddot.f90
> as: unrecognized option '-triple=amdgcn--amdhsa'
> mkoffload: fatal error: x86_64-pc-linux-gnu-accel-amdgcn-unknown-amdhsa-gcc returned 1 exit status
> compilation terminated.
> lto-wrapper: fatal error: /home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-12.2.0/gcc-12.2.0-w7lclfarefmge3uegn2a5vw37bnwhwto/libexec/gcc/x86_64-pc-linux-gnu/12.2.0//accel/amdgcn-unknown-amdhsa/mkoffload returned 1 exit status
> compilation terminated.
> /usr/bin/ld: error: lto-wrapper failed
> collect2: error: ld returned 1 exit status

My guess is that it's trying to use the wrong assembler. Usually this 
means there is a problem with your installation procedure and/or your 
PATH. I think you should be able to investigate further using -v and/or 
strace. The correct one should be named 
$DESTDIR/usr/local/amdgcn-amdhsa/bin/as, but this will be different if 
you configured GCC with a custom --prefix location. If you have 
relocated the toolchain since installation then the toolchain will 
attempt to locate libraries and tools relative to the gcc binary. If it 
does not find them there then it looks in the "usual places", and those 
usually contain an "as" suitable only for the host system.

If you find an error on the Wiki instructions please let me know and I 
will correct them.

Andrew

RE: Clarification on newlib version for building AMDGCN offloading backend

[Stubbs, Andrew]

Hi Wil,

Our toolchains installations are relocatable, and therefore can't have an rpath hardcoded into them. Instead, we provide instructions in the user manual how to set LD_LIBRARY_PATH to pick up the correct libraries.

If you're creating OS packages then you could probably add the lib64 directory to the ldcache so it Just Works without any explicit rpath.

If you really want to modify the specs, you can do this by patching the gcc/config/i386/linux-common.h sourcefile to add it to the LINK_SPEC there, or you can build and install the compiler, run "gcc -dumpspecs", and write the edited text to a file named "specs" in the correct install directory. You'll want it in the x86_64 compiler directory, not the amdgcn compiler though, as the dynamic libraries are the host-side implementation, not the GPU side.

There might be a configure option too, but I've never investigated that. If there isn't and you would like to add one, then I expect upstream GCC would be happy to accept a patch.

Andrew

> -----Original Message-----
> From: Wileam Yonatan Phan <wileamyp@outlook.com>
> Sent: Friday, June 30, 2023 10:06 PM
> To: Andrew Stubbs <ams@codesourcery.com>; gcc@gcc.gnu.org
> Subject: Re: Clarification on newlib version for building AMDGCN offloading
> backend
>
> Hi Andrew,
>
> Just wanna follow up on the progress of this endeavor of enabling GCC with
> AMDGCN offloading in Spack. So far I think I've got everything working, except
> for the part where libgomp is pulled from the wrong place at runtime, because
> Spack prefers using RPATH to LD_LIBRARY_PATH. As outlined in the Spack PR
> comments, the proposed fix is modifying the *link_gomp field inside <install-
> prefix>/lib/gcc/amdgcn-amdhsa/<gcc-version>/specs to add the rpath flags for
> libgomp. But I'm honestly unsure if this should be done at configure time, build
> time, or install time.
>
> The Spack PR can be accessed here:
> <https://github.c/
> om%2Fspack%2Fspack%2Fpull%2F35919&data=05%7C01%7Candrew.stubbs%4
> 0siemens.com%7C673557f0a6c04a52c1c608db79ade602%7C38ae3bcd95794f
> d4addab42e1495d55a%7C1%7C0%7C638237560023809014%7CUnknown%7C
> TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
> XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UrKryjKfE6uH7tJR4jNbSfK4AfnTR
> %2FeJg3tiKgR8LeQ%3D&reserved=0>
>
> Please advise,
> Wil
>
> From: Andrew Stubbs <ams@codesourcery.com>
> Sent: Thursday, March 30, 2023 04:45
> To: Wileam Yonatan Phan <wileamyp@outlook.com>; gcc@gcc.gnu.org
> <gcc@gcc.gnu.org>
> Subject: Re: Clarification on newlib version for building AMDGCN offloading
> backend
>
> On 29/03/2023 19:18, Wileam Yonatan Phan wrote:
> > Hi Andrew,
> >
> > I just built GCC 12.2.0 with AMDGCN offloading successfully with Spack!
> > However, when I tried to test it with an OpenACC test code that I have, I
> encountered the following error message:
> >
> > wyp@basecamp:~/work/testcodes/f90-acc-ddot$ gfortran -fopenacc
> > -foffload=amdgcn-unknown-amdhsa="-march=gfx900" ddot.f90
> > as: unrecognized option '-triple=amdgcn--amdhsa'
> > mkoffload: fatal error:
> > x86_64-pc-linux-gnu-accel-amdgcn-unknown-amdhsa-gcc returned 1 exit
> status compilation terminated.
> > lto-wrapper: fatal error:
> > /home/wyp/work/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-12.2.0/gcc-
> 12.2.0-w7lclfarefmge3uegn2a5vw37bnwhwto/libexec/gcc/x86_64-pc-linux-
> gnu/12.2.0//accel/amdgcn-unknown-amdhsa/mkoffload returned 1 exit status
> compilation terminated.
> > /usr/bin/ld: error: lto-wrapper failed
> > collect2: error: ld returned 1 exit status
>
> My guess is that it's trying to use the wrong assembler. Usually this means
> there is a problem with your installation procedure and/or your PATH. I think
> you should be able to investigate further using -v and/or strace. The correct one
> should be named $DESTDIR/usr/local/amdgcn-amdhsa/bin/as, but this will be
> different if you configured GCC with a custom --prefix location. If you have
> relocated the toolchain since installation then the toolchain will attempt to
> locate libraries and tools relative to the gcc binary. If it does not find them there
> then it looks in the "usual places", and those usually contain an "as" suitable
> only for the host system.
>
> If you find an error on the Wiki instructions please let me know and I will
> correct them.
>
> Andrew