RISC-V 64bit and 32bit binaries on the same system

RISC-V 64bit and 32bit binaries on the same system

[Andreas K. Hüttel]

[-- Attachment #1: Type: text/plain, Size: 2846 bytes --]

Hi all, 

as you probably know, the gcc multilib settings for riscv build and install 4 
ABI (rv64/lp64d, rv64/lp64, rv32/ilp32d, rv32/ilp32). Now that rv32 support is 
in glibc, I gave this a try (*) in qemu-user. 

gcc and glibc build and install fine. However, there's a fundamental problem 
with this configuration. ld.so does not filter by ELFCLASS (as it does, e.g., 
for x64-64 vs i686), and the order of directories in the dynamic linker config 
determines e.g. which libstdc++ a binary sees first. [#]

Which means, either rv32 or rv64 binaries will terminate with a "wrong 
ELFCLASS" fatal error.

Now, what to do?

* Introduce a special cache flag as for x86-64 FLAG_X8664_LIB64?

* Make ld.so ignore wrong-ELFCLASS binaries instead of producing an error? 
(Same result handled at different point...)

* Propose to change the gcc multilib profiles so we don't pretend that this 
works?

Cheers, 
Andreas


(*) I know that there is no real hardware that can run both rv64 and rv32 at 
the moment. However, it would be useful for me to bootstrap different single-
ABI environments.

[#]
(riscv-main chroot) farino /lib # ldconfig -p|grep c++
        libstdc++.so.6 (libc6,double-float) => /usr/lib/gcc/riscv64-unknown-
linux-gnu/10.2.0/lib64/lp64d/libstdc++.so.6
        libstdc++.so.6 (libc6,double-float) => /usr/lib/gcc/riscv64-unknown-
linux-gnu/10.2.0/lib32/ilp32d/libstdc++.so.6
        libstdc++.so.6 (libc6,soft-float) => /usr/lib/gcc/riscv64-unknown-
linux-gnu/10.2.0/lib64/lp64/libstdc++.so.6
        libstdc++.so.6 (libc6,soft-float) => /usr/lib/gcc/riscv64-unknown-
linux-gnu/10.2.0/lib32/ilp32/libstdc++.so.6
        libstdc++.so (libc6,double-float) => /usr/lib/gcc/riscv64-unknown-
linux-gnu/10.2.0/lib64/lp64d/libstdc++.so
        libstdc++.so (libc6,double-float) => /usr/lib/gcc/riscv64-unknown-
linux-gnu/10.2.0/lib32/ilp32d/libstdc++.so
        libstdc++.so (libc6,soft-float) => /usr/lib/gcc/riscv64-unknown-linux-
gnu/10.2.0/lib64/lp64/libstdc++.so
        libstdc++.so (libc6,soft-float) => /usr/lib/gcc/riscv64-unknown-linux-
gnu/10.2.0/lib32/ilp32/libstdc++.so

versus

farino ~ # ldconfig -p |grep c++
        libstdc++.so.6 (libc6,x86-64) => /usr/lib/gcc/x86_64-pc-linux-gnu/
9.3.0/libstdc++.so.6
        libstdc++.so.6 (libc6) => /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/32/
libstdc++.so.6
        libstdc++.so (libc6,x86-64) => /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/
libstdc++.so
        libstdc++.so (libc6) => /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/32/
libstdc++.so
        libnetcdf_c++.so.4 (libc6,x86-64) => /usr/lib64/libnetcdf_c++.so.4
        libnetcdf_c++.so (libc6,x86-64) => /usr/lib64/libnetcdf_c++.so


-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, qa, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: RISC-V 64bit and 32bit binaries on the same system

[Joseph Myers]

On Mon, 7 Sep 2020, Andreas K. Hüttel via Libc-alpha wrote:

> gcc and glibc build and install fine. However, there's a fundamental problem 
> with this configuration. ld.so does not filter by ELFCLASS (as it does, e.g., 
> for x64-64 vs i686), and the order of directories in the dynamic linker config 
> determines e.g. which libstdc++ a binary sees first. [#]

Whenever I asked about such configurations, the answer was always that the 
Linux kernel port did not support running RV32I processes on RV64I 
processors, although the architecture definition supported it as an 
optional feature.  I don't know if that Linux kernel support has since 
been implemented.

<https://sourceware.org/legacy-ml/libc-alpha/2018-01/msg00008.html> is 
where I described what you need to do in glibc to support that kind of 
configuration for an architecture.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: RISC-V 64bit and 32bit binaries on the same system

[Andreas K. Hüttel]

[-- Attachment #1: Type: text/plain, Size: 1299 bytes --]

> > gcc and glibc build and install fine. However, there's a fundamental
> > problem with this configuration. ld.so does not filter by ELFCLASS (as it
> > does, e.g., for x64-64 vs i686), and the order of directories in the
> > dynamic linker config determines e.g. which libstdc++ a binary sees
> > first. [#]
> 
> Whenever I asked about such configurations, the answer was always that the
> Linux kernel port did not support running RV32I processes on RV64I
> processors, although the architecture definition supported it as an
> optional feature.  I don't know if that Linux kernel support has since
> been implemented.

Not to my knowledge.

> <https://sourceware.org/legacy-ml/libc-alpha/2018-01/msg00008.html> is
> where I described what you need to do in glibc to support that kind of
> configuration for an architecture.

Yeah, that's the more knowledgeable answer compared to my newbie digging at 
what is needed. 

(And it looks like a comparatively large amount of work for a situation that 
is right now fairly hypothetical, unless you try silly things with qemu like 
me).

So maybe the multilib defaults of gcc should be changed?

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, qa, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: RISC-V 64bit and 32bit binaries on the same system

[Palmer Dabbelt]

On Mon, 07 Sep 2020 10:05:22 PDT (-0700), libc-alpha@sourceware.org wrote:
>> > gcc and glibc build and install fine. However, there's a fundamental
>> > problem with this configuration. ld.so does not filter by ELFCLASS (as it
>> > does, e.g., for x64-64 vs i686), and the order of directories in the
>> > dynamic linker config determines e.g. which libstdc++ a binary sees
>> > first. [#]
>> 
>> Whenever I asked about such configurations, the answer was always that the
>> Linux kernel port did not support running RV32I processes on RV64I
>> processors, although the architecture definition supported it as an
>> optional feature.  I don't know if that Linux kernel support has since
>> been implemented.
>
> Not to my knowledge.

We don't have any plans to do it soon, as there's no hardware out there and
nobody is clamoring for it.  The last glibc post I see on this is from January
2018, and since then we have sufficient ISA support to implement this sort of
thing (the UXL field) but I know of any hardware that implements it or anyone
who's particularly interested in doing so.

>> <https://sourceware.org/legacy-ml/libc-alpha/2018-01/msg00008.html> is
>> where I described what you need to do in glibc to support that kind of
>> configuration for an architecture.
>
> Yeah, that's the more knowledgeable answer compared to my newbie digging at 
> what is needed. 
>
> (And it looks like a comparatively large amount of work for a situation that 
> is right now fairly hypothetical, unless you try silly things with qemu like 
> me).

It'd be great to bootstrap all the multilib stuff, but it's just a matter of
priorities.  It's a fairly large amount of work to put this all together, but
it shouldn't be all that hard to add writable XLEN to QEMU which would allow
Linux to spin up rv32 processes.  There'd be a big pile of work left at that
point, though...

> So maybe the multilib defaults of gcc should be changed?

Either way, that seems reasonable to me.  If we ever implement it we can always
add the multilibs back, but it doesn't seem reasonable to default to something
broken.  I bet you guys are the first to take a shot at the Linux multilib
stuff, so I anticipate a bunch of stuff will be broken.

Do you want to send a GCC patch, or do you want me to?

> -- 
> Andreas K. Hüttel
> dilfridge@gentoo.org
> Gentoo Linux developer 
> (council, qa, toolchain, base-system, perl, libreoffice)

Re: RISC-V 64bit and 32bit binaries on the same system

[Andreas K. Hüttel]

[-- Attachment #1: Type: text/plain, Size: 2128 bytes --]

> > (And it looks like a comparatively large amount of work for a situation
> > that is right now fairly hypothetical, unless you try silly things with
> > qemu like me).
> 
> It'd be great to bootstrap all the multilib stuff, but it's just a matter of
> priorities.  It's a fairly large amount of work to put this all together,
> but it shouldn't be all that hard to add writable XLEN to QEMU which would
> allow Linux to spin up rv32 processes.  There'd be a big pile of work left
> at that point, though...

Well it kinda-works half in a Gentoo qemu-user chroot right now, which is of 
course not the most useful production environment. :)

What looks good in my experiments:
* building and installing gcc with all 4 ABI and multilib paths
* building and installing glibc with all 4 ABI and multilib paths

Then, the Gentoo multilib system still works *if* I order the ld.so search 
paths right (all the executables are lp64d, so that comes first, and the 
ilp32[d] libs are ignored). All four ld.so variants are installed.

In principle, *if* ld.so were to ignore wrong elfclass (with all changes that 
this requires), a mixed multilib Gentoo install could work more or less out of 
the box. We already use /usr/lib64/lp64[d], and I hook into the same path 
adjustment mechanisms and multi-abi builds we have made for x86-64/i686 and 
others.

> > So maybe the multilib defaults of gcc should be changed?
> 
> Either way, that seems reasonable to me.  If we ever implement it we can
> always add the multilibs back, but it doesn't seem reasonable to default to
> something broken.  I bet you guys are the first to take a shot at the Linux
> multilib stuff, so I anticipate a bunch of stuff will be broken.
> 
> Do you want to send a GCC patch, or do you want me to?

I agree; we can always slowly prepare the support anyway so it's ready once 
there is hardware.

It's probably better if the gcc patch comes from you ("more official").

Best, 
Andreas

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, qa, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 963 bytes --]