[RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Palmer Dabbelt]

Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
---
So this is very much an RFC.  I had written a commit message as

    This will hopefully be a widely used configuration in the future, but
    it's not all that common right now.  That said, let's be proactive and
    at start testing it.

but I even that's kind of a strong statement here so I'm just stashing
it off to the side to be sure.

A few of us were talking about the GCC meeting last week, but I figured
I'd send a glibc patch to at least start some discussion on the lists as
something similar had come up in the TLSDESC thread and a while ago when
we talked about glibc-hwcaps.

I'm not really sure what the right thing to do here is: we don't have
broadly available hardware that supports these new extensions, but
they're pretty much table stakes for hardware that's competitive with
arm64/x86.  Without B and V we're going to end up piling up a ton of
IFUNC-based routines to try and work around the base ISA, but those
extensions are so fundamental to good codegen that it doesn't feel like
we're going to get where we want with just IFUNCs.

Unfortunately I don't think we can drop support for the other base ISAs:
the distros appear to be targeting rv64gc and that's the only common
base for most hardware that's out there.  I guess we could drop support
for the rv*ima-based base ISAs, but I'm not sure that buys us much:
we've got a lot of embedded users so we won't be able to drop the GCC
support for soft float, and dropping the glibc support puts us in a
clunky spot for testing.

So we're probably stuck in a bit of a performance hole for a while.
That said, we could at least put a stake in the ground and start testing
some target with some newer extensions.  That way distros that want to
try a larger base than rv64gc have something that's getting tested and
is thus likely to work.  For GCC we ended up adding some test targets
with pretty much all the non-embedded, but at least B and V seem like
the bare minimum.

I'm not sure if all that is worth the extra build/test resource burden
for everyone, though, particularly given the lack of hardware.
---
 scripts/build-many-glibcs.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
index ecc743e672..21da46dc4a 100755
--- a/scripts/build-many-glibcs.py
+++ b/scripts/build-many-glibcs.py
@@ -411,7 +411,9 @@ class Context(object):
                         os_name='linux-gnu',
                         variant='rv64imafdc-lp64d',
                         gcc_cfg=['--with-arch=rv64imafdc', '--with-abi=lp64d',
-                                 '--disable-multilib'])
+                                 '--disable-multilib']),
+                        extra_glibcs=[{'variant': 'rv64gcbv-rv64gc-lp64d',
+                                       'ccopts': '-march=rv64gcv_zba_zbb_zbs'}]
         self.add_config(arch='s390x',
                         os_name='linux-gnu',
                         glibcs=[{},
-- 
2.44.0

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[enh]

On Mon, Apr 15, 2024 at 12:27 PM Palmer Dabbelt <palmer@rivosinc.com> wrote:
>
> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
> ---
> So this is very much an RFC.  I had written a commit message as
>
>     This will hopefully be a widely used configuration in the future, but
>     it's not all that common right now.  That said, let's be proactive and
>     at start testing it.
>
> but I even that's kind of a strong statement here so I'm just stashing
> it off to the side to be sure.
>
> A few of us were talking about the GCC meeting last week, but I figured
> I'd send a glibc patch to at least start some discussion on the lists as
> something similar had come up in the TLSDESC thread and a while ago when
> we talked about glibc-hwcaps.
>
> I'm not really sure what the right thing to do here is: we don't have
> broadly available hardware that supports these new extensions, but
> they're pretty much table stakes for hardware that's competitive with
> arm64/x86.  Without B and V we're going to end up piling up a ton of
> IFUNC-based routines to try and work around the base ISA, but those
> extensions are so fundamental to good codegen that it doesn't feel like
> we're going to get where we want with just IFUNCs.
>
> Unfortunately I don't think we can drop support for the other base ISAs:
> the distros appear to be targeting rv64gc and that's the only common
> base for most hardware that's out there.  I guess we could drop support
> for the rv*ima-based base ISAs, but I'm not sure that buys us much:
> we've got a lot of embedded users so we won't be able to drop the GCC
> support for soft float, and dropping the glibc support puts us in a
> clunky spot for testing.
>
> So we're probably stuck in a bit of a performance hole for a while.
> That said, we could at least put a stake in the ground and start testing
> some target with some newer extensions.  That way distros that want to
> try a larger base than rv64gc have something that's getting tested and
> is thus likely to work.  For GCC we ended up adding some test targets
> with pretty much all the non-embedded, but at least B and V seem like
> the bare minimum.
>
> I'm not sure if all that is worth the extra build/test resource burden
> for everyone, though, particularly given the lack of hardware.

for Android -- which is in a similar situation -- having the build
system enable these extensions been pretty useful for finding
toolchain and qemu bugs, which seems useful in its own right, if only
for CI purposes...

> ---
>  scripts/build-many-glibcs.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
> index ecc743e672..21da46dc4a 100755
> --- a/scripts/build-many-glibcs.py
> +++ b/scripts/build-many-glibcs.py
> @@ -411,7 +411,9 @@ class Context(object):
>                          os_name='linux-gnu',
>                          variant='rv64imafdc-lp64d',
>                          gcc_cfg=['--with-arch=rv64imafdc', '--with-abi=lp64d',
> -                                 '--disable-multilib'])
> +                                 '--disable-multilib']),
> +                        extra_glibcs=[{'variant': 'rv64gcbv-rv64gc-lp64d',
> +                                       'ccopts': '-march=rv64gcv_zba_zbb_zbs'}]
>          self.add_config(arch='s390x',
>                          os_name='linux-gnu',
>                          glibcs=[{},
> --
> 2.44.0
>

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Palmer Dabbelt]

On Mon, 15 Apr 2024 13:05:53 PDT (-0700), enh@google.com wrote:
> On Mon, Apr 15, 2024 at 12:27 PM Palmer Dabbelt <palmer@rivosinc.com> wrote:
>>
>> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
>> ---
>> So this is very much an RFC.  I had written a commit message as
>>
>>     This will hopefully be a widely used configuration in the future, but
>>     it's not all that common right now.  That said, let's be proactive and
>>     at start testing it.
>>
>> but I even that's kind of a strong statement here so I'm just stashing
>> it off to the side to be sure.
>>
>> A few of us were talking about the GCC meeting last week, but I figured
>> I'd send a glibc patch to at least start some discussion on the lists as
>> something similar had come up in the TLSDESC thread and a while ago when
>> we talked about glibc-hwcaps.
>>
>> I'm not really sure what the right thing to do here is: we don't have
>> broadly available hardware that supports these new extensions, but
>> they're pretty much table stakes for hardware that's competitive with
>> arm64/x86.  Without B and V we're going to end up piling up a ton of
>> IFUNC-based routines to try and work around the base ISA, but those
>> extensions are so fundamental to good codegen that it doesn't feel like
>> we're going to get where we want with just IFUNCs.
>>
>> Unfortunately I don't think we can drop support for the other base ISAs:
>> the distros appear to be targeting rv64gc and that's the only common
>> base for most hardware that's out there.  I guess we could drop support
>> for the rv*ima-based base ISAs, but I'm not sure that buys us much:
>> we've got a lot of embedded users so we won't be able to drop the GCC
>> support for soft float, and dropping the glibc support puts us in a
>> clunky spot for testing.
>>
>> So we're probably stuck in a bit of a performance hole for a while.
>> That said, we could at least put a stake in the ground and start testing
>> some target with some newer extensions.  That way distros that want to
>> try a larger base than rv64gc have something that's getting tested and
>> is thus likely to work.  For GCC we ended up adding some test targets
>> with pretty much all the non-embedded, but at least B and V seem like
>> the bare minimum.
>>
>> I'm not sure if all that is worth the extra build/test resource burden
>> for everyone, though, particularly given the lack of hardware.
>
> for Android -- which is in a similar situation -- having the build
> system enable these extensions been pretty useful for finding
> toolchain and qemu bugs, which seems useful in its own right, if only
> for CI purposes...

Ya, that's kind of my theory here.  Hopefully hardware vendors will 
eventually align to some interesting set of extensions, maybe split 
between a few markets, but at least we can do _something_ here.  I'd be 
super surprised if we end up with any sort of long-term successful 
hardware that lacks B and V, so it seems like a safe target for testing.

>> ---
>>  scripts/build-many-glibcs.py | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
>> index ecc743e672..21da46dc4a 100755
>> --- a/scripts/build-many-glibcs.py
>> +++ b/scripts/build-many-glibcs.py
>> @@ -411,7 +411,9 @@ class Context(object):
>>                          os_name='linux-gnu',
>>                          variant='rv64imafdc-lp64d',
>>                          gcc_cfg=['--with-arch=rv64imafdc', '--with-abi=lp64d',
>> -                                 '--disable-multilib'])
>> +                                 '--disable-multilib']),
>> +                        extra_glibcs=[{'variant': 'rv64gcbv-rv64gc-lp64d',
>> +                                       'ccopts': '-march=rv64gcv_zba_zbb_zbs'}]
>>          self.add_config(arch='s390x',
>>                          os_name='linux-gnu',
>>                          glibcs=[{},
>> --
>> 2.44.0
>>

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Jeff Law]

On 4/15/24 2:22 PM, Palmer Dabbelt wrote:
> On Mon, 15 Apr 2024 13:05:53 PDT (-0700), enh@google.com wrote:
>> On Mon, Apr 15, 2024 at 12:27 PM Palmer Dabbelt <palmer@rivosinc.com> 
>> wrote:
>>>
>>> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
>>> ---
>>> So this is very much an RFC.  I had written a commit message as
>>>
>>>     This will hopefully be a widely used configuration in the future, 
>>> but
>>>     it's not all that common right now.  That said, let's be 
>>> proactive and
>>>     at start testing it.
>>>
>>> but I even that's kind of a strong statement here so I'm just stashing
>>> it off to the side to be sure.
>>>
>>> A few of us were talking about the GCC meeting last week, but I figured
>>> I'd send a glibc patch to at least start some discussion on the lists as
>>> something similar had come up in the TLSDESC thread and a while ago when
>>> we talked about glibc-hwcaps.
>>>
>>> I'm not really sure what the right thing to do here is: we don't have
>>> broadly available hardware that supports these new extensions, but
>>> they're pretty much table stakes for hardware that's competitive with
>>> arm64/x86.  Without B and V we're going to end up piling up a ton of
>>> IFUNC-based routines to try and work around the base ISA, but those
>>> extensions are so fundamental to good codegen that it doesn't feel like
>>> we're going to get where we want with just IFUNCs.
>>>
>>> Unfortunately I don't think we can drop support for the other base ISAs:
>>> the distros appear to be targeting rv64gc and that's the only common
>>> base for most hardware that's out there.  I guess we could drop support
>>> for the rv*ima-based base ISAs, but I'm not sure that buys us much:
>>> we've got a lot of embedded users so we won't be able to drop the GCC
>>> support for soft float, and dropping the glibc support puts us in a
>>> clunky spot for testing.
>>>
>>> So we're probably stuck in a bit of a performance hole for a while.
>>> That said, we could at least put a stake in the ground and start testing
>>> some target with some newer extensions.  That way distros that want to
>>> try a larger base than rv64gc have something that's getting tested and
>>> is thus likely to work.  For GCC we ended up adding some test targets
>>> with pretty much all the non-embedded, but at least B and V seem like
>>> the bare minimum.
>>>
>>> I'm not sure if all that is worth the extra build/test resource burden
>>> for everyone, though, particularly given the lack of hardware.
>>
>> for Android -- which is in a similar situation -- having the build
>> system enable these extensions been pretty useful for finding
>> toolchain and qemu bugs, which seems useful in its own right, if only
>> for CI purposes...
> 
> Ya, that's kind of my theory here.  Hopefully hardware vendors will 
> eventually align to some interesting set of extensions, maybe split 
> between a few markets, but at least we can do _something_ here.  I'd be 
> super surprised if we end up with any sort of long-term successful 
> hardware that lacks B and V, so it seems like a safe target for testing.
Definitely starting to see more B variants showing up.  V variants are 
still quite scarce.  I'm hoping the sweet spot is going to be rv64gcbv 
fairly soon, but I don't see it happening in time for the spring distro 
refreshes, obviously.

jeff

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Darius Rad]

Is this really lp64d?  What is the status of the vector ABI?  If we need to
have (and test) more targets that are actually useful, so be it.  But we
don't want to be stuck testing targets nobody uses or that provide no
testing value, and removing them is usually harder than adding them.


On Mon, Apr 15, 2024 at 12:24:15PM -0700, Palmer Dabbelt wrote:
> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
> ---
> So this is very much an RFC.  I had written a commit message as
> 
>     This will hopefully be a widely used configuration in the future, but
>     it's not all that common right now.  That said, let's be proactive and
>     at start testing it.
> 
> but I even that's kind of a strong statement here so I'm just stashing
> it off to the side to be sure.
> 
> A few of us were talking about the GCC meeting last week, but I figured
> I'd send a glibc patch to at least start some discussion on the lists as
> something similar had come up in the TLSDESC thread and a while ago when
> we talked about glibc-hwcaps.
> 
> I'm not really sure what the right thing to do here is: we don't have
> broadly available hardware that supports these new extensions, but
> they're pretty much table stakes for hardware that's competitive with
> arm64/x86.  Without B and V we're going to end up piling up a ton of
> IFUNC-based routines to try and work around the base ISA, but those
> extensions are so fundamental to good codegen that it doesn't feel like
> we're going to get where we want with just IFUNCs.
> 
> Unfortunately I don't think we can drop support for the other base ISAs:
> the distros appear to be targeting rv64gc and that's the only common
> base for most hardware that's out there.  I guess we could drop support
> for the rv*ima-based base ISAs, but I'm not sure that buys us much:
> we've got a lot of embedded users so we won't be able to drop the GCC
> support for soft float, and dropping the glibc support puts us in a
> clunky spot for testing.
> 
> So we're probably stuck in a bit of a performance hole for a while.
> That said, we could at least put a stake in the ground and start testing
> some target with some newer extensions.  That way distros that want to
> try a larger base than rv64gc have something that's getting tested and
> is thus likely to work.  For GCC we ended up adding some test targets
> with pretty much all the non-embedded, but at least B and V seem like
> the bare minimum.
> 
> I'm not sure if all that is worth the extra build/test resource burden
> for everyone, though, particularly given the lack of hardware.
> ---
>  scripts/build-many-glibcs.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
> index ecc743e672..21da46dc4a 100755
> --- a/scripts/build-many-glibcs.py
> +++ b/scripts/build-many-glibcs.py
> @@ -411,7 +411,9 @@ class Context(object):
>                          os_name='linux-gnu',
>                          variant='rv64imafdc-lp64d',
>                          gcc_cfg=['--with-arch=rv64imafdc', '--with-abi=lp64d',
> -                                 '--disable-multilib'])
> +                                 '--disable-multilib']),
> +                        extra_glibcs=[{'variant': 'rv64gcbv-rv64gc-lp64d',
> +                                       'ccopts': '-march=rv64gcv_zba_zbb_zbs'}]
>          self.add_config(arch='s390x',
>                          os_name='linux-gnu',
>                          glibcs=[{},
> -- 
> 2

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Palmer Dabbelt]

On Mon, 15 Apr 2024 13:24:14 PDT (-0700), Darius Rad wrote:
> Is this really lp64d?

I guess that depend on what you mean by "ABI".  Passing `-march=rv64gv 
-mabi=lp64d` will result in binaries that require the V extension to 
run, but can be linked with any other binary that is compiled with 
`-mabi=lp64d`.  It's essentially the same as `-march=rv64g -mabi=lp64`, 
except for vector instead of float.

> What is the status of the vector ABI?

We don't have a global vector ABI switch for GCC (ie, `-mabi=lp64dv`) 
and we decided not to put on in for GCC-14.  We've essentially got all 
the machinery there for it, as we've got the vector calling convention 
attribute, but that came in pretty late and we decided it'd be better to 
wait.

I don't see any reason it'd miss GCC-15, but no promises as there's not 
even a patch on the lists yet...

> If we need to
> have (and test) more targets that are actually useful, so be it.  But we
> don't want to be stuck testing targets nobody uses or that provide no
> testing value, and removing them is usually harder than adding them.

Ya, that's kind of my worry here.  We'd be going out on a limb here by 
targeting something without hardware availability, we've already got 
some baggage floating around from lightly-used targets and it'd be best 
to avoid too much of that.

IMO it's really more of a question for users/distros -- and I'm 
definitely a crazy person on that front, I run Gentoo ;)

> On Mon, Apr 15, 2024 at 12:24:15PM -0700, Palmer Dabbelt wrote:
>> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
>> ---
>> So this is very much an RFC.  I had written a commit message as
>>
>>     This will hopefully be a widely used configuration in the future, but
>>     it's not all that common right now.  That said, let's be proactive and
>>     at start testing it.
>>
>> but I even that's kind of a strong statement here so I'm just stashing
>> it off to the side to be sure.
>>
>> A few of us were talking about the GCC meeting last week, but I figured
>> I'd send a glibc patch to at least start some discussion on the lists as
>> something similar had come up in the TLSDESC thread and a while ago when
>> we talked about glibc-hwcaps.
>>
>> I'm not really sure what the right thing to do here is: we don't have
>> broadly available hardware that supports these new extensions, but
>> they're pretty much table stakes for hardware that's competitive with
>> arm64/x86.  Without B and V we're going to end up piling up a ton of
>> IFUNC-based routines to try and work around the base ISA, but those
>> extensions are so fundamental to good codegen that it doesn't feel like
>> we're going to get where we want with just IFUNCs.
>>
>> Unfortunately I don't think we can drop support for the other base ISAs:
>> the distros appear to be targeting rv64gc and that's the only common
>> base for most hardware that's out there.  I guess we could drop support
>> for the rv*ima-based base ISAs, but I'm not sure that buys us much:
>> we've got a lot of embedded users so we won't be able to drop the GCC
>> support for soft float, and dropping the glibc support puts us in a
>> clunky spot for testing.
>>
>> So we're probably stuck in a bit of a performance hole for a while.
>> That said, we could at least put a stake in the ground and start testing
>> some target with some newer extensions.  That way distros that want to
>> try a larger base than rv64gc have something that's getting tested and
>> is thus likely to work.  For GCC we ended up adding some test targets
>> with pretty much all the non-embedded, but at least B and V seem like
>> the bare minimum.
>>
>> I'm not sure if all that is worth the extra build/test resource burden
>> for everyone, though, particularly given the lack of hardware.
>> ---
>>  scripts/build-many-glibcs.py | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
>> index ecc743e672..21da46dc4a 100755
>> --- a/scripts/build-many-glibcs.py
>> +++ b/scripts/build-many-glibcs.py
>> @@ -411,7 +411,9 @@ class Context(object):
>>                          os_name='linux-gnu',
>>                          variant='rv64imafdc-lp64d',
>>                          gcc_cfg=['--with-arch=rv64imafdc', '--with-abi=lp64d',
>> -                                 '--disable-multilib'])
>> +                                 '--disable-multilib']),
>> +                        extra_glibcs=[{'variant': 'rv64gcbv-rv64gc-lp64d',
>> +                                       'ccopts': '-march=rv64gcv_zba_zbb_zbs'}]
>>          self.add_config(arch='s390x',
>>                          os_name='linux-gnu',
>>                          glibcs=[{},
>> --
>> 2

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Darius Rad]

On Mon, Apr 15, 2024 at 02:36:59PM -0700, Palmer Dabbelt wrote:
> On Mon, 15 Apr 2024 13:24:14 PDT (-0700), Darius Rad wrote:
> > Is this really lp64d?
> 
> I guess that depend on what you mean by "ABI".  Passing `-march=rv64gv
> -mabi=lp64d` will result in binaries that require the V extension to run,
> but can be linked with any other binary that is compiled with `-mabi=lp64d`.
> It's essentially the same as `-march=rv64g -mabi=lp64`, except for vector
> instead of float.
> 

Got it.  I thought that might be the case, but wasn't sure if the vector
calling convention leaked in with the arch switch only (and not the abi).

> > What is the status of the vector ABI?
> 
> We don't have a global vector ABI switch for GCC (ie, `-mabi=lp64dv`) and we
> decided not to put on in for GCC-14.  We've essentially got all the
> machinery there for it, as we've got the vector calling convention
> attribute, but that came in pretty late and we decided it'd be better to
> wait.
> 
> I don't see any reason it'd miss GCC-15, but no promises as there's not even
> a patch on the lists yet...
> 

Understood.  Though when this hits, won't it be the most likely target for
vector?  Meaning at that point, rv64gcbv/lp64d would be less useful or
important.

> > If we need to
> > have (and test) more targets that are actually useful, so be it.  But we
> > don't want to be stuck testing targets nobody uses or that provide no
> > testing value, and removing them is usually harder than adding them.
> 
> Ya, that's kind of my worry here.  We'd be going out on a limb here by
> targeting something without hardware availability, we've already got some
> baggage floating around from lightly-used targets and it'd be best to avoid
> too much of that.
> 

Maybe make an explicit note that if/when the vector ABI is in gcc, the
target will be switched to that, rather than another target being added.
Then there's some notice for people to not rely on rv64gcbv/lp64d, or at
least put up support and/or evidence of a use case to retain the target.

> IMO it's really more of a question for users/distros -- and I'm definitely a
> crazy person on that front, I run Gentoo ;)
> 

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Palmer Dabbelt]

On Tue, 16 Apr 2024 07:55:38 PDT (-0700), Darius Rad wrote:
> On Mon, Apr 15, 2024 at 02:36:59PM -0700, Palmer Dabbelt wrote:
>> On Mon, 15 Apr 2024 13:24:14 PDT (-0700), Darius Rad wrote:
>> > Is this really lp64d?
>>
>> I guess that depend on what you mean by "ABI".  Passing `-march=rv64gv
>> -mabi=lp64d` will result in binaries that require the V extension to run,
>> but can be linked with any other binary that is compiled with `-mabi=lp64d`.
>> It's essentially the same as `-march=rv64g -mabi=lp64`, except for vector
>> instead of float.
>>
>
> Got it.  I thought that might be the case, but wasn't sure if the vector
> calling convention leaked in with the arch switch only (and not the abi).

The intent was to avoid that, but GCC-14 will be the first release with 
any meaningful vector codegen so I guess we'll find out if we have any 
bugs ;)

>> > What is the status of the vector ABI?
>>
>> We don't have a global vector ABI switch for GCC (ie, `-mabi=lp64dv`) and we
>> decided not to put on in for GCC-14.  We've essentially got all the
>> machinery there for it, as we've got the vector calling convention
>> attribute, but that came in pretty late and we decided it'd be better to
>> wait.
>>
>> I don't see any reason it'd miss GCC-15, but no promises as there's not even
>> a patch on the lists yet...
>>
>
> Understood.  Though when this hits, won't it be the most likely target for
> vector?  Meaning at that point, rv64gcbv/lp64d would be less useful or
> important.

I'm not really sure.  If I was building a system I'd want an 
lp64dv-flavored ABI, but if distros are already rv64gc/lp64d then 
rv64gcv/lp64d would provide a binary compatible upgrade path for them.  
Assuming we end up with some sort of glibc-hwcaps the rv64gcv/lp64d 
builds might be common.

It's kind of all just a guess until HW shows up, though...

>> > If we need to
>> > have (and test) more targets that are actually useful, so be it.  But we
>> > don't want to be stuck testing targets nobody uses or that provide no
>> > testing value, and removing them is usually harder than adding them.
>>
>> Ya, that's kind of my worry here.  We'd be going out on a limb here by
>> targeting something without hardware availability, we've already got some
>> baggage floating around from lightly-used targets and it'd be best to avoid
>> too much of that.
>>
>
> Maybe make an explicit note that if/when the vector ABI is in gcc, the
> target will be switched to that, rather than another target being added.
> Then there's some notice for people to not rely on rv64gcbv/lp64d, or at
> least put up support and/or evidence of a use case to retain the target.

Ya, I guess so far we've not really been explicit about what's supported 
in glibc and what's not -- there's some code in there to ban stuff like 
lv64f and some default ISAs in build-many-glibcs, but I don't know if 
any of that is really binding.

So maybe we should do a wiki entry with what configurations are actively 
tested and thus likely to work?

>> IMO it's really more of a question for users/distros -- and I'm definitely a
>> crazy person on that front, I run Gentoo ;)
>>

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Darius Rad]

On Tue, Apr 16, 2024 at 09:04:24AM -0700, Palmer Dabbelt wrote:
> On Tue, 16 Apr 2024 07:55:38 PDT (-0700), Darius Rad wrote:
> > On Mon, Apr 15, 2024 at 02:36:59PM -0700, Palmer Dabbelt wrote:
> > > On Mon, 15 Apr 2024 13:24:14 PDT (-0700), Darius Rad wrote:
> > > > Is this really lp64d?
> > > 
> > > I guess that depend on what you mean by "ABI".  Passing `-march=rv64gv
> > > -mabi=lp64d` will result in binaries that require the V extension to run,
> > > but can be linked with any other binary that is compiled with `-mabi=lp64d`.
> > > It's essentially the same as `-march=rv64g -mabi=lp64`, except for vector
> > > instead of float.
> > > 
> > 
> > Got it.  I thought that might be the case, but wasn't sure if the vector
> > calling convention leaked in with the arch switch only (and not the abi).
> 
> The intent was to avoid that, but GCC-14 will be the first release with any
> meaningful vector codegen so I guess we'll find out if we have any bugs ;)
> 
> > > > What is the status of the vector ABI?
> > > 
> > > We don't have a global vector ABI switch for GCC (ie, `-mabi=lp64dv`) and we
> > > decided not to put on in for GCC-14.  We've essentially got all the
> > > machinery there for it, as we've got the vector calling convention
> > > attribute, but that came in pretty late and we decided it'd be better to
> > > wait.
> > > 
> > > I don't see any reason it'd miss GCC-15, but no promises as there's not even
> > > a patch on the lists yet...
> > > 
> > 
> > Understood.  Though when this hits, won't it be the most likely target for
> > vector?  Meaning at that point, rv64gcbv/lp64d would be less useful or
> > important.
> 
> I'm not really sure.  If I was building a system I'd want an lp64dv-flavored
> ABI, but if distros are already rv64gc/lp64d then rv64gcv/lp64d would
> provide a binary compatible upgrade path for them.  Assuming we end up with
> some sort of glibc-hwcaps the rv64gcv/lp64d builds might be common.
> 
> It's kind of all just a guess until HW shows up, though...
> 
> > > > If we need to
> > > > have (and test) more targets that are actually useful, so be it.  But we
> > > > don't want to be stuck testing targets nobody uses or that provide no
> > > > testing value, and removing them is usually harder than adding them.
> > > 
> > > Ya, that's kind of my worry here.  We'd be going out on a limb here by
> > > targeting something without hardware availability, we've already got some
> > > baggage floating around from lightly-used targets and it'd be best to avoid
> > > too much of that.
> > > 
> > 
> > Maybe make an explicit note that if/when the vector ABI is in gcc, the
> > target will be switched to that, rather than another target being added.
> > Then there's some notice for people to not rely on rv64gcbv/lp64d, or at
> > least put up support and/or evidence of a use case to retain the target.
> 
> Ya, I guess so far we've not really been explicit about what's supported in
> glibc and what's not -- there's some code in there to ban stuff like lv64f
> and some default ISAs in build-many-glibcs, but I don't know if any of that
> is really binding.
> 

At least build-many-glibcs, and more importantly the release notes,
indicate what is actually tested, which says something.

> So maybe we should do a wiki entry with what configurations are actively
> tested and thus likely to work?
> 

Sure, if we think that can be kept up to date.  I think I would rather take
an official position on what build-many-glibcs or the release testing means
for supported targets (if there isn't something glibc-wide already), then
we don't have another thing to maintain.

> > > IMO it's really more of a question for users/distros -- and I'm definitely a
> > > crazy person on that front, I run Gentoo

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Palmer Dabbelt]

On Tue, 16 Apr 2024 13:41:15 PDT (-0700), Darius Rad wrote:
> On Tue, Apr 16, 2024 at 09:04:24AM -0700, Palmer Dabbelt wrote:
>> On Tue, 16 Apr 2024 07:55:38 PDT (-0700), Darius Rad wrote:
>> > On Mon, Apr 15, 2024 at 02:36:59PM -0700, Palmer Dabbelt wrote:
>> > > On Mon, 15 Apr 2024 13:24:14 PDT (-0700), Darius Rad wrote:
>> > > > Is this really lp64d?
>> > >
>> > > I guess that depend on what you mean by "ABI".  Passing `-march=rv64gv
>> > > -mabi=lp64d` will result in binaries that require the V extension to run,
>> > > but can be linked with any other binary that is compiled with `-mabi=lp64d`.
>> > > It's essentially the same as `-march=rv64g -mabi=lp64`, except for vector
>> > > instead of float.
>> > >
>> >
>> > Got it.  I thought that might be the case, but wasn't sure if the vector
>> > calling convention leaked in with the arch switch only (and not the abi).
>>
>> The intent was to avoid that, but GCC-14 will be the first release with any
>> meaningful vector codegen so I guess we'll find out if we have any bugs ;)
>>
>> > > > What is the status of the vector ABI?
>> > >
>> > > We don't have a global vector ABI switch for GCC (ie, `-mabi=lp64dv`) and we
>> > > decided not to put on in for GCC-14.  We've essentially got all the
>> > > machinery there for it, as we've got the vector calling convention
>> > > attribute, but that came in pretty late and we decided it'd be better to
>> > > wait.
>> > >
>> > > I don't see any reason it'd miss GCC-15, but no promises as there's not even
>> > > a patch on the lists yet...
>> > >
>> >
>> > Understood.  Though when this hits, won't it be the most likely target for
>> > vector?  Meaning at that point, rv64gcbv/lp64d would be less useful or
>> > important.
>>
>> I'm not really sure.  If I was building a system I'd want an lp64dv-flavored
>> ABI, but if distros are already rv64gc/lp64d then rv64gcv/lp64d would
>> provide a binary compatible upgrade path for them.  Assuming we end up with
>> some sort of glibc-hwcaps the rv64gcv/lp64d builds might be common.
>>
>> It's kind of all just a guess until HW shows up, though...
>>
>> > > > If we need to
>> > > > have (and test) more targets that are actually useful, so be it.  But we
>> > > > don't want to be stuck testing targets nobody uses or that provide no
>> > > > testing value, and removing them is usually harder than adding them.
>> > >
>> > > Ya, that's kind of my worry here.  We'd be going out on a limb here by
>> > > targeting something without hardware availability, we've already got some
>> > > baggage floating around from lightly-used targets and it'd be best to avoid
>> > > too much of that.
>> > >
>> >
>> > Maybe make an explicit note that if/when the vector ABI is in gcc, the
>> > target will be switched to that, rather than another target being added.
>> > Then there's some notice for people to not rely on rv64gcbv/lp64d, or at
>> > least put up support and/or evidence of a use case to retain the target.
>>
>> Ya, I guess so far we've not really been explicit about what's supported in
>> glibc and what's not -- there's some code in there to ban stuff like lv64f
>> and some default ISAs in build-many-glibcs, but I don't know if any of that
>> is really binding.
>>
>
> At least build-many-glibcs, and more importantly the release notes,
> indicate what is actually tested, which says something.

Certainly the list of tested targets per release is a pretty strong sign 
of what's actually working now, which is great (and thanks for testing 
basically every release...).

>> So maybe we should do a wiki entry with what configurations are actively
>> tested and thus likely to work?
>>
>
> Sure, if we think that can be kept up to date.  I think I would rather take
> an official position on what build-many-glibcs or the release testing means
> for supported targets (if there isn't something glibc-wide already), then
> we don't have another thing to maintain.

Ya, that's best.  I guess that's more of a glibc-wide question, though, 
so we'll have to see if someone else chimes in?

>> > > IMO it's really more of a question for users/distros -- and I'm definitely a
>> > > crazy person on that front, I run Gentoo

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Michael Hudson-Doyle]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

On Tue, 16 Apr 2024 at 07:28, Palmer Dabbelt <palmer@rivosinc.com> wrote:

> Unfortunately I don't think we can drop support for the other base ISAs:
> the distros appear to be targeting rv64gc and that's the only common
> base for most hardware that's out there.
>

This is certainly the case for Ubuntu for now but I certainly hope that by
26.04 we are able to be a bit more aggressive here, fwiw.

Cheers,
mwh

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Palmer Dabbelt]

On Tue, 16 Apr 2024 00:01:31 PDT (-0700), michael.hudson@canonical.com wrote:
> On Tue, 16 Apr 2024 at 07:28, Palmer Dabbelt <palmer@rivosinc.com> wrote:
>
>> Unfortunately I don't think we can drop support for the other base ISAs:
>> the distros appear to be targeting rv64gc and that's the only common
>> base for most hardware that's out there.
>>
>
> This is certainly the case for Ubuntu for now but I certainly hope that by
> 26.04 we are able to be a bit more aggressive here, fwiw.

That'd be awesome.  Any idea if something like this would help?  

I think we're all a bit worried about being stuck with a very small 
baseline and thus a bunch of vendor-specific behavior.  That's certainly 
the way things are trending right now in RISC-V land, it's already super 
complicated to manage and it looks like it's going to get worse before 
it gets better.

If we can pick up some extra testing burden now as a way to reduce 
fragmentation then it's probably a win in the long run, even if we have 
to go through a few generations of this before things calm down.  If 
you're planning on moving to something completely different, though, 
then maybe it's not worth the extra builds.

> Cheers,
> mwh

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Florian Weimer]

* Palmer Dabbelt:

> If we can pick up some extra testing burden now as a way to reduce
> fragmentation then it's probably a win in the long run, even if we
> have to go through a few generations of this before things calm down.
> If you're planning on moving to something completely different,
> though, then maybe it's not worth the extra builds.

A run-time tester integrated with patchwork might be useful, assuming
it's stable and produces clean results for the current tree.

Thanks,
Florian

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Michael Hudson-Doyle]

[-- Attachment #1: Type: text/plain, Size: 2318 bytes --]

On Wed, 17 Apr 2024 at 06:12, Palmer Dabbelt <palmer@rivosinc.com> wrote:

> On Tue, 16 Apr 2024 00:01:31 PDT (-0700), michael.hudson@canonical.com
> wrote:
> > On Tue, 16 Apr 2024 at 07:28, Palmer Dabbelt <palmer@rivosinc.com>
> wrote:
> >
> >> Unfortunately I don't think we can drop support for the other base ISAs:
> >> the distros appear to be targeting rv64gc and that's the only common
> >> base for most hardware that's out there.
> >>
> >
> > This is certainly the case for Ubuntu for now but I certainly hope that
> by
> > 26.04 we are able to be a bit more aggressive here, fwiw.
>
> That'd be awesome.  Any idea if something like this would help?


Well having the non baseline builds tested and so not having everything
explode as/when we shift the baseline would probably help, yes :-)


> I think we're all a bit worried about being stuck with a very small
> baseline and thus a bunch of vendor-specific behavior.


Yes. That would be unfortunate.


> That's certainly
> the way things are trending right now in RISC-V land, it's already super
> complicated to manage and it looks like it's going to get worse before
> it gets better.
>

I think for Ubuntu we are obviously not interested in supporting every
RISC-V core out there, we are only going to be aiming for what would be a
"cortex a" CPU in the ARM world and I think the baseline for those is
probably going to be a bit more than just rv64gc (like, we could almost
certainly just copy Google's baseline for Android).


> If we can pick up some extra testing burden now as a way to reduce
> fragmentation then it's probably a win in the long run, even if we have
> to go through a few generations of this before things calm down.  If
> you're planning on moving to something completely different, though,
> then maybe it's not worth the extra builds.
>

We certainly don't have any concrete plans beyond "we don't want to be
stuck on rv64gc forever". We also don't want to be building every package
in Ubuntu for every core either! It's obviously hard to see where things
are going but I can imagine a world where there are different baselines for
different application domains in practice (e.g. iot, routers, servers) with
different baselines in practice and it's worth building some core packages
to target those. But who knows!

Cheers,
mwh

Re: [RFC] build-many-glibcs: Add a rv64gcbv-on-rv64gc/lp64d sub-variant

[Florian Weimer]

* Michael Hudson-Doyle:

> On Tue, 16 Apr 2024 at 07:28, Palmer Dabbelt <palmer@rivosinc.com> wrote:
>
>> Unfortunately I don't think we can drop support for the other base
>> ISAs: the distros appear to be targeting rv64gc and that's the only
>> common base for most hardware that's out there.
>
> This is certainly the case for Ubuntu for now but I certainly hope
> that by 26.04 we are able to be a bit more aggressive here, fwiw.

You might want to add an early check to ld.so to report CPU
compatibilities, similar to what we do using gcc-macros.h in
sysdeps/powerpc/powerpc64/le/dl-hwcap-check.h,
sysdeps/s390/s390-64/dl-hwcap-check.h,
sysdeps/x86/dl-get-cpu-features.c.  This requires configure glibc withe
the appropriate --with-rtld-early-cflags option, as GCC does not provide
a way to force the compiler to use the baseline ISA once the defaults
have changed (either within GCC itself, based on its configuration, or
earlier on the command line).

Thanks,
Florian