RISC-V support

RISC-V support

[Jim Wilson]

I'm looking at the RISC-V elfutils support to help the Debian folks.
I see four testcases failing, same as Kurt Roeckx reported about 6
weeks ago.  I'm testing on a Fedora Core 29 system.

I found a trivial bug in backends/riscv_corenote.c.  It has ".offset =
1" but this is a byte offset not a register offset, so it needs to be
".offset = 8" instead.  I also added in the missing PC support.  These
two fixes then require a fix for tests/run-readelf-mixed-corenote.sh
because the eu-readelf output is now more correct than before.

There is also a missing backends/riscv_retval.c file.  I have an
initial implementation for this, but I haven't implemented the support
for structures with one or two float fields yet, as this gets a little
complicated.

With these patches, I now see two failures.  One is a glibc bug that
Andreas Schwab already fixed, where _start fails to terminate the
unwind chain.  I just don't have this patch on my system.  The other
failure is the same glibc bug in __thread_start, which apparently
isn't fixed yet, and needs the same fix Andreas already added to
_start.  So with the appropriate glibc fixes, the elfutils testsuite
should run without error on a riscv64/lp64d system using the patches I
have.

There is a problem here though.  The riscv support was written to try
to handle both 32-bit and 64-bit targets with a single elfutils
backend.  But I have 6 ABIs I need to (theoretically) handle in
riscv_retval.c.  The return_value_location function doesn't take any
ebl or elf pointer, so I can't handle it there.  I can handle it in
riscv_init.c by checking ebl and elf pointers there, and calling an
appropriate function, but I'm not sure if that is OK.  Currently, none
of the *_init.c files are using the elf pointer argument.

I noticed another problem which is that riscv_corenote.c is only
correct for riscv64, because it assumes that registers are 64-bits.
But I see that sparc has a solution for that, so I will have to take a
closer look at that and see if I can make it work for riscv.

I unfortunately can't test the 32-bit riscv support.  We don't have
working upstream support for 32-bit linux yet.  I can only test the
64-bit LP64D riscv support.

I haven't contributed to elfutils before.  So I'm looking for advice
on how to proceed.  I can send out my work in progress patches if that
is useful.  I probably should try to chop them up a bit first. I think
I have 3 parts at the moment.  One part should be OK, and one part
needs more work to be complete (but maybe incomplete is OK?), and one
part I haven't written yet.

Jim

Re: RISC-V support

[Mark Wielaard]

Hi Jim,

Apologies for the late reply, holiday season.
Seeing some patches already posted I think you already found the
answers to your questions, but just to be sure, lets answer this email
before reviewing the actual patches.

On Wed, 2018-12-26 at 18:29 -0800, Jim Wilson wrote:
> I'm looking at the RISC-V elfutils support to help the Debian folks.
> I see four testcases failing, same as Kurt Roeckx reported about 6
> weeks ago.  I'm testing on a Fedora Core 29 system.

This is very exciting, I had no idea RISC-V was already so complete. I
just installed a fedora stage 4 image using libvirt/qemu. It is
somewhat slow (elfutils build still running), but seems very complete.

> I found a trivial bug in backends/riscv_corenote.c.  It has ".offset =
> 1" but this is a byte offset not a register offset, so it needs to be
> ".offset = 8" instead.  I also added in the missing PC support.  These
> two fixes then require a fix for tests/run-readelf-mixed-corenote.sh
> because the eu-readelf output is now more correct than before.

Thanks.

> There is also a missing backends/riscv_retval.c file.  I have an
> initial implementation for this, but I haven't implemented the support
> for structures with one or two float fields yet, as this gets a little
> complicated.

Support for just the basic types is already very useful. This is used
for example by systemtap for return probes.

> With these patches, I now see two failures.  One is a glibc bug that
> Andreas Schwab already fixed, where _start fails to terminate the
> unwind chain.  I just don't have this patch on my system.  The other
> failure is the same glibc bug in __thread_start, which apparently
> isn't fixed yet, and needs the same fix Andreas already added to
> _start.  So with the appropriate glibc fixes, the elfutils testsuite
> should run without error on a riscv64/lp64d system using the patches
> I
> have.

The _start one seems to be:
https://sourceware.org/bugzilla/show_bug.cgi?id=23125
So that is fixed with glibc 2.29.

Do you have a bug for the second issue with __thread_start?

My build finally finished. autoreconf && configure && make -j2 && make
check -j2 took 25 minutes. It is using glibc 2.27.9000. I'll see if I
can upgrade it somehow.

It would be nice to have some riscv setup for our buildbot. Do you
happen to have recommendations for something like that? Any distro that
gets regular toolchain updates? Is a libvirt/qemu setup reliable enough
or would you recommend trying to get real hardware?

> There is a problem here though.  The riscv support was written to try
> to handle both 32-bit and 64-bit targets with a single elfutils
> backend.  But I have 6 ABIs I need to (theoretically) handle in
> riscv_retval.c.  The return_value_location function doesn't take any
> ebl or elf pointer, so I can't handle it there.  I can handle it in
> riscv_init.c by checking ebl and elf pointers there, and calling an
> appropriate function, but I'm not sure if that is OK.  Currently,
> none
> of the *_init.c files are using the elf pointer argument.

The ppc64 init does (to lookup the odp table which is necessary for
ppc64[be], but not ppc64le). It is allowed. And the backends/ebl
interface is completely internal, so feel free to suggest changes if
they make sense for riscv. If it is necessary we'll just update the
other backends.

> I noticed another problem which is that riscv_corenote.c is only
> correct for riscv64, because it assumes that registers are 64-bits.
> But I see that sparc has a solution for that, so I will have to take
> a
> closer look at that and see if I can make it work for riscv.
> 
> I unfortunately can't test the 32-bit riscv support.  We don't have
> working upstream support for 32-bit linux yet.  I can only test the
> 64-bit LP64D riscv support.
> 
> I haven't contributed to elfutils before.  So I'm looking for advice
> on how to proceed.  I can send out my work in progress patches if
> that
> is useful.  I probably should try to chop them up a bit first. I
> think
> I have 3 parts at the moment.  One part should be OK, and one part
> needs more work to be complete (but maybe incomplete is OK?), and one
> part I haven't written yet.

Looking at the patches you did sent it looks you already found the
CONTRIBUTING file:
https://sourceware.org/git/?p=elfutils.git;a=blob_plain;f=CONTRIBUTING
If not, please look through it, we are fairly easy on contributions, just make sure you read and agree to the Developer's Certfificate of Origin.

Partial patches are OK. But it is preferred to have them somewhat testable. Please don't contribute code that is completely theoretical and cannot be validated at all.

Thanks,

Mark

Re: RISC-V support

[Jim Wilson]

On Tue, Jan 8, 2019 at 5:52 AM Mark Wielaard <mark@klomp.org> wrote:
> The _start one seems to be:
> https://sourceware.org/bugzilla/show_bug.cgi?id=23125
> So that is fixed with glibc 2.29.
>
> Do you have a bug for the second issue with __thread_start?

https://sourceware.org/bugzilla/show_bug.cgi?id=24040

I plan to write a patch for it, if Andreas Schwab doesn't get to it
first, but the Fedora koji package server was down for a few weeks
which prevented me from installing the packages I needed to do this
work.  It is back up and I am looking at this now.

> It would be nice to have some riscv setup for our buildbot. Do you
> happen to have recommendations for something like that? Any distro that
> gets regular toolchain updates? Is a libvirt/qemu setup reliable enough
> or would you recommend trying to get real hardware?

I've been using Fedora, but Debian and OpenSuse are both usable too.
Fedora is just a little more convenient for me as they have images I
can download and boot, and a git tree for building a kernel with NBD
support which makes the HiFive Unleashed much more stable than when
using the SDcard for the root file system.  Plus I used to work for
Cygnus/Red Hat so maybe I am a little biased.

QEMU works too.  There are some known bugs in the RISC-V qemu FP
support, I wouldn't use it for numerical work.  If you stress the
system you might find some bugs, but overall it is certainly usable.
I used qemu for my gdb work, because it was easier to try kernel
patches that way, and I didn't want to risk trying untested kernel
patches on my main RISC-V development system.  The linux distro folks
are using qemu for builds also, to supplement the builds on hardware,
as they have limited numbers of boards at present.

There is only one ASIC option for running linux that I know of at this
time, and that is the SiFive HiFive Unleashed.  It is faster than qemu
running on a fast x86_64 machine.  It is stable if you don't use the
SDcard for anything other than booting.  I've hit 60 days of uptime on
my board, running a patched 4.15 kernel.  There are also FPGA options,
but these are more complicated and probably more expensive than using
the HiFive Unleashed.  I expect that more hardware options will be
available in the future.

Jim

Re: RISC-V support

[Karsten Merker]

On Tue, Jan 08, 2019 at 02:52:33PM +0100, Mark Wielaard wrote:

> It would be nice to have some riscv setup for our buildbot.  Do
> you happen to have recommendations for something like that? 
> Any distro that gets regular toolchain updates?

Hello,

the Debian riscv64 port is based on the Debian "unstable" suite
(i.e. Debian's development branch), so it usually follows
upstream's toolchain development rather closely.  Things will
slow down a bit during the next months, though, as Debian will
enter the staged freeze for the next stable release.

Debian doesn't have ready-made system images for riscv64 right
now.  Our plan is to provide a regular Debian-Installer image
for riscv64, but one of the last big blockers for that are
exactly the elfutils issues that Jim is trying to address in
this thread :-).

If you have access to a Debian/unstable system somewhere,
setting up a Debian-based riscv64 VM can be done rather easily. 
There is a step-by-step documentation in the Debian Wiki at

  https://wiki.debian.org/RISC-V#Setting_up_a_riscv64_virtual_machine

If you should happen to have further questions, feel free to
contact me.

> Is a libvirt/qemu setup reliable enough or would you recommend
> trying to get real hardware?

Qemu works quite well (both user-mode as well as full-system
emulation).  Debian currently runs a large number of riscv64
package builders in qemu-system instances and they have shown
to be very reliable.

Regards,
Karsten
-- 
Gem. Par. 28 Abs. 4 Bundesdatenschutzgesetz widerspreche ich der Nutzung
sowie der Weitergabe meiner personenbezogenen Daten für Zwecke der
Werbung sowie der Markt- oder Meinungsforschung.

Re: RISC-V support

[Kurt Roeckx]

On Tue, Jan 08, 2019 at 02:52:33PM +0100, Mark Wielaard wrote:
> > There is a problem here though.  The riscv support was written to try
> > to handle both 32-bit and 64-bit targets with a single elfutils
> > backend.  But I have 6 ABIs I need to (theoretically) handle in
> > riscv_retval.c.  The return_value_location function doesn't take any
> > ebl or elf pointer, so I can't handle it there.  I can handle it in
> > riscv_init.c by checking ebl and elf pointers there, and calling an
> > appropriate function, but I'm not sure if that is OK.  Currently,
> > none
> > of the *_init.c files are using the elf pointer argument.
> 
> The ppc64 init does (to lookup the odp table which is necessary for
> ppc64[be], but not ppc64le). It is allowed. And the backends/ebl
> interface is completely internal, so feel free to suggest changes if
> they make sense for riscv. If it is necessary we'll just update the
> other backends.

I've been looking at mips, and it seems to have many different
ABIs too. A patch I've received does this:
int
mips_return_value_location (Dwarf_Die *functypedie, const Dwarf_Op **locp)
{
  /* First find the ABI used by the elf object */
  enum mips_abi abi = find_mips_abi(functypedie->cu->dbg->elf);

The patch only supports 6 ABIs, but I think there are really over
10 ABIs.

Maybe it would be good that we only need to determine the ABI
once, instead of each time the function is called.


Kurt

Re: RISC-V support

[Mark Wielaard]

On Sat, 2019-01-12 at 00:23 +0100, Kurt Roeckx wrote:
> I've been looking at mips, and it seems to have many different
> ABIs too. A patch I've received does this:
> int
> mips_return_value_location (Dwarf_Die *functypedie, const Dwarf_Op
> **locp)
> {
>   /* First find the ABI used by the elf object */
>   enum mips_abi abi = find_mips_abi(functypedie->cu->dbg->elf);
> 
> The patch only supports 6 ABIs, but I think there are really over
> 10 ABIs.

But how many are actually used? Which does Debian support?

> Maybe it would be good that we only need to determine the ABI
> once, instead of each time the function is called.

Yes, so instead of having each hook find the correct abi, you can do
that in the init hook once. And then either assigning a abi specific
callback hook, or setting some struct ebl data field (ppc64 added one
for function descriptor support for example).

Cheers,

Mark

Re: RISC-V support

[Kurt Roeckx]

On Sat, Jan 12, 2019 at 11:35:48PM +0100, Mark Wielaard wrote:
> On Sat, 2019-01-12 at 00:23 +0100, Kurt Roeckx wrote:
> > I've been looking at mips, and it seems to have many different
> > ABIs too. A patch I've received does this:
> > int
> > mips_return_value_location (Dwarf_Die *functypedie, const Dwarf_Op
> > **locp)
> > {
> >   /* First find the ABI used by the elf object */
> >   enum mips_abi abi = find_mips_abi(functypedie->cu->dbg->elf);
> > 
> > The patch only supports 6 ABIs, but I think there are really over
> > 10 ABIs.
> 
> But how many are actually used? Which does Debian support?

I'm not at all an export of mips, I really don't know that much
about it.

We have 3 mips ports, but mips and mipsel are the same ABI, it's
just big endian vs little endian. But as far as I understand it,
we transitioned those 2 ports from one ABI to an other, so they
are compatible with yet an other ABI. I think that at least makes
4 ABIs we care about. And I think gcc supports many different ones
depending on compiler flags you give to it, but I really don't know.


Kurt

Re: RISC-V support

[Jim Wilson]

On Sat, Jan 12, 2019 at 3:21 PM Kurt Roeckx <kurt@roeckx.be> wrote:
> > But how many are actually used? Which does Debian support?
>
> I'm not at all an export of mips, I really don't know that much
> about it.

It depends on how you count ABIs, but yes there have unfortunately
been a lot of them over the years.

As a practical matter, you should only need support for the (old) 32
bit ABI with pic support, the n32 ABI (which is 32-bit types on a
64-bit machine like the x86_64 x32 ABI), and the (new) 64-bit ABI.
Those are the only ones that gcc supports for linux and other POSIX
operating systems.  These exist in both big-endian and little-endian
forms.

There are a bunch of other ABIs or ABI variants that were only ever
supported for embedded systems, or were used on old pre-shared library
systems, or were never implemented in FSF GCC.  So you can ignore all
of those.

Jim

Re: RISC-V support

[Kurt Roeckx]

On Sat, Jan 12, 2019 at 05:06:18PM -0800, Jim Wilson wrote:
> On Sat, Jan 12, 2019 at 3:21 PM Kurt Roeckx <kurt@roeckx.be> wrote:
> > > But how many are actually used? Which does Debian support?
> >
> > I'm not at all an export of mips, I really don't know that much
> > about it.
> 
> It depends on how you count ABIs, but yes there have unfortunately
> been a lot of them over the years.
> 
> As a practical matter, you should only need support for the (old) 32
> bit ABI with pic support, the n32 ABI (which is 32-bit types on a
> 64-bit machine like the x86_64 x32 ABI), and the (new) 64-bit ABI.
> Those are the only ones that gcc supports for linux and other POSIX
> operating systems.  These exist in both big-endian and little-endian
> forms.

O32 really has at least the following variants:
- O32 FP32
- O32 FPXX
- O32 FP64
- O32 FP64A

They are all different ABIs related to the floating points. It's my
understanding that Debian's mips and mipsel port was O32 FP32 and is now
using O32 FPXX, and that mips64el is using N64.


Kurt

Re: RISC-V support

[Jim Wilson]

On Sat, Jan 12, 2019 at 5:23 PM Kurt Roeckx <kurt@roeckx.be> wrote:
> O32 really has at least the following variants:
> - O32 FP32
> - O32 FPXX
> - O32 FP64
> - O32 FP64A

The FPXX and FP64A stuff is new, after I stopped tracking MIPS stuff,
so I hadn't noticed it.  I see why they added these, but if you count
soft-float, single-float, no-float, and a now deprecated 64-bit FP reg
mode, there are 8 different ways to handle FP code for o32.  Crazy.
Most of them aren't relevant to linux though.  And I suppose 20-30
years from now RISC-V may have the same problem.

Jim