Plugin-based opcode table

Plugin-based opcode table

[Dmitry Selyutin]

Hello folks,

recently we've been discussing the opcode allocation for SVP64[0]. Either
the exact opcodes should be reserved for SVP64 needs, or there must be some
established and documented way to determine programmatically that the code
uses SVP64 extensions, and not other extension. Either way, we're waiting
for OpenPOWER Foundation decision.

Before this decision is taken, and OPF approves and documents it, we'd like
to have some option to adjust the opcodes dynamically, effectively
populating the opcodes table with SVP64 instructions. Or, alternatively,
have a way to provide the alternative opcode table. This way, we would not
pollute the opcodes table and vanilla binutils with opcodes not blessed by
OPF yet. We've been thinking of some plugin, with obvious candidates being
dlopen/dlsym combo. Sure the code will be public, as well as the whole work
around libresoc and SVP64. Also, we might opt to tune few other things via
plugin as well: SVP64 prefixed opcodes are a perfect example of the place
where we fork.

I recall there are plugins in binutils, but I'm not sure whether these are
designed to areas other than linker. I'm also not sure whether this is
supported, and if there is a sufficient documentation on API. A quick
search yields include/plugin-api.h, which might be not what we want.

We'd like to consult with binutils developers on how to approach this
issue, and collect a feedback. Thank you in advance!

[0] https://sourceware.org/pipermail/binutils/2022-May/120775.html

Best regards,
Dmitry

Re: Plugin-based opcode table

[lkcl]

On May 17, 2022 1:37:36 PM GMT+01:00, Dmitry Selyutin
<ghostmansd@gmail.com> wrote:

> We've been thinking of some plugin, with obvious candidates being
>dlopen/dlsym combo. Sure the code will be public, as well as the whole
>work around libresoc and SVP64.

... but it would not be "endorsed" - implicitly and accidentally - by
the binutils team making decisions that are at odds with the direction
set by the OpenPOWER Foundation.

prior to approval yes-or-no by OPF, a plugin system would not, i feel,
actually need endorsement *by* the OpenPOWER Foundation to go ahead:
that's more down to you, Alan.

we're quite happy to do the work (and have the funding from NLnet to do it).

btw one big question, i note that v3.1 prefixed instructions have started
to be added:

   https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=8acf14351c818d956babe50e61711740f378c941

we would need to work out how - in a *plugin* context - how to bring in
SVP64 which uses some of EXT001's [currently] reserved space.

and also how to reserve a known range of the opindex without the plugin
trampling all over the non-plugin (OPF-approved) area.  that's a lot
easier once opindex is 16-bit.

l.

Re: Plugin-based opcode table

[Alan Modra]

On Tue, May 17, 2022 at 03:01:48PM +0100, lkcl wrote:
> On May 17, 2022 1:37:36 PM GMT+01:00, Dmitry Selyutin
> <ghostmansd@gmail.com> wrote:
> 
> > We've been thinking of some plugin, with obvious candidates being
> >dlopen/dlsym combo. Sure the code will be public, as well as the whole
> >work around libresoc and SVP64.
> 
> ... but it would not be "endorsed" - implicitly and accidentally - by
> the binutils team making decisions that are at odds with the direction
> set by the OpenPOWER Foundation.
> 
> prior to approval yes-or-no by OPF, a plugin system would not, i feel,
> actually need endorsement *by* the OpenPOWER Foundation to go ahead:
> that's more down to you, Alan.

I don't see any need for the artificial boundaries you might get with
a plugin, whatever that might look like.  ppc-opc.c already supports
multiple incompatible powerpc cpu implementations, including ones that
depart significantly from current powerpc ISA documentation.  Your new
SVP64 design is not an orphan in that respect.  I do not want to see
"plugin" silliness out of fear of stepping on OPF's toes.

The fact that some cpu is supported in ppc-opc.c has no bearing on ISA
direction.  The binutils project doesn't have that power, nor should
it.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Plugin-based opcode table

[Dmitry Selyutin]

On Mon, May 23, 2022 at 3:59 PM Alan Modra <amodra@gmail.com> wrote:
> I don't see any need for the artificial boundaries you might get with
> a plugin, whatever that might look like.  ppc-opc.c already supports
> multiple incompatible powerpc cpu implementations, including ones that
> depart significantly from current powerpc ISA documentation.

Alan, please, correct me if I'm wrong: does it mean that we can just
stick to marking new instructions and facilities with a custom per-CPU
flag? For now I introduced the -mlibresoc CPU which enabled the
PPC_OPCODE_SVP64 flag, and marked some new instructions appropriately
with this flag. Would that be OK if we follow this approach?

Re: Plugin-based opcode table

[Alan Modra]

On Mon, May 23, 2022 at 04:06:17PM +0300, Dmitry Selyutin wrote:
> On Mon, May 23, 2022 at 3:59 PM Alan Modra <amodra@gmail.com> wrote:
> > I don't see any need for the artificial boundaries you might get with
> > a plugin, whatever that might look like.  ppc-opc.c already supports
> > multiple incompatible powerpc cpu implementations, including ones that
> > depart significantly from current powerpc ISA documentation.
> 
> Alan, please, correct me if I'm wrong: does it mean that we can just
> stick to marking new instructions and facilities with a custom per-CPU
> flag? For now I introduced the -mlibresoc CPU which enabled the
> PPC_OPCODE_SVP64 flag, and marked some new instructions appropriately
> with this flag. Would that be OK if we follow this approach?

Yes, that would be fine.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Plugin-based opcode table

[Dmitry Selyutin]

On Tue, May 24, 2022, 08:41 Alan Modra <amodra@gmail.com> wrote:

> Yes, that would be fine.
>

Thank you for the reply and patience, Alan! I'll stick to this approach
then.

Re: Plugin-based opcode table

[Peter Bergner]

On 5/23/22 8:06 AM, Dmitry Selyutin via Binutils wrote:
> On Mon, May 23, 2022 at 3:59 PM Alan Modra <amodra@gmail.com> wrote:
>> I don't see any need for the artificial boundaries you might get with
>> a plugin, whatever that might look like.  ppc-opc.c already supports
>> multiple incompatible powerpc cpu implementations, including ones that
>> depart significantly from current powerpc ISA documentation.
> 
> Alan, please, correct me if I'm wrong: does it mean that we can just
> stick to marking new instructions and facilities with a custom per-CPU
> flag? For now I introduced the -mlibresoc CPU which enabled the
> PPC_OPCODE_SVP64 flag, and marked some new instructions appropriately
> with this flag. Would that be OK if we follow this approach?

I'm just catching up on my email backlog after being on vacation last
week, but I agree with everything Alan said and also agree with how
you're going about things.

That said, I assume your -mlibresoc flag enables more than just the
PPC_OPCODE_SVP64 flag, since there is a lot of "base" POWER instructions
like addi, etc. that I'm sure your cpu implements.  If for some reason,
one of those "base" instructions enabled by -mlibresoc isn't implemented
by your cpu or conflicts in some way with your new instructions, you
can add the PPC_OPCODE_SVP64 to the deprecated field for the problematic
instruction to disable it when using -mlibresoc.

For example, POWER9 uses all the flags POWER8 enables plus PPC_OPCODE_POWER9.
However, the "lxvx" instruction changed between POWER8 (where it was an
extended mnemonic for lxvd2x) and POWER9 where is became a real instruction
with a different opcode than lxvd2x (I'm ignoring that doing that was
a very bad idea!).  How that was solved, was adding PPC_OPCODE_POWER9
to the old lxvx instruction's deprecated field to disable that on POWER9
and later cpus.  POWER9 and later cpus get the new lxvx instruction,
because we enable it with the PPC_OPCODE_POWER9.

Peter

Re: Plugin-based opcode table

[Luke Kenneth Casson Leighton]

On Sun, May 29, 2022 at 3:30 AM Peter Bergner <bergner@linux.ibm.com> wrote:

> I'm just catching up on my email backlog after being on vacation last
> week,

nice. i mean the vacation :)

> but I agree with everything Alan said and also agree with how
> you're going about things.

appreciated the reassurance.  i'm not going to get over being
nervous about adding to EXT01 (25% of v3.1 prefix space),
and to EXT019, 005 and 004, before all these have been
approved by the OPF ISA Working Group.

certainly, the 005 and 022 (sandbox) opcodes will *not* be
permanently in the places we'll initially submit them in.  004 we can
make a good case for madded and divmod2du [0] staying where
proposed.

still, the good news is that people can go "oh, er, so that's what
the Libre-SOC team plans to propose to add to Power ISA,
i didn't quite have time to work it out before".

> That said, I assume your -mlibresoc flag enables more than just the
> PPC_OPCODE_SVP64 flag, since there is a lot of "base" POWER instructions
> like addi, etc. that I'm sure your cpu implements.

yes, we aim for the SFFS (Scalar Fixed Floating Point Subset) subset,
described right at the very beginning of the Power ISA Spec (page viii)
and have confirmed with Mendy that it's fine to add whatever you like:
we therefore choose to add BE, AMO, TLBIE and a RADIX MMU in
order to run scalar Linux. following microwatt, basically [1]

keenly aware that a corresponding revival of runtime capability detection
is needed in glibc6 there (removed a few years ago because noone was
using it, because there was only POWER8/POWER9) but one step at
a time.

> If for some reason,
> one of those "base" instructions enabled by -mlibresoc isn't implemented
> by your cpu or conflicts in some way with your new instructions, you
> can add the PPC_OPCODE_SVP64 to the deprecated field for the problematic
> instruction to disable it when using -mlibresoc.

fortunately as an entirely from-scratch design with a focus on mass-volume
general-purpose compute we have no legacy instructions to support.

> For example, POWER9 uses all the flags POWER8 enables plus PPC_OPCODE_POWER9.
> However, the "lxvx" instruction changed between POWER8 (where it was an
> extended mnemonic for lxvd2x) and POWER9 where is became a real instruction
> with a different opcode than lxvd2x (I'm ignoring that doing that was
> a very bad idea!).

yowser that would have been painful. and a costly mistake that won't go
away for at least another decade [i'm aware that IBM still gets 3rd parties
asking to license POWER8, *for new designs*]

>  How that was solved, was adding PPC_OPCODE_POWER9
> to the old lxvx instruction's deprecated field to disable that on POWER9
> and later cpus.  POWER9 and later cpus get the new lxvx instruction,
> because we enable it with the PPC_OPCODE_POWER9.

ok. so this is good to know, because you know what it costs to have to
support a conflicting opcode, long-term.

now imagine that a binary program needs *both* the old *and* the
new behaviour of lxvx, in the same binary - in public usage. part
of upstream binutils, part of upstream glibc6, and so on.

a solution here is to set the PCR bit that flips back to POWER8
compatibility. if that's protected it would be necessary to have a
kernel syscall call to flip back and forth. performance would
suck, but at least runtime binary compatibility would be achieved
and people don't scream [except to hoot somewhat in derision
at such a terrible hack, but hey].

now imagine - even worse - that there's no PCR register bit that
can be deployed to flip back and forth between POWER8 and POWER9
compatiblity.

now imagine that this happens on two or more high-profile public uses of
the Sandbox (EXT022).  the spec said it was "perfectly ok". page xii

    "Facilities described in proposals that are not adopted
    into the architecture may be implemented as Custom Extensions
    using the architecture sandbox"

this is the nightmare scenario that i'd like to see avoided [understatement]

we've another meeting coming up, wed 01jun2022, i have an idea to
put in a proposal for some "pre-reservations" of opcode space as ISA
RFCs. my biggest concern is that there's no two-way communications
path to the IBM Engineers submitting instructions for future IBM
POWER 11/12 designs. we have no idea - at all - whether there's
already a conflict.

l.

[0] https://libre-soc.org/openpower/sv/biginteger/
[1] https://shenki.github.io/boot-linux-on-microwatt/

Re: Plugin-based opcode table

[Paul Mackerras]

Just to clarify something below...

On Sun, May 29, 2022 at 10:13:06AM +0100, Luke Kenneth Casson Leighton wrote:
> On Sun, May 29, 2022 at 3:30 AM Peter Bergner <bergner@linux.ibm.com> wrote:
[snip]
> > However, the "lxvx" instruction changed between POWER8 (where it was an
> > extended mnemonic for lxvd2x) and POWER9 where is became a real instruction
> > with a different opcode than lxvd2x (I'm ignoring that doing that was
> > a very bad idea!).
> 
> yowser that would have been painful. and a costly mistake that won't go
> away for at least another decade [i'm aware that IBM still gets 3rd parties
> asking to license POWER8, *for new designs*]
> 
> >  How that was solved, was adding PPC_OPCODE_POWER9
> > to the old lxvx instruction's deprecated field to disable that on POWER9
> > and later cpus.  POWER9 and later cpus get the new lxvx instruction,
> > because we enable it with the PPC_OPCODE_POWER9.
> 
> ok. so this is good to know, because you know what it costs to have to
> support a conflicting opcode, long-term.
> 
> now imagine that a binary program needs *both* the old *and* the
> new behaviour of lxvx, in the same binary - in public usage. part
> of upstream binutils, part of upstream glibc6, and so on.

The problem Peter was describing was at the assembly level not the
machine code level.  The lxvd2x instruction still exists in POWER9 and
does the same thing that it does in POWER8.  POWER9 added a new
instruction which took over the "lxvx" mnemonic.  In big-endian mode,
lxvx and lxvd2x do the same thing (in little-endian, they differ by a
double-word swap).

That is, the difference is in what bits you get out of the assembler
when you write "lxvx", rather than a binary compatibility difference.

Paul.

Re: Plugin-based opcode table

[lkcl]

On Tue, May 31, 2022 at 1:14 AM Paul Mackerras <paulus@ozlabs.org> wrote:
>
> Just to clarify something below...

always appreciated, Paul.

> The problem Peter was describing was at the assembly level not the
> machine code level.  The lxvd2x instruction still exists in POWER9 and
> does the same thing that it does in POWER8.

ahh right. so an older program compiled prior to the assembler mnemonic
rename will still be binary-compatible.

> That is, the difference is in what bits you get out of the assembler
> when you write "lxvx", rather than a binary compatibility difference.

subtle, and annoying/quirky, but not something that breaks older
programs, thank goodness. important to have clarified because it
would give the false impression (to any future reader of the archives)
of an instability in the Power ISA, which would be bad.

l.