ABI, I don't get it...

ABI, I don't get it...

[ANDY KENNEDY]

I've sent a couple of private messages to Maciej about this, so this
won't be a huge surprise to at least one on this list...

I am working with the Broadcom XLP processor.  We are number crunching
big-time, for which this processor was made.  The original port that we
found of binutils of this was from a few years ago (the link to which
can be found in another thread on this list) seems incomplete.  In the
hopes that this patch was completely upstreamed, I grabbed the latest
binutils, and the rest of the toolchain, the objective of which was to
get a fully working 64 bit ABI based toolchain for the XLP.  Turns out
that the added patch to binutils is not complete at all.  There is a
comment in one file that says something like "the XLP is just like the
XLR except that it is a mips64r2".  Unfortunately, this is incorrect
as there are at least a dozen instructions that aren't defined for the
XLR that are required for the XLP (things like -- for lack of a better
term -- callbacks for various operations).

So, after working with the new toolchain for about a week, and
attempting to port the changes (which, from the 2.24 release of binutils
conflicts with the changes put in for the Octeon3), I gave up and
reverted back to my 2.24 binutils.  I then found a patch from Maciej to
allow me to set the default ABI of the toolchain to 64, however, it
didn't seem to work.  Following the spirit of the patch, I made
additional changes to Gcc, (all of the support packages for binutils),
GDB, and eglibc.  Nothing has seemed to work.

Next, I focused in on my build system (which is a highly modified
version of an old BuildRoot -- everything stripped out of it an only our
proprietary stuff being built by a heavy modification of it -- really,
the only thing we kept was the configuration.  Having said that, there
were adaptations of the toolchain settings that we kept from BR.
Adjusting these values (selecting the ABI for the platform, soft/hard
float etc) has led me into a situation in which I get the following
error routinely:

ABI is incompatible with that of the selected emulation

This comes from the toolchain that I ACTUALLY GOT our IP to build with,
however, it core dumps immediately (as I expected given the linker
warning).

Another error, from a mixture that I cannot get to build correctly,
gives me the following:

/opt/toolchains/mips64-LinuxBSP-linux-gnu/mips64-LinuxBSP-linux-gnu/sysroot/usr/include/gnu/stubs.h:20:33: fatal error: gnu/stubs-n64_soft.h: No such file or directory

with a toolchain that was supposedly 64-bit native, hard-float.  Why
the floating point selection is soft, I cannot say.

If you have ANY idea that might help me on figuring this one out,
please provide a nugget.

Also, to answer another question from my other thread:  I have contacted
Broadcom about the original toolchain.  I have requested copyright
privileges from them, but have not received an answer on that one yet.
I get the sinking feeling that the FAE that recently left took with him
the only knowledge of how this processor works.

So, if I'm to get this working in the newer toolchain (assuming I get
permissions) I'll beg each of your help to do it the "right way".

Now, to be clear on what I'm asking in this e-mail:

1) Is there a way to determine the ABI calls being used in a particular
object file?

One thing I see from objdump is ".mdebug.abi64", but I see this in all
of the object files... which indicates to me that this is not a good
predictor.


 --- my take on this is to build three int main () { return 0 ;} object
files with -mabi={64,o32,n32} and then link EACH object file from our IP
against each of these, to show which ones are being linked in with the
wrong ABI.
1.a) Does it matter (speaking of ABI) on the hard/soft float?

2) How can I determine whether an object file is built using hard-float
ABI calls?  I know that the XLP has a HW FPU.

Thanks in advance for any help you can provide me,

Andy

Re: ABI, I don't get it...

[Paul_Koning]

> On Aug 12, 2016, at 6:46 PM, ANDY KENNEDY <ANDY.KENNEDY@adtran.com> wrote:
> 
> ...
> ABI is incompatible with that of the selected emulation

I've run into that message all too often.  I no longer remember what is needed to cure it.  It's a pain in the neck, because the message gives no information and the solution tends to be obscure.

> ...
> with a toolchain that was supposedly 64-bit native, hard-float.  Why
> the floating point selection is soft, I cannot say.

Perhaps because some other MIPS64 machines don't have float?  The Sibyte series comes to mind.  We use soft-float for that reason, even though XLR and XLP do have it.

> ...
> Also, to answer another question from my other thread:  I have contacted
> Broadcom about the original toolchain.  I have requested copyright
> privileges from them, but have not received an answer on that one yet.
> I get the sinking feeling that the FAE that recently left took with him
> the only knowledge of how this processor works.

That could be.  It's also possible that they don't understand open source well enough.

> So, if I'm to get this working in the newer toolchain (assuming I get
> permissions) I'll beg each of your help to do it the "right way".
> ...
> --- my take on this is to build three int main () { return 0 ;} object
> files with -mabi={64,o32,n32} and then link EACH object file from our IP
> against each of these, to show which ones are being linked in with the
> wrong ABI.
> 1.a) Does it matter (speaking of ABI) on the hard/soft float?

Yes, at least in the sense that the supporting libraries obviously have to be different.  So if you want to cover all bases  you are looking at 6 flavors (8 if you include O64, which isn't really worth the trouble).

> 2) How can I determine whether an object file is built using hard-float
> ABI calls?  I know that the XLP has a HW FPU.

I think it shows up in the 32-bit MIPS-specific flags word that objdump can display, but I don't remember the meaning of the bits.  If they don't match that may be the issue, in other words, aim for them to match across all the object files you're linking together.

	paul

Re: ABI, I don't get it...

[Maciej W. Rozycki]

On Sun, 14 Aug 2016, Paul_Koning@Dell.com wrote:

> > with a toolchain that was supposedly 64-bit native, hard-float.  Why
> > the floating point selection is soft, I cannot say.
> 
> Perhaps because some other MIPS64 machines don't have float?  The 
> Sibyte series comes to mind.  We use soft-float for that reason, even 
> though XLR and XLP do have it.

 FYI, the SiByte/Broadcom SB-1 processor cores do have an FPU albeit IIRC 
early (A0?) steppings had fatal errata making the unit less than usable 
(they also had other issues making them usable even less).  This may have 
been the reason you have chosen a software configuration which avoids the 
use of the FPU with these processors.  Later revisions were fine.  I've 
used the FPU with a B2 stepping SB-1 myself with no issues; there was a 
later C0 revision as well.

 Ralf (cc-ed) may remember a bit more about the story here.

 FWIW,

  Maciej

Re: ABI, I don't get it...

[Paul_Koning]

> On Aug 24, 2016, at 11:46 AM, Maciej W. Rozycki <macro@imgtec.com> wrote:
> 
> On Sun, 14 Aug 2016, Paul_Koning@Dell.com wrote:
> 
>>> with a toolchain that was supposedly 64-bit native, hard-float.  Why
>>> the floating point selection is soft, I cannot say.
>> 
>> Perhaps because some other MIPS64 machines don't have float?  The 
>> Sibyte series comes to mind.  We use soft-float for that reason, even 
>> though XLR and XLP do have it.
> 
> FYI, the SiByte/Broadcom SB-1 processor cores do have an FPU albeit IIRC 
> early (A0?) steppings had fatal errata making the unit less than usable 
> (they also had other issues making them usable even less).  This may have 
> been the reason you have chosen a software configuration which avoids the 
> use of the FPU with these processors.  Later revisions were fine.  I've 
> used the FPU with a B2 stepping SB-1 myself with no issues; there was a 
> later C0 revision as well.

It's been a while, but yes, that sounds familiar.  Part of the consideration was that we switched to soft-float for that reason, and it worked well enough for what we do that it wasn't interesting to switch back.

	paul

RE: ABI, I don't get it...

[ANDY KENNEDY]

> From: Maciej W. Rozycki [mailto:macro@imgtec.com]
> Subject: Re: ABI, I don't get it...
> 
> On Sun, 14 Aug 2016, Paul_Koning@Dell.com wrote:
> 
> > > with a toolchain that was supposedly 64-bit native, hard-float.  Why
> > > the floating point selection is soft, I cannot say.
> >
> > Perhaps because some other MIPS64 machines don't have float?  The
> > Sibyte series comes to mind.  We use soft-float for that reason, even
> > though XLR and XLP do have it.
> 
>  FYI, the SiByte/Broadcom SB-1 processor cores do have an FPU albeit IIRC
> early (A0?) steppings had fatal errata making the unit less than usable
> (they also had other issues making them usable even less).  This may have
> been the reason you have chosen a software configuration which avoids the
> use of the FPU with these processors.  Later revisions were fine.  I've
> used the FPU with a B2 stepping SB-1 myself with no issues; there was a
> later C0 revision as well.

Not sure what that processor is.  This one is the Broadcom/NetLogic XLP
which have fairly high-end gpu's as the FPU on each core.  Fairly slick
processor and supposedly a powerhouse number cruncher.

I would not think that the NetLogic flagship (the main draw being the FPU)
would have issues like this -- though, who knows!

Thanks for the info!

Andy

> 
>  Ralf (cc-ed) may remember a bit more about the story here.
> 
>  FWIW,
> 
>   Maciej

RE: ABI, I don't get it...

[Maciej W. Rozycki]

On Wed, 24 Aug 2016, ANDY KENNEDY wrote:

> Not sure what that processor is.  This one is the Broadcom/NetLogic XLP
> which have fairly high-end gpu's as the FPU on each core.  Fairly slick
> processor and supposedly a powerhouse number cruncher.

 See: 
<http://www.broadcom.com/products/Processors/Carrier-and-Service-Provider/BCM1250> 
for an SB-1 based SOC information.  It used to be a high-end networking 
embedded device of some 15-10 years ago and has been since long obsoleted.

  Maciej

Re: ABI, I don't get it...

[Andrew Pinski]

On Wed, Aug 24, 2016 at 8:46 AM, Maciej W. Rozycki <macro@imgtec.com> wrote:
> On Sun, 14 Aug 2016, Paul_Koning@Dell.com wrote:
>
>> > with a toolchain that was supposedly 64-bit native, hard-float.  Why
>> > the floating point selection is soft, I cannot say.
>>
>> Perhaps because some other MIPS64 machines don't have float?  The
>> Sibyte series comes to mind.  We use soft-float for that reason, even
>> though XLR and XLP do have it.

Also FYI. Octeon 1/+ and Octeon 2 do not have a FPU while Octeon 3
does have one.
Maybe most people now days are targeting the Octeon family :).

Thanks,
Andrew Pinski


>
>  FYI, the SiByte/Broadcom SB-1 processor cores do have an FPU albeit IIRC
> early (A0?) steppings had fatal errata making the unit less than usable
> (they also had other issues making them usable even less).  This may have
> been the reason you have chosen a software configuration which avoids the
> use of the FPU with these processors.  Later revisions were fine.  I've
> used the FPU with a B2 stepping SB-1 myself with no issues; there was a
> later C0 revision as well.
>
>  Ralf (cc-ed) may remember a bit more about the story here.
>
>  FWIW,
>
>   Maciej

Re: ABI, I don't get it...

[Maciej W. Rozycki]

On Fri, 12 Aug 2016, ANDY KENNEDY wrote:

> So, after working with the new toolchain for about a week, and
> attempting to port the changes (which, from the 2.24 release of binutils
> conflicts with the changes put in for the Octeon3), I gave up and
> reverted back to my 2.24 binutils.  I then found a patch from Maciej to
> allow me to set the default ABI of the toolchain to 64, however, it
> didn't seem to work.  Following the spirit of the patch, I made
> additional changes to Gcc, (all of the support packages for binutils),
> GDB, and eglibc.  Nothing has seemed to work.

 NB these days you can configure GCC with the default ABI of your choice, 
by using the `--with-abi=' option, e.g.:

$ /path/to/configure --target=mips64-linux-gnu --with-abi=64 ...

and the resulting compiler will then default to the n64 ABI.  The same ABI 
names are supported as with the `-mabi=' compiler option.  See the GCC 
manual and the GCC installation manual for further details.

 Therefore as long as you always use the GCC driver to run individual 
programs from within the toolchain, including GAS and LD (which is 
recommended), you don't need to modify the configuration of other pieces.  
The build process of GLIBC (which is preferred to EGLIBC these days, long 
unmaintained) will handle such a configuration normally, although the 
FHS-defined library paths will be used for your libraries, so e.g. 64-bit 
libraries will land in /lib64, etc.  That hopefully shouldn't be a 
problem.  If it is, then you'll have to look yourself into the guts of 
configury.

> Next, I focused in on my build system (which is a highly modified
> version of an old BuildRoot -- everything stripped out of it an only our
> proprietary stuff being built by a heavy modification of it -- really,
> the only thing we kept was the configuration.  Having said that, there
> were adaptations of the toolchain settings that we kept from BR.
> Adjusting these values (selecting the ABI for the platform, soft/hard
> float etc) has led me into a situation in which I get the following
> error routinely:
> 
> ABI is incompatible with that of the selected emulation

 This means that the linker emulation chosen with LD invocation cannot 
handle the input files presented.  With the MIPS ABIs the linker emulation 
selected with the `-m' option has to match the ABI of the input files.  
The usual selections for the Linux targets are: `elf32btsmip', 
`elf32btsmipn32' and `elf64btsmip' for the big-endian o32, n32 and n64 
ABIs respectively.  Substitute `lt' for `bt' for the little endianness.

 Run `ld --help' for the complete list of emulations configured.  The GCC 
driver is supposed to select the correct emulation on linker invocation, 
based on the ABI selection (so you need to use the same `-mabi=' driver 
option with linker invocation as you used with source compilation).

> Another error, from a mixture that I cannot get to build correctly,
> gives me the following:
> 
> /opt/toolchains/mips64-LinuxBSP-linux-gnu/mips64-LinuxBSP-linux-gnu/sysroot/usr/include/gnu/stubs.h:20:33: fatal error: gnu/stubs-n64_soft.h: No such file or directory
> 
> with a toolchain that was supposedly 64-bit native, hard-float.  Why
> the floating point selection is soft, I cannot say.

 I can't help you with that offhand, you might want to seek a GLIBC 
expert.  Perhaps you have a soft-float multilib inadvertently included in 
your configuration.  Depending on the exact GCC target chosen a different 
list of multilibs is configured which then affects the build process of 
other components.

 FYI, multilibs are library configuration variants covering different 
endiannesses, ABIs, ISAs, floating-point and other architecture 
peculiarities and the configured list for a given GCC binary can be listed 
with `gcc -print-multi-lib'.  Some may be incompatible with each other and 
some hardware, while some may just provide different optimisations.

> Now, to be clear on what I'm asking in this e-mail:
> 
> 1) Is there a way to determine the ABI calls being used in a particular
> object file?

 You'll get basic ABI/ISA information with the usual *nix `file' command. 
Experiment with `readelf' for further details, but beware that information 
given will be dense and may require some expertise to interpret.  The `-h' 
and `-A' options will be particularly relevant here.  You may ignore any 
and all the GOT information listed with the latter option.

> One thing I see from objdump is ".mdebug.abi64", but I see this in all
> of the object files... which indicates to me that this is not a good
> predictor.

 That itself actually indicates the n64 ABI, but you'll need to peek at 
the objects with `readelf' for further information.

>  --- my take on this is to build three int main () { return 0 ;} object
> files with -mabi={64,o32,n32} and then link EACH object file from our IP
> against each of these, to show which ones are being linked in with the
> wrong ABI.

 A 64-bit Linux toolchain will normally support all the three ABIs, with 
the default set according to `--with-abi=' as noted above.  For the 
remaining two you'll need to use `-mabi=' explicitly.

> 1.a) Does it matter (speaking of ABI) on the hard/soft float?

 It does, as the calling convention is different and therefore you can't 
link soft-float (`-msoft-float') objects and hard float (`-mhard-float') 
objects together.  Please note that there are further hard-float ABI 
variations, however with the XLP processor the compiler defaults will 
probably be right for you.

> 2) How can I determine whether an object file is built using hard-float
> ABI calls?  I know that the XLP has a HW FPU.

 The MIPS FPU is a part of the Linux user ABI, so you can always use hard 
float in user programs as the kernel will emulate the FPU if missing in 
hardware.  Use `readelf -A' to determine the floating-point convention of 
an ELF file; it will be printed at the top, e.g.:

Attribute Section: gnu
File Attributes
  Tag_GNU_MIPS_ABI_FP: Hard float (32-bit CPU, Any FPU)

Old versions of `readelf' won't print this information.

 HTH,

  Maciej

RE: ABI, I don't get it...

[Matthew Fortune]

Hi Andy,

Have you now got enough information to understand what is going on? If
not please say which aspects are still confusing and I will try and
offer any insight I can.

I have to say that the linker talking about emulations in error messages
is completely meaningless to many toolchain developers let alone end
users so those messages are borderline useless. That's not to say many
of the other ABI related error messages are much more helpful but some
hint that it is an 'ABI' issue is a good start as user's do know that
there are multiple ABIs in general I think.

Matthew

RE: ABI, I don't get it...

[ANDY KENNEDY]

> From: Matthew Fortune [mailto:Matthew.Fortune@imgtec.com]
> Subject: RE: ABI, I don't get it...
> 
> Hi Andy,
> 
> Have you now got enough information to understand what is going on? If
> not please say which aspects are still confusing and I will try and
> offer any insight I can.

Eh, sort of.  I had build flags set in my (yes, MY) build system that
I had forgotten about.  I don't have the toolchain building natively as
64 bit ABI, however, at this point it doesn't matter as I am controlling
it from my build system.  Felt pretty bad when I found that I was passing
-msoft-float and -mabit=n32 through my own build system.

> 
> I have to say that the linker talking about emulations in error messages
> is completely meaningless to many toolchain developers let alone end
> users so those messages are borderline useless. That's not to say many
> of the other ABI related error messages are much more helpful but some
> hint that it is an 'ABI' issue is a good start as user's do know that
> there are multiple ABIs in general I think.

It turned out to be both the flags above mixing up with the
way I wanted from another package.  That was soooo last week.  I've
put out so many fires since then :).

I really appreciate that you asked about it, though.  I should have
answered back to close the loop on my own question but I've been swamped
at home and work.  Sorry about that.

Thanks again!
Andy

> 
> Matthew
> 

RE: ABI, I don't get it...

[Maciej W. Rozycki]

On Wed, 24 Aug 2016, ANDY KENNEDY wrote:

> I really appreciate that you asked about it, though.  I should have
> answered back to close the loop on my own question but I've been swamped
> at home and work.  Sorry about that.

 I'm glad you made progress.  Good luck and feel free to ask again in the 
future.

  Maciej

RE: ABI, I don't get it...

[Maciej W. Rozycki]

On Wed, 24 Aug 2016, Matthew Fortune wrote:

> I have to say that the linker talking about emulations in error messages
> is completely meaningless to many toolchain developers let alone end
> users so those messages are borderline useless.

 Well, to be fair to binutils documentation maintainers I have to observe 
that the term "emulation" is used throughout the LD user manual, so there 
is at least a point of reference, although I do agree the definition 
itself presented there of what a linker emulation is is a bit lacking to 
say the least.

 You and anyone else are of course welcome to contribute documentation 
improvements; perhaps merging this old chapter: 
<http://ftp.gnu.org/old-gnu/Manuals/binutils-2.12/html_node/binutils_20.html>, 
which has been lost to history for some reason, will be a good way to 
start.

> That's not to say many
> of the other ABI related error messages are much more helpful but some
> hint that it is an 'ABI' issue is a good start as user's do know that
> there are multiple ABIs in general I think.

 You may actually only have support for a single ABI configured and still 
get the error message.  Although to be fair you won't get this message 
unless you mess up GCC driver options (which as a rule of thumb need to be 
the same throughout all the software build stages), invoke one part of the 
toolchain in the build process directly rather than via the GCC driver 
(which is not recommended unless you know what you're doing), or use 
binutils built from incorrectly modified sources.  Some or all of these 
aspects may have already been properly documented, beyond just being 
common GNU toolchain lore.

 In any case, again, both documentation and error reporting improvements 
are always welcome; sometimes being concise and precise both at a time -- 
as required for a good error message -- is quite a challenge.

 FWIW,

  Maciej