GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[Darryl L. Miles]

Is there a document or standard (or group of standards) that define the 
collective ABIs of GNU/Linux systems using ELF binary formats of various 
CPU architectures, including at least:
  IA32 (i386/i686/AMD64/EMT64/etc...)
  ARM (v5, v5t, v7, etc...)

What is the policy of the GNU toolchain, does it attempt to support a 
super-set of features where code was contributed but not directly 
set/enforce policy itself.  This being a matter for a distribution 
creator to establish.



My concern comes from what looks to be non-backwards compatible changes 
being made by the MeeGo distribution of Linux.  Specifically the 
enablement of SSSE3 IA32 instructions for "general purpose code 
generation", one possible motive for this is that a particular hardware 
vendor can claim in marketing the platform is optimized for XYZ 
technology/product, the thing they are actively trying to sell of the 
day.  This doesn't necessarily make for a good engineering choice.

While I accept that any distribution can do what they want, given the 
choice and resources I might wish to rebuild the entire open source 
project based around a better set of rules over this matter, since I 
have an interest in using the software but being a developer want the 
least amount of headaches looking into the future.  I also wish to 
achieve the goals to be able to provide a fully native SSSE3 optimized 
complete system but want other systems to behave in a repeatable and 
consistent way when binaries ultimately end up shared across them 
(desktop Linux).



Does GCC support use of newer CPU instructions for "general purpose code 
generation" ?  If so what kind of situations might they get selected for 
use ?  It is possible the situation is being misinterpreted by me, if 
GCC can not actually schedule newer instructions in code generation.



So my next question is what support is there in the various formats, 
technologies and runtime libraries to provide a backwards compatible 
solution, such that a binary from one system when put on another can 
have any hardware incompatibilities detected at the soonest opportunity, 
for example upon execution, soon after execution, during DSO loading:

1) ELF magic or hwcap (this would allow the kernel to error during the 
exec() system call, knowing that the format is not supported by the 
system).  DSOs loaded would also error with the same checking.  Ideally 
a hwcap system ideally doesn't want to be a rigid bitmask but some kind 
of extensible ASN.1 system where anyone can register for their own 
hierarchical domain and assign whatever they want from within it.

2) Use of a bespoke/custom Dynamic Linker path, I would guess any system 
doing this would be free to implement an alternative ABI.  Mixing 
binaries between systems would result in them not working due to lack of 
Dynamic Linker at that path.

3) Do the ABIs directly discuss of explain how such matters should be 
addressed, i.e. by guarding the execution of new CPU instructions by 
runtime checking.  This might mean whole optimized DSOs are loaded 
instead, or it might mean bunch of symbols would be redirected to 
optimized code within the same DSO, or it might mean an inline change of 
the flow of execution.  While I understand ABIs maybe 
open/loose/ambigious to allow new technologies and ideas to exist when 
known interoperability problems appear then guidelines should exist that 
provide technical answers to guide implementers so that good citizenship 
may follow.

4) Use of some ELF section to describe additional runtime checking rules.



The problem stems from hit and miss users getting SIGILL due to use of 
unguarded IA32 instructions being executed on non compatible (older) 
CPUs.  The kernel doesn't provide any trap and emulation, so the general 
purpose applications abort resulting is possible data loss.  Do 
guidelines exist within the GNU/Linux ABI on how to be a "good citizen" 
and help systems differentiate incompatible binaries so they simply 
don't run instead of causing a SIGILL potentially some months after 
execution started, because over the entire executable only a tiny 
handful of these instructions got selected by the compiler for use and 
that code didn't get run for a long time after the executable started.



The next matter is has anyone done any studies on the performance 
difference when enablement of newer instructions is possible for 
"general purpose code generation".  I'm not so interested in specialized 
use cases such as codecs, compression, encryption, graphics, etc... I 
consider these specialized use cases for which many applications and 
libraries already have a workable solution by "guarding" the execution 
of instructions that optimize such algorithms by checking the CPU 
runtime support.  I'm interested in the facts on how much benefit 
regular code gets from this choice.

Thanks,

Darryl

Re: GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[Jan Hubicka]

>
> Is there a document or standard (or group of standards) that define the  
> collective ABIs of GNU/Linux systems using ELF binary formats of various  
> CPU architectures, including at least:
>  IA32 (i386/i686/AMD64/EMT64/etc...)
>  ARM (v5, v5t, v7, etc...)

x86_64 ABI is at www.x86-64.org/documentation/abi.pdf
You can find similar processor supplements for other architectures.

> My concern comes from what looks to be non-backwards compatible changes  
> being made by the MeeGo distribution of Linux.  Specifically the  
> enablement of SSSE3 IA32 instructions for "general purpose code  
> generation", one possible motive for this is that a particular hardware  

GCC has -march switch specifying what ISA you want to generate for.
By default, GCC won't use SSSE3 or other extensions on i386 nor x86_64
target, but distribution, at its own choice, might change the default.
When configured to use extended ISA, GCC is trying to autogenerate new instructions
in generic code.  It naturally depends on type of instructions: many of new SSE
instructions are very exotic and not fitting generic codegen and are available
only via special builtins.

Honza

Re: GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[Paul Brook]

> So my next question is what support is there in the various formats,
> technologies and runtime libraries to provide a backwards compatible
> solution, such that a binary from one system when put on another can
> have any hardware incompatibilities detected at the soonest opportunity,
> for example upon execution, soon after execution, during DSO loading:

You can configure GCC to default to whatever ISA you like.  Unless you (the 
person building GCC) specified otherwise, it will usually choose something 
fairly conservative.

If you have been given a toolchain or binaries that is not suitable for you 
system, then that is something you should probably take up with whoever 
supplied them.

My understanding is that glibc has limited facilites for picking different set 
of libraries at runtime, but will not disgnose mismatches in a graceful 
fasion.  Either way this is probably an assembler/linker problem rather than a 
compiler problem. GNU as/ld do support thin kind of object file annotation, 
though the level of support varies between targets.

Paul

Re: GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[Ian Lance Taylor]

"Darryl L. Miles" <darryl-mailinglists@netbauds.net> writes:

> So my next question is what support is there in the various formats,
> technologies and runtime libraries to provide a backwards compatible
> solution, such that a binary from one system when put on another can
> have any hardware incompatibilities detected at the soonest
> opportunity, for example upon execution, soon after execution, during
> DSO loading:

Different architectures have adopted different approaches here.  I
believe the most common one is to record the required architecture in
the e_flags field of the ELF header.  ARM has adoped a more baroque
solution involving a special attributes section.  Either way, the linker
combines this information such that the resulting executable is marked
for the required architecture.  The kernel or dynamic linker may then
check the architecture, and give an error if the system does not support
it.

Unfortunately, as far as I know, no such solution was ever adopted for
the x86 family.  So this does not help your immediate problem, and will
not help it until somebody implements such an approach for x86.


> The next matter is has anyone done any studies on the performance
> difference when enablement of newer instructions is possible for
> "general purpose code generation".  I'm not so interested in
> specialized use cases such as codecs, compression, encryption,
> graphics, etc... I consider these specialized use cases for which many
> applications and libraries already have a workable solution by
> "guarding" the execution of instructions that optimize such algorithms
> by checking the CPU runtime support.  I'm interested in the facts on
> how much benefit regular code gets from this choice.

I don't know of any studies, but it's clear that automatic vectorization
using the SSE instructions can help a range of different programs.

Ian

Re: GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[Rainer Orth]

Ian Lance Taylor <iant@google.com> writes:

> Unfortunately, as far as I know, no such solution was ever adopted for
> the x86 family.  So this does not help your immediate problem, and will
> not help it until somebody implements such an approach for x86.

The Sun assembler, linker, and runtime linker implement this, cf.

	http://docs.sun.com/app/docs/doc/819-0690/chapter7-28?l=en&a=view

Hope this helps.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[Ian Lance Taylor]

Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> writes:

> Ian Lance Taylor <iant@google.com> writes:
>
>> Unfortunately, as far as I know, no such solution was ever adopted for
>> the x86 family.  So this does not help your immediate problem, and will
>> not help it until somebody implements such an approach for x86.
>
> The Sun assembler, linker, and runtime linker implement this, cf.
>
> 	http://docs.sun.com/app/docs/doc/819-0690/chapter7-28?l=en&a=view
>
> Hope this helps.

Sounds good to me.  Anybody want to add this to the GNU tools?

Ian

Re: GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[H.J. Lu]

On Tue, Jul 13, 2010 at 4:26 AM, Ian Lance Taylor <iant@google.com> wrote:
> Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> writes:
>
>> Ian Lance Taylor <iant@google.com> writes:
>>
>>> Unfortunately, as far as I know, no such solution was ever adopted for
>>> the x86 family.  So this does not help your immediate problem, and will
>>> not help it until somebody implements such an approach for x86.
>>
>> The Sun assembler, linker, and runtime linker implement this, cf.
>>
>>       http://docs.sun.com/app/docs/doc/819-0690/chapter7-28?l=en&a=view
>>
>> Hope this helps.
>
> Sounds good to me.  Anybody want to add this to the GNU tools?
>

Not very interesting to Linux. We don't use hardware capability filters.
However, it is nice to mark a binary which ISAs it uses. But it should
be specified in the psABI.

-- 
H.J.

Re: GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[H.J. Lu]

On Tue, Jul 13, 2010 at 6:39 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Tue, Jul 13, 2010 at 4:26 AM, Ian Lance Taylor <iant@google.com> wrote:
>> Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> writes:
>>
>>> Ian Lance Taylor <iant@google.com> writes:
>>>
>>>> Unfortunately, as far as I know, no such solution was ever adopted for
>>>> the x86 family.  So this does not help your immediate problem, and will
>>>> not help it until somebody implements such an approach for x86.
>>>
>>> The Sun assembler, linker, and runtime linker implement this, cf.
>>>
>>>       http://docs.sun.com/app/docs/doc/819-0690/chapter7-28?l=en&a=view
>>>
>>> Hope this helps.
>>
>> Sounds good to me.  Anybody want to add this to the GNU tools?
>>
>
> Not very interesting to Linux. We don't use hardware capability filters.
> However, it is nice to mark a binary which ISAs it uses. But it should
> be specified in the psABI.
>

On x86, it is OK for binary to have ISAs beyond the base ISA
as long as proper ISA check is used before the new ISA code is
entered. However, we should have a way to mark the base ISA.


-- 
H.J.

Re: GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[Rainer Orth]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On x86, it is OK for binary to have ISAs beyond the base ISA
> as long as proper ISA check is used before the new ISA code is
> entered. However, we should have a way to mark the base ISA.

That's what the Sun assembler does by default.  If you need to override
its findings if e.g. runtime checks are in place, you can do so with
linker maps, see gcc.target/i386/clearcap.map for an example.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: GNU/Linux ABI documentation ? GCC supports SSSE3 in general purpose code generation ?

[Paul Brook]

> Ian Lance Taylor <iant@google.com> writes:
> > Unfortunately, as far as I know, no such solution was ever adopted for
> > the x86 family.  So this does not help your immediate problem, and will
> > not help it until somebody implements such an approach for x86.
> 
> The Sun assembler, linker, and runtime linker implement this, cf.
> 
> 	http://docs.sun.com/app/docs/doc/819-0690/chapter7-28?l=en&a=view
> 
> Hope this helps.

Better IMO would be to implement GNU object attributes.

The assembler and linker already have all the framework for these, you just 
need to define an appropriate set of attributes. Their design is based on the 
ARM EABI object attributes, which provide a more complete example of how they 
can be used.

While a simple bitmask may be sufficient for the runtime linker, it's often 
useful to have a more flexible system for relocatable objects.

Paul