RE: Software Convention Proposal

RE: Software Convention Proposal

[Kirkegaard, Knud J]

I would like to change the command line options after seeing Richard's
response.
The original proposed command-line option with an optional argument does not
have a clean semantic and is difficult to parse.

Therefore using Richard's suggestion:

-fvisibility=<visibility> 

where visibility is one of: default, protected, hidden, or internal.
This option will, by default, cause all global symbols to get the ELF
visibility
specified by the option.

-fvisibility-default=<file>
-fvisibility-protected=<file>
-fvisibility-hidden=<file>
-fvisibility-internal=<file>

The options listed will all take a file as an argument. Space separated
symbols listed in
the file will get the visibility specified by the options. The precedence
order of 
the visibility attributes remains as described in the original proposal.


With respect to -fminshared I still feel it makes sense to have it imply 
-fvisibility=protected since it allows the compiler to generate absolute
addressing for
any symbols not explicitly marked default (either through using the
attribute syntax or
the -fvisibility-default=<file>). That in turn means symbol preemption is
not possible
and therefore it should imply that the default visibility is protected. It
is not
exactly the same as -fno-pic since the default is still PIC, just that most
of the
symbols can be addressed absolute since we guarantee it will be linked into
the main
executable image. If due diligence is not applied the worst case should be a
link time
error when a symbol marked protected turns out to be resolved from a shared
library.

Regarding architectures that will benefit from this besides IPF and the ones
listed in other responses, we can add that PA-RISC already benefits from
this. Otherwise any architecture using ELF and having PIC and symbol
preemption as the default behavior should benefit.


- Knud

Re: Software Convention Proposal

[Richard Henderson]

On Fri, Aug 23, 2002 at 02:47:04PM -0700, Kirkegaard, Knud J wrote:
> It is not exactly the same as -fno-pic since the default is still PIC...

This is not actually correct.  For gcc, the presence or abscense of
-fpic on the command line *does* change the symbol resoution rules
used inside the compiler.  It is already the case that shared libraries
must be compiled with -fpic or you'll get link errors of the form

  file.o: @gprel relocation against dynamic symbol foo

Thus, from my point of view, your -fminshared is exactly the same as
-fvisibility=protected and without -fpic.



r~

Re: Software Convention Proposal

[Richard Henderson]

On Thu, Sep 05, 2002 at 11:18:14AM -0700, Sehr, David C wrote:
> If I understand what you are saying about building a shared library,
> then without -fpic the following global symbol visibility rules apply:
> 1) if a symbol is defined, it is treated as protected (and made gprel if
> short)
> 2) if a symbol is extern (or, I presume, common), it is treated as default

Correct.

> The error from the linker then comes from having an extern reference
> to a defined symbol in another .o.

Not exactly.  The symbol might not be defined anywhere else.  But
if the symbol appears in the dynamic symbol table, we'll generate
an error because we cannot (and don't want to) represent the gprel
relocation in the dynamic relocation table.


r~

RE: Software Convention Proposal

[Sehr, David C]

Richard,

Sorry for the long latency.  I'm a little confused by your example,
and haven't quite been able to find the right man page to answer my
question.

If I understand what you are saying about building a shared library,
then without -fpic the following global symbol visibility rules apply:
1) if a symbol is defined, it is treated as protected (and made gprel if
short)
2) if a symbol is extern (or, I presume, common), it is treated as default
The error from the linker then comes from having an extern reference
to a defined symbol in another .o.  Is this what you were saying?
Adding -fvisibility=protected then alters this to minshared by
turning the extern symbol to protected (and hence gprel).

Thanks for the clarification.

Best regards,

David
  
-----Original Message-----
From: Richard Henderson [mailto:rth@redhat.com]
Sent: Wednesday, August 28, 2002 9:54 AM
To: Kirkegaard, Knud J
Cc: 'gcc@gcc.gnu.org'; Sehr, David C; Saxena, Sunil; 'cary@cup.hp.com';
Winalski, Paul
Subject: Re: Software Convention Proposal


On Fri, Aug 23, 2002 at 02:47:04PM -0700, Kirkegaard, Knud J wrote:
> It is not exactly the same as -fno-pic since the default is still PIC...

This is not actually correct.  For gcc, the presence or abscense of
-fpic on the command line *does* change the symbol resoution rules
used inside the compiler.  It is already the case that shared libraries
must be compiled with -fpic or you'll get link errors of the form

  file.o: @gprel relocation against dynamic symbol foo

Thus, from my point of view, your -fminshared is exactly the same as
-fvisibility=protected and without -fpic.



r~

Re: Software Convention Proposal

[Richard Henderson]

On Wed, Aug 21, 2002 at 10:08:48PM +0200, Jakub Jelinek wrote:
> How is this different from just sourcing another header with
> #define H(x) extern __typeof(x) x __attribute__((visibility("hidden")));
> H(foo)
> H(bar)
> H(baz)

I suppose in theory it isn't.  The biggest difference is being
able to enumerate the symbols that you _do_ want exported rather
than the symbols you _don't_.


r~

Re: Software Convention Proposal

[Richard Henderson]

On Wed, Aug 21, 2002 at 09:19:42PM +0200, Andreas Jaeger wrote:
> So, for what architectures will this proposal bring any benefit?  Any
> other archs besides ia64?

Alpha, certainly (32-bit immediate gp-relative relocs vs a got load).
PPC64, probably.  MIPS, probably, though I'm not certain that they 
have the relocations to make proper use of the info.


r~

RE: Software Convention Proposal

[Kirkegaard, Knud J]

Thanks to Richard for the clarification of the rationale.

We should also have acknowledged Richard Henderson (Redhat), Cary Coutant
(HP) for valuable input and inspiration to the proposal. 

Once we review the feedback we will send out a response and the proposal
updated as
appropriate.

Knud

-----Original Message-----
From: Richard Henderson [ mailto:rth@redhat.com ]
Sent: Wednesday, August 21, 2002 12:04 PM
To: Kirkegaard, Knud J
Cc: 'gcc@gcc.gnu.org'; Sehr, David C; Saxena, Sunil
Subject: Re: Software Convention Proposal


On Tue, Aug 20, 2002 at 03:31:06PM -0700, Kirkegaard, Knud J wrote:
> This is a proposal to take advantage of the performance opportunities
> available to several applications that don't require position independent
> code and the default symbol preemption model described in the ELF ABI.

It would have been helpful if you'd included some rationale.
Since I met with yall to discuss this, I'll summarize quickly:

IA-64, by default, uses PIC data references.  This avoids the
horror of the .dynbss hack used on x86 (and others) to handle
references from the main application to data in shared libraries.
However, this does introduce unnecessary overhead when referencing
data that does in fact reside in the main application.

We do, at present, use gp-relative relocations to the small-data
area when we can show that the symbol must bind locally; this can
be extended trivially to use "movl" to load absolute addresses 
for symbols outside the small-data area, but we must still
restrict this to symbols know to be bound locally.

  [ As a quick data point, Intel prototyped something like
    this in their compiler and got (iirc) a 10-15% improvement
    for a particular application.  ]

So the object of the game here is to extend the set of symbols
that we know binds locally.

With one's own application, it's possible (though not always
trivial, given the size of some applications) to modify all
declarations to include the visibility attribute.

With some buy-in from the OS vendor, it's possible to get the
system headers marked such that if the default binding were
changed on the compiler command line, the system libraries
would continue to bind correctly.

A problem area, however, are third-party libraries.  In that
case, given that we cannot modify the headers, the best we can
do is create a list of the symbols that should be bound a
particular way and present it to the compiler through some side
channel.  E.g. a file associated with a command line switch.

>  -fprotected -fdefault <file> 

I'm not thrilled about using the same switch to mean two
different things.  Certainly you couldn't just leave the
filename unassociated like that.  So at minimum you'd need

	-fprotected -fdefault=<file>

But I'd prefer switches that differed slightly.  Perhaps

	-fvisibility=protected
	-fvisibility-default=<file>

I'd like input from others on this.

> Compiling for "main program" option
> -------------------------------------
> 
> -fminshared
> 
>  This option indicates to the compiler that the module is being compiled
>  for the main program. This allows for absolute addressing of symbols that
>  are protected or hidden.

Err, this is what the lack of -fpic implies.

>  The option implies -fprotected.

This doesn't seem like a good idea.  If you've not done the due dilligence
above, how do you know your system libraries are properly marked?



r~

Re: Software Convention Proposal

[Jakub Jelinek]

On Wed, Aug 21, 2002 at 12:03:34PM -0700, Richard Henderson wrote:
> A problem area, however, are third-party libraries.  In that
> case, given that we cannot modify the headers, the best we can
> do is create a list of the symbols that should be bound a
> particular way and present it to the compiler through some side
> channel.  E.g. a file associated with a command line switch.
> 
> >  -fprotected -fdefault <file> 
> 
> I'm not thrilled about using the same switch to mean two
> different things.  Certainly you couldn't just leave the
> filename unassociated like that.  So at minimum you'd need
> 
> 	-fprotected -fdefault=<file>
> 
> But I'd prefer switches that differed slightly.  Perhaps
> 
> 	-fvisibility=protected
> 	-fvisibility-default=<file>
> 
> I'd like input from others on this.

How is this different from just sourcing another header with
#define H(x) extern __typeof(x) x __attribute__((visibility("hidden")));
H(foo)
H(bar)
H(baz)
? The only difference I can think of is if the headers then define
macros with the same names as the functions...

	Jakub

Re: Software Convention Proposal

[David Edelsohn]

>>>>> Andreas Jaeger writes:

Andreas> So, for what architectures will this proposal bring any benefit?  Any
Andreas> other archs besides ia64?

	Probably 64-bit PowerPC Linux and AIX.

David

Re: Software Convention Proposal

[Andreas Jaeger]

Richard Henderson <rth@redhat.com> writes:

> On Tue, Aug 20, 2002 at 03:31:06PM -0700, Kirkegaard, Knud J wrote:
>> This is a proposal to take advantage of the performance opportunities
>> available to several applications that don't require position independent
>> code and the default symbol preemption model described in the ELF ABI.
>
> It would have been helpful if you'd included some rationale.
> Since I met with yall to discuss this, I'll summarize quickly:
>
> IA-64, by default, uses PIC data references.  This avoids the
> horror of the .dynbss hack used on x86 (and others) to handle
> references from the main application to data in shared libraries.
> However, this does introduce unnecessary overhead when referencing
> data that does in fact reside in the main application.

So, for what architectures will this proposal bring any benefit?  Any
other archs besides ia64?

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj

Re: Software Convention Proposal

[Richard Henderson]

On Tue, Aug 20, 2002 at 03:31:06PM -0700, Kirkegaard, Knud J wrote:
> This is a proposal to take advantage of the performance opportunities
> available to several applications that don't require position independent
> code and the default symbol preemption model described in the ELF ABI.

It would have been helpful if you'd included some rationale.
Since I met with yall to discuss this, I'll summarize quickly:

IA-64, by default, uses PIC data references.  This avoids the
horror of the .dynbss hack used on x86 (and others) to handle
references from the main application to data in shared libraries.
However, this does introduce unnecessary overhead when referencing
data that does in fact reside in the main application.

We do, at present, use gp-relative relocations to the small-data
area when we can show that the symbol must bind locally; this can
be extended trivially to use "movl" to load absolute addresses 
for symbols outside the small-data area, but we must still
restrict this to symbols know to be bound locally.

  [ As a quick data point, Intel prototyped something like
    this in their compiler and got (iirc) a 10-15% improvement
    for a particular application.  ]

So the object of the game here is to extend the set of symbols
that we know binds locally.

With one's own application, it's possible (though not always
trivial, given the size of some applications) to modify all
declarations to include the visibility attribute.

With some buy-in from the OS vendor, it's possible to get the
system headers marked such that if the default binding were
changed on the compiler command line, the system libraries
would continue to bind correctly.

A problem area, however, are third-party libraries.  In that
case, given that we cannot modify the headers, the best we can
do is create a list of the symbols that should be bound a
particular way and present it to the compiler through some side
channel.  E.g. a file associated with a command line switch.

>  -fprotected -fdefault <file> 

I'm not thrilled about using the same switch to mean two
different things.  Certainly you couldn't just leave the
filename unassociated like that.  So at minimum you'd need

	-fprotected -fdefault=<file>

But I'd prefer switches that differed slightly.  Perhaps

	-fvisibility=protected
	-fvisibility-default=<file>

I'd like input from others on this.

> Compiling for "main program" option
> -------------------------------------
> 
> -fminshared
> 
>  This option indicates to the compiler that the module is being compiled
>  for the main program. This allows for absolute addressing of symbols that
>  are protected or hidden.

Err, this is what the lack of -fpic implies.

>  The option implies -fprotected.

This doesn't seem like a good idea.  If you've not done the due dilligence
above, how do you know your system libraries are properly marked?



r~

Software Convention Proposal

[Kirkegaard, Knud J]

Proposal for Linux Software Convention Options. 
Knud J Kirkegaard, Intel Corporation
David C Sehr, Intel Corporation

Introduction
------------

This is a proposal to take advantage of the performance opportunities
available to several applications that don't require position independent
code and the default symbol preemption model described in the ELF ABI.

The intent is to maintain binary compatibility by using existing ELF gABI
symbol visibility attributes as well as options to the compiler that can
change the default software convention used. The linker will report errors
if an attempt is made to link objects that have incompatible visibility
attributes on symbols.


ELF Symbol Visibility Attribute 
-------------------------------

Currently gcc supports a function and variable attribute corresponding to
the ELF symbol visibility attributes protected, hidden, and internal. These
in
turn correspond to the ELB ABI symbol visibility attributes STV_PROTECTED,
STV_HIDDEN, and STV_INTERNAL (see Appendix).

We propose to add the visibility attribute "default" corresponding to
STV_DEFAULT.  The idea is that through compiler command line options it
will be possible to change the default symbol visibility and then set the
"default" attribute on functions and variables that require the
"default" setting. 

 Example:

   int i __attribute__ ((visibility ("default")));
   void __attribute__ ((visibility ("default"))) foo () {  }


In addition to providing a source syntax to set the complete range of ELF
visibility attributes it is proposed to add compiler options to provide
command line control of the visibility attributes. The will enable immediate
access to the feature and not depend on header file modifications.


-fdefault   [<filename>]
-fprotected [<filename>]
-fhidden    [<filename>]
-finternal  [<filename>]

 The options will cause all global symbols to get the visibility specified
 by the option. If no file is specified as argument to the option all
symbols
 will get the specified attribute. If a file argument is specified, then
only
 space separated symbols listed in the file will get the visibility
attribute
 specified.

 Explicitly setting the visibility to "default", either using the attribute
 syntax or the command line option, to "default" will override any setting
 to "protected", "hidden", or "internal"

 Explicitly setting the visibility to "protected" will override any setting
 to "hidden", or "internal"

 Explicitly setting the visibility to "hidden" will override any setting
 to "internal"

 Since "internal" is a processor-specific attribute is may not be desirable
 to have a general option for it. This proposal does not depend on support
 for -finternal. It is included for completeness.


 Example:

 -fprotected -fdefault <file> 

 All symbols will be protected except symbols listed in the file argument
 to the fdefault option or symbols having the "default" visibility attribute
 set in the source.



Compiling for "main program" option
-------------------------------------

-fminshared

 This option indicates to the compiler that the module is being compiled
 for the main program. This allows for absolute addressing of symbols that
 are protected or hidden. The option implies -fprotected. Use of this option
 will allow the compiler to generate non-position independent code and 
 therefore use absolute addressing.



libc Header File Changes
------------------------

To increase the acceptance of the options to change software conventions
it is proposed that the glibc headers are modified to include the "default"
attribute on every extern symbol declaration. Applications that rely only
on using glibc as a shared library will not need to specify any symbol lists
to -fdefault to get the performance benefits.

It is recommended that other shared libraries also add the "default"
attribute
to the header files. Third party library providers would be encouraged to
add the "default" attribute as well.

It is proposed that the setting of the "default" attribute for exported 
data and functions can be overridden through the definition of a macro or
similar control.  Since the default behavior is dynamic linking an extra
macro definition or similar control is only needed to override the
"default" attribute, if an application wants to take full advantage of the
performance opportunities of statically linking in glibc.

Example:

The printf prototype will be:

  extern int __VISIBILITY printf (__const char *__restrict __format, ...)
__THROW;

By default the prototype for printf when including stdio.h will be expanded
to:

  extern int __attribute__ ((visibility ("default"))) printf (__const char
*__restrict __format, ...) __THROW;

When static behavior is desired the prototype will be expanded to:

  extern int printf (__const char *__restrict __format, ...) __THROW;



APPENDIX
--------

From the GCC documentation on current development:

Not all target machines support this attribute.

visibility ("visibility_type") 
The visibility attribute on ELF targets causes the declaration to be emitted
with hidden, protected or internal visibility. 

void __attribute__ ((visibility ("protected")))
f () { /* Do something. */; }
int i __attribute__ ((visibility ("hidden")));

See the ELF gABI for complete details, but the short story is 

hidden 
Hidden visibility indicates that the symbol will not be placed into the
dynamic symbol table, so no other module (executable or shared library) can
reference it directly. 

protected 
Protected visibility indicates that the symbol will be placed in the dynamic
symbol table, but that references within the defining module will bind to
the local symbol. That is, the symbol cannot be overridden by another
module. 

internal 
Internal visibility is like hidden visibility, but with additional processor
specific semantics. Unless otherwise specified by the psABI, gcc defines
internal visibility to mean that the function is never called from another
module. Note that hidden symbols, while then cannot be referenced directly
by other modules, can be referenced indirectly via function pointers. By
indicating that a symbol cannot be called from outside the module, gcc may
for instance omit the load of a PIC register since it is known that the
calling function loaded the correct value. 
Not all ELF targets support this attribute. 





----------------------------------------------------------------------------
--------------------------
Knud Joergen Bjerre Kirkegaard	phone	(408) 765 5069
Intel Corporation			fax:	(408) 765 5165
3600 Juliette Lane		email:	Knud.J.Kirkegaard@intel.com
Santa Clara, CA 95052-8119	mailstop: SC12-301
----------------------------------------------------------------------------
--------------------------