RFC: Add STB_GNU_SECONDARY

RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

Hi,

We have a need to define a secondary symbol as backup in
case there isn't a primary one.  Here is a proposal for
STB_GNU_SECONDARY.  Any comments?

Thanks.

-- 
H.J.
---

STB_GNU_SECONDARY

      Secondary symbols are similar to weak symbols, but their definitions
      have even lower precedence.  Secondary symbols can only appear in a
      relocatable object.  They must be either removed or converted to
      global or local symbols once it has become part of an executable or
      shared object.  The difference between secondary symbols and weak
      symbols are

	1.  When the link editor searches an archive library, it must
	extracts archive members that contain the global, weak or common
	definition of the secondary symbol with the same name and ignore
	the secondary one.
	2.  When the link editor searches a shared object, it must honor
	the global or weak definition in the shared object and ignore the
	secondary one with the same name.
	3. The link editor ignores the secondary definition if there is
	a global, weak or common definition with the same name.

      The purpose of this symbol binding is to provide the primary
      definition as a global, weak or common symbol in an archive library
      or a shared object while keeping a secondary definition in a
      relocatable object.  If there is no primary definition, the
      secondary definition will be used.

STB_GNU_SECONDARY is defined in OS-specific range:

#define STB_LOOS		10	/* OS-specific semantics */
#define STB_GNU_UNIQUE		10	/* Symbol is unique in namespace */
#define STB_GNU_SECONDARY	11	/* Secondary symbol */
#define STB_LOOS		12	/* OS-specific semantics */

Re: RFC: Add STB_GNU_SECONDARY

[Roland McGrath]

Please provide an example that illustrates why you think you need this.


Thanks,
Roland

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 12:50 PM, Roland McGrath <roland@hack.frob.com> wrote:
> Please provide an example that illustrates why you think you need this.
>

Currently we use weak undefined symbol, foo, to do

if (&foo != 0)
 foo is defined.
else
 foo isn't defined.

We want is to define foo as a secondary symbol so that
we can always use foo without checking.  If there is a primary
one in a .o file and .so file, we will get the primary one,
otherwise, we will use the secondary one.


-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[Roland McGrath]

> Currently we use weak undefined symbol, foo, to do
> 
> if (&foo != 0)
>  foo is defined.
> else
>  foo isn't defined.
> 
> We want is to define foo as a secondary symbol so that
> we can always use foo without checking.  If there is a primary
> one in a .o file and .so file, we will get the primary one,
> otherwise, we will use the secondary one.

Why not use a weak definition in the file where you make the call?

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 1:26 PM, Roland McGrath <roland@hack.frob.com> wrote:
>> Currently we use weak undefined symbol, foo, to do
>>
>> if (&foo != 0)
>>  foo is defined.
>> else
>>  foo isn't defined.
>>
>> We want is to define foo as a secondary symbol so that
>> we can always use foo without checking.  If there is a primary
>> one in a .o file and .so file, we will get the primary one,
>> otherwise, we will use the secondary one.
>
> Why not use a weak definition in the file where you make the call?

It doesn't work for us since a weak definition can't be overridden
by another definition in a .so file.

-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[Petr Baudis]

On Fri, Apr 20, 2012 at 01:11:34PM -0700, H.J. Lu wrote:
> On Fri, Apr 20, 2012 at 12:50 PM, Roland McGrath <roland@hack.frob.com> wrote:
> > Please provide an example that illustrates why you think you need this.
> >
> 
> Currently we use weak undefined symbol, foo, to do
> 
> if (&foo != 0)
>  foo is defined.
> else
>  foo isn't defined.
> 
> We want is to define foo as a secondary symbol so that
> we can always use foo without checking.  If there is a primary
> one in a .o file and .so file, we will get the primary one,
> otherwise, we will use the secondary one.

This is still a very general example. Does this concern a particular
software package?

Couldn't you use IFUNC to perform the if-statement above and supply
the proper function?

				Petr "Pasky" Baudis

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 3:54 PM, Petr Baudis <pasky@ucw.cz> wrote:
> On Fri, Apr 20, 2012 at 01:11:34PM -0700, H.J. Lu wrote:
>> On Fri, Apr 20, 2012 at 12:50 PM, Roland McGrath <roland@hack.frob.com> wrote:
>> > Please provide an example that illustrates why you think you need this.
>> >
>>
>> Currently we use weak undefined symbol, foo, to do
>>
>> if (&foo != 0)
>>  foo is defined.
>> else
>>  foo isn't defined.
>>
>> We want is to define foo as a secondary symbol so that
>> we can always use foo without checking.  If there is a primary
>> one in a .o file and .so file, we will get the primary one,
>> otherwise, we will use the secondary one.
>
> This is still a very general example. Does this concern a particular
> software package?

We have a compiler optimization feature which requires a backup
definition just in case that the primary one doesn't exist in an archive
or a DSO.


> Couldn't you use IFUNC to perform the if-statement above and supply
> the proper function?

No.


-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[Joern Rennecke]

Quoting "H.J. Lu" <hjl.tools@gmail.com>:

> Hi,
>
> We have a need to define a secondary symbol as backup in
> case there isn't a primary one.  Here is a proposal for
> STB_GNU_SECONDARY.  Any comments?

If two levels of prevedence (ordinary and weak) are not enough, why will
three levels be so much better?
If you use a signed fractional or even floating-point precedence value,
you have a lot more space to accomodate afterthoughts - above, below,
and in-between in precedence to existing values.

Even better, you could use symbolic tags, and have the linker script
assign precedence values to these tags.

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 1:55 PM, Joern Rennecke <amylaar@spamcop.net> wrote:
> Quoting "H.J. Lu" <hjl.tools@gmail.com>:
>
>> Hi,
>>
>> We have a need to define a secondary symbol as backup in
>> case there isn't a primary one.  Here is a proposal for
>> STB_GNU_SECONDARY.  Any comments?
>
>
> If two levels of prevedence (ordinary and weak) are not enough, why will
> three levels be so much better?

The main issues with weak symbols are

1.  A global definition in shared object won't override a weak definition
in relocatable file.
2. A global definition in archive won't override a weak definition
in relocatable file.

> If you use a signed fractional or even floating-point precedence value,
> you have a lot more space to accomodate afterthoughts - above, below,
> and in-between in precedence to existing values.

We only have very few bits to in STB_XXX field.

> Even better, you could use symbolic tags, and have the linker script
> assign precedence values to these tags.

It won't help us.


-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[Cary Coutant]

> We only have very few bits to in STB_XXX field.

This is exactly why I'm not in favor of this extension. The feature
doesn't seem compelling enough to use up one of these precious
reserved values (in fact, you're using the next-to-last one that's
reserved for OS use).

You want a backup definition? Put a weak def at the end of the link line.

-cary

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 3:10 PM, Cary Coutant <ccoutant@google.com> wrote:
>> We only have very few bits to in STB_XXX field.
>
> This is exactly why I'm not in favor of this extension. The feature
> doesn't seem compelling enough to use up one of these precious
> reserved values (in fact, you're using the next-to-last one that's
> reserved for OS use).
>
> You want a backup definition? Put a weak def at the end of the link line.
>

It doesn't work for us since the backup definition is
always used even if there is a normal definition in
a shared library or an archive.


-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 3:47 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Fri, Apr 20, 2012 at 3:10 PM, Cary Coutant <ccoutant@google.com> wrote:
>>> We only have very few bits to in STB_XXX field.
>>
>> This is exactly why I'm not in favor of this extension. The feature
>> doesn't seem compelling enough to use up one of these precious
>> reserved values (in fact, you're using the next-to-last one that's
>> reserved for OS use).
>>
>> You want a backup definition? Put a weak def at the end of the link line.
>>
>
> It doesn't work for us since the backup definition is
> always used even if there is a normal definition in
> a shared library or an archive.
>
>

In our usage, the backup definition may not be at the end of
link command line.


-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Fri, Apr 20, 2012 at 3:10 PM, Cary Coutant <ccoutant@google.com> wrote:
>>> We only have very few bits to in STB_XXX field.
>>
>> This is exactly why I'm not in favor of this extension. The feature
>> doesn't seem compelling enough to use up one of these precious
>> reserved values (in fact, you're using the next-to-last one that's
>> reserved for OS use).
>>
>> You want a backup definition? Put a weak def at the end of the link line.
>>
>
> It doesn't work for us since the backup definition is
> always used even if there is a normal definition in
> a shared library or an archive.

Can you expand on that?  How can you refer to the backup definition if
there is a normal definition?

Ian

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 3:59 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Fri, Apr 20, 2012 at 3:10 PM, Cary Coutant <ccoutant@google.com> wrote:
>>>> We only have very few bits to in STB_XXX field.
>>>
>>> This is exactly why I'm not in favor of this extension. The feature
>>> doesn't seem compelling enough to use up one of these precious
>>> reserved values (in fact, you're using the next-to-last one that's
>>> reserved for OS use).
>>>
>>> You want a backup definition? Put a weak def at the end of the link line.
>>>
>>
>> It doesn't work for us since the backup definition is
>> always used even if there is a normal definition in
>> a shared library or an archive.
>
> Can you expand on that?  How can you refer to the backup definition if
> there is a normal definition?
>

We need a definition for symbol, foo.  Since we don't know if there
is a definition of foo at the final link time.  We provide the backup
definition for foo.  The backup one is ignored if there is a normal one in
an archive or DSO at link time.


-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Fri, Apr 20, 2012 at 3:59 PM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>
>>> On Fri, Apr 20, 2012 at 3:10 PM, Cary Coutant <ccoutant@google.com> wrote:
>>>>> We only have very few bits to in STB_XXX field.
>>>>
>>>> This is exactly why I'm not in favor of this extension. The feature
>>>> doesn't seem compelling enough to use up one of these precious
>>>> reserved values (in fact, you're using the next-to-last one that's
>>>> reserved for OS use).
>>>>
>>>> You want a backup definition? Put a weak def at the end of the link line.
>>>>
>>>
>>> It doesn't work for us since the backup definition is
>>> always used even if there is a normal definition in
>>> a shared library or an archive.
>>
>> Can you expand on that?  How can you refer to the backup definition if
>> there is a normal definition?
>>
>
> We need a definition for symbol, foo.  Since we don't know if there
> is a definition of foo at the final link time.  We provide the backup
> definition for foo.  The backup one is ignored if there is a normal one in
> an archive or DSO at link time.

That use case would be satisfied by Cary's suggestion of adding a weak
definition of the symbol in an object included at the end of the link
line.

Ian

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 4:40 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Fri, Apr 20, 2012 at 3:59 PM, Ian Lance Taylor <iant@google.com> wrote:
>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>
>>>> On Fri, Apr 20, 2012 at 3:10 PM, Cary Coutant <ccoutant@google.com> wrote:
>>>>>> We only have very few bits to in STB_XXX field.
>>>>>
>>>>> This is exactly why I'm not in favor of this extension. The feature
>>>>> doesn't seem compelling enough to use up one of these precious
>>>>> reserved values (in fact, you're using the next-to-last one that's
>>>>> reserved for OS use).
>>>>>
>>>>> You want a backup definition? Put a weak def at the end of the link line.
>>>>>
>>>>
>>>> It doesn't work for us since the backup definition is
>>>> always used even if there is a normal definition in
>>>> a shared library or an archive.
>>>
>>> Can you expand on that?  How can you refer to the backup definition if
>>> there is a normal definition?
>>>
>>
>> We need a definition for symbol, foo.  Since we don't know if there
>> is a definition of foo at the final link time.  We provide the backup
>> definition for foo.  The backup one is ignored if there is a normal one in
>> an archive or DSO at link time.
>
> That use case would be satisfied by Cary's suggestion of adding a weak
> definition of the symbol in an object included at the end of the link
> line.

In our usage, the backup definition may not be at the end of
command line since it may reference library symbols.



-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> In our usage, the backup definition may not be at the end of
> command line since it may reference library symbols.

You could write out the backup function you need under a different name.
Then have the backup symbol at the end of the link call the new name of
the backup function.

I agree that this kind of approach is less convenient, but this doesn't
seem like a persuasive choice for one of the two remaining OS-dependent
symbol binding numbers.  Perhaps you should try to sell this idea to the
ELF ABI, the generic space has seven available numbers.

Ian

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 5:49 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> In our usage, the backup definition may not be at the end of
>> command line since it may reference library symbols.
>
> You could write out the backup function you need under a different name.
> Then have the backup symbol at the end of the link call the new name of
> the backup function.

isn't it replacing one backup with another backup?

> I agree that this kind of approach is less convenient, but this doesn't
> seem like a persuasive choice for one of the two remaining OS-dependent
> symbol binding numbers.  Perhaps you should try to sell this idea to the
> ELF ABI, the generic space has seven available numbers.
>

We are looking for a solution to have the backup definition
which can be overridden by a normal definition from an archive
or shared object.  Weak symbol doesn't work here.  I will ask
in the gABI group.

Thanks.

-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Fri, Apr 20, 2012 at 5:49 PM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>
>>> In our usage, the backup definition may not be at the end of
>>> command line since it may reference library symbols.
>>
>> You could write out the backup function you need under a different name.
>> Then have the backup symbol at the end of the link call the new name of
>> the backup function.
>
> isn't it replacing one backup with another backup?

Let's say your symbol is f.  I'm proposing that in the object file you
write out

void
f_implementation ()
{
  /* Whatever you want.  */
  /* Call library functions, etc. */
}

Then in an object at the end of the link, put

void
f () __attribute__ ((weak))
{
  f_implementation ();
}

Ian

Re: RFC: Add STB_GNU_SECONDARY

[Joern Rennecke]

Quoting "H.J. Lu" <hjl.tools@gmail.com>:

> We only have very few bits to in STB_XXX field.

Well, you could put the information somewhere else.  E.g. a special  
relocation,
or a special elf section.  Or you could mangle the information into
the section name in which the symbol is present.

>> Even better, you could use symbolic tags, and have the linker script
>> assign precedence values to these tags.
>
> It won't help us.

Maybe it wouldn't buy you more than the secondary symbols right now, but
it would give a lot of flexibility to rearrange and combine different
peoples ideas of link priority.

Another thought on symbols priorities: with pthread, using symbols of
different priorities is really a crutch - and it doesn't really work
that well for static linking.  What is really wanted is that
whenever one of a set of object files is linked in, a whole set of symbols
should be satisfied from a different place.  Maybe if, for a static pthread,
we'd put the pthread-aware I/O etc. into a special .text.pthread section
or somesuch, which would normally discarded, but had a mechanism to use them
if a special symbol is set (or used?) from an object file that provides
a pthread API function.

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Fri, Apr 20, 2012 at 6:08 PM, Joern Rennecke <amylaar@spamcop.net> wrote:
>>> Even better, you could use symbolic tags, and have the linker script
>>> assign precedence values to these tags.
>>
>>
>> It won't help us.
>
>
> Maybe it wouldn't buy you more than the secondary symbols right now, but
> it would give a lot of flexibility to rearrange and combine different
> peoples ideas of link priority.
>
> Another thought on symbols priorities: with pthread, using symbols of
> different priorities is really a crutch - and it doesn't really work
> that well for static linking.  What is really wanted is that
> whenever one of a set of object files is linked in, a whole set of symbols
> should be satisfied from a different place.  Maybe if, for a static pthread,
> we'd put the pthread-aware I/O etc. into a special .text.pthread section
> or somesuch, which would normally discarded, but had a mechanism to use them
> if a special symbol is set (or used?) from an object file that provides
> a pthread API function.

We want to provide a .o file which can take advantage of all versions of
Linux we support.  For a function, foo, in glibc, we can use it only if it is
available on all versions of glibc or we provide our own implementation of
foo inside the .o file.  With our own foo, the one in glibc will never be used.
When our own foo is marked as secondary, the one in glibc will be
used if it exists.  However we do want to use foo in glibc if it exists.

Putting our own foo in a section with a special prefix in section name,
like .secondary_*, works with linker support.  But it isn't very reliable.

-- 
H.J.

Re: RFC: Add STB_GNU_SECONDARY

[Joern Rennecke]

Quoting "H.J. Lu" <hjl.tools@gmail.com>:

> Putting our own foo in a section with a special prefix in section name,
> like .secondary_*, works with linker support.  But it isn't very reliable.

In what way is requiring linker support for STB_GNU_SECONDARY more reliable
than requiring linker support for .secondary_* sections?

Re: RFC: Add STB_GNU_SECONDARY

[H.J. Lu]

On Sat, Apr 21, 2012 at 12:01 PM, Joern Rennecke <amylaar@spamcop.net> wrote:
> Quoting "H.J. Lu" <hjl.tools@gmail.com>:
>
>> Putting our own foo in a section with a special prefix in section name,
>> like .secondary_*, works with linker support.  But it isn't very reliable.
>
>
> In what way is requiring linker support for STB_GNU_SECONDARY more reliable
> than requiring linker support for .secondary_* sections?

It may lead to conflict with section __attribute__ in C source and section
directive in assembly code.

-- 
H.J.