non-conformant ELF symbol table on MIPS?

non-conformant ELF symbol table on MIPS?

[Jan Beulich]

Hello,

extending the gas elf/common3.s testcase (in the context of trying to fix
PR gas/30688) to

	.comm foobar,30,4
	.comm buf1, 5-1, 8
	.comm buf2, 4, 9-1
	.lcomm lbuf, 9-1

I notice a violation of the ELF spec on most (all?) MIPS targets I would
routinely test for common code changes: "lbuf" is emitted last in the
symbol table. The spec is quite clear: "In each symbol table, all
symbols with STB_LOCAL binding precede the weak and global symbols." I
guess I want/need to xfail that test for mips*-*-* for the time being,
but it would be nice if you could shed some light on this behavior.

Thanks, Jan

Re: non-conformant ELF symbol table on MIPS?

[Ian Lance Taylor]

On Thu, Jul 27, 2023 at 7:28 AM Jan Beulich via Binutils
<binutils@sourceware.org> wrote:
>
> extending the gas elf/common3.s testcase (in the context of trying to fix
> PR gas/30688) to
>
>         .comm foobar,30,4
>         .comm buf1, 5-1, 8
>         .comm buf2, 4, 9-1
>         .lcomm lbuf, 9-1
>
> I notice a violation of the ELF spec on most (all?) MIPS targets I would
> routinely test for common code changes: "lbuf" is emitted last in the
> symbol table. The spec is quite clear: "In each symbol table, all
> symbols with STB_LOCAL binding precede the weak and global symbols." I
> guess I want/need to xfail that test for mips*-*-* for the time being,
> but it would be nice if you could shed some light on this behavior.

This is historical behavior that dates back to the SGI Irix operating
system.  Search for elf_bad_symtab in the BFD sources.  Here is the
commend from elf-bfd.h:

  /* Irix 5 often screws up the symbol table, sorting local symbols
     after global symbols.  This flag is set if the symbol table in
     this BFD appears to be screwed up.  If it is, we ignore the
     sh_info field in the symbol table header, and always read all the
     symbols.  */
  unsigned int bad_symtab : 1;

I believe we currently set elf_bad_symtab true by default for most MIPS targets.

Ian

Re: non-conformant ELF symbol table on MIPS?

[Maciej W. Rozycki]

On Thu, 27 Jul 2023, Ian Lance Taylor wrote:

> > I notice a violation of the ELF spec on most (all?) MIPS targets I would
> > routinely test for common code changes: "lbuf" is emitted last in the
> > symbol table. The spec is quite clear: "In each symbol table, all
> > symbols with STB_LOCAL binding precede the weak and global symbols." I
> > guess I want/need to xfail that test for mips*-*-* for the time being,
> > but it would be nice if you could shed some light on this behavior.
> 
> This is historical behavior that dates back to the SGI Irix operating
> system.  Search for elf_bad_symtab in the BFD sources.  Here is the
> commend from elf-bfd.h:
> 
>   /* Irix 5 often screws up the symbol table, sorting local symbols
>      after global symbols.  This flag is set if the symbol table in
>      this BFD appears to be screwed up.  If it is, we ignore the
>      sh_info field in the symbol table header, and always read all the
>      symbols.  */
>   unsigned int bad_symtab : 1;
> 
> I believe we currently set elf_bad_symtab true by default for most 
> MIPS targets.

 Maybe, though not sure if really, by the number of individual targets 
supported, but certainly not by the userbase.  IRIX emulation is only 
used by IRIX targets (naturally, hardly used now) and some, especially 
older bare-metal targets.  Then Linux, *BSD and the remaining bare-metal 
targets use traditional emulation.

 In particular IRIX emulation is incompatible with symbol versioning 
required by glibc 2.x, which was the motivation back in the day to add 
traditional emulation support to the MIPS backend.

  Maciej

Re: non-conformant ELF symbol table on MIPS?

[Jan Beulich]

On 27.07.2023 21:04, Ian Lance Taylor wrote:
> On Thu, Jul 27, 2023 at 7:28 AM Jan Beulich via Binutils
> <binutils@sourceware.org> wrote:
>>
>> extending the gas elf/common3.s testcase (in the context of trying to fix
>> PR gas/30688) to
>>
>>         .comm foobar,30,4
>>         .comm buf1, 5-1, 8
>>         .comm buf2, 4, 9-1
>>         .lcomm lbuf, 9-1
>>
>> I notice a violation of the ELF spec on most (all?) MIPS targets I would
>> routinely test for common code changes: "lbuf" is emitted last in the
>> symbol table. The spec is quite clear: "In each symbol table, all
>> symbols with STB_LOCAL binding precede the weak and global symbols." I
>> guess I want/need to xfail that test for mips*-*-* for the time being,
>> but it would be nice if you could shed some light on this behavior.
> 
> This is historical behavior that dates back to the SGI Irix operating
> system.  Search for elf_bad_symtab in the BFD sources.  Here is the
> commend from elf-bfd.h:
> 
>   /* Irix 5 often screws up the symbol table, sorting local symbols
>      after global symbols.  This flag is set if the symbol table in
>      this BFD appears to be screwed up.  If it is, we ignore the
>      sh_info field in the symbol table header, and always read all the
>      symbols.  */
>   unsigned int bad_symtab : 1;
> 
> I believe we currently set elf_bad_symtab true by default for most MIPS targets.

That explains consuming behavior. But gas is a producer, and from what
you say and quote I can't see why gas also needs to produce non-
conforming symbol tables.

Jan

Re: non-conformant ELF symbol table on MIPS?

[Ian Lance Taylor]

On Thu, Jul 27, 2023 at 11:16 PM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 27.07.2023 21:04, Ian Lance Taylor wrote:
> > On Thu, Jul 27, 2023 at 7:28 AM Jan Beulich via Binutils
> > <binutils@sourceware.org> wrote:
> >>
> >> extending the gas elf/common3.s testcase (in the context of trying to fix
> >> PR gas/30688) to
> >>
> >>         .comm foobar,30,4
> >>         .comm buf1, 5-1, 8
> >>         .comm buf2, 4, 9-1
> >>         .lcomm lbuf, 9-1
> >>
> >> I notice a violation of the ELF spec on most (all?) MIPS targets I would
> >> routinely test for common code changes: "lbuf" is emitted last in the
> >> symbol table. The spec is quite clear: "In each symbol table, all
> >> symbols with STB_LOCAL binding precede the weak and global symbols." I
> >> guess I want/need to xfail that test for mips*-*-* for the time being,
> >> but it would be nice if you could shed some light on this behavior.
> >
> > This is historical behavior that dates back to the SGI Irix operating
> > system.  Search for elf_bad_symtab in the BFD sources.  Here is the
> > commend from elf-bfd.h:
> >
> >   /* Irix 5 often screws up the symbol table, sorting local symbols
> >      after global symbols.  This flag is set if the symbol table in
> >      this BFD appears to be screwed up.  If it is, we ignore the
> >      sh_info field in the symbol table header, and always read all the
> >      symbols.  */
> >   unsigned int bad_symtab : 1;
> >
> > I believe we currently set elf_bad_symtab true by default for most MIPS targets.
>
> That explains consuming behavior. But gas is a producer, and from what
> you say and quote I can't see why gas also needs to produce non-
> conforming symbol tables.

It was required for the linker, because otherwise the SGI IRIX dynamic
linker would not work correctly.

I don't recall whether it was required for gas.  That may have been an
accidental carry-over.  Or it may have been required for the IRIX
static linker to work, I don't remember.

Doing it for the plan mips-elf target is probably not the best choice today.

Ian

Re: non-conformant ELF symbol table on MIPS?

[Alan Modra]

On Fri, Jul 28, 2023 at 10:11:50AM -0700, Ian Lance Taylor via Binutils wrote:
> On Thu, Jul 27, 2023 at 11:16 PM Jan Beulich <jbeulich@suse.com> wrote:
> >
> > On 27.07.2023 21:04, Ian Lance Taylor wrote:
> > > On Thu, Jul 27, 2023 at 7:28 AM Jan Beulich via Binutils
> > > <binutils@sourceware.org> wrote:
> > >>
> > >> extending the gas elf/common3.s testcase (in the context of trying to fix
> > >> PR gas/30688) to
> > >>
> > >>         .comm foobar,30,4
> > >>         .comm buf1, 5-1, 8
> > >>         .comm buf2, 4, 9-1
> > >>         .lcomm lbuf, 9-1
> > >>
> > >> I notice a violation of the ELF spec on most (all?) MIPS targets I would
> > >> routinely test for common code changes: "lbuf" is emitted last in the
> > >> symbol table. The spec is quite clear: "In each symbol table, all
> > >> symbols with STB_LOCAL binding precede the weak and global symbols." I
> > >> guess I want/need to xfail that test for mips*-*-* for the time being,
> > >> but it would be nice if you could shed some light on this behavior.
> > >
> > > This is historical behavior that dates back to the SGI Irix operating
> > > system.  Search for elf_bad_symtab in the BFD sources.  Here is the
> > > commend from elf-bfd.h:
> > >
> > >   /* Irix 5 often screws up the symbol table, sorting local symbols
> > >      after global symbols.  This flag is set if the symbol table in
> > >      this BFD appears to be screwed up.  If it is, we ignore the
> > >      sh_info field in the symbol table header, and always read all the
> > >      symbols.  */
> > >   unsigned int bad_symtab : 1;
> > >
> > > I believe we currently set elf_bad_symtab true by default for most MIPS targets.
> >
> > That explains consuming behavior. But gas is a producer, and from what
> > you say and quote I can't see why gas also needs to produce non-
> > conforming symbol tables.
> 
> It was required for the linker, because otherwise the SGI IRIX dynamic
> linker would not work correctly.
> 
> I don't recall whether it was required for gas.  That may have been an
> accidental carry-over.  Or it may have been required for the IRIX
> static linker to work, I don't remember.

I think it is probably required.  It's not just the sorting within
symbol tables but also the meaning of sh_info.  For mips irix5 a
symbol table sh_info is the section symbol count, not the local symbol
count.  I say "probably" because I guess it might be possible to have
SHT_SYMTAB and SHT_DYNSYM differ, but that seems even more weird.

> Doing it for the plain mips-elf target is probably not the best choice today.

I agree.

-- 
Alan Modra
Australia Development Lab, IBM

Re: non-conformant ELF symbol table on MIPS?

[Maciej W. Rozycki]

On Thu, 27 Jul 2023, Jan Beulich wrote:

> extending the gas elf/common3.s testcase (in the context of trying to fix
> PR gas/30688) to
> 
> 	.comm foobar,30,4
> 	.comm buf1, 5-1, 8
> 	.comm buf2, 4, 9-1
> 	.lcomm lbuf, 9-1
> 
> I notice a violation of the ELF spec on most (all?) MIPS targets I would
> routinely test for common code changes: "lbuf" is emitted last in the
> symbol table. The spec is quite clear: "In each symbol table, all
> symbols with STB_LOCAL binding precede the weak and global symbols." I
> guess I want/need to xfail that test for mips*-*-* for the time being,
> but it would be nice if you could shed some light on this behavior.

 Which specific MIPS target is it?

 There are two kinds of MIPS targets, ones that use the older IRIX symbol 
sorting rules and ones that use the newer "traditional" symbol sorting 
rules.  The IRIX symbol sorting rules mandate that section symbols precede 
all other symbols, see e.g. bfd/elf32-mips.c:mips_elf_sym_is_global.  
Mind that SGI was an early adopter of the ELF standard and things weren't 
as settled as they are nowadays, hence e.g. the odd solution for composed 
relocations with the n64 psABI (later n32 uses standard ELF gABI ones) and 
other peculiarities here and there.

 Determining whether a MIPS target uses the IRIX emulation is complex, see 
e.g. how binutils/testsuite/binutils-all/mips/mips.exp sets `irixemul'.  
If you provide me with test case details (input and output), then I may be 
able to give you further advice, but otherwise you may want to mark the 
test case with `notarget' or suchlike rather than `xfail', because that's 
not a bug with the backend that the symbol ordering varies.

  Maciej

Re: non-conformant ELF symbol table on MIPS?

[Jan Beulich]

On 27.07.2023 21:28, Maciej W. Rozycki wrote:
> On Thu, 27 Jul 2023, Jan Beulich wrote:
> 
>> extending the gas elf/common3.s testcase (in the context of trying to fix
>> PR gas/30688) to
>>
>> 	.comm foobar,30,4
>> 	.comm buf1, 5-1, 8
>> 	.comm buf2, 4, 9-1
>> 	.lcomm lbuf, 9-1
>>
>> I notice a violation of the ELF spec on most (all?) MIPS targets I would
>> routinely test for common code changes: "lbuf" is emitted last in the
>> symbol table. The spec is quite clear: "In each symbol table, all
>> symbols with STB_LOCAL binding precede the weak and global symbols." I
>> guess I want/need to xfail that test for mips*-*-* for the time being,
>> but it would be nice if you could shed some light on this behavior.
> 
>  Which specific MIPS target is it?

Among the 9 I test, it's mips-unknown-elf, mipsisa32r2el-unknown-elf,
mips-sgi-irix6, and mipstx39-unknown-elf.

>  There are two kinds of MIPS targets, ones that use the older IRIX symbol 
> sorting rules and ones that use the newer "traditional" symbol sorting 
> rules.  The IRIX symbol sorting rules mandate that section symbols precede 
> all other symbols, see e.g. bfd/elf32-mips.c:mips_elf_sym_is_global.  
> Mind that SGI was an early adopter of the ELF standard and things weren't 
> as settled as they are nowadays, hence e.g. the odd solution for composed 
> relocations with the n64 psABI (later n32 uses standard ELF gABI ones) and 
> other peculiarities here and there.
> 
>  Determining whether a MIPS target uses the IRIX emulation is complex, see 
> e.g. how binutils/testsuite/binutils-all/mips/mips.exp sets `irixemul'.  
> If you provide me with test case details (input and output), then I may be 
> able to give you further advice,

I've provided you the input (see context above), and you'll be able to
generate the object file from it yourself. But I don't see what this
would bring, when you already indicate that this is "expected" behavior.

> but otherwise you may want to mark the 
> test case with `notarget' or suchlike rather than `xfail', because that's 
> not a bug with the backend that the symbol ordering varies.

I'm okay making mips*-*-irix* "notarget", but I don't view this as
appropriate for e.g. mips*-unknown-elf. Instead I would prefer leaving
further tweaking (if so needed) to you. The immediate goal is to get a
regression fix (with an accompanying extension to a testcase) in
quickly, such that hopefully Nick would still be able to pick it up for
2.41 (as in principle it affects most if not all targets).

Jan

Re: non-conformant ELF symbol table on MIPS?

[Jan Beulich]

On 28.07.2023 08:13, Jan Beulich wrote:
> On 27.07.2023 21:28, Maciej W. Rozycki wrote:
>>  Determining whether a MIPS target uses the IRIX emulation is complex, see 
>> e.g. how binutils/testsuite/binutils-all/mips/mips.exp sets `irixemul'.  
>> If you provide me with test case details (input and output), then I may be 
>> able to give you further advice,
> 
> I've provided you the input (see context above), and you'll be able to
> generate the object file from it yourself. But I don't see what this
> would bring, when you already indicate that this is "expected" behavior.
> 
>> but otherwise you may want to mark the 
>> test case with `notarget' or suchlike rather than `xfail', because that's 
>> not a bug with the backend that the symbol ordering varies.
> 
> I'm okay making mips*-*-irix* "notarget", but I don't view this as
> appropriate for e.g. mips*-unknown-elf.

Actually you're right, it needs to be notarget, or else we'd see
"unexpected successes" on the other mips*-*-* targets. I'll refer
to irixemul in the accompanying comment.

Jan

Re: non-conformant ELF symbol table on MIPS?

[Jan Beulich]

On 27.07.2023 21:28, Maciej W. Rozycki wrote:
> On Thu, 27 Jul 2023, Jan Beulich wrote:
> 
>> extending the gas elf/common3.s testcase (in the context of trying to fix
>> PR gas/30688) to
>>
>> 	.comm foobar,30,4
>> 	.comm buf1, 5-1, 8
>> 	.comm buf2, 4, 9-1
>> 	.lcomm lbuf, 9-1
>>
>> I notice a violation of the ELF spec on most (all?) MIPS targets I would
>> routinely test for common code changes: "lbuf" is emitted last in the
>> symbol table. The spec is quite clear: "In each symbol table, all
>> symbols with STB_LOCAL binding precede the weak and global symbols." I
>> guess I want/need to xfail that test for mips*-*-* for the time being,
>> but it would be nice if you could shed some light on this behavior.
> 
>  Which specific MIPS target is it?
> 
>  There are two kinds of MIPS targets, ones that use the older IRIX symbol 
> sorting rules and ones that use the newer "traditional" symbol sorting 
> rules.  The IRIX symbol sorting rules mandate that section symbols precede 
> all other symbols, see e.g. bfd/elf32-mips.c:mips_elf_sym_is_global.  

Along the lines of my reply to Ian: This still doesn't explain the
behavior I observe. What you say is required could still be met without
violating ELF rules.

Jan

> Mind that SGI was an early adopter of the ELF standard and things weren't 
> as settled as they are nowadays, hence e.g. the odd solution for composed 
> relocations with the n64 psABI (later n32 uses standard ELF gABI ones) and 
> other peculiarities here and there.
> 
>  Determining whether a MIPS target uses the IRIX emulation is complex, see 
> e.g. how binutils/testsuite/binutils-all/mips/mips.exp sets `irixemul'.  
> If you provide me with test case details (input and output), then I may be 
> able to give you further advice, but otherwise you may want to mark the 
> test case with `notarget' or suchlike rather than `xfail', because that's 
> not a bug with the backend that the symbol ordering varies.
> 
>   Maciej

Re: non-conformant ELF symbol table on MIPS?

[Maciej W. Rozycki]

On Fri, 28 Jul 2023, Jan Beulich wrote:

> >  There are two kinds of MIPS targets, ones that use the older IRIX symbol 
> > sorting rules and ones that use the newer "traditional" symbol sorting 
> > rules.  The IRIX symbol sorting rules mandate that section symbols precede 
> > all other symbols, see e.g. bfd/elf32-mips.c:mips_elf_sym_is_global.  
> 
> Along the lines of my reply to Ian: This still doesn't explain the
> behavior I observe. What you say is required could still be met without
> violating ELF rules.

 Yes, we could put section symbols first, then local ones, then external 
ones, and it was discussed many years ago, but it would require rather a 
heavy overhaul of generic parts of BFD.  Someone would have to do it just 
to handle a corner case, so it was concluded that it wouldn't be the best 
use of time, which would better be spent on something more important.

  Maciej