[MIPS] Is it legal for the assembler to generate more than 64K sections?

[MIPS] Is it legal for the assembler to generate more than 64K sections?

[Jack Carter]

My question is: shouldn't the assembler barf and if not, shouldn't the consuming elf readers scream?

I am debugging a test case where it looks like there are 77298 sections, but there is only 2 bytes to hold the section count in the ehdr and symbol header. Both relocations and the sections themselves seem to be able to hole 32 bits of section count.

The assembler produces the .o without complaint. The linker consumes the object without complaint, but sometimes barfs because relocation/symbol info is bad. Readelf also consumes the object without complaint except is a little wierd on the section count:

% readelf -h bad.o
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           MIPS R3000
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          22654764 (bytes into file)
  Flags:                             0x70001007, noreorder, pic, cpic, o32, mips32r2
  Size of this header:               52 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           40 (bytes)
  Number of section headers:         0 (77298)
  Section header string table index: 65535 (77294)

My home grown elfdump refused to read the object in the first place. 

The test case is c++ with macros and templates: llvm/tools/clang/unittests/Tooling/RecursiveASTVisitorTest.cpp.
I'm not really interested why so many sections got created, but in why the gnu assembler would allow this and why readobj and or the linker don't alert me to the fact things are not well in ELF land.

If this is a bug, I can submit it as bug and try to come up with the necessary patches to at least catch the section issue since I have a ready test case.

Thanks,

Jack

Re: [MIPS] Is it legal for the assembler to generate more than 64K sections?

[H.J. Lu]

On Fri, Jan 31, 2014 at 3:22 PM, Jack Carter <Jack.Carter@imgtec.com> wrote:
> My question is: shouldn't the assembler barf and if not, shouldn't the consuming elf readers scream?
>
> I am debugging a test case where it looks like there are 77298 sections, but there is only 2 bytes to hold the section count in the ehdr and symbol header. Both relocations and the sections themselves seem to be able to hole 32 bits of section count.
>
> The assembler produces the .o without complaint. The linker consumes the object without complaint, but sometimes barfs because relocation/symbol info is bad. Readelf also consumes the object without complaint except is a little wierd on the section count:
>
> % readelf -h bad.o
> ELF Header:
>   Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
>   Class:                             ELF32
>   Data:                              2's complement, little endian
>   Version:                           1 (current)
>   OS/ABI:                            UNIX - System V
>   ABI Version:                       0
>   Type:                              REL (Relocatable file)
>   Machine:                           MIPS R3000
>   Version:                           0x1
>   Entry point address:               0x0
>   Start of program headers:          0 (bytes into file)
>   Start of section headers:          22654764 (bytes into file)
>   Flags:                             0x70001007, noreorder, pic, cpic, o32, mips32r2
>   Size of this header:               52 (bytes)
>   Size of program headers:           0 (bytes)
>   Number of program headers:         0
>   Size of section headers:           40 (bytes)
>   Number of section headers:         0 (77298)
>   Section header string table index: 65535 (77294)
>
> My home grown elfdump refused to read the object in the first place.

This is perfectly fine with the current gABI which supports >64K
sections.  You need to find out why MIPS backend can't handle
it properly.  Check how SHN_LORESERVE and SHN_XINDEX
are handled.

> The test case is c++ with macros and templates: llvm/tools/clang/unittests/Tooling/RecursiveASTVisitorTest.cpp.
> I'm not really interested why so many sections got created, but in why the gnu assembler would allow this and why readobj and or the linker don't alert me to the fact things are not well in ELF land.
>
> If this is a bug, I can submit it as bug and try to come up with the necessary patches to at least catch the section issue since I have a ready test case.
>
> Thanks,
>
> Jack



-- 
H.J.

RE: [MIPS] Is it legal for the assembler to generate more than 64K sections?

[Jack Carter]

________________________________________
>From: H.J. Lu [hjl.tools@gmail.com]
>Sent: Friday, January 31, 2014 4:25 PM
>To: Jack Carter
>Cc: binutils@sourceware.org
>Subject: Re: [MIPS] Is it legal for the assembler to generate more than 64K sections?
>
>On Fri, Jan 31, 2014 at 3:22 PM, Jack Carter <Jack.Carter@imgtec.com> wrote:
>> My question is: shouldn't the assembler barf and if not, shouldn't the consuming elf readers scream?
>>
>> I am debugging a test case where it looks like there are 77298 sections, but there is only 2 bytes to hold the section count in the ehdr and symbol header. Both relocations and the sections themselves seem to be able to hole 32 bits of section count.
>>
>> The assembler produces the .o without complaint. The linker consumes the object without complaint, but sometimes barfs because relocation/symbol info is bad. Readelf also consumes the object without complaint except is a little wierd on the section count:
>>
> % readelf -h bad.o
>> ELF Header:
>>   Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
>>   Class:                             ELF32
>>   Data:                              2's complement, little endian
>>   Version:                           1 (current)
>>   OS/ABI:                            UNIX - System V
>>   ABI Version:                       0
>>   Type:                              REL (Relocatable file)
>>   Machine:                           MIPS R3000
>>   Version:                           0x1
>>   Entry point address:               0x0
>>   Start of program headers:          0 (bytes into file)
>>   Start of section headers:          22654764 (bytes into file)
>>   Flags:                             0x70001007, noreorder, pic, cpic, o32, mips32r2
>>   Size of this header:               52 (bytes)
>>   Size of program headers:           0 (bytes)
>>   Number of program headers:         0
>>   Size of section headers:           40 (bytes)
>>   Number of section headers:         0 (77298)
>>   Section header string table index: 65535 (77294)
>>
>> My home grown elfdump refused to read the object in the first place.
>
>This is perfectly fine with the current gABI which supports >64K
>sections.  You need to find out why MIPS backend can't handle
>it properly.  Check how SHN_LORESERVE and SHN_XINDEX
>are handled.

How is this fine if the ABI it is reading into only supports 16 bits?

typedef struct
{
  unsigned char	e_ident[EI_NIDENT];	/* Magic number and other info */
  Elf32_Half	e_type;			/* Object file type */
  Elf32_Half	e_machine;		/* Architecture */
  Elf32_Word	e_version;		/* Object file version */
  Elf32_Addr	e_entry;		/* Entry point virtual address */
  Elf32_Off	e_phoff;		/* Program header table file offset */
  Elf32_Off	e_shoff;		/* Section header table file offset */
  Elf32_Word	e_flags;		/* Processor-specific flags */
  Elf32_Half	e_ehsize;		/* ELF header size in bytes */
  Elf32_Half	e_phentsize;		/* Program header table entry size */
  Elf32_Half	e_phnum;		/* Program header table entry count */
  Elf32_Half	e_shentsize;		/* Section header table entry size */
  Elf32_Half	e_shnum;		/* Section header table entry count */
  Elf32_Half	e_shstrndx;		/* Section header string table index */
} Elf32_Ehdr;

/* Symbol table entry.  */

typedef struct
{
  Elf32_Word	st_name;		/* Symbol name (string tbl index) */
  Elf32_Addr	st_value;		/* Symbol value */
  Elf32_Word	st_size;		/* Symbol size */
  unsigned char	st_info;		/* Symbol type and binding */
  unsigned char	st_other;		/* Symbol visibility */
  Elf32_Section	st_shndx;		/* Section index */
} Elf32_Sym;

Does this mean everywhere in binutils and glibc for MIPS that there is a data structure that could store a section index that it needs to be increased to 32 bits?

Or does it mean that even though the gABI seems to handled 32 bit sections, for MIPS we should reject that amount because it will be illegal for the MIPS ABI?

Thanks,

Jack

>
>> The test case is c++ with macros and templates: llvm/tools/clang/unittests/Tooling/RecursiveASTVisitorTest.cpp.
>> I'm not really interested why so many sections got created, but in why the gnu assembler would allow this and why readobj and or the linker don't alert me to the fact things are not well in ELF land.
>>
>> If this is a bug, I can submit it as bug and try to come up with the necessary patches to at least catch the section issue since I have a ready test case.
>>
>> Thanks,
>>
>> Jack
>
>
>
>--
>H.J.

Re: [MIPS] Is it legal for the assembler to generate more than 64K sections?

[Richard Sandiford]

Jack Carter <Jack.Carter@imgtec.com> writes:
> ________________________________________
>>From: H.J. Lu [hjl.tools@gmail.com]
>>Sent: Friday, January 31, 2014 4:25 PM
>>To: Jack Carter
>>Cc: binutils@sourceware.org
>>Subject: Re: [MIPS] Is it legal for the assembler to generate more than
>> 64K sections?
>>
>>On Fri, Jan 31, 2014 at 3:22 PM, Jack Carter <Jack.Carter@imgtec.com> wrote:
>>> My question is: shouldn't the assembler barf and if not, shouldn't
>>> the consuming elf readers scream?
>>>
>>> I am debugging a test case where it looks like there are 77298
>>> sections, but there is only 2 bytes to hold the section count in the
>>> ehdr and symbol header. Both relocations and the sections themselves
>>> seem to be able to hole 32 bits of section count.
>>>
>>> The assembler produces the .o without complaint. The linker consumes
>>> the object without complaint, but sometimes barfs because
>>> relocation/symbol info is bad. Readelf also consumes the object
>>> without complaint except is a little wierd on the section count:
>>>
>> % readelf -h bad.o
>>> ELF Header:
>>>   Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
>>>   Class:                             ELF32
>>>   Data:                              2's complement, little endian
>>>   Version:                           1 (current)
>>>   OS/ABI:                            UNIX - System V
>>>   ABI Version:                       0
>>>   Type:                              REL (Relocatable file)
>>>   Machine:                           MIPS R3000
>>>   Version:                           0x1
>>>   Entry point address:               0x0
>>>   Start of program headers:          0 (bytes into file)
>>>   Start of section headers:          22654764 (bytes into file)
>>>   Flags: 0x70001007, noreorder, pic, cpic, o32, mips32r2
>>>   Size of this header:               52 (bytes)
>>>   Size of program headers:           0 (bytes)
>>>   Number of program headers:         0
>>>   Size of section headers:           40 (bytes)
>>>   Number of section headers:         0 (77298)
>>>   Section header string table index: 65535 (77294)
>>>
>>> My home grown elfdump refused to read the object in the first place.
>>
>>This is perfectly fine with the current gABI which supports >64K
>>sections.  You need to find out why MIPS backend can't handle
>>it properly.  Check how SHN_LORESERVE and SHN_XINDEX
>>are handled.
>
> How is this fine if the ABI it is reading into only supports 16 bits?
>
> typedef struct
> {
>   unsigned char	e_ident[EI_NIDENT];	/* Magic number and other info */
>   Elf32_Half	e_type;			/* Object file type */
>   Elf32_Half	e_machine;		/* Architecture */
>   Elf32_Word	e_version;		/* Object file version */
>   Elf32_Addr	e_entry;		/* Entry point virtual address */
>   Elf32_Off	e_phoff;		/* Program header table file offset */
>   Elf32_Off	e_shoff;		/* Section header table file offset */
>   Elf32_Word	e_flags;		/* Processor-specific flags */
>   Elf32_Half	e_ehsize;		/* ELF header size in bytes */
>   Elf32_Half	e_phentsize;		/* Program header table entry size */
>   Elf32_Half	e_phnum;		/* Program header table entry count */
>   Elf32_Half	e_shentsize;		/* Section header table entry size */
>   Elf32_Half	e_shnum;		/* Section header table entry count */
>   Elf32_Half	e_shstrndx;		/* Section header string table index */
> } Elf32_Ehdr;
>
> /* Symbol table entry.  */
>
> typedef struct
> {
>   Elf32_Word	st_name;		/* Symbol name (string tbl index) */
>   Elf32_Addr	st_value;		/* Symbol value */
>   Elf32_Word	st_size;		/* Symbol size */
>   unsigned char	st_info;		/* Symbol type and binding */
>   unsigned char	st_other;		/* Symbol visibility */
>   Elf32_Section	st_shndx;		/* Section index */
> } Elf32_Sym;
>
> Does this mean everywhere in binutils and glibc for MIPS that there is a
> data structure that could store a section index that it needs to be
> increased to 32 bits?

See:

    http://www.sco.com/developers/gabi/latest/ch4.eheader.html

specifically:

e_shnum
    This member holds the number of entries in the section header
    table. Thus the product of e_shentsize and e_shnum gives the section
    header table's size in bytes. If a file has no section header table,
    e_shnum holds the value zero.

    If the number of sections is greater than or equal to SHN_LORESERVE
    (0xff00), this member has the value zero and the actual number of
    section header table entries is contained in the sh_size field of
    the section header at index 0. (Otherwise, the sh_size member of the
    initial entry contains 0.)

What's the problem you're seeing exactly?  I wasn't sure from your
original message.

Thanks,
Richard

RE: [MIPS] Is it legal for the assembler to generate more than 64K sections?

[Jack Carter]

________________________________________
From: Richard Sandiford [rdsandiford@googlemail.com]
Sent: Monday, February 03, 2014 12:29 PM
To: Jack Carter
Cc: H.J. Lu; binutils@sourceware.org
Subject: Re: [MIPS] Is it legal for the assembler to generate more than 64K sections?

>>
>> Does this mean everywhere in binutils and glibc for MIPS that there is a
>> data structure that could store a section index that it needs to be
>> increased to 32 bits?
>
>See:
>
>    http://www.sco.com/developers/gabi/latest/ch4.eheader.html
>
>specifically:
>
>e_shnum
>    This member holds the number of entries in the section header
>    table. Thus the product of e_shentsize and e_shnum gives the section
>    header table's size in bytes. If a file has no section header table,
>    e_shnum holds the value zero.
>
>    If the number of sections is greater than or equal to SHN_LORESERVE
>    (0xff00), this member has the value zero and the actual number of
>    section header table entries is contained in the sh_size field of
>    the section header at index 0. (Otherwise, the sh_size member of the
>    initial entry contains 0.)
>
>What's the problem you're seeing exactly?  I wasn't sure from your
>original message.
>
>Thanks,
>Richard

We have a "canadian_cross" build of LLVM and have been getting off and on link time errors from one of the input object files RecursiveASTVisitorTest.o. :

(.text._ZN4llvm25SmallVectorTemplateCommonIN5clang19RecursiveASTVisitorINS1_11AttrVisitorEE10EnqueueJobEvE4backEv[_ZN4llvm25SmallVectorTemplateCommonIN5clang19RecursiveASTVisitorINS1_11AttrVisitorEE10EnqueueJobEvE4backEv]+0x64): relocation truncated to fit: R_MIPS_GOT16 against `no symbol'

Basically building LLVM with a MIPS GCC. I noticed that the section count was way above 16 bits and seized on that as being the culprit.

From the information you and H.J. have pointed me to, I need to dig further. Why I never knew about this trap door for section overflow is beyond me :-| Probably since I never hit the limit myself.

Thanks,

Jack

Re: [MIPS] Is it legal for the assembler to generate more than 64K sections?

[Richard Sandiford]

Jack Carter <Jack.Carter@imgtec.com> writes:
> We have a "canadian_cross" build of LLVM and have been getting off and
> on link time errors from one of the input object files
> RecursiveASTVisitorTest.o. :
>
> (.text._ZN4llvm25SmallVectorTemplateCommonIN5clang19RecursiveASTVisitorINS1_11AttrVisitorEE10EnqueueJobEvE4backEv[_ZN4llvm25SmallVectorTemplateCommonIN5clang19RecursiveASTVisitorINS1_11AttrVisitorEE10EnqueueJobEvE4backEv]+0x64):
> relocation truncated to fit: R_MIPS_GOT16 against `no symbol'
>
> Basically building LLVM with a MIPS GCC. I noticed that the section
> count was way above 16 bits and seized on that as being the culprit.

Ah, OK.  Looks like a multigot failure at face value.  Are you using
recent binutils (2.24)?  It has some fixes in that area.

Thanks,
Richard

RE: [MIPS] Is it legal for the assembler to generate more than 64K sections?

[Jack Carter]

>(.text._ZN4llvm25SmallVectorTemplateCommonIN5clang19RecursiveASTVisitorINS1_11AttrVisitorEE10EnqueueJobEvE4backEv[_ZN4llvm25SmallVectorTemplateCommonIN5clang19RecursiveASTVisitorINS1_11AttrVisitorEE10EnqueueJobEvE4backEv]+0x64):
>> relocation truncated to fit: R_MIPS_GOT16 against `no symbol'
>>
>> Basically building LLVM with a MIPS GCC. I noticed that the section
>> count was way above 16 bits and seized on that as being the culprit.
>
>Ah, OK.  Looks like a multigot failure at face value.  Are you using
>recent binutils (2.24)?  It has some fixes in that area.

I rebased on Thursday.

Multigot is a good area for me to look since I am mucking with the got with ifunc anyway.

Cheers,

Jack

>
>Thanks,
>Richard

Re: [MIPS] Is it legal for the assembler to generate more than 64K sections?

[Andrew Pinski]

On Mon, Feb 3, 2014 at 1:12 PM, Jack Carter <Jack.Carter@imgtec.com> wrote:
>>(.text._ZN4llvm25SmallVectorTemplateCommonIN5clang19RecursiveASTVisitorINS1_11AttrVisitorEE10EnqueueJobEvE4backEv[_ZN4llvm25SmallVectorTemplateCommonIN5clang19RecursiveASTVisitorINS1_11AttrVisitorEE10EnqueueJobEvE4backEv]+0x64):
>>> relocation truncated to fit: R_MIPS_GOT16 against `no symbol'
>>>
>>> Basically building LLVM with a MIPS GCC. I noticed that the section
>>> count was way above 16 bits and seized on that as being the culprit.
>>
>>Ah, OK.  Looks like a multigot failure at face value.  Are you using
>>recent binutils (2.24)?  It has some fixes in that area.
>
> I rebased on Thursday.
>
> Multigot is a good area for me to look since I am mucking with the got with ifunc anyway.


It could also be that the GOT for that object file is just too big.  I
bet that is the case which means you need to use the option -mxgot .
Multigot only works with different object files, it does not work
inside one object file as far as I know.

Thanks,
Andrew Pinski

>
> Cheers,
>
> Jack
>
>>
>>Thanks,
>>Richard
>