Best reference for understanding ELF format

Best reference for understanding ELF format

[Peng Yu]

Hi,

I am wondering what is the best reference for understanding ELF
format. By best, I mean update-to-date, easy-to-following, and
complete.

Could anybody recommend references based on this criterion? Thanks.

-- 
Regards,
Peng

Re: Best reference for understanding ELF format

[Mike Frysinger]

On 15 Apr 2021 22:38, Peng Yu via Libc-help wrote:
> I am wondering what is the best reference for understanding ELF
> format. By best, I mean update-to-date, easy-to-following, and
> complete.
> 
> Could anybody recommend references based on this criterion? Thanks.

read "Linkers & Loaders" by John R. Levine
-mike

Re: Best reference for understanding ELF format

[Peng Yu]

That is a book. By definition, a book will not be complete since a
book is not written for completeness?

For the best reference (as "reference" may have muliple meanings maybe
I should use the word "spec" to avoid ambiguity), I found the
following two documents. One is for 32-bit, the other is for 64-bit.
Are they the most current ones? Other there any other documents that
cover the things missed by them?

https://refspecs.linuxfoundation.org/elf/elf.pdf
https://www.uclibc.org/docs/elf-64-gen.pdf

On 4/15/21, Mike Frysinger <vapier@gentoo.org> wrote:
> On 15 Apr 2021 22:38, Peng Yu via Libc-help wrote:
>> I am wondering what is the best reference for understanding ELF
>> format. By best, I mean update-to-date, easy-to-following, and
>> complete.
>>
>> Could anybody recommend references based on this criterion? Thanks.
>
> read "Linkers & Loaders" by John R. Levine
> -mike
>


-- 
Regards,
Peng

Re: Best reference for understanding ELF format

[Szabolcs Nagy]

The 04/17/2021 07:31, Peng Yu via Libc-help wrote:
> That is a book. By definition, a book will not be complete since a
> book is not written for completeness?
> 
> For the best reference (as "reference" may have muliple meanings maybe
> I should use the word "spec" to avoid ambiguity), I found the
> following two documents. One is for 32-bit, the other is for 64-bit.
> Are they the most current ones? Other there any other documents that
> cover the things missed by them?
> 
> https://refspecs.linuxfoundation.org/elf/elf.pdf
> https://www.uclibc.org/docs/elf-64-gen.pdf

elf spec is part of the system v abi document.
the generic part is in
http://www.sco.com/developers/gabi/
the processor specific part is published separately by
whoever does the tooling for the particular processor.

(all this can be found out by 5 minute research on wikipedia)

but since you asked for easy-to-follow and for
understanding elf i think the book is better.

> On 4/15/21, Mike Frysinger <vapier@gentoo.org> wrote:
> > On 15 Apr 2021 22:38, Peng Yu via Libc-help wrote:
> >> I am wondering what is the best reference for understanding ELF
> >> format. By best, I mean update-to-date, easy-to-following, and
> >> complete.
> >>
> >> Could anybody recommend references based on this criterion? Thanks.
> >
> > read "Linkers & Loaders" by John R. Levine
> > -mike
> >
> 
> 
> -- 
> Regards,
> Peng

-- 

Re: Best reference for understanding ELF format

[Florian Weimer]

* Szabolcs Nagy via Libc-help:

> but since you asked for easy-to-follow and for
> understanding elf i think the book is better.

Agreed.  Linkers and Loaders also covers non-ELF architectures, which
sometimes helps to understand why things are in done in particular way
in ELF.

Thanks,
Florian

Re: Best reference for understanding ELF format

[Peng Yu]

On 4/19/21, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> The 04/17/2021 07:31, Peng Yu via Libc-help wrote:
>> That is a book. By definition, a book will not be complete since a
>> book is not written for completeness?
>>
>> For the best reference (as "reference" may have muliple meanings maybe
>> I should use the word "spec" to avoid ambiguity), I found the
>> following two documents. One is for 32-bit, the other is for 64-bit.
>> Are they the most current ones? Other there any other documents that
>> cover the things missed by them?
>>
>> https://refspecs.linuxfoundation.org/elf/elf.pdf
>> https://www.uclibc.org/docs/elf-64-gen.pdf
>
> elf spec is part of the system v abi document.
> the generic part is in
> http://www.sco.com/developers/gabi/
> the processor specific part is published separately by
> whoever does the tooling for the particular processor.
>
> (all this can be found out by 5 minute research on wikipedia)
>
> but since you asked for easy-to-follow and for
> understanding elf i think the book is better.

Updated is also needed. An outdated reference is not preferred even it
is easy to follow.

I found this book is quite outdated. It does not explain the current
gcc and clang.

For example, gcc mostly use RELA instead of REL. But this book says this.

"If the architecture has room for all the addends in the object code
like the x86 does, it uses REL, if not it uses RELA. But in principle
a compiler could save some space on architectures that need addends by
putting all the relocations with zero addends, e.g., procedure
references, in a SHT_REL section and the rest in a SHT_RELA."

64-bit details are not covered much.

So a better reference in terms of all three aspects that I mentioned is needed.

- up-to-date
- easy-to-follow
- complete

-- 
Regards,
Peng

Re: Best reference for understanding ELF format

[tomas]

[-- Attachment #1: Type: text/plain, Size: 417 bytes --]

On Mon, Apr 19, 2021 at 08:58:02PM -0500, Peng Yu via Libc-help wrote:

[...]

> Updated is also needed. An outdated reference is not preferred even it
> is easy to follow.

[...]

> So a better reference in terms of all three aspects that I mentioned is needed.
> 
> - up-to-date
> - easy-to-follow
> - complete

That sounds awesome. Would you like to write one? I would
read it :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Best reference for understanding ELF format

[Jeffrey Walton]

On Sat, Apr 17, 2021 at 9:15 AM Peng Yu via Libc-help
<libc-help@sourceware.org> wrote:
>
> That is a book. By definition, a book will not be complete since a
> book is not written for completeness?
>
> For the best reference (as "reference" may have muliple meanings maybe
> I should use the word "spec" to avoid ambiguity), I found the
> following two documents. One is for 32-bit, the other is for 64-bit.
> Are they the most current ones? Other there any other documents that
> cover the things missed by them?

For a spec, see Dreeper's DSO HowTo at
https://www.akkadia.org/drepper/dsohowto.pdf.

Jeff

Re: Best reference for understanding ELF format

[Mike Frysinger]

On 19 Apr 2021 20:58, Peng Yu via Libc-help wrote:
> On 4/19/21, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> > The 04/17/2021 07:31, Peng Yu via Libc-help wrote:
> >> That is a book. By definition, a book will not be complete since a
> >> book is not written for completeness?
> >>
> >> For the best reference (as "reference" may have muliple meanings maybe
> >> I should use the word "spec" to avoid ambiguity), I found the
> >> following two documents. One is for 32-bit, the other is for 64-bit.
> >> Are they the most current ones? Other there any other documents that
> >> cover the things missed by them?
> >>
> >> https://refspecs.linuxfoundation.org/elf/elf.pdf
> >> https://www.uclibc.org/docs/elf-64-gen.pdf
> >
> > elf spec is part of the system v abi document.
> > the generic part is in
> > http://www.sco.com/developers/gabi/
> > the processor specific part is published separately by
> > whoever does the tooling for the particular processor.
> >
> > (all this can be found out by 5 minute research on wikipedia)
> >
> > but since you asked for easy-to-follow and for
> > understanding elf i think the book is better.
> 
> Updated is also needed. An outdated reference is not preferred even it
> is easy to follow.

pretty much all the content in the book is still relevant.  the fact that it
doesn't cover the latest ELF extensions doesn't mean that the fundamentals are
no longer relevant.

> I found this book is quite outdated. It does not explain the current
> gcc and clang.

gcc & clang don't produce ELFs.  so, at best, they'd be mentioned in passing
even in the latest docs.

> For example, gcc mostly use RELA instead of REL. But this book says this.

gcc doesn't generate relocations.

also "mostly" is a bit skewed.  it depends on the architecture.

> So a better reference in terms of all three aspects that I mentioned is needed.

you need to learn to walk before you can hope to run.
-mike

Re: Best reference for understanding ELF format

[Peng Yu]

On 4/20/21, Mike Frysinger <vapier@gentoo.org> wrote:
> On 19 Apr 2021 20:58, Peng Yu via Libc-help wrote:
>> On 4/19/21, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>> > The 04/17/2021 07:31, Peng Yu via Libc-help wrote:
>> >> That is a book. By definition, a book will not be complete since a
>> >> book is not written for completeness?
>> >>
>> >> For the best reference (as "reference" may have muliple meanings maybe
>> >> I should use the word "spec" to avoid ambiguity), I found the
>> >> following two documents. One is for 32-bit, the other is for 64-bit.
>> >> Are they the most current ones? Other there any other documents that
>> >> cover the things missed by them?
>> >>
>> >> https://refspecs.linuxfoundation.org/elf/elf.pdf
>> >> https://www.uclibc.org/docs/elf-64-gen.pdf
>> >
>> > elf spec is part of the system v abi document.
>> > the generic part is in
>> > http://www.sco.com/developers/gabi/
>> > the processor specific part is published separately by
>> > whoever does the tooling for the particular processor.
>> >
>> > (all this can be found out by 5 minute research on wikipedia)
>> >
>> > but since you asked for easy-to-follow and for
>> > understanding elf i think the book is better.
>>
>> Updated is also needed. An outdated reference is not preferred even it
>> is easy to follow.
>
> pretty much all the content in the book is still relevant.  the fact that
> it
> doesn't cover the latest ELF extensions doesn't mean that the fundamentals
> are
> no longer relevant.
>
>> I found this book is quite outdated. It does not explain the current
>> gcc and clang.
>
> gcc & clang don't produce ELFs.

Then what produces ELFs? gcc/clang output files in ELFs. I don't
understand what you mean here.

>  so, at best, they'd be mentioned in
> passing
> even in the latest docs.
>
>> For example, gcc mostly use RELA instead of REL. But this book says this.
>
> gcc doesn't generate relocations.

Again. gcc produces files with relocations. Do you mean gcc does not
produce them directly (instead gcc some other tools to produce an ELF
file with relocations)?

> also "mostly" is a bit skewed.  it depends on the architecture.
>
>> So a better reference in terms of all three aspects that I mentioned is
>> needed.
>
> you need to learn to walk before you can hope to run.

Please provide useful information instead of nonsense like this.

-- 
Regards,
Peng

Re: Best reference for understanding ELF format

[Mike Frysinger]

On 20 Apr 2021 08:40, Peng Yu via Libc-help wrote:
> On 4/20/21, Mike Frysinger <vapier@gentoo.org> wrote:
> > On 19 Apr 2021 20:58, Peng Yu via Libc-help wrote:
> >> On 4/19/21, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> >> > The 04/17/2021 07:31, Peng Yu via Libc-help wrote:
> >> >> That is a book. By definition, a book will not be complete since a
> >> >> book is not written for completeness?
> >> >>
> >> >> For the best reference (as "reference" may have muliple meanings maybe
> >> >> I should use the word "spec" to avoid ambiguity), I found the
> >> >> following two documents. One is for 32-bit, the other is for 64-bit.
> >> >> Are they the most current ones? Other there any other documents that
> >> >> cover the things missed by them?
> >> >>
> >> >> https://refspecs.linuxfoundation.org/elf/elf.pdf
> >> >> https://www.uclibc.org/docs/elf-64-gen.pdf
> >> >
> >> > elf spec is part of the system v abi document.
> >> > the generic part is in
> >> > http://www.sco.com/developers/gabi/
> >> > the processor specific part is published separately by
> >> > whoever does the tooling for the particular processor.
> >> >
> >> > (all this can be found out by 5 minute research on wikipedia)
> >> >
> >> > but since you asked for easy-to-follow and for
> >> > understanding elf i think the book is better.
> >>
> >> Updated is also needed. An outdated reference is not preferred even it
> >> is easy to follow.
> >
> > pretty much all the content in the book is still relevant.  the fact that
> > it
> > doesn't cover the latest ELF extensions doesn't mean that the fundamentals
> > are
> > no longer relevant.
> >
> >> I found this book is quite outdated. It does not explain the current
> >> gcc and clang.
> >
> > gcc & clang don't produce ELFs.
> 
> Then what produces ELFs? gcc/clang output files in ELFs. I don't
> understand what you mean here.

that is incorrect.  gcc & clang are compiler drivers & language frontends.
they produce assembly for the assembler (which is not part of gcc or clang)
which creates object files for the linker (which is not part of gcc or clang)
which is turned into the final ELF image (e.g. program or library).

> Please provide useful information instead of nonsense like this.

you've been given so much already, and yet you balk at it.  if you aren't
going to accept valid pointers to relevant information, then stop wasting
all of our time and go annoy someone else.  like stack overflow.
-mike

Re: Best reference for understanding ELF format

[Peng Yu]

On 4/20/21, Mike Frysinger <vapier@gentoo.org> wrote:
> On 20 Apr 2021 08:40, Peng Yu via Libc-help wrote:
>> On 4/20/21, Mike Frysinger <vapier@gentoo.org> wrote:
>> > On 19 Apr 2021 20:58, Peng Yu via Libc-help wrote:
>> >> On 4/19/21, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>> >> > The 04/17/2021 07:31, Peng Yu via Libc-help wrote:
>> >> >> That is a book. By definition, a book will not be complete since a
>> >> >> book is not written for completeness?
>> >> >>
>> >> >> For the best reference (as "reference" may have muliple meanings
>> >> >> maybe
>> >> >> I should use the word "spec" to avoid ambiguity), I found the
>> >> >> following two documents. One is for 32-bit, the other is for
>> >> >> 64-bit.
>> >> >> Are they the most current ones? Other there any other documents
>> >> >> that
>> >> >> cover the things missed by them?
>> >> >>
>> >> >> https://refspecs.linuxfoundation.org/elf/elf.pdf
>> >> >> https://www.uclibc.org/docs/elf-64-gen.pdf
>> >> >
>> >> > elf spec is part of the system v abi document.
>> >> > the generic part is in
>> >> > http://www.sco.com/developers/gabi/
>> >> > the processor specific part is published separately by
>> >> > whoever does the tooling for the particular processor.
>> >> >
>> >> > (all this can be found out by 5 minute research on wikipedia)
>> >> >
>> >> > but since you asked for easy-to-follow and for
>> >> > understanding elf i think the book is better.
>> >>
>> >> Updated is also needed. An outdated reference is not preferred even it
>> >> is easy to follow.
>> >
>> > pretty much all the content in the book is still relevant.  the fact
>> > that
>> > it
>> > doesn't cover the latest ELF extensions doesn't mean that the
>> > fundamentals
>> > are
>> > no longer relevant.
>> >
>> >> I found this book is quite outdated. It does not explain the current
>> >> gcc and clang.
>> >
>> > gcc & clang don't produce ELFs.
>>
>> Then what produces ELFs? gcc/clang output files in ELFs. I don't
>> understand what you mean here.
>
> that is incorrect.  gcc & clang are compiler drivers & language frontends.
> they produce assembly for the assembler (which is not part of gcc or clang)
> which creates object files for the linker (which is not part of gcc or
> clang)
> which is turned into the final ELF image (e.g. program or library).

People don't usually directly call ld to generate object, executable
or shared library files.

Instead, gcc/clang are called. So there is nothing wrong to say those
files are produced by gcc/clang, at least at a superficially level.
You can say that gcc/clang don't directly produce the ELF files.
Nevertheless, this doesn't add too much to the topic of this thread.

>> Please provide useful information instead of nonsense like this.
>
> you've been given so much already, and yet you balk at it.

I've already give justifications why they don't satisfy the three
requirements. If there are no such resources, just say so. Don't
pretend there is one.

> if you aren't
> going to accept valid pointers to relevant information, then stop wasting
> all of our time and go annoy someone else.  like stack overflow.

-- 
Regards,
Peng

Re: Best reference for understanding ELF format

[Jeffrey Walton]

On Tue, Apr 20, 2021 at 12:29 PM Peng Yu via Libc-help
<libc-help@sourceware.org> wrote:
> ...
> People don't usually directly call ld to generate object, executable
> or shared library files.
>
> Instead, gcc/clang are called. So there is nothing wrong to say those
> files are produced by gcc/clang, at least at a superficially level.
> You can say that gcc/clang don't directly produce the ELF files.
> Nevertheless, this doesn't add too much to the topic of this thread.

I think Clang can produce object files directly. Clang has an
integrated assembler.

On OS X, sometimes you need to use -Wa,-q to enlist the Clang
Integrated Assembler because GNU toolchain is too old to consume code
for some ISA's, like AVX and AVX2. (That's assuming you're using a
modern GCC, like from Macports or Homebrew). That's why -Wa,-q shows
up in places like
https://github.com/weidai11/cryptopp/blob/master/GNUmakefile#L109.

Jeff

Re: Best reference for understanding ELF format

[Carlos O'Donell]

On Tue, Apr 20, 2021 at 12:39 PM Jeffrey Walton via Libc-help
<libc-help@sourceware.org> wrote:
>
> On Tue, Apr 20, 2021 at 12:29 PM Peng Yu via Libc-help
> <libc-help@sourceware.org> wrote:
> > ...
> > People don't usually directly call ld to generate object, executable
> > or shared library files.
> >
> > Instead, gcc/clang are called. So there is nothing wrong to say those
> > files are produced by gcc/clang, at least at a superficially level.
> > You can say that gcc/clang don't directly produce the ELF files.
> > Nevertheless, this doesn't add too much to the topic of this thread.
>
> I think Clang can produce object files directly. Clang has an
> integrated assembler.

Conceptually the point is the same though.

The compiler relies on an assembler and static linker to generate the
object files.

Whether that assembler is a library and the invocation is a library
call, or it's a secondary process via a pipe, the concept being
discussed is the same.

The compiler itself (language front end, or driver) doesn't produce
ELF format object files. Technical clarity around these distinctions
helps at a later stage when we're trying to explain other concepts
where the distinction matters or where the distinction is muddled e.g.
LTO, ThinLTO etc.

Cheers,
Carlos.

Re: Best reference for understanding ELF format

[Mike Frysinger]

On 20 Apr 2021 12:39, Jeffrey Walton via Libc-help wrote:
> On Tue, Apr 20, 2021 at 12:29 PM Peng Yu via Libc-help wrote:
> > People don't usually directly call ld to generate object, executable
> > or shared library files.
> >
> > Instead, gcc/clang are called. So there is nothing wrong to say those
> > files are produced by gcc/clang, at least at a superficially level.
> > You can say that gcc/clang don't directly produce the ELF files.
> > Nevertheless, this doesn't add too much to the topic of this thread.
> 
> I think Clang can produce object files directly. Clang has an
> integrated assembler.

the assembler is a distinct project from clang.  they're both under the llvm
umbrella, but still distinct.  that clang has an option to statically link a
tool into itself rather than invoke it via fork+exec+pipes is a bit immaterial
to the overall division of duties.  i could statically link a C library into
clang too, but i wouldn't claim that clang provides a C library.
-mike

Re: Best reference for understanding ELF format

[Mike Frysinger]

On 20 Apr 2021 10:12, Peng Yu via Libc-help wrote:
> On 4/20/21, Mike Frysinger <vapier@gentoo.org> wrote:
> > On 20 Apr 2021 08:40, Peng Yu via Libc-help wrote:
> >> On 4/20/21, Mike Frysinger <vapier@gentoo.org> wrote:
> >> > On 19 Apr 2021 20:58, Peng Yu via Libc-help wrote:
> >> >> I found this book is quite outdated. It does not explain the current
> >> >> gcc and clang.
> >> >
> >> > gcc & clang don't produce ELFs.
> >>
> >> Then what produces ELFs? gcc/clang output files in ELFs. I don't
> >> understand what you mean here.
> >
> > that is incorrect.  gcc & clang are compiler drivers & language frontends.
> > they produce assembly for the assembler (which is not part of gcc or clang)
> > which creates object files for the linker (which is not part of gcc or
> > clang)
> > which is turned into the final ELF image (e.g. program or library).
> 
> People don't usually directly call ld to generate object, executable
> or shared library files.

you might not, but you are not all people.  glibc def invokes it directly.

> Instead, gcc/clang are called. So there is nothing wrong to say those
> files are produced by gcc/clang, at least at a superficially level.
> You can say that gcc/clang don't directly produce the ELF files.
> Nevertheless, this doesn't add too much to the topic of this thread.

you simultaneously claim you want in-depth detail for how ELFs work and how
they're produced, and you don't want to understand how the toolchains work.
this is like saying you want to know how the sausage is made, but you don't
want to hear about the cows being slaughtered & ground up, you just want to
eat the hotdog.

toolchains have well segmented duties spread between projects.  if you were
to go ask the gcc folks about ELF relocations, they'd (correctly) tell you
you're at the wrong place: you want to go ask the binutils projects instead
as they produce the assembler & linker & related tools.

> >> Please provide useful information instead of nonsense like this.
> >
> > you've been given so much already, and yet you balk at it.
> 
> I've already give justifications why they don't satisfy the three
> requirements. If there are no such resources, just say so. Don't
> pretend there is one.

sounds good: everything you want wrapped up in a bow doesn't exist.
feel free to write it yourself and then get it published on amazon.
-mike

Re: Best reference for understanding ELF format

[Peng Yu]

On Tue, Apr 20, 2021 at 11:51 AM Carlos O'Donell
<carlos@systemhalted.org> wrote:
>
> On Tue, Apr 20, 2021 at 12:39 PM Jeffrey Walton via Libc-help
> <libc-help@sourceware.org> wrote:
> >
> > On Tue, Apr 20, 2021 at 12:29 PM Peng Yu via Libc-help
> > <libc-help@sourceware.org> wrote:
> > > ...
> > > People don't usually directly call ld to generate object, executable
> > > or shared library files.
> > >
> > > Instead, gcc/clang are called. So there is nothing wrong to say those
> > > files are produced by gcc/clang, at least at a superficially level.
> > > You can say that gcc/clang don't directly produce the ELF files.
> > > Nevertheless, this doesn't add too much to the topic of this thread.
> >
> > I think Clang can produce object files directly. Clang has an
> > integrated assembler.
>
> Conceptually the point is the same though.
>
> The compiler relies on an assembler and static linker to generate the
> object files.
>
> Whether that assembler is a library and the invocation is a library
> call, or it's a secondary process via a pipe, the concept being
> discussed is the same.
>
> The compiler itself (language front end,

I think the "front end" is actually the cc1 program which is not
exposed to PATH by default for gcc. In my system, it is the following
one, which transforms a .c file to a .s file.

/usr/lib/gcc/x86_64-linux-gnu/10/cc1

Then the .s file is processed by as, to generate a .o file. Then, the
final a.out file is generated from the .o file by ld.

The command gcc is just a wrapper of the three programs (cc1/as/ld).
The following is a minimal work example that demonstrates this.

$ cat main.c
#include <stdio.h>
int main() {
    puts("Hello World!");
    return 0;
}
$ /usr/lib/gcc/x86_64-linux-gnu/10/cc1 -quiet main.c -o main.s
$ as -o main.o main.s
$ ld -dynamic-linker /lib64/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o
/usr/lib/x86_64-linux-gnu/crtn.o main.o -lc

> or driver) doesn't produce
> ELF format object files. Technical clarity around these distinctions
> helps at a later stage when we're trying to explain other concepts
> where the distinction matters or where the distinction is muddled e.g.
> LTO, ThinLTO etc.

As these (LTO, ThinLTO) specific to the llvm pipeline?

-- 
Regards,
Peng