dwfl_module_relocate_address() versus base address

dwfl_module_relocate_address() versus base address

[Mark Wielaard]

Hi,

I am slightly confused when dwfl_module_relocate_address() strips off
the base address and when not. In translate.cxx we have
dump_unwindsyms() which is the callback for dwfl_getmodules(). We are
trying to build a symbol table here that is relative to the base
address. We do this by calling dwfl_module_relocate_address(), which
will return the address without the given base, but only for shared
libraries where there is a dwbias (according to dwfl_module_info()).
This is the case for example for glibc. But for other shared libraries
(*) dwbias is -1 and so the address isn't made relative to the base
address. So we end up in stap-symbols.h with some shared library modules
having a symbol section that is relative to the module base address and
others that aren't.

To work around this I now have the following patch which seems to work
in all cases, but I am not completely clear why:

diff --git a/translate.cxx b/translate.cxx
index 27f6a04..88f01ac 100644
--- a/translate.cxx
+++ b/translate.cxx
@@ -4523,9 +4523,17 @@ dump_unwindsyms (Dwfl_Module *m,
 
               if (n > 0) // only try to relocate if there exist relocation base
                 {
+                  Dwarf_Addr dwbias;
                   int ki = dwfl_module_relocate_address (m, &sym_addr);
                   dwfl_assert ("dwfl_module_relocate_address", ki >= 0);
                   secname = dwfl_module_relocation_info (m, ki, NULL);
+
+                  // Check whether the relocation took dwbias into account,
+                  // if not, we need to adjust the address by hand.
+                  dwfl_module_info (m, NULL, NULL, NULL, &dwbias,
+                                    NULL, NULL, NULL);
+                  if (dwbias == (Dwarf_Addr) -1)
+                    extra_offset = base;
                 }
 
               if (n == 1 && modname == "kernel")


If anybody could enlighten me that would be appreciated.

Cheers,

Mark

(*) A simple reproducer, without the above patch applied, is compiling
this little library:

$ gcc usymbols_lib.c -fPIC -shared -o libusymbols.so

And doing:
$ stap -k -d /lib/libc.so.6 -d `pwd`/libusymbols.so \
	-e 'probe begin {exit()}'

Where usymbols_lib.c contains just one function like:

void lib_main () {}

You will see that the stap-symbols.h file will contain libusymbols
addresses that have dwarf_module_base added, while the libc symbol
addresses don't have those.

Re: dwfl_module_relocate_address() versus base address

[Mark Wielaard]

On Fri, 2008-12-12 at 18:37 +0100, Mark Wielaard wrote:
> (*) A simple reproducer, without the above patch applied, is compiling
> this little library:
> 
> $ gcc usymbols_lib.c -fPIC -shared -o libusymbols.so
> 
> And doing:
> $ stap -k -d /lib/libc.so.6 -d `pwd`/libusymbols.so \
> 	-e 'probe begin {exit()}'
> 
> Where usymbols_lib.c contains just one function like:
> 
> void lib_main () {}
> 
> You will see that the stap-symbols.h file will contain libusymbols
> addresses that have dwarf_module_base added, while the libc symbol
> addresses don't have those.

So just for reference. This is what you will get:

struct _stp_symbol _stp_module_1_symbols_0[] = {
  { 0x200428, "_init" },
  { 0x200460, "call_gmon_start" },
  { 0x200480, "__do_global_dtors_aux" },
  { 0x200500, "frame_dummy" },
  { 0x200534, "lib_main" },
  { 0x200540, "__do_global_ctors_aux" },
  { 0x200578, "_fini" },
};
struct _stp_section _stp_module_1_sections[] = {
{
.name = ".dynamic",
.symbols = _stp_module_1_symbols_0,
.num_symbols = sizeof(_stp_module_1_symbols_0)/sizeof(struct
_stp_symbol)
},
};
struct _stp_module _stp_module_1 = {
.name = "/tmp/libusymbols.so",
.dwarf_module_base = 0x200000,
[...]

Note how the dwarf_module_base is included in the addresses.
But for libc you will get:

struct _stp_symbol _stp_module_0_symbols_0[] = {
  { 0x7e74b0, "__libc_global_ctors" },
  { 0x7e74e0, "__libc_fini" },
  { 0x7e750f, "__i686.get_pc_thunk.bx" },
  { 0x7e7520, "__libc_init_first" },
  { 0x7e7530, "_dl_start" },
  { 0x7e7540, "_init" },
[...]
};
struct _stp_section _stp_module_0_sections[] = {
{
.name = ".dynamic",
.symbols = _stp_module_0_symbols_0,
.num_symbols = sizeof(_stp_module_0_symbols_0)/sizeof(struct
_stp_symbol)
},
};
struct _stp_module _stp_module_0 = {
.name = "/lib/libc.so.6", 
.dwarf_module_base = 0x7d1000, 

So the dwarf base is already extracted from the symbol addresses.

Re: dwfl_module_relocate_address() versus base address

[Roland McGrath]

When dwfl_module_info yields a dwbias of -1, that means there is no DWARF
info known (yet); also the debug file name (last arg) would be NULL.  This
means you haven't asked for DWARF for this module yet, or that no debuginfo
was found.

Important things to note when looking at this stuff are the phdrs
(eu-readelf -l) of the DSOs in question.  The big likely differences
between libc and your example is that libc might be prelinked, and
that it has a separate debuginfo file.

When prelinked, the base bias (phdrs vs runtime address) usually winds up
0, whereas a plain DSO always has a nonzero base.

When debuginfo is separate, then the dwbias can be different when the DSO
was prelinked, because the addresses in the .debug file (i.e. in DWARF, and
in symtabs if using .debug's symtab rather than stripped-file's symtab)
were not changed by prelink though the DSO itself was changed.

I think dwfl_module_relocate_address might be applying the wrong bias.  It
subtracts the dwbias, but it should subtract the main file bias.  The
address you pass it is an absolute address in that Dwfl's address space,
and what it should yield is the address relative to the first p_vaddr in
the DSO (i.e. 0 if not prelinked).

diff --git a/libdwfl/derelocate.c b/libdwfl/derelocate.c
index 402bc06..7f390c7 100644
--- a/libdwfl/derelocate.c
+++ b/libdwfl/derelocate.c
@@ -358,7 +358,7 @@ dwfl_module_relocate_address (Dwfl_Module *mod, Dwarf_Addr *addr)
 
   if (mod->e_type != ET_REL)
     {
-      *addr -= mod->debug.bias;
+      *addr -= mod->main.bias;
       return 0;
     }


Thanks,
Roland

Re: dwfl_module_relocate_address() versus base address

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

Hi Roland,

On Fri, 2008-12-12 at 12:06 -0800, Roland McGrath wrote:
> I think dwfl_module_relocate_address might be applying the wrong bias.  It
> subtracts the dwbias, but it should subtract the main file bias.  The
> address you pass it is an absolute address in that Dwfl's address space,
> and what it should yield is the address relative to the first p_vaddr in
> the DSO (i.e. 0 if not prelinked).

Aha! Doh. Yes, things make a lot more sense now. Since older/current
versions of dwfl_module_relocate_address() will have this bug I am using
dwfl_module_address_section() now, which does seem to do the right
thing. Then to get the relative address of the symbol when the code is
mapped in all we need is the section offset. This seems to work fine in
all cases. Although I am still confused how this ever worked before.
Also for the kernel we still do things slightly differently (making
things relative to _stext) . I checked in the following patch to my
pr6866 branch and will be testing more with it.

Thanks,

Mark

[-- Attachment #2: translate.patch --]
[-- Type: text/x-patch, Size: 1481 bytes --]

diff --git a/translate.cxx b/translate.cxx
index 27f6a04..c60d2da 100644
--- a/translate.cxx
+++ b/translate.cxx
@@ -4521,11 +4521,24 @@ dump_unwindsyms (Dwfl_Module *m,
               Dwarf_Addr sym_addr = sym.st_value;
               const char *secname = NULL;
 
-              if (n > 0) // only try to relocate if there exist relocation bases
+              // only try to relocate if there exist relocation bases
+              // and not for the kernel, we readjust against _stext,
+              // through extra_offset above.
+              if (n > 0 && modname != "kernel")
                 {
-                  int ki = dwfl_module_relocate_address (m, &sym_addr);
-                  dwfl_assert ("dwfl_module_relocate_address", ki >= 0);
-                  secname = dwfl_module_relocation_info (m, ki, NULL);
+		  // libdwfl up to 0.137 could miscalculate the relocated
+		  // address for shared ET_DYN files (n > 0). Luckily
+		  // dwfl_module_address_section does work correctly.
+		  // Then we just need to add the section offset to get
+		  // the (relative) address of the symbol when mapped in.
+		  Dwarf_Addr bias;
+		  Elf_Scn *sec;
+		  GElf_Shdr *shdr, shdr_mem;
+		  sec = dwfl_module_address_section (m, &sym_addr, &bias);
+		  dwfl_assert ("dwfl_module_address_section", sec != NULL);
+		  shdr = gelf_getshdr(sec, &shdr_mem);
+		  sym_addr += shdr->sh_offset;
+		  secname = "";
                 }
 
               if (n == 1 && modname == "kernel")

Re: dwfl_module_relocate_address() versus base address

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]

Hi,

On Mon, 2008-12-15 at 13:06 +0100, Mark Wielaard wrote:
> Also for the kernel we still do things slightly differently (making
> things relative to _stext) . I checked in the following patch to my
> pr6866 branch and will be testing more with it.

This is not completely true. I was confused since we also rely on the
secname produced by dwfl_module_relocation_info() (being empty for
ET_DYN). So that is now also replicated by hand in the new patch. It is
getting slightly messy now :{

Cheers,

Mark

[-- Attachment #2: translate.patch --]
[-- Type: text/x-patch, Size: 1394 bytes --]

diff --git a/translate.cxx b/translate.cxx
index 27f6a04..bb976d5 100644
--- a/translate.cxx
+++ b/translate.cxx
@@ -4523,9 +4523,26 @@ dump_unwindsyms (Dwfl_Module *m,
 
               if (n > 0) // only try to relocate if there exist relocation bases
                 {
-                  int ki = dwfl_module_relocate_address (m, &sym_addr);
-                  dwfl_assert ("dwfl_module_relocate_address", ki >= 0);
-                  secname = dwfl_module_relocation_info (m, ki, NULL);
+		  // libdwfl up to 0.137 could miscalculate the relocated
+		  // address for shared ET_DYN files (n > 0). Luckily
+		  // dwfl_module_address_section does work correctly.
+		  // Then we just need to add the section offset to get
+		  // the (relative) address of the symbol when mapped in.
+		  Dwarf_Addr bias;
+		  Elf_Scn *sec;
+		  GElf_Shdr *shdr, shdr_mem;
+		  GElf_Ehdr *ehdr, ehdr_mem;
+		  Elf *elf;
+		  sec = dwfl_module_address_section (m, &sym_addr, &bias);
+		  dwfl_assert ("dwfl_module_address_section", sec != NULL);
+		  shdr = gelf_getshdr(sec, &shdr_mem);
+		  sym_addr += shdr->sh_offset;
+		  elf = dwfl_module_getelf (m, &bias);
+		  ehdr = gelf_getehdr(elf, &ehdr_mem);
+		  if (ehdr->e_type == ET_DYN)
+		    secname = "";
+		  else
+		    secname = elf_strptr (elf, ehdr->e_shstrndx, shdr->sh_name);
                 }
 
               if (n == 1 && modname == "kernel")

Re: dwfl_module_relocate_address() versus base address

[Roland McGrath]

> Aha! Doh. Yes, things make a lot more sense now. Since older/current
> versions of dwfl_module_relocate_address() will have this bug I am using
> dwfl_module_address_section() now, which does seem to do the right
> thing. 

Note that for ET_REL that call also incurs a big cost you don't otherwise
need (applying relocs to the section data).  

> Although I am still confused how this ever worked before.

The buggy code only affects ET_DYN(-like) with separate debuginfo, not
ET_REL.  It probably wasn't noticed before because usually vmlinux is not
separate debuginfo, it's a whole unstripped copy, and .ko are ET_REL.

> Also for the kernel we still do things slightly differently (making
> things relative to _stext) .

The kernel is treated as ET_DYN by libdwfl (hence the -like).  Though the
file says ET_EXEC, it really behaves like ET_DYN on CONFIG_RELOCATABLE=y
kernels.  This is just FYI, I guess.

> This is not completely true. I was confused since we also rely on the
> secname produced by dwfl_module_relocation_info() (being empty for
> ET_DYN). So that is now also replicated by hand in the new patch. It is
> getting slightly messy now :{

Since this code does getelf anyway (which, incidentally, imposes all the
costs of dwfl_module_address_section on every section), you can just do a
different workaround.  You can even skip the bad overhead, because you
don't need to do any workaround for ET_REL files anyway:

	if (ki == 0 && secname != NULL && secname[0] == '\0')
	  {
	    check for bug
	  }

So, you can do getelf + getdwarf, to get mainbias and dwbias, respectively.
If mainbias == dwbias, you are already correct.  To check for the bug,
you'd see if the original sym_addr and the sym_addr adjusted by
dwfl_module_relocate_address mainbias or by dwbias.  But, if you just want
to work right rather than test for the bug, you can just do:

	Dwarf_Addr save_addr = sym_addr;
	int ki = dwfl_module_relocate_address (m, &sym_addr);
	dwfl_assert ("dwfl_module_relocate_address", ki >= 0);
	secname = dwfl_module_relocation_info (m, ki, NULL);
	if (ki == 0 && secname != NULL && secname[0] == '\0')
	  {
	    Dwarf_Addr bias;
	    dwfl_assert ("dwfl_getelf", dwfl_module_getelf (m, &bias) != NULL);
	    sym_addr = save_addr - bias;
	  }


Thanks,
Roland

Re: dwfl_module_relocate_address() versus base address

[Mark Wielaard]

Hi Roland,

On Tue, 2008-12-16 at 01:11 -0800, Roland McGrath wrote:
> Since this code does getelf anyway (which, incidentally, imposes all the
> costs of dwfl_module_address_section on every section), you can just do a
> different workaround.  You can even skip the bad overhead, because you
> don't need to do any workaround for ET_REL files anyway:
> 
> 	if (ki == 0 && secname != NULL && secname[0] == '\0')
> 	  {
> 	    check for bug
> 	  }
> 
> So, you can do getelf + getdwarf, to get mainbias and dwbias, respectively.
> If mainbias == dwbias, you are already correct.  To check for the bug,
> you'd see if the original sym_addr and the sym_addr adjusted by
> dwfl_module_relocate_address mainbias or by dwbias.  But, if you just want
> to work right rather than test for the bug, you can just do:
> 
> 	Dwarf_Addr save_addr = sym_addr;
> 	int ki = dwfl_module_relocate_address (m, &sym_addr);
> 	dwfl_assert ("dwfl_module_relocate_address", ki >= 0);
> 	secname = dwfl_module_relocation_info (m, ki, NULL);
> 	if (ki == 0 && secname != NULL && secname[0] == '\0')
> 	  {
> 	    Dwarf_Addr bias;
> 	    dwfl_assert ("dwfl_getelf", dwfl_module_getelf (m, &bias) != NULL);
> 	    sym_addr = save_addr - bias;
> 	  }

Thanks again. Slowly I will actually understand what libdwfl does :)

But this is not fully working for me against my libc. Lets trace what I
do and what this workaround does (aka what dwfl_module_relocate_address
is supposed to do). I might just not fully comprehend the address
adjustments that both these functions are doing.

What I want is the address of a symbol relative to the start of where
the elf segment is (will be) loaded in memory.

What dwfl_module_relocate_address gives me is the address compensated
for the main elf file bias. That is the difference between the v_addr
and p_addr of the segment. That seems fine, if the v_addr is zero.
However for my libc, the bias is zero (v_addr and p_addr are equal), but
they are non-zero (they are both 0x0000003688a00040 in my copy). So then
I need to hand adjust for the v_addr afterwards to get the (relative)
address I want. (This is the dwfl_module_info start address [so without
bias], which I subtract after the dwfl_module_relocate_address
call/workaround - actually I just subtract the start address and call it
a day since we don't care about the bias in this case.)

What dwfl_module_relocate_address gives me is the address relative to
the start of the elf section that the symbol lives in. That would be
fine if that section starts at the beginning of the segment to be mapped
in. Otherwise I need to adjust for the section offset (which is why I
got the shdr from the section returned by dwfl_module_relocate_address
in my original patch to add sh_off).

Does the above make sense? Does the above description explain what these
functions are supposed to do to the given address? And are the fixups
that I need do expected for my particular use case?

Thanks,

Mark

Re: dwfl_module_relocate_address() versus base address

[Roland McGrath]

> What dwfl_module_relocate_address gives me is the address compensated
> for the main elf file bias. That is the difference between the v_addr
> and p_addr of the segment. That seems fine, if the v_addr is zero.

p_paddr is wholly irrelevant to anything you'll ever do.  Ignore it.  
(Its only use is in loading the kernel before MMU hardware is enabled.)

The main ELF file's bias is the difference between what the file's headers
like p_vaddr and sh_addr say and what the actual address is in the address
space described by your Dwfl object.

> However for my libc, the bias is zero (v_addr and p_addr are equal), but
> they are non-zero (they are both 0x0000003688a00040 in my copy). 

That is normal for a prelinked DSO.  The true bias is the difference
between 0x0000003688a00040 and where that PT_LOAD actually sits in the
runtime view of your Dwfl object.  (For offline processing, that's an
arbitrary address for the arbitrary layout chosen by libdwfl.  In live
processing, it would be the actual address where the DSO got mapped.)

> So then I need to hand adjust for the v_addr afterwards to get the
> (relative) address I want. (This is the dwfl_module_info start address
> [so without bias], which I subtract after the
> dwfl_module_relocate_address call/workaround - actually I just subtract
> the start address and call it a day since we don't care about the bias in
> this case.)

The module start address is not useful here.  If the DSO's lowest p_vaddr
is 0 (i.e., it's not prelinked), then the module start address will be the
same as the main file bias.  But that does not apply to your case, which
illustrates why it is never the right term to use in a bias calculation.

The dwarf_module_getelf call gives you the main file's bias.  This is what
you need to adjust a p_vaddr/sh_addr address by to make it an absolute
address in your Dwfl's space (or vice versa, in your use here).

> What dwfl_module_relocate_address gives me is the address relative to
> the start of the elf section that the symbol lives in. 

No, it's not.  It gives you the "relocation base index", and the address
relative to that base.  In an ET_REL file, the relocation bases are ELF
sections, so there are often > 1 of them.  In an ET_DYN(-like) file, there
is only one relocation base and it's the main file's bias.

Note that the relocation base indices are tightly-packed (so you can use an
array of the size returned by dwfl_module_relocations with no waste).  A
relocation base index is purely a small number to pass to
dwfl_module_relocation_info, and is not the same as an ELF section index.

dwfl_module_relocation_info tells you the ELF section index and name for a
relocation base index.  For ET_REL this is a proper index and that
section's name.  For ET_DYN(-like) there is only one valid relocation base
index (0), for which dwfl_module_relocation_info yields SHN_ABS and "".

> That would be fine if that section starts at the beginning of the segment
> to be mapped in. Otherwise I need to adjust for the section offset (which
> is why I got the shdr from the section returned by
> dwfl_module_relocate_address in my original patch to add sh_off).

You never need this.  (sh_offset is the sections's file position, and is
never relevant to any of this.)  dwfl_module_address_section yields the
bias to apply to the sh_addr in the returned section's shdr.  Hence
gelf_getshdr(scn)->sh_addr + bias yields the absolute address of the start
of that section's contents in the Dwfl address space.

Is that all clear?


Thanks,
Roland

Re: dwfl_module_relocate_address() versus base address

[Mark Wielaard]

Hi Roland,

On Tue, 2008-12-16 at 15:05 -0800, Roland McGrath wrote:
> The main ELF file's bias is the difference between what the file's
> headers like p_vaddr and sh_addr say and what the actual address is in
> the address space described by your Dwfl object.
>[...]
> The dwarf_module_getelf call gives you the main file's bias.  This is
> what you need to adjust a p_vaddr/sh_addr address by to make it an
> absolute address in your Dwfl's space (or vice versa, in your use
> here).
>[...]
> dwfl_module_relocation_info tells you the ELF section index and name
> for a relocation base index.  For ET_REL this is a proper index and
> that section's name.  For ET_DYN(-like) there is only one valid
> relocation base index (0), for which dwfl_module_relocation_info
> yields SHN_ABS and "".
>[...]
>Is that all clear?

So many addresses spaces, biases, offsets and elf/section/program
headers to get wrong :{ And I see in my write up I actually switched
dwfl_module_relocation_info and dwfl_module_address_section in a couple
of places, sigh. But yes, I think it is all clear now. Thanks.

The one remaining issue is finding the correct p_vaddr. I must be
missing the obvious. Currently I am just going through all program
headers to try and find the correct one (after adjusting sym_addr for
the elf main file bias):

      ehdr = gelf_getehdr(elf, &ehdr_mem);
      for (idx = 0; idx < ehdr->e_phnum; idx++)
        {
          phdr = gelf_getphdr (elf, idx, &phdr_mem);
          if (phdr != NULL
              && phdr->p_type == PT_LOAD
              && phdr->p_flags & PF_X
              && sym_addr >= phdr->p_vaddr
              && sym_addr < phdr->p_vaddr + phdr->p_memsz)
            break;
        }
      if (phdr != NULL && idx < ehdr->e_phnum)
        sym_addr -= phdr->p_vaddr;

Is there a simpler way?

Cheers,

Mark

Re: dwfl_module_relocate_address() versus base address

[Roland McGrath]

> The one remaining issue is finding the correct p_vaddr. I must be
> missing the obvious. Currently I am just going through all program
> headers to try and find the correct one (after adjusting sym_addr for
> the elf main file bias):

Something is not all clear to at least one of us, because I have no idea
why you think you need to do anything like this.

Thanks,
Roland

Re: dwfl_module_relocate_address() versus base address

[Mark Wielaard]

Hi Roland,

On Wed, 2008-12-17 at 15:08 -0800, Roland McGrath wrote:
> > The one remaining issue is finding the correct p_vaddr. I must be
> > missing the obvious. Currently I am just going through all program
> > headers to try and find the correct one (after adjusting sym_addr for
> > the elf main file bias):
> 
> Something is not all clear to at least one of us, because I have no idea
> why you think you need to do anything like this.

OK. Lets go to the start and see if from that it does or doesn't follow
that I want to adjust for the p_vaddr of the segment that the address is
in to get a relative offset.

What we do at translation time is create a table of symbol addresses
that we can look up against at run time so that when we see an address
in a segment that was mapped in we can use that lookup table for that
module and provide the corresponding symbol name. (This was my earlier
work mapping the vma table from the task finder to the corresponding
_stp_module). Since we don't know where the segment will be mapped into
the user address space beforehand we need the symbol addresses to be
relative to the start of segment that gets loaded.

So to get these relative addresses for shared libraries for each
stp_module in translate.cxx we have dump_unwindsyms() which is the
callback for dwfl_getmodules(). To build up a symbol table (as put in
stap-symbols.h) that is relative to the start of where the elf segment
is (will be) loaded in memory we do the following there:
- dwfl_module_getsymtab() to get the number of symbols in the module.
  Then for each symbol:
  - dwfl_module_getsym() to get the symbol values.
  - dwfl_module_relocate_address() to adjust the address to the base.
  - dwfl_module_relocation_info() to get the base information
    (so we know the section name, etc so we know whether it is a
     dynamic address.)
  - Get the p_vaddr of the segment that the symbol address is in
    so we can make the address relative to the segment load address.

It is the last step that seems a bit cumbersome. So if we can come up
with a way to avoid it that would be nice.

Cheers,

Mark

Re: dwfl_module_relocate_address() versus base address

[Roland McGrath]

> OK. Lets go to the start and see if from that it does or doesn't follow
> that I want to adjust for the p_vaddr of the segment that the address is
> in to get a relative offset.

We can start out by saying that it doesn't, because libdwfl's purpose in
life is to make this easier for you than that.  Of course, that faith might
imply that libdwfl is broken or wrong-headed or needs more calls to help you.
But you happen to have its authors at your beck and call, so your first
answer to such issues can always be, "Hey libdwfl, help me over here!"
That's what it's there for.  (And then once we've resolved that, we can
figure out if you need a workaround/backport solution for an interim while
waiting for libdwfl improvements.)

> [...] Since we don't know where the segment will be mapped into
> the user address space beforehand we need the symbol addresses to be
> relative to the start of segment that gets loaded.

What you need is addresses that you know how you'll adjust at runtime.

Some useful facts:

The ELF format spec says about PT_LOAD phdrs: "Loadable segment entries in
the program header table appear in ascending order, sorted on the p_vaddr
member."  This means the first PT_LOAD segment is the lowest-addressed,
unless the ELF file is invalid (and then all bets are off anyway).  (In
actual fact, Linux ELF files have phdrs sorted by p_paddr.  This is the
desireable thing for directing boot loaders to load the kernel into
physical memory.  It never affects a case you're thinking about, because in
all normal binaries, p_paddr==p_vaddr, so the effect is that Linux ELF
files follow the spec in ordering by p_vaddr except for the special case of
the kernel binary, which is special in plenty of other ways too.)

The lowest-addressed (first) PT_LOAD segment (phdrs[i]) is placed at some
runtime-chosen address (p) regardless of its given p_vaddr.  Each
successive PT_LOAD (phdrs[j]) is then placed exactly at:
	phdrs[j].p_vaddr - phdrs[i].p_vaddr + p
This is the only way that ld.so behaves, even on ia64 where the ABI spec is
looser.  (Any intervening holes in the address space are filled with
PROT_NONE pages.  This comes up often on x86_64, where ABI p_align is 2MB
but actual page size is normally 4KB.  In actual fact, what ld.so does is
reserve the whole range from p to PAGE_ALIGN(phdrs[last_j].p_vaddr +
phdrs[last_j].p_memsz - phdrs[i].p_vaddr + p) in the mmap call that loads
the first segment (made without MAP_FIXED), and then calls mmap with
MAP_FIXED for each successive segment (splitting/overwriting the initial
mapping just made), interleaving mprotect calls for each hole of a whole
page or more, to set those to PROT_NONE.)

(Those asides had lots of detail not directly apropos, but I know you often
find the gritty details helpfully elucidating.)  The upshot is that for
ET_DYN libraries it makes sense to think of a single "load address" for the
library, being the address at which its lowest-addressed PT_LOAD segment
was actually loaded.  The "ELF bias" is the difference between that address
and the p_vaddr of that PT_LOAD.  (In a normal DSO, that p_vaddr is 0, so
the bias is exactly the load address.  In a prelinked DSO, that p_vaddr is
nonzero, and the actual load address may be equal, greater, or lesser.)

In libdwfl, the "module start" address for a DSO module is that load address.

> - dwfl_module_getsymtab() to get the number of symbols in the module.
>   Then for each symbol:
>   - dwfl_module_getsym() to get the symbol values.

This yields an absolute address (st_value) in the Dwfl address space.  In
offline mode (which stap always uses), that means an arbitrary placement.

>   - dwfl_module_relocate_address() to adjust the address to the base.

Modulo the recent bug, this yields an address R such that:

	B + R = st_value

where B is the "relocation base" for which dwfl_module_relocate_address
returns the "relocation base index".

>   - dwfl_module_relocation_info() to get the base information
>     (so we know the section name, etc so we know whether it is a
>      dynamic address.)

For ET_DYN, this yields SHN_ABS, "" to describe the relocation base (with
index 0, the only valid index in this case).  This means that the relocation
base is the module's start address (so sayeth the libdwfl.h comment).

For DSOs that weren't prelinked, and for DSOs that were prelinked after
separating their debuginfo (i.e. prelinking of a package-installed DSO, the
normal case for libc et al), the old code returned this correctly.  The
debuginfo (or unprelinked original) is relative to load address 0, so the
debug.bias is exactly the load address.  The fix I did the other day was in
fact wrong for prelinked DSOs.  The right fix is to subtract low_addr (the
module start address), not main.bias.  (main.bias == low_addr for a DSO
that was not prelinked.)

http://git.fedorahosted.org/git/elfutils.git?p=elfutils.git;a=commitdiff;h=7d9b821db6bc494417a57321b419c6b9481a2128

>   - Get the p_vaddr of the segment that the symbol address is in
>     so we can make the address relative to the segment load address.
> 
> It is the last step that seems a bit cumbersome. So if we can come up
> with a way to avoid it that would be nice.

You are not supposed to need this, since a fixed
dwfl_module_relocate_address wlll yield the right relative address in the
first place.  To work around the old one, you only need to do what the
fixed one does.  That is, take the absolute address (the input to
dwfl_module_relocate_address) and substract the module start address.
This address is an argument to all module callback functions,
and also queryable with dwfl_module_info.

Now you have an address relative to a base intended to be straightforward
for you.  For ET_DYN, this means the load address of the DSO, which is the
start of the lowest-addressed mapping of the file's offset 0.  If you were
later also using libdwfl at runtime, then your runtime Dwfl would be
populated as by dwfl_linux_proc_report, so the module start addresses for
DSO modules would yield what you want.  (This is the whole idea of why the
offline/relocation interfaces are supposed to be intuitive to use, which
evidently hasn't worked out so well, though perhaps that's really only due
to the bugs leading you astray.)  Of course you're not using it, since your
runtime reality is in kernel code.  But the hope was that this view of it
makes natural sense to what you're looking at.  i.e., compare your runtime
layout tracking to eu-unstrip -n -p PID, which shows you the libdwfl view
of things at runtime.  Also compare that to eu-unstrip -n -e foo.so, which
is the same case the stap translator is using internally when resolving a
user module.  (Or -k to compare the kernel case to the offline kernel case,
which is -K, the same case the stap translator is using internally.)

This stuff really hasn't been exercised much for the user-mode ET_DYN cases
(i.e. coping with all permutations of prelinking and separate debuginfo).
So there might well be more bugs in there.

I've been working through a cold this week, so I can't entirely vouch for
the clarity of my thinking either yesterday or today (and they don't match).
But I hope now I've given enough information in enough corners that you can
validate for yourself whether what I've said last is really true.

(I haven't gone into detail on the ET_REL cases in this message.  But those
are already dealt with fine AFAIK.  I can elaborate those differences if
you ask.)


Thanks,
Roland