MIPS and -msym32

MIPS and -msym32

[Crestez Dan Leonard]

Hello,

I've been playing around with systemtap on mips.

I used the elfutils mips patch from debian:
	http://sources.debian.net/src/elfutils/0.159-4/debian/patches/mips_backend.diff

For systemtap I ported some old patches from cisco on top of release 2.5. Here is the link to the patches on top of 1.6:
	https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=log;h=refs/heads/systemtap-1.6-cisco-patches

One of the issues I encountered is that by default 64-bit mips kernels are compiled with -msym32. This causes gcc to emit dwarf information with a pointer size of 4, even though the file is 64bit. In theory this can be disabled by passing KBUILD_SYM32=no to the kernel make commandline. This is a bad idea. It would disable some optimizations which have been enabled on mips for many years. I actually tried to do it locally but it broke module loading and I was unable to boot.

The generated dwarf files confuse systemtap is multiple ways. 

When fetching some parameters systemtap relies on address_size from dwarf_diecu to determine max_fetch_size. This caused an error like "semantic error: single register too big for fetch/store ???: identifier '$data' at ..." for an unsigned long variable.

Here is a hack I used to get around this:
--- a/libdw/dwarf_diecu.c
+++ b/libdw/dwarf_diecu.c
@@ -47,7 +47,22 @@ dwarf_diecu (die, result, address_sizep, offset_sizep)
   *result = CUDIE (die->cu);
 
   if (address_sizep != NULL)
+  {
     *address_sizep = die->cu->address_size;
+    /* Hack: */
+    if (1)
+    {
+      struct Elf *elf = die->cu->dbg->elf;
+      GElf_Ehdr ehdr_mem;
+      GElf_Ehdr* ehdr = gelf_getehdr (elf, &ehdr_mem);
+      if (ehdr &&
+              ehdr->e_machine == EM_MIPS &&
+              ehdr->e_ident[EI_CLASS] == ELFCLASS64)
+      {
+        *address_sizep = 8;
+      }
+    }
+  }
   if (offset_sizep != NULL)
     *offset_sizep = die->cu->offset_size;

This is obviously evil. What is worse is that -msym32 seems to break systemtap address mapping. Earlier I had to do the following hack inside systemtap.

diff --git a/tapsets.cxx b/tapsets.cxx
index 4e42403..94ee230 100644
--- a/tapsets.cxx
+++ b/tapsets.cxx
@@ -1265,6 +1265,22 @@ dwarf_query::add_probe_point(const string& dw_funcname,
       else
         {
           assert (has_kernel || has_module);
+          if (1)
+            {
+              Dwarf_Addr bias;
+              Elf* elf = (dwarf_getelf (dwfl_module_getdwarf (dw.module, &bias))
+                 ?: dwfl_module_getelf (dw.module, &bias));
+              GElf_Ehdr ehdr_mem;
+              GElf_Ehdr* em = gelf_getehdr (elf, &ehdr_mem);
+              int elf_machine = em->e_machine;
+              if (elf_machine == EM_MIPS)
+                {
+                  if (sess.verbose > 1)
+                    clog << "Hack reloc=" << hex << reloc_addr
+                      << " from addr=" << hex << addr;
+                  reloc_addr = addr - 0xc0000400;
+                }
+            }
           results.push_back (new dwarf_derived_probe(funcname, filename, line,
                                                      module, reloc_section, addr, reloc_addr,
                                                      *this, scope_die));

At the time I did the hack I was unaware of -msym32, I just wanted something to work. If I compile with KBUILD_SYM32=no it seems that the above hack might no longer be necessary (the addresses "look" right). I can't actually test the generated probes because my system won't boot with KBUILD_SYM32=no.

Apparently the gcc folks decided that this -msym32 behavior was too confusing and changed it to generate dwarf with a pointer size of 8:
https://gcc.gnu.org/ml/gcc/2009-01/msg00611.html

In the future this might not matter, but my cross-compiler is based on gcc-4.3.3. I'm posting this here because maybe somebody has a better idea about how to deal with this strange flavor of dwarf.

If this could be detected and handled inside systemtap fully then maybe you could apply the patches. Elfutils seems mostly blameless to me, it's just reading the information that GCC wrote.

Regards,
Leonard

Re: MIPS and -msym32

[Mark Wielaard]

Hi Leonard,

On Tue, Jul 22, 2014 at 05:44:48PM +0300, Crestez Dan Leonard wrote:
> I used the elfutils mips patch from debian:
> 	http://sources.debian.net/src/elfutils/0.159-4/debian/patches/mips_backend.diff

It would be convenient if the MIPS port was integrated upstream.
Do you happen to know whether the porters might want to contribute it?
The contribution policy for elfutils is documented at:
https://git.fedorahosted.org/cgit/elfutils.git/tree/CONTRIBUTING

> The generated dwarf files confuse systemtap is multiple ways. 

Do you happen to have one such DWARF file around? I like to better
understand which address size is set where.

> Here is a hack I used to get around this:
> --- a/libdw/dwarf_diecu.c
> +++ b/libdw/dwarf_diecu.c
> @@ -47,7 +47,22 @@ dwarf_diecu (die, result, address_sizep, offset_sizep)
>    *result = CUDIE (die->cu);
>  
>    if (address_sizep != NULL)
> +  {
>      *address_sizep = die->cu->address_size;
> +    /* Hack: */
> +    if (1)
> +    {
> +      struct Elf *elf = die->cu->dbg->elf;
> +      GElf_Ehdr ehdr_mem;
> +      GElf_Ehdr* ehdr = gelf_getehdr (elf, &ehdr_mem);
> +      if (ehdr &&
> +              ehdr->e_machine == EM_MIPS &&
> +              ehdr->e_ident[EI_CLASS] == ELFCLASS64)
> +      {
> +        *address_sizep = 8;
> +      }
> +    }
> +  }
>    if (offset_sizep != NULL)
>      *offset_sizep = die->cu->offset_size;
> 
> This is obviously evil.

Yes, it is :)
Assuming that address size used in the CU is correct, it seems the
above check should be in systemtap instead.

> Apparently the gcc folks decided that this -msym32 behavior was too
> confusing and changed it to generate dwarf with a pointer size of 8:
> https://gcc.gnu.org/ml/gcc/2009-01/msg00611.html

Hmmm. That helps, but is technically wrong IMHO.
It would be better to fix binutils instead.

Thanks,

Mark

Re: MIPS and -msym32

[Crestez Dan Leonard]

On 08/01/2014 01:59 PM, Mark Wielaard wrote:
> Hi Leonard,
> 
> On Tue, Jul 22, 2014 at 05:44:48PM +0300, Crestez Dan Leonard wrote:
>> I used the elfutils mips patch from debian:
>> 	http://sources.debian.net/src/elfutils/0.159-4/debian/patches/mips_backend.diff
> 
> It would be convenient if the MIPS port was integrated upstream.
> Do you happen to know whether the porters might want to contribute it?
> The contribution policy for elfutils is documented at:
> https://git.fedorahosted.org/cgit/elfutils.git/tree/CONTRIBUTING

The patch I found does not list any author. I just "found it". Maybe you should email
the debian maintainer to ask where it's from?

Regards,
Leonard

Re: MIPS and -msym32

[Mark Wielaard]

Hi Kurt,

On Fri, 2014-08-01 at 15:21 +0300, Crestez Dan Leonard wrote:
> On 08/01/2014 01:59 PM, Mark Wielaard wrote:
> > On Tue, Jul 22, 2014 at 05:44:48PM +0300, Crestez Dan Leonard wrote:
> >> I used the elfutils mips patch from debian:
> >> 	http://sources.debian.net/src/elfutils/0.159-4/debian/patches/mips_backend.diff
> > 
> > It would be convenient if the MIPS port was integrated upstream.
> > Do you happen to know whether the porters might want to contribute it?
> > The contribution policy for elfutils is documented at:
> > https://git.fedorahosted.org/cgit/elfutils.git/tree/CONTRIBUTING
> 
> The patch I found does not list any author. I just "found it". Maybe you should email
> the debian maintainer to ask where it's from?

Could you help us out with this? Do you know who the authors are of the
elfutils mips patch in debian and whether they would be willing to
contribute it upstream as described in the CONTRIBUTING document?

Thanks,

Mark

Re: MIPS and -msym32

[Kurt Roeckx]

On Fri, Aug 15, 2014 at 04:38:22PM +0200, Mark Wielaard wrote:
> Hi Kurt,
> 
> On Fri, 2014-08-01 at 15:21 +0300, Crestez Dan Leonard wrote:
> > On 08/01/2014 01:59 PM, Mark Wielaard wrote:
> > > On Tue, Jul 22, 2014 at 05:44:48PM +0300, Crestez Dan Leonard wrote:
> > >> I used the elfutils mips patch from debian:
> > >> 	http://sources.debian.net/src/elfutils/0.159-4/debian/patches/mips_backend.diff
> > > 
> > > It would be convenient if the MIPS port was integrated upstream.
> > > Do you happen to know whether the porters might want to contribute it?
> > > The contribution policy for elfutils is documented at:
> > > https://git.fedorahosted.org/cgit/elfutils.git/tree/CONTRIBUTING
> > 
> > The patch I found does not list any author. I just "found it". Maybe you should email
> > the debian maintainer to ask where it's from?
> 
> Could you help us out with this? Do you know who the authors are of the
> elfutils mips patch in debian and whether they would be willing to
> contribute it upstream as described in the CONTRIBUTING document?

It's been on my TODO list for a very long time to properly
document the author and get things merged.

The changelog indicates this was written by Christian Aichinger
<Greek0@gmx.net>

There is an other mips related patch in Debian.  I have code for
that, it was mailed to this list, I needed to make some changes,
I think I've done that but I didn't really finish them so it was
never send.  I can send what I currently have if that's useful.


Kurt