[Bug nptl/25890] New: Endianness detection in libthread_db fails with some type sizes

[Bug nptl/25890] New: Endianness detection in libthread_db fails with some type sizes

[simark at simark dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=25890

            Bug ID: 25890
           Summary: Endianness detection in libthread_db fails with some
                    type sizes
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: nptl
          Assignee: unassigned at sourceware dot org
          Reporter: simark at simark dot ca
                CC: drepper.fsp at gmail dot com
  Target Milestone: ---

I found this while trying to debug a powerpc64 (big-endian) core file on an
x86-64 (little endian) machine.  In order to read TLS variables, I need a
working libthread_db.

At first I wasn't sure if this was a supported use case.  If libthread_db reads
the data structures from the inferior using its own native type sizes / offset
/ endianness, there's no way it can work.  However, I found that libthread_db
actually uses some descriptors stored in libpthread, which describe the type
sizes, and the size and offset of fields within structures.  So that makes it
work in cross-architecture scenarios.

I have built a libthread_db for the x86-64 architecture from the same sources
as the libc used on the powerpc64 machine.  When I load the core in GDB, I get:

(gdb) core ./core.ppc64 
warning: core file may not match specified executable file.
[New LWP 200]
[New LWP 201]
Warning: couldn't activate thread debugging using libthread_db: Cannot find new
threads: generic error

I drilled down, and found this bug in libthread_db.

To detect whether the inferior has an endianness different than its own,
libthread_db uses the `size` property of descriptors.  It does this check (in
_td_check_sizeof, but there are similar checks elsewhere):

      if (*sizep & 0xff000000U)
        *sizep = bswap_32 (*sizep);

The logic behind it is probably that since sizes are expected to be small, if
the bytes are reversed, the size will be found in the high-order bits.  On
powerpc64, the pthread structure happens to have the very unfortunate size of
0x700 bytes.  When reversed, the value is 0x00070000.  As you now understand,
the aforementioned check fails to identify that the bytes are in the wrong
order.  libthread_db continues thinking that a pthread structure is 0x70000
bytes long, instead of 0x700, and it eventually breaks down.

As a quick fix, I changed the check to be:

      if (*sizep & 0xffff0000U)
        *sizep = bswap_32 (*sizep);

... and things started to work.

So I think that these `size & 0xff000000` checks will have to be replaced with
something else.

-- 
You are receiving this mail because:
You are on the CC list for the bug.