Glibc commit causing regression in gdb

Glibc commit causing regression in gdb

[Emil Velikov]

Greetings everyone,

We have recently received a regression report in Arch Linux, where gdb
attach fails with some applications.
In particular, the issue originates in glibc 2.33, while gdb used is
10.1 and is 100% reproducible (here) with steam and gimp.

Note that explicitly rebuilding gdb explicitly against the glibc does not help.

After some digging I found the offending glibc commit
1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move stack list
variables into _rtld_global"). With the said commit the
td_ta_map_lwp2thr() call in GDB fails, resulting in the infamous

Error while reading shared library symbols for /usr/lib/libpthread.so.0:
Cannot find user-level thread for LWP 2999060: generic error


Can anyone confirm if the issue is reproducible with other distros? Is
there some gdb patch to fix this, or alternatively is it safe to ship
a revert upstream and/or within the Arch Linux package?

Bear in mind that I'm happy to get this issue resolved properly, yet
this is my first time seriously looking at the glibc codebase. For my
initial investigation I force-enabled LOG() and added a few extra
notations.

- Vanilla glibc 2.33
td_init
td_ta_new
td_init
td_ta_new
td_ta_map_lwp2thr
__td_ta_stack_user has failed // newly added LOG

- Vanilla glibc 2.33 with offending commit reverted
td_init
td_ta_new
td_init
td_ta_new
td_ta_map_lwp2thr
__td_ta_lookup_th_unique // newly added LOG entry
td_thr_get_info
... above three are repeated a lot, until the end

Thanks
Emil
[1] https://bugs.archlinux.org/task/69657

Re: Glibc commit causing regression in gdb

[Florian Weimer]

* Emil Velikov via Libc-alpha:

> We have recently received a regression report in Arch Linux, where gdb
> attach fails with some applications.
> In particular, the issue originates in glibc 2.33, while gdb used is
> 10.1 and is 100% reproducible (here) with steam and gimp.
>
> Note that explicitly rebuilding gdb explicitly against the glibc does not help.
>
> After some digging I found the offending glibc commit
> 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move stack list
> variables into _rtld_global"). With the said commit the
> td_ta_map_lwp2thr() call in GDB fails, resulting in the infamous
>
> Error while reading shared library symbols for /usr/lib/libpthread.so.0:
> Cannot find user-level thread for LWP 2999060: generic error

> Can anyone confirm if the issue is reproducible with other distros?

There's a separate thread about this on the gdb-patches list, under the
slightly misleading subject of “GDB 10.2 Release -- 2021-04-11 update”:

  <https://sourceware.org/pipermail/gdb-patches/2021-April/thread.html>

I believe there was an earlier discussion as well, but I can't find it
right now.

Thanks,
Florian

Re: Glibc commit causing regression in gdb

[Emil Velikov]

Hi Florian,

On Fri, 16 Apr 2021 at 09:35, Florian Weimer <fweimer@redhat.com> wrote:
>
> * Emil Velikov via Libc-alpha:
>
> > We have recently received a regression report in Arch Linux, where gdb
> > attach fails with some applications.
> > In particular, the issue originates in glibc 2.33, while gdb used is
> > 10.1 and is 100% reproducible (here) with steam and gimp.
> >
> > Note that explicitly rebuilding gdb explicitly against the glibc does not help.
> >
> > After some digging I found the offending glibc commit
> > 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move stack list
> > variables into _rtld_global"). With the said commit the
> > td_ta_map_lwp2thr() call in GDB fails, resulting in the infamous
> >
> > Error while reading shared library symbols for /usr/lib/libpthread.so.0:
> > Cannot find user-level thread for LWP 2999060: generic error
>
> > Can anyone confirm if the issue is reproducible with other distros?
>
> There's a separate thread about this on the gdb-patches list, under the
> slightly misleading subject of “GDB 10.2 Release -- 2021-04-11 update”:
>
>   <https://sourceware.org/pipermail/gdb-patches/2021-April/thread.html>
>
> I believe there was an earlier discussion as well, but I can't find it
> right now.
>
Thanks for the link and tips. Reading through the thread and related
discussions it seems that a gdb-side fix is far from trivial.

With your glibc developer hat on, how crazy^Wstupid of an idea is to
carry a revert locally until a proper gdb fix is available? Say on the
scale of 1 to 10.
I am going through the glibc/gdb test suites with the revert, although
I don't know if there are extra subtleties which won't be caught by
the tests.

Thanks again,
Emil

Re: Glibc commit causing regression in gdb

[Florian Weimer]

* Emil Velikov:

> With your glibc developer hat on, how crazy^Wstupid of an idea is to
> carry a revert locally until a proper gdb fix is available? Say on the
> scale of 1 to 10.  I am going through the glibc/gdb test suites with
> the revert, although I don't know if there are extra subtleties which
> won't be caught by the tests.

The revert changes the internal GLIBC_PRIVATE ABI, so some processes
will fail if the update is applied without a reboot.

(This assumes that the patch can be reverted in isolation.)

Thanks,
Florian