[Bug threads/27690] New: FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: libpthread.so not initialized (pattern 2)

[Bug threads/27690] New: FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: libpthread.so not initialized (pattern 2)

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27690

            Bug ID: 27690
           Summary: FAIL: gdb.threads/check-libthread-db.exp:
                    user-initiated check: libpthread.so not initialized
                    (pattern 2)
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: threads
          Assignee: unassigned at sourceware dot org
          Reporter: vries at gcc dot gnu.org
  Target Milestone: ---

On openSUSE tumbleweed, I see:
...
FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: libpthread.so
not initialized (pattern 2)
FAIL: gdb.threads/check-libthread-db.exp: automated load-time check:
libpthread.so not initialized: check debug libthread-db output (pattern 2)
...

In more detail:
...
(gdb) PASS: gdb.threads/check-libthread-db.exp: user-initiated check: continue
gdb_expect_list pattern: /[^M
]+Running libthread_db integrity checks:maint check libthread-db^M
Running libthread_db integrity checks:^M
  Got thread 0x7ffff7c77b80gdb_expect_list pattern: /[^M
]+[ ]+Got thread 0x0 => [0-9]+ => 0x0 ... OK => 4560 => 0x7ffff7c77b80; errno =
0 ... OK^M
libthread_db integrity checks passed.^M
(gdb) FAIL: gdb.threads/check-libthread-db.exp: user-initiated check:
libpthread.so not initialized (pattern 2)
...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug threads/27690] FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: libpthread.so not initialized (pattern 2)

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27690

--- Comment #1 from Tom de Vries <vries at gcc dot gnu.org> ---
This does not improve from patch
https://sourceware.org/pipermail/gdb-patches/2021-March/177369.html, the RFC
patch for PR27526.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/27690] FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: libpthread.so not initialized (pattern 2)

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27690

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|threads                     |testsuite

--- Comment #2 from Tom de Vries <vries at gcc dot gnu.org> ---
https://sourceware.org/pipermail/gdb-patches/2021-July/180757.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/27690] FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: libpthread.so not initialized (pattern 2)

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27690

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |simark at simark dot ca

--- Comment #3 from Tom de Vries <vries at gcc dot gnu.org> ---
Managed to bisect this in glibc, as follows.

I. 

Build glibc using pretty basic script:
...
$ more ~/glibc/scripts/build.sh
#!/bin/sh

set -e

pwd=$(pwd -P)

src=$pwd/src
build=$pwd/build
install=$pwd/install

rm -Rf $build $install
mkdir $build $install

cd $build

$src/configure --prefix=$install

make -j5
...

II.

Run test-case using custom build glibc (using recipe from
https://sourceware.org/glibc/wiki/Testing/Builds#Compile_against_glibc_build_tree):
...
diff --git a/gdb/testsuite/gdb.threads/check-libthread-db.exp
b/gdb/testsuite/gdb.threads/c
heck-libthread-db.exp
index dfb3bad3712..78bb881c6b3 100644
--- a/gdb/testsuite/gdb.threads/check-libthread-db.exp
+++ b/gdb/testsuite/gdb.threads/check-libthread-db.exp
@@ -27,15 +27,23 @@ if [gdb_debug_enabled] {

 standard_testfile

+set GLIBC "/home/vries/glibc/build"
+set f1
"-Wl,-rpath=${GLIBC}:${GLIBC}/math:${GLIBC}/elf:${GLIBC}/dlfcn:${GLIBC}/nss:
${GLIBC}/nis:${GLIBC}/rt:${GLIBC}/resolv:${GLIBC}/crypt:${GLIBC}/nptl:${GLIBC}/dfp" 
+
+set f2 "-Wl,--dynamic-linker=${GLIBC}/elf/ld.so"
+
 if {[gdb_compile_pthreads "${srcdir}/${subdir}/${srcfile}" "${binfile}" \
-        executable debug] != "" } {
+        executable [list debug additional_flags=$f1 additional_flags=$f2]] !=
"" } {
     return -1
 }

+
 with_test_prefix "user-initiated check" {

     # User-initiated check with libthread_db not loaded.
     clean_restart ${binfile}
+    gdb_test "set auto-load safe-path /"
+    gdb_test "set libthread-db-search-path /home/vries/glibc/build/nptl_db"

     gdb_test "maint show check-libthread-db" \
        "Whether to check libthread_db at load time is off."
@@ -81,6 +89,8 @@ with_test_prefix "automated load-time check" {
     # Automated load-time check with NPTL uninitialized.
     with_test_prefix "libpthread.so not initialized" {
        clean_restart ${binfile}
+       gdb_test "set auto-load safe-path /"
+       gdb_test "set libthread-db-search-path /home/vries/glibc/build/nptl_db"

        gdb_test_no_output "maint set check-libthread-db 1"
        gdb_test_no_output "set debug libthread-db 1"
@@ -99,6 +109,8 @@ with_test_prefix "automated load-time check" {
     # Automated load-time check with NPTL fully operational.
     with_test_prefix "libpthread.so fully initialized" {
        clean_restart ${binfile}
+       gdb_test "set auto-load safe-path /"
+       gdb_test "set libthread-db-search-path /home/vries/glibc/build/nptl_db"

        gdb_test_no_output "maint set check-libthread-db 1"
        gdb_test_no_output "set debug libthread-db 1"
...

Bisects to (as indeed suggested here
https://sourceware.org/pipermail/gdb-patches/2021-July/180867.html by Simon):
...
$ git bisect bad
1daccf403b1bd86370eb94edca794dc106d02039 is the first bad commit
commit 1daccf403b1bd86370eb94edca794dc106d02039
Author: Florian Weimer <fweimer@redhat.com>
Date:   Mon Nov 16 19:33:30 2020 +0100

    nptl: Move stack list variables into _rtld_global
...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/27690] FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: libpthread.so not initialized (pattern 2)

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27690

--- Comment #4 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Tom de Vries <vries@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=6b66b5a5a29e18e1c1ddd29ea0749050d532de65

commit 6b66b5a5a29e18e1c1ddd29ea0749050d532de65
Author: Tom de Vries <tdevries@suse.de>
Date:   Tue Jul 13 15:56:49 2021 +0200

    [gdb/testsuite] Fix check-libthread-db.exp FAILs with glibc 2.33

    When running test-case gdb.threads/check-libthread-db.exp on openSUSE
    Tumbleweed with glibc 2.33, I get:
    ...
    (gdb) maint check libthread-db^M
    Running libthread_db integrity checks:^M
      Got thread 0x7ffff7c79b80 => 9354 => 0x7ffff7c79b80; errno = 0 ... OK^M
    libthread_db integrity checks passed.^M
    (gdb) FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: \
      libpthread.so not initialized (pattern 2)
    ...

    The test-case expects instead:
    ...
      Got thread 0x0 => 9354 => 0x0 ... OK^M
    ...
    which is what I get on openSUSE Leap 15.2 with glibc 2.26, and what is
    described in the test-case like this:
    ...
        # libthread_db should fake a single thread with th_unique == NULL.
    ...

    Using a breakpoint on check_thread_db_callback we can compare the two
    scenarios, and find that in the latter case we hit this code in glibc
function
    iterate_thread_list in nptl_db/td_ta_thr_iter.c:
    ...
      if (next == 0 && fake_empty)
        {
          /* __pthread_initialize_minimal has not run.  There is just the main
             thread to return.  We cannot rely on its thread register.  They
             sometimes contain garbage that would confuse us, left by the
             kernel at exec.  So if it looks like initialization is incomplete,
             we only fake a special descriptor for the initial thread.  */
          td_thrhandle_t th = { ta, 0 };
          return callback (&th, cbdata_p) != 0 ? TD_DBERR : TD_OK;
        }
    ...
    while in the former case we don't because this preceding statement doesn't
    result in next == 0:
    ...
      err = DB_GET_FIELD (next, ta, head, list_t, next, 0);
    ...

    Note that the comment mentions __pthread_initialize_minimal, but in both
cases
    it has already run before we hit the callback, so it's possible the comment
is
    no longer accurate.

    The change in behaviour bisect to glibc commit 1daccf403b "nptl: Move stack
    list variables into _rtld_global", which moves the initialization of stack
    list variables such as __stack_user to an earlier moment, which explains
well
    enough the observed difference.

    Fix this by updating the regexp patterns to agree with what libthread-db is
    telling us.

    Tested on x86_64-linux, both with glibc 2.33 and 2.26.

    gdb/testsuite/ChangeLog:

    2021-07-07  Tom de Vries  <tdevries@suse.de>

            PR testsuite/27690
            * gdb.threads/check-libthread-db.exp: Update patterns for glibc
2.33.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/27690] FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: libpthread.so not initialized (pattern 2)

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27690

--- Comment #5 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The gdb-11-branch branch has been updated by Tom de Vries
<vries@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=58a136f3e739f5858c22ef62f981606a47cada66

commit 58a136f3e739f5858c22ef62f981606a47cada66
Author: Tom de Vries <tdevries@suse.de>
Date:   Tue Jul 13 16:11:38 2021 +0200

    [gdb/testsuite] Fix check-libthread-db.exp FAILs with glibc 2.33

    When running test-case gdb.threads/check-libthread-db.exp on openSUSE
    Tumbleweed with glibc 2.33, I get:
    ...
    (gdb) maint check libthread-db^M
    Running libthread_db integrity checks:^M
      Got thread 0x7ffff7c79b80 => 9354 => 0x7ffff7c79b80; errno = 0 ... OK^M
    libthread_db integrity checks passed.^M
    (gdb) FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: \
      libpthread.so not initialized (pattern 2)
    ...

    The test-case expects instead:
    ...
      Got thread 0x0 => 9354 => 0x0 ... OK^M
    ...
    which is what I get on openSUSE Leap 15.2 with glibc 2.26, and what is
    described in the test-case like this:
    ...
        # libthread_db should fake a single thread with th_unique == NULL.
    ...

    Using a breakpoint on check_thread_db_callback we can compare the two
    scenarios, and find that in the latter case we hit this code in glibc
function
    iterate_thread_list in nptl_db/td_ta_thr_iter.c:
    ...
      if (next == 0 && fake_empty)
        {
          /* __pthread_initialize_minimal has not run.  There is just the main
             thread to return.  We cannot rely on its thread register.  They
             sometimes contain garbage that would confuse us, left by the
             kernel at exec.  So if it looks like initialization is incomplete,
             we only fake a special descriptor for the initial thread.  */
          td_thrhandle_t th = { ta, 0 };
          return callback (&th, cbdata_p) != 0 ? TD_DBERR : TD_OK;
        }
    ...
    while in the former case we don't because this preceding statement doesn't
    result in next == 0:
    ...
      err = DB_GET_FIELD (next, ta, head, list_t, next, 0);
    ...

    Note that the comment mentions __pthread_initialize_minimal, but in both
cases
    it has already run before we hit the callback, so it's possible the comment
is
    no longer accurate.

    The change in behaviour bisect to glibc commit 1daccf403b "nptl: Move stack
    list variables into _rtld_global", which moves the initialization of stack
    list variables such as __stack_user to an earlier moment, which explains
well
    enough the observed difference.

    Fix this by updating the regexp patterns to agree with what libthread-db is
    telling us.

    Tested on x86_64-linux, both with glibc 2.33 and 2.26.

    gdb/testsuite/ChangeLog:

    2021-07-13  Tom de Vries  <tdevries@suse.de>

            PR testsuite/27690
            * gdb.threads/check-libthread-db.exp: Update patterns for glibc
2.33.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/27690] FAIL: gdb.threads/check-libthread-db.exp: user-initiated check: libpthread.so not initialized (pattern 2)

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27690

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED
   Target Milestone|---                         |11.1

--- Comment #6 from Tom de Vries <vries at gcc dot gnu.org> ---
Patch committed to trunk and gdb-11-branch, marking resolved-fixed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.