Re: [PATCH] 0001-Fix-multi-thread-debug-bug-in-AIX.patch

[Aditya Kamath1 <Aditya.Kamath1@ibm.com>]

[-- Attachment #1.1: Type: text/plain, Size: 20554 bytes --]

Hi Ulrich, Tom and community,

Please find attached the patch. {See: 0001-Fix-multi-thread-debug-bug-in-AIX.patch}

The previous email could not get to the community mailing list as it had a size constraint.

So, the bug is fixed, and outputs are the same as pasted below in this email.

In the patch commit message I have provided specific and detailed explaination of every change as much as possible.

So honestly there was a bug left to fix in the previous email of the patch while we followed the child in particular. I figured it out as I was testing more deeply. Kindly see section "Solution Part 1: - " in the patch commit message where I have explained the same.

Kindly suggest me for changes if needed. Otherwise kindly let me know if this is ready for commit.

>Now if the application is
>theraded and we only pass ptid_t (user_current_pid) to switch_to_thread ()
>it will crash as main thread looks different or is ptid_t (pid, 0, tid).

> This part I don't quite understand yet - how/why does it crash?

Kindly check the section "Solution Part 2: - " of the patch, where I have explained this.

>Similarly, I agree that everything may currently "work" without
>adding the equivalent change to pdc_write_memory, but most likely
>this is simply because that callback may just not be used very much.

Yes, I agree.  We have the changed user_current_pid variable to thread so that we always switch in the right context. Kindly let me know if it is alright and any changes are necessary here.

>Can you come up with a
>message that maybe starts out with the high-level change
>and goes from there into the specific
>details.... Thanks!

Sure, so this patch has it. Kindly suggest me if we can do this better.

Have a nice day ahead.

Thanks and regards,
Aditya.

--------------------------------------------------------------------------------
Code:-


#include <stdio.h>

#include <unistd.h>

#include <stdlib.h>

#include <pthread.h>

#include <assert.h>


pthread_barrier_t barrier;


#define NUM_THREADS 2


void *

thread_function (void *arg)

{

  /* This ensures that the breakpoint is only hit after both threads

     are created, so the test can always switch to the non-event

     thread when the breakpoint triggers.  */


  pthread_barrier_wait (&barrier);

  pid_t child;


  child = fork ();

  if (child > 0)

    printf ("I am parent \n");

  else

  {

    child = fork ();

    if (child > 0)

      printf ("I am child \n");

    else

      printf ("I am grandchild \n");

  }

  while (1); /* break here */

}


int

main (void)

{

  int i;

  pthread_t thread[NUM_THREADS];


  alarm (300);


  pthread_barrier_init (&barrier, NULL, NUM_THREADS);


  for (i = 0; i < NUM_THREADS; i++)

    {

      int res;


      res = pthread_create (&thread[i], NULL,

                            thread_function, NULL);

      assert (res == 0);

    }


  while (1)

  {

    sleep (15);

  }


  return 0;

}

-------------------------------------------------
Output with patch:-


Reading symbols from /home/aditya/gdb_tests/ultimate-multi-thread-fork...

(gdb) set detach-on-fork off

(gdb) r

Starting program: /home/aditya/gdb_tests/ultimate-multi-thread-fork

[New Thread 258]

[New Thread 515]

[New inferior 2 (process 15335754)]

[New inferior 3 (process 8061404)]

I am parent

I am parent

^C

Thread 1.1 received signal SIGINT, Interrupt.

[Switching to Thread 1]

0xd0595fb0 in _p_nsleep () from /usr/lib/libpthread.a(shr_xpg5.o)

(gdb) info sharedlibrary

From        To          Syms Read   Shared Object Library

0xd05bc124  0xd05bf194  Yes (*)     /usr/lib/libpthreads.a(shr_comm.o)

0xd05bb240  0xd05bb9a1  Yes (*)     /usr/lib/libcrypt.a(shr.o)

0xd0576180  0xd05ba731  Yes (*)     /usr/lib/libpthread.a(shr_xpg5.o)

0xd0100e00  0xd0575123  Yes (*)     /usr/lib/libc.a(shr.o)

(*): Shared library is missing debugging information.

(gdb) info threads

  Id   Target Id                          Frame

* 1.1  Thread 1 (tid 30671243, running)   0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  1.2  Thread 258 (tid 34406781, running) thread_function (arg=0x0)

    at /home/aditya/gdb_tests/ultimate-multi-thread-fork.c:32

  1.3  Thread 515 (tid 36307315, running) thread_function (arg=warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)


0x0)

    at /home/aditya/gdb_tests/ultimate-multi-thread-fork.c:32

  2.1  process 15335754                   0xd0594fc8 in _sigsetmask ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  3.1  process 8061404                    0xd0594fc8 in _sigsetmask ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

(gdb) inferior 2

[Switching to inferior 2 [process 15335754] (/home/aditya/gdb_tests/ultimate-multi-thread-fork)]

[Switching to thread 2.1 (process 15335754)]

#0  0xd0594fc8 in _sigsetmask () from /usr/lib/libpthread.a(shr_xpg5.o)

(gdb)

-----------------------------------------------------------------------
Output without patch:-
Reading symbols from /home/aditya/gdb_tests/ultimate-multi-thread-fork...

(gdb) set detach-on-fork off

(gdb) r

Starting program: /home/aditya/gdb_tests/ultimate-multi-thread-fork

[New Thread 1]

[New Thread 258]

[New Thread 515]

[New inferior 2 (process 11731200)]

I am parent

[New inferior 3 (process 16843200)]

I am parent

^C

Thread 1.1 received signal SIGINT, Interrupt.

0xd0595fb0 in _p_nsleep () from /usr/lib/libpthread.a(shr_xpg5.o)

(gdb) inferior 2

[Switching to inferior 2 [process 11731200] (/home/aditya/gdb_tests/ultimate-multi-thread-fork)]

[Switching to thread 2.1 (process 11731200)]

#0  0xd0594fc8 in ?? ()

(gdb) info threads

  Id   Target Id         Frame

  1.1  process 15270316  0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  1.2  process 15270316  0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  1.3  process 15270316  0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  1.4  process 15270316  0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

* 2.1  process 11731200  0xd0594fc8 in ?? ()

  3.1  process 16843200  0xd0594fc8 in ?? ()

(gdb) info sharedlibrary

warning: "/usr/lib/libpthreads.a": member "shr_comm.o" missing.

warning: "/usr/lib/libcrypt.a": member "shr.o" missing.

warning: "/usr/lib/libpthread.a": member "shr_xpg5.o" missing.

warning: "/usr/lib/libc.a": member "shr.o" missing.

warning: Could not load shared library symbols for 4 libraries, e.g. /usr/lib/libpthreads.a(shr_comm.o).

Use the "info sharedlibrary" command to see the complete listing.

Do you need "set solib-search-path" or "set sysroot"?

From        To          Syms Read   Shared Object Library

                        No          /usr/lib/libpthreads.a(shr_comm.o)

                        No          /usr/lib/libcrypt.a(shr.o)

                        No          /usr/lib/libpthread.a(shr_xpg5.o)

                        No          /usr/lib/libc.a(shr.o)
(gdb)



________________________________
From: Aditya Kamath1 <Aditya.Kamath1@ibm.com>
Sent: 10 February 2023 22:03
To: Ulrich Weigand <Ulrich.Weigand@de.ibm.com>; simark@simark.ca <simark@simark.ca>; gdb-patches@sourceware.org <gdb-patches@sourceware.org>
Cc: Sangamesh Mallayya <sangamesh.swamy@in.ibm.com>
Subject: Re: [PATCH] 0001-Fix-multi-thread-debug-bug-in-AIX.patch

Hi Ulrich, Tom and community,

Please find attached the patch. {See: 0001-Fix-multi-thread-debug-bug-in-AIX.patch}

Also find attached a document that I have proposed as a commit message. {See: Fix Multi Thread debug fix for AIX.pdf}.. This same document is used in the commit message of this patch.

So the bug is fixed and test cases run alright. Kindly check the sample output of the same pasted below this email.

In the document I have provided specific and detailed explaination of every change as much as possible.

So honestly there was a bug left to fix in the previous email of the patch while we followed the child in particular. I figured it out as I was testing more deeply. Kindly see section "Solution Part 1: - " where I have explained the same.

Kindly suggest me for changes if needed. Otherwise kindly let me know if this is ready for commit.

>Previously we used switch_to_thread ().. Now if the application is
>theraded and we only pass ptid_t (user_current_pid) to switch_to_thread ()
>it will crash as main thread looks different or is ptid_t (pid, 0, tid).

> This part I don't quite understand yet - how/why does it crash?

Kindly check "Solution Part 2: - " of the document, where I have explained this.

>Similarly, I agree that everything may currently "work" without
>adding the equivalent change to pdc_write_memory, but most likely
>this is simply because that callback may just not be used very much.

Yes, I agree.  We have the changed user_current_pid variable to thread so that we always switch in the right context. Kindly let me know if it is alright and any changes are necessary here.

>Can you come up with a
>message that maybe starts out with the high-level change
>(along the lines of "update aix-thread.c to handle threads in
>multiple inferiors"), and goes from there into the specific
>details (aix_thread_variables structure, handling only a
>single inferior per sync_threadlists invocation, solib fixes
>for multiple inferiors, ...)?  Thanks!

Sure, so the pdf attached in this email has it. Kindly suggest me if we can do this better.

Have a nice day ahead.

Thanks and regards,
Aditya.


--------------------------------------------------------------------------------
Code:-


#include <stdio.h>

#include <unistd.h>

#include <stdlib.h>

#include <pthread.h>

#include <assert.h>


pthread_barrier_t barrier;


#define NUM_THREADS 2


void *

thread_function (void *arg)

{

  /* This ensures that the breakpoint is only hit after both threads

     are created, so the test can always switch to the non-event

     thread when the breakpoint triggers.  */


  pthread_barrier_wait (&barrier);

  pid_t child;


  child = fork ();

  if (child > 0)

    printf ("I am parent \n");

  else

  {

    child = fork ();

    if (child > 0)

      printf ("I am child \n");

    else

      printf ("I am grandchild \n");

  }

  while (1); /* break here */

}


int

main (void)

{

  int i;

  pthread_t thread[NUM_THREADS];


  alarm (300);


  pthread_barrier_init (&barrier, NULL, NUM_THREADS);


  for (i = 0; i < NUM_THREADS; i++)

    {

      int res;


      res = pthread_create (&thread[i], NULL,

                            thread_function, NULL);

      assert (res == 0);

    }


  while (1)

  {

    sleep (15);

  }


  return 0;

}

-------------------------------------------------
Output with patch:-


Reading symbols from /home/aditya/gdb_tests/ultimate-multi-thread-fork...

(gdb) set detach-on-fork off

(gdb) r

Starting program: /home/aditya/gdb_tests/ultimate-multi-thread-fork

[New Thread 258]

[New Thread 515]

[New inferior 2 (process 15335754)]

[New inferior 3 (process 8061404)]

I am parent

I am parent

^C

Thread 1.1 received signal SIGINT, Interrupt.

[Switching to Thread 1]

0xd0595fb0 in _p_nsleep () from /usr/lib/libpthread.a(shr_xpg5.o)

(gdb) info sharedlibrary

From        To          Syms Read   Shared Object Library

0xd05bc124  0xd05bf194  Yes (*)     /usr/lib/libpthreads.a(shr_comm.o)

0xd05bb240  0xd05bb9a1  Yes (*)     /usr/lib/libcrypt.a(shr.o)

0xd0576180  0xd05ba731  Yes (*)     /usr/lib/libpthread.a(shr_xpg5.o)

0xd0100e00  0xd0575123  Yes (*)     /usr/lib/libc.a(shr.o)

(*): Shared library is missing debugging information.

(gdb) info threads

  Id   Target Id                          Frame

* 1.1  Thread 1 (tid 30671243, running)   0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  1.2  Thread 258 (tid 34406781, running) thread_function (arg=0x0)

    at /home/aditya/gdb_tests/ultimate-multi-thread-fork.c:32

  1.3  Thread 515 (tid 36307315, running) thread_function (arg=warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)


0x0)

    at /home/aditya/gdb_tests/ultimate-multi-thread-fork.c:32

  2.1  process 15335754                   0xd0594fc8 in _sigsetmask ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  3.1  process 8061404                    0xd0594fc8 in _sigsetmask ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

(gdb) inferior 2

[Switching to inferior 2 [process 15335754] (/home/aditya/gdb_tests/ultimate-multi-thread-fork)]

[Switching to thread 2.1 (process 15335754)]

#0  0xd0594fc8 in _sigsetmask () from /usr/lib/libpthread.a(shr_xpg5.o)

(gdb)

-----------------------------------------------------------------------
Output without patch:-
Reading symbols from /home/aditya/gdb_tests/ultimate-multi-thread-fork...

(gdb) set detach-on-fork off

(gdb) r

Starting program: /home/aditya/gdb_tests/ultimate-multi-thread-fork

[New Thread 1]

[New Thread 258]

[New Thread 515]

[New inferior 2 (process 11731200)]

I am parent

[New inferior 3 (process 16843200)]

I am parent

^C

Thread 1.1 received signal SIGINT, Interrupt.

0xd0595fb0 in _p_nsleep () from /usr/lib/libpthread.a(shr_xpg5.o)

(gdb) inferior 2

[Switching to inferior 2 [process 11731200] (/home/aditya/gdb_tests/ultimate-multi-thread-fork)]

[Switching to thread 2.1 (process 11731200)]

#0  0xd0594fc8 in ?? ()

(gdb) info threads

  Id   Target Id         Frame

  1.1  process 15270316  0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  1.2  process 15270316  0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  1.3  process 15270316  0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

  1.4  process 15270316  0xd0595fb0 in _p_nsleep ()

   from /usr/lib/libpthread.a(shr_xpg5.o)

* 2.1  process 11731200  0xd0594fc8 in ?? ()

  3.1  process 16843200  0xd0594fc8 in ?? ()

(gdb) info sharedlibrary

warning: "/usr/lib/libpthreads.a": member "shr_comm.o" missing.

warning: "/usr/lib/libcrypt.a": member "shr.o" missing.

warning: "/usr/lib/libpthread.a": member "shr_xpg5.o" missing.

warning: "/usr/lib/libc.a": member "shr.o" missing.

warning: Could not load shared library symbols for 4 libraries, e.g. /usr/lib/libpthreads.a(shr_comm.o).

Use the "info sharedlibrary" command to see the complete listing.

Do you need "set solib-search-path" or "set sysroot"?

From        To          Syms Read   Shared Object Library

                        No          /usr/lib/libpthreads.a(shr_comm.o)

                        No          /usr/lib/libcrypt.a(shr.o)

                        No          /usr/lib/libpthread.a(shr_xpg5.o)

                        No          /usr/lib/libc.a(shr.o)
(gdb)
________________________________
From: Ulrich Weigand <Ulrich.Weigand@de.ibm.com>
Sent: 09 February 2023 00:14
To: simark@simark.ca <simark@simark.ca>; Aditya Kamath1 <Aditya.Kamath1@ibm.com>; gdb-patches@sourceware.org <gdb-patches@sourceware.org>
Cc: Sangamesh Mallayya <sangamesh.swamy@in.ibm.com>
Subject: Re: [PATCH] 0001-Fix-multi-thread-debug-bug-in-AIX.patch

Aditya Kamath1 <Aditya.Kamath1@ibm.com> wrote:

>>This seems unrelated to the rest of the changes at first glance.
>>Why is this necessary?
>
>So, when we need to be in the right context when we read memory. Before
>coming into the target wait, we switch_to_no_thread () due to which our
>inferior_ptid is set to null. Our target_memory needs the correct
>inferior_ptid.  Also, in case we don't have a ptid_t (pid) and the
>application is threaded we need the inferior_ptid to be set correctly
>like shown in the patch.

Understood.

>Previously we used switch_to_thread ().. Now if the application is
>theraded and we only pass ptid_t (user_current_pid) to switch_to_thread ()
>it will crash as main thread looks different or is ptid_t (pid, 0, tid).

This part I don't quite understand yet - how/why does it crash?

>>By comparison, the Linux version of this in proc-service.c also
>>switches the current inferior and address space:
> > scoped_restore_current_inferior restore_inferior;
> > set_current_inferior (ph->thread->inf);
> > scoped_restore_current_program_space restore_current_progspace;
> > set_current_program_space (ph->thread->inf->pspace);
> > scoped_restore save_inferior_ptid = make_scoped_restore (&inferior_ptid);
> > inferior_ptid = ph->thread->ptid;
>> so we should probably do the same for consistency.

>So, kindly allow me to disagree with you on this. What is happening is in
>inferior.c in do_target_wait1 () we call switch_to_inferior_no_thread ()..
[snip]
>Here we already set the correct current inferior and program space to
>the same thing as that if we set in pdc_read_memory like linux.
>So, it does not make any difference to add the changes like linux does

Well, it does look like if you entered the callback in this particular
context, the inferior may have already been set up correctly.  However,
in theory the callback could also be called in different contexts, and
just as a precaution it would be preferable to have it always work
correctly.  The semantics of the callback is to read memory of a
particular process as identified via the pthdb_user_t argument, and
we should write the routine so that it always does what's needed to
implement that semantics correctly.

>Secondly, things work if we do not do the same for pdc_write_memory.
>I have not seen anything not work. So, I don't think it is good to
>add it there. What say??

Similarly, I agree that everything may currently "work" without
adding the equivalent change to pdc_write_memory, but most likely
this is simply because that callback may just not be used very much.

But as a precaution, and to accommodate potential future changes
e.g. in the libpthdebug.a library, if would be preferable to
implement the semantics correctly.  (Also, it just looks surprising
to see the read and write implementation differ when there is no
apparent reason why that should be the case.)

>>This looks unnecessarily complicated.  Isn't this just
>  > *g++ = tp;
>
>This I have changed.

The code now looks like:
>+  for (thread_info *tp : all_threads (proc_target, ptid_t (pid)))
>+  {
>+    *g = tp;
>+    *g++;
>+  }

Which is weird, as *g++ dereferences g for no reason.  This should
simply be:

  for (thread_info *tp : all_threads (proc_target, ptid_t (pid)))
    *g++ = tp;


>As far as the check gptid.is_pid () is concerned, I will suggest we
>keep it there. If cmp_result is > 0 and we have a main process swap
>to create a thread. Rest is same in the loop. The reason being handling
>pi and gi variables becomes complex otherwise. When this swap happens,
>we need to increment both pi and gi.. Because we have taken care of the
>main threads in both pthread library and GDB. And this for loop is
>executed only once. So, the first event is main process being
>pthreaded. Once the swap happens pi and gi become one and since
>gcount = pcount = 1 we exit the for loop. Thread addition events comes
>after this.

Hmm, handling the initial switch of a single PID-only thread
to the PID/TID-style ptid_t separately before still seems
a bit clearer to me.  But in the end your proposed code looks
correct now so I'd be fine with it as is, if you prefer.


Except for the few things mentioned above, this now looks ready to
be committed to me.  However, I'm not sure the commit message
fully describes the latest version of the patch, after we've
gone through all those iterations ...  Can you come up with a
message that maybe starts out with the high-level change
(along the lines of "update aix-thread.c to handle threads in
multiple inferiors"), and goes from there into the specific
details (aix_thread_variables structure, handling only a
single inferior per sync_threadlists invocation, solib fixes
for multiple inferiors, ...)?  Thanks!

Bye,
Ulrich



[-- Attachment #2: 0001-Fix-multi-thread-debug-bug-in-AIX.patch --]
[-- Type: application/octet-stream, Size: 44777 bytes --]

From ae2f4812d5cf561ac24bbd51cfdaa532c73ea900 Mon Sep 17 00:00:00 2001
From: Aditya Vidyadhar Kamath <Aditya.Kamath1@ibm.com>
Date: Fri, 10 Feb 2023 09:36:11 -0600
Subject: [PATCH] Fix Multi Thread debug fix for AIX
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The bug:- In the recent commit 98ed24fb35d89eb20179edf6c12f599c7a9e228e there is a change in aix-thread.c file that changes 



static_cast<aix_thread_info *> in gdb to gdb::checked_static_cast<aix_thread_info *>



AIX folks using the latest version thus will not be able to debug multi thread programs as a result of it. 

The error in AIX is as follows: - 

internal error: checked_static_cast: Assertion `result! = nullptr' failed.

The root cause of the issue:-  The private data was not set for the first thread or the main thread of a process. In AIX when we run an “info threads” command, we showed main process as “process <pid>” without private data set and added a new thread Thread<Tid> representing the same with private set. When we iterate_over_threads () we call get_aix_thread_info (). This leads to the crash as we had the main process thread “process <pid>” with no private data. Hence the checked static cast will not allow us to debug any further which is rightly so as we had a thread with no private data.

What should be the fix: - Removing the main process thread i.e. “process <pid> “was the first proposed solution as the “Thread <tid>” representing the same already exists with private data set. This was happening in the sync_threadlists () code of AIX.

Solution Part 1: -

Why the change?

The delete_thread () with the cmp_result > 0 block of the for loop in the sync_threadlists () function which applies the difference between the pthread and GDB threadlist, will fail to delete the main process thread. The reason is that it “process <pid>” is the current process and thus GDB core will not delete it despite we are calling it. Hence even if we add the “thread <tid>” representing the same “process <pid>” in the next iteration of the for loop we will not be successful.

Hence this forces us to change the main process thread “process <pid>” to “thread <tid>” via the thread_change_ptid () and the private data set. These changes can be seen in the sync_threadlists () part.

However, we also need to keep in mind that before we think this will work, our libpthread library is only ready when the following condition in the wait () of aix-thread.c is satisfied.

/* Check whether libpthdebug might be ready to be initialized.  */
  if (!data->pd_active && status->kind () == TARGET_WAITKIND_STOPPED
      && status->sig () == GDB_SIGNAL_TRAP)

Until then changing the “process <pid>” to “thread <tid>” is incorrect. Even though the session is ready and initalised via pd_enable () and pd_activate () functions respectively. Therfore this made us to keep a variable pthdebugready in all functions that lead to sync_threadlists () so that we change the process thread to a thread with private data only when libpthdebug is initialised for a particular process.

The first if condition below this paragraph change in the sync_threadlists () as shown below means the pthread debug library is not intialised. This is just to set priv to main process thread.

if (gbuf[0]->ptid.is_pid () && !pthdebugready)
    {
      aix_thread_info *priv = new aix_thread_info;
      tp->priv.reset (priv);
    }

The second if condition below this paragraph change is for changing “process <pid>” to “thread <tid>” as the pthread debug library is intialised.

if (gptid.is_pid () && pthdebugready)
                {
                  thread_change_ptid (proc_target, gptid, pptid);
                  aix_thread_info *priv = new aix_thread_info;
                  priv->pdtid = pbuf[pi].pdtid;
                  priv->tid = pbuf[pi].tid;
                  tp->priv.reset (priv);
                  gi++;
                  pi++;
                }

Failing to do so leads us to two problems. One while we fetch_registers () our regcache-> ptid though changed to ptid_t (pid, 0, tid) will not be able to get the private data in a case where we switch to a child process from the parent process via “inferior 2” command leading to the crash that private data was not set for a thread. Because we incorrectly changed the “process <pid>” to “thread <tid>” before the process itself could raise a trap and tell the debugger we are now ready to debug threads.

Example of the crash:-
(gdb) set detach-on-fork off
(gdb) r
Starting program:
[New Thread 258]
[New Thread 515]
[New inferior 2 (process 21627386)]
I am parent
[New inferior 3 (process 9372064)]
I am parent
^C
Thread 1.1 received signal SIGINT, Interrupt.
[Switching to Thread 1]
0xd0595fb0 in _p_nsleep () from /usr/lib/libpthread.a(shr_xpg5.o)
(gdb) inferior 2
[Switching to inferior 2 [process 21627386] (/home /gdb_tests/ultimate-multi-thread-fork)]
[Switching to thread 2.1 (Thread 515)]
(gdb) c
Continuing.
./../gdbsupport/gdb-checked-static-cast.h:58: internal-error: checked_static_cast: Assertion `result != nullptr' failed.

The process stack of the crash due to the is as below: -

0x000000010059ef60  aix_thread_info* gdb::checked_static_cast<aix_thread_info*, private_thread_info>(private_thread_info*)(0x0) + 0x7c
0x0000000100596ea0  get_aix_thread_info(thread_info*)(0x0) + 0x34
0x000000010059b778  aix_thread_target::fetch_registers(regcache*, int)(0x11001f3f8, 0x1107c5030, 0x4000000000) + 0xf8
0x00000001003675f0  target_fetch_registers(regcache*, int)(0x1107c5030, 0x40e0ddf00d) + 0x6c
0x00000001005817c0  regcache::raw_update(int)(0x1107c5030, 0x401001f3f8) + 0x94
0x0000000100581904  readable_regcache::raw_read(int, unsigned char*)(0x1107c5030, 0x4000000203, 0xfffffffffffebc0) + 0x8c
0x0000000100581f54  readable_regcache::cooked_read(int, unsigned char*)(0x1107c5030, 0x40ffffeb90, 0xfffffffffffebc0) + 0xec
0x0000000100daba10  register_status readable_regcache::cooked_read<unsigned long, void>(int, unsigned long*)(0x1107c5030, 0x40ffffec50, 0xfffffffffffed10) + 0xd4
0x00000001005826a0  regcache_cooked_read_unsigned(regcache*, int, unsigned long*)(0x1107c5030, 0x40ffffecd0, 0xfffffffffffed10) + 0x70
0x0000000100584e2c  regcache_read_pc(regcache*)(0x1107c5030) + 0xa4
0x0000000100387614  handle_signal_stop(execution_control_state*)(0xffffffffffff3a8) + 0x158
0x00000001003864e4

Secondly in a case where, if we follow the child instead of the parent and we end up changing our “process <pid>” to “thread <tid>” before the process itself raises a trap and tells the debugger “I am ready for threads”, then when we switch_to_thread in the follow_fork () we end up not finding the “process <pid>” and thus leading to an assertion failure as shown below and rightly so, because we changed threads without the library being initialised. This happens when the follow_fork () is called, and we switch to the child thread there.

(gdb) set detach-on-fork off
(gdb) set follow-fork-mode child
(gdb) r
Starting program:
[New Thread 258]
[New Thread 515]
[Attaching after Thread 515 fork to child process 18809098]
[New inferior 2 (process 18809098)]
thread.c:1337: internal-error: switch_to_thread: Assertion `thr != NULL' failed.

The process stack is as follows:-
0x0000000100036590  internal_error_loc(char const*, int, char const*, ...)(0x10192ba70, 0x53900000000, 0x10192b970) + 0x58
0x0000000100619918  switch_to_thread(thread_info*)(0x0) + 0x48
0x000000010037635c  follow_fork()() + 0x4c8
0x0000000100385af8  handle_inferior_event(execution_control_state*)(0xffffffffffff3a8) + 0xda8
0x00000001003809d0  fetch_inferior_event()() + 0x2f8
0x0000000100719a0c  inferior_event_handler(inferior_event_type)(0x10207a50) + 0x38
0x000000010039228c  infrun_async_inferior_event_handler(void*)(0x0) + 0x30
0x0000000100671d18  check_async_event_handlers()() + 0x94
0x000000010066e32c  gdb_do_one_event(int)(0xfffffffffffff840) + 0xb4
0x0000000100001dcc  start_event_loop()() + 0x28
0x0000000100001fd4  captured_command_loop()() + 0x58
0x000000010000414c  captured_main(void*)(0xffffffffffffa60) + 0x2c
0x0000000100004220  gdb_main(captured_main_args*)(0xffffffffffffa60) + 0x20

So, the changes in the sync_threadlists () with parameter and the for loop justifies the same.

Also, we now do not use iterate_over_threads to count our GDB threads. We instead do it via for (thread_info *tp : all_threads (proc_target, ptid_t (pid))) inline.

Solution Part 2: -

Since we switch_to_no_thread before a wait (), on an event of a thread detection or any other event which makes us use the thread call-backs, we need to be in the right context while we read and write data for threads. That is why we switch our inferior_ptid, current_inferior and program space in pdc_read_data () and pdc_write_data and now pdc_write_data.

So why did we make this change
-    if (user_current_pid != 0)
-      switch_to_thread (current_inferior ()->process_target (),
-                       ptid_t (user_current_pid));
 in pdc_read_data and change our user variable which was the process ID to a thread? Wasn’t it already doing the job?

Consider an event where the parent process is threaded, and we have a fork (). When we do a pd_update () after the beneath->wait () in thread wait () we call sync_threadlists () as well. Over there we call pthdb_pthread (data->pd_session, &pdtid, cmd);

This now will use the ptid_t (user_current_pid) to switch the thread (). However, our parent process or main thread of it, is threaded i.e is ptid_t (user_current_pid, 0, tid). Hence, we will crash with an assertion failure that thread ptid_t (user_current_pid
) has not been found.

In order to avoid the same, we now pass the thread directly. So, on any event after the main process looks like a main thread, there will be no confusion on which thread space or inferior_ptid or program space to switch, especially when a process is multi-threaded.

Solution Part 3: - In AIX we use a lot of variables for different purposes like pd_active, pd_able, arch64, pd_brk_addr and pd_session. These variables are unique per inferior. Hence, we need to keep them in a map <inferior, structure> where structure can hold all these variables per inferior. This is where we use the inbuilt GDB registry for every inferior. This change exists in this patch.

Solution Part 4: -

We figured out that the top target for a new inferior born after the main inferior was incorrect post the process being threaded.

The root cause was that the shared library was not being loaded for new process. The reason being we change our shared library file name in the BFD registry from member name to path(member_name).

Hence the changes in solib-aix takes care of the new pattern so that the shared library can be loaded correctly for every new inferior born as well via pattern matching the ‘(‘character and checking if the member_name exists after that in the new pattern registered in the BFD registry as shown in solib-aix.c changes in this patch.

-----------------------------------------------------------------------------------------------------------------

These 4 solution parts together fixes the bug.
---
 gdb/aix-thread.c | 431 ++++++++++++++++++++++++++++-------------------
 gdb/solib-aix.c  |  14 ++
 2 files changed, 271 insertions(+), 174 deletions(-)

diff --git a/gdb/aix-thread.c b/gdb/aix-thread.c
index e556c153576..f86d9429f71 100644
--- a/gdb/aix-thread.c
+++ b/gdb/aix-thread.c
@@ -68,10 +68,6 @@ static bool debug_aix_thread;
 #define pthdb_tid_t	tid_t
 #endif
 
-/* Return whether to treat PID as a debuggable thread id.  */
-
-#define PD_TID(ptid)	(pd_active && ptid.tid () != 0)
-
 /* Success and failure values returned by pthdb callbacks.  */
 
 #define PDC_SUCCESS	PTHDB_SUCCESS
@@ -144,38 +140,20 @@ class aix_thread_target final : public target_ops
 
 static aix_thread_target aix_thread_ops;
 
-/* Address of the function that libpthread will call when libpthdebug
-   is ready to be initialized.  */
-
-static CORE_ADDR pd_brk_addr;
-
-/* Whether the current application is debuggable by pthdb.  */
-
-static int pd_able = 0;
-
-/* Whether a threaded application is being debugged.  */
-
-static int pd_active = 0;
-
-/* Whether the current architecture is 64-bit.  
-   Only valid when pd_able is true.  */
-
-static int arch64;
-
 /* Forward declarations for pthdb callbacks.  */
 
-static int pdc_symbol_addrs (pthdb_user_t, pthdb_symbol_t *, int);
-static int pdc_read_data (pthdb_user_t, void *, pthdb_addr_t, size_t);
-static int pdc_write_data (pthdb_user_t, void *, pthdb_addr_t, size_t);
-static int pdc_read_regs (pthdb_user_t user, pthdb_tid_t tid,
+static int pdc_symbol_addrs (thread_info*, pthdb_symbol_t *, int);
+static int pdc_read_data (thread_info*, void *, pthdb_addr_t, size_t);
+static int pdc_write_data (thread_info*, void *, pthdb_addr_t, size_t);
+static int pdc_read_regs (thread_info* user, pthdb_tid_t tid,
 			  unsigned long long flags, 
 			  pthdb_context_t *context);
-static int pdc_write_regs (pthdb_user_t user, pthdb_tid_t tid,
+static int pdc_write_regs (thread_info* user, pthdb_tid_t tid,
 			   unsigned long long flags, 
 			   pthdb_context_t *context);
-static int pdc_alloc (pthdb_user_t, size_t, void **);
-static int pdc_realloc (pthdb_user_t, void *, size_t, void **);
-static int pdc_dealloc (pthdb_user_t, void *);
+static int pdc_alloc (thread_info*, size_t, void **);
+static int pdc_realloc (thread_info*, void *, size_t, void **);
+static int pdc_dealloc (thread_info*, void *);
 
 /* pthdb callbacks.  */
 
@@ -191,9 +169,66 @@ static pthdb_callbacks_t pd_callbacks = {
   NULL
 };
 
-/* Current pthdb session.  */
+/* Aix variable structure.  */
+struct aix_thread_variables 
+{
+  /* Whether the current application is debuggable by pthdb.  */
+  int pd_able;
+
+  /* Whether a threaded application is being debugged.  */
+  int pd_active;
+
+  /* Current pthdb session.  */
+  pthdb_session_t pd_session;
+
+  /* Address of the function that libpthread will call when libpthdebug
+   is ready to be initialized.  */
+  CORE_ADDR pd_brk_addr;
+
+  /* Whether the current architecture is 64-bit.
+   Only valid when pd_able is true.  */
+  int arch64;
+};
+
+/* Key to our per-inferior data.  */
+static const registry<inferior>::key<aix_thread_variables>
+  aix_thread_variables_handle;
+
+/* Function to Get aix_thread_variables data.  */
+static struct aix_thread_variables*
+get_aix_thread_variables_data (struct inferior *inf)
+{
+  if (inf == NULL)
+    return NULL;
+
+  struct aix_thread_variables* data;
+
+  data = aix_thread_variables_handle.get (inf);
+  if (data == NULL)
+    data = aix_thread_variables_handle.emplace (inf);
+
+  return data;
+}
+
+/* Helper to get data for ptid in a function.  */
 
-static pthdb_session_t pd_session;
+static struct aix_thread_variables*
+get_thread_data_helper_for_ptid (ptid_t ptid)
+{
+  inferior *inf = find_inferior_ptid (current_inferior ()->process_target (),
+					ptid);
+  return get_aix_thread_variables_data (inf);
+}
+
+/* Helper to get data for pid in a function.  */
+
+static struct aix_thread_variables*
+get_thread_data_helper_for_pid (pid_t pid)
+{
+  inferior *inf = find_inferior_pid (current_inferior ()->process_target (),
+                                        pid);
+  return get_aix_thread_variables_data (inf);
+}
 
 /* Return a printable representation of pthdebug function return
    STATUS.  */
@@ -318,7 +353,7 @@ pid_to_prc (ptid_t *ptidp)
   ptid_t ptid;
 
   ptid = *ptidp;
-  if (PD_TID (ptid))
+  if (ptid.tid () != 0)
     *ptidp = ptid_t (ptid.pid ());
 }
 
@@ -326,7 +361,7 @@ pid_to_prc (ptid_t *ptidp)
    the address of SYMBOLS[<i>].name.  */
 
 static int
-pdc_symbol_addrs (pthdb_user_t user_current_pid, pthdb_symbol_t *symbols, int count)
+pdc_symbol_addrs (thread_info *user_current_thread, pthdb_symbol_t *symbols, int count)
 {
   struct bound_minimal_symbol ms;
   int i;
@@ -334,8 +369,8 @@ pdc_symbol_addrs (pthdb_user_t user_current_pid, pthdb_symbol_t *symbols, int co
 
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog,
-		"pdc_symbol_addrs (user_current_pid = %ld, symbols = 0x%lx, count = %d)\n",
-		user_current_pid, (long) symbols, count);
+		"pdc_symbol_addrs (user_current_pid = %d, symbols = 0x%lx, count = %d)\n",
+		user_current_thread->ptid.pid (), (long) symbols, count);
 
   for (i = 0; i < count; i++)
     {
@@ -373,7 +408,7 @@ pdc_symbol_addrs (pthdb_user_t user_current_pid, pthdb_symbol_t *symbols, int co
    If successful return 0, else non-zero is returned.  */
 
 static int
-pdc_read_regs (pthdb_user_t user_current_pid,
+pdc_read_regs (thread_info *user_current_thread,
 	       pthdb_tid_t tid,
 	       unsigned long long flags,
 	       pthdb_context_t *context)
@@ -389,6 +424,9 @@ pdc_read_regs (pthdb_user_t user_current_pid,
   double fprs[ppc_num_fprs];
   struct ptxsprs sprs64;
   struct ptsprs sprs32;
+  struct aix_thread_variables *data;
+  
+  data = get_thread_data_helper_for_pid (user_current_thread->ptid.pid ());
   
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog, "pdc_read_regs tid=%d flags=%s\n",
@@ -397,7 +435,7 @@ pdc_read_regs (pthdb_user_t user_current_pid,
   /* General-purpose registers.  */
   if (flags & PTHDB_FLAG_GPRS)
     {
-      if (arch64)
+      if (data->arch64)
 	{
 	  if (!ptrace64aix (PTT_READ_GPRS, tid, 
 			    (unsigned long) gprs64, 0, NULL))
@@ -423,7 +461,7 @@ pdc_read_regs (pthdb_user_t user_current_pid,
   /* Special-purpose registers.  */
   if (flags & PTHDB_FLAG_SPRS)
     {
-      if (arch64)
+      if (data->arch64)
 	{
 	  if (!ptrace64aix (PTT_READ_SPRS, tid, 
 			    (unsigned long) &sprs64, 0, NULL))
@@ -445,7 +483,7 @@ pdc_read_regs (pthdb_user_t user_current_pid,
    If successful return 0, else non-zero is returned.  */
 
 static int
-pdc_write_regs (pthdb_user_t user_current_pid,
+pdc_write_regs (thread_info *user_current_thread,
 		pthdb_tid_t tid,
 		unsigned long long flags,
 		pthdb_context_t *context)
@@ -456,6 +494,10 @@ pdc_write_regs (pthdb_user_t user_current_pid,
      this is needed, I have implemented what I think it should do,
      however this code is untested.  */
 
+  struct aix_thread_variables *data;
+  
+  data = get_thread_data_helper_for_pid (user_current_thread->ptid.pid ());
+
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog, "pdc_write_regs tid=%d flags=%s\n",
 		(int) tid, hex_string (flags));
@@ -463,7 +505,7 @@ pdc_write_regs (pthdb_user_t user_current_pid,
   /* General-purpose registers.  */
   if (flags & PTHDB_FLAG_GPRS)
     {
-      if (arch64)
+      if (data->arch64)
 	ptrace64aix (PTT_WRITE_GPRS, tid, 
 		     (unsigned long) context->gpr, 0, NULL);
       else
@@ -479,7 +521,7 @@ pdc_write_regs (pthdb_user_t user_current_pid,
   /* Special-purpose registers.  */
   if (flags & PTHDB_FLAG_SPRS)
     {
-      if (arch64)
+      if (data->arch64)
 	{
 	  ptrace64aix (PTT_WRITE_SPRS, tid, 
 		       (unsigned long) &context->msr, 0, NULL);
@@ -495,27 +537,30 @@ pdc_write_regs (pthdb_user_t user_current_pid,
 /* pthdb callback: read LEN bytes from process ADDR into BUF.  */
 
 static int
-pdc_read_data (pthdb_user_t user_current_pid, void *buf,
+pdc_read_data (thread_info *user_current_thread, void *buf,
 	       pthdb_addr_t addr, size_t len)
 {
   int status, ret;
 
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog,
-		"pdc_read_data (user_current_pid = %ld, buf = 0x%lx, addr = %s, len = %ld)\n",
-		user_current_pid, (long) buf, hex_string (addr), len);
+		"pdc_read_data (user_current_pid = %d, buf = 0x%lx, addr = %s, len = %ld)\n",
+		user_current_thread->ptid.pid (), (long) buf, hex_string (addr), len);
 
   /* This is needed to eliminate the dependency of current thread
      which is null so that thread reads the correct target memory.  */
   {
-    scoped_restore_current_thread restore_current_thread;
+    scoped_restore save_inferior_ptid = make_scoped_restore (&inferior_ptid);
     /* Before the first inferior is added, we pass inferior_ptid.pid ()
        from pd_enable () which is 0.  There is no need to switch threads
        during first initialisation.  In the rest of the callbacks the
        current thread needs to be correct.  */
-    if (user_current_pid != 0)
-      switch_to_thread (current_inferior ()->process_target (),
-			ptid_t (user_current_pid));
+    inferior_ptid = user_current_thread->ptid;
+    scoped_restore_current_inferior restore_inferior;
+    set_current_inferior (user_current_thread->inf);
+
+    scoped_restore_current_program_space restore_current_progspace;
+    set_current_program_space (user_current_thread->inf->pspace);
     status = target_read_memory (addr, (gdb_byte *) buf, len);
   }
   ret = status == 0 ? PDC_SUCCESS : PDC_FAILURE;
@@ -529,17 +574,27 @@ pdc_read_data (pthdb_user_t user_current_pid, void *buf,
 /* pthdb callback: write LEN bytes from BUF to process ADDR.  */
 
 static int
-pdc_write_data (pthdb_user_t user_current_pid, void *buf,
+pdc_write_data (thread_info *user_current_thread, void *buf,
 		pthdb_addr_t addr, size_t len)
 {
   int status, ret;
 
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog,
-		"pdc_write_data (user_current_pid = %ld, buf = 0x%lx, addr = %s, len = %ld)\n",
-		user_current_pid, (long) buf, hex_string (addr), len);
+		"pdc_write_data (user_current_pid = %d, buf = 0x%lx, addr = %s, len = %ld)\n",
+		user_current_thread->ptid.pid (), (long) buf, hex_string (addr), len);
+
+  {
+    scoped_restore save_inferior_ptid = make_scoped_restore (&inferior_ptid);
+    inferior_ptid = user_current_thread->ptid;
+    scoped_restore_current_inferior restore_inferior;
+    set_current_inferior (user_current_thread->inf);
+
+    scoped_restore_current_program_space restore_current_progspace;
+    set_current_program_space (user_current_thread->inf->pspace);
+    status = target_write_memory (addr, (gdb_byte *) buf, len);
+  }
 
-  status = target_write_memory (addr, (gdb_byte *) buf, len);
   ret = status == 0 ? PDC_SUCCESS : PDC_FAILURE;
 
   if (debug_aix_thread)
@@ -552,12 +607,12 @@ pdc_write_data (pthdb_user_t user_current_pid, void *buf,
    in BUFP.  */
 
 static int
-pdc_alloc (pthdb_user_t user_current_pid, size_t len, void **bufp)
+pdc_alloc (thread_info *user_current_thread, size_t len, void **bufp)
 {
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog,
-		"pdc_alloc (user_current_pid = %ld, len = %ld, bufp = 0x%lx)\n",
-		user_current_pid, len, (long) bufp);
+		"pdc_alloc (user_current_pid = %d, len = %ld, bufp = 0x%lx)\n",
+		user_current_thread->ptid.pid (), len, (long) bufp);
   *bufp = xmalloc (len);
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog, 
@@ -574,12 +629,12 @@ pdc_alloc (pthdb_user_t user_current_pid, size_t len, void **bufp)
    pointer to the result in BUFP.  */
 
 static int
-pdc_realloc (pthdb_user_t user_current_pid, void *buf, size_t len, void **bufp)
+pdc_realloc (thread_info *user_current_thread, void *buf, size_t len, void **bufp)
 {
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog,
-		"pdc_realloc (user_current_pid = %ld, buf = 0x%lx, len = %ld, bufp = 0x%lx)\n",
-		user_current_pid, (long) buf, len, (long) bufp);
+		"pdc_realloc (user_current_pid = %d, buf = 0x%lx, len = %ld, bufp = 0x%lx)\n",
+		user_current_thread->ptid.pid (), (long) buf, len, (long) bufp);
   *bufp = xrealloc (buf, len);
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog, 
@@ -591,11 +646,11 @@ pdc_realloc (pthdb_user_t user_current_pid, void *buf, size_t len, void **bufp)
    realloc callback.  */
 
 static int
-pdc_dealloc (pthdb_user_t user_current_pid, void *buf)
+pdc_dealloc (thread_info *user_current_thread, void *buf)
 {
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog, 
-		"pdc_free (user_current_pid = %ld, buf = 0x%lx)\n", user_current_pid,
+		"pdc_free (user_current_pid = %d, buf = 0x%lx)\n", user_current_thread->ptid.pid (),
 		(long) buf);
   xfree (buf);
   return PDC_SUCCESS;
@@ -639,39 +694,6 @@ pcmp (const void *p1v, const void *p2v)
   return p1->pthid < p2->pthid ? -1 : p1->pthid > p2->pthid;
 }
 
-/* iterate_over_threads() callback for counting GDB threads.
-
-   Do not count the main thread (whose tid is zero).  This matches
-   the list of threads provided by the pthreaddebug library, which
-   does not include that main thread either, and thus allows us
-   to compare the two lists.  */
-
-static int
-giter_count (struct thread_info *thread, void *countp)
-{
-  if (PD_TID (thread->ptid))
-    (*(int *) countp)++;
-  return 0;
-}
-
-/* iterate_over_threads() callback for accumulating GDB thread pids.
-
-   Do not include the main thread (whose tid is zero).  This matches
-   the list of threads provided by the pthreaddebug library, which
-   does not include that main thread either, and thus allows us
-   to compare the two lists.  */
-
-static int
-giter_accum (struct thread_info *thread, void *bufp)
-{
-  if (PD_TID (thread->ptid))
-    {
-      **(struct thread_info ***) bufp = thread;
-      (*(struct thread_info ***) bufp)++;
-    }
-  return 0;
-}
-
 /* ptid comparison function */
 
 static int
@@ -719,7 +741,10 @@ get_signaled_thread (int pid)
 		    sizeof (thrinf), &ktid, 1) != 1)
 	break;
 
-      if (thrinf.ti_cursig == SIGTRAP)
+      /* We also need to keep in mind Trap and interrupt or any
+         signal that needs to be handled in pd_update ().  */
+
+      if (thrinf.ti_cursig)
 	return thrinf.ti_tid;
     }
 
@@ -741,7 +766,7 @@ get_signaled_thread (int pid)
        have difficulty with certain call patterns */
 
 static void
-sync_threadlists (int pid)
+sync_threadlists (pid_t pid, int pthdebugready)
 {
   int cmd, status;
   int pcount, psize, pi, gcount, gi;
@@ -750,6 +775,11 @@ sync_threadlists (int pid)
   pthdb_pthread_t pdtid;
   pthread_t pthid;
   pthdb_tid_t tid;
+  process_stratum_target *proc_target
+            = current_inferior ()->process_target ();
+  thread_info  *tp;
+  struct aix_thread_variables *data;
+  data = get_thread_data_helper_for_pid (pid);
 
   /* Accumulate an array of libpthdebug threads sorted by pthread id.  */
 
@@ -759,11 +789,11 @@ sync_threadlists (int pid)
 
   for (cmd = PTHDB_LIST_FIRST;; cmd = PTHDB_LIST_NEXT)
     {
-      status = pthdb_pthread (pd_session, &pdtid, cmd);
+      status = pthdb_pthread (data->pd_session, &pdtid, cmd);
       if (status != PTHDB_SUCCESS || pdtid == PTHDB_INVALID_PTHREAD)
 	break;
 
-      status = pthdb_pthread_ptid (pd_session, pdtid, &pthid);
+      status = pthdb_pthread_ptid (data->pd_session, pdtid, &pthid);
       if (status != PTHDB_SUCCESS || pthid == PTHDB_INVALID_PTID)
 	continue;
 
@@ -780,7 +810,7 @@ sync_threadlists (int pid)
 
   for (pi = 0; pi < pcount; pi++)
     {
-      status = pthdb_pthread_tid (pd_session, pbuf[pi].pdtid, &tid);
+      status = pthdb_pthread_tid (data->pd_session, pbuf[pi].pdtid, &tid);
       if (status != PTHDB_SUCCESS)
 	tid = PTHDB_INVALID_TID;
       pbuf[pi].tid = tid;
@@ -790,13 +820,32 @@ sync_threadlists (int pid)
 
   /* Accumulate an array of GDB threads sorted by pid.  */
 
+  /* gcount is GDB thread count and pcount is pthreadlib thread count.  */
+
   gcount = 0;
-  iterate_over_threads (giter_count, &gcount);
+  for (thread_info *tp : all_threads (proc_target, ptid_t (pid)))
+    gcount++;
   g = gbuf = XNEWVEC (struct thread_info *, gcount);
-  iterate_over_threads (giter_accum, &g);
+  for (thread_info *tp : all_threads (proc_target, ptid_t (pid)))
+    *g++ = tp;
   qsort (gbuf, gcount, sizeof *gbuf, gcmp);
 
+  tp = find_thread_ptid (proc_target, ptid_t (pid));
+
+  /* If the pthreadlibrary is not ready to debug 
+     then this is just a main process which needs 
+     a priv to be set.  The if condition below does 
+     the same.  Otherwise we go to the for loop to 
+     sync the pthread and GDB thread lists.  */
+
+  if (gbuf[0]->ptid.is_pid () && !pthdebugready)
+    {
+      aix_thread_info *priv = new aix_thread_info;
+      tp->priv.reset (priv);
+    }
+
   /* Apply differences between the two arrays to GDB's thread list.  */
+  else  
   for (pi = gi = 0; pi < pcount || gi < gcount;)
     {
       if (pi == pcount)
@@ -810,8 +859,6 @@ sync_threadlists (int pid)
 	  priv->pdtid = pbuf[pi].pdtid;
 	  priv->tid = pbuf[pi].tid;
 
-	  process_stratum_target *proc_target
-	    = current_inferior ()->process_target ();
 	  thread = add_thread_with_info (proc_target,
 					 ptid_t (pid, 0, pbuf[pi].pthid),
 					 priv);
@@ -841,13 +888,27 @@ sync_threadlists (int pid)
 	    }
 	  else if (cmp_result > 0)
 	    {
-	      delete_thread (gbuf[gi]);
-	      gi++;
+	      /* This is to make the main process thread now look
+                 like a thread.  */
+
+	      if (gptid.is_pid () && pthdebugready)
+		{
+		  thread_change_ptid (proc_target, gptid, pptid);
+		  aix_thread_info *priv = new aix_thread_info;
+                  priv->pdtid = pbuf[pi].pdtid;
+                  priv->tid = pbuf[pi].tid;
+		  tp->priv.reset (priv);
+		  gi++;
+		  pi++;
+		}
+	      else
+		{
+		  delete_thread (gbuf[gi]);
+		  gi++;
+		}
 	    }
 	  else
 	    {
-	      process_stratum_target *proc_target
-		= current_inferior ()->process_target ();
 	      thread = add_thread (proc_target, pptid);
 
 	      aix_thread_info *priv = new aix_thread_info;
@@ -881,21 +942,24 @@ iter_tid (struct thread_info *thread, void *tidp)
    return a pid-only ptid with PID.  */
 
 static ptid_t
-pd_update (int pid)
+pd_update (pid_t pid, int pthdebugready)
 {
   int status;
   ptid_t ptid;
   pthdb_tid_t tid;
   struct thread_info *thread = NULL;
+  struct aix_thread_variables *data;
 
-  if (!pd_active)
+  data = get_thread_data_helper_for_pid (pid);
+
+  if (!data->pd_active)
     return ptid_t (pid);
 
-  status = pthdb_session_update (pd_session);
+  status = pthdb_session_update (data->pd_session);
   if (status != PTHDB_SUCCESS)
     return ptid_t (pid);
 
-  sync_threadlists (pid);
+  sync_threadlists (pid, pthdebugready);
 
   /* Define "current thread" as one that just received a trap signal.  */
 
@@ -915,32 +979,22 @@ pd_update (int pid)
    for that thread.  Otherwise, return a ptid-only ptid using PID.  */
 
 static ptid_t
-pd_activate (int pid)
+pd_activate (pid_t pid, int pthdebugready)
 {
   int status;
-		
-  status = pthdb_session_init (pid, arch64 ? PEM_64BIT : PEM_32BIT,
+  struct aix_thread_variables *data;
+  data = get_thread_data_helper_for_pid (pid);
+  thread_info *thread = find_thread_ptid (current_inferior (), ptid_t (pid));
+  
+  status = pthdb_session_init (thread, data->arch64 ? PEM_64BIT : PEM_32BIT,
 			       PTHDB_FLAG_REGS, &pd_callbacks, 
-			       &pd_session);
+			       &data->pd_session);
   if (status != PTHDB_SUCCESS)
     {
       return ptid_t (pid);
     }
-  pd_active = 1;
-  return pd_update (pid);
-}
-
-/* Undo the effects of pd_activate().  */
-
-static void
-pd_deactivate (void)
-{
-  if (!pd_active)
-    return;
-  pthdb_session_destroy (pd_session);
-  
-  pid_to_prc (&inferior_ptid);
-  pd_active = 0;
+  data->pd_active = 1;
+  return pd_update (pid, pthdebugready);
 }
 
 /* An object file has just been loaded.  Check whether the current
@@ -952,17 +1006,24 @@ pd_enable (void)
   int status;
   char *stub_name;
   struct bound_minimal_symbol ms;
+  struct aix_thread_variables *data;
+
+  if (!inferior_ptid.pid ())
+    return;
+  
+  data = get_thread_data_helper_for_ptid (inferior_ptid);
 
   /* Don't initialize twice.  */
-  if (pd_able)
+  if (data->pd_able)
     return;
 
   /* Check application word size.  */
-  arch64 = register_size (target_gdbarch (), 0) == 8;
+  data->arch64 = register_size (target_gdbarch (), 0) == 8;
 
   /* Check whether the application is pthreaded.  */
   stub_name = NULL;
-  status = pthdb_session_pthreaded (inferior_ptid.pid (), PTHDB_FLAG_REGS,
+  thread_info *thread = find_thread_ptid (current_inferior (), inferior_ptid);
+  status = pthdb_session_pthreaded (thread, PTHDB_FLAG_REGS,
 				    &pd_callbacks, &stub_name);
   if ((status != PTHDB_SUCCESS
        && status != PTHDB_NOT_PTHREADED) || !stub_name)
@@ -972,18 +1033,18 @@ pd_enable (void)
   ms = lookup_minimal_symbol (stub_name, NULL, NULL);
   if (ms.minsym == NULL)
     return;
-  pd_brk_addr = ms.value_address ();
-  if (!create_thread_event_breakpoint (target_gdbarch (), pd_brk_addr))
+  data->pd_brk_addr = ms.value_address ();
+  if (!create_thread_event_breakpoint (target_gdbarch (), data->pd_brk_addr))
     return;
 
   /* Prepare for thread debugging.  */
   current_inferior ()->push_target (&aix_thread_ops);
-  pd_able = 1;
+  data->pd_able = 1; 
 
   /* If we're debugging a core file or an attached inferior, the
      pthread library may already have been initialized, so try to
      activate thread debugging.  */
-  pd_activate (inferior_ptid.pid ());
+  pd_activate (inferior_ptid.pid (), 0);
 }
 
 /* Undo the effects of pd_enable().  */
@@ -991,28 +1052,31 @@ pd_enable (void)
 static void
 pd_disable (void)
 {
-  if (!pd_able)
+  struct aix_thread_variables *data;
+  data = get_thread_data_helper_for_ptid (inferior_ptid);
+
+  if (!data->pd_able)
+    return;
+  if (!data->pd_active)
     return;
-  if (pd_active)
-    pd_deactivate ();
-  pd_able = 0;
+  pthdb_session_destroy (data->pd_session);
+ 
+  pid_to_prc (&inferior_ptid);
+  data->pd_active = 0;
+  data->pd_able = 0;
   current_inferior ()->unpush_target (&aix_thread_ops);
 }
 
 /* new_objfile observer callback.
 
    If OBJFILE is non-null, check whether a threaded application is
-   being debugged, and if so, prepare for thread debugging.
-
-   If OBJFILE is null, stop debugging threads.  */
+   being debugged, and if so, prepare for thread debugging.  */
 
 static void
 new_objfile (struct objfile *objfile)
 {
   if (objfile)
     pd_enable ();
-  else
-    pd_disable ();
 }
 
 /* Attach to process specified by ARGS.  */
@@ -1042,8 +1106,11 @@ aix_thread_target::resume (ptid_t ptid, int step, enum gdb_signal sig)
 {
   struct thread_info *thread;
   pthdb_tid_t tid[2];
+  struct aix_thread_variables *data;
+  
+  data = get_thread_data_helper_for_ptid (ptid);
 
-  if (!PD_TID (ptid))
+  if (ptid.tid () == 0)
     {
       scoped_restore save_inferior_ptid = make_scoped_restore (&inferior_ptid);
       
@@ -1065,7 +1132,7 @@ aix_thread_target::resume (ptid_t ptid, int step, enum gdb_signal sig)
 	       ptid.lwp ());
       tid[1] = 0;
 
-      if (arch64)
+      if (data->arch64)
 	ptrace64aix (PTT_CONTINUE, tid[0], (long long) 1,
 		     gdb_signal_to_host (sig), (PTRACE_TYPE_ARG5) tid);
       else
@@ -1082,6 +1149,7 @@ ptid_t
 aix_thread_target::wait (ptid_t ptid, struct target_waitstatus *status,
 			 target_wait_flags options)
 {
+  struct aix_thread_variables *data;
   {
     pid_to_prc (&ptid);
 
@@ -1095,8 +1163,10 @@ aix_thread_target::wait (ptid_t ptid, struct target_waitstatus *status,
      pid-only ptids.  */
   gdb_assert (ptid.is_pid ());
 
+  data = get_thread_data_helper_for_ptid (ptid);
+
   /* Check whether libpthdebug might be ready to be initialized.  */
-  if (!pd_active && status->kind () == TARGET_WAITKIND_STOPPED
+  if (!data->pd_active && status->kind () == TARGET_WAITKIND_STOPPED
       && status->sig () == GDB_SIGNAL_TRAP)
     {
       process_stratum_target *proc_target
@@ -1105,11 +1175,11 @@ aix_thread_target::wait (ptid_t ptid, struct target_waitstatus *status,
       struct gdbarch *gdbarch = regcache->arch ();
 
       if (regcache_read_pc (regcache)
-	  - gdbarch_decr_pc_after_break (gdbarch) == pd_brk_addr)
-	return pd_activate (ptid.pid ());
+	  - gdbarch_decr_pc_after_break (gdbarch) == data->pd_brk_addr)
+	return pd_activate (ptid.pid (), 1);
     }
 
-  return pd_update (ptid.pid ());
+  return pd_update (ptid.pid (), 0);
 }
 
 /* Record that the 64-bit general-purpose registers contain VALS.  */
@@ -1229,18 +1299,20 @@ fetch_regs_user_thread (struct regcache *regcache, pthdb_pthread_t pdtid)
   ppc_gdbarch_tdep *tdep = gdbarch_tdep<ppc_gdbarch_tdep> (gdbarch);
   int status, i;
   pthdb_context_t ctx;
+  struct aix_thread_variables *data;
+  data = get_thread_data_helper_for_ptid (inferior_ptid);
 
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog, 
 		"fetch_regs_user_thread %lx\n", (long) pdtid);
-  status = pthdb_pthread_context (pd_session, pdtid, &ctx);
+  status = pthdb_pthread_context (data->pd_session, pdtid, &ctx);
   if (status != PTHDB_SUCCESS)
     error (_("aix-thread: fetch_registers: pthdb_pthread_context returned %s"),
 	   pd_status2str (status));
 
   /* General-purpose registers.  */
 
-  if (arch64)
+  if (data->arch64)
     supply_gprs64 (regcache, ctx.gpr);
   else
     for (i = 0; i < ppc_num_gprs; i++)
@@ -1253,7 +1325,7 @@ fetch_regs_user_thread (struct regcache *regcache, pthdb_pthread_t pdtid)
 
   /* Special registers.  */
 
-  if (arch64)
+  if (data->arch64)
     supply_sprs64 (regcache, ctx.iar, ctx.msr, ctx.cr, ctx.lr, ctx.ctr,
 			     ctx.xer, ctx.fpscr);
   else
@@ -1288,18 +1360,21 @@ fetch_regs_kernel_thread (struct regcache *regcache, int regno,
   struct ptxsprs sprs64;
   struct ptsprs sprs32;
   int i;
+  struct aix_thread_variables *data;
+  
+  data = get_thread_data_helper_for_ptid (regcache->ptid ());
 
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog,
 		"fetch_regs_kernel_thread tid=%lx regno=%d arch64=%d\n",
-		(long) tid, regno, arch64);
+		(long) tid, regno, data->arch64);
 
   /* General-purpose registers.  */
   if (regno == -1
       || (tdep->ppc_gp0_regnum <= regno
 	  && regno < tdep->ppc_gp0_regnum + ppc_num_gprs))
     {
-      if (arch64)
+      if (data->arch64)
 	{
 	  if (!ptrace64aix (PTT_READ_GPRS, tid, 
 			    (unsigned long) gprs64, 0, NULL))
@@ -1331,7 +1406,7 @@ fetch_regs_kernel_thread (struct regcache *regcache, int regno,
 
   if (regno == -1 || special_register_p (gdbarch, regno))
     {
-      if (arch64)
+      if (data->arch64)
 	{
 	  if (!ptrace64aix (PTT_READ_SPRS, tid, 
 			    (unsigned long) &sprs64, 0, NULL))
@@ -1363,7 +1438,7 @@ aix_thread_target::fetch_registers (struct regcache *regcache, int regno)
   struct thread_info *thread;
   pthdb_tid_t tid;
 
-  if (!PD_TID (regcache->ptid ()))
+  if (regcache->ptid ().tid () == 0)
     beneath ()->fetch_registers (regcache, regno);
   else
     {
@@ -1511,6 +1586,8 @@ store_regs_user_thread (const struct regcache *regcache, pthdb_pthread_t pdtid)
   pthdb_context_t ctx;
   uint32_t int32;
   uint64_t int64;
+  struct aix_thread_variables *data;
+  data = get_thread_data_helper_for_ptid (inferior_ptid);
 
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog, 
@@ -1518,7 +1595,7 @@ store_regs_user_thread (const struct regcache *regcache, pthdb_pthread_t pdtid)
 
   /* Retrieve the thread's current context for its non-register
      values.  */
-  status = pthdb_pthread_context (pd_session, pdtid, &ctx);
+  status = pthdb_pthread_context (data->pd_session, pdtid, &ctx);
   if (status != PTHDB_SUCCESS)
     error (_("aix-thread: store_registers: pthdb_pthread_context returned %s"),
 	   pd_status2str (status));
@@ -1528,7 +1605,7 @@ store_regs_user_thread (const struct regcache *regcache, pthdb_pthread_t pdtid)
   for (i = 0; i < ppc_num_gprs; i++)
     if (REG_VALID == regcache->get_register_status (tdep->ppc_gp0_regnum + i))
       {
-	if (arch64)
+	if (data->arch64)
 	  {
 	    regcache->raw_collect (tdep->ppc_gp0_regnum + i, (void *) &int64);
 	    ctx.gpr[i] = int64;
@@ -1545,7 +1622,7 @@ store_regs_user_thread (const struct regcache *regcache, pthdb_pthread_t pdtid)
     fill_fprs (regcache, ctx.fpr);
 
   /* Special registers (always kept in ctx as 64 bits).  */
-  if (arch64)
+  if (data->arch64)
     {
       fill_sprs64 (regcache, &ctx.iar, &ctx.msr, &ctx.cr, &ctx.lr, &ctx.ctr,
 			     &ctx.xer, &ctx.fpscr);
@@ -1576,7 +1653,7 @@ store_regs_user_thread (const struct regcache *regcache, pthdb_pthread_t pdtid)
 	ctx.fpscr = tmp_fpscr;
     }
 
-  status = pthdb_pthread_setcontext (pd_session, pdtid, &ctx);
+  status = pthdb_pthread_setcontext (data->pd_session, pdtid, &ctx);
   if (status != PTHDB_SUCCESS)
     error (_("aix-thread: store_registers: "
 	     "pthdb_pthread_setcontext returned %s"),
@@ -1602,6 +1679,9 @@ store_regs_kernel_thread (const struct regcache *regcache, int regno,
   double fprs[ppc_num_fprs];
   struct ptxsprs sprs64;
   struct ptsprs  sprs32;
+  struct aix_thread_variables *data;
+  
+  data = get_thread_data_helper_for_ptid (regcache->ptid ());
 
   if (debug_aix_thread)
     gdb_printf (gdb_stdlog, 
@@ -1613,7 +1693,7 @@ store_regs_kernel_thread (const struct regcache *regcache, int regno,
       || (tdep->ppc_gp0_regnum <= regno
 	  && regno < tdep->ppc_gp0_regnum + ppc_num_fprs))
     {
-      if (arch64)
+      if (data->arch64)
 	{
 	  /* Pre-fetch: some regs may not be in the cache.  */
 	  ptrace64aix (PTT_READ_GPRS, tid, (unsigned long) gprs64, 0, NULL);
@@ -1646,7 +1726,7 @@ store_regs_kernel_thread (const struct regcache *regcache, int regno,
 
   if (regno == -1 || special_register_p (gdbarch, regno))
     {
-      if (arch64)
+      if (data->arch64)
 	{
 	  /* Pre-fetch: some registers won't be in the cache.  */
 	  ptrace64aix (PTT_READ_SPRS, tid, 
@@ -1703,7 +1783,7 @@ aix_thread_target::store_registers (struct regcache *regcache, int regno)
   struct thread_info *thread;
   pthdb_tid_t tid;
 
-  if (!PD_TID (regcache->ptid ()))
+  if (regcache->ptid ().tid () == 0)
     beneath ()->store_registers (regcache, regno);
   else
     {
@@ -1741,7 +1821,7 @@ aix_thread_target::mourn_inferior ()
 {
   target_ops *beneath = this->beneath ();
 
-  pd_deactivate ();
+  pd_disable ();
   beneath->mourn_inferior ();
 }
 
@@ -1750,7 +1830,7 @@ aix_thread_target::mourn_inferior ()
 bool
 aix_thread_target::thread_alive (ptid_t ptid)
 {
-  if (!PD_TID (ptid))
+  if (ptid.tid () == 0)
     return beneath ()->thread_alive (ptid);
 
   /* We update the thread list every time the child stops, so all
@@ -1766,7 +1846,7 @@ aix_thread_target::thread_alive (ptid_t ptid)
 std::string
 aix_thread_target::pid_to_str (ptid_t ptid)
 {
-  if (!PD_TID (ptid))
+  if (ptid.tid () == 0)
     return beneath ()->pid_to_str (ptid);
 
   return string_printf (_("Thread %s"), pulongest (ptid.tid ()));
@@ -1786,8 +1866,11 @@ aix_thread_target::extra_thread_info (struct thread_info *thread)
   pthdb_detachstate_t detachstate;
   int cancelpend;
   static char *ret = NULL;
+  struct aix_thread_variables *data;
+
+  data = get_thread_data_helper_for_ptid (thread->ptid);
 
-  if (!PD_TID (thread->ptid))
+  if (thread->ptid.tid () == 0)
     return NULL;
 
   string_file buf;
@@ -1800,24 +1883,24 @@ aix_thread_target::extra_thread_info (struct thread_info *thread)
     /* i18n: Like "thread-identifier %d, [state] running, suspended" */
     buf.printf (_("tid %d"), (int)tid);
 
-  status = pthdb_pthread_state (pd_session, pdtid, &state);
+  status = pthdb_pthread_state (data->pd_session, pdtid, &state);
   if (status != PTHDB_SUCCESS)
     state = PST_NOTSUP;
   buf.printf (", %s", state2str (state));
 
-  status = pthdb_pthread_suspendstate (pd_session, pdtid, 
+  status = pthdb_pthread_suspendstate (data->pd_session, pdtid, 
 				       &suspendstate);
   if (status == PTHDB_SUCCESS && suspendstate == PSS_SUSPENDED)
     /* i18n: Like "Thread-Id %d, [state] running, suspended" */
     buf.printf (_(", suspended"));
 
-  status = pthdb_pthread_detachstate (pd_session, pdtid, 
+  status = pthdb_pthread_detachstate (data->pd_session, pdtid, 
 				      &detachstate);
   if (status == PTHDB_SUCCESS && detachstate == PDS_DETACHED)
     /* i18n: Like "Thread-Id %d, [state] running, detached" */
     buf.printf (_(", detached"));
 
-  pthdb_pthread_cancelpend (pd_session, pdtid, &cancelpend);
+  pthdb_pthread_cancelpend (data->pd_session, pdtid, &cancelpend);
   if (status == PTHDB_SUCCESS && cancelpend)
     /* i18n: Like "Thread-Id %d, [state] running, cancel pending" */
     buf.printf (_(", cancel pending"));
diff --git a/gdb/solib-aix.c b/gdb/solib-aix.c
index f483f54de13..6be81064ebd 100644
--- a/gdb/solib-aix.c
+++ b/gdb/solib-aix.c
@@ -618,6 +618,20 @@ solib_aix_bfd_open (const char *pathname)
       if (member_name == bfd_get_filename (object_bfd.get ()))
 	break;
 
+      std::string s = bfd_get_filename (object_bfd.get ());
+
+      /* For every inferior after first int bfd system we 
+	 will have the pathname instead of the member name
+	 registered. Hence the below condition exists.  */
+
+      if (s.find ('(') != std::string::npos)
+	{
+	  int pos = s.find ('(');
+	  int len = s.find (')') - s.find ('(');
+	  if (s.substr (pos+1, len-1) == member_name) 
+	    return object_bfd;
+	}
+
       object_bfd = gdb_bfd_openr_next_archived_file (archive_bfd.get (),
 						     object_bfd.get ());
     }
-- 
2.31.1