debug of multithreaded program turns to be impossible during execution

debug of multithreaded program turns to be impossible during execution

[Avi Gozlan]

Hello,

I'm debugging a multithreaded program using GDB. Various threads are created and exit during execution. GDB copes successfully with the multiple threads (e.g. 'thread apply 3 bt' reports successfully backtrace of thread 3), but then at one point loses control of the program:

(gdb) c
Continuing.
[New Thread 0x41401940 (LWP 13370)]
 [Thread 0x40a00940 (LWP 13333) exited]
[New Thread 0x41e02940 (LWP 13371)]
...
[New Thread 0x45a08940 (LWP 13377)]
[New Thread 0x46409940 (LWP 13378)]
 [Thread 0x41e02940 (LWP 13371) exited]
 [Switching to Thread 0x45a08940 (LWP 13377)]
Cannot remove breakpoints because program is no longer writable.
Further execution is probably impossible.
0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
ptrace: No such process.

Surprisingly, although debugging is no more possible, the program is not dead:

(gdb) q
A debugging session is active.

        Inferior 1 [process 13296] will be killed.

Quit anyway? (y or n) y
[Thread 0x46409940 (LWP 13378) exited]
[Thread 0x45a08940 (LWP 13377) exited]
...
[Thread 0x2aaaaccbe580 (LWP 13296) exited]

Some details:
GDB version: 7.6
OS: RHEL 5.1 based
GLIBC: glibc-2.5-18
/lib64/libthread-1.0.so is not stripped.

Your assistance will be appreciated.
Thanks,
Avi

Re: debug of multithreaded program turns to be impossible during execution

[Pedro Alves]

On 07/31/2014 10:27 AM, Avi Gozlan wrote:
> Hello,
> 
> I'm debugging a multithreaded program using GDB. Various threads are created and exit during execution. GDB copes successfully with the multiple threads (e.g. 'thread apply 3 bt' reports successfully backtrace of thread 3), but then at one point loses control of the program:
> 
> (gdb) c
> Continuing.
> [New Thread 0x41401940 (LWP 13370)]
>  [Thread 0x40a00940 (LWP 13333) exited]
> [New Thread 0x41e02940 (LWP 13371)]
> ...
> [New Thread 0x45a08940 (LWP 13377)]
> [New Thread 0x46409940 (LWP 13378)]
>  [Thread 0x41e02940 (LWP 13371) exited]
>  [Switching to Thread 0x45a08940 (LWP 13377)]
> Cannot remove breakpoints because program is no longer writable.
> Further execution is probably impossible.
> 0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
> ptrace: No such process.
> 
> Surprisingly, although debugging is no more possible, the program is not dead:

I have a vague recollection that this was fixed a while ago.  Like,
GDB was picking the wrong thread to access memory, and it ended up
picking a running or exited thread, and that fails (in either
case ptrace complains the same say).

> GDB version: 7.6

Is there a chance you could try a more recent GDB version?  Ideally
7.8, which we just released.

Thanks,
Pedro Alves

RE: debug of multithreaded program turns to be impossible during execution

[Avi Gozlan]

GDB 7.8 yields a similar behavior:

(gdb) c
Continuing.
...
[New Thread 0x43c05940 (LWP 3311)]
[New Thread 0x44606940 (LWP 3312)]
...
[Thread 0x44606940 (LWP 3312) exited]
[Switching to Thread 0x43c05940 (LWP 3311)]
Cannot remove breakpoints because program is no longer writable.
Further execution is probably impossible.
0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
ptrace: No such process.
(gdb) [Thread 0x46409940 (LWP 3315) exited]
[Thread 0x45a08940 (LWP 3314) exited]
[Thread 0x43c05940 (LWP 3311) exited]
[Inferior 1 (process 3268) exited normally]

The only difference is that in 7.8 case, GDB continues automatically after giving the ptrace message.

Avi

-----Original Message-----
From: Pedro Alves [mailto:palves@redhat.com] 
Sent: Thursday, July 31, 2014 1:17 PM
To: Avi Gozlan; gdb@sourceware.org
Subject: Re: debug of multithreaded program turns to be impossible during execution

On 07/31/2014 10:27 AM, Avi Gozlan wrote:
> Hello,
> 
> I'm debugging a multithreaded program using GDB. Various threads are created and exit during execution. GDB copes successfully with the multiple threads (e.g. 'thread apply 3 bt' reports successfully backtrace of thread 3), but then at one point loses control of the program:
> 
> (gdb) c
> Continuing.
> [New Thread 0x41401940 (LWP 13370)]
>  [Thread 0x40a00940 (LWP 13333) exited] [New Thread 0x41e02940 (LWP 
> 13371)] ...
> [New Thread 0x45a08940 (LWP 13377)]
> [New Thread 0x46409940 (LWP 13378)]
>  [Thread 0x41e02940 (LWP 13371) exited]  [Switching to Thread 
> 0x45a08940 (LWP 13377)] Cannot remove breakpoints because program is 
> no longer writable.
> Further execution is probably impossible.
> 0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
> ptrace: No such process.
> 
> Surprisingly, although debugging is no more possible, the program is not dead:

I have a vague recollection that this was fixed a while ago.  Like, GDB was picking the wrong thread to access memory, and it ended up picking a running or exited thread, and that fails (in either case ptrace complains the same say).

> GDB version: 7.6

Is there a chance you could try a more recent GDB version?  Ideally 7.8, which we just released.

Thanks,
Pedro Alves


Email secured by Check Point.

Re: debug of multithreaded program turns to be impossible during execution

[Pedro Alves]

On 08/14/2014 04:52 PM, Avi Gozlan wrote:
> GDB 7.8 yields a similar behavior:
> 
> (gdb) c
> Continuing.
> ...
> [New Thread 0x43c05940 (LWP 3311)]
> [New Thread 0x44606940 (LWP 3312)]
> ...
> [Thread 0x44606940 (LWP 3312) exited]
> [Switching to Thread 0x43c05940 (LWP 3311)]
> Cannot remove breakpoints because program is no longer writable.
> Further execution is probably impossible.
> 0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
> ptrace: No such process.
> (gdb) [Thread 0x46409940 (LWP 3315) exited]
> [Thread 0x45a08940 (LWP 3314) exited]
> [Thread 0x43c05940 (LWP 3311) exited]
> [Inferior 1 (process 3268) exited normally]
> 
> The only difference is that in 7.8 case, GDB continues automatically after giving the ptrace message.

Any chance you could try your reproducer with a more recent
system?  Mainly, I'm wondering whether kernel/glibc are at play here.

Also, a run with:

 set debug infrun 1
 set debug target 1
 set debug lin-lwp 1

may help diagnose the issue.

Thanks,
Pedro Alves

Re: debug of multithreaded program turns to be impossible during execution

[Alex Eckert]

Hi, did you ever resolve this? I'm seeing the same issue I think.