[Bug gdb/18368] New: gdb with nsenter attaches to wrong process

[Bug gdb/18368] New: gdb with nsenter attaches to wrong process

[raghu.prabhu13 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18368

            Bug ID: 18368
           Summary: gdb with nsenter attaches to wrong process
           Product: gdb
           Version: 7.9
            Status: NEW
          Severity: normal
          Priority: P2
         Component: gdb
          Assignee: unassigned at sourceware dot org
          Reporter: raghu.prabhu13 at gmail dot com
  Target Milestone: ---

When trying to attach to a process in a different PID namespace, gdb correctly
identifies it and warns as "warning: Target and debugger are in different PID
namespaces; thread lists and other data are likely unreliable"

however, when used with nsenter (-p for pid namespace), it attaches to the
wrong process.


nsenter  -t 600  -p gdb -p 1      (where 600 is the PID of docker daemon, and 1
is the PID of process inside it, whose host pid is 29120).

GNU gdb (GDB) 7.9
.......
......
Attaching to process 1
Reading symbols from /usr/lib/systemd/systemd...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/libcap.so.2...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/libpam.so.0...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/libkmod.so.2...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/libmount.so.1...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/libseccomp.so.2...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/librt.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Reading symbols from /usr/lib/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/libattr.so.1...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/liblzma.so.5...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libblkid.so.1...(no debugging symbols
found)...done.
Reading symbols from /usr/lib/libuuid.so.1...(no debugging symbols
found)...done.
0x00007f1439c951d3 in __epoll_wait_nocancel () from /usr/lib/libc.so.6
(gdb) quit
....

As you can see it attaches to systemd of the host than the process inside the
container.

nsenter  -t 600  -p gdb -p 29120 also gives the PID namespace warning and
doesn't work.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/18368] gdb with nsenter attaches to wrong process

[palves at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18368

Pedro Alves <palves at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |gbenson at redhat dot com,
                   |                            |palves at redhat dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/18368] gdb with nsenter attaches to wrong process

[gbenson at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18368

--- Comment #1 from Gary Benson <gbenson at redhat dot com> ---
You're using nsenter to enter the PID namespace but you're not entering the
corresponding mount namespace; GDB is looking in the wrong /proc.  Try adding
"-m" to your nsenter options.

Also, I can't imagine how using the docker daemon as a reference can work.  As
I understand it the docker daemon does not switch namespaces, so "nsenter -t
$DOCKERPID" should just stay in whatever namespaces you were already in.

If you're building from source, try this branch:

  git clone -b namespaces https://github.com/gbenson/binutils-gdb.git

With that GDB you should be able to attach with "gdb -p $HOSTPID" (where
$HOSTPID is 29120 in your example).

Let me know if/how you get this working please.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/18368] gdb with nsenter attaches to wrong process

[raghu.prabhu13 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18368

--- Comment #2 from Raghavendra <raghu.prabhu13 at gmail dot com> ---
>Also, I can't imagine how using the docker daemon as a reference can work.  As >I understand it the docker daemon does not switch namespaces, so "nsenter -t 
>$DOCKERPID" should just stay in whatever namespaces you were already in

You are right. The namespace remains same for daemon as is the docker client. 


>You're using nsenter to enter the PID namespace but you're not entering the >corresponding mount namespace; GDB is looking in the wrong /proc.  Try adding >"-m" to your nsenter options.

I did try - nsenter-t 25057 -p -m  gdb -p 1 - but this fails since gdb is not
present there.
(where 25057 is the pid of process inside container - in this case sleep
100000)


>If you're building from source, try this branch:

>  git clone -b namespaces https://github.com/gbenson/binutils-gdb.git

I will try this and get back.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/18368] gdb with nsenter attaches to wrong process

[gbenson at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18368

--- Comment #3 from Gary Benson <gbenson at redhat dot com> ---
You do need to have GDB or gdbserver inside the container if you're going to
use nsenter.  (If just gdbserver, start GDB with no options and do "target
remote | nsenter ... gdbserver - --attach 1")

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/18368] gdb with nsenter attaches to wrong process

[raghu.prabhu13 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18368

--- Comment #4 from Raghavendra <raghu.prabhu13 at gmail dot com> ---
So, I did try with namespaces branch and it works great.


sudo gdb -p 31802
GNU gdb (GDB) 7.9.50.20150429-cvs
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 31802
Reading symbols from target:/usr/bin/sleep...Reading symbols from
target:/usr/bin/sleep...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Reading symbols from target:/lib64/libc.so.6...(no debugging symbols
found)...done.
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.

warning: Target and debugger are in different PID namespaces; thread lists and
other data are likely unreliable
0x00007fdaa0a74470 in __nanosleep_nocancel () from target:/lib64/libc.so.6
(gdb) bt
#0  0x00007fdaa0a74470 in __nanosleep_nocancel () from target:/lib64/libc.so.6
#1  0x0000000000403e5f in rpl_nanosleep ()
#2  0x0000000000403cc0 in xnanosleep ()
#3  0x00000000004016cd in main ()
(gdb) quit
A debugging session is active.

        Inferior 1 [process 31802] will be detached.

Quit anyway? (y or n) y
Detaching from program: target:/usr/bin/sleep, process 31802

==================

a) I also verified this from inside container by installing gdb there. (You
need a privileged container for that which is strange).

b) It still shows  the warning - "warning: Target and debugger are in different
PID namespaces; thread lists and other data are likely unreliable", is it still
required?


c) Also, -Werror build fails for that branch:
https://github.com/gbenson/binutils-gdb/issues/1


Thanks.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/18368] gdb with nsenter attaches to wrong process

[gbenson at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18368

Gary Benson <gbenson at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

--- Comment #5 from Gary Benson <gbenson at redhat dot com> ---
(In reply to Raghavendra from comment #4)
> So, I did try with namespaces branch and it works great.
[snip]

Great!

> a) I also verified this from inside container by installing gdb there.
> (You need a privileged container for that which is strange).

What error do you get if the container isn't privileged?

> b) It still shows  the warning - "warning: Target and debugger are in
> different PID namespaces; thread lists and other data are likely
> unreliable", is it still required?

I'm not sure.  The way libthread_db works I think everything should be
consistent, but I'm still experimenting.

> c) Also, -Werror build fails for that branch:
> https://github.com/gbenson/binutils-gdb/issues/1

I'll look into it.

I'm closing this bug now (the original issue isn't a GDB bug) but we can carry
on discussing things here.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/18368] gdb with nsenter attaches to wrong process

[raghu.prabhu13 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18368

--- Comment #6 from Raghavendra <raghu.prabhu13 at gmail dot com> ---
For unprivileged container (ie. without --privileged or --cap-add sys_ptrace):

I see

gdb -p 1
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 1
ptrace: Operation not permitted.
(gdb) bt
No stack.
(gdb)


=========



>I'm not sure.  The way libthread_db works I think everything should be
>consistent, but I'm still experimenting.

Does this require just the gdb to be installed or any other binutil components
updated as well? 


Regarding

>You do need to have GDB or gdbserver inside the container if you're going to >use nsenter.  (If just gdbserver, start GDB with no options and do "target >remote | nsenter ... gdbserver - --attach 1")

I haven't been able to get this working since it required gdbserver inside
container.  Can you show how this is done?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/18368] gdb with nsenter attaches to wrong process

[gbenson at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18368

--- Comment #7 from Gary Benson <gbenson at redhat dot com> ---
You just need to install the gdbserver command (/usr/bin/gdbserver usually).
For Fedora and RHEL this is "yum install gdb-gdbserver", for Debian etc I think
it's "apt-get install gdbserver".

-- 
You are receiving this mail because:
You are on the CC list for the bug.