ptrace PEEKTEXT IO error?!

ptrace PEEKTEXT IO error?!

[David Lecomber]

Dear all,

I have a multithreaded code which is misbehaving in GDB (verified on
latest CVS).  After threads are created, the target suddenly seems to
become unwriteable, and even unreadable in some bits.

(gdb) n
Single stepping until exit from function sleep,
which has no line number information.
main (argc=1, argv=0x7ffffffff3d8, environ=0x7ffffffff3e8) at hello.c:89
89              MPI_Init(&argc, &argv);
(gdb) n
[New Thread 32769 (LWP 18096)]
[New Thread 16386 (LWP 18097)]
[New Thread 32771 (LWP 18098)]
90              MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
(gdb) info threads
  4 Thread 32771 (LWP 18098)  0x00002aaaab5f8bc9 in ioctl ()
from /lib64/libc.so.6
  3 Thread 16386 (LWP 18097)  0x00002aaaab5f8bc9 in ioctl ()
from /lib64/libc.so.6
  2 Thread 32769 (LWP 18096)  0x00002aaaab5f7829 in poll ()
from /lib64/libc.so.6
* 1 Thread 16384 (LWP 17994)  main (argc=1, argv=0x7ffffffff3d8,
environ=0x7ffffffff3e8) at hello.c:90
(gdb) info address hello.c:90
No symbol "hello.c:90" in current context.
(gdb) info line
Line 90 of "hello.c" starts at address 0x402042 <main+316> and ends at
0x402056 <main+336>.
(gdb) n
Warning:
Cannot insert breakpoint 0.
Error accessing memory address 0x402056: Input/output error.
 

If I then strace the running GDB, I see the following errors (and a few
more ptrace errors).. 

ptrace(PTRACE_PEEKTEXT, 17994, 0x407480, [0]) = -1 EIO (Input/output
error)
ptrace(PTRACE_POKEDATA, 17994, 0x407480, 0xffffffffffffccff) = -1 EIO
(Input/output error)


Anyone know what could be going on here?  It's a  2.6.11  kernel, x86_64
based on Debian.

Cheers
David

Re: ptrace PEEKTEXT IO error?!

[Daniel Jacobowitz]

On Wed, Oct 05, 2005 at 06:04:00PM +0100, David Lecomber wrote:
> Dear all,
> 
> I have a multithreaded code which is misbehaving in GDB (verified on
> latest CVS).  After threads are created, the target suddenly seems to
> become unwriteable, and even unreadable in some bits.

This usually means the thread is not stopped.  Failing that, check with
your kernel.

> If I then strace the running GDB, I see the following errors (and a few
> more ptrace errors).. 
> 
> ptrace(PTRACE_PEEKTEXT, 17994, 0x407480, [0]) = -1 EIO (Input/output
> error)
> ptrace(PTRACE_POKEDATA, 17994, 0x407480, 0xffffffffffffccff) = -1 EIO
> (Input/output error)

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: ptrace PEEKTEXT IO error?!

[David Lecomber]

On Wed, 2005-10-05 at 13:11 -0400, Daniel Jacobowitz wrote:
> On Wed, Oct 05, 2005 at 06:04:00PM +0100, David Lecomber wrote:
> > Dear all,
> > 
> > I have a multithreaded code which is misbehaving in GDB (verified on
> > latest CVS).  After threads are created, the target suddenly seems to
> > become unwriteable, and even unreadable in some bits.
> 
> This usually means the thread is not stopped.  Failing that, check with
> your kernel.

'ps' says the processes is stopped - so I guess that makes it a kernel
woe.

I've knocked up a ptrace test code, and it also shows the same behaviour
independently of GDB.   

It seems ok at reading memory from addresses on the heap and stack --
but it barfs at reading memory from the text segment.  Is there
something that the user code  (one of the libraries called by my
program) could do to prevent access to memory by a ptracing program?


Cheers
David

Re: ptrace PEEKTEXT IO error?!

[David Lecomber]

On Wed, 2005-10-05 at 19:20 +0100, David Lecomber wrote:
> 
> It seems ok at reading memory from addresses on the heap and stack --
> but it barfs at reading memory from the text segment.  Is there
> something that the user code  (one of the libraries called by my
> program) could do to prevent access to memory by a ptracing program?
> 

and I should have asked.. or is it only possible for the kernel, or a
kernel module to do that?

Cheers
David
-- 
David Lecomber, CTO, Allinea Software
tel: +44 1926 623231  fax: +44 1926 623232

Re: ptrace PEEKTEXT IO error?!

[Daniel Jacobowitz]

On Wed, Oct 05, 2005 at 07:20:11PM +0100, David Lecomber wrote:
> On Wed, 2005-10-05 at 13:11 -0400, Daniel Jacobowitz wrote:
> > On Wed, Oct 05, 2005 at 06:04:00PM +0100, David Lecomber wrote:
> > > Dear all,
> > > 
> > > I have a multithreaded code which is misbehaving in GDB (verified on
> > > latest CVS).  After threads are created, the target suddenly seems to
> > > become unwriteable, and even unreadable in some bits.
> > 
> > This usually means the thread is not stopped.  Failing that, check with
> > your kernel.
> 
> 'ps' says the processes is stopped - so I guess that makes it a kernel
> woe.
> 
> I've knocked up a ptrace test code, and it also shows the same behaviour
> independently of GDB.   
> 
> It seems ok at reading memory from addresses on the heap and stack --
> but it barfs at reading memory from the text segment.  Is there
> something that the user code  (one of the libraries called by my
> program) could do to prevent access to memory by a ptracing program?

No, this is most likely a kernel bug.  This is mapped code, right, not
some mmaped device file (which is not generally accessible by ptrace)?

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: ptrace PEEKTEXT IO error?!

[David Lecomber]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]


> > 
> > It seems ok at reading memory from addresses on the heap and stack --
> > but it barfs at reading memory from the text segment.  [snip]

> No, this is most likely a kernel bug.  This is mapped code, right, not
> some mmaped device file (which is not generally accessible by ptrace)?

The code I'm trying to read is an ordinary subroutine in an ordinary C
file.  The program does call a library handling infiniband network
cards, and there are loads of kernel modules on this machine looking
after that.  

There are some mmap commands in the strace output -- they seem to be in
the wrong range to be causing the trouble (I've attached an strace log
of the binary, those mmap commands might make sense to someone..) - FYI,
the memory address it can't read is  0x401e18 (4202008), which is right
in the middle of my ordinary code..



Cheers
David

-- 
David Lecomber 
CTO Allinea Software
Tel: +44 1926 623231
Fax: +44 1926 623232

[-- Attachment #2: infi.log --]
[-- Type: text/x-log, Size: 21999 bytes --]

execve("/opt/mpi/bin/mpirun", ["mpirun", "-np", "1", "/usr/bin/strace", "-f", "./mpi"], [/* 47 vars */]) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
execve("./mpi", ["./mpi"], [/* 56 vars */]) = 0
uname({sys="Linux", node="snode164", ...}) = 0
brk(0)                                  = 0x54c000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaaaabf000
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory)
open("/opt/intel_cce_81-027/lib/tls/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/intel_cce_81-027/lib/tls/x86_64", 0x7fffffffea10) = -1 ENOENT (No such file or directory)
open("/opt/intel_cce_81-027/lib/tls/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/intel_cce_81-027/lib/tls", 0x7fffffffea10) = -1 ENOENT (No such file or directory)
open("/opt/intel_cce_81-027/lib/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/intel_cce_81-027/lib/x86_64", 0x7fffffffea10) = -1 ENOENT (No such file or directory)
open("/opt/intel_cce_81-027/lib/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/intel_cce_81-027/lib", {st_mode=S_IFDIR|0755, st_size=944, ...}) = 0
open("/apps/prod/openib/x86_64/driver/infinihost/lib/tls/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/apps/prod/openib/x86_64/driver/infinihost/lib/tls/x86_64", 0x7fffffffea10) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/tls/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/apps/prod/openib/x86_64/driver/infinihost/lib/tls", 0x7fffffffea10) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/apps/prod/openib/x86_64/driver/infinihost/lib/x86_64", 0x7fffffffea10) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/apps/prod/openib/x86_64/driver/infinihost/lib", {st_mode=S_IFDIR|0755, st_size=280, ...}) = 0
open("/home/rzsunhome/rz51/rz51031/tls/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/rzsunhome/rz51/rz51031/tls/x86_64", 0x7fffffffea10) = -1 ENOENT (No such file or directory)
open("/home/rzsunhome/rz51/rz51031/tls/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/rzsunhome/rz51/rz51031/tls", 0x7fffffffea10) = -1 ENOENT (No such file or directory)
open("/home/rzsunhome/rz51/rz51031/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/rzsunhome/rz51/rz51031/x86_64", 0x7fffffffea10) = -1 ENOENT (No such file or directory)
open("/home/rzsunhome/rz51/rz51031/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/rzsunhome/rz51/rz51031", {st_mode=S_IFDIR|0700, st_size=1536, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY)      = 6
fstat(6, {st_mode=S_IFREG|0644, st_size=10801, ...}) = 0
mmap(NULL, 10801, PROT_READ, MAP_PRIVATE, 6, 0) = 0x2aaaaaac0000
close(6)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib64/libdl.so.2", O_RDONLY)     = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\35\0"..., 640) = 640
fstat(6, {st_mode=S_IFREG|0644, st_size=12248, ...}) = 0
mmap(NULL, 1058904, PROT_READ|PROT_EXEC, MAP_PRIVATE, 6, 0) = 0x2aaaaabc2000
mprotect(0x2aaaaabc5000, 1046616, PROT_NONE) = 0
mmap(0x2aaaaacc2000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x2aaaaacc2000
close(6)                                = 0
open("/opt/intel_cce_81-027/lib/libmtl_common.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/libmtl_common.so", O_RDONLY) = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\n\0"..., 640) = 640
fstat(6, {st_mode=S_IFREG|0755, st_size=10894, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaaacc5000
mmap(NULL, 1056488, PROT_READ|PROT_EXEC, MAP_PRIVATE, 6, 0) = 0x2aaaaacc6000
mprotect(0x2aaaaacc8000, 1048296, PROT_NONE) = 0
mmap(0x2aaaaadc6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x2aaaaadc6000
close(6)                                = 0
open("/opt/intel_cce_81-027/lib/libvapi.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/libvapi.so", O_RDONLY) = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\200"..., 640) = 640
fstat(6, {st_mode=S_IFREG|0755, st_size=232369, ...}) = 0
mmap(NULL, 1252944, PROT_READ|PROT_EXEC, MAP_PRIVATE, 6, 0) = 0x2aaaaadc8000
mprotect(0x2aaaaadf2000, 1080912, PROT_NONE) = 0
mmap(0x2aaaaaec8000, 204800, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x2aaaaaec8000
close(6)                                = 0
open("/opt/intel_cce_81-027/lib/libmosal.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/libmosal.so", O_RDONLY) = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320(\0\0"..., 640) = 640
fstat(6, {st_mode=S_IFREG|0755, st_size=34902, ...}) = 0
mmap(NULL, 1074632, PROT_READ|PROT_EXEC, MAP_PRIVATE, 6, 0) = 0x2aaaaaefa000
mprotect(0x2aaaaaf00000, 1050056, PROT_NONE) = 0
mmap(0x2aaaaaffa000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x2aaaaaffa000
close(6)                                = 0
open("/opt/intel_cce_81-027/lib/libmpga.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/libmpga.so", O_RDONLY) = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\32\0\0"..., 640) = 640
fstat(6, {st_mode=S_IFREG|0755, st_size=35636, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaab001000
mmap(NULL, 1078456, PROT_READ|PROT_EXEC, MAP_PRIVATE, 6, 0) = 0x2aaaab002000
mprotect(0x2aaaab008000, 1053880, PROT_NONE) = 0
mmap(0x2aaaab102000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x2aaaab102000
close(6)                                = 0
open("/opt/intel_cce_81-027/lib/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/rzsunhome/rz51/rz51031/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib64/libpthread.so.0", O_RDONLY) = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260S\0\0"..., 640) = 640
fstat(6, {st_mode=S_IFREG|0644, st_size=92805, ...}) = 0
mmap(NULL, 1654688, PROT_READ|PROT_EXEC, MAP_PRIVATE, 6, 0) = 0x2aaaab10a000
mprotect(0x2aaaab118000, 1597344, PROT_NONE) = 0
mmap(0x2aaaab20a000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x2aaaab20a000
mmap(0x2aaaab21a000, 540576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2aaaab21a000
close(6)                                = 0
open("/opt/intel_cce_81-027/lib/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/rzsunhome/rz51/rz51031/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib64/libm.so.6", O_RDONLY)      = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`@\0\0\0"..., 640) = 640
fstat(6, {st_mode=S_IFREG|0644, st_size=563296, ...}) = 0
mmap(NULL, 1609160, PROT_READ|PROT_EXEC, MAP_PRIVATE, 6, 0) = 0x2aaaab29e000
mprotect(0x2aaaab321000, 1072584, PROT_NONE) = 0
mmap(0x2aaaab39e000, 561152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x2aaaab39e000
close(6)                                = 0
open("/opt/intel_cce_81-027/lib/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/rzsunhome/rz51/rz51031/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib64/libgcc_s.so.1", O_RDONLY)  = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\37\0\0"..., 640) = 640
fstat(6, {st_mode=S_IFREG|0644, st_size=44768, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaab427000
mmap(NULL, 1091744, PROT_READ|PROT_EXEC, MAP_PRIVATE, 6, 0) = 0x2aaaab428000
mprotect(0x2aaaab432000, 1050784, PROT_NONE) = 0
mmap(0x2aaaab528000, 45056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x2aaaab528000
close(6)                                = 0
open("/opt/intel_cce_81-027/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/apps/prod/openib/x86_64/driver/infinihost/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/rzsunhome/rz51/rz51031/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib64/libc.so.6", O_RDONLY)      = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\332\1"..., 640) = 640
fstat(6, {st_mode=S_IFREG|0644, st_size=1292528, ...}) = 0
mmap(NULL, 2352776, PROT_READ|PROT_EXEC, MAP_PRIVATE, 6, 0) = 0x2aaaab533000
mprotect(0x2aaaab64d000, 1197704, PROT_NONE) = 0
mmap(0x2aaaab733000, 237568, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0x100000) = 0x2aaaab733000
mmap(0x2aaaab76d000, 18056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2aaaab76d000
close(6)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaab772000
arch_prctl(0x1002, 0x2aaaab772540)      = 0
munmap(0x2aaaaaac0000, 10801)           = 0
getpid()                                = 22774
rt_sigaction(SIGRTMIN, {0x2aaaab112980, [], 0x4000000}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x2aaaab112a60, [], 0x4000000}, NULL, 8) = 0
rt_sigaction(SIGRT_2, {0x2aaaab113290, [], 0x4000000}, NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [RTMIN], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1], NULL, 8) = 0
_sysctl({{CTL_KERN, KERN_VERSION}, 2, 0x7ffffffff010, 36, (nil), 0}) = 0
open("/dev/mosal", O_RDONLY)            = 6
brk(0)                                  = 0x54c000
brk(0x56d000)                           = 0x56d000
brk(0)                                  = 0x56d000
uname({sys="Linux", node="snode164", ...}) = 0
open("/dev/vipkl_dev", O_RDONLY)        = 7
ioctl(7, 0x80087801, 0x7ffffffff1a0)    = 0
ioctl(7, 0x80087801, 0x7ffffffff180)    = 0
ioctl(7, 0x80087801, 0x7ffffffff160)    = 0
ioctl(7, 0x80087801, 0x7ffffffff1b0)    = 0
ioctl(7, 0x80087801, 0x7ffffffff120)    = 0
ioctl(7, 0x80087801, 0x7ffffffff0a0)    = 0
ioctl(7, 0x80087801, 0x7ffffffff100)    = 0
ioctl(7, 0x80087801, 0x7ffffffff0c0)    = 0
ioctl(7, 0x80087801, 0x7ffffffff0c0)    = 0
getrlimit(0x3, 0x7fffffffeeb0)          = 0
pipe([8, 9])                            = 0
clone(Process 22775 attached
child_stack=0x560040, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND) = 22775
[pid 22774] write(9, "\300\360\377\377\377\177\0\0\5\0\0\0\252*\0\0\260|B\253"..., 168 <unfinished ...>
[pid 22775] arch_prctl(0x1002, 0x560100 <unfinished ...>
[pid 22774] <... write resumed> )       = 168
[pid 22775] <... arch_prctl resumed> )  = 0
[pid 22774] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid 22775] rt_sigprocmask(SIG_SETMASK, ~[TRAP RT_1],  <unfinished ...>
[pid 22774] <... rt_sigprocmask resumed> [RTMIN], 8) = 0
[pid 22775] <... rt_sigprocmask resumed> NULL, 8) = 0
[pid 22774] write(9, "@%w\253\252*\0\0\0\0\0\0\252*\0\0000\361\377\377\377\177"..., 168 <unfinished ...>
[pid 22775] read(8,  <unfinished ...>
[pid 22774] <... write resumed> )       = 168
[pid 22775] <... read resumed> "\300\360\377\377\377\177\0\0\5\0\0\0\252*\0\0\260|B\253"..., 168) = 168
[pid 22774] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid 22775] poll( <unfinished ...>
[pid 22774] <... rt_sigprocmask resumed> [RTMIN], 8) = 0
[pid 22775] <... poll resumed> [{fd=8, events=POLLIN, revents=POLLIN}], 1, 2000) = 1
[pid 22774] rt_sigsuspend([] <unfinished ...>
[pid 22775] getppid()                   = 22774
[pid 22775] read(8, "@%w\253\252*\0\0\0\0\0\0\252*\0\0000\361\377\377\377\177"..., 168) = 168
[pid 22775] mmap(NULL, 2097152, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaab773000
[pid 22775] mprotect(0x2aaaab773000, 4096, PROT_NONE) = 0
[pid 22775] clone(Process 22776 attached
child_stack=0x2aaaab972ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|SIGRT_1) = 22776
[pid 22775] kill(22774, SIGRTMIN <unfinished ...>
[pid 22774] --- SIGRTMIN (Unknown signal 32) @ 0 (0) ---
[pid 22775] <... kill resumed> )        = 0
[pid 22774] <... rt_sigsuspend resumed> ) = -1 EINTR (Interrupted system call)
[pid 22775] poll( <unfinished ...>
[pid 22774] rt_sigreturn(0x20)          = -1 EINTR (Interrupted system call)
[pid 22776] arch_prctl(0x1002, 0x560840) = 0
[pid 22776] getpid()                    = 22776
[pid 22776] rt_sigprocmask(SIG_SETMASK, [RTMIN], NULL, 8) = 0
[pid 22776] sched_setscheduler(0x58f8, 0, 0x5609e4) = 0
[pid 22776] ioctl(7, 0x80087801, 0x2aaaab972e40) = 0
[pid 22774] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid 22776] ioctl(7, 0x80087801 <unfinished ...>
[pid 22774] <... rt_sigprocmask resumed> [RTMIN], 8) = 0
[pid 22774] write(9, "@%w\253\252*\0\0\0\0\0\0\252*\0\0000\361\377\377\377\177"..., 168 <unfinished ...>
[pid 22775] <... poll resumed> [{fd=8, events=POLLIN, revents=POLLIN}], 1, 2000) = 1
[pid 22774] <... write resumed> )       = 168
[pid 22775] getppid( <unfinished ...>
[pid 22774] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid 22775] <... getppid resumed> )     = 22774
[pid 22775] read(8, "@%w\253\252*\0\0\0\0\0\0\252*\0\0000\361\377\377\377\177"..., 168) = 168
[pid 22774] <... rt_sigprocmask resumed> [RTMIN], 8) = 0
[pid 22775] mmap(NULL, 2097152, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
[pid 22774] rt_sigsuspend([] <unfinished ...>
[pid 22775] <... mmap resumed> )        = 0x2aaaab973000
[pid 22775] mprotect(0x2aaaab973000, 4096, PROT_NONE) = 0
[pid 22775] clone(Process 22777 attached
child_stack=0x2aaaabb72ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|SIGRT_1) = 22777
[pid 22775] kill(22774, SIGRTMIN <unfinished ...>
[pid 22777] arch_prctl(0x1002, 0x560f20 <unfinished ...>
[pid 22774] --- SIGRTMIN (Unknown signal 32) @ 0 (0) ---
[pid 22775] <... kill resumed> )        = 0
[pid 22777] <... arch_prctl resumed> )  = 0
[pid 22774] <... rt_sigsuspend resumed> ) = -1 EINTR (Interrupted system call)
[pid 22775] poll( <unfinished ...>
[pid 22777] getpid( <unfinished ...>
[pid 22774] rt_sigreturn(0x20)          = -1 EINTR (Interrupted system call)
[pid 22777] <... getpid resumed> )      = 22777
[pid 22777] rt_sigprocmask(SIG_SETMASK, [RTMIN], NULL, 8) = 0
[pid 22777] sched_setscheduler(0x58f9, 0, 0x5610c4) = 0
[pid 22777] ioctl(7, 0x80087801, 0x2aaaabb72e40) = 0
[pid 22777] ioctl(7, 0x80087801 <unfinished ...>
[pid 22774] brk(0)                      = 0x56d000
[pid 22774] brk(0x5c2000)               = 0x5c2000
[pid 22774] brk(0)                      = 0x5c2000
[pid 22774] brk(0x5e4000)               = 0x5e4000
[pid 22774] brk(0)                      = 0x5e4000
[pid 22774] brk(0x606000)               = 0x606000
[pid 22774] brk(0)                      = 0x606000
[pid 22774] brk(0x628000)               = 0x628000
[pid 22774] brk(0)                      = 0x628000
[pid 22774] brk(0x64a000)               = 0x64a000
[pid 22774] brk(0)                      = 0x64a000
[pid 22774] brk(0x66c000)               = 0x66c000
[pid 22774] brk(0)                      = 0x66c000
[pid 22774] brk(0x68e000)               = 0x68e000
[pid 22774] brk(0)                      = 0x68e000
[pid 22774] brk(0x6b0000)               = 0x6b0000
[pid 22774] ioctl(7, 0x80087801, 0x7ffffffff1b0) = 0
[pid 22774] ioctl(7, 0x80087801, 0x7fffffffefd0) = 0
[pid 22774] brk(0)                      = 0x6b0000
[pid 22774] brk(0x8c5000)               = 0x8c5000
[pid 22774] ioctl(7, 0x80087801, 0x7fffffffefc0) = 0
[pid 22774] ioctl(7, 0x80087801, 0x7ffffffff1b0) = 0
[pid 22774] getpid()                    = 22774
[pid 22774] open("/etc/resolv.conf", O_RDONLY) = 10
[pid 22774] fstat(10, {st_mode=S_IFREG|0644, st_size=79, ...}) = 0
[pid 22774] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaabb75000
[pid 22774] read(10, "search rrze.uni-erlangen.de\nname"..., 4096) = 79
[pid 22774] read(10, "", 4096)          = 0
[pid 22774] close(10)                   = 0
[pid 22774] munmap(0x2aaaabb75000, 4096) = 0
[pid 22774] socket(PF_UNIX, SOCK_STREAM, 0) = 10
[pid 22774] connect(10, {sa_family=AF_UNIX, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 22774] close(10)                   = 0
[pid 22774] open("/etc/nsswitch.conf", O_RDONLY) = 10
[pid 22774] fstat(10, {st_mode=S_IFREG|0644, st_size=425, ...}) = 0
[pid 22774] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaabb75000
[pid 22774] read(10, "# /etc/nsswitch.conf\n#\n# Example"..., 4096) = 425
[pid 22774] read(10, "", 4096)          = 0
[pid 22774] close(10)                   = 0
[pid 22774] munmap(0x2aaaabb75000, 4096) = 0
[pid 22774] open("/opt/intel_cce_81-027/lib/libnss_files.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 22774] open("/apps/prod/openib/x86_64/driver/infinihost/lib/libnss_files.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 22774] open("/home/rzsunhome/rz51/rz51031/libnss_files.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 22774] open("/etc/ld.so.cache", O_RDONLY) = 10
[pid 22774] fstat(10, {st_mode=S_IFREG|0644, st_size=10801, ...}) = 0
[pid 22774] mmap(NULL, 10801, PROT_READ, MAP_PRIVATE, 10, 0) = 0x2aaaabb75000
[pid 22774] close(10)                   = 0
[pid 22774] access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
[pid 22774] open("/lib64/libnss_files.so.2", O_RDONLY) = 10
[pid 22774] read(10, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340%\0\0"..., 640) = 640
[pid 22774] fstat(10, {st_mode=S_IFREG|0644, st_size=44736, ...}) = 0
[pid 22774] mmap(NULL, 1092296, PROT_READ|PROT_EXEC, MAP_PRIVATE, 10, 0) = 0x2aaaabb78000
[pid 22774] mprotect(0x2aaaabb81000, 1055432, PROT_NONE) = 0
[pid 22774] mmap(0x2aaaabc78000, 45056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 10, 0) = 0x2aaaabc78000
[pid 22774] close(10)                   = 0
[pid 22774] munmap(0x2aaaabb75000, 10801) = 0
[pid 22774] open("/etc/host.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 22774] open("/etc/hosts", O_RDONLY) = 10
[pid 22774] fcntl(10, F_GETFD)          = 0
[pid 22774] fcntl(10, F_SETFD, FD_CLOEXEC) = 0
[pid 22774] fstat(10, {st_mode=S_IFREG|0644, st_size=74, ...}) = 0
[pid 22774] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaabb75000
[pid 22774] read(10, "127.0.0.1\tlocalhost\n192.168.80.1"..., 4096) = 74
[pid 22774] close(10)                   = 0
[pid 22774] munmap(0x2aaaabb75000, 4096) = 0
[pid 22774] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 10
[pid 22774] socket(PF_UNIX, SOCK_STREAM, 0) = 11
[pid 22774] connect(11, {sa_family=AF_UNIX, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 22774] close(11)                   = 0
[pid 22774] open("/etc/hosts", O_RDONLY) = 11
[pid 22774] fcntl(11, F_GETFD)          = 0
[pid 22774] fcntl(11, F_SETFD, FD_CLOEXEC) = 0
[pid 22774] fstat(11, {st_mode=S_IFREG|0644, st_size=74, ...}) = 0
[pid 22774] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaabb75000
[pid 22774] read(11, "127.0.0.1\tlocalhost\n192.168.80.1"..., 4096) = 74
[pid 22774] close(11)                   = 0
[pid 22774] munmap(0x2aaaabb75000, 4096) = 0
[pid 22774] connect(10, {sa_family=AF_INET, sin_port=htons(35556), sin_addr=inet_addr("192.168.80.164")}, 16) = 0
[pid 22774] write(10, "\2\0\0\0", 4)    = 4
[pid 22774] write(10, "\0\0\0\0", 4)    = 4
[pid 22774] write(10, "\10\0\0\0", 4)   = 4
[pid 22774] write(10, "!\1\0\0\300\250P\244", 8) = 8
[pid 22774] read(10, "!\1\0\0!\1\0\0\300\250P\244", 12) = 12
[pid 22774] brk(0)                      = 0x8c5000
[pid 22774] brk(0x4361000)              = 0x4361000
[pid 22774] ioctl(7, 0x80087801, 0x7fffffffee40) = 0
[pid 22774] write(10, "\0\0\0\0", 4)    = 4
[pid 22774] read(10, "\0\0\0\0", 4)     = 4
[pid 22774] close(10)                   = 0
[pid 22774] getuid()                    = 36231
[pid 22774] open("/tmp/ib_shmem-snode164-36231.tmp", O_RDWR|O_CREAT, 0777) = 10
[pid 22774] ftruncate(10, 1940)         = 0
[pid 22774] write(10, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1940) = 1940
[pid 22774] lseek(10, 0, SEEK_SET)      = 0
[pid 22774] fstat(10, {st_mode=S_IFREG|0700, st_size=1940, ...}) = 0
[pid 22774] mmap(NULL, 1940, PROT_READ|PROT_WRITE, MAP_SHARED, 10, 0) = 0x2aaaabb75000
[pid 22774] unlink("/tmp/ib_shmem-snode164-36231.tmp") = 0
[pid 22774] getpid()                    = 22774
[pid 22774] getpid()                    = 22774
[pid 22774] write(1, "./ptrace_test 22774 70597968 140"..., 136) = 136
[pid 22774] rt_sigprocmask(SIG_BLOCK, [CHLD], [RTMIN], 8) = 0
[pid 22774] rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8) = 0
[pid 22774] rt_sigprocmask(SIG_SETMASK, [RTMIN], NULL, 8) = 0
[pid 22774] nanosleep({20, 0},  <unfinished ...>
[pid 22775] <... poll resumed> [{fd=8, events=POLLIN}], 1, 2000) = 0
[pid 22775] getppid()                   = 22774
[pid 22775] poll(

Re: ptrace PEEKTEXT IO error?!

[Daniel Jacobowitz]

On Wed, Oct 05, 2005 at 09:06:20PM +0100, David Lecomber wrote:
> 
> > > 
> > > It seems ok at reading memory from addresses on the heap and stack --
> > > but it barfs at reading memory from the text segment.  [snip]
> 
> > No, this is most likely a kernel bug.  This is mapped code, right, not
> > some mmaped device file (which is not generally accessible by ptrace)?
> 
> The code I'm trying to read is an ordinary subroutine in an ordinary C
> file.  The program does call a library handling infiniband network
> cards, and there are loads of kernel modules on this machine looking
> after that.  

The infiniband code does all sorts of funky things to VM.  It's
extremely likely that this is repsonsible.


-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: ptrace PEEKTEXT IO error?!

[David Lecomber]

On Wed, 2005-10-05 at 15:54 -0400, Daniel Jacobowitz wrote:
> On Wed, Oct 05, 2005 at 09:06:20PM +0100, David Lecomber wrote:
> > 
> > > > 
> > > > It seems ok at reading memory from addresses on the heap and stack --
> > > > but it barfs at reading memory from the text segment.  [snip]
> > 
> > > No, this is most likely a kernel bug.  This is mapped code, right, not
> > > some mmaped device file (which is not generally accessible by ptrace)?
> > 
> > The code I'm trying to read is an ordinary subroutine in an ordinary C
> > file.  The program does call a library handling infiniband network
> > cards, and there are loads of kernel modules on this machine looking
> > after that.  
> 
> The infiniband code does all sorts of funky things to VM.  It's
> extremely likely that this is repsonsible.

Great, I bet this bug'll cheer them up!  Thanks for the info, I'll chase
their mailing list now.

Cheers
David