[Bug translator/3785] New: systmtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6 kernel

[Bug translator/3785] New: systmtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6 kernel

[wcohen at redhat dot com]

The systemtap.sample/poll_map.stp test built with the current systemtap
translator (20061222) dies on FC6 machines. The test is getting a NULL
dereference when running:

BUG: unable to handle kernel NULL pointer dere8 printing eip:
c0460579
*pde = 00000000
Oops: 0000 [#1]
SMP
last sysfs file: /module/uhci_hcd/sections/.text
Modules linked in: stap_a428a285aca60807a64f09f4e76207fd_35809(U) autofs4 hidp
dCPU:    0
EIP:    0060:[<c0460579>]    Not tainted VLI
EFLAGS: 00210202   (2.6.18-1.2868.fc6 #1)

Commenting out the entire probe kernel.function( "sys_*" ) avoids crashing the
machine. However, just commenting out the body of the  probe kernel.function(
"sys_*" ) still crashes.

-- 
           Summary: systmtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6
                    kernel
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: critical
          Priority: P2
         Component: translator
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: wcohen at redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/3785] systmtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6 kernel

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2007-01-01 16:51 -------
This is an IBM T41 Thinkpad with a Pentium M processor running FC6 updated via
yum. The machine has has 512M of memory.

The problem is still triggered on the system with the latest checkout of systemtap:

BUG: unable to handle kernel NULL pointer dere8
 printing eip:                                                                  
c0460579                                                                        
*pde = 06579067                                                                 
Oops: 0000 [#1]                                                                 
SMP                                                                             
last sysfs file: /module/uhci_hcd/sections/.text                                
Modules linked in: stap_a058fcd7029f5c9bb738cfca0ac4c0fc_35799(U) autofs4 hidp d
CPU:    0                                                                       
EIP:    0060:[<c0460579>]    Not tainted VLI                                    
EFLAGS: 00210202   (2.6.18-1.2868.fc6 #1)

I don't yet have a kdump from the problem.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/3785] systmtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6 kernel

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2007-01-01 17:37 -------
I added some text to http://sourceware.org/systemtap/wiki/HowToReportBugs to
help gather needed info.  See the bottom few points of 'System crashes'.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/3785] systemtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6 kernel

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2007-01-01 19:39 -------
I have thought about this a bit more and I am wondering if the elfutils-0.123 on
the fc6 machine might be causing the problem. The systemtap is built using
elfutils 0.124 and has it's own shared library installed in a local directory.

Looking at the position of the EIP it looks like it is in sys_munlockall of this
kernel. However, having a probe that only instruments that one function doesn't
crash. The probe is set at 0xc046054aUL the EIP is reported at c0460579:

c0460549 <sys_munlockall>:
c0460549:	53                   	push   %ebx
c046054a:	89 e0                	mov    %esp,%eax
c046054c:	25 00 f0 ff ff       	and    $0xfffff000,%eax
c0460551:	8b 00                	mov    (%eax),%eax
c0460553:	8b 80 84 00 00 00    	mov    0x84(%eax),%eax
c0460559:	83 c0 38             	add    $0x38,%eax
c046055c:	e8 e7 8e fd ff       	call   c0439448 <down_write>
c0460561:	31 c0                	xor    %eax,%eax
c0460563:	e8 65 fd ff ff       	call   c04602cd <do_mlockall>
c0460568:	89 c3                	mov    %eax,%ebx
c046056a:	89 e0                	mov    %esp,%eax
c046056c:	25 00 f0 ff ff       	and    $0xfffff000,%eax
c0460571:	8b 00                	mov    (%eax),%eax
c0460573:	8b 80 84 00 00 00    	mov    0x84(%eax),%eax
c0460579:	83 c0 38             	add    $0x38,%eax
c046057c:	e8 b1 8e fd ff       	call   c0439432 <up_write>
c0460581:	89 d8                	mov    %ebx,%eax
c0460583:	5b                   	pop    %ebx
c0460584:	c3                   	ret    


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|systmtap.sample/poll_map.stp|systemtap.sample/poll_map.st
                   |dies on 2.6.18-1.2868.fc6   |p dies on 2.6.18-1.2868.fc6
                   |kernel                      |kernel


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/3785] systemtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6 kernel

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2007-01-01 20:10 -------
Get exactly the same crash with using the stock systemtap rpm with the
poll_map.stp. The machine has virtually no load on it when the example is run
(and crashes).  This machine is normal installation of fedora core 6 updated
with "yum update". What was the i686 configuration used to attempt to replicate
this problem? Below are details about the machine.

$ rpm -q systemtap gcc elfutils
systemtap-0.5.10-1.fc6
gcc-4.1.1-30
elfutils-0.123-1.fc6
$ uname -a
Linux montague.devel.redhat.com 2.6.18-1.2868.fc6 #1 SMP Fri Dec 15 17:31:29 EST
2006 i686 i686 i386 GNU/Linux
[wcohen@montague systemtap.samples]$ more /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 9
model name      : Intel(R) Pentium(R) M processor 1600MHz
stepping        : 5
cpu MHz         : 1594.855
cache size      : 1024 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr mce cx8 mtrr pge mca cmov pat clflush d
ts acpi mmx fxsr sse sse2 tm pbe up est tm2
bogomips        : 3190.87
$ free
             total       used       free     shared    buffers     cached
Mem:        514652     236688     277964          0      32188     150120
-/+ buffers/cache:      54380     460272
Swap:      1050800          0    1050800

$ stap -v -k
/home/wcohen/systemtap_write/src/testsuite/systemtap.samples/poll_map.stp
Pass 1: parsed user script and 53 library script(s) in 330usr/10sys/391real ms.
Pass 2: analyzed script: 3 probe(s), 2 function(s), 2 global(s) in
300usr/30sys/1889real ms.
Pass 3: translated to C into "/tmp/stapvZHsjn/stap_2669.c" in
80usr/30sys/204real ms.
Pass 4: compiled C into "stap_2669.ko" in 2860usr/260sys/4843real ms.
Pass 5: starting run.

The same messages appears on the console:

BUG: unable to handle kernel NULL pointer dere8
 printing eip:                                                                  
c0460579                                                                        
*pde = 0e445067                                                                 
Oops: 0000 [#1]                                                                 
SMP                                                                             
last sysfs file: /block/hda/removable                                           
Modules linked in: stap_2669(U) autofs4 hidp rfcomm l2cap bluetooth sunrpc ip_cd
CPU:    0                                                                       
EIP:    0060:[<c0460579>]    Not tainted VLI                                    
EFLAGS: 00210202   (2.6.18-1.2868.fc6 #1)    

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/3785] systemtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6 kernel

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2007-01-02 21:31 -------
The stock kdump in the fc6 doesn't work with Pentium M because the processor
doesn't have the PAE support and the fc6 kernel-kdump requires PAE support. :(

Suspect the problem is related to Pentium M or the laptop environment.



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/3785] systemtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6 kernel

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2007-01-05 15:46 -------
Narrowed down the problem to working with FC5 kernel but failing with the FC6
kernel.

Looking through the archive of test results the poll_map.stp test last ran
successfully was 

stap_testing_200611070930/obj/testsuite/systemtap.log:PASS: poll_map (1)

The source code for this is the same as the one that is currently crashing.
Attempted to use the stap translator built at that time. It crashes in the same way.

/home/wcohen/stap_testing_200611070930/install/bin/stap -v
/home/wcohen/stap_testing_200611070930/src/testsuite/systemtap.samples/poll_map.stp 

When the switch on the machine from FC5 to FC6 occurred the test stopped
working. Trying the FC5 kernel, 2.6.18-1.2200.fc5, on the machine and things
work. It appears to be an issue with the kernel.

2.6.18-1.2200.fc5 i686 kernel works, but the 2.6.18-1.2869.fc6 i686 kernel fails.





-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|translator                  |kprobes


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/3785] systemtap.sample/poll_map.stp dies on 2.6.18-1.2868.fc6 kernel

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2007-01-05 20:42 -------
Built a kernel locally with the same configuration as the stock kernel from the
source code from kernel-2.6.18-1.2869.fc6.src.rpm. The test ran fine with the
locally built kernel. There are differences in the locations of functions in the
System.map files.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/3785] mismatched kernel and debuginfo architecture causing system crashes

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2007-01-09 15:13 -------
This is caused by FC6 installing a i586 kernel and kernel-devel rather than a
i686 kernel and kernel-devel on the laptop and i686 debuginfo being installed.
Systemtap checks that the debuginfo version but not the architecture. The
mismatch causes some probes to be off. Can check that architecture information
is reasonable with

 rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n" | grep kernel

The kernel-debuginfo, kernel, and kernel-devel should be for the same arch.

It appears that anaconda installed the incorrect architecture kernel on the machine.

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=211941



-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |FIXED
            Summary|systemtap.sample/poll_map.st|mismatched kernel and
                   |p dies on 2.6.18-1.2868.fc6 |debuginfo architecture
                   |kernel                      |causing system crashes


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/3785] mismatched kernel and debuginfo architecture causing system crashes

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2007-01-09 15:59 -------
It is unfortunate that there appears to be no way, based on the ELF files only,
to tell whether they come from the i586 vs. i686 builds.  If it were possible,
then the recently added checks in tapsets.cxx (query_module) could look for it.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=3785

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Controlling a systemtap script from userspace

[David A Sperry]

Hi,
    I am interested in logging kernel stuff only during small
portions of a long test. About 10 milliseconds of logging every
10 minutes. I have a C++ program than knows when to do the
logging, the question is how to get that user space C++ program to set a
global in a systemTap script?  Is there an obscure syscall
or something else I can use as a probe to set a global?

For example:


#!/usr/bin/env stap
#
# This script sporatically lists the top 20 systemcalls on the system
#

global syscalls isLogging

# turn on logging
probe ### something from user space ### {isLogging = 1}

# turn off logging
probe ### something else from user space #### {isLogging = 0}

# probe gated with isLogging
probe kernel.function("sys_*") {
        if (isLogging == 1) {
      syscalls[probefunc()]++
        }
}

# print top syscalls every 5 seconds
probe timer.ms(5000) {
      print_top ()
}

function print_top () {
      cnt=0
      printf ("SYSCALL\t\t\t\tCOUNT\n")
      foreach ([name] in syscalls-) {
            printf("%-20s\t\t%5d\n",name, syscalls[name])
            if (cnt++ == 20)
                  break
      }
      printf("--------------------------------------\n")
      delete syscalls
}

Re: Controlling a systemtap script from userspace

[James Dickens]

On 1/9/07, David A Sperry <David_A_Sperry@raytheon.com> wrote:
> Hi,
>     I am interested in logging kernel stuff only during small
> portions of a long test. About 10 milliseconds of logging every
> 10 minutes. I have a C++ program than knows when to do the
> logging, the question is how to get that user space C++ program to set a
> global in a systemTap script?  Is there an obscure syscall
> or something else I can use as a probe to set a global?
>

one easy way is to have the userland app execute a signal, and your
systemtap script can watch for the signal and verify that it was
called by your application via its commandname you could use standard
signals such as

 SIGUSR1   30,10,16    Term    User-defined signal 1
       SIGUSR2   31,12,17    Term    User-defined signal 2

or you could fire a non normal one such as signal 255 or something if
you are worried your signal being thought of as a normal signal.

James Dickens
uadmin.blogspot.com

> For example:
>
>
> #!/usr/bin/env stap
> #
> # This script sporatically lists the top 20 systemcalls on the system
> #
>
> global syscalls isLogging
>
> # turn on logging
> probe ### something from user space ### {isLogging = 1}
>
> # turn off logging
> probe ### something else from user space #### {isLogging = 0}
>
> # probe gated with isLogging
> probe kernel.function("sys_*") {
>         if (isLogging == 1) {
>       syscalls[probefunc()]++
>         }
> }
>
> # print top syscalls every 5 seconds
> probe timer.ms(5000) {
>       print_top ()
> }
>
> function print_top () {
>       cnt=0
>       printf ("SYSCALL\t\t\t\tCOUNT\n")
>       foreach ([name] in syscalls-) {
>             printf("%-20s\t\t%5d\n",name, syscalls[name])
>             if (cnt++ == 20)
>                   break
>       }
>       printf("--------------------------------------\n")
>       delete syscalls
> }
>
>

Re: Controlling a systemtap script from userspace

[Martin Hunt]

On Tue, 2007-01-09 at 17:12 -0500, David A Sperry wrote:
>  I have a C++ program than knows when to do the
> logging, the question is how to get that user space C++ program to set a
> global in a systemTap script?  

I recommend something like the following:

# This probe checks for some unusual filenames which are
# actually signals from userspace to toggle logging on and off.
probe syscall.open {
	if (filename == "####ON####") {
		isLogging = 1
		printf("Logging ON\n")
	} else if (filename == "####OFF####") {
		isLogging = 0
		printf("Logging OFF\n")
	}
}

Then in C or C++, you simple toggle logging with calls to open and the
appropriate special filename.

Martin

Re: Controlling a systemtap script from userspace

[Frank Ch. Eigler]

> [...] the question is how to get that user space C++ program to set
> a global in a systemTap script?  [...]

Other than hunt's solution, someone will eventually implement
our old bug #1154, which would allow direct expression thusly:

> # turn on logging
> probe ### something from user space ### {isLogging = 1}
> # turn off logging
> probe ### something else from user space #### {isLogging = 0}

global logging
probe procfs("stap/myscript/logging").write {
  logging = ($value == "1")
}
...


# echo 1 > /proc/stap/myscript/logging


- FChE