[WIP] uprobe tests

[WIP] uprobe tests

[Srikar Dronamraju]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

***********************
Warning: Your file, uprobes_srikar.tgz, contains more than 32 files after decompression and cannot be scanned.
***********************


Hi, 

Attached is the current set of tests for uprobes.  A few more test
scenarios have been identified and will be added to this testsuite. 

--
Thanks and Regards
Srikar 


[-- Attachment #2: uprobes_srikar.tgz --]
[-- Type: application/x-tar-gz, Size: 31869 bytes --]

Re: [WIP] uprobe tests

[Srikar Dronamraju]

> Hi, 
> 
> Attached is the current set of tests for uprobes.  A few more test
> scenarios have been identified and will be added to this testsuite. 

Please refer http://sourceware.org/systemtap/wiki/UprobesTestScenarios
for the test scenarios planned/identified. 

Do let me know your comments and suggestions on the test scenarios and
the existing testsuite. 

--
Thanks and Regards
Srikar 

Re: [WIP] uprobe tests

[Frank Ch. Eigler]

Srikar Dronamraju <srikar@linux.vnet.ibm.com> writes:

> Attached is the current set of tests for uprobes.  A few more test
> scenarios have been identified and will be added to this testsuite.

Thanks, it's a good start.  I'll add some test ideas to the wiki
version.

When I run this test suite on i686 2.6.21-rc6-mm1 + may25-uprobes
kernel configured with a bunch of debugging options, we get a bunch of
kernel errors.  A sampling:

slab error in verify_redzone_free(): cache `size-32': memory outside object was 
overwritten
 [<c04787f2>] cache_free_debugcheck+0xb2/0x1a6
 [<c0478b5e>] kfree+0x90/0xe0
 [<f098208b>] u_dbfs_cleanup+0x4b/0x4d [blink2]
 [<f09820d6>] cleanup_module+0x49/0x4b [blink2]

slab error in cache_alloc_debugcheck_after(): cache `size-32': double free, or m
emory outside object was overwritten
 [<c0478674>] cache_alloc_debugcheck_after+0x89/0x155
 [<c0479b44>] kmem_cache_zalloc+0xb5/0xe7
 [<c045c77e>] register_uprobe+0x2ea/0x6cc
 [<f09822f3>] init_module+0x6b/0xa1 [blink]

kfree_debugcheck: out of range ptr 170fc2a5h.
------------[ cut here ]------------
kernel BUG at /home/fche/Private/DEVEL/DEVEL-linux/linux-2.6/mm/slab.c:2851!
Process rmmod (pid: 2225, ti=c38d8000 task=c34ff0a0 task.ti=c38d8000)
Stack: c0683915 170fc2a5 c38d9f20 c0478af2 00000000 bfd09ed0 00000206 f0983700 
       00000000 bfd09ed0 c38d9f30 f0982041 00000000 bfd09ed0 c38d9f50 f098209f 
       f0982449 000008a6 080485a7 c05eebc3 fffffff0 bfd09ed0 c38d9fb0 c0448034 
Call Trace:
 [<c0478af2>] kfree+0x24/0xe0
 [<f0982041>] u_dbfs_cleanup+0x41/0x4d [blink]
 [<f098209f>] cleanup_module+0x52/0x54 [blink]
 [<c0448034>] sys_delete_module+0x180/0x1a8
 [<c0404eb4>] syscall_call+0x7/0xb
INFO: lockdep is turned off.
Code: 5d c3 55 89 c2 8d 80 00 00 00 40 89 e5 c1 e8 0c 83 ec 08 3b 05 84 d8 9f c0
 72 14 89 54 24 04 c7 04 24 15 39 68 c0 e8 dc be fa ff <0f> 0b eb fe c9 c3 55 89
 e5 57 31 ff 56 89 c6 53 89 d3 83 ec 1c 

BUG: sleeping function called from invalid context at /home/fche/Private/DEVEL/D
EVEL-linux/linux-2.6/kernel/rwsem.c:20
in_atomic():0, irqs_disabled():1
INFO: lockdep is turned off.
irq event stamp: 0
hardirqs last  enabled at (0): [<00000000>] 0x0
hardirqs last disabled at (0): [<c0421b2f>] copy_process+0x2f1/0x12c3
softirqs last  enabled at (0): [<c0421b2f>] copy_process+0x2f1/0x12c3
softirqs last disabled at (0): [<00000000>] 0x0
 [<c0405ee1>] dump_trace+0x63/0x1eb
 [<c0406083>] show_trace_log_lvl+0x1a/0x30

The dejagnu results are 14 passes and 5 failures:

FAIL: test for 2 probes one probepoint
FAIL: test for suicide attempt
FAIL: 10 thread test
FAIL: 10k iterations
FAIL: suicide test


- FChE

Re: [WIP] uprobe tests

[Jim Keniston]

On Sat, 2007-06-09 at 17:01 -0400, Frank Ch. Eigler wrote:
> Srikar Dronamraju <srikar@linux.vnet.ibm.com> writes:
> 
> > Attached is the current set of tests for uprobes.  A few more test
> > scenarios have been identified and will be added to this testsuite.
> 
> Thanks, it's a good start.  I'll add some test ideas to the wiki
> version.
> 
> When I run this test suite on i686 2.6.21-rc6-mm1 + may25-uprobes
> kernel configured with a bunch of debugging options, we get a bunch of
> kernel errors.  A sampling:

The suite runs cleanly for me (even on SMP, with today's fix applied).
Could you please provide your .config file, or at least that portion
that shows which debugging options you have enabled?

Most of these problems appear to be associated with the test harness
(which includes u_dbfs_cleanup), but there's one associated with
register_uprobe().

> 
> slab error in verify_redzone_free(): cache `size-32': memory outside object was 
> overwritten
>  [<c04787f2>] cache_free_debugcheck+0xb2/0x1a6
>  [<c0478b5e>] kfree+0x90/0xe0
>  [<f098208b>] u_dbfs_cleanup+0x4b/0x4d [blink2]
>  [<f09820d6>] cleanup_module+0x49/0x4b [blink2]
> 
> slab error in cache_alloc_debugcheck_after(): cache `size-32': double free, or m
> emory outside object was overwritten
>  [<c0478674>] cache_alloc_debugcheck_after+0x89/0x155
>  [<c0479b44>] kmem_cache_zalloc+0xb5/0xe7
>  [<c045c77e>] register_uprobe+0x2ea/0x6cc
>  [<f09822f3>] init_module+0x6b/0xa1 [blink]
> 
> kfree_debugcheck: out of range ptr 170fc2a5h.
> ------------[ cut here ]------------
> kernel BUG at /home/fche/Private/DEVEL/DEVEL-linux/linux-2.6/mm/slab.c:2851!
> Process rmmod (pid: 2225, ti=c38d8000 task=c34ff0a0 task.ti=c38d8000)
> Stack: c0683915 170fc2a5 c38d9f20 c0478af2 00000000 bfd09ed0 00000206 f0983700 
>        00000000 bfd09ed0 c38d9f30 f0982041 00000000 bfd09ed0 c38d9f50 f098209f 
>        f0982449 000008a6 080485a7 c05eebc3 fffffff0 bfd09ed0 c38d9fb0 c0448034 
> Call Trace:
>  [<c0478af2>] kfree+0x24/0xe0
>  [<f0982041>] u_dbfs_cleanup+0x41/0x4d [blink]
>  [<f098209f>] cleanup_module+0x52/0x54 [blink]
>  [<c0448034>] sys_delete_module+0x180/0x1a8
>  [<c0404eb4>] syscall_call+0x7/0xb
> INFO: lockdep is turned off.
> Code: 5d c3 55 89 c2 8d 80 00 00 00 40 89 e5 c1 e8 0c 83 ec 08 3b 05 84 d8 9f c0
>  72 14 89 54 24 04 c7 04 24 15 39 68 c0 e8 dc be fa ff <0f> 0b eb fe c9 c3 55 89
>  e5 57 31 ff 56 89 c6 53 89 d3 83 ec 1c 
> 
> BUG: sleeping function called from invalid context at /home/fche/Private/DEVEL/D
> EVEL-linux/linux-2.6/kernel/rwsem.c:20
> in_atomic():0, irqs_disabled():1
> INFO: lockdep is turned off.
> irq event stamp: 0
> hardirqs last  enabled at (0): [<00000000>] 0x0
> hardirqs last disabled at (0): [<c0421b2f>] copy_process+0x2f1/0x12c3
> softirqs last  enabled at (0): [<c0421b2f>] copy_process+0x2f1/0x12c3
> softirqs last disabled at (0): [<00000000>] 0x0
>  [<c0405ee1>] dump_trace+0x63/0x1eb
>  [<c0406083>] show_trace_log_lvl+0x1a/0x30

It's hard to tell what this refers to.

> 
> The dejagnu results are 14 passes and 5 failures:
> 
> FAIL: test for 2 probes one probepoint
> FAIL: test for suicide attempt
> FAIL: 10 thread test
> FAIL: 10k iterations
> FAIL: suicide test
> 
> 
> - FChE

Jim

Re: [WIP] uprobe tests

[Jim Keniston]

On Sat, 2007-06-09 at 15:55 -0700, Jim Keniston wrote:
> On Sat, 2007-06-09 at 17:01 -0400, Frank Ch. Eigler wrote:
> > Srikar Dronamraju <srikar@linux.vnet.ibm.com> writes:
> > 
...
> > 
> > When I run this test suite on i686 2.6.21-rc6-mm1 + may25-uprobes
> > kernel configured with a bunch of debugging options, we get a bunch of
> > kernel errors.  A sampling:
> 
...
> 
> Most of these problems appear to be associated with the test harness
> (which includes u_dbfs_cleanup), but there's one associated with
> register_uprobe().
...
> > 
> > slab error in verify_redzone_free(): cache `size-32': memory outside object was 
> > overwritten
> >  [<c04787f2>] cache_free_debugcheck+0xb2/0x1a6
> >  [<c0478b5e>] kfree+0x90/0xe0
> >  [<f098208b>] u_dbfs_cleanup+0x4b/0x4d [blink2]
> >  [<f09820d6>] cleanup_module+0x49/0x4b [blink2]
...

I rebuilt kernels with Frank's debugging options enabled.  Here's a
simple fix that yields clean test runs for me.  In the test suite, in
include/udbgfs.c, in the line
	print_buf = kmalloc(sizeof(print_buf),GFP_KERNEL);
change
	sizeof(print_buf)
to
	sizeof(*print_buf)

BTW, I don't think test_printk() handles buffer overflows correctly.
When we reach the end of the buffer, it's possible for vsnprintf() to
return a number greater than print_buf->bytes_left (see "Return value"
in the man page), which means print_buf->bytes_left can underflow to a
very big number, telling the next call to vsnprintf() that we have a
very big buffer.  I don't see any overflows in the test suite that would
test my hypothesis, though, so I'll leave that investigation and fix to
Srikar.

> > 
> > 
> > - FChE

Jim

Re: [WIP] uprobe tests

[Srikar Dronamraju]

> > > slab error in verify_redzone_free(): cache `size-32': memory outside object was 
> > > overwritten
> > >  [<c04787f2>] cache_free_debugcheck+0xb2/0x1a6
> > >  [<c0478b5e>] kfree+0x90/0xe0
> > >  [<f098208b>] u_dbfs_cleanup+0x4b/0x4d [blink2]
> > >  [<f09820d6>] cleanup_module+0x49/0x4b [blink2]
> ...

I was also able to reproduce this with Frank's Kernel Hacking options.


> 
> I rebuilt kernels with Frank's debugging options enabled.  Here's a
> simple fix that yields clean test runs for me.  In the test suite, in
> include/udbgfs.c, in the line
> 	print_buf = kmalloc(sizeof(print_buf),GFP_KERNEL);
> change
> 	sizeof(print_buf)
> to
> 	sizeof(*print_buf)

I have added the fix as suggested by Jim.

> 
> BTW, I don't think test_printk() handles buffer overflows correctly.
> When we reach the end of the buffer, it's possible for vsnprintf() to
> return a number greater than print_buf->bytes_left (see "Return value"
> in the man page), which means print_buf->bytes_left can underflow to a
> very big number, telling the next call to vsnprintf() that we have a
> very big buffer.  I don't see any overflows in the test suite that would
> test my hypothesis, though, so I'll leave that investigation and fix to
> Srikar.

To take care of the overflow that Jim has pointed out, I have modified 
test_printk function to check for the overflow as an interim solution. 
Once we start seeing tests failing due to overflow we can investigate at
a more feasible solution. Please do let me know if this is acceptable.

Here is the modified test_printk function.

int test_printk(const char *fmt, ...)
{
        va_list args;
        int len;

#ifdef UPROBE_DEBUGFS_DEBUG
        printk (KERN_ERR "calling test_printk\n");
#endif  
        if (print_buf->bytes_left == 0) {
                printk (KERN_ERR "test_printk: bytes left is 0\n");
                return -1;
        }

        va_start(args, fmt);

        spin_lock(&print_buf->lock);
        len = vsnprintf(print_buf->cur, print_buf->bytes_left, fmt,
args);

        print_buf->cur += len;
        print_buf->bytes_in_buf += len;
        if (len > print_buf->bytes_left)
                print_buf->bytes_left = 0;
        else
                print_buf->bytes_left -= len;
        spin_unlock(&print_buf->lock);
        va_end(args);

#ifdef UPROBE_DEBUGFS_DEBUG
        printk (KERN_ERR "exiting test_printk\n");
#endif

        return len;
}


--
Thanks and Regards
Srikar 

utrace & systemtap scripts

[Wenji Huang]

Hi guru,

  In trying uprobe, I downloaded mainline kernel linux-2.6.21.tar.bz2 
and got linux-2.6-utrace.patch from 
http://people.redhat.com/roland/utrace/2.6-current/.

  Unfortunately, some failed in patching.
       patching file arch/s390/kernel/process.c
       Hunk #1 FAILED at 321.

       patching file arch/s390/kernel/compat_linux.c
       Hunk #1 FAILED at 513.

       patching file arch/powerpc/kernel/asm-offsets.c
       Hunk #1 FAILED at 59.

       patching file arch/powerpc/kernel/ptrace.c
       Hunk #2 FAILED at 265.
   
       patching file arch/avr32/kernel/ptrace.c
       Hunk #1 FAILED at 5.
    
       patching file kernel/fork.c
       Hunk #11 FAILED at 1364.

       patching file kernel/exit.c
       Hunk #6 FAILED at 296.

       patching file kernel/ptrace.c
       Hunk #1 FAILED at 18.

       patching file kernel/signal.c
       Hunk #4 FAILED at 100.
       Hunk #11 FAILED at 1498.

       patching file include/linux/sched.h
       Hunk #1 FAILED at 823.

    Although I switched "infrastructure of tracing and debugging users 
processes" on, make can't be passed. So I wonder whether there are other 
patches needed or I am not in the right building steps?


BTW:
   I wrote two stap scripts and uploaded them to wiki, you can refer to 
them.

Best regards,

***Wenji Huang
* *
Oracle Asia Research & Development Center
Open Source Technologies Development
*

Re: utrace & systemtap scripts

[Roland McGrath]

> Hi guru,
> 
>   In trying uprobe, I downloaded mainline kernel linux-2.6.21.tar.bz2 
> and got linux-2.6-utrace.patch from 
> http://people.redhat.com/roland/utrace/2.6-current/.

From utrace/README.txt:

	See README.backport for details on supporting older kernel versions.
	[...]
	Patches in the 2.6-current/ subdirectory are against the current Linus
	tree, or a pretty recent one from the last time I rebased it.

You'll need at least 2.6.22-rc4 to use the 2.6-current/ patches.
If you want to use 2.6.21, try the 2.6.21/ patches.


Thanks,
Roland