Notes from stap meeting 20060803

Notes from stap meeting 20060803

[Elena Zannoni]

Will took notes this time

---------------------------------------------

kprobes portion-------------

userspace probes
	attaching on two fronts:
	-modified version from spring (prassana addressing issues)
	-utrace other approach (jim and ananth working on that of/on)

some effort testing utrace on s390
need priorities  in notify structure maybe put into utrace
some issues building utrace testing structures
any request to test utrace on ppc64 from redhat?
dave wilder doing some testing, gdb on ptrace on utrace ppc64
performance measurements on userspace probes approaches
	-still need to doing measurements on utrace

someone (anil?) is working on ia64 utrace
Need to agree on which tests to run for utrace testing
	i386/x86_64 redhat (what is rh doing for testing?)
	ia64 intel
	ppc64/s390 ibm
what is plan to get utrace upstream?


bz2091 why failing only on some power64 architectures

bug with itrace similar works on power5, crashes on power4
	mike mason looking at this


systemtap--------------------

redhat: david, will, elena
ibm: jim, david wilder
intel: josh


talked about inviting Mathieu Desnoyes


s390 kprobes should be in rawhide, hasn't been tested
dave wilder will test

any news about userspace probes?
working on getting gdb fixed with utrace
roland has a number of patches to clean things up
need to rerun tests
run tests in gdb source, might need to compare utrace results with 
ptrace run

anil has been testing utrace ia64
elena says utrace incomplete on ia64, need some additional work
frysk uses other parts of utrace, and gdb testing single threaded
is there an strace testsuite?

userspace probes?
prasanna persuing the two approaches.
prasanna has done some perf testing of earlier approach
nothing on utrace implementation
what are chances of getting into rhel5 in the next month?
	gate is closing on rhel5. might get a few features in
have something that works (prasanna approach)
	no code review in 3/4 weeks
	less politically acceptable than utrace approach (jim)
	want to know what kabis these patches will break

dave smith: cross compilation support module location information
	resubmitted kernel patch, and got comments on it.
	will submit upstream
	what about testing it? need two machines?
	also working on probe merge patch
		can low module size significantly

will cohen:
	success on i686 xen running of systemtap tests
	discussion with stephane eranian about perfmon2 testing
	perfmon2 systemtap
	

frank eigler:
	moving testsuite into same package
	adding examples into wiki

josh stone:
	trying to get xen working
	x86-64 both seems to work xen
	timer.profile doesn't work (dynamic ticks) need to investigate
	should xen get the registration to fail?
	or have on dynamic tick code path
	oprofile uses same hook, Will Cohen should check works

bryan
	working on dashboard, collection of scripts other people found useful
	dashboard for new users
	eager to get additional scripts
	how does gui chart data, documentation on this? uses regex

static kernel markers
	frank: has there been any progress been porting block trace code
		to static marker
	satoshi: mentions static probe mechanism still have to be careful
		with changing code at static marker. could cause fault
	frank: no changes in code space to avoid these problems
	tomz: where to get code to do this? stap_mark.h in systemtap
		documented in man page? also described in ols paper
	vara: do we want to put static tracing in rhel5?
		frank doesn't see it happening

satoshi: lastest rawhide kernel	split of debuginfo. which rpms are required?
	yum install just the "kernel package name" -debuginfo	

Re: Notes from stap meeting 20060803

[Roland McGrath]

> satoshi: lastest rawhide kernel	split of debuginfo. which rpms are required?
> 	yum install just the "kernel package name" -debuginfo	

yum install KERNELPACKAGENAME-debuginfo.ARCH will get you all the rpms you need
by yum dependency magic.  Use KERNELPACKAGENAME-debuginfo-VER-REL.ARCH to
be paranoidly sure you are downloading the right version for your kernel
before you get the 150MB.  You may need --enablerepo=core-debuginfo
--enablerepo=updates-debuginfo or --enablerepo=development-debuginfo
options to yum if you have not editted your /etc/yum.repos.d files.


Thanks,
Roland

Re: Notes from stap meeting 20060803

[Roland McGrath]

These utrace issues are not really specific to systemtap, though of
course they are important if utrace will be used by systemtap, and
I'm glad for anyone's involvement in the utrace work.  Anyone who is
also involved in the Frysk project will be having the same
discussions with me over there.

> some effort testing utrace on s390

David Wilder and I have been working on this.  It looks like we
broke things again the other day, but things have been substantially
working already.  The main work is done.  The code is enabled in the
rawhide s390 kernel.

> need priorities  in notify structure maybe put into utrace

This is probably a reasonable addition to the interface.  I prefer
to wait until there are actually two utrace engines in the world
that can concretely vie for priority before implementing the solution.

> any request to test utrace on ppc64 from redhat?

I have a Mac G5 with Fedora and so can debug on this platform
myself, and tested and debugged the port when I did it.  But I have
not done any regular regime of testing.  I haven't tested other
kinds of ppc64 machines, or configurations with more than 2 CPUs
(nor fewer).  I have not done any heavy stress-testing.  Having
someone concerned with each platform doing regular testing on
the machine configurations of interest is certainly worthwhile.

> someone (anil?) is working on ia64 utrace

Anil Keshavamurthy and Bibo Mao have begun work on this, after some
administrative delay beyond their personal control.  We are still hashing
out the best implementation approaches for ia64, but it seems to be coming
along now.

> Need to agree on which tests to run for utrace testing

As yet the main means of regression and stress testing for the core
code is via things that use ptrace.  gdb is the only canonical free
thing with a large test suite stressing ptrace, of which I am aware.
I can separately supply technical info on using gdb to test the
kernel behavior, for those who are going to do it.  I expect that we
will get some automated form of this put together for the Fedora
test system, but I don't know any details about that yet.  Right now
I can give instruction for developers to do one-off testing by hand.

I have a small test suite specifically for utrace.  That will get
specific regression cases as they come up, and will eventually get
to be close to exhaustive, but will probably not be stressful.  So
far it doesn't test very much, and the gdb testing is far more useful.

If other folks have existing applications (nonfree or whatever) that
exercise ptrace, then regression testing those on current rawhide
kernels should by all means be done.

In the long run, a lot of serious stress testing would be very
valuable.  That is, running numerous gdb test suites and other such
stressers simultaneously, doing that and other miscellaneous
workloads while running some pervasive, noninvasive tracing modules
monitoring those processes, etc.  

> what is plan to get utrace upstream?

I am working through this with upstream kernel folks.  Getting the
arch work still in progress into good shape is an important factor.


Thanks,
Roland