LKML watch

LKML watch

[Masami Hiramatsu]

Hi,

Here are issues posted on LKML in the last week.

OOM-killer too aggressive?
http://lkml.org/lkml/2006/2/26/58
We can track the memory allocation requests and find which request
actually kicks the OOM killer.

udevd is killing file write performance.
http://lkml.org/lkml/2006/2/22/80
The inotify locks a directory and udevd consumes a lot of cpu time.
http://lkml.org/lkml/2006/2/23/97
And in this mail, he traced back the stack to get the function arguments
by using kdb. I think we can provide some tracing scripts to do that.

NFS Still broken in 2.6.x?
http://lkml.org/lkml/2006/2/23/282
The performance of NFS server using linux2.6 is slower than linux2.4.
http://lkml.org/lkml/2006/2/24/151
And in this mail, he found the NFS performance degradation occurred
only with the "cfq" I/O scheduler. I think we can obtain how NFS
server works with I/O schedulers by using the systemtap.

Bio & Biovec-1 increasing cache size, never freed during disk IO
http://lkml.org/lkml/2006/2/25/184
This problem was solved. I think if we use the systemtap, we can find
the resource leaks by checking reference counter on enter and exit of
functions.

Re: lockd: couldn't create RPC handle for (host)
http://lkml.org/lkml/2006/2/27/302
NFS lockd has had a problem. It reported since Dec 18, 2005.
But the oops still occurs with 2.6.15.4. Some arguments or variables
may be corrupted. I think the systemtap can solve this problem
by tracing those variables.

-- 
Masami HIRAMATSU
2nd Research Dept.
Hitachi, Ltd., Systems Development Laboratory
E-mail: hiramatu@sdl.hitachi.co.jp

LKML Watch

[Satoshi Oshima]

I think the cases that systemtap might help to solve are below.

(1)
OOM kills if swappiness set to 0, swap storms otherwise
http://lkml.org/lkml/2006/3/27/250

I am simply trying to run a Gnome desktop (Gnome 2.14, Evolution,
Firefox, gtk-gnutella usually open) without swapping or getting OOM
killed.
I have to set the swappiness to 0 or else I get swap storms when simply
browsing the web and reading my mail.  I think this is insane as I have
512MB of RAM.  It seems as if the kernel will OOM kill firefox rather
than shrink the file cache!


(2)
dcache leak in 2.6.16-git8
http://lkml.org/lkml/2006/3/27/7

A 2GB x86-64 desktop system here is currently swapping itself to death after
a few days uptime.
Some investigation shows this:

inode_cache         1287   1337    568    7    1 : tunables   54   27    8 : slabdata    191    191      0
dentry_cache      1867436 1867643    208   19    1 : tunables  120   60    8 : slabdata  98297  98297      0

Going to reboot it now.

comment> One of the most difficult debug is slab related bug. 
comment> I hope systemtap can help this type of bug.


(3)
KERNEL: assertion (!sk->sk_forward_alloc) failed at net/...
http://lkml.org/lkml/2006/3/29/4

Kernel logs assertions below.
----
Mar 27 16:16:45 king kernel: KERNEL: assertion (!sk->sk_forward_alloc) failed at net/core/stream.c (283)
Mar 27 16:16:45 king kernel: KERNEL: assertion (!sk->sk_forward_alloc) failed at net/ipv4/af_inet.c (150)
----
commnet> Systemtap will help to track down the root cause.


(4)
DTrace for Linux
http://lkml.org/lkml/2006/3/29/6

I am planning to port DTrace (Solaris 10) utility to Linux. I have
done good enough research for same. Is this port of DTrace to Linux
already been started/completed ?
Please let me know your comments on this.

comment> We should give more publicity. 8)


Satoshi

LKML Watch

[Hien Nguyen]

Doubt about scheduler-
I have to know some things of the scheduler. The runqueue struct, and so on
http://marc.theaimsgroup.com/?l=linux-kernel&m=114133213804909&w=2

Readahead value 128K?
Does this happen with a seek call as well, or is this limited
to fseek?
http://marc.theaimsgroup.com/?l=linux-kernel&m=114159989109853&w=2

TCP control block interdependence
I have to investigate some link layer protocol impacts on TCP 
performance...so I have some emulator working on linux, and I was 
investigating the impact of delay spikes.
http://marc.theaimsgroup.com/?l=linux-kernel&m=114138206106106&w=2

LKML watch

[Hien Nguyen]

network delays, mysteriuos push packets
http://marc.theaimsgroup.com/?l=linux-kernel&m=113926568811275&w=2
This person tries to figure out why
Occasionally, at some point, the transmission slows way down

Stale NFS file handle
http://marc.theaimsgroup.com/?l=sun-managers&m=113760926704519&w=2

This person looking for a way to track down NFS errors. Someone
suggested him to get the newer kernel that would solve his problem.

time function behaving strane
http://marc.theaimsgroup.com/?l=linux-kernel&m=113894024731534&w=2
 
Measure the Accept Queuing time
http://marc.theaimsgroup.com/?l=linux-kernel&m=113892007931597&w=2

He wants measure the amount of time when a open request is in 
the accept queue. Frank has posted a reply to this thread and offer some solution using systemtap

LKML watch

[Hien Nguyen]

Stale NFS file handle
http://marc.theaimsgroup.com/?l=sun-managers&m=113760926704519&w=2

This person looking for a way to track down NFS errors. Someone
suggested him to get the newer kernel that would solve his problem.

time function behaving strane
http://marc.theaimsgroup.com/?l=linux-kernel&m=113894024731534&w=2
 
Measure the Accept Queuing time
http://marc.theaimsgroup.com/?l=linux-kernel&m=113892007931597&w=2

He wants measure the amount of time when a open request is in 
the accept queue. The only problem is his kernel is 2.4.25.

LKML Watch

[Stone, Joshua I]

It was my turn to watch the LKML list for SystemTap opportunities.

I didn't see anything that screamed SystemTap to me, but here's one that
I thought might be applicable...


Subject: Size-128 slab leak
http://lkml.org/lkml/2006/1/30/387

This was followed up later by a patch to detect slab leaks.
http://lkml.org/lkml/2006/2/2/47

One could probably write a quick SystemTap script that does the same
thing as the given patch, without needing to recompile the kernel.
Simply put a probe on the point to instrument (within slabinfo_write)
and use embedded-C to walk the structures and report the same
information.


On a related note, it might be nice to put together a BKM for monitoring
the LKML, or at least hear others' methods for doing so.  I found it
difficult to sift through all the traffic, so I'm wondering how others
have been going at it...

Thanks,

Josh

Re: LKML watch

[William Cohen]

Looking over LKML Found the two threads that have stuff of interest to
SystemTap work.


sendfile() with 100 simultaneous 100MB files
http://www.ussg.iu.edu/hypermail/linux/kernel/0601.2/2077.html

Performance problems when sending many large files. Not clear what the
root cause is. Currently, workaround in userspace. Ben LaHaise
had a hypthesis that it was requests overrun in the disk elevator:

http://www.ussg.iu.edu/hypermail/linux/kernel/0601.2/2378.html

There is a bugzilla entry for this problem:

http://bugzilla.kernel.org/show_bug.cgi?id=5949

---

RCU latency regression in 2.6.16-rc1
http://www.ussg.iu.edu/hypermail/linux/kernel/0601.3/0022.html

Port of latency tracer used to show latency problem. This is the type
of problem that people would be interested in using SystemTap for.

The latency tracer patches:
http://people.redhat.com/mingo/latency-tracing-patches/

-Will

LKML watch

[Hien Nguyen]

A way to verify that shared memory is actually being used
http://marc.theaimsgroup.com/?l=linux-kernel&m=113742102209908&w=2

Out of memory: Killed process...
process got killed when executing dd command.
http://marc.theaimsgroup.com/?t=113766047000002&r=1&w=2

IO performance
http://marc.theaimsgroup.com/?l=linux-kernel&m=113740069508429&w=2

reiserfs mount time - why does it take so long to mount a reiserfs?
(Already, some answers were given)
http://marc.theaimsgroup.com/?l=linux-kernel&m=113675913317424&w=2