relayfs changes needed

relayfs changes needed

[Martin Hunt]

We need some relayfs-related changes in the transport and stpd. I know
there has been some discussion on this, but I'm not clear on what needs
to be done. 

Currently, we print to buffers which are then copied to the relayfs
buffers. At the start of every print buffer is a sequence number. stpd
reads the relayfs buffers, and simply dumps them to files. Later the
files are optionally merged by reading the sequence numbers.
Unfortunately, the data format is simply 4-bytes sequence numbers
followed by null-terminated data. This does not work with binary data.

So changes are needed. But before we decide what to change, can we
review how and why relayfs is used?  On typical desktop machines, there
seems to be no reason to use relayfs. On larger systems, especially NUMA
systems, I expect that relayfs will have better performance. For most
scripts this won't matter, but in the case of more complex kernel
tracing packages built upon systemtap, the improved performance and
per-cpu files will both be useful. And in the cases, I expect the output
will be carefully formatted binary.

So if this is the case, I'm wondering if maybe the right thing to do is
to remove all sequence numbers, formatting, and merging in the transport
layer for relayfs and simply let it run full-speed dumping data into
per-cpu files. This gives the highest performance and moves the
responsibility for any merging or other postprocessing to the
application layer, where it can be done most efficiently. So some script
(or program) will invoke systemtap, and on exit, take the binary data
files and process them. Systemtap will not timestamp, sequence, or
otherwise process the data. 

Removing some of the complexity from the transport will greatly simplify
it, allowing us to speed it up further.

We probably need a way to specify the base names for the per-cpu files.
Maybe changing the meaning of "stap -o" for relayfs.

Re: relayfs changes needed

[Gui,Jian]

> So if this is the case, I'm wondering if maybe the right thing to do is
> to remove all sequence numbers, formatting, and merging in the transport
> layer for relayfs and simply let it run full-speed dumping data into
> per-cpu files. This gives the highest performance and moves the
> responsibility for any merging or other postprocessing to the
> application layer, where it can be done most efficiently. So some script
> (or program) will invoke systemtap, and on exit, take the binary data
> files and process them. Systemtap will not timestamp, sequence, or
> otherwise process the data. 
> 
How about adding an option to control this? 
e.g.
  "stap -b -K a.stp" will write to relayfs buffer directly, remove all 
sequence numbers and keep the pre-cpu files;
  "stap -b a.stp" will also write to relayfs buffer directly, but 
still keep current plain ascii sequencing scheme and merge pre-cpu files
to probe.out as it does now.

I think it can keep the convenience of current sequencing method 
in cases where ascii tracing is good enough, and also keep the
performance advantage of relayfs.