more transport questions

more transport questions

[Frank Ch. Eigler]

Hi -

I'm just beginning to look deeper into this portion of the runtime
code, as I'm planning to pull in the core parts of stpd into the
translator/driver.

I wonder: is there some reason (ideally: data) to believe that the
intricate multi-threaded relayfs copying and subsequent cpu_merging is
necessary?  For instance, how bad would a simpler, single-threaded
multi-fd poll() work in the relayfs reading loop?  Is merging
on-the-fly impractical?  On the transmission kernel-probe side, would
a relay_flush() after every probe handler be impractical?


- FChE

Re: more transport questions

[Tom Zanussi]

Frank Ch. Eigler writes:
 > Hi -
 > 
 > I'm just beginning to look deeper into this portion of the runtime
 > code, as I'm planning to pull in the core parts of stpd into the
 > translator/driver.
 > 
 > I wonder: is there some reason (ideally: data) to believe that the
 > intricate multi-threaded relayfs copying and subsequent cpu_merging is
 > necessary?  For instance, how bad would a simpler, single-threaded
 > multi-fd poll() work in the relayfs reading loop?  Is merging

It probably doesn't need to be done using multiple threads - it's
based on the relay-apps code that does it that way, and I didn't see
any reason to change it.  BTW, the per-cpu threads are used only for
writing the per-cpu data as it comes in - the merging is done later in
the main thread.

 > on-the-fly impractical?  On the transmission kernel-probe side, would
 > a relay_flush() after every probe handler be impractical?

Merging on-the-fly would be practical if you flushed every per-cpu
channel at the same time i.e. forced each per-cpu buffer to finish the
current sub-buffer so it could be read, but that would defeat the
purpose of using relayfs for high-speed logging.  relay_flush() also
isn't safe to call unless you know there's nothing actively logging
into a buffer.

Martin did create a more efficient sort/merge of the data, but one
question that came up then was whether the per-cpu data should be
merged or sorted at all - it makes sense for relatively small amounts,
but if you're talking about huge quantities of data, say many
gigabytes, it seems impractical to merge it all into a single file,
and why is it necessary?  Wouldn't any postprocessing tool that wanted
to make sense of that much data also need to be able to deal with it
in a more efficient way than by reading it linearly from a single
file?  That would argue for some sort of indexing scheme where for
instance data is written in constant-sized blocks and each block would
contain start/end event identifiers identifying the events contained
within that block.  In that case it wouldn't really matter whether the
data is in a single file or in multiple files (probably better to keep
it in multiple files actually).

Tom

Re: more transport questions

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 1343 bytes --]

Hi -


On Wed, Jul 06, 2005 at 02:55:38PM -0500, Tom Zanussi wrote:
> [...]
>  > on-the-fly impractical?  On the transmission kernel-probe side, would
>  > a relay_flush() after every probe handler be impractical?
>
> Merging on-the-fly would be practical if you flushed every per-cpu
> channel at the same time i.e. forced each per-cpu buffer to finish
> the current sub-buffer so it could be read but that would defeat the
> purpose of using relayfs for high-speed logging.  [...]

OK, I didn't really mean relay_flush but rather stp_print_flush.
Besides, with the presence of a globally unique message serial number,
it would be straightforward to merge on the fly, even if the
processors generate trace messages (apprx == run probe functions) at
very different intervals/rates.


> Martin did create a more efficient sort/merge of the data, but one
> question that came up then was whether the per-cpu data should be
> merged or sorted at all [...]

That's a reasonable question.  In the future, when we get some
experience with such data firehoses, we consider providing more
options.


This reminds me - has anyone made any progress on carrying out some of
the testing outlined by Brad's plan of two months ago?  In particular,
do there now exist kprobes stress tests (numerous / frequent /
densely-packed / long-lived probes)?


- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: more transport questions

[Martin Hunt]

On Thu, 2005-07-07 at 15:53 -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> 
> On Wed, Jul 06, 2005 at 02:55:38PM -0500, Tom Zanussi wrote:
> > [...]
> >  > on-the-fly impractical?  On the transmission kernel-probe side, would
> >  > a relay_flush() after every probe handler be impractical?
> >
> > Merging on-the-fly would be practical if you flushed every per-cpu
> > channel at the same time i.e. forced each per-cpu buffer to finish
> > the current sub-buffer so it could be read but that would defeat the
> > purpose of using relayfs for high-speed logging.  [...]
> 
> OK, I didn't really mean relay_flush but rather stp_print_flush.
> Besides, with the presence of a globally unique message serial number,
> it would be straightforward to merge on the fly, even if the
> processors generate trace messages (apprx == run probe functions) at
> very different intervals/rates.

Currently, we use _stp_print_flush() to tell the runtime to put a
timestamp in front of accumulated data and send it to relayfs or
netlink.  I have wanted for a long time to make this more efficient. Tom
and I discussed it last month and I just haven't finished testing
(waiting for better netlink and relayfs tuning info from Tom).

What I would like to do is have each probed function that outputs
anything do _stp_entry() at the start.  That puts a sequence number in
the output buffer. No flush is done until a buffer fills.  We write data
sequentially, allowing us to possibly remove the internal buffer in
runtime and write directly to the relayfs buffers.

Does this make sense and is it too dosruptive to make such a change at
this date?

Martin

Re: more transport questions

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

Hi -

hunt wrote:

> [...]  What I would like to do is have each probed function that
> outputs anything do _stp_entry() at the start.  [...]  Does this
> make sense and is it too dosruptive to make such a change at this
> date?

Putting in calls to arbitrary functions just before and just after a
probe handler is easy.

It would be most straightforward for the translator to call an entry &
exit function unconditionally, that is regardless of whether a probe
handler contains any actual trace/print calls.  In this case, the
runtime code may have to make some extra effort to optimize the "null"
case where a probe handler has no trace/print calls.

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]