[Bug translator/2293] New: confirm preemption/interrupt blocking in systemtap probes

[Bug translator/2293] New: confirm preemption/interrupt blocking in systemtap probes

[fche at redhat dot com]

For each probe type supported in tapsets.cxx, confirm that the probe handlers
are run in an atomic manner: preemption + interrupts disabled.

This could tie in with bug #1884, by detecting whether we are already in an
interrupt context, and if so, reducing the MAXACTION budget for this probe hit.

-- 
           Summary: confirm preemption/interrupt blocking in systemtap
                    probes
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: translator
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: fche at redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2006-02-22 15:05 -------
Added a local_irq_save/restore into every probe handler prologue/epilogue.  The
runtime should no longer need to do this sort of thing for functions only called
from within probe handlers.


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hunt at redhat dot com
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[hunt at redhat dot com]

------- Additional Comments From hunt at redhat dot com  2006-02-22 20:07 -------
(In reply to comment #1)
>The
> runtime should no longer need to do this sort of thing for functions only called
> from within probe handlers.

The runtime functions callable by the translator never disable interrupts. 



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2006-02-23 20:42 -------
Other runtime protective measure still exist though: get_cpu/put_cpu can simply
be smp_processor_id.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[hunt at redhat dot com]

------- Additional Comments From hunt at redhat dot com  2006-02-27 17:32 -------
(In reply to comment #3)
> Other runtime protective measure still exist though: get_cpu/put_cpu can simply
> be smp_processor_id.

Yeah, I forgot about those.

I've been trying to solve a  problem created by this change. The problem is with
scripts that collect a large amount of data then dump it all at probe end. You
have now disabled interrupts, so the data fills the buffers but stpd cannot
empty them. So we have effectively limited systemtap output to whatever space is
remaining in  buffer when "probe end" is hit.  Which might not be much.

That's fine for little test scripts, but a real solution is needed. Ideas?





-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

disabled interrupts (Was bug #2293

[Frank Ch. Eigler]

hunt wrote:

> [...] You have now disabled interrupts, so the data [produced by end
> probes] fills the buffers but stpd cannot empty them.

How could stpd have emptied them before, in a reliable way?  During an
"end" probe, or when?  stpd can't run until the next timeslice; we
never cooperatively schedule out of a probe.

> So we have effectively limited systemtap output to whatever space is
> remaining in buffer when "probe end" is hit.  [...]

A couple of possible solutions: Use bigger buffers; put a buffer aside
for end probes; get stpd to empty all the buffers early during the
STP_EXIT exchange and thus before "end" probes run.

- FChE

Re: disabled interrupts (Was bug #2293

[Martin Hunt]

On Mon, 2006-02-27 at 13:56 -0500, Frank Ch. Eigler wrote:
> hunt wrote:
> 
> > [...] You have now disabled interrupts, so the data [produced by end
> > probes] fills the buffers but stpd cannot empty them.
> 
> How could stpd have emptied them before, in a reliable way?  During an
> "end" probe, or when?  stpd can't run until the next timeslice; we
> never cooperatively schedule out of a probe.

The runtime calls probe_end(), and it used to know it was safe to sleep
while in probe_end(). So if it attempted to send data and no buffers
were available, it would sleep a bit (allowing stpd to precess the data)
and retry (up to a limit), thus ensuring 100% of the data got sent.

Other solutions require fixed limits on the amount of data that can be
sent during probe end.

Re: disabled interrupts (Was bug #2293

[Frank Ch. Eigler]

hunt wrote:

> The runtime calls probe_end(), and it used to know it was safe to sleep
> while in probe_end(). So if it attempted to send data and no buffers
> were available, it would sleep a bit (allowing stpd to precess the data)
> and retry (up to a limit), thus ensuring 100% of the data got sent.

You must be referring to the msleep() loop in _stp_transport_send.
Isn't that conditional on the "exit is pending" flag, which is set
considerably earlier than the beginning of first "end" probe handler?
Also, if that 250ms total sleep elapses without stpd waking up and
taking action, data is still lost, so this isn't a 100% solution
either.  Let's accept the best one of the 99% solutions.

- FChE

Re: disabled interrupts (Was bug #2293

[Martin Hunt]

On Mon, 2006-02-27 at 15:01 -0500, Frank Ch. Eigler wrote:
> hunt wrote:
> 
> > The runtime calls probe_end(), and it used to know it was safe to sleep
> > while in probe_end(). So if it attempted to send data and no buffers
> > were available, it would sleep a bit (allowing stpd to precess the data)
> > and retry (up to a limit), thus ensuring 100% of the data got sent.
> 
> You must be referring to the msleep() loop in _stp_transport_send.
> Isn't that conditional on the "exit is pending" flag, which is set
> considerably earlier than the beginning of first "end" probe handler?

You are thinking of _stp_exit. _stp_exit_called is the flag we are
interested in.

int _stp_transport_write (void *data, int len)  
{
  /* when _stp_exit_called is set, we are in probe_exit() */
  /* and we can sleep */
  if (_stp_exit_called)
      return _stp_transport_send (STP_REALTIME_DATA, data, len);
  return _stp_write(STP_REALTIME_DATA, data, len);
}

From _stp_cleanup_and_exit()

	if (!_stp_exit_called) {
		_stp_exit_called = 1;
		probe_exit();

I'm going to move the _stp_exit_called to the other side of probe_exit()
for now. And I'm going to put a loop to wait on empty buffers before
probe_exit().  That's about all I can do.  


Martin

[Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[hunt at redhat dot com]

------- Additional Comments From hunt at redhat dot com  2006-03-29 07:06 -------
Still trying to figure out the transport problem. It appears to not be solvable;
if you disable interrupts, stpd will not be processing any output and output
from "probe end" will get truncated if the user tries to write too much.

However, simply removing the local_irq_save() for "probe begin" and "probe end"
would still leave us vulnerable to the possibility of preemption causing the cpu
to change during the probe execution. One possibility would be to replace all
smp_processor_id() calls with _stp_processor_id() which would do

if (STAP_SESSION_STARTING or STAP_SESSION_STOPPING)
    return 0
else
    return smp_processor_id()




-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2006-03-29 11:54 -------
I am skeptical that games with the processor ID are appropriate, or that
allowing end probes to block is appropriate.

I recommend opening a new bug against the runtime, addressing specifically the
issue of I/O buffering near the time of shutdown.  I recall suggesting looking
into whether stpd and the kernel-side runtime message handler can work together
to drain the buffers before starting the module_exit process, to provide the
maximum static space to the end probes.  (That space amount would
uncoincidentally match the command line option "-s NUM" to the initial
compilation stage, and thus make some intuitive sense to the user.)  Did you try
that?

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[Martin Hunt]

On Wed, 2006-03-29 at 11:54 +0000, fche at redhat dot com wrote:

> I recommend opening a new bug against the runtime, addressing specifically the
> issue of I/O buffering near the time of shutdown.  I recall suggesting looking
> into whether stpd and the kernel-side runtime message handler can work together
> to drain the buffers before starting the module_exit process, to provide the
> maximum static space to the end probes.  (That space amount would
> uncoincidentally match the command line option "-s NUM" to the initial
> compilation stage, and thus make some intuitive sense to the user.)  Did you try
> that?

I think I originally suggested it, and I have thought further about it.
I hoped to find a better solution than imposing another limit users have
to compute. For collecting large amounts of data, MAXMAPENTRIES needs
raised and then you have to calculate how much space that data will take
up when "printed" into the output buffers.  Another problem is that for
relayfs the buffer is divided into per-cpu sub-buffers. So the maximum
data that can be sent is NUM/cpus. 

Martin

[Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[hunt at redhat dot com]

------- Additional Comments From hunt at redhat dot com  2006-03-29 18:05 -------
Subject: Re:  confirm preemption/interrupt blocking in
	systemtap probes

On Wed, 2006-03-29 at 11:54 +0000, fche at redhat dot com wrote:

> I recommend opening a new bug against the runtime, addressing specifically the
> issue of I/O buffering near the time of shutdown.  I recall suggesting looking
> into whether stpd and the kernel-side runtime message handler can work together
> to drain the buffers before starting the module_exit process, to provide the
> maximum static space to the end probes.  (That space amount would
> uncoincidentally match the command line option "-s NUM" to the initial
> compilation stage, and thus make some intuitive sense to the user.)  Did you try
> that?

I think I originally suggested it, and I have thought further about it.
I hoped to find a better solution than imposing another limit users have
to compute. For collecting large amounts of data, MAXMAPENTRIES needs
raised and then you have to calculate how much space that data will take
up when "printed" into the output buffers.  Another problem is that for
relayfs the buffer is divided into per-cpu sub-buffers. So the maximum
data that can be sent is NUM/cpus. 

Martin



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[Frank Ch. Eigler]

hunt wrote:

> [...]  For collecting large amounts of data, MAXMAPENTRIES needs
> raised and then you have to calculate how much space that data will
> take up when "printed" into the output buffers.

I suppose MAXACTION too, in order just to run enough printf's.

But really, can you imagine a reasonable scenario where an "end" probe
should produce megabytes of traffic?

> Another problem is that for relayfs the buffer is divided into
> per-cpu sub-buffers. [...]

Alternately, the relayfs buffers could each be of the given size.


- FChE

[Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2006-05-01 14:20 -------
Please open a new bug for more work along the lines of
<http://sourceware.org/ml/systemtap/2006-q1/msg00653.html>


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |FIXED


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[Vara Prasad]

Martin Hunt wrote:

>On Wed, 2006-03-29 at 11:54 +0000, fche at redhat dot com wrote:
>
>  
>
>>I recommend opening a new bug against the runtime, addressing specifically the
>>issue of I/O buffering near the time of shutdown.  I recall suggesting looking
>>into whether stpd and the kernel-side runtime message handler can work together
>>to drain the buffers before starting the module_exit process, to provide the
>>maximum static space to the end probes.  (That space amount would
>>uncoincidentally match the command line option "-s NUM" to the initial
>>compilation stage, and thus make some intuitive sense to the user.)  Did you try
>>that?
>>    
>>
>
>I think I originally suggested it, and I have thought further about it.
>I hoped to find a better solution than imposing another limit users have
>to compute. For collecting large amounts of data, MAXMAPENTRIES needs
>raised and then you have to calculate how much space that data will take
>up when "printed" into the output buffers.  Another problem is that for
>relayfs the buffer is divided into per-cpu sub-buffers. So the maximum
>data that can be sent is NUM/cpus. 
>
>Martin
>
>  
>
There is a generic problem that we have to solve in SystemTap to support 
long running or large number of probes. The common problem with these 
scenarios is, they generate lot more data than the maps can hold. There 
are two solutions i can think of to help address this area
1) We should have capability to say truncate the map by leaving only the 
top "n" entries based on the key. If one is looking to get general 
trends top few is more than enough hence this solution could be useful.
2) Second solution is able to periodically dump the maps to userspace 
and then stpd during the post processing can reconstruct the full maps 
from the dumps. I have not looked at if there are any maps that have 
some specific mathematical properties that doesn't lend into post 
aggregation, we have to look into that aspect.

Coming to relayfs i believe stpd has an option to specify the overall 
buffer size but not the no. of sub buffers and size of each. As Frank 
mentioned I think it may be a good idea able to specify the no. of sub 
buffers as well along with the overall buffer size.  I think script 
writers are likely to have a better idea of how much data their script 
collects rather than the executor of the script, that makes me think we 
should also have an option for the script writer to specify the the type 
of transport used procfs or relayfs and if it is relayfs they should 
also have an option to specify the no. of sub buffers and size of the 
total buffers. If the size is specified on the command line and as well 
in the script, i think we should use the max of the two.

bye,
Vara Prasad

[Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[prasadav at us dot ibm dot com]

------- Additional Comments From prasadav at us dot ibm dot com  2006-05-05 16:09 -------
Subject: Re:  confirm preemption/interrupt blocking in
 	systemtap probes

Martin Hunt wrote:

>On Wed, 2006-03-29 at 11:54 +0000, fche at redhat dot com wrote:
>
>  
>
>>I recommend opening a new bug against the runtime, addressing specifically the
>>issue of I/O buffering near the time of shutdown.  I recall suggesting looking
>>into whether stpd and the kernel-side runtime message handler can work together
>>to drain the buffers before starting the module_exit process, to provide the
>>maximum static space to the end probes.  (That space amount would
>>uncoincidentally match the command line option "-s NUM" to the initial
>>compilation stage, and thus make some intuitive sense to the user.)  Did you try
>>that?
>>    
>>
>
>I think I originally suggested it, and I have thought further about it.
>I hoped to find a better solution than imposing another limit users have
>to compute. For collecting large amounts of data, MAXMAPENTRIES needs
>raised and then you have to calculate how much space that data will take
>up when "printed" into the output buffers.  Another problem is that for
>relayfs the buffer is divided into per-cpu sub-buffers. So the maximum
>data that can be sent is NUM/cpus. 
>
>Martin
>
>  
>
There is a generic problem that we have to solve in SystemTap to support 
long running or large number of probes. The common problem with these 
scenarios is, they generate lot more data than the maps can hold. There 
are two solutions i can think of to help address this area
1) We should have capability to say truncate the map by leaving only the 
top "n" entries based on the key. If one is looking to get general 
trends top few is more than enough hence this solution could be useful.
2) Second solution is able to periodically dump the maps to userspace 
and then stpd during the post processing can reconstruct the full maps 
from the dumps. I have not looked at if there are any maps that have 
some specific mathematical properties that doesn't lend into post 
aggregation, we have to look into that aspect.

Coming to relayfs i believe stpd has an option to specify the overall 
buffer size but not the no. of sub buffers and size of each. As Frank 
mentioned I think it may be a good idea able to specify the no. of sub 
buffers as well along with the overall buffer size.  I think script 
writers are likely to have a better idea of how much data their script 
collects rather than the executor of the script, that makes me think we 
should also have an option for the script writer to specify the the type 
of transport used procfs or relayfs and if it is relayfs they should 
also have an option to specify the no. of sub buffers and size of the 
total buffers. If the size is specified on the command line and as well 
in the script, i think we should use the max of the two.

bye,
Vara Prasad



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2293

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Bug translator/2293] confirm preemption/interrupt blocking in systemtap probes

[Frank Ch. Eigler]

varap wrote:

> There is a generic problem that we have to solve in SystemTap to
> support long running or large number of probes. The common problem
> with these scenarios is, they generate lot more data than the maps
> can hold.

Let's turn the question around for a moment: how big arrays can we
safely support?  What's the largest MAXMAPENTRIES people have tried?

> 1) We should have capability to say truncate the map by leaving only
> the top "n" entries based on the key. If one is looking to get
> general trends top few is more than enough hence this solution could
> be useful.

That's an interesting idea.  Such a partial-delete operation would
have to be used carefully, since e.g.  it destroys information that
could raise a low-ranked value from rising into the preserved few.  In
other words, truncating periodically is not equivalent to truncating
once at the end.


> 2) Second solution is able to periodically dump the maps to
> userspace and then stpd during the post processing can reconstruct
> the full maps from the dumps. [...]

For "write-only" maps such as dtrace aggregates, this sort of thing
could work.  In systemtap, all arrays are in general read/write, so
this would apply only in restricted cases and/or with moving some
processing over along with the data.

Ordinary trace data could of course be post-processed by user-space
scripts.


- FChE