RE: FW: locking timeout error ?

RE: FW: locking timeout error ?

[Stone, Joshua I]

Frank Ch. Eigler wrote:
>> [...] Imagine the performance if the probe handles perhaps 5
>> different locks in its body, which is well within reason.
> 
> If the locking scheme is reorganized as outlined in #2060, by pulling
> all lock/unlock operations to the outermost level of a probe, then the
> overall (rather than per-lock) timeout could be easily bounded.  Plus
> each global variable would be locked at most once per probe handler
> run, rather than around each appearance in an expression.

I really like this idea, because it treats probes as truly atomic - if
you don't get all of the locks you need, don't do anything.
Implementing this will also require full function analysis to propogate
the locks back to the probe.

> Actually, even without that, we could accumulate locking iteration
> counts in a new context variable, and limit the cumulative (rather
> than individual) total to MAXTRYLOCK.

I think this would be a much more meaningful metric, and it should be an
easy thing to implement...


Josh

Re: FW: locking timeout error ?

[Frank Ch. Eigler]

joshua.i.stone wrote:

> > If the locking scheme is reorganized as outlined in #2060, by pulling
> > all lock/unlock operations to the outermost level of a probe [...]
> 
> I really like this idea, because it treats probes as truly atomic [...]

This is now done.

- FChE

Re: FW: locking timeout error ?

[Martin Hunt]

On Mon, 2006-01-16 at 15:50 -0500, Frank Ch. Eigler wrote:
> Hi -
> 
> > [...] Imagine the performance if the probe handles perhaps 5
> > different locks in its body, which is well within reason.
> 
> If the locking scheme is reorganized as outlined in #2060, by pulling
> all lock/unlock operations to the outermost level of a probe, then the
> overall (rather than per-lock) timeout could be easily bounded.  Plus
> each global variable would be locked at most once per probe handler
> run, rather than around each appearance in an expression.

If you reorganize the locks as proposed, you should be able to actually
reduce MAXTRYLOCK because the consequences of a lock timeout will simply
be an increment of the skipped probe counter instead of a fatal error.
Maps are not scalable anyway and pmaps will be unaffected, so I expect
performance will not suffer at all.

Martin

RE: FW: locking timeout error ?

[Stone, Joshua I]

James Dickens wrote:
> perhaps it would be better if systemtap took no locks, other than
> private ones related to systemtap data structures, [...]

AFAIK this is already the case - all of the locks we are talking about
here are 'private' systemtap locks.

Josh

Re: FW: locking timeout error ?

[Frank Ch. Eigler]

Hi -

On Mon, Jan 16, 2006 at 03:02:20PM -0600, James Dickens wrote:

> [...]  perhaps it would be better if systemtap took no locks, other
> than private ones related to systemtap data structures [...]

It is exactly these locks we are talking about.

- FChE

Re: FW: locking timeout error ?

[James Dickens]

On 1/16/06, Frank Ch. Eigler <fche@redhat.com> wrote:
> Hi -
>
> > [...] Imagine the performance if the probe handles perhaps 5
> > different locks in its body, which is well within reason.
>
> If the locking scheme is reorganized as outlined in #2060, by pulling
> all lock/unlock operations to the outermost level of a probe, then the
> overall (rather than per-lock) timeout could be easily bounded.  Plus
> each global variable would be locked at most once per probe handler
> run, rather than around each appearance in an expression.
>
> Actually, even without that, we could accumulate locking iteration
> counts in a new context variable, and limit the cumulative (rather
> than individual) total to MAXTRYLOCK.
>
>
> > Perhaps the better way to approach this is to estimate how long we might
> > expect the person holding the lock to take.  Do we have any
> > microbenchmarks on how long various operations take, like indexing a
> > map?
>
perhaps it would be better if systemtap took no locks, other than
private ones related to systemtap data structures, sure it will be
harder to write, but you can be sure that your script will work long
term. Not having to worry about some part of the kernel taking a lock
you depend on and holding it or worse yet, taking a lock and then have
a probe fire that wants to take the lock being held.

James Dickens
uadmin.blogspot.com



> Unfortunately, no.  Bug #1884 and #2060 could benefit from the
> development of a microbenchmark suite.
>
>
> - FChE
>

Re: FW: locking timeout error ?

[Frank Ch. Eigler]

Hi -

> [...] Imagine the performance if the probe handles perhaps 5
> different locks in its body, which is well within reason.

If the locking scheme is reorganized as outlined in #2060, by pulling
all lock/unlock operations to the outermost level of a probe, then the
overall (rather than per-lock) timeout could be easily bounded.  Plus
each global variable would be locked at most once per probe handler
run, rather than around each appearance in an expression.

Actually, even without that, we could accumulate locking iteration
counts in a new context variable, and limit the cumulative (rather
than individual) total to MAXTRYLOCK.


> Perhaps the better way to approach this is to estimate how long we might
> expect the person holding the lock to take.  Do we have any
> microbenchmarks on how long various operations take, like indexing a
> map?

Unfortunately, no.  Bug #1884 and #2060 could benefit from the
development of a microbenchmark suite.


- FChE

RE: FW: locking timeout error ?

[Stone, Joshua I]

Frank Ch. Eigler wrote:
>> Actually, I meant to ask you about this.  Is it safe to be waiting so
>> long for a lock if the probe is in interrupt context?  If I
>> understand it, the lock will wait about MAXTRYLOCK*TRYLOCKDELAY
>> nanoseconds = 100 microseconds.  That seems like an awfully long
>> time to poll the lock. 
> 
> The current figures are not meant as definitive.  It may be wise to
> wait less if in_interrupt().  Do you have alternate numbers to
> suggest? 

I'll leave it to the experts to decide what is appropriate.  But even
when not in_interrupt, I think this may be too long.  Do we have
estimates for what a "reasonable" runtime for a probe?  We can assume
that in the worst case, the lock will almost timeout everytime, but
still succeed.  Imagine the performance if the probe handles perhaps 5
different locks in its body, which is well within reason.

Perhaps the better way to approach this is to estimate how long we might
expect the person holding the lock to take.  Do we have any
microbenchmarks on how long various operations take, like indexing a
map?

Josh

Re: FW: locking timeout error ?

[Frank Ch. Eigler]

Hi -

joshua.i.stone wrote:

> Actually, I meant to ask you about this.  Is it safe to be waiting so
> long for a lock if the probe is in interrupt context?  If I understand
> it, the lock will wait about MAXTRYLOCK*TRYLOCKDELAY nanoseconds = 100
> microseconds.  That seems like an awfully long time to poll the lock.

The current figures are not meant as definitive.  It may be wise to
wait less if in_interrupt().  Do you have alternate numbers to suggest?


- FChE