Embedded C code - functions that might sleep

Embedded C code - functions that might sleep

[Turgis, Frederic]

Hi,

systemtap doc states: "Note that all SystemTap functions and probes run with interrupts disabled, thus you cannot call functions that might sleep within the embedded C."

This would explain issues we get with below probe on ARM platform (OMAP):
- open a GP timer through function that calls mutex_lock() in "begin" probe -> it gives "BUG: sleeping function called from invalid context".

Do you confirm interpretation and what can we do ? This would not be used in production so use it with a risk of lock ?


Note that I think I have similar issue with "iounmap" embedded C in "end" probe but I must re-setup a platform to double check error I recently got.


Regards
Fred

Frederic Turgis

OMAP Platform Business Unit - OMAP System Engineering - Platform Enablement - System Multimedia


Texas Instruments France SA, 821 Avenue Jack Kilby, 06270 Villeneuve Loubet. 036 420 040 R.C.S Antibes. Capital de EUR 753.920

Re: Embedded C code - functions that might sleep

[Mark Wielaard]

Hi Frederic,

On Wed, 2012-02-01 at 12:27 +0000, Turgis, Frederic wrote:
> systemtap doc states: "Note that all SystemTap functions and probes run with interrupts disabled, thus you cannot call functions that might sleep within the embedded C."
> 
> This would explain issues we get with below probe on ARM platform (OMAP):
> - open a GP timer through function that calls mutex_lock() in "begin" probe -> it gives "BUG: sleeping function called from invalid context".

In general probes can occur anywhere, so might be triggered in interrupt
context. You would see the above message also if you are holding any
locks (sleeping while holding locks is bad, since there is no guarantee
the lock will ever be released again). If you are running a kernel with
lockdep enabled it should tell you exactly why it is yelling at you.
(Note that only very recently, like 30 minutes ago, we fixed some
lockdep issue with user probes, see the thread on the mailinglist).

Cheers,

Mark

RE: Embedded C code - functions that might sleep

[Turgis, Frederic]

Thanks for the explanation and thread is giving good light on this.

My point is that systemtap design and doc seem to imply I can't call some functions in embedded C because of relevant systemtap design choices. In our script, we are in fact users, not developers of this called function. My goal is to know if there is no easy trick or "specific" mode of systemtap not mentioned in doc where I could still call them or if these functions (like opening OMAP GP timer) are definitely forbidden. Or if this is bad using them, but this is just profiling script on a dev mobile phone so who cares about hanging from time to time ? ;-)

Eventually, we would need to rewrite the function or mimic its functionality in embedded C, respecting the specific constraints. This can be far more work based on our expertise/time requirements so we need to check benefits.


Regards
Fred

Frederic Turgis

OMAP Platform Business Unit - OMAP System Engineering - Platform Enablement - System Multimedia



Texas Instruments France SA, 821 Avenue Jack Kilby, 06270 Villeneuve Loubet. 036 420 040 R.C.S Antibes. Capital de EUR 753.920

-----Original Message-----
From: Mark Wielaard [mailto:mjw@redhat.com]
Sent: Wednesday, February 01, 2012 2:34 PM
To: Turgis, Frederic
Cc: systemtap@sourceware.org
Subject: Re: Embedded C code - functions that might sleep

Hi Frederic,

On Wed, 2012-02-01 at 12:27 +0000, Turgis, Frederic wrote:
> systemtap doc states: "Note that all SystemTap functions and probes run with interrupts disabled, thus you cannot call functions that might sleep within the embedded C."
>
> This would explain issues we get with below probe on ARM platform (OMAP):
> - open a GP timer through function that calls mutex_lock() in "begin" probe -> it gives "BUG: sleeping function called from invalid context".

In general probes can occur anywhere, so might be triggered in interrupt context. You would see the above message also if you are holding any locks (sleeping while holding locks is bad, since there is no guarantee the lock will ever be released again). If you are running a kernel with lockdep enabled it should tell you exactly why it is yelling at you.
(Note that only very recently, like 30 minutes ago, we fixed some lockdep issue with user probes, see the thread on the mailinglist).

Cheers,

Mark

RE: Embedded C code - functions that might sleep

[Mark Wielaard]

On Wed, 2012-02-01 at 14:47 +0000, Turgis, Frederic wrote:
> Thanks for the explanation and thread is giving good light on this.
> 
> My point is that systemtap design and doc seem to imply I can't call
> some functions in embedded C because of relevant systemtap design
> choices. In our script, we are in fact users, not developers of this
> called function. My goal is to know if there is no easy trick or
> "specific" mode of systemtap not mentioned in doc where I could still
> call them or if these functions (like opening OMAP GP timer) are
> definitely forbidden. Or if this is bad using them, but this is just
> profiling script on a dev mobile phone so who cares about hanging from
> time to time ? ;-)

I am afraid that in general you just cannot sleep in a probe handler. If
you still try "bad things" might happen. Like hanging or even crashing.
In some cases there are variants that also work in atomic context (for
example you can do some allocations with GFP_ATOMIC) but I don't know if
the OMAP GP timer provides such functionality/mode.

That said, if you are using guru mode already you are kind of on your
own anyway. In a lot of cases you might just get away with a little
badness. Just hope the function doesn't really sleep or schedule the
current task away...

Cheers,

Mark

RE: Embedded C code - functions that might sleep

[Turgis, Frederic]

Hi,

Well, "little badness" could be fun !

Within the team, I got interesting question if interrupt disabling could be relaxed for begin/end probes (through compilation flag), which would be executed in safer contexts as not being kprobes. That would be sufficient for our needs.
In the code, I see that begin probe is called when module receives STP_START command and end probe on STP_EXIT or module unload.

Still "enter_be_probe()" in generated code relies on preempt_disable()/enable() or local_irq_save()/restore(). Is there some other constraint to take into account ?

Regards
Fred

Frederic Turgis
OMAP Platform Business Unit - OMAP System Engineering - Platform Enablement - System Multimedia



Texas Instruments France SA, 821 Avenue Jack Kilby, 06270 Villeneuve Loubet. 036 420 040 R.C.S Antibes. Capital de EUR 753.920

-----Original Message-----
From: Mark Wielaard [mailto:mjw@redhat.com]
Sent: Wednesday, February 01, 2012 5:07 PM
To: Turgis, Frederic
Cc: systemtap@sourceware.org
Subject: RE: Embedded C code - functions that might sleep

On Wed, 2012-02-01 at 14:47 +0000, Turgis, Frederic wrote:
> Thanks for the explanation and thread is giving good light on this.
>
> My point is that systemtap design and doc seem to imply I can't call
> some functions in embedded C because of relevant systemtap design
> choices. In our script, we are in fact users, not developers of this
> called function. My goal is to know if there is no easy trick or
> "specific" mode of systemtap not mentioned in doc where I could still
> call them or if these functions (like opening OMAP GP timer) are
> definitely forbidden. Or if this is bad using them, but this is just
> profiling script on a dev mobile phone so who cares about hanging from
> time to time ? ;-)

I am afraid that in general you just cannot sleep in a probe handler. If you still try "bad things" might happen. Like hanging or even crashing.
In some cases there are variants that also work in atomic context (for example you can do some allocations with GFP_ATOMIC) but I don't know if the OMAP GP timer provides such functionality/mode.

That said, if you are using guru mode already you are kind of on your own anyway. In a lot of cases you might just get away with a little badness. Just hope the function doesn't really sleep or schedule the current task away...

Cheers,

Mark

Re: Embedded C code - functions that might sleep

[Frank Ch. Eigler]

"Turgis, Frederic" <f-turgis@ti.com> writes:

> My point is that systemtap design and doc seem to imply I can't call
> some functions in embedded C because of relevant systemtap design
> choices. [...]

Not exactly.  Generally, the conditions are imposed upon us through
the kernel callback mechanisms used by the respective probes.  "probe
begin" has no material constraints (or should not; PR13641 indicates
something unintentionally is impeding this), so from there you can
generally do whatever you need to.  From kprobes, or tracepoints, the
kernel context is much more restricted and we must do nothing more
than atomic type operations.

- FChE

RE: Embedded C code - functions that might sleep

[Turgis, Frederic]

> Not exactly.  Generally, the conditions are imposed upon us through the kernel callback mechanisms used > by the respective probes.  "probe begin" has no material constraints (or should not; PR13641 indicates

I didn't mean to say bad things about systemtap "design choices" ;-) "relevant" was added to say there are good reasons to do that, i.e. respect constraints from the rest of the world. But well, I agree this is not really a choice, so let's talk about design constraints !

Texas Instruments France SA, 821 Avenue Jack Kilby, 06270 Villeneuve Loubet. 036 420 040 R.C.S Antibes. Capital de EUR 753.920