context[2] stuck: (null)

context[2] stuck: (null)

[Arkady]

Hi,

I am getting context[2] stuck: (null) error. The cause of error is
likely the "unmanaged" code I have added to the driver. Specifically I
have a shared memory (mmap) in the driver. The failure happens
randomly every 50-200 module restarts The failure happens only on the
multicore CPUs, or happens often enough to be caught.

I tried to force the the wait function with
STAP_OVERRIDE_STUCK_CONTEXT - kernel panics in one of the (probably
random) probes.

While debugging the issue I  patched the SystemTap source code - added
an argument to the _stp_runtime_entryfn_get_context(int) like in this
commit https://github.com/larytet/SystemTap/commit/61a284732893fa6f201e07f9f12f5e1820e7c26f
In the function _stp_runtime_context_wait() I print the line in the
source code which called the _stp_runtime_entryfn_get_context()

The "bad" context is enter_be_probe(). I checked the source code of
enter_be_probe() and there is not much there.

I struggle with the problem for some time and I will greatly
appreciate any tip.

Thank you, Arkady.

Re: context[2] stuck: (null)

[Arkady]

Update. The failure happens consistently in the same context

"context[1] stuck: (null), line_get=36848, line_put=36914
last_err=(null) last_stmt=identifier 'probe_begin'"
where line_get and line_put are lines in the enter_be_probe()

36843     #endif
36844     goto probe_epilogue;
36845   }
36846   if (atomic_read (session_state()) != stp->state)
36847     goto probe_epilogue;
36848   c = _stp_runtime_entryfn_get_context(__LINE__);
36849   if (!c) {
36850     #if !INTERRUPTIBLE
36851     atomic_inc (skipped_count());
36852     #endif
36853     #ifdef STP_TIMING
36854     atomic_inc (skipped_count_reentrant());
36855     #endif
36856     goto probe_epilogue;
36857   }
..................
36907     }
36908   }
36909 probe_epilogue:
36910   if (unlikely (atomic_read (skipped_count()) > MAXSKIPPED)) {
36911     if (unlikely (pseudo_atomic_cmpxchg(session_state(),
STAP_SESSION_RUNNING, STAP_SESSION_ERROR) == STAP_SESSION_RUNNING))
36912     _stp_error ("Skipped too many probes, check MAXSKIPPED or
try again with stap -t for more details.");
36913   }
36914   _stp_runtime_entryfn_put_context(c, __LINE__);
36915   #if !INTERRUPTIBLE
36916   local_irq_restore (flags);
36917   #endif
36918   #endif // STP_ALIBI

On Mon, Jul 10, 2017 at 6:16 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Hi,
>
> I am getting context[2] stuck: (null) error. The cause of error is
> likely the "unmanaged" code I have added to the driver. Specifically I
> have a shared memory (mmap) in the driver. The failure happens
> randomly every 50-200 module restarts The failure happens only on the
> multicore CPUs, or happens often enough to be caught.
>
> I tried to force the the wait function with
> STAP_OVERRIDE_STUCK_CONTEXT - kernel panics in one of the (probably
> random) probes.
>
> While debugging the issue I  patched the SystemTap source code - added
> an argument to the _stp_runtime_entryfn_get_context(int) like in this
> commit https://github.com/larytet/SystemTap/commit/61a284732893fa6f201e07f9f12f5e1820e7c26f
> In the function _stp_runtime_context_wait() I print the line in the
> source code which called the _stp_runtime_entryfn_get_context()
>
> The "bad" context is enter_be_probe(). I checked the source code of
> enter_be_probe() and there is not much there.
>
> I struggle with the problem for some time and I will greatly
> appreciate any tip.
>
> Thank you, Arkady.

Re: context[2] stuck: (null)

[Arkady]

Update. Some of the system calls I am doing in the begin probe are
blocking. I understand that it will break things on multicore systems.
Am I right?

On Tue, Jul 11, 2017 at 9:24 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Update. The failure happens consistently in the same context
>
> "context[1] stuck: (null), line_get=36848, line_put=36914
> last_err=(null) last_stmt=identifier 'probe_begin'"
> where line_get and line_put are lines in the enter_be_probe()
>
> 36843     #endif
> 36844     goto probe_epilogue;
> 36845   }
> 36846   if (atomic_read (session_state()) != stp->state)
> 36847     goto probe_epilogue;
> 36848   c = _stp_runtime_entryfn_get_context(__LINE__);
> 36849   if (!c) {
> 36850     #if !INTERRUPTIBLE
> 36851     atomic_inc (skipped_count());
> 36852     #endif
> 36853     #ifdef STP_TIMING
> 36854     atomic_inc (skipped_count_reentrant());
> 36855     #endif
> 36856     goto probe_epilogue;
> 36857   }
> ..................
> 36907     }
> 36908   }
> 36909 probe_epilogue:
> 36910   if (unlikely (atomic_read (skipped_count()) > MAXSKIPPED)) {
> 36911     if (unlikely (pseudo_atomic_cmpxchg(session_state(),
> STAP_SESSION_RUNNING, STAP_SESSION_ERROR) == STAP_SESSION_RUNNING))
> 36912     _stp_error ("Skipped too many probes, check MAXSKIPPED or
> try again with stap -t for more details.");
> 36913   }
> 36914   _stp_runtime_entryfn_put_context(c, __LINE__);
> 36915   #if !INTERRUPTIBLE
> 36916   local_irq_restore (flags);
> 36917   #endif
> 36918   #endif // STP_ALIBI
>
> On Mon, Jul 10, 2017 at 6:16 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> Hi,
>>
>> I am getting context[2] stuck: (null) error. The cause of error is
>> likely the "unmanaged" code I have added to the driver. Specifically I
>> have a shared memory (mmap) in the driver. The failure happens
>> randomly every 50-200 module restarts The failure happens only on the
>> multicore CPUs, or happens often enough to be caught.
>>
>> I tried to force the the wait function with
>> STAP_OVERRIDE_STUCK_CONTEXT - kernel panics in one of the (probably
>> random) probes.
>>
>> While debugging the issue I  patched the SystemTap source code - added
>> an argument to the _stp_runtime_entryfn_get_context(int) like in this
>> commit https://github.com/larytet/SystemTap/commit/61a284732893fa6f201e07f9f12f5e1820e7c26f
>> In the function _stp_runtime_context_wait() I print the line in the
>> source code which called the _stp_runtime_entryfn_get_context()
>>
>> The "bad" context is enter_be_probe(). I checked the source code of
>> enter_be_probe() and there is not much there.
>>
>> I struggle with the problem for some time and I will greatly
>> appreciate any tip.
>>
>> Thank you, Arkady.

Re: context[2] stuck: (null)

[Arkady]

Is there a chance to get a user defined hook in the context of
systemtap_module_init()

Something like this, for example:

diff --git a/translate.cxx b/translate.cxx
index 0b9fc45..ba7088b 100644
--- a/translate.cxx
+++ b/translate.cxx
@@ -1774,6 +1774,10 @@ c_unparser::emit_module_init ()
   o->newline() << "#include \"linux/stp_tracepoint.c\"";
   o->newline() << "#endif";

+  o->newline() << "#ifdef STAP_NEED_USER_INIT";
+  o->newline() << "static int stp_user_init(void);";
+  o->newline() << "#endif";
+
   o->newline();
   o->newline() << "static int systemtap_module_init (void) {";
   o->newline(1) << "int rc = 0;";
@@ -1922,6 +1926,15 @@ c_unparser::emit_module_init ()
   o->newline(1) << "goto out;";
   o->indent(-1);

+  // user init hook
+  o->newline() << "#ifdef STAP_NEED_USER_INIT";
+  o->newline() << "rc = stp_user_init();";
+  o->newline() << "if (rc) {";
+  o->newline(1) << "_stp_error (\"couldn't initialize user init\");";
+  o->newline() << "goto out;";
+  o->newline(-1) << "}";
+  o->newline() << "#endif";
+





On Tue, Jul 11, 2017 at 9:46 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Update. Some of the system calls I am doing in the begin probe are
> blocking. I understand that it will break things on multicore systems.
> Am I right?
>
> On Tue, Jul 11, 2017 at 9:24 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> Update. The failure happens consistently in the same context
>>
>> "context[1] stuck: (null), line_get=36848, line_put=36914
>> last_err=(null) last_stmt=identifier 'probe_begin'"
>> where line_get and line_put are lines in the enter_be_probe()
>>
>> 36843     #endif
>> 36844     goto probe_epilogue;
>> 36845   }
>> 36846   if (atomic_read (session_state()) != stp->state)
>> 36847     goto probe_epilogue;
>> 36848   c = _stp_runtime_entryfn_get_context(__LINE__);
>> 36849   if (!c) {
>> 36850     #if !INTERRUPTIBLE
>> 36851     atomic_inc (skipped_count());
>> 36852     #endif
>> 36853     #ifdef STP_TIMING
>> 36854     atomic_inc (skipped_count_reentrant());
>> 36855     #endif
>> 36856     goto probe_epilogue;
>> 36857   }
>> ..................
>> 36907     }
>> 36908   }
>> 36909 probe_epilogue:
>> 36910   if (unlikely (atomic_read (skipped_count()) > MAXSKIPPED)) {
>> 36911     if (unlikely (pseudo_atomic_cmpxchg(session_state(),
>> STAP_SESSION_RUNNING, STAP_SESSION_ERROR) == STAP_SESSION_RUNNING))
>> 36912     _stp_error ("Skipped too many probes, check MAXSKIPPED or
>> try again with stap -t for more details.");
>> 36913   }
>> 36914   _stp_runtime_entryfn_put_context(c, __LINE__);
>> 36915   #if !INTERRUPTIBLE
>> 36916   local_irq_restore (flags);
>> 36917   #endif
>> 36918   #endif // STP_ALIBI
>>
>> On Mon, Jul 10, 2017 at 6:16 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> Hi,
>>>
>>> I am getting context[2] stuck: (null) error. The cause of error is
>>> likely the "unmanaged" code I have added to the driver. Specifically I
>>> have a shared memory (mmap) in the driver. The failure happens
>>> randomly every 50-200 module restarts The failure happens only on the
>>> multicore CPUs, or happens often enough to be caught.
>>>
>>> I tried to force the the wait function with
>>> STAP_OVERRIDE_STUCK_CONTEXT - kernel panics in one of the (probably
>>> random) probes.
>>>
>>> While debugging the issue I  patched the SystemTap source code - added
>>> an argument to the _stp_runtime_entryfn_get_context(int) like in this
>>> commit https://github.com/larytet/SystemTap/commit/61a284732893fa6f201e07f9f12f5e1820e7c26f
>>> In the function _stp_runtime_context_wait() I print the line in the
>>> source code which called the _stp_runtime_entryfn_get_context()
>>>
>>> The "bad" context is enter_be_probe(). I checked the source code of
>>> enter_be_probe() and there is not much there.
>>>
>>> I struggle with the problem for some time and I will greatly
>>> appreciate any tip.
>>>
>>> Thank you, Arkady.

Re: context[2] stuck: (null)

[David Smith]

On Tue, Jul 11, 2017 at 1:46 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Update. Some of the system calls I am doing in the begin probe are
> blocking. I understand that it will break things on multicore systems.
> Am I right?

Yes. Systemtap probes can never block.

Perhaps we can suggest some ideas if you tell us a bit more of the
background of what you are trying to do here.

-- 
David Smith
Principal Software Engineer
Red Hat

Re: context[2] stuck: (null)

[David Smith]

On Tue, Jul 11, 2017 at 5:30 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Is there a chance to get a user defined hook in the context of
> systemtap_module_init()

Probably not. It seems like too big of a security/safety hole.

However, if you are developing a new feature that requires runtime
support, we'd have no problems adding some initialization code where
you need it.

-- 
David Smith
Principal Software Engineer
Red Hat

Re: context[2] stuck: (null)

[Arkady]

On Wed, Jul 12, 2017 at 3:58 PM, David Smith <dsmith@redhat.com> wrote:
> On Tue, Jul 11, 2017 at 1:46 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> Update. Some of the system calls I am doing in the begin probe are
>> blocking. I understand that it will break things on multicore systems.
>> Am I right?
>
> Yes. Systemtap probes can never block.
>
> Perhaps we can suggest some ideas if you tell us a bit more of the
> background of what you are trying to do here.

My goal is to allocate the shared memory - one or more large (100s of
MBs) vzallocs,
add a dozen files to the sysfs and debugfs.
Some of the things I am doing involve kernel APIs which block. I was thinking
about the following options

*  Add a user defined hook in the systemtap_module_init() before "probe begin"
*  Make probes begin and end a special case and allow blocking
*  A separate driver which creates things I need in the kernel and "exports" API
*  Ask a kernel tasklet/workq/whatever to do the initialization

The first was easiest to implement and allowed me to solve the problem here and
now.
The second option involves SystemTap hacking which I am trying to avoid
because of the maintenance overhead in the future
A separate driver complicates boot and introduce security risks which I would
like to avoid for my specific application unless absolutely necessary.
The last option - take the initialization offline - a tasklet could do
the work. It
was longer to develop and the hour was late.

>
> --
> David Smith
> Principal Software Engineer
> Red Hat

Re: context[2] stuck: (null)

[David Smith]

On Wed, Jul 12, 2017 at 8:16 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> On Wed, Jul 12, 2017 at 3:58 PM, David Smith <dsmith@redhat.com> wrote:
>> On Tue, Jul 11, 2017 at 1:46 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> Update. Some of the system calls I am doing in the begin probe are
>>> blocking. I understand that it will break things on multicore systems.
>>> Am I right?
>>
>> Yes. Systemtap probes can never block.
>>
>> Perhaps we can suggest some ideas if you tell us a bit more of the
>> background of what you are trying to do here.
>
> My goal is to allocate the shared memory - one or more large (100s of
> MBs) vzallocs,
> add a dozen files to the sysfs and debugfs.

Note that systemtap already has procfs 'probes', which create procfs files.

> Some of the things I am doing involve kernel APIs which block. I was thinking
> about the following options
>
> *  Add a user defined hook in the systemtap_module_init() before "probe begin"
> *  Make probes begin and end a special case and allow blocking
> *  A separate driver which creates things I need in the kernel and "exports" API
> *  Ask a kernel tasklet/workq/whatever to do the initialization
>
> The first was easiest to implement and allowed me to solve the problem here and
> now.
> The second option involves SystemTap hacking which I am trying to avoid
> because of the maintenance overhead in the future
> A separate driver complicates boot and introduce security risks which I would
> like to avoid for my specific application unless absolutely necessary.
> The last option - take the initialization offline - a tasklet could do
> the work. It
> was longer to develop and the hour was late.

If we wanted to include this functionality with systemtap, we'd have
to generalize this a bit and make it less domain specific. Can you
think of a script language extension that would express what you are
trying to do here?

-- 
David Smith
Principal Software Engineer
Red Hat

Re: context[2] stuck: (null)

[Arkady]

On Wed, Jul 12, 2017 at 4:55 PM, David Smith <dsmith@redhat.com> wrote:
> On Wed, Jul 12, 2017 at 8:16 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> My goal is to allocate the shared memory - one or more large (100s of
>> MBs) vzallocs,
>> add a dozen files to the sysfs and debugfs.
>
> Note that systemtap already has procfs 'probes', which create procfs files.
>

procfs API has limitations and inconveniences.

>> Some of the things I am doing involve kernel APIs which block. I was thinking
>> about the following options
>> The last option - take the initialization offline - a tasklet could do
>> the work. It
>> was longer to develop and the hour was late.
>
> If we wanted to include this functionality with systemtap, we'd have
> to generalize this a bit and make it less domain specific. Can you
> think of a script language extension that would express what you are
> trying to do here?
>

I would like to have an Initialization probe which allows use of blocking APIs.


> --
> David Smith
> Principal Software Engineer
> Red Hat

Re: context[2] stuck: (null)

[Arkady]

On Wed, Jul 12, 2017 at 5:40 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> On Wed, Jul 12, 2017 at 4:55 PM, David Smith <dsmith@redhat.com> wrote:
>> On Wed, Jul 12, 2017 at 8:16 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> My goal is to allocate the shared memory - one or more large (100s of
>>> MBs) vzallocs,
>>> add a dozen files to the sysfs and debugfs.
>>
>> Note that systemtap already has procfs 'probes', which create procfs files.
>>
>
> procfs API has limitations and inconveniences.
>
>> If we wanted to include this functionality with systemtap, we'd have
>> to generalize this a bit and make it less domain specific. Can you
>> think of a script language extension that would express what you are
>> trying to do here?
>>
>
> I would like to have an Initialization probe which allows use of blocking APIs.

.... for example probe begin_interruptable {}  and probe end_interruptable {}

>
>> --
>> David Smith
>> Principal Software Engineer
>> Red Hat

Re: context[2] stuck: (null)

[Arkady]

In the "probe begin" I can offload the init to a just spawned thread
and return from the "probe begin" immediately. The rest of the probes
will test a global atomic and exit if the data structures are not
ready yet.

In the "probe end" I have another problem. I have to wait for the
offloaded task to complete. I am polling an atomic in a tight loop.
This is not quite a solution I like.

On Wed, Jul 12, 2017 at 6:39 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> On Wed, Jul 12, 2017 at 5:40 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> On Wed, Jul 12, 2017 at 4:55 PM, David Smith <dsmith@redhat.com> wrote:
>>> On Wed, Jul 12, 2017 at 8:16 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>> My goal is to allocate the shared memory - one or more large (100s of
>>>> MBs) vzallocs,
>>>> add a dozen files to the sysfs and debugfs.
>>>
>>> Note that systemtap already has procfs 'probes', which create procfs files.
>>>
>>
>> procfs API has limitations and inconveniences.
>>
>>> If we wanted to include this functionality with systemtap, we'd have
>>> to generalize this a bit and make it less domain specific. Can you
>>> think of a script language extension that would express what you are
>>> trying to do here?
>>>
>>
>> I would like to have an Initialization probe which allows use of blocking APIs.
>
> .... for example probe begin_interruptable {}  and probe end_interruptable {}
>
>>
>>> --
>>> David Smith
>>> Principal Software Engineer
>>> Red Hat

Re: context[2] stuck: (null)

[David Smith]

On Wed, Jul 12, 2017 at 10:39 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> On Wed, Jul 12, 2017 at 5:40 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> On Wed, Jul 12, 2017 at 4:55 PM, David Smith <dsmith@redhat.com> wrote:
>>> On Wed, Jul 12, 2017 at 8:16 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>> My goal is to allocate the shared memory - one or more large (100s of
>>>> MBs) vzallocs,
>>>> add a dozen files to the sysfs and debugfs.
>>>
>>> Note that systemtap already has procfs 'probes', which create procfs files.
>>>
>>
>> procfs API has limitations and inconveniences.
>>
>>> If we wanted to include this functionality with systemtap, we'd have
>>> to generalize this a bit and make it less domain specific. Can you
>>> think of a script language extension that would express what you are
>>> trying to do here?
>>>
>>
>> I would like to have an Initialization probe which allows use of blocking APIs.
>
> .... for example probe begin_interruptable {}  and probe end_interruptable {}

I think you misunderstood me, probably because I didn't express myself
well. Let's try again.

If you wanted to include the functionality of the allocating the
shared memory to store data and retrieve it via sysfs and debugfs
files in systemtap, what would the user experience look like? Let's
say I'm in a syscall probe. How would I add something to the shared
memory? What would the sysfs/debugfs probe look like?

-- 
David Smith
Principal Software Engineer
Red Hat

Re: context[2] stuck: (null)

[Arkady]

>>>
>>> I would like to have an Initialization probe which allows use of blocking APIs.
>>
>> .... for example probe begin_interruptable {}  and probe end_interruptable {}
>
> I think you misunderstood me, probably because I didn't express myself
> well. Let's try again.
>
> If you wanted to include the functionality of the allocating the
> shared memory to store data and retrieve it via sysfs and debugfs
> files in systemtap, what would the user experience look like? Let's
> say I'm in a syscall probe. How would I add something to the shared
> memory? What would the sysfs/debugfs probe look like?
>

I would prefer a generic interface allowing me to call blocking APIs

> --
> David Smith
> Principal Software Engineer
> Red Hat

Re: context[2] stuck: (null)

[Arkady]

On Fri, Jul 14, 2017 at 7:38 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>
>>>> I would like to have an Initialization probe which allows use of blocking APIs.
>>>
>>> .... for example probe begin_interruptable {}  and probe end_interruptable {}
>>
>> I think you misunderstood me, probably because I didn't express myself
>> well. Let's try again.
>>
>> If you wanted to include the functionality of the allocating the
>> shared memory to store data and retrieve it via sysfs and debugfs
>> files in systemtap, what would the user experience look like? Let's
>> say I'm in a syscall probe. How would I add something to the shared
>> memory? What would the sysfs/debugfs probe look like?
>>
>
> I would prefer a generic interface allowing me to call blocking APIs

This is, for example, my allocation function
https://gist.github.com/larytet/643d4a2b932b3d133f6dfca4a4ee8b60
I do not know how I could define a memory allocation API which does
everything I will ever need. The same problem with debugfs/sysfs API.

I understand that I am trying to use STAP in a not intended for the
product application.

>
>> --
>> David Smith
>> Principal Software Engineer
>> Red Hat

Re: context[2] stuck: (null)

[Arkady]

Another approach can be to allow "probe proc" to call blocking APIs

On Fri, Jul 14, 2017 at 8:39 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> On Fri, Jul 14, 2017 at 7:38 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>>
>>>>> I would like to have an Initialization probe which allows use of blocking APIs.
>>>>
>>>> .... for example probe begin_interruptable {}  and probe end_interruptable {}
>>>
>>> I think you misunderstood me, probably because I didn't express myself
>>> well. Let's try again.
>>>
>>> If you wanted to include the functionality of the allocating the
>>> shared memory to store data and retrieve it via sysfs and debugfs
>>> files in systemtap, what would the user experience look like? Let's
>>> say I'm in a syscall probe. How would I add something to the shared
>>> memory? What would the sysfs/debugfs probe look like?
>>>
>>
>> I would prefer a generic interface allowing me to call blocking APIs
>
> This is, for example, my allocation function
> https://gist.github.com/larytet/643d4a2b932b3d133f6dfca4a4ee8b60
> I do not know how I could define a memory allocation API which does
> everything I will ever need. The same problem with debugfs/sysfs API.
>
> I understand that I am trying to use STAP in a not intended for the
> product application.
>
>>
>>> --
>>> David Smith
>>> Principal Software Engineer
>>> Red Hat