No output from trace file after a few hours

No output from trace file after a few hours

[Daniel Doron]

Hi,

I built my module to probe a few system calls and functions. Nothing
fancy, no big work in it, just prints some data. Then I wrote a simple
python script to read data directly off the
/sys/kernel/debug/systemtap/mymodule/trace0 file (of course running as
sudo)
I run stap in Flight recorder mode.
After a few hours (varies from 8 to 20) I stop getting any output from
the module...
Any idea how to check why that is?
dmesg does not report anything unusual...

Thanks.
Daniel.

Re: No output from trace file after a few hours

[Arkady]

Please send the stap script

On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
<danielmeirdoron@gmail.com> wrote:
> Hi,
>
> I built my module to probe a few system calls and functions. Nothing
> fancy, no big work in it, just prints some data. Then I wrote a simple
> python script to read data directly off the
> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course running as
> sudo)
> I run stap in Flight recorder mode.
> After a few hours (varies from 8 to 20) I stop getting any output from
> the module...
> Any idea how to check why that is?
> dmesg does not report anything unusual...
>
> Thanks.
> Daniel.

Re: No output from trace file after a few hours

[Arkady]

Do you have the kernel version pinned?

On Thu, Jun 15, 2017 at 10:12 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Please send the stap script
>
> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
> <danielmeirdoron@gmail.com> wrote:
>> Hi,
>>
>> I built my module to probe a few system calls and functions. Nothing
>> fancy, no big work in it, just prints some data. Then I wrote a simple
>> python script to read data directly off the
>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course running as
>> sudo)
>> I run stap in Flight recorder mode.
>> After a few hours (varies from 8 to 20) I stop getting any output from
>> the module...
>> Any idea how to check why that is?
>> dmesg does not report anything unusual...
>>
>> Thanks.
>> Daniel.

Re: No output from trace file after a few hours

[Daniel Doron]

Sorry, I do not know what you mean by that...

On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Do you have the kernel version pinned?
>
> On Thu, Jun 15, 2017 at 10:12 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> Please send the stap script
>>
>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>> <danielmeirdoron@gmail.com> wrote:
>>> Hi,
>>>
>>> I built my module to probe a few system calls and functions. Nothing
>>> fancy, no big work in it, just prints some data. Then I wrote a simple
>>> python script to read data directly off the
>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course running as
>>> sudo)
>>> I run stap in Flight recorder mode.
>>> After a few hours (varies from 8 to 20) I stop getting any output from
>>> the module...
>>> Any idea how to check why that is?
>>> dmesg does not report anything unusual...
>>>
>>> Thanks.
>>> Daniel.

Re: No output from trace file after a few hours

[Daniel Doron]

if you mean that the kernel version does not get upgraded, then yes.

On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
<danielmeirdoron@gmail.com> wrote:
> Sorry, I do not know what you mean by that...
>
> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> Do you have the kernel version pinned?
>>
>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> Please send the stap script
>>>
>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>> <danielmeirdoron@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I built my module to probe a few system calls and functions. Nothing
>>>> fancy, no big work in it, just prints some data. Then I wrote a simple
>>>> python script to read data directly off the
>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course running as
>>>> sudo)
>>>> I run stap in Flight recorder mode.
>>>> After a few hours (varies from 8 to 20) I stop getting any output from
>>>> the module...
>>>> Any idea how to check why that is?
>>>> dmesg does not report anything unusual...
>>>>
>>>> Thanks.
>>>> Daniel.

Re: No output from trace file after a few hours

[Daniel Doron]

BTW kernel version: 2.6.32-431.el6.x86_64

On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
<danielmeirdoron@gmail.com> wrote:
> if you mean that the kernel version does not get upgraded, then yes.
>
> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
> <danielmeirdoron@gmail.com> wrote:
>> Sorry, I do not know what you mean by that...
>>
>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> Do you have the kernel version pinned?
>>>
>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>> Please send the stap script
>>>>
>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>> <danielmeirdoron@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I built my module to probe a few system calls and functions. Nothing
>>>>> fancy, no big work in it, just prints some data. Then I wrote a simple
>>>>> python script to read data directly off the
>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course running as
>>>>> sudo)
>>>>> I run stap in Flight recorder mode.
>>>>> After a few hours (varies from 8 to 20) I stop getting any output from
>>>>> the module...
>>>>> Any idea how to check why that is?
>>>>> dmesg does not report anything unusual...
>>>>>
>>>>> Thanks.
>>>>> Daniel.

Re: No output from trace file after a few hours

[Daniel Doron]

Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
Copyright (C) 2005-2015 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP

On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
<danielmeirdoron@gmail.com> wrote:
> BTW kernel version: 2.6.32-431.el6.x86_64
>
> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
> <danielmeirdoron@gmail.com> wrote:
>> if you mean that the kernel version does not get upgraded, then yes.
>>
>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>> <danielmeirdoron@gmail.com> wrote:
>>> Sorry, I do not know what you mean by that...
>>>
>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>> Do you have the kernel version pinned?
>>>>
>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>> Please send the stap script
>>>>>
>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I built my module to probe a few system calls and functions. Nothing
>>>>>> fancy, no big work in it, just prints some data. Then I wrote a simple
>>>>>> python script to read data directly off the
>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course running as
>>>>>> sudo)
>>>>>> I run stap in Flight recorder mode.
>>>>>> After a few hours (varies from 8 to 20) I stop getting any output from
>>>>>> the module...
>>>>>> Any idea how to check why that is?
>>>>>> dmesg does not report anything unusual...
>>>>>>
>>>>>> Thanks.
>>>>>> Daniel.

Re: No output from trace file after a few hours

[Arkady]

How long time it runs for you?

On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
<danielmeirdoron@gmail.com> wrote:
> Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
> Copyright (C) 2005-2015 Red Hat, Inc. and others
> This is free software; see the source for copying conditions.
> enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>
> On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
> <danielmeirdoron@gmail.com> wrote:
>> BTW kernel version: 2.6.32-431.el6.x86_64
>>
>> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>> <danielmeirdoron@gmail.com> wrote:
>>> if you mean that the kernel version does not get upgraded, then yes.
>>>
>>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>>> <danielmeirdoron@gmail.com> wrote:
>>>> Sorry, I do not know what you mean by that...
>>>>
>>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>> Do you have the kernel version pinned?
>>>>>
>>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>>> Please send the stap script
>>>>>>
>>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I built my module to probe a few system calls and functions. Nothing
>>>>>>> fancy, no big work in it, just prints some data. Then I wrote a simple
>>>>>>> python script to read data directly off the
>>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course running as
>>>>>>> sudo)
>>>>>>> I run stap in Flight recorder mode.
>>>>>>> After a few hours (varies from 8 to 20) I stop getting any output from
>>>>>>> the module...
>>>>>>> Any idea how to check why that is?
>>>>>>> dmesg does not report anything unusual...
>>>>>>>
>>>>>>> Thanks.
>>>>>>> Daniel.

Re: No output from trace file after a few hours

[Arkady]

Do you do something like

sudo stap -F -o ./test.log -S 1,2  test.stp ?

On Thu, Jun 15, 2017 at 11:25 AM, Daniel Doron
<danielmeirdoron@gmail.com> wrote:
> On that specific kernel about 8-10 hours.
> On my machine with latest kernel, been running for 24 hours.
> I need to check other distros.
>
> On Jun 15, 2017 11:18, "Arkady" <arkady.miasnikov@gmail.com> wrote:
>>
>> How long time it runs for you?
>>
>> On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
>> <danielmeirdoron@gmail.com> wrote:
>> > Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
>> > Copyright (C) 2005-2015 Red Hat, Inc. and others
>> > This is free software; see the source for copying conditions.
>> > enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>> >
>> > On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
>> > <danielmeirdoron@gmail.com> wrote:
>> >> BTW kernel version: 2.6.32-431.el6.x86_64
>> >>
>> >> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>> >> <danielmeirdoron@gmail.com> wrote:
>> >>> if you mean that the kernel version does not get upgraded, then yes.
>> >>>
>> >>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>> >>> <danielmeirdoron@gmail.com> wrote:
>> >>>> Sorry, I do not know what you mean by that...
>> >>>>
>> >>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com>
>> >>>> wrote:
>> >>>>> Do you have the kernel version pinned?
>> >>>>>
>> >>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady
>> >>>>> <arkady.miasnikov@gmail.com> wrote:
>> >>>>>> Please send the stap script
>> >>>>>>
>> >>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>> >>>>>> <danielmeirdoron@gmail.com> wrote:
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>> I built my module to probe a few system calls and functions.
>> >>>>>>> Nothing
>> >>>>>>> fancy, no big work in it, just prints some data. Then I wrote a
>> >>>>>>> simple
>> >>>>>>> python script to read data directly off the
>> >>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course
>> >>>>>>> running as
>> >>>>>>> sudo)
>> >>>>>>> I run stap in Flight recorder mode.
>> >>>>>>> After a few hours (varies from 8 to 20) I stop getting any output
>> >>>>>>> from
>> >>>>>>> the module...
>> >>>>>>> Any idea how to check why that is?
>> >>>>>>> dmesg does not report anything unusual...
>> >>>>>>>
>> >>>>>>> Thanks.
>> >>>>>>> Daniel.

Re: No output from trace file after a few hours

[Daniel Doron]

On that specific kernel about 8-10 hours.
On my machine with latest kernel, been running for 24 hours.
I need to check other distros.

sudo stap -v -F -m mymodule test.stp

On Thu, Jun 15, 2017 at 11:26 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Do you do something like
>
> sudo stap -F -o ./test.log -S 1,2  test.stp ?
>
> On Thu, Jun 15, 2017 at 11:25 AM, Daniel Doron
> <danielmeirdoron@gmail.com> wrote:
>> On that specific kernel about 8-10 hours.
>> On my machine with latest kernel, been running for 24 hours.
>> I need to check other distros.
>>
>> On Jun 15, 2017 11:18, "Arkady" <arkady.miasnikov@gmail.com> wrote:
>>>
>>> How long time it runs for you?
>>>
>>> On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
>>> <danielmeirdoron@gmail.com> wrote:
>>> > Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
>>> > Copyright (C) 2005-2015 Red Hat, Inc. and others
>>> > This is free software; see the source for copying conditions.
>>> > enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>> >
>>> > On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
>>> > <danielmeirdoron@gmail.com> wrote:
>>> >> BTW kernel version: 2.6.32-431.el6.x86_64
>>> >>
>>> >> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>>> >> <danielmeirdoron@gmail.com> wrote:
>>> >>> if you mean that the kernel version does not get upgraded, then yes.
>>> >>>
>>> >>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>>> >>> <danielmeirdoron@gmail.com> wrote:
>>> >>>> Sorry, I do not know what you mean by that...
>>> >>>>
>>> >>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com>
>>> >>>> wrote:
>>> >>>>> Do you have the kernel version pinned?
>>> >>>>>
>>> >>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady
>>> >>>>> <arkady.miasnikov@gmail.com> wrote:
>>> >>>>>> Please send the stap script
>>> >>>>>>
>>> >>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>> >>>>>> <danielmeirdoron@gmail.com> wrote:
>>> >>>>>>> Hi,
>>> >>>>>>>
>>> >>>>>>> I built my module to probe a few system calls and functions.
>>> >>>>>>> Nothing
>>> >>>>>>> fancy, no big work in it, just prints some data. Then I wrote a
>>> >>>>>>> simple
>>> >>>>>>> python script to read data directly off the
>>> >>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course
>>> >>>>>>> running as
>>> >>>>>>> sudo)
>>> >>>>>>> I run stap in Flight recorder mode.
>>> >>>>>>> After a few hours (varies from 8 to 20) I stop getting any output
>>> >>>>>>> from
>>> >>>>>>> the module...
>>> >>>>>>> Any idea how to check why that is?
>>> >>>>>>> dmesg does not report anything unusual...
>>> >>>>>>>
>>> >>>>>>> Thanks.
>>> >>>>>>> Daniel.

Re: No output from trace file after a few hours

[Arkady]

Interesting. I have reproduced the problem by running two tight bash
loops simultaneously

sudo stap -F -o ./test.log -S 1,2  test.stp
while [ 1 ];do ls > /dev/null;done &
while [ 1 ];do echo -en "\033[0;0H";ls -l | grep "test.log";echo -en
"\033[0;0H";done;

Linux centos6 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux



On Thu, Jun 15, 2017 at 11:59 AM, Daniel Doron
<danielmeirdoron@gmail.com> wrote:
> On that specific kernel about 8-10 hours.
> On my machine with latest kernel, been running for 24 hours.
> I need to check other distros.
>
> sudo stap -v -F -m mymodule test.stp
>
> On Thu, Jun 15, 2017 at 11:26 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> Do you do something like
>>
>> sudo stap -F -o ./test.log -S 1,2  test.stp ?
>>
>> On Thu, Jun 15, 2017 at 11:25 AM, Daniel Doron
>> <danielmeirdoron@gmail.com> wrote:
>>> On that specific kernel about 8-10 hours.
>>> On my machine with latest kernel, been running for 24 hours.
>>> I need to check other distros.
>>>
>>> On Jun 15, 2017 11:18, "Arkady" <arkady.miasnikov@gmail.com> wrote:
>>>>
>>>> How long time it runs for you?
>>>>
>>>> On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
>>>> <danielmeirdoron@gmail.com> wrote:
>>>> > Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
>>>> > Copyright (C) 2005-2015 Red Hat, Inc. and others
>>>> > This is free software; see the source for copying conditions.
>>>> > enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>>> >
>>>> > On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
>>>> > <danielmeirdoron@gmail.com> wrote:
>>>> >> BTW kernel version: 2.6.32-431.el6.x86_64
>>>> >>
>>>> >> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>>>> >> <danielmeirdoron@gmail.com> wrote:
>>>> >>> if you mean that the kernel version does not get upgraded, then yes.
>>>> >>>
>>>> >>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>>>> >>> <danielmeirdoron@gmail.com> wrote:
>>>> >>>> Sorry, I do not know what you mean by that...
>>>> >>>>
>>>> >>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com>
>>>> >>>> wrote:
>>>> >>>>> Do you have the kernel version pinned?
>>>> >>>>>
>>>> >>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady
>>>> >>>>> <arkady.miasnikov@gmail.com> wrote:
>>>> >>>>>> Please send the stap script
>>>> >>>>>>
>>>> >>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>> >>>>>> <danielmeirdoron@gmail.com> wrote:
>>>> >>>>>>> Hi,
>>>> >>>>>>>
>>>> >>>>>>> I built my module to probe a few system calls and functions.
>>>> >>>>>>> Nothing
>>>> >>>>>>> fancy, no big work in it, just prints some data. Then I wrote a
>>>> >>>>>>> simple
>>>> >>>>>>> python script to read data directly off the
>>>> >>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course
>>>> >>>>>>> running as
>>>> >>>>>>> sudo)
>>>> >>>>>>> I run stap in Flight recorder mode.
>>>> >>>>>>> After a few hours (varies from 8 to 20) I stop getting any output
>>>> >>>>>>> from
>>>> >>>>>>> the module...
>>>> >>>>>>> Any idea how to check why that is?
>>>> >>>>>>> dmesg does not report anything unusual...
>>>> >>>>>>>
>>>> >>>>>>> Thanks.
>>>> >>>>>>> Daniel.

Re: No output from trace file after a few hours

[Arkady]

STAP simply quits in a couple of seconds

sudo stap -F -o ./test.log -S 1,2  test.stp
94089
ps -ef | grep -v "grep" | grep  94089

On Thu, Jun 15, 2017 at 1:53 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Interesting. I have reproduced the problem by running two tight bash
> loops simultaneously
>
> sudo stap -F -o ./test.log -S 1,2  test.stp
> while [ 1 ];do ls > /dev/null;done &
> while [ 1 ];do echo -en "\033[0;0H";ls -l | grep "test.log";echo -en
> "\033[0;0H";done;
>
> Linux centos6 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux
>
>
>
> On Thu, Jun 15, 2017 at 11:59 AM, Daniel Doron
> <danielmeirdoron@gmail.com> wrote:
>> On that specific kernel about 8-10 hours.
>> On my machine with latest kernel, been running for 24 hours.
>> I need to check other distros.
>>
>> sudo stap -v -F -m mymodule test.stp
>>
>> On Thu, Jun 15, 2017 at 11:26 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> Do you do something like
>>>
>>> sudo stap -F -o ./test.log -S 1,2  test.stp ?
>>>
>>> On Thu, Jun 15, 2017 at 11:25 AM, Daniel Doron
>>> <danielmeirdoron@gmail.com> wrote:
>>>> On that specific kernel about 8-10 hours.
>>>> On my machine with latest kernel, been running for 24 hours.
>>>> I need to check other distros.
>>>>
>>>> On Jun 15, 2017 11:18, "Arkady" <arkady.miasnikov@gmail.com> wrote:
>>>>>
>>>>> How long time it runs for you?
>>>>>
>>>>> On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>> > Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
>>>>> > Copyright (C) 2005-2015 Red Hat, Inc. and others
>>>>> > This is free software; see the source for copying conditions.
>>>>> > enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>>>> >
>>>>> > On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
>>>>> > <danielmeirdoron@gmail.com> wrote:
>>>>> >> BTW kernel version: 2.6.32-431.el6.x86_64
>>>>> >>
>>>>> >> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>>>>> >> <danielmeirdoron@gmail.com> wrote:
>>>>> >>> if you mean that the kernel version does not get upgraded, then yes.
>>>>> >>>
>>>>> >>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>>>>> >>> <danielmeirdoron@gmail.com> wrote:
>>>>> >>>> Sorry, I do not know what you mean by that...
>>>>> >>>>
>>>>> >>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com>
>>>>> >>>> wrote:
>>>>> >>>>> Do you have the kernel version pinned?
>>>>> >>>>>
>>>>> >>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady
>>>>> >>>>> <arkady.miasnikov@gmail.com> wrote:
>>>>> >>>>>> Please send the stap script
>>>>> >>>>>>
>>>>> >>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>>> >>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>> >>>>>>> Hi,
>>>>> >>>>>>>
>>>>> >>>>>>> I built my module to probe a few system calls and functions.
>>>>> >>>>>>> Nothing
>>>>> >>>>>>> fancy, no big work in it, just prints some data. Then I wrote a
>>>>> >>>>>>> simple
>>>>> >>>>>>> python script to read data directly off the
>>>>> >>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course
>>>>> >>>>>>> running as
>>>>> >>>>>>> sudo)
>>>>> >>>>>>> I run stap in Flight recorder mode.
>>>>> >>>>>>> After a few hours (varies from 8 to 20) I stop getting any output
>>>>> >>>>>>> from
>>>>> >>>>>>> the module...
>>>>> >>>>>>> Any idea how to check why that is?
>>>>> >>>>>>> dmesg does not report anything unusual...
>>>>> >>>>>>>
>>>>> >>>>>>> Thanks.
>>>>> >>>>>>> Daniel.

Re: No output from trace file after a few hours

[Arkady]

This works fine

sudo stap -D MAXSKIPPED=0 -D MAXTRYLOCK=1000 -D TRYLOCKDELAY=100  -F
-o ./test.log -S 1,2  test.stp

On Thu, Jun 15, 2017 at 2:03 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> STAP simply quits in a couple of seconds
>
> sudo stap -F -o ./test.log -S 1,2  test.stp
> 94089
> ps -ef | grep -v "grep" | grep  94089
>
> On Thu, Jun 15, 2017 at 1:53 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> Interesting. I have reproduced the problem by running two tight bash
>> loops simultaneously
>>
>> sudo stap -F -o ./test.log -S 1,2  test.stp
>> while [ 1 ];do ls > /dev/null;done &
>> while [ 1 ];do echo -en "\033[0;0H";ls -l | grep "test.log";echo -en
>> "\033[0;0H";done;
>>
>> Linux centos6 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC
>> 2016 x86_64 x86_64 x86_64 GNU/Linux
>>
>>
>>
>> On Thu, Jun 15, 2017 at 11:59 AM, Daniel Doron
>> <danielmeirdoron@gmail.com> wrote:
>>> On that specific kernel about 8-10 hours.
>>> On my machine with latest kernel, been running for 24 hours.
>>> I need to check other distros.
>>>
>>> sudo stap -v -F -m mymodule test.stp
>>>
>>> On Thu, Jun 15, 2017 at 11:26 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>> Do you do something like
>>>>
>>>> sudo stap -F -o ./test.log -S 1,2  test.stp ?
>>>>
>>>> On Thu, Jun 15, 2017 at 11:25 AM, Daniel Doron
>>>> <danielmeirdoron@gmail.com> wrote:
>>>>> On that specific kernel about 8-10 hours.
>>>>> On my machine with latest kernel, been running for 24 hours.
>>>>> I need to check other distros.
>>>>>
>>>>> On Jun 15, 2017 11:18, "Arkady" <arkady.miasnikov@gmail.com> wrote:
>>>>>>
>>>>>> How long time it runs for you?
>>>>>>
>>>>>> On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>> > Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
>>>>>> > Copyright (C) 2005-2015 Red Hat, Inc. and others
>>>>>> > This is free software; see the source for copying conditions.
>>>>>> > enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>>>>> >
>>>>>> > On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
>>>>>> > <danielmeirdoron@gmail.com> wrote:
>>>>>> >> BTW kernel version: 2.6.32-431.el6.x86_64
>>>>>> >>
>>>>>> >> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>>>>>> >> <danielmeirdoron@gmail.com> wrote:
>>>>>> >>> if you mean that the kernel version does not get upgraded, then yes.
>>>>>> >>>
>>>>>> >>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>>>>>> >>> <danielmeirdoron@gmail.com> wrote:
>>>>>> >>>> Sorry, I do not know what you mean by that...
>>>>>> >>>>
>>>>>> >>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com>
>>>>>> >>>> wrote:
>>>>>> >>>>> Do you have the kernel version pinned?
>>>>>> >>>>>
>>>>>> >>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady
>>>>>> >>>>> <arkady.miasnikov@gmail.com> wrote:
>>>>>> >>>>>> Please send the stap script
>>>>>> >>>>>>
>>>>>> >>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>>>> >>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>> >>>>>>> Hi,
>>>>>> >>>>>>>
>>>>>> >>>>>>> I built my module to probe a few system calls and functions.
>>>>>> >>>>>>> Nothing
>>>>>> >>>>>>> fancy, no big work in it, just prints some data. Then I wrote a
>>>>>> >>>>>>> simple
>>>>>> >>>>>>> python script to read data directly off the
>>>>>> >>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course
>>>>>> >>>>>>> running as
>>>>>> >>>>>>> sudo)
>>>>>> >>>>>>> I run stap in Flight recorder mode.
>>>>>> >>>>>>> After a few hours (varies from 8 to 20) I stop getting any output
>>>>>> >>>>>>> from
>>>>>> >>>>>>> the module...
>>>>>> >>>>>>> Any idea how to check why that is?
>>>>>> >>>>>>> dmesg does not report anything unusual...
>>>>>> >>>>>>>
>>>>>> >>>>>>> Thanks.
>>>>>> >>>>>>> Daniel.

Re: No output from trace file after a few hours

[Daniel Doron]

I will try that.
BTW, this also just happened after almost 24 hours on kernel 4.11.3-1-ARCH

On Thu, Jun 15, 2017 at 2:05 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> This works fine
>
> sudo stap -D MAXSKIPPED=0 -D MAXTRYLOCK=1000 -D TRYLOCKDELAY=100  -F
> -o ./test.log -S 1,2  test.stp
>
> On Thu, Jun 15, 2017 at 2:03 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> STAP simply quits in a couple of seconds
>>
>> sudo stap -F -o ./test.log -S 1,2  test.stp
>> 94089
>> ps -ef | grep -v "grep" | grep  94089
>>
>> On Thu, Jun 15, 2017 at 1:53 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> Interesting. I have reproduced the problem by running two tight bash
>>> loops simultaneously
>>>
>>> sudo stap -F -o ./test.log -S 1,2  test.stp
>>> while [ 1 ];do ls > /dev/null;done &
>>> while [ 1 ];do echo -en "\033[0;0H";ls -l | grep "test.log";echo -en
>>> "\033[0;0H";done;
>>>
>>> Linux centos6 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC
>>> 2016 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>>
>>>
>>> On Thu, Jun 15, 2017 at 11:59 AM, Daniel Doron
>>> <danielmeirdoron@gmail.com> wrote:
>>>> On that specific kernel about 8-10 hours.
>>>> On my machine with latest kernel, been running for 24 hours.
>>>> I need to check other distros.
>>>>
>>>> sudo stap -v -F -m mymodule test.stp
>>>>
>>>> On Thu, Jun 15, 2017 at 11:26 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>> Do you do something like
>>>>>
>>>>> sudo stap -F -o ./test.log -S 1,2  test.stp ?
>>>>>
>>>>> On Thu, Jun 15, 2017 at 11:25 AM, Daniel Doron
>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>> On that specific kernel about 8-10 hours.
>>>>>> On my machine with latest kernel, been running for 24 hours.
>>>>>> I need to check other distros.
>>>>>>
>>>>>> On Jun 15, 2017 11:18, "Arkady" <arkady.miasnikov@gmail.com> wrote:
>>>>>>>
>>>>>>> How long time it runs for you?
>>>>>>>
>>>>>>> On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
>>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>> > Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
>>>>>>> > Copyright (C) 2005-2015 Red Hat, Inc. and others
>>>>>>> > This is free software; see the source for copying conditions.
>>>>>>> > enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>>>>>> >
>>>>>>> > On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
>>>>>>> > <danielmeirdoron@gmail.com> wrote:
>>>>>>> >> BTW kernel version: 2.6.32-431.el6.x86_64
>>>>>>> >>
>>>>>>> >> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>>>>>>> >> <danielmeirdoron@gmail.com> wrote:
>>>>>>> >>> if you mean that the kernel version does not get upgraded, then yes.
>>>>>>> >>>
>>>>>>> >>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>>>>>>> >>> <danielmeirdoron@gmail.com> wrote:
>>>>>>> >>>> Sorry, I do not know what you mean by that...
>>>>>>> >>>>
>>>>>>> >>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com>
>>>>>>> >>>> wrote:
>>>>>>> >>>>> Do you have the kernel version pinned?
>>>>>>> >>>>>
>>>>>>> >>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady
>>>>>>> >>>>> <arkady.miasnikov@gmail.com> wrote:
>>>>>>> >>>>>> Please send the stap script
>>>>>>> >>>>>>
>>>>>>> >>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>>>>> >>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>> >>>>>>> Hi,
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> I built my module to probe a few system calls and functions.
>>>>>>> >>>>>>> Nothing
>>>>>>> >>>>>>> fancy, no big work in it, just prints some data. Then I wrote a
>>>>>>> >>>>>>> simple
>>>>>>> >>>>>>> python script to read data directly off the
>>>>>>> >>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course
>>>>>>> >>>>>>> running as
>>>>>>> >>>>>>> sudo)
>>>>>>> >>>>>>> I run stap in Flight recorder mode.
>>>>>>> >>>>>>> After a few hours (varies from 8 to 20) I stop getting any output
>>>>>>> >>>>>>> from
>>>>>>> >>>>>>> the module...
>>>>>>> >>>>>>> Any idea how to check why that is?
>>>>>>> >>>>>>> dmesg does not report anything unusual...
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Thanks.
>>>>>>> >>>>>>> Daniel.

Re: No output from trace file after a few hours

[Arkady]

One possible reason is a system update which caused lot of probes to trigger.

On Thu, Jun 15, 2017 at 2:13 PM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
> I will try that.
> BTW, this also just happened after almost 24 hours on kernel 4.11.3-1-ARCH
>
> On Thu, Jun 15, 2017 at 2:05 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> This works fine
>>
>> sudo stap -D MAXSKIPPED=0 -D MAXTRYLOCK=1000 -D TRYLOCKDELAY=100  -F
>> -o ./test.log -S 1,2  test.stp
>>
>> On Thu, Jun 15, 2017 at 2:03 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> STAP simply quits in a couple of seconds
>>>
>>> sudo stap -F -o ./test.log -S 1,2  test.stp
>>> 94089
>>> ps -ef | grep -v "grep" | grep  94089
>>>
>>> On Thu, Jun 15, 2017 at 1:53 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>> Interesting. I have reproduced the problem by running two tight bash
>>>> loops simultaneously
>>>>
>>>> sudo stap -F -o ./test.log -S 1,2  test.stp
>>>> while [ 1 ];do ls > /dev/null;done &
>>>> while [ 1 ];do echo -en "\033[0;0H";ls -l | grep "test.log";echo -en
>>>> "\033[0;0H";done;
>>>>
>>>> Linux centos6 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC
>>>> 2016 x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>>
>>>>
>>>> On Thu, Jun 15, 2017 at 11:59 AM, Daniel Doron
>>>> <danielmeirdoron@gmail.com> wrote:
>>>>> On that specific kernel about 8-10 hours.
>>>>> On my machine with latest kernel, been running for 24 hours.
>>>>> I need to check other distros.
>>>>>
>>>>> sudo stap -v -F -m mymodule test.stp
>>>>>
>>>>> On Thu, Jun 15, 2017 at 11:26 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>>> Do you do something like
>>>>>>
>>>>>> sudo stap -F -o ./test.log -S 1,2  test.stp ?
>>>>>>
>>>>>> On Thu, Jun 15, 2017 at 11:25 AM, Daniel Doron
>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>> On that specific kernel about 8-10 hours.
>>>>>>> On my machine with latest kernel, been running for 24 hours.
>>>>>>> I need to check other distros.
>>>>>>>
>>>>>>> On Jun 15, 2017 11:18, "Arkady" <arkady.miasnikov@gmail.com> wrote:
>>>>>>>>
>>>>>>>> How long time it runs for you?
>>>>>>>>
>>>>>>>> On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
>>>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>> > Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
>>>>>>>> > Copyright (C) 2005-2015 Red Hat, Inc. and others
>>>>>>>> > This is free software; see the source for copying conditions.
>>>>>>>> > enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>>>>>>> >
>>>>>>>> > On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
>>>>>>>> > <danielmeirdoron@gmail.com> wrote:
>>>>>>>> >> BTW kernel version: 2.6.32-431.el6.x86_64
>>>>>>>> >>
>>>>>>>> >> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>>>>>>>> >> <danielmeirdoron@gmail.com> wrote:
>>>>>>>> >>> if you mean that the kernel version does not get upgraded, then yes.
>>>>>>>> >>>
>>>>>>>> >>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>>>>>>>> >>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>> >>>> Sorry, I do not know what you mean by that...
>>>>>>>> >>>>
>>>>>>>> >>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com>
>>>>>>>> >>>> wrote:
>>>>>>>> >>>>> Do you have the kernel version pinned?
>>>>>>>> >>>>>
>>>>>>>> >>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady
>>>>>>>> >>>>> <arkady.miasnikov@gmail.com> wrote:
>>>>>>>> >>>>>> Please send the stap script
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>>>>>> >>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>> >>>>>>> Hi,
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> I built my module to probe a few system calls and functions.
>>>>>>>> >>>>>>> Nothing
>>>>>>>> >>>>>>> fancy, no big work in it, just prints some data. Then I wrote a
>>>>>>>> >>>>>>> simple
>>>>>>>> >>>>>>> python script to read data directly off the
>>>>>>>> >>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course
>>>>>>>> >>>>>>> running as
>>>>>>>> >>>>>>> sudo)
>>>>>>>> >>>>>>> I run stap in Flight recorder mode.
>>>>>>>> >>>>>>> After a few hours (varies from 8 to 20) I stop getting any output
>>>>>>>> >>>>>>> from
>>>>>>>> >>>>>>> the module...
>>>>>>>> >>>>>>> Any idea how to check why that is?
>>>>>>>> >>>>>>> dmesg does not report anything unusual...
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Thanks.
>>>>>>>> >>>>>>> Daniel.

Re: No output from trace file after a few hours

[Arkady]

you can also try  --suppress-time-limits

On Thu, Jun 15, 2017 at 2:32 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> One possible reason is a system update which caused lot of probes to trigger.
>
> On Thu, Jun 15, 2017 at 2:13 PM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
>> I will try that.
>> BTW, this also just happened after almost 24 hours on kernel 4.11.3-1-ARCH
>>
>> On Thu, Jun 15, 2017 at 2:05 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> This works fine
>>>
>>> sudo stap -D MAXSKIPPED=0 -D MAXTRYLOCK=1000 -D TRYLOCKDELAY=100  -F
>>> -o ./test.log -S 1,2  test.stp
>>>
>>> On Thu, Jun 15, 2017 at 2:03 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>> STAP simply quits in a couple of seconds
>>>>
>>>> sudo stap -F -o ./test.log -S 1,2  test.stp
>>>> 94089
>>>> ps -ef | grep -v "grep" | grep  94089
>>>>
>>>> On Thu, Jun 15, 2017 at 1:53 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>> Interesting. I have reproduced the problem by running two tight bash
>>>>> loops simultaneously
>>>>>
>>>>> sudo stap -F -o ./test.log -S 1,2  test.stp
>>>>> while [ 1 ];do ls > /dev/null;done &
>>>>> while [ 1 ];do echo -en "\033[0;0H";ls -l | grep "test.log";echo -en
>>>>> "\033[0;0H";done;
>>>>>
>>>>> Linux centos6 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC
>>>>> 2016 x86_64 x86_64 x86_64 GNU/Linux
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 15, 2017 at 11:59 AM, Daniel Doron
>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>> On that specific kernel about 8-10 hours.
>>>>>> On my machine with latest kernel, been running for 24 hours.
>>>>>> I need to check other distros.
>>>>>>
>>>>>> sudo stap -v -F -m mymodule test.stp
>>>>>>
>>>>>> On Thu, Jun 15, 2017 at 11:26 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>>>> Do you do something like
>>>>>>>
>>>>>>> sudo stap -F -o ./test.log -S 1,2  test.stp ?
>>>>>>>
>>>>>>> On Thu, Jun 15, 2017 at 11:25 AM, Daniel Doron
>>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>> On that specific kernel about 8-10 hours.
>>>>>>>> On my machine with latest kernel, been running for 24 hours.
>>>>>>>> I need to check other distros.
>>>>>>>>
>>>>>>>> On Jun 15, 2017 11:18, "Arkady" <arkady.miasnikov@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> How long time it runs for you?
>>>>>>>>>
>>>>>>>>> On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
>>>>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>>> > Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
>>>>>>>>> > Copyright (C) 2005-2015 Red Hat, Inc. and others
>>>>>>>>> > This is free software; see the source for copying conditions.
>>>>>>>>> > enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>>>>>>>> >
>>>>>>>>> > On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
>>>>>>>>> > <danielmeirdoron@gmail.com> wrote:
>>>>>>>>> >> BTW kernel version: 2.6.32-431.el6.x86_64
>>>>>>>>> >>
>>>>>>>>> >> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>>>>>>>>> >> <danielmeirdoron@gmail.com> wrote:
>>>>>>>>> >>> if you mean that the kernel version does not get upgraded, then yes.
>>>>>>>>> >>>
>>>>>>>>> >>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>>>>>>>>> >>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>>> >>>> Sorry, I do not know what you mean by that...
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com>
>>>>>>>>> >>>> wrote:
>>>>>>>>> >>>>> Do you have the kernel version pinned?
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady
>>>>>>>>> >>>>> <arkady.miasnikov@gmail.com> wrote:
>>>>>>>>> >>>>>> Please send the stap script
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>>>>>>> >>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>>> >>>>>>> Hi,
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> I built my module to probe a few system calls and functions.
>>>>>>>>> >>>>>>> Nothing
>>>>>>>>> >>>>>>> fancy, no big work in it, just prints some data. Then I wrote a
>>>>>>>>> >>>>>>> simple
>>>>>>>>> >>>>>>> python script to read data directly off the
>>>>>>>>> >>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course
>>>>>>>>> >>>>>>> running as
>>>>>>>>> >>>>>>> sudo)
>>>>>>>>> >>>>>>> I run stap in Flight recorder mode.
>>>>>>>>> >>>>>>> After a few hours (varies from 8 to 20) I stop getting any output
>>>>>>>>> >>>>>>> from
>>>>>>>>> >>>>>>> the module...
>>>>>>>>> >>>>>>> Any idea how to check why that is?
>>>>>>>>> >>>>>>> dmesg does not report anything unusual...
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Thanks.
>>>>>>>>> >>>>>>> Daniel.

Re: No output from trace file after a few hours

[Daniel Doron]

ok

On Thu, Jun 15, 2017 at 2:59 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> you can also try  --suppress-time-limits
>
> On Thu, Jun 15, 2017 at 2:32 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> One possible reason is a system update which caused lot of probes to trigger.
>>
>> On Thu, Jun 15, 2017 at 2:13 PM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
>>> I will try that.
>>> BTW, this also just happened after almost 24 hours on kernel 4.11.3-1-ARCH
>>>
>>> On Thu, Jun 15, 2017 at 2:05 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>> This works fine
>>>>
>>>> sudo stap -D MAXSKIPPED=0 -D MAXTRYLOCK=1000 -D TRYLOCKDELAY=100  -F
>>>> -o ./test.log -S 1,2  test.stp
>>>>
>>>> On Thu, Jun 15, 2017 at 2:03 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>> STAP simply quits in a couple of seconds
>>>>>
>>>>> sudo stap -F -o ./test.log -S 1,2  test.stp
>>>>> 94089
>>>>> ps -ef | grep -v "grep" | grep  94089
>>>>>
>>>>> On Thu, Jun 15, 2017 at 1:53 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>>> Interesting. I have reproduced the problem by running two tight bash
>>>>>> loops simultaneously
>>>>>>
>>>>>> sudo stap -F -o ./test.log -S 1,2  test.stp
>>>>>> while [ 1 ];do ls > /dev/null;done &
>>>>>> while [ 1 ];do echo -en "\033[0;0H";ls -l | grep "test.log";echo -en
>>>>>> "\033[0;0H";done;
>>>>>>
>>>>>> Linux centos6 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC
>>>>>> 2016 x86_64 x86_64 x86_64 GNU/Linux
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 15, 2017 at 11:59 AM, Daniel Doron
>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>> On that specific kernel about 8-10 hours.
>>>>>>> On my machine with latest kernel, been running for 24 hours.
>>>>>>> I need to check other distros.
>>>>>>>
>>>>>>> sudo stap -v -F -m mymodule test.stp
>>>>>>>
>>>>>>> On Thu, Jun 15, 2017 at 11:26 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>>>>> Do you do something like
>>>>>>>>
>>>>>>>> sudo stap -F -o ./test.log -S 1,2  test.stp ?
>>>>>>>>
>>>>>>>> On Thu, Jun 15, 2017 at 11:25 AM, Daniel Doron
>>>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>>> On that specific kernel about 8-10 hours.
>>>>>>>>> On my machine with latest kernel, been running for 24 hours.
>>>>>>>>> I need to check other distros.
>>>>>>>>>
>>>>>>>>> On Jun 15, 2017 11:18, "Arkady" <arkady.miasnikov@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> How long time it runs for you?
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 15, 2017 at 10:38 AM, Daniel Doron
>>>>>>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>>>> > Systemtap translator/driver (version 2.9/0.164, rpm 2.9-7.el6)
>>>>>>>>>> > Copyright (C) 2005-2015 Red Hat, Inc. and others
>>>>>>>>>> > This is free software; see the source for copying conditions.
>>>>>>>>>> > enabled features: AVAHI LIBRPM LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>>>>>>>>> >
>>>>>>>>>> > On Thu, Jun 15, 2017 at 10:37 AM, Daniel Doron
>>>>>>>>>> > <danielmeirdoron@gmail.com> wrote:
>>>>>>>>>> >> BTW kernel version: 2.6.32-431.el6.x86_64
>>>>>>>>>> >>
>>>>>>>>>> >> On Thu, Jun 15, 2017 at 10:36 AM, Daniel Doron
>>>>>>>>>> >> <danielmeirdoron@gmail.com> wrote:
>>>>>>>>>> >>> if you mean that the kernel version does not get upgraded, then yes.
>>>>>>>>>> >>>
>>>>>>>>>> >>> On Thu, Jun 15, 2017 at 10:34 AM, Daniel Doron
>>>>>>>>>> >>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>>>> >>>> Sorry, I do not know what you mean by that...
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Thu, Jun 15, 2017 at 10:32 AM, Arkady <arkady.miasnikov@gmail.com>
>>>>>>>>>> >>>> wrote:
>>>>>>>>>> >>>>> Do you have the kernel version pinned?
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> On Thu, Jun 15, 2017 at 10:12 AM, Arkady
>>>>>>>>>> >>>>> <arkady.miasnikov@gmail.com> wrote:
>>>>>>>>>> >>>>>> Please send the stap script
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> On Thu, Jun 15, 2017 at 10:10 AM, Daniel Doron
>>>>>>>>>> >>>>>> <danielmeirdoron@gmail.com> wrote:
>>>>>>>>>> >>>>>>> Hi,
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> I built my module to probe a few system calls and functions.
>>>>>>>>>> >>>>>>> Nothing
>>>>>>>>>> >>>>>>> fancy, no big work in it, just prints some data. Then I wrote a
>>>>>>>>>> >>>>>>> simple
>>>>>>>>>> >>>>>>> python script to read data directly off the
>>>>>>>>>> >>>>>>> /sys/kernel/debug/systemtap/mymodule/trace0 file (of course
>>>>>>>>>> >>>>>>> running as
>>>>>>>>>> >>>>>>> sudo)
>>>>>>>>>> >>>>>>> I run stap in Flight recorder mode.
>>>>>>>>>> >>>>>>> After a few hours (varies from 8 to 20) I stop getting any output
>>>>>>>>>> >>>>>>> from
>>>>>>>>>> >>>>>>> the module...
>>>>>>>>>> >>>>>>> Any idea how to check why that is?
>>>>>>>>>> >>>>>>> dmesg does not report anything unusual...
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> Thanks.
>>>>>>>>>> >>>>>>> Daniel.