[PATCH 0/5] Synchronize nd_syscalls.stp with syscalls.stp

[PATCH 0/5] Synchronize nd_syscalls.stp with syscalls.stp

[Przemyslaw Pawelczyk]

Hi,

Here are a few patches for nd_syscalls.stp to sync it with syscalls.stp.

Fourth patch is problematic. From a theoretical viewpoint it isn't flawed.
Though applying it causes my strace-like script almost inoperative.
http://research.pawelczyk.it/systemtap/scripts/strace-nd.stp

Typical use case:
./strace-nd.stp -vc /bin/ls

It hangs after reporting many probe registration errors (rc -22), the last
one is kprobe.function("SyS_epoll_create").return!
After typing Ctrl-C script is stopped, but no data was collected and
process /bin/ls probably wasn't started -- there is no output from it.

Other scenario:
./strace-nd.stp -v

Now only fstat hits the probe.

Any ideas where is the source of problem?

Regards.

Przemyslaw Pawelczyk (5):
  Unify formatting of nd_syscalls.stp.
  Remove return probes for exit[_group] in nd_syscalls.stp.
  Add missing probe points in nd_syscalls.stp.
  Fix nd_syscalls.stp for architectures using SYSCALL_WRAPPERS.
  Uncomment 'name' variable in nd_syscall.lseek probe point.

 tapset/nd_syscalls.stp | 1207 ++++++++++++++++++++++++++++++++----------------
 1 files changed, 809 insertions(+), 398 deletions(-)

Re: [PATCH 0/5] Synchronize nd_syscalls.stp with syscalls.stp

[Josh Stone]

On 05/22/2009 08:46 AM, Przemyslaw Pawelczyk wrote:
> Hi,
> 
> Here are a few patches for nd_syscalls.stp to sync it with syscalls.stp.
> 
> Fourth patch is problematic. From a theoretical viewpoint it isn't flawed.
> Though applying it causes my strace-like script almost inoperative.
> http://research.pawelczyk.it/systemtap/scripts/strace-nd.stp
> 
> Typical use case:
> ./strace-nd.stp -vc /bin/ls
> 
> It hangs after reporting many probe registration errors (rc -22), the last
> one is kprobe.function("SyS_epoll_create").return!
> After typing Ctrl-C script is stopped, but no data was collected and
> process /bin/ls probably wasn't started -- there is no output from it.
> 
> Other scenario:
> ./strace-nd.stp -v
> 
> Now only fstat hits the probe.
> 
> Any ideas where is the source of problem?

kprobe.function can't resolve whether the function actually exists until
runtime.  So when the translator sees a kprobe.function("foo")!, we
assume that it will work, and so the rest of the alternatives aren't
tried.  Notice you have:

> +probe nd_syscall.accept = kprobe.function("SyS_accept") !,
> +                          kprobe.function("sys_accept") ?

and

>  probe nd_syscall.fstat = kprobe.function("sys_fstat") ?,
> +                         kprobe.function("SyS_fstat64") ?,
>                           kprobe.function("sys_fstat64") ?,
>                           kprobe.function("sys32_fstat64") ?,
> +                         kprobe.function("SyS_newfstat") ?,
>                           kprobe.function("sys_newfstat") ?,
>                           kprobe.function("sys_oabi_fstat64") ?,
>                           kprobe.function("compat_sys_newfstat") ?

So for nd_syscall.accept, the translator will only output a probe on
SyS_accept, since it has a '!', but for nd_syscall.fstat the translator
will output them all.

Therefore, using '!' on kprobe.function is currently useless, since the
translator always considers them resolved.  For the same reason, the '?'
isn't really doing anything here either.

We could try to suppress the registration warning at runtime for probes
marked with a '?' though.  Then you could just mark all of these as
optional, and while every one would be attempted, the user won't be
bothered with the failures.

Josh

Re: [PATCH 0/5] Synchronize nd_syscalls.stp with syscalls.stp

[Przemysław Pawełczyk]

On Fri, May 22, 2009 at 19:05, Josh Stone <jistone@redhat.com> wrote:
> On 05/22/2009 08:46 AM, Przemyslaw Pawelczyk wrote:

>> Here are a few patches for nd_syscalls.stp to sync it with syscalls.stp.
>>
>> Fourth patch is problematic. From a theoretical viewpoint it isn't flawed.
>> Though applying it causes my strace-like script almost inoperative.
>> http://research.pawelczyk.it/systemtap/scripts/strace-nd.stp
>>
>> Typical use case:
>> ./strace-nd.stp -vc /bin/ls
>>
>> It hangs after reporting many probe registration errors (rc -22), the last
>> one is kprobe.function("SyS_epoll_create").return!
>> After typing Ctrl-C script is stopped, but no data was collected and
>> process /bin/ls probably wasn't started -- there is no output from it.
>>
>> Other scenario:
>> ./strace-nd.stp -v
>>
>> Now only fstat hits the probe.

> kprobe.function can't resolve whether the function actually exists until
> runtime.  So when the translator sees a kprobe.function("foo")!, we
> assume that it will work, and so the rest of the alternatives aren't
> tried.  Notice you have:
>
>> +probe nd_syscall.accept = kprobe.function("SyS_accept") !,
>> +                          kprobe.function("sys_accept") ?
>
> and
>
>>  probe nd_syscall.fstat = kprobe.function("sys_fstat") ?,
>> +                         kprobe.function("SyS_fstat64") ?,
>>                           kprobe.function("sys_fstat64") ?,
>>                           kprobe.function("sys32_fstat64") ?,
>> +                         kprobe.function("SyS_newfstat") ?,
>>                           kprobe.function("sys_newfstat") ?,
>>                           kprobe.function("sys_oabi_fstat64") ?,
>>                           kprobe.function("compat_sys_newfstat") ?
>
> So for nd_syscall.accept, the translator will only output a probe on
> SyS_accept, since it has a '!', but for nd_syscall.fstat the translator
> will output them all.
>
> Therefore, using '!' on kprobe.function is currently useless, since the
> translator always considers them resolved.  For the same reason, the '?'
> isn't really doing anything here either.

I sent another version of fourth patch removing sufficiency of new
probe points, but problem still exists in my first use case.

> Josh

Regards.

-- 
Przemysław Pawełczyk

Re: [PATCH 0/5] Synchronize nd_syscalls.stp with syscalls.stp

[Josh Stone]

On 05/22/2009 08:46 AM, Przemyslaw Pawelczyk wrote:
> Hi,
>
> Here are a few patches for nd_syscalls.stp to sync it with syscalls.stp.
> [...]
>    Unify formatting of nd_syscalls.stp.
>    Remove return probes for exit[_group] in nd_syscalls.stp.
>    Add missing probe points in nd_syscalls.stp.
>    Fix nd_syscalls.stp for architectures using SYSCALL_WRAPPERS.
>    Uncomment 'name' variable in nd_syscall.lseek probe point.

As we discussed on IRC, I have committed patches 1, 2, 3, and 5.  We'll 
wait on the SYSCALL_WRAPPERS addition until bugzillas 10189 and/or 10190 
are resolved.

Josh

Re: [PATCH 0/5] Synchronize nd_syscalls.stp with syscalls.stp

[Josh Stone]

On 05/22/2009 03:51 PM, Josh Stone wrote:
> On 05/22/2009 08:46 AM, Przemyslaw Pawelczyk wrote:
>> Hi,
>>
>> Here are a few patches for nd_syscalls.stp to sync it with syscalls.stp.
>> [...]
>>     Unify formatting of nd_syscalls.stp.
>>     Remove return probes for exit[_group] in nd_syscalls.stp.
>>     Add missing probe points in nd_syscalls.stp.
>>     Fix nd_syscalls.stp for architectures using SYSCALL_WRAPPERS.
>>     Uncomment 'name' variable in nd_syscall.lseek probe point.
>
> As we discussed on IRC, I have committed patches 1, 2, 3, and 5.  We'll
> wait on the SYSCALL_WRAPPERS addition until bugzillas 10189 and/or 10190
> are resolved.

I committed a fix for PR10190, so now an optional probe won't print a 
registration WARNING.  However, I think we need another iteration on the 
SYSCALL_WRAPPERS patch.

> +probe nd_syscall.access = kprobe.function("SyS_access") ?,
> +                          kprobe.function("sys_access")

On a kernel without wrappers, SyS_access will fail and be silently 
ignored, and sys_access will be the only thing probed.  (good)

On a kernel with wrappers, SyS_access will be probed, and sys_access 
will fail loudly.  (not good)

Unfortunately, I think the only way around this is to mark everything 
with '?'.

Josh