[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[penguin-kernel@i-love.sakura.ne.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |penguin-kernel@i-love.sakur
                   |                            |a.ne.jp

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/22155] New: kernel panic due to NULL vma_cache_p->f_path.dentry

[penguin-kernel@i-love.sakura.ne.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

            Bug ID: 22155
           Summary: kernel panic due to NULL vma_cache_p->f_path.dentry
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: penguin-kernel@i-love.sakura.ne.jp
  Target Milestone: ---

I encountered an oops with systemtap-3.1-3.el7.x86_64. The location was

  if (path->dentry->d_op && path->dentry->d_op->d_dname &&

in d_path(), which means that path->dentry == NULL for some reason.

Looking at __stp_call_mmap_callbacks_for_task() (runtime/linux/task_finder.c
or runtime/linux/task_finder2.c , which one is in use?), it counts number of
vma->vm_file != NULL entries in tsk->mm->mmap and allocates memory for such
entries and remembers the address of vma->vm_file->f_path like

  vma_cache_p->f_path = &(vma->vm_file->f_path);

with tsk->mm->mmap_sem held for read, and the address of f_path
(instead of f_path.dentry and f_path.mnt) is later passed to d_path()
after tsk->mm->mmap_sem is released.

Is tsk->mm->mmap_sem held for read sufficient for guaranteeing that

  // First find the number of file-based vmas.
  vma = mm->mmap;
  while (vma) {
    if (vma->vm_file)
      file_based_vmas++;
      vma = vma->vm_next;
  }

and

  vma = mm->mmap;
  vma_cache_p = vma_cache;
  while (vma) {
    if (vma->vm_file) {
      // Notice we're increasing the reference
      // count for 'f_path'.  This way it won't
      // get deleted from out under us.
      vma_cache_p->f_path = &(vma->vm_file->f_path);
      path_get(vma_cache_p->f_path);
      vma_cache_p->dentry = vma->vm_file->f_path.dentry;
      vma_cache_p->addr = vma->vm_start;
      vma_cache_p->length = vma->vm_end - vma->vm_start;
      vma_cache_p->offset = (vma->vm_pgoff << PAGE_SHIFT);
      vma_cache_p->vm_flags = vma->vm_flags;
      vma_cache_p++;
    }
    vma = vma->vm_next;
  }

fills all entries in vma_cache_p ?

Assuming that above is correct, although we took reference on
f_path.dentry and f_path.mnt via path_get(), what guarantees that
vma->vm_file->f_path is still valid (i.e. f_path.dentry and f_path.mnt
do not change) between tsk->mm->mmap_sem is released and d_path() is called?
Isn't there a race window that the memory region pointed by vma->vm_file
changes (i.e. vma->vm_file->f_path == { garbage, garbage }) ?

I feel that "struct vma_cache_t" needs to use "struct path" rather than
"struct path *".

----------
[    0.000000] Linux version 3.10.0-514.26.2.el7.x86_64
(builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
4.8.5-11) (GCC) ) #1 SMP Tue Jul 4 15:04:05 UTC 2017
(...snipped...)
[ 2670.046497] Kprobes globally unoptimized
[ 2670.047387] stap_c80c762b8a8bcff4b69879d41fae40ef_1_2980: loading
out-of-tree module taints kernel.
[ 2670.047830] stap_c80c762b8a8bcff4b69879d41fae40ef_1_2980: module
verification failed: signature and/or required key missing - tainting kernel
[ 2670.135913] stap_c80c762b8a8bcff4b69879d41fae40ef_1_2980: systemtap:
3.1/0.166, base: ffffffffa03eb000, memory:
214data/116text/241ctx/2063net/33348alloc kb, probes: 48
[ 3726.377000] stap_c80c762b8a8bcff4b69879d41fae40ef_1_4019: systemtap:
3.1/0.166, base: ffffffffa043f000, memory:
214data/116text/241ctx/2063net/33348alloc kb, probes: 48
[10281.381269] stap_c80c762b8a8bcff4b69879d41fae40ef_1_6247: systemtap:
3.1/0.166, base: ffffffffa03e6000, memory:
214data/116text/241ctx/2063net/33348alloc kb, probes: 48
[15632.200303] stap_c80c762b8a8bcff4b69879d41fae40ef_1_8026: systemtap:
3.1/0.166, base: ffffffffa043a000, memory:
214data/116text/241ctx/2063net/33348alloc kb, probes: 48
[16109.057836] stap_c80c762b8a8bcff4b69879d41fae40ef_1_8883: systemtap:
3.1/0.166, base: ffffffffa03e6000, memory:
214data/116text/241ctx/2063net/33348alloc kb, probes: 48
[16578.618229] stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881: systemtap:
3.1/0.166, base: ffffffffa043a000, memory:
214data/116text/241ctx/2063net/33348alloc kb, probes: 48
[81608.487805] BUG: unable to handle kernel NULL pointer dereference at
0000000000000060
[81608.487995] IP: [<ffffffff812168f8>] d_path+0x38/0x170
[81608.488145] PGD 5e1fd067 PUD 5deb7067 PMD 0 
[81608.488220] Oops: 0000 [#1] SMP 
[81608.488276] Modules linked in:
stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881(OE) vmw_vsock_vmci_transport vsock
intel_powerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon sg pcspkr shpchp
parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom
sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul
crct10dif_common vmwgfx drm_kms_helper crc32c_intel syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm serio_raw mptspi drm scsi_transport_spi mptscsih
vmxnet3 mptbase ata_piix libata i2c_core floppy fjes dm_mirror dm_region_hash
dm_log dm_mod [last unloaded: stap_c80c762b8a8bcff4b69879d41fae40ef_1_8883]
[81608.491292] CPU: 0 PID: 25868 Comm: java Tainted: G           OE 
------------   3.10.0-514.26.2.el7.x86_64 #1
[81608.494534] Hardware name: VMware, Inc. VMware Virtual Platform/440BX
Desktop Reference Platform, BIOS 6.00 04/14/2014
[81608.497591] task: ffff880137320000 ti: ffff880137da0000 task.ti:
ffff880137da0000
[81608.499174] RIP: 0010:[<ffffffff812168f8>]  [<ffffffff812168f8>]
d_path+0x38/0x170
[81608.500896] RSP: 0018:ffff880137da3d90  EFLAGS: 00010246
[81608.502863] RAX: ffff880137e0f000 RBX: ffff8800b709e310 RCX:
0000000000000000
[81608.504573] RDX: 0000000000001000 RSI: ffff880137e0e000 RDI:
0000000000000000
[81608.506223] RBP: ffff880137da3dd0 R08: 00007f475ad31000 R09:
0000000000001000
[81608.508052] R10: 00000000000000e4 R11: ffffea0004df8200 R12:
ffff880137e0e000
[81608.510484] R13: ffff88013722a010 R14: ffff880137e0efab R15:
ffff880137228c60
[81608.513263] FS:  00007f471583f700(0000) GS:ffff88013fc00000(0000)
knlGS:0000000000000000
[81608.516228] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[81608.519215] CR2: 0000000000000060 CR3: 000000006e07a000 CR4:
00000000000407f0
[81608.522110] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[81608.525089] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[81608.527954] Stack:
[81608.530743]  0000100008100073 ffff880137e0f000 00007f475ad31000
ffff880137320000
[81608.533765]  0000000052e3c232 ffff880137320000 ffff880137e0e000
ffff88013722a010
[81608.536778]  ffff880137da3e30 ffffffffa0453629 0000000000017000
0000000008100073
[81608.539867] Call Trace:
[81608.542637]  [<ffffffffa0453629>]
__stp_call_mmap_callbacks_for_task+0x169/0x240
[stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881]
[81608.548067]  [<ffffffffa045385c>]
__stp_utrace_task_finder_target_quiesce+0x15c/0x2b0
[stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881]
[81608.553490]  [<ffffffffa043ec7e>] start_callback.isra.48+0x7e/0x100
[stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881]
[81608.559129]  [<ffffffffa0442a09>] utrace_resume+0x109/0x410
[stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881]
[81608.564108]  [<ffffffff810ad1e7>] task_work_run+0xa7/0xe0
[81608.566703]  [<ffffffff8102ab22>] do_notify_resume+0x92/0xb0
[81608.569549]  [<ffffffff81697abd>] int_signal+0x12/0x17
[81608.571756] Code: 89 e5 41 55 41 54 53 48 89 fb 48 83 ec 28 48 8b 7f 08 48
89 44 24 08 89 54 24 04 65 48 8b 0c 25 28 00 00 00 48 89 4c 24 20 31 c9 <48> 8b
47 60 48 85 c0 74 2f 48 8b 40 40 48 85 c0 74 26 ff d0 48 
[81608.578430] RIP  [<ffffffff812168f8>] d_path+0x38/0x170
[81608.581039]  RSP <ffff880137da3d90>
[81608.583132] CR2: 0000000000000060
----------

----------
crash> bt -l
PID: 25868  TASK: ffff880137320000  CPU: 0   COMMAND: "java"
 #0 [ffff880137da3a20] machine_kexec at ffffffff81059beb
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/kernel/machine_kexec_64.c:
319
 #1 [ffff880137da3a80] __crash_kexec at ffffffff81105822
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/kernel/kexec.c:
1491
 #2 [ffff880137da3b50] crash_kexec at ffffffff81105910
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/include/asm/atomic.h:
38
 #3 [ffff880137da3b68] oops_end at ffffffff81690008
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/kernel/dumpstack.c:
225
 #4 [ffff880137da3b90] no_context at ffffffff8167fc96
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/mm/fault.c:
703
 #5 [ffff880137da3be0] __bad_area_nosemaphore at ffffffff8167fd2c
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/mm/fault.c:
782
 #6 [ffff880137da3c28] bad_area at ffffffff81680050
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/mm/fault.c:
811
 #7 [ffff880137da3c50] __do_page_fault at ffffffff81692f4f
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/mm/fault.c:
1164
 #8 [ffff880137da3cb0] do_page_fault at ffffffff81692ff5
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/mm/fault.c:
1237
 #9 [ffff880137da3ce0] page_fault at ffffffff8168f208
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/kernel/entry_64.S:
1316
    [exception RIP: d_path+56]
    RIP: ffffffff812168f8  RSP: ffff880137da3d90  RFLAGS: 00010246
    RAX: ffff880137e0f000  RBX: ffff8800b709e310  RCX: 0000000000000000
    RDX: 0000000000001000  RSI: ffff880137e0e000  RDI: 0000000000000000
    RBP: ffff880137da3dd0   R8: 00007f475ad31000   R9: 0000000000001000
    R10: 00000000000000e4  R11: ffffea0004df8200  R12: ffff880137e0e000
    R13: ffff88013722a010  R14: ffff880137e0efab  R15: ffff880137228c60
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/fs/dcache.c:
2940
#10 [ffff880137da3dd8] __stp_call_mmap_callbacks_for_task at ffffffffa0453629
[stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881]
#11 [ffff880137da3e38] __stp_utrace_task_finder_target_quiesce at
ffffffffa045385c [stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881]
#12 [ffff880137da3e70] start_callback at ffffffffa043ec7e
[stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881]
#13 [ffff880137da3ea8] utrace_resume at ffffffffa0442a09
[stap_c80c762b8a8bcff4b69879d41fae40ef_1_9881]
#14 [ffff880137da3f00] task_work_run at ffffffff810ad1e7
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/kernel/task_work.c:
89
#15 [ffff880137da3f30] do_notify_resume at ffffffff8102ab22
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/include/linux/tracehook.h:
196
#16 [ffff880137da3f50] int_signal at ffffffff81697abd
   
/usr/src/debug/kernel-3.10.0-514.26.2.el7/linux-3.10.0-514.26.2.el7.x86_64/arch/x86/kernel/entry_64.S:
620
    RIP: 00007f47752fd3dc  RSP: 00007f471583c6b0  RFLAGS: 00000293
    RAX: 0000000000000000  RBX: 00007f47200069e0  RCX: ffffffffffffffff
    RDX: 00000000ffffd9a8  RSI: 0000000000002658  RDI: 00007f4772ba8541
    RBP: 00007f471583c6d0   R8: 0000000000000000   R9: 000000000000000c
    R10: 00007f475d33b18f  R11: 0000000000000293  R12: 00007f471c058b40
    R13: 00007f471c059f50  R14: 00007f471c0582e0  R15: 00007f471c05b7e0
    ORIG_RAX: 000000000000003a  CS: 0033  SS: 002b
----------

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com

--- Comment #1 from David Smith <dsmith at redhat dot com> ---
(In reply to Tetsuo Handa from comment #0)
> I encountered an oops with systemtap-3.1-3.el7.x86_64. The location was
> 
>   if (path->dentry->d_op && path->dentry->d_op->d_dname &&
> 
> in d_path(), which means that path->dentry == NULL for some reason.
> 
> Looking at __stp_call_mmap_callbacks_for_task() (runtime/linux/task_finder.c
> or runtime/linux/task_finder2.c , which one is in use?),

Knowing whether you are using task_finder.c or task_finder2.c is based on your
kernel. Since you are using 3.10.0-514.26.2.el7.x86_64 which doesn't have
CONFIG_UTRACE, you'll be using task_finder2.c.

> it counts number of
> vma->vm_file != NULL entries in tsk->mm->mmap and allocates memory for such
> entries and remembers the address of vma->vm_file->f_path like
> 
>   vma_cache_p->f_path = &(vma->vm_file->f_path);
> 
> with tsk->mm->mmap_sem held for read, and the address of f_path
> (instead of f_path.dentry and f_path.mnt) is later passed to d_path()
> after tsk->mm->mmap_sem is released.
> 
> Is tsk->mm->mmap_sem held for read sufficient for guaranteeing that
> 
>   // First find the number of file-based vmas.
>   vma = mm->mmap;
>   while (vma) {
>     if (vma->vm_file)
>       file_based_vmas++;
>       vma = vma->vm_next;
>   }
> 
> and
> 
>   vma = mm->mmap;
>   vma_cache_p = vma_cache;
>   while (vma) {
>     if (vma->vm_file) {
>       // Notice we're increasing the reference
>       // count for 'f_path'.  This way it won't
>       // get deleted from out under us.
>       vma_cache_p->f_path = &(vma->vm_file->f_path);
>       path_get(vma_cache_p->f_path);
>       vma_cache_p->dentry = vma->vm_file->f_path.dentry;
>       vma_cache_p->addr = vma->vm_start;
>       vma_cache_p->length = vma->vm_end - vma->vm_start;
>       vma_cache_p->offset = (vma->vm_pgoff << PAGE_SHIFT);
>       vma_cache_p->vm_flags = vma->vm_flags;
>       vma_cache_p++;
>     }
>     vma = vma->vm_next;
>   }
> 
> fills all entries in vma_cache_p ?

We used to call get_task_mm() and mmput() in that function, but since mmput()
can sleep and that function can get called in atomic context, we had to remove
those calls.

But, when that function gets called, the task is stopped. So, hopefully the
task->mm won't be changing while the task is stopped.

> Assuming that above is correct, although we took reference on
> f_path.dentry and f_path.mnt via path_get(), what guarantees that
> vma->vm_file->f_path is still valid (i.e. f_path.dentry and f_path.mnt
> do not change) between tsk->mm->mmap_sem is released and d_path() is called?
> Isn't there a race window that the memory region pointed by vma->vm_file
> changes (i.e. vma->vm_file->f_path == { garbage, garbage }) ?

If path_get() isn't sufficient to keep the f_path.dentry and f_path.mnt valid,
the there is some generic bug in the kernel. That's kind of the point for a
reference-counted structure.

> I feel that "struct vma_cache_t" needs to use "struct path" rather than
> "struct path *".

Hmm, that's an interesting suggestion.

... console output deleted ...

Can you show us the script that caused this panic?

Is this panic intermittent or does it happen every time?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[penguin-kernel@i-love.sakura.ne.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

--- Comment #2 from Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> ---
> But, when that function gets called, the task is stopped.
> So, hopefully the task->mm won't be changing while the task is stopped.

Java is a multi-threaded process.
Are all threads stopped when that function gets called?

> Can you show us the script that caused this panic?

I don't have the script right now (I need to ask permission for disclosure).
But the script hooks Java methods using java().class().method() and file opens.
Guru mode is not used.

> Is this panic intermittent or does it happen every time?

So far only once, while running stability test for 8 hours.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

--- Comment #3 from David Smith <dsmith at redhat dot com> ---
(In reply to Tetsuo Handa from comment #2)
> > But, when that function gets called, the task is stopped.
> > So, hopefully the task->mm won't be changing while the task is stopped.
> 
> Java is a multi-threaded process.
> Are all threads stopped when that function gets called?

No, all threads aren't stopped. But, each userspace thread is represented by a
task structure in the kernel.

> > Can you show us the script that caused this panic?
> 
> I don't have the script right now (I need to ask permission for disclosure).
> But the script hooks Java methods using java().class().method() and file
> opens.
> Guru mode is not used.
> 
> > Is this panic intermittent or does it happen every time?
> 
> So far only once, while running stability test for 8 hours.

Unless you can get us a reproducer, debugging becomes quite difficult.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[penguin-kernel@i-love.sakura.ne.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

--- Comment #4 from Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> ---
(In reply to David Smith from comment #3)
> (In reply to Tetsuo Handa from comment #2)
> > > But, when that function gets called, the task is stopped.
> > > So, hopefully the task->mm won't be changing while the task is stopped.
> > 
> > Java is a multi-threaded process.
> > Are all threads stopped when that function gets called?
> 
> No, all threads aren't stopped. But, each userspace thread is represented by
> a task structure in the kernel.

Suppose process P1 has two threads T1 and T2, it is possible that this function
is called when T1 is stopped but T2 is running, isn't it?

You think T1->mm will not go away because T1 is stopped, that part is fine.

My question is, what prevents T2 from updating (or removing) vma->vm_file
after this function released mmap_sem? I think there is a possibility that
vma->vm_file is released (or vma->vm_file.dentry becomes NULL) before
&vma->vm_file
is dereferenced at d_path().

If the reason "struct vma_cache_t" saves

        struct dentry *dentry;
        unsigned long addr;
        unsigned long length;
        unsigned long offset;
        unsigned long vm_flags;

before releasing mmap_sem is that vma might change after releasing mmap_sem,
why is it safe to save the address of vma->vm_file like

        vma_cache_p->f_path = &(vma->vm_file->f_path);

? I think it is not safe unless we copy "struct path" like

        struct path f_path;

        vma_cache_p->f_path = vma->vm_file->f_path;
        path_get(&vma_cache_p->f_path);

and use like

        mmpath = d_path(&vma_cache_p->f_path, mmpath_buf,
                        PATH_MAX);
        path_put(&vma_cache_p->f_path);

.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[penguin-kernel@i-love.sakura.ne.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

--- Comment #5 from Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> ---
(In reply to David Smith from comment #3)
> Unless you can get us a reproducer, debugging becomes quite difficult.

Below is a reproducer written in C and a patch in order to make it easier to
test this problem. I don't know side effect of removing "tgt->mmap_events == 1"
check, but I think I succeeded to reproduce the problem because the backtrace
is same.

(1) Patch runtime/linux/task_finder2.c as below.

[root@localhost ~]# rpm -V systemtap-devel-3.1-3.el7.x86_64
S.5....T.    /usr/share/systemtap/runtime/linux/task_finder2.c
[root@localhost ~]# diff -u
/usr/share/systemtap/runtime/linux/task_finder2.c.orig
/usr/share/systemtap/runtime/linux/task_finder2.c
--- /usr/share/systemtap/runtime/linux/task_finder2.c.orig
+++ /usr/share/systemtap/runtime/linux/task_finder2.c
@@ -1210,6 +1210,9 @@
        if (vma_cache) {
                int i;

+               printk(KERN_INFO "Injecting mdelay() at
__stp_call_mmap_callbacks_for_task()\n");
+               mdelay(10);
+
                // Loop over our cached information and make callbacks
                // based on it.
                vma_cache_p = vma_cache;
@@ -1361,7 +1364,7 @@
                   group leader, don't bother inform map callback
                   clients about its memory map, since they will
                   simply duplicate each other. */
-               if (tgt->mmap_events == 1 && tsk->tgid == tsk->pid) {
+               if (/*tgt->mmap_events == 1 && */ tsk->tgid == tsk->pid) {
                        __stp_call_mmap_callbacks_for_task(tgt, tsk);
                }
        }

(2) Compile a.out and run SystemTap probe as below.

[root@localhost ~]# gcc -Wall -O0 -g -o a.out -x c - << "EOF"
#define _GNU_SOURCE
#include <sys/stat.h>
#include <fcntl.h>
#include <sched.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/poll.h>
#include <sys/mman.h>

static int thread_func(void *unused)
{
        sleep(1);
        const int fd = open("/proc/self/exe", O_RDONLY);
        void *ptr = mmap(NULL, 4096, PROT_READ, MAP_PRIVATE, fd, 0);
        poll(NULL, 0, 1);
        munmap(ptr, 4096);
        close(fd);
        return 0;
}

static void leader_func(void)
{
        poll(NULL, 0, 1);
}

int main(int argc, char *argv[])
{
        int i;
        for (i = 0; i < 1024; i++)
                clone(thread_func, malloc(4096) + 4096, CLONE_VM, NULL);
        while (1)
                leader_func();
        return 0;
}
EOF
[root@localhost ~]# stap -e 'global count; probe
process("a.out").function("leader_func") { printf("leader_func %u\n", ++count);
}'

(3) Run ./a.out from another shell. If the problem do not reproduce, terminate
it
    with Ctrl-C and try again. The problem should be reproduced within a few
trials.

----------
[  647.845554] BUG: unable to handle kernel NULL pointer dereference at
0000000000000060
[  647.847323] IP: [<ffffffff81219b04>] d_path+0x34/0x150
[  647.848538] PGD 29ff067 PUD 29fe067 PMD 0
[  647.849618] Oops: 0000 [#1] SMP
[  647.850496] Modules linked in:
stap_b9cc31d25bb1799d76699c0d4e9b27d_10606(OE) nf_conntrack_netbios_ns
nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT
nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge
stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6
ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security
iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
vmw_vsock_vmci_transport vsock coretemp iosf_mbi crc32_pclmul
ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev
vmw_balloon cryptd nfit pcspkr joydev sg parport_pc parport libnvdimm shpchp
vmw_vmci i2c_piix4 ip_tables xfs libcrc32c
[  647.878391]  sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic
pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper
ahci libahci syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm
ata_piix drm mptspi e1000 scsi_transport_spi mptscsih libata mptbase i2c_core
[last unloaded: stap_234fabfb03ce8e31588682ff44f0b625_8281]
[  647.890978] CPU: 0 PID: 11136 Comm: a.out Tainted: G           OE 
------------   3.10.0-693.2.2.el7.x86_64 #1
[  647.897707] Hardware name: VMware, Inc. VMware Virtual Platform/440BX
Desktop Reference Platform, BIOS 6.00 07/02/2015
[  647.900578] task: ffff880002e95ee0 ti: ffff880002bd8000 task.ti:
ffff880002bd8000
[  647.902828] RIP: 0010:[<ffffffff81219b04>]  [<ffffffff81219b04>]
d_path+0x34/0x150
[  647.909394] RSP: 0018:ffff880002bdbd90  EFLAGS: 00010246
[  647.911286] RAX: ffff88007704c000 RBX: ffff880003285a10 RCX:
ffff880035e5bc80
[  647.917906] RDX: 0000000000001000 RSI: ffff88007704b000 RDI:
0000000000000000
[  647.920215] RBP: ffff880002bdbdc0 R08: 00007f5fce020000 R09:
0000000000021000
[  647.922530] R10: 00000000000008a1 R11: 0000000000aaaaaa R12:
ffff880002bfda40
[  647.929254] R13: ffff880002e95ee0 R14: ffff880002bfd980 R15:
ffff88007704bfea
[  647.931577] FS:  00007f5fce22f740(0000) GS:ffff88007b600000(0000)
knlGS:0000000000000000
[  647.938604] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  647.940735] CR2: 0000000000000060 CR3: 000000007830c000 CR4:
00000000001407f0
[  647.943176] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  647.950390] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[  647.952865] Stack:
[  647.958860]  00001000ce020000 ffff88007704c000 ffff880002bfda40
ffff880002e95ee0
[  647.961448]  000000007e98da24 ffff88007704b000 ffff880002bdbe20
ffffffffc050f06b
[  647.968495]  0000000000000000 0000000008000875 ffff880002bfd800
ffffffffc0516a88
[  647.971015] Call Trace:
[  647.972424]  [<ffffffffc050f06b>]
__stp_call_mmap_callbacks_for_task+0x1bb/0x270
[stap_b9cc31d25bb1799d76699c0d4e9b27d_10606]
[  647.979902]  [<ffffffffc050f275>]
__stp_utrace_task_finder_target_quiesce+0x155/0x2a0
[stap_b9cc31d25bb1799d76699c0d4e9b27d_10606]
[  647.983069]  [<ffffffffc050940e>] start_callback.isra.40+0x7e/0x100
[stap_b9cc31d25bb1799d76699c0d4e9b27d_10606]
[  647.990284]  [<ffffffffc050af09>] utrace_resume+0x109/0x390
[stap_b9cc31d25bb1799d76699c0d4e9b27d_10606]
[  647.993002]  [<ffffffff810ad247>] task_work_run+0xa7/0xf0
[  647.999038]  [<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[  648.000915]  [<ffffffff816b52bd>] int_signal+0x12/0x17
[  648.002663] Code: 48 01 f0 48 89 e5 53 48 89 fb 48 83 ec 28 65 48 8b 3c 25
28 00 00 00 48 89 7c 24 20 31 ff 48 8b 7b 08 89 54 24 04 48 89 44 24 08 <48> 8b
4f 60 48 85 c9 74 43 48 8b 49 40 48 85 c9 74 3a 48 3b 7f
[  648.018302] RIP  [<ffffffff81219b04>] d_path+0x34/0x150
[  648.024164]  RSP <ffff880002bdbd90>
[  648.029608] CR2: 0000000000000060
----------

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[penguin-kernel@i-love.sakura.ne.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

--- Comment #6 from Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> ---
F.Y.I. KASAN reports that this is use-after-free bug.

[  103.454147] Injecting mdelay() at __stp_call_mmap_callbacks_for_task()
[  103.519926]
==================================================================
[  103.519940] BUG: KASAN: use-after-free in d_path+0xb0/0x420
[  103.519943] Read of size 8 at addr ffff880067e09fd8 by task a.out/3147
[  103.519945] 
[  103.519949] CPU: 2 PID: 3147 Comm: a.out Tainted: G           O    4.13.0
#150
[  103.519952] Hardware name: VMware, Inc. VMware Virtual Platform/440BX
Desktop Reference Platform, BIOS 6.00 07/02/2015
[  103.519953] Call Trace:
[  103.519959]  dump_stack+0x68/0x99
[  103.519965]  print_address_description+0x5d/0x270
[  103.519971]  kasan_report+0x25a/0x370
[  103.519974]  ? d_path+0xb0/0x420
[  103.519981]  __asan_load8+0x54/0x90
[  103.519984]  d_path+0xb0/0x420
[  103.519988]  ? lock_downgrade+0x2c0/0x2c0
[  103.519994]  ? prepend_path+0x790/0x790
[  103.520010]  ? __stp_call_mmap_callbacks.part.40+0xd0/0x100
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520026]  __stp_call_mmap_callbacks_for_task+0x2e6/0x380
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520043]  __stp_utrace_task_finder_target_quiesce+0x32d/0x380
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520055]  ? __stp_tf_quiesce_worker+0x1e0/0x1e0
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520063]  start_callback.isra.18+0xd7/0x180
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520077]  utrace_resume+0x20b/0x580
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520079]  ? __lock_is_held+0x2d/0x100
[  103.520090]  ? utrace_control+0x470/0x470
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520102]  task_work_run+0xa7/0xe0
[  103.520110]  exit_to_usermode_loop+0xaa/0xb0
[  103.520116]  syscall_return_slowpath+0xc3/0x110
[  103.520121]  ret_from_fork+0x15/0x40
[  103.520124] RIP: 0033:0x7ff1a0764311
[  103.520126] RSP: 002b:00000000022bd320 EFLAGS: 00000202 ORIG_RAX:
0000000000000038
[  103.520129] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
00007ff1a0764311
[  103.520131] RDX: 0000000000001000 RSI: 00000000022bd320 RDI:
0000000000000100
[  103.520133] RBP: 00007ffde649b740 R08: 0000000000001000 R09:
0000000000001000
[  103.520135] R10: 00007ffde649b828 R11: 0000000000000202 R12:
0000000000400640
[  103.520137] R13: 00007ffde649b820 R14: 0000000000000000 R15:
0000000000000000
[  103.520150] 
[  103.520152] Allocated by task 2842:
[  103.520157]  save_stack_trace+0x16/0x20
[  103.520159]  kasan_kmalloc+0xee/0x190
[  103.520161]  kasan_slab_alloc+0x12/0x20
[  103.520165]  kmem_cache_alloc+0xf8/0x370
[  103.520169]  get_empty_filp+0x68/0x260
[  103.520172]  path_openat+0xb8/0x1180
[  103.520175]  do_filp_open+0x121/0x1c0
[  103.520177]  do_sys_open+0x1a4/0x2a0
[  103.520180]  SyS_open+0x19/0x20
[  103.520182]  do_syscall_64+0xe5/0x250
[  103.520185]  return_from_SYSCALL_64+0x0/0x7a
[  103.520186] 
[  103.520187] Freed by task 2841:
[  103.520190]  save_stack_trace+0x16/0x20
[  103.520192]  kasan_slab_free+0xaf/0x190
[  103.520195]  kmem_cache_free+0x9c/0x330
[  103.520197]  file_free_rcu+0x46/0x70
[  103.520201]  rcu_process_callbacks+0x425/0xae0
[  103.520204]  __do_softirq+0x104/0x5e3
[  103.520205] 
[  103.520207] The buggy address belongs to the object at ffff880067e09fc0
[  103.520207]  which belongs to the cache filp of size 456
[  103.520209] The buggy address is located 24 bytes inside of
[  103.520209]  456-byte region [ffff880067e09fc0, ffff880067e0a188)
[  103.520211] The buggy address belongs to the page:
[  103.520214] page:ffffea00019f8200 count:1 mapcount:0 mapping:         
(null) index:0xffff880067e0b4c0 compound_mapcount: 0
[  103.520219] flags: 0x1ffe00000008100(slab|head)
[  103.520223] raw: 01ffe00000008100 0000000000000000 ffff880067e0b4c0
000000010024001d
[  103.520227] raw: ffffea000196f620 ffff88006a81c2c0 ffff88006cd8de80
0000000000000000
[  103.520228] page dumped because: kasan: bad access detected
[  103.520229] 
[  103.520230] Memory state around the buggy address:
[  103.520233]  ffff880067e09e80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
fc
[  103.520235]  ffff880067e09f00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
fc
[  103.520237] >ffff880067e09f80: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb
fb
[  103.520239]                                                     ^
[  103.520241]  ffff880067e0a000: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
fb
[  103.520243]  ffff880067e0a080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
fb
[  103.520244]
==================================================================
[  103.520246] Disabling lock debugging due to kernel taint
[  103.520256] BUG: unable to handle kernel NULL pointer dereference at
00000000000000c0
[  103.520260] IP: d_path+0xc1/0x420
[  103.520262] PGD 69187067 
[  103.520262] P4D 69187067 
[  103.520263] PUD 69186067 
[  103.520264] PMD 0 
[  103.520265] 
[  103.520267] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN
[  103.520269] Modules linked in: stap_0bb496b0c0998e4178af5110538c49d_2830(O)
coretemp pcspkr sg vmw_vmci i2c_piix4 shpchp sd_mod ata_generic pata_acpi
serio_raw vmwgfx drm_kms_helper syscopyarea ahci sysfillrect libahci sysimgblt
fb_sys_fops e1000 ttm drm ata_piix mptspi scsi_transport_spi mptscsih mptbase
i2c_core libata ipv6
[  103.520295] CPU: 2 PID: 3147 Comm: a.out Tainted: G    B      O    4.13.0
#150
[  103.520297] Hardware name: VMware, Inc. VMware Virtual Platform/440BX
Desktop Reference Platform, BIOS 6.00 07/02/2015
[  103.520298] task: ffff880061214b40 task.stack: ffff880061230000
[  103.520300] RIP: 0010:d_path+0xc1/0x420
[  103.520302] RSP: 0018:ffff880061237be8 EFLAGS: 00010282
[  103.520304] RAX: 0000000000000000 RBX: 1ffff1000c246f81 RCX:
ffffffff8136c701
[  103.520305] RDX: 0000000000000000 RSI: 0000000000000008 RDI:
00000000000000c0
[  103.520306] RBP: ffff880061237cd8 R08: 0000000000000000 R09:
0000000000000000
[  103.520308] R10: fbfbfbfbfbfbfbfb R11: fffffbfff067fc25 R12:
ffff880067e09fd0
[  103.520309] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000001000
[  103.520311] FS:  00007ff1a0c3e740(0000) GS:ffff88006d280000(0000)
knlGS:0000000000000000
[  103.520317] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  103.520319] CR2: 00000000000000c0 CR3: 000000006c4d4000 CR4:
00000000001406e0
[  103.520320] Call Trace:
[  103.520323]  ? lock_downgrade+0x2c0/0x2c0
[  103.520327]  ? prepend_path+0x790/0x790
[  103.520338]  ? __stp_call_mmap_callbacks.part.40+0xd0/0x100
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520348]  __stp_call_mmap_callbacks_for_task+0x2e6/0x380
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520360]  __stp_utrace_task_finder_target_quiesce+0x32d/0x380
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520369]  ? __stp_tf_quiesce_worker+0x1e0/0x1e0
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520376]  start_callback.isra.18+0xd7/0x180
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520385]  utrace_resume+0x20b/0x580
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520387]  ? __lock_is_held+0x2d/0x100
[  103.520396]  ? utrace_control+0x470/0x470
[stap_0bb496b0c0998e4178af5110538c49d_2830]
[  103.520402]  task_work_run+0xa7/0xe0
[  103.520406]  exit_to_usermode_loop+0xaa/0xb0
[  103.520410]  syscall_return_slowpath+0xc3/0x110
[  103.520413]  ret_from_fork+0x15/0x40
[  103.520414] RIP: 0033:0x7ff1a0764311
[  103.520416] RSP: 002b:00000000022bd320 EFLAGS: 00000202 ORIG_RAX:
0000000000000038
[  103.520418] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
00007ff1a0764311
[  103.520419] RDX: 0000000000001000 RSI: 00000000022bd320 RDI:
0000000000000100
[  103.520421] RBP: 00007ffde649b740 R08: 0000000000001000 R09:
0000000000001000
[  103.520422] R10: 00007ffde649b828 R11: 0000000000000202 R12:
0000000000400640
[  103.520423] R13: 00007ffde649b820 R14: 0000000000000000 R15:
0000000000000000
[  103.520429] Code: 48 89 85 50 ff ff ff 48 8d 47 08 48 89 c7 48 89 85 18 ff
ff ff e8 40 96 f9 ff 4d 8b 74 24 08 49 8d be c0 00 00 00 e8 2f 96 f9 ff <4d> 8b
ae c0 00 00 00 4d 85 ed 0f 84 9c 00 00 00 49 8d 7d 48 e8 
[  103.520471] RIP: d_path+0xc1/0x420 RSP: ffff880061237be8
[  103.520472] CR2: 00000000000000c0
[  103.520474] ---[ end trace 1cb14de0bcaa41fd ]---

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[penguin-kernel@i-love.sakura.ne.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

--- Comment #7 from Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> ---
Created attachment 10432
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10432&action=edit
A patch that fixes this problem

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/22155] kernel panic due to NULL vma_cache_p->f_path.dentry

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22155

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED

--- Comment #8 from David Smith <dsmith at redhat dot com> ---
I ran the testsuite with your patch and didn't see any issues. Checked in as
commit 589859d30.

-- 
You are receiving this mail because:
You are the assignee for the bug.