[Bug nptl/10184] New: setuid functions can stall pthread exit code

[Bug nptl/10184] New: setuid functions can stall pthread exit code

[samandbernie at guarana dot org]

It appears that calling any of the setuid functions from one thread while
another thread is exiting can sometimes cause the exiting thread to get stuck.
The stuck thread is not visible to gdb but does appear in the output of ps. If
another thread is trying to join the stuck thread, it will wait forever.

It can become stuck either waiting for a futex wake or busy looping. In both
cases the location is in thread_start, from pthread_create.c (around line 388 in
the git trunk code). As follows:

      do
	lll_futex_wait (&pd->setxid_futex, 0, LLL_PRIVATE);
      while (pd->cancelhandling & SETXID_BITMASK);

>From our investigation, it happens with a tgkill from setxid_signal_thread
fails. This causes the SETDIX_BIT to be set in the target thread, but no signal
is sent and no other thread is actually waiting on it. It seems like a naive fix
would be to change the last statement in that function from:

  if (!INTERNAL_SYSCALL_ERROR_P (val, err))
    atomic_increment (&cmdp->cntr);

To:

  if (INTERNAL_SYSCALL_ERROR_P (val, err))
    t->cancelhandling &= ~SETXID_BITMASK;
  else
    atomic_increment (&cmdp->cntr);

This change appears to correct the problem on our machines.

On our machines (Ubuntu 9.04 with 2 or 4 way SMP) the following program
replicates the problem within a few minutes (just run it and watch for the
output to change to a continuous stream of the letter u):

#include <sys/types.h>
#include <unistd.h>
#include <pthread.h>
#include <stdio.h>

void *noop(void *arg) {
        usleep(rand() % 10000);
        return 0;
}

void *spawner(void *arg) {
        pthread_t t;

        for (;;) {
                fprintf(stderr, "c");
                pthread_create(&t, 0, noop, 0);
                fprintf(stderr, "j");
                pthread_join(t, 0);
        }
}

int main() {
        pthread_t spawner_id;

        pthread_create(&spawner_id, 0, spawner, 0);
        for(;;) {
                fprintf(stderr, "u");
                setuid(getuid());
                usleep(10000);
        }
        return 0;
}

We've run the test code on several RedHat machines, with the bug happening on
machines with glibc-2.3.4-2.41, glibc-2.5-18.el5_1.1 or glibc-2.5-24. For some
reason it takes several minutes to fail on 8-way SMP machines.

-- 
           Summary: setuid functions can stall pthread exit code
           Product: glibc
           Version: 2.9
            Status: NEW
          Severity: normal
          Priority: P2
         Component: nptl
        AssignedTo: drepper at redhat dot com
        ReportedBy: samandbernie at guarana dot org
                CC: glibc-bugs at sources dot redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=10184

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/10184] setuid functions can stall pthread exit code

[pasky at suse dot cz]

------- Additional Comments From pasky at suse dot cz  2009-05-22 13:35 -------
I remember seeing something like this, but it was fixed. Therefore, you need to
check if you can still reproduce your bug with latest glibc version, or ask your
vendor for support instead.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING


http://sourceware.org/bugzilla/show_bug.cgi?id=10184

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/10184] setuid functions can stall pthread exit code

[samandbernie at guarana dot org]

------- Additional Comments From samandbernie at guarana dot org  2009-05-22 23:18 -------
(In reply to comment #1)
> I remember seeing something like this, but it was fixed. Therefore, you need to
> check if you can still reproduce your bug with latest glibc version, or ask your
> vendor for support instead.

Before filing the bug we did check the bug tracker for similar bugs and also
that the latest git source was unchanged at the point we beleave to have the bug
(in allocatestack.c).

We have not yet built all of glibc from the trunk but will do so as soon as
possible (probably Monday) and re-test.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10184

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/10184] setuid functions can stall pthread exit code

[samandbernie at guarana dot org]

------- Additional Comments From samandbernie at guarana dot org  2009-05-25 01:43 -------
We've just confirmed that the issue is still present in the latest code from git
and that the same patch to setxid_signal_thread in allocatestack.c appears to
resolve it.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10184

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/10184] setuid functions can stall pthread exit code

[samandbernie at guarana dot org]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW


http://sourceware.org/bugzilla/show_bug.cgi?id=10184

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/10184] setuid functions can stall pthread exit code

[schwab at linux-m68k dot org]

------- Additional Comments From schwab at linux-m68k dot org  2009-10-29 16:53 -------


*** This bug has been marked as a duplicate of 3270 ***

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |DUPLICATE


http://sourceware.org/bugzilla/show_bug.cgi?id=10184

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/10184] setuid functions can stall pthread exit code

[soda at sra dot co dot jp]

------- Additional Comments From soda at sra dot co dot jp  2010-04-20 09:21 -------
It seems this bug still exists.

I think that the following source RPM includes the fix, is my guess right?
ftp://ftp.redhat.com/redhat/linux/enterprise/5Server/en/os/SRPMS/glibc-2.5-49.src.rpm

I built and installed the glibc-2.5-49 RPM on CentOS 5.4,
and run the test program in the description field of this bug report
under gdb, then the test program stucked nearly immediately as follows:

(gdb) run
Starting program: /tmp/setxid_test 
malloc: using debugging hooks
[Thread debugging using libthread_db enabled]
malloc: using debugging hooks
[New Thread 0x2acde66e98a0 (LWP 18528)]
[New Thread 0x41177940 (LWP 18529)]
uc[New Thread 0x41b78940 (LWP 18530)]
juc[Thread 0x41b78940 (LWP 18530) exited]
[New Thread 0x41b78940 (LWP 18531)]
jc[Thread 0x41b78940 (LWP 18531) exited]
[New Thread 0x41b78940 (LWP 18532)]
jc[Thread 0x41b78940 (LWP 18532) exited]
[New Thread 0x41b78940 (LWP 18533)]
ju[Thread 0x41b78940 (LWP 18533) exited]
c[New Thread 0x41b78940 (LWP 18534)]
ju[Thread 0x41b78940 (LWP 18534) exited]
c[New Thread 0x41b78940 (LWP 18535)]
j[Thread 0x41b78940 (LWP 18535) exited]
c[New Thread 0x41b78940 (LWP 18536)]
juc[Thread 0x41b78940 (LWP 18536) exited]
[New Thread 0x41b78940 (LWP 18537)]
j[Thread 0x41b78940 (LWP 18537) exited]
c[New Thread 0x41b78940 (LWP 18538)]
juc[Thread 0x41b78940 (LWP 18538) exited]
[New Thread 0x41b78940 (LWP 18539)]
jc[Thread 0x41b78940 (LWP 18539) exited]
[New Thread 0x41b78940 (LWP 18540)]
j[Thread 0x41b78940 (LWP 18540) exited]
c[New Thread 0x41b78940 (LWP 18541)]
ju[Thread 0x41b78940 (LWP 18541) exited]
c[New Thread 0x41b78940 (LWP 18542)]
ju[Thread 0x41b78940 (LWP 18542) exited]

(gdb) thr app all bt

Thread 2 (Thread 0x41177940 (LWP 18529)):
#0  0x000000319720d48e in __lll_lock_wait_private ()
   from /lib64/libpthread.so.0
#1  0x000000319720775d in _L_lock_2600 () from /lib64/libpthread.so.0
#2  0x00000031972064bb in __deallocate_stack (pd=<value optimized out>)
    at allocatestack.c:733
#3  0x0000003197207afa in pthread_join (threadid=<value optimized out>, 
    thread_return=<value optimized out>) at pthread_join.c:110
#4  0x00000000004007a2 in spawner (arg=0x0) at setxid_test.c:18
#5  0x000000319720673d in start_thread (arg=<value optimized out>)
    at pthread_create.c:301
#6  0x0000003196ad3d1d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2acde66e98a0 (LWP 18528)):
#0  __nptl_setxid (cmdp=<value optimized out>) at allocatestack.c:1096
#1  0x0000003196a9afed in __setuid (uid=<value optimized out>)
    at ../sysdeps/unix/sysv/linux/setuid.c:26
#2  0x00000000004007e1 in main () at setxid_test.c:28

It seems running this test program without gdb works fine, but I'm not
running the test program long enough.


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|DUPLICATE                   |


http://sourceware.org/bugzilla/show_bug.cgi?id=10184

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/10184] setuid functions can stall pthread exit code

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2010-04-20 12:06 -------
(In reply to comment #5)
> I think that the following source RPM includes the fix, is my guess right?
>
ftp://ftp.redhat.com/redhat/linux/enterprise/5Server/en/os/SRPMS/glibc-2.5-49.src.rpm

This is no support site for vendor versions.  Contact the vendor.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |FIXED


http://sourceware.org/bugzilla/show_bug.cgi?id=10184

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.