pthread_mutex_lock doesn't interrupt

pthread_mutex_lock doesn't interrupt

[James Cotton]

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

Hello, I've been trying to get the FreeRTOS Posix simulator to run on
Cygwin and have run into a few bugs in the signal handling/pthread
libraries and have isolated test cases

I believe when pthread_mutex_lock receives an interrupt it should run
it then resume trying to get a lock, the signal handler appears to
never run (at least as inferred by no output.  I tried running under
cygwin gdb but get uniformative backtraces).

Thanks,
James

[-- Attachment #2: test_case_4_mutex_lock.c --]
[-- Type: application/octet-stream, Size: 724 bytes --]

/**
 * small etst program whether signals between threads work as they should
 */

#include <pthread.h>
#include <signal.h>

static pthread_mutex_t Mutex = PTHREAD_MUTEX_INITIALIZER;

/**
 * actual test program
 */

void sighandler(int sig) {
	write(2,".",1);
	while(1);
	return;
}

void* threadstart(void* arg) {

	while (1) {
		pthread_mutex_lock(&Mutex);
	}
}

int main(char** argc, int argv) {

	pthread_t testthread1;	
	struct sigaction action;

	action.sa_handler=sighandler;
	action.sa_flags=0;
	sigfillset( &action.sa_mask );
	sigaction(SIGUSR1,&action,NULL);

	pthread_mutex_lock(&Mutex);
	pthread_create(&testthread1,NULL,threadstart,NULL);
	while (1) {
		write(2,"S",1);
		pthread_kill(testthread1,SIGUSR1);
	}
}

[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: pthread_mutex_lock doesn't interrupt

[Larry Hall (Cygwin)]

On 7/12/2010 12:56 PM, James Cotton wrote:
> Hello, I've been trying to get the FreeRTOS Posix simulator to run on
> Cygwin and have run into a few bugs in the signal handling/pthread
> libraries and have isolated test cases
>
> I believe when pthread_mutex_lock receives an interrupt it should run
> it then resume trying to get a lock, the signal handler appears to
> never run (at least as inferred by no output.  I tried running under
> cygwin gdb but get uniformative backtraces).

Did you build a debuggable Cygwin DLL or download one from
<http://cygwin.com/snapshots/>?

-- 
Larry Hall                              http://www.rfk.com
RFK Partners, Inc.                      (508) 893-9779 - RFK Office
216 Dalton Rd.                          (508) 893-9889 - FAX
Holliston, MA 01746

_____________________________________________________________________

A: Yes.
> Q: Are you sure?
>> A: Because it reverses the logical flow of conversation.
>>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Sending SIGUSR1 to thread in nanosleep causes segfault

[Corvus Corax]

[-- Attachment #1: Type: text/plain, Size: 3737 bytes --]

FYI

Just to update some details.

I am working with James Cotton on the OpenPilot project.
We are trying to port the POSIX simulation port of FreeRTOS to cygwin.

We run into issues with signal handling in cygwin 1.7.5 that are
currently blocking our development, since every method to suspend a
running thread via signals either causes:

- random segfaults within Cygwin DLL.
or
- failure to execute the signal handler.

For the sake of narrowing down the source of our problems,
we created 8 demo test cases.

Each of them works the same way:

1. A thread is spawned via pthread_create(), which then executes some
   code that is supposed to "sleep the thread" in background.
2. Signals are repeatedly sent to that thread using pthread_kill()
   which are supposed to be handled by a signal handler.
3. Each succesful call to the signal handler writes a dot (.) to stderr
   (using the write() system call to be thread safe)

Obviously what this is supposed to create is a neverending line of dots.
(Tested and working on Linux and Mac OS X on several architectures)

We believe this is a sign of a major flaw in cygwin signal handling and
may possibly affect many applications.
(Any application that uses signal handlers is potentially affected!)

SEGFAULTS usually occur only once in several thousand signals, so it is
a problem that won't appear on normal applications in a very frequent
way. However it is definitely reproducable!

All 8 test cases usually reach a stable state (segfault or freeze of
output) within less than 10 seconds (usually less than 1s) of run time.


The 8 test cases are as follows:

----

test_case_1_sleep.c

- The thread sleeps (literally) calling sleep() from <unistd.h>
Bug:
- After a couple thousand dots, the program ends with SIGSEGV in
  cygwin1.dll. The backtrace leads to garbage due to stack corruption!

----

test_case_2_nanosleep.c

- The thread sleeps calling nanosleep() from <time.h>
Bug:
- After a couple thousand dots, the program ends with SIGSEGV in
  cygwin1.dll. The backtrace leads to garbage due to stack corruption!

----

test_case_3_select.c

- The thread sleeps calling select() from <unistd.h>
Bug:
- Once select() is executed, cygwin blocks signals, failing to execute
  the signal handler. No dots are printed (There is a violation to
  POSIX specification. select() is supposed to return prematurely on
  receiving any signal, whether handled or not. This does not happen.)

----

test_case_4_mutex_lock.c

- The thread sleeps calling pthread_mutex_lock() from <pthread.h>
Bug:
- Once mutex_lock() is executed, cygwin blocks signals, failing to
  execute the signal handler. No dots are printed. This is a major
  issue for the porting of FreeRTOS, because it causes deadlocks we
  cannot resolve.

----

test_case_5_pselect.c

- The thread sleeps calling pselect() from <unistd.h>. Pselect allows
  to specify a signal mask of signals to be (or not to be) blocked
  during the pselect() call. The mask is deliberate set EMPTY to make
  pselect() interrupt on ALL signals.
Bug:
- Once pselect() is executed, cygwin blocks signals, failing to execute
  the signal handler. No dots are printed (pselect() completely
  ignores its signal mask argument !!!!! )

----

test_case_6_sched_yield.c

- The thread "sleeps" repeatedly calling sched_yield() from <pthread.h>
Bug:
- The signal handler is not executed in a reliable fashion.

----

test_case_7_busy_waiting.c
- The thread runs like crazy in a busy loop.
Bug:
- The signal handler is not executed in a reliable fashion.

----

test_case_8_io.c
- The thread "sleeps" by executing a blocking "read()" call from stdin
Bug:
- The signal handler is not executed in a reliable fashion.
- Occasional segfaults in cygwin dll.


[-- Attachment #2: test_case_1_sleep.c --]
[-- Type: text/x-c++src, Size: 577 bytes --]

/**
 * small test program whether signals between threads work as they should
 */

#include <unistd.h>
#include <pthread.h>
#include <signal.h>

void sighandler(int sig) {
	write(2,".",1);
	return;
}

void* threadstart(void* arg) {

	while (1) {
		sleep(1);
	}
}

int main(char** argc, int argv) {

	pthread_t testthread1;	
	struct sigaction action;

	action.sa_handler=sighandler;
	action.sa_flags=0;
	sigfillset( &action.sa_mask );
	sigaction(SIGUSR1,&action,NULL);

	pthread_create(&testthread1,NULL,threadstart,NULL);
	while (1) {
		pthread_kill(testthread1,SIGUSR1);
	}
}

[-- Attachment #3: test_case_2_nanosleep.c --]
[-- Type: text/x-c++src, Size: 665 bytes --]

/**
 * small test program whether signals between threads work as they should
 */

#include <time.h>
#include <pthread.h>
#include <signal.h>

void sighandler(int sig) {
	write(2,".",1);
	return;
}

void* threadstart(void* arg) {
struct timespec sleeptime;

	while (1) {
		sleeptime.tv_sec=1;
		sleeptime.tv_nsec=0;
		nanosleep(&sleeptime,NULL);
	}
}

int main(char** argc, int argv) {

	pthread_t testthread1;	
	struct sigaction action;

	action.sa_handler=sighandler;
	action.sa_flags=0;
	sigfillset( &action.sa_mask );
	sigaction(SIGUSR1,&action,NULL);

	pthread_create(&testthread1,NULL,threadstart,NULL);
	while (1) {
		pthread_kill(testthread1,SIGUSR1);
	}
}

[-- Attachment #4: test_case_3_select.c --]
[-- Type: text/x-c++src, Size: 720 bytes --]

/**
 * small test program whether signals between threads work as they should
 */

#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#include <pthread.h>
#include <signal.h>

void sighandler(int sig) {
	write(2,".",1);
	return;
}

void* threadstart(void* arg) {
struct timeval sleeptime;

	while (1) {
		sleeptime.tv_sec=1;
		sleeptime.tv_usec=0;
		select(0,NULL,NULL,NULL,&sleeptime);
	}
}

int main(char** argc, int argv) {

	pthread_t testthread1;	
	struct sigaction action;

	action.sa_handler=sighandler;
	action.sa_flags=0;
	sigfillset( &action.sa_mask );
	sigaction(SIGUSR1,&action,NULL);

	pthread_create(&testthread1,NULL,threadstart,NULL);
	while (1) {
		pthread_kill(testthread1,SIGUSR1);
	}
}

[-- Attachment #5: test_case_4_mutex_lock.c --]
[-- Type: text/x-c++src, Size: 695 bytes --]

/**
 * small etst program whether signals between threads work as they should
 */

#include <pthread.h>
#include <signal.h>

static pthread_mutex_t Mutex = PTHREAD_MUTEX_INITIALIZER;

/**
 * actual test program
 */

void sighandler(int sig) {
	write(2,".",1);
	return;
}

void* threadstart(void* arg) {

	while (1) {
		pthread_mutex_lock(&Mutex);
	}
}

int main(char** argc, int argv) {

	pthread_t testthread1;	
	struct sigaction action;

	action.sa_handler=sighandler;
	action.sa_flags=0;
	sigfillset( &action.sa_mask );
	sigaction(SIGUSR1,&action,NULL);

	pthread_mutex_lock(&Mutex);
	pthread_create(&testthread1,NULL,threadstart,NULL);
	while (1) {
		pthread_kill(testthread1,SIGUSR1);
	}
}

[-- Attachment #6: test_case_5_pselect.c --]
[-- Type: text/x-c++src, Size: 771 bytes --]

/**
 * small test program whether signals between threads work as they should
 */

#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#include <pthread.h>
#include <signal.h>

void sighandler(int sig) {
	write(2,".",1);
	return;
}

void* threadstart(void* arg) {
sigset_t sigset;
struct timespec sleeptime;

	while (1) {
		sleeptime.tv_sec=1;
		sleeptime.tv_nsec=0;
		sigemptyset(&sigset);
		pselect(0,NULL,NULL,NULL,&sleeptime,&sigset);
	}
}

int main(char** argc, int argv) {

	pthread_t testthread1;	
	struct sigaction action;

	action.sa_handler=sighandler;
	action.sa_flags=0;
	sigfillset( &action.sa_mask );
	sigaction(SIGUSR1,&action,NULL);

	pthread_create(&testthread1,NULL,threadstart,NULL);
	while (1) {
		pthread_kill(testthread1,SIGUSR1);
	}
}

[-- Attachment #7: test_case_6_sched_yield.c --]
[-- Type: text/x-c++src, Size: 582 bytes --]

/**
 * small test program whether signals between threads work as they should
 */

#include <unistd.h>
#include <pthread.h>
#include <signal.h>

void sighandler(int sig) {
	write(2,".",1);
	return;
}

void* threadstart(void* arg) {

	while (1) {
		sched_yield();
	}
}

int main(char** argc, int argv) {

	pthread_t testthread1;	
	struct sigaction action;

	action.sa_handler=sighandler;
	action.sa_flags=0;
	sigfillset( &action.sa_mask );
	sigaction(SIGUSR1,&action,NULL);

	pthread_create(&testthread1,NULL,threadstart,NULL);
	while (1) {
		pthread_kill(testthread1,SIGUSR1);
	}
}

[-- Attachment #8: test_case_7_busy_waiting.c --]
[-- Type: text/x-c++src, Size: 565 bytes --]

/**
 * small test program whether signals between threads work as they should
 */

#include <unistd.h>
#include <pthread.h>
#include <signal.h>

void sighandler(int sig) {
	write(2,".",1);
	return;
}

void* threadstart(void* arg) {

	while (1) {
	}
}

int main(char** argc, int argv) {

	pthread_t testthread1;	
	struct sigaction action;

	action.sa_handler=sighandler;
	action.sa_flags=0;
	sigfillset( &action.sa_mask );
	sigaction(SIGUSR1,&action,NULL);

	pthread_create(&testthread1,NULL,threadstart,NULL);
	while (1) {
		pthread_kill(testthread1,SIGUSR1);
	}
}

[-- Attachment #9: test_case_8_io.c --]
[-- Type: text/x-c++src, Size: 602 bytes --]

/**
 * small test program whether signals between threads work as they should
 */

#include <unistd.h>
#include <pthread.h>
#include <signal.h>

void sighandler(int sig) {
	write(2,".",1);
	return;
}

void* threadstart(void* arg) {
	char buf[1024];

	while (1) {
		read(1,buf,512);
	}
}

int main(char** argc, int argv) {

	pthread_t testthread1;	
	struct sigaction action;

	action.sa_handler=sighandler;
	action.sa_flags=0;
	sigfillset( &action.sa_mask );
	sigaction(SIGUSR1,&action,NULL);

	pthread_create(&testthread1,NULL,threadstart,NULL);
	while (1) {
		pthread_kill(testthread1,SIGUSR1);
	}
}


[-- Attachment #10: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: pthread_mutex_lock doesn't interrupt

[Andy Koppe]

On 12 July 2010 17:56, James Cotton wrote:
> Hello, I've been trying to get the FreeRTOS Posix simulator to run on
> Cygwin and have run into a few bugs in the signal handling/pthread
> libraries and have isolated test cases
>
> I believe when pthread_mutex_lock receives an interrupt it should run
> it then resume trying to get a lock

Have you got chapter and verse from the POSIX standard on that? By
'interrupt' do you mean SIGINT or any signal? Is it possible that the
signal is just being deferred and then subsumed by later signals?

Andy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Sending SIGUSR1 to thread in nanosleep causes segfault

[Andy Koppe]

On 12 July 2010 22:05, Corvus Corax wrote:
> Just to update some details.
>
> I am working with James Cotton on the OpenPilot project.
> We are trying to port the POSIX simulation port of FreeRTOS to cygwin.
>
> We run into issues with signal handling in cygwin 1.7.5 that are
> currently blocking our development, since every method to suspend a
> running thread via signals either causes:
>
> - random segfaults within Cygwin DLL.
> or
> - failure to execute the signal handler.

I'm afraid you're unlikely to get a quick and comprehensive answer,
because the Cygwin signal guru currently is on a well-deserved
vacation. Thanks for the testcases, hopefully they'll prove useful.

You could also try the latest snapshot from
http://cygwin.com/snapshots to see whether that makes a difference. A
debug version of the DLL is available there as well. And if you're
brave enough, you could try getting stuck into the Cygwin sources
yourself.

The trouble with Cygwin signals is that Windows provides practically
no support for them, so everything has to be manually mapped to
Windows concepts. Have a look at the Signals section of
http://www.cygwin.com/cygwin-ug-net/highlights.html for some of the
tricks. Obviously pthreads further complicate matters, and the sort of
thing you're doing here is not something that tends to get a lot of
use by "normal" programs, so shortcomings are quite likely.

Andy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: pthread_mutex_lock doesn't interrupt

[Corvus Corax]

Hi

I did read your other response, regarding the signal developer being on
vacation. I'm sending this just for reference, since I had already
looked it up anyway :)

Andy wrote:

> Have you got chapter and verse from the POSIX standard on that? By
> 'interrupt' do you mean SIGINT or any signal? Is it possible that the
> signal is just being deferred and then subsumed by later signals?
> 
> Andy

PTHREAD_MUTEX_LOCK() has been defined in IEEE POSIX 1003.1c.

From the POSIX Programmers Manual:

http://www.opengroup.org/onlinepubs/9699919799/functions/pthread_mutex_lock.html

> If  a  signal is delivered to a thread waiting for a mutex, upon
> return from the signal handler the thread shall resume waiting
> for  the  mutex as if it was not interrupted.

As you can see from the test cases (attached to the other mail)
only one type of signal (SIGUSR1) is repeatedly sent, but not handled.

I am not sure if I understood what you mean by "deferred".

It is possible (though in our case irrelevant) that the cygwin code
WOULD still call the signal handler AFTER pthread_mutex_lock() returns.
However in the FreeRTOS code this never happens, since the failure of
executing the signal handler even on repeated signals results in the
mutex in question not being released -> deadlock!

(Another thread holds the mutex but waits for the signal handler
 to acknowledge the signal reception before releasing it)

regards

Corvus

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: pthread_mutex_lock doesn't interrupt

[Andy Koppe]

On 13 July 2010 08:07, Corvus Corax wrote:
>> Have you got chapter and verse from the POSIX standard on that? By
>> 'interrupt' do you mean SIGINT or any signal? Is it possible that the
>> signal is just being deferred and then subsumed by later signals?
>>
>> Andy
>
> PTHREAD_MUTEX_LOCK() has been defined in IEEE POSIX 1003.1c.
>
> From the POSIX Programmers Manual:
>
> http://www.opengroup.org/onlinepubs/9699919799/functions/pthread_mutex_lock.html
>
>> If  a  signal is delivered to a thread waiting for a mutex, upon
>> return from the signal handler the thread shall resume waiting
>> for  the  mutex as if it was not interrupted.
>
> As you can see from the test cases (attached to the other mail)
> only one type of signal (SIGUSR1) is repeatedly sent, but not handled.
>
> I am not sure if I understood what you mean by "deferred".

Signal generation and signal delivery aren't the same thing. With
asynchronous signals as in your test case, there's always some sort of
delay until the signal is delivered, and I don't know whether the
standard actually requires any upper bound on that. One example where
a signal in Cygwin gets deferred is when your thread happens to be
blocked in a Win32 function (although of course POSIX doesn't apply to
that anyway).

Having said all that, if it works in Linux, it ought to work in
Cygwin. No idea what the issue could be though.

Andy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: pthread_mutex_lock doesn't interrupt

[Corvus Corax]

> Signal generation and signal delivery aren't the same thing. With
> asynchronous signals as in your test case, there's always some sort of
> delay until the signal is delivered, and I don't know whether the
> standard actually requires any upper bound on that. One example where
> a signal in Cygwin gets deferred is when your thread happens to be
> blocked in a Win32 function (although of course POSIX doesn't apply to
> that anyway).
> 
> Having said all that, if it works in Linux, it ought to work in
> Cygwin. No idea what the issue could be though.

Well as you already said, it could be calling a win32 api function to
implement the mutex_lock and miss the signal because of that.

for that I actually might have a work around - a busy loop with

> while (0!=mutex_trylock()) sched_yield();

You must agree though that's a damn ugly solution just to lock a mutex
in a non signal-blocking way. Busy loops are a no brainer. And due to
other bugs with signaling in combination to sched_yield() as in the
other test case - this isn't even stable right now.

However even pselect() with its sigmask argument explicitly
sigemptyset()-ed does completely block signals and doesn't execute the
handler.

pselect() allows the user to specify which signals should be deferred
until the end of pselect() and which should be handled and end
pselect() with errno=EINTR -- and of course also call the signal
handler.

I suspect the reason of this behaviour on the entire group of blocking
functions I found (select(), pselect(), pthread_mutex_lock(), ...)
could possibly be the same.

I haven't found any information about how long signals may be deferred
by POSIX functions without explicit definition of signal blockage.

However POSIX usually explicitly defines the functions which may defer
signals - and in many cases even introduces a mask to control that
behaviour. I think it's a safe assumption however that if a signal is
received and a signal handler is set up, the signal handler shall
EVENTUALLY be called unless specified otherwise.

With pthread_mutex_lock() not being released until the signal had been
handled and cygwin not executing the handler until the end of the lock,
the signal is however deferred FOREVER, which brakes this assumption.

One could probably argue that this is not explicitly forbidden by the
POSIX standard, however it's not how Linux or any other semi-posix-like
system works, and it makes certain programming constructs 
(in this case a user space scheduler) basically impossible to implement.


Well long story short, I am trying to get a backtrace of the segfault
caused by the other signaling bug with a current debugging enabled
cygwin.dll and post it here, if you don't already have that.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: pthread_mutex_lock doesn't interrupt

[Andy Koppe]

On 13 July 2010 01:08, James Cotton wrote:
> Sorry, I don't have access to the actual POSIX standard

It's at http://www.opengroup.org/onlinepubs/9699919799.

> but my
> interpretation is based on man pages:
>
> https://computing.llnl.gov/tutorials/pthreads/man/pthread_mutex_lock.txt
>
> And certainly nowhere have I see "pthread_mutex_lock blocks signals".
> The signal we are using is SIGUSR1.  I don't see why the signal would
> be deferred since there are no signals being blocked.  Possibly the
> first signal gets caught and another one comes while in the signal
> handler (where usual that signal should then get blocked) and is
> disrupting the processing.

Makes sense.

Andy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: pthread_mutex_lock doesn't interrupt

[James Cotton]

Sorry, I don't have access to the actual POSIX standard, but my
interpretation is based on man pages:

https://computing.llnl.gov/tutorials/pthreads/man/pthread_mutex_lock.txt

And certainly nowhere have I see "pthread_mutex_lock blocks signals".
The signal we are using is SIGUSR1.  I don't see why the signal would
be deferred since there are no signals being blocked.  Possibly the
first signal gets caught and another one comes while in the signal
handler (where usual that signal should then get blocked) and is
disrupting the processing.

James

>On 12 July 2010 17:56, James Cotton wrote:
>> Hello, I've been trying to get the FreeRTOS Posix simulator to run on
>> Cygwin and have run into a few bugs in the signal handling/pthread
>> libraries and have isolated test cases
>>
>> I believe when pthread_mutex_lock receives an interrupt it should run
>> it then resume trying to get a lock
>
>Have you got chapter and verse from the POSIX standard on that? By
>'interrupt' do you mean SIGINT or any signal? Is it possible that the
>signal is just being deferred and then subsumed by later signals?
>
>Andy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple