1.5.9-1: socket() appears NOT to be thread-safe

1.5.9-1: socket() appears NOT to be thread-safe

[Enzo Michelangeli]

While porting to Cygwin the Linux application "sipd"
(http://www.sxdesign.com/index.php?page=developer&submnu=sipd ), which
uses pthreads, I noticed that socket() calls issued concurrently by
several threads often failed, with strerror(errno) saying "operation not
permitted". Once I protected all the calls with mutex locks, such errors
went away. Is Cygwin's implementation of socket() known to be
thread-unsafe?

Enzo

P.S. I have written an implementation of the missing gethostbyname_r(),
based on a mutex-protected call to gethostbyname(). If useful, I may
gladly contribute the code.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Corinna Vinschen]

On Apr 14 18:42, Enzo Michelangeli wrote:
> While porting to Cygwin the Linux application "sipd"
> (http://www.sxdesign.com/index.php?page=developer&submnu=sipd ), which
> uses pthreads, I noticed that socket() calls issued concurrently by
> several threads often failed, with strerror(errno) saying "operation not
> permitted". Once I protected all the calls with mutex locks, such errors
> went away. Is Cygwin's implementation of socket() known to be
> thread-unsafe?

No.  If you can prepare a brief testcase to demonstrate the problem,
I'm happy to fix it.  Of course, patches would be most welcome.  See
below.

> P.S. I have written an implementation of the missing gethostbyname_r(),
> based on a mutex-protected call to gethostbyname(). If useful, I may
> gladly contribute the code.

Sure!  Unfortunately, that requires us to ask you for a copyright
assignment first.  See http://cygwin.com/contrib.html for details.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Developer                                mailto:cygwin@cygwin.com
Red Hat, Inc.

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Brian Ford]

On Wed, 14 Apr 2004, Enzo Michelangeli wrote:

> While porting to Cygwin the Linux application "sipd"
> (http://www.sxdesign.com/index.php?page=developer&submnu=sipd ), which
> uses pthreads, I noticed that socket() calls issued concurrently by
> several threads often failed, with strerror(errno) saying "operation not
> permitted". Once I protected all the calls with mutex locks, such errors
> went away. Is Cygwin's implementation of socket() known to be
> thread-unsafe?

FWIW, I have seen this symptom as well, primarily on a dual CPU,
Hyperthreaded XP box under 1.5.[8|9]+.  In fact, I just confirmed it
still exists in a current CVS build from this morning.

It seems like a race condition.  Running under strace fixes it, so it has
been difficult to isolate.  If I have time, I'll try to look at it again
soon.  Would you like to work on it together?

-- 
Brian Ford
Senior Realtime Software Engineer
VITAL - Visual Simulation Systems
FlightSafety International
Phone: 314-551-8460
Fax:   314-551-8444

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Enzo Michelangeli]

----- Original Message ----- 
From: "Brian Ford" <ford@vss.fsi.com>
To: "Enzo Michelangeli" <em@em.no-ip.com>
Cc: <cygwin@cygwin.com>
Sent: Thursday, April 15, 2004 1:05 AM
Subject: Re: 1.5.9-1: socket() appears NOT to be thread-safe

> On Wed, 14 Apr 2004, Enzo Michelangeli wrote:
>
> > While porting to Cygwin the Linux application "sipd"
> > (http://www.sxdesign.com/index.php?page=developer&submnu=sipd ),
> > which uses pthreads, I noticed that socket() calls issued
> > concurrently by several threads often failed, with
> > strerror(errno) saying "operation not permitted". Once I
> > protected all the calls with mutex locks, such errors
> > went away. Is Cygwin's implementation of socket() known to be
> > thread-unsafe?
>
> FWIW, I have seen this symptom as well, primarily on a dual CPU,
> Hyperthreaded XP box under 1.5.[8|9]+.  In fact, I just confirmed
> it still exists in a current CVS build from this morning.
>
> It seems like a race condition.  Running under strace fixes it,
> so it has been difficult to isolate.  If I have time, I'll try
> to look at it again soon.  Would you like to work on it together?

What I can do for the time being is to provide the testcase that Corinna
asked for (attached below). Here are the results:

em@emnb ~/unsafesocket
$ cc -o unsafesocket unsafesocket.c

em@emnb ~/unsafesocket
$ ./unsafesocket
usage: ./unsafesocket [n|y]

em@emnb ~/unsafesocket
$ ./unsafesocket n
NOT protecting socket() call with mutex:
Thread 0: socket() returned 3, strerror(errno)="No error"
Thread 1: socket() returned -1, strerror(errno)="Operation not permitted"
Thread 2: socket() returned -1, strerror(errno)="Operation not permitted"
Thread 3: socket() returned 3, strerror(errno)="No error"

em@emnb ~/unsafesocket
$ ./unsafesocket n
NOT protecting socket() call with mutex:
Thread 0: socket() returned -1, strerror(errno)="Operation not permitted"
Thread 1: socket() returned 3, strerror(errno)="No error"
Thread 2: socket() returned -1, strerror(errno)="Operation not permitted"
Thread 3: socket() returned -1, strerror(errno)="Operation not permitted"

em@emnb ~/unsafesocket
$ ./unsafesocket y
Protecting socket() call with mutex:
Thread 0: socket() returned 3, strerror(errno)="No error"
Thread 1: socket() returned 4, strerror(errno)="No error"
Thread 2: socket() returned 5, strerror(errno)="No error"
Thread 3: socket() returned 6, strerror(errno)="No error"

em@emnb ~/unsafesocket
$

By the way, even in case of no error the socket calls return the same
value of fd. Is this OK??

Cheers --

Enzo

--------- begin unsafesocket.c ---------

#include <sys/socket.h>
#include <netinet/in.h>
#include <errno.h>
#include <stdio.h>
#include <pthread.h>

#define MAXTHREADS 4

static int sync_with_mutex = 0;

struct thrdata {
 int thrnum;
 pthread_t t;
 int socket_fd;
 int socket_errno;
} th[MAXTHREADS];

static void *thread_code(void *p);

main(int argc, char *argv[]) {
 int i;

 if(argc == 1) {
  printf("usage: %s [n|y]\n", argv[0]);
  exit(1);
 }

 if(argv[1][0] == 'y' || argv[1][0] == 'Y') {
  printf("Protecting socket() call with mutex:\n");
  sync_with_mutex = 1;
 } else {
  printf("NOT protecting socket() call with mutex:\n");
  sync_with_mutex = 0;
 }

 for(i=0; i<MAXTHREADS; i++)
 {
  th[i].thrnum = i;
  pthread_create(&th[i].t, NULL, thread_code, (void *)&(th[i]));
 }

 for(i=0; i<MAXTHREADS; i++) {
  pthread_join(th[i].t, NULL);
 }

 for(i=0; i<MAXTHREADS; i++) {
  printf("Thread %d: socket() returned %d, strerror(errno)=\"%s\"\n",
    i, th[i].socket_fd, strerror(th[i].socket_errno));

  if(th[i].socket_fd > 0)
   close(th[i].socket_fd);
 }

 exit(0);
}


static void *thread_code(void *p) {
 struct thrdata *pt = (struct thrdata *)p;
 int fd;
 struct sockaddr_in sa;
 static pthread_mutex_t __mutex = PTHREAD_MUTEX_INITIALIZER;

 bzero(&(sa), sizeof(sa));
 sa.sin_family = AF_INET;
 sa.sin_addr.s_addr = htonl(INADDR_ANY);
 sa.sin_port = 1024 + pt->thrnum;

 if(sync_with_mutex) {
  pthread_mutex_lock(&__mutex); /* begin critical area */
 }

 fd = socket(AF_INET, SOCK_DGRAM, 0);

 if(sync_with_mutex) {
  pthread_mutex_unlock(&__mutex); /* begin critical area */
 }

 /* report results to main thread */
 pt->socket_fd = fd;
 pt->socket_errno = errno;

 return(NULL);
}

--------- end unsafesocket.c ---------


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Christopher Faylor]

On Thu, Apr 15, 2004 at 11:16:03AM +0800, Enzo Michelangeli wrote:
>What I can do for the time being is to provide the testcase that Corinna
>asked for (attached below). Here are the results:

I can duplicate the behavior noted here.

I sent Corinna a patch in private email.  I'm not thrilled with it but
it does seem to alleviate the problem.

The problem seems to be that winsock's socket is not multi-thread safe.
That is very hard for me to believe but the strace output seems to
confirm it.

My "fix" was to just move an existing mutex lock earlier in the socket
call.  It makes me wonder, if this is really needed for socket, might it
also needed for other network calls like connect, listen, etc.?

Btw, Brian, I could duplicate this behavior with strace.  My usual trick
for doing this is to use a large buffer size:

  strace -oc:/tmp/strace.out -b1000000 threadsocket.exe n

This makes strace a little faster.  This method showed the failure for
me pretty reliably.
--
Christopher Faylor			spammer? ->	aaaspam@sourceware.org
Cygwin Co-Project Leader				aaaspam@duffek.com
TimeSys, Inc.

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Christopher Faylor]

On Thu, Apr 15, 2004 at 12:14:55AM -0400, Christopher Faylor wrote:
>On Thu, Apr 15, 2004 at 11:16:03AM +0800, Enzo Michelangeli wrote:
>>What I can do for the time being is to provide the testcase that Corinna
>>asked for (attached below). Here are the results:
>
>I can duplicate the behavior noted here.
>
>I sent Corinna a patch in private email.  I'm not thrilled with it but
>it does seem to alleviate the problem.
>
>The problem seems to be that winsock's socket is not multi-thread safe.
>That is very hard for me to believe but the strace output seems to
>confirm it.

Corinna showed me that this was a problem in my autoload code rather than
a problem with winsock.  That's comforting.  I guess I've grown too quick
to judge Windows.

I've checked in a fix and am regenerating a snapshot.  The fix consisted of
deleting a few lines of code so that's always nice...

Thanks for the test case.  It helped a lot in tracking this problem down.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Brian Ford]

I assume this email is hopeless, but I'm desperately searching for a
lead...

On Thu, 15 Apr 2004, Christopher Faylor wrote:

> Corinna showed me that this was a problem in my autoload code rather than
> a problem with winsock.  That's comforting.  I guess I've grown too quick
> to judge Windows.
>
> I've checked in a fix and am regenerating a snapshot.  The fix consisted of
> deleting a few lines of code so that's always nice...
>
> Thanks for the test case.  It helped a lot in tracking this problem down.

I still see the same symptom (ie. socket randomly returns "Operation not
permitted" at application startup) with current CVS, but not with the
original test case, and only on a dual CPU box :-(.  So far, I have been
unsuccessful at catching it with strace, even when -b 1000000 is supplied.
It is unfortunately a complicated series of events that cause the problem.

1.) X app forks
2.) child execlp's a /bin/sh script
3.) script exec's a program that calls socket

About 30% of the time, socket returns the error above.  I tried
replacing the exec line in the shell script with:

exec strace -o tracefile -b 1000000 socket_error.exe

but then it doesn't fail.  It also doesn't fail if socket_error.exe is
launched directly from the bash prompt.

I will keep trying to come up with a test case that I can actually study,
but I was hoping someone might have an idea about how to catch it better
or where to look.

Is it possible that the autoload code needs to be made dual CPU safe?

Thanks anyway.

-- 
Brian Ford
Senior Realtime Software Engineer
VITAL - Visual Simulation Systems
FlightSafety International
the best safety device in any aircraft is a well-trained pilot...

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Christopher Faylor]

On Fri, May 21, 2004 at 05:21:19PM -0500, Brian Ford wrote:
>On Thu, 15 Apr 2004, Christopher Faylor wrote:
>>Corinna showed me that this was a problem in my autoload code rather
>>than a problem with winsock.  That's comforting.  I guess I've grown
>>too quick to judge Windows.
>>
>>I've checked in a fix and am regenerating a snapshot.  The fix
>>consisted of deleting a few lines of code so that's always nice...
>>
>>Thanks for the test case.  It helped a lot in tracking this problem
>>down.
>
>I still see the same symptom (ie.  socket randomly returns "Operation
>not permitted" at application startup) with current CVS, but not with
>the original test case, and only on a dual CPU box :-(.

It's not usually helpful to see a "it doesn't work" a month after the
announcement of a fix.  Call me absent minded but I don't even remember
what I did to supposedly fix this.

>About 30% of the time, socket returns the error above.  I tried
>replacing the exec line in the shell script with:
>
>exec strace -o tracefile -b 1000000 socket_error.exe
>
>but then it doesn't fail.  It also doesn't fail if socket_error.exe is
>launched directly from the bash prompt.
>
>I will keep trying to come up with a test case that I can actually study,
>but I was hoping someone might have an idea about how to catch it better
>or where to look.

Put a call to the debugger at the offending error message and look around.

>Is it possible that the autoload code needs to be made dual CPU safe?

No.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Brian Ford]

On Fri, 21 May 2004, Christopher Faylor wrote:

> On Fri, May 21, 2004 at 05:21:19PM -0500, Brian Ford wrote:
> >On Thu, 15 Apr 2004, Christopher Faylor wrote:
> >>Corinna showed me that this was a problem in my autoload code rather
> >>than a problem with winsock.  That's comforting.  I guess I've grown
> >>too quick to judge Windows.
> >>
> >>I've checked in a fix and am regenerating a snapshot.  The fix
> >>consisted of deleting a few lines of code so that's always nice...
> >>
> >>Thanks for the test case.  It helped a lot in tracking this problem
> >>down.
> >
> >I still see the same symptom (ie.  socket randomly returns "Operation
> >not permitted" at application startup) with current CVS, but not with
> >the original test case, and only on a dual CPU box :-(.

I'm now seeing it on some Hyperthreaded boxes too.  It is still a problem
in current CVS.

> It's not usually helpful to see a "it doesn't work" a month after the
> announcement of a fix.  Call me absent minded but I don't even remember
> what I did to supposedly fix this.

I know, but I was unable to get time on the particular box that exhibts
the problem with the particular software scenerio that does the same in a
timely fashion.  Sorry.

> >About 30% of the time, socket returns the error above.  I tried
> >replacing the exec line in the shell script with:
> >
> >exec strace -o tracefile -b 1000000 socket_error.exe
> >
> >but then it doesn't fail.  It also doesn't fail if socket_error.exe is
> >launched directly from the bash prompt.
> >
> >I will keep trying to come up with a test case that I can actually study,
> >but I was hoping someone might have an idea about how to catch it better
> >or where to look.
>
> Put a call to the debugger at the offending error message and look around.

Ok, I found out a tiny bit more.

I put a call to try_to_debug in find_winsock_errno when it returned the
default EPERM.  Then, by setting
CYGWIN="error_start=c:\cygwin\bin\gdb.exe" I can get a gdb pop up and do a
back trace.  The error was:

10093
Either the application has not called WSAStartup, or WSAStartup failed.

and came from the socket call in cygwin_socket.

> >Is it possible that the autoload code needs to be made dual CPU safe?
>
> No.

So, given this is some sort of race condition, how do I debug the autoload
code and find out if WSAStartup was actually called or if it failed?

I'm still digging, but very slowly given the nature of the problem and my
time to debug it.

-- 
Brian Ford
Senior Realtime Software Engineer
VITAL - Visual Simulation Systems
FlightSafety International
the best safety device in any aircraft is a well-trained pilot...

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Brian Ford]

On Thu, 10 Jun 2004, Brian Ford wrote:

> I put a call to try_to_debug in find_winsock_errno when it returned the
> default EPERM.  Then, by setting
> CYGWIN="error_start=c:\cygwin\bin\gdb.exe" I can get a gdb pop up and do a
> back trace.  The error was:
>
> 10093
> Either the application has not called WSAStartup, or WSAStartup failed.
>
> and came from the socket call in cygwin_socket.
>
> So, given this is some sort of race condition, how do I debug the autoload
> code and find out if WSAStartup was actually called or if it failed?

Ok, I put another try_to_debug call in wsock_init just inside the if
(!wsock_started).  Sometimes I hit it first and the socket call succeeds.
But, sometimes I hit the one above first and the socket call fails.  In
the later case, wsock_started=1.  Any idea how that could happen?

-- 
Brian Ford
Senior Realtime Software Engineer
VITAL - Visual Simulation Systems
FlightSafety International
the best safety device in any aircraft is a well-trained pilot...

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.9-1: socket() appears NOT to be thread-safe

[Enzo Michelangeli]

I wrote:

----- Original Message ----- 
From: "Enzo Michelangeli" <em@em.no-ip.com>
To: <cygwin@cygwin.com>
Cc: ""Brian Ford"" <ford@vss.fsi.com>
Sent: Thursday, April 15, 2004 11:16 AM
Subject: Re: 1.5.9-1: socket() appears NOT to be thread-safe

[...]
> By the way, even in case of no error the socket calls return the same
> value of fd. Is this OK??

Please ignore the two lines above: I forgot to remove them after fixing
the bug that had prompted me to write them in first place :-) The fd are
obviously different, as shown in the sample output.

Enzo

P.S. By the way, Corinna: couldn't I just put my gethostbyname_r() in the
public domain, rather than going through the bureaucratic chore of the
copyright assignment? Also because I feel that implementing it through
mutex-protection of gethostbyname(), as I did, is just a quick hack, as it
unnecessarily blocks other threads that could access the name server in
parallel (with separate network I/O and properly re-entrant code). It may
help other implementors to solve an urgent problem, but I don't think it
should be released as part of the Cygwin code.



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/