Cygwin socket option SO_REUSEADDR operates unlike Linux

[Mark Geisert <mark@maxrnd.com>]

This report is based on a series of recent list emails with Subject: lines
"RPC clnt_create() adress already in use" which date back to last 
September but are unfortunately not chained together...  They contain a 
discussion I've been having with OP Raimund Paulus.

I believe I've distilled the issue(s) down as far as I can.  A 
self-contained STC is included at the end of this email.

On the latest 64-bit Cygwin, running the STC shows:

~ netstat -an|grep :111
   TCP    0.0.0.0:111            0.0.0.0:0              LISTENING
   TCP    [::]:111               [::]:0                 LISTENING
   UDP    0.0.0.0:111            *:*
   UDP    [::]:111               *:*

~ ./bindtest
1st socket is 3
1st bind OK
1st connect OK
2nd socket is 3
2nd bind OK
2nd connect: Address already in use

~ ./bindtest
1st socket is 3
1st bind OK
1st connect: Address already in use

On Fedora 27, running the same STC shows:

[mark@lux ~]$ netstat -an|grep :111
tcp        0      0 0.0.0.0:111         0.0.0.0:*        LISTEN
tcp6       0      0 :::111              :::*             LISTEN
udp        0      0 0.0.0.0:111         0.0.0.0:*
udp6       0      0 :::111              :::*
[mark@lux ~]$ ./bindtest
1st socket is 3
1st bind OK
1st connect OK
2nd socket is 3
2nd bind OK
2nd connect OK
[mark@lux ~]$ ./bindtest
1st socket is 3
1st bind OK
1st connect OK
2nd socket is 3
2nd bind OK
2nd connect OK

The STC source code is given below.  It assumes you're running rpcbind on 
the local machine at TCP port 111.  Remember to abort rpcbind after 
testing if you don't need it running for other RPC services.

Two issues are visible.  (1) The 2nd connect attempt elicits an EADDRINUSE 
on Cygwin even though SO_REUSEADDR has been set.  Fedora allows the 2nd 
connect to succeed.  (2) The EADDRINUSE is being reported by connect(), 
not by the preceding bind(), which is where one usually sees it.

I've spent some time inside Cygwin's net.cc and fhandler_socket.cc and see 
how Cygwin deals with Winsock's peculiar notion of SO_REUSEADDR.  Between 
that unfortunate but necessary workaround and what I see on MSDN now, at
https://msdn.microsoft.com/en-us/library/windows/desktop/ms740621(v=vs.85).aspx 
I can't help wondering if the workaround has maybe stopped working.  The 
web page's section "Enhanced Socket Security" is what made me wonder this.

Here's the STC, a single source file bindtest.c to be compiled with
     gcc -g -Wall -o bindtest bindtest.c

Thanks,

..mark

--------8<--------
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define LCLADDR "127.0.0.1"
#define LCLPORT 8888
#define RMTADDR LCLADDR
#define RMTPORT 111

void
checkresult (int value, char *string)
{
     if (value == -1) {
         perror (string);
         exit (1);
     }
}

int
main (int argc, char **argv)
{
     const int one = 1;
     int res;
     int sock;
     struct sockaddr lcladdr, rmtaddr;
     struct sockaddr_in addr;
     socklen_t alen = sizeof (addr);

     memset (&addr, 0, alen);
     memset (&rmtaddr, 0, sizeof (rmtaddr));
     memset (&lcladdr, 0, sizeof (lcladdr));

     addr.sin_family = AF_INET;
     addr.sin_port = htons (RMTPORT);
     addr.sin_addr.s_addr = inet_addr (RMTADDR);
     memcpy (&rmtaddr, &addr, alen);

     addr.sin_port = htons (LCLPORT);
     addr.sin_addr.s_addr = inet_addr (LCLADDR);
     memcpy (&lcladdr, &addr, alen);

//  FIRST CONNECTION ATTEMPT
     sock = socket (AF_INET, SOCK_STREAM, 0);
     checkresult (sock, "1st socket");
     fprintf (stderr, "1st socket is %d\n", sock);

     res = setsockopt (sock, SOL_SOCKET, SO_REUSEADDR, &one, sizeof (one));
     checkresult (res, "1st setsockopt");

     res = bind (sock, &lcladdr, alen);
     checkresult (res, "1st bind");
     fprintf (stderr, "1st bind OK\n");

     res = connect (sock, &rmtaddr, alen);
     checkresult (res, "1st connect");
     fprintf (stderr, "1st connect OK\n");

     res = close (sock);
     checkresult (res, "1st close");

//  SECOND CONNECTION ATTEMPT
     sock = socket (AF_INET, SOCK_STREAM, 0);
     checkresult (sock, "2nd socket");
     fprintf (stderr, "2nd socket is %d\n", sock);

     res = setsockopt (sock, SOL_SOCKET, SO_REUSEADDR, &one, sizeof (one));
     checkresult (res, "2nd setsockopt");

     res = bind (sock, &lcladdr, alen);
     checkresult (res, "2nd bind");
     fprintf (stderr, "2nd bind OK\n");

     res = connect (sock, &rmtaddr, alen);
     checkresult (res, "2nd connect");
     fprintf (stderr, "2nd connect OK\n");

     res = close (sock);
     checkresult (res, "2nd close");

     return 0;
}

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple