Re: AW: RPC clnt_create() adress already in use

Re: AW: RPC clnt_create() adress already in use

[PAULUS, Raimund, TI-ABN]

Hello Mark,

you wrote:
"Just so I understand more fully: you are compiling with those options on Cygwin?"

Answer: yes

you wrote:
"The idea there is to use a different port on each run of the program.  If you're looping around and reaching that code again with port==0, then yes you'll get the same port number 
each time because getpid() is not changing."

Answer:
That's correct. But you can see, the following bind() is embedded in a loop: if the bind() would work correctly, it would deliver res < 0 because errno = EADDRINUSE. In this case 
the port would be incremented and the next loop pass is made. This would happen till an unused port is found or all ports are used. I think, the problem here is, bind() does not 
deliver the correct return value. Therefore my two approaches.

you wrote:
"I don't know enough to set up an RPC test bed to try your client program 
locally.  *Unless* it's sufficient to run your client program and aim at the 
local machine even though it has no RPC server listening.  If that would 
demonstrate your issue, let me know and I'll have another look.  Otherwise, I 
would need to set up an RPC server on my Cygwin machine.  If you've done that on 
your Cygwin machine, what are the steps?  I think I can follow Unix-style 
instructions but will have to allow for Windows Firewall so if you've done this 
already on Cygwin your procedure would help a lot."

Answer:
To make a connection the RPC server must be running. My RPC server is running on a linux machine since several years. I know, it is reliable and i would not have an additional possible error source.
I didn't do anything with the Windows firewall.
I found a simple RPC example here: https://www.unf.edu/~sahuja/cnt5505/rpcexample.html.
I did some minimal changes to better show the problem (client uses tcp instead udp, connection establishment in a loop). I could reproduce the problem in this example. Here the sources and the compile directives:

date.h
================================
/*
 * Please do not edit this file.
 * It was generated using rpcgen.
 */

#ifndef _DATE_H_RPCGEN
#define _DATE_H_RPCGEN

#include <rpc/rpc.h>

#ifdef __cplusplus
extern "C" {
#endif

#define DATEPROG 0x3012225
#define DATEVERS 1

#if defined(__STDC__) || defined(__cplusplus)
#define BINDATE 1
extern  long * bindate_1(void *, CLIENT *);
extern  long * bindate_1_svc(void *, struct svc_req *);
extern int dateprog_1_freeresult (SVCXPRT *, xdrproc_t, caddr_t);

#else /* K&R C */
#define BINDATE 1
extern  long * bindate_1();
extern  long * bindate_1_svc();
extern int dateprog_1_freeresult ();
#endif /* K&R C */

#ifdef __cplusplus
}
#endif

#endif /* !_DATE_H_RPCGEN */

date_client.c
===================
/*
 * Please do not edit this file.
 * It was generated using rpcgen.
 */

#include <memory.h> /* for memset */
#include "date.h"

/* Default timeout can be changed using clnt_control() */
static struct timeval TIMEOUT = { 25, 0 };

long *
bindate_1(void *argp, CLIENT *clnt)
{
        static long clnt_res;

        memset((char *)&clnt_res, 0, sizeof(clnt_res));
        if (clnt_call (clnt, BINDATE,
                (xdrproc_t) xdr_void, (caddr_t) argp,
                (xdrproc_t) xdr_long, (caddr_t) &clnt_res,
                TIMEOUT) != RPC_SUCCESS) {
                return (NULL);
        }
        return (&clnt_res);
}


date_svc.c
==================
/*
 * Please do not edit this file.
 * It was generated using rpcgen.
 */

#include "date.h"
#include <stdio.h>
#include <stdlib.h>
#include <rpc/pmap_clnt.h>
#include <string.h>
#include <memory.h>
#include <sys/socket.h>
#include <netinet/in.h>

#ifndef SIG_PF
#define SIG_PF void(*)(int)
#endif

static void
dateprog_1(struct svc_req *rqstp, register SVCXPRT *transp)
{
        union {
                int fill;
        } argument;
        char *result;
        xdrproc_t _xdr_argument, _xdr_result;
        char *(*local)(char *, struct svc_req *);

        switch (rqstp->rq_proc) {
        case NULLPROC:
                (void) svc_sendreply (transp, (xdrproc_t) xdr_void, (char *)NULL);
                return;
        case BINDATE:
                _xdr_argument = (xdrproc_t) xdr_void;
                _xdr_result = (xdrproc_t) xdr_long;
                local = (char *(*)(char *, struct svc_req *)) bindate_1_svc;
                break;

        default:
                svcerr_noproc (transp);
                return;
        }
        memset ((char *)&argument, 0, sizeof (argument));
        if (!svc_getargs (transp, (xdrproc_t) _xdr_argument, (caddr_t) &argument)) {
                svcerr_decode (transp);
                return;
        }
        result = (*local)((char *)&argument, rqstp);
        if (result != NULL && !svc_sendreply(transp, (xdrproc_t) _xdr_result, result)) {
                svcerr_systemerr (transp);
        }
        if (!svc_freeargs (transp, (xdrproc_t) _xdr_argument, (caddr_t) &argument)) {
                fprintf (stderr, "%s", "unable to free arguments");
                exit (1);
        }
        return;
}

int
main (int argc, char **argv)
{
        register SVCXPRT *transp;

        pmap_unset (DATEPROG, DATEVERS);

        transp = svctcp_create(RPC_ANYSOCK, 0, 0);
        if (transp == NULL) {
                fprintf (stderr, "%s", "cannot create tcp service.");
                exit(1);
        }
        if (!svc_register(transp, DATEPROG, DATEVERS, dateprog_1, IPPROTO_UDP)) {
                fprintf (stderr, "%s", "unable to register (DATEPROG, DATEVERS, udp).");
                exit(1);
        }

        transp = svctcp_create(RPC_ANYSOCK, 0, 0);
        if (transp == NULL) {
                fprintf (stderr, "%s", "cannot create tcp service.");
                exit(1);
        }
        if (!svc_register(transp, DATEPROG, DATEVERS, dateprog_1, IPPROTO_TCP)) {
                fprintf (stderr, "%s", "unable to register (DATEPROG, DATEVERS, tcp).");
                exit(1);
        }

        svc_run ();
        fprintf (stderr, "%s", "svc_run returned");
        exit (1);
        /* NOTREACHED */
}

dateproc.c
=======================
/*********************************************************************/
/* dateproc.c - remote procedures; called by server stub             */
#include <stdio.h>
#include <stdlib.h>
#include <rpc/rpc.h>
#include "date.h"

/* return the binary date and time */
/* In Linux: long * bindate_1_svc(void* arg1, struct svc_req *arg2) {
*/

/* In Dec Unix: long * bindate_1() {
*/

long * bindate_1_svc(void* arg1, struct svc_req *arg2) {
  static long timeval; /* must be static */
  timeval = time((long *) 0);
  return (&timeval);
}

rdate.c
=================================
/****************************************************************/
/* rdate.c - client program for remote date service             */
#include <stdio.h>
#include <time.h>
#include <rpc/rpc.h>
#include <stdlib.h>
#include <unistd.h>
#include "date.h"

int main(int argc, char *argv[])
{
  CLIENT *cl;
  char *server;
  long *lres;
  if (argc != 2) {
    fprintf(stderr, "usage: %s hostname\n", argv[0]);
    exit(1);
  }

  server = argv[1];

int i;
for(i=0; i< 100; i++)
{
  /* create client handle */
  if ((cl = clnt_create(server, DATEPROG, DATEVERS, "tcp")) == NULL) {
    /* couldn't establish connection with server */
    printf("can't establish connection with host %s\n", server);
    exit(2);
    }

  /* first call the remote procedure bindate() */
  if (( lres = bindate_1(NULL, cl)) == NULL){
    printf(" remote procedure bindate() failure\n");
    exit(3);
  }

  printf("Loop %d: time on host %s = %ld == %s",
         i, server, *lres, ctime(lres));
  clnt_destroy(cl); /* done with handle */

sleep(1);
}

  return 0;
}


****************************************************
Commands:
gcc -c date_clnt.c
gcc -c date_svc.c
gcc -c dateproc.c
gcc -c rdate.c
gcc -o client date_clnt.o rdate.o
gcc -o server date_svc.o dateproc.o

for Cygwin
gcc -I/usr/include/tirpc -c date_clnt.c
gcc -I/usr/include/tirpc -c date_svc.c
gcc -I/usr/include/tirpc -c dateproc.c
gcc -I/usr/include/tirpc -c rdate.c
gcc -o client date_clnt.o rdate.o -ltirpc
gcc -o server date_svc.o dateproc.o -ltirpc


I hope this will help

Raimund


-----Ursprüngliche Nachricht-----
Von: cygwin-owner@cygwin.com [mailto:cygwin-owner@cygwin.com] Im Auftrag von Mark Geisert
Gesendet: Mittwoch, 27. Dezember 2017 10:39
An: cygwin@cygwin.com
Betreff: Re: RPC clnt_create() adress already in use

Hi Raimund,
Comments embedded below...

PAULUS, Raimund, TI-ABN wrote:
> Hello Mark,
>
> in my earlier tests the lib and header files for tirpc was still there from the Cygwin-installation. I think, this could influence the results of the tests.
> Therefore i removed these files. From now on i used only the downloaded files from sourceforge (https://sourceforge.net/projects/libtirpc/files/libtirpc/). I tested libtirpc-0.3.2, libtirpc-1.0.1 and libtirpc-1.0.2. The results for all releases are the same.
>
> I needed the following sources:
>
> auth_none.c, auth_unix.c, authunix_prot.c, bindresvport.c, clnt_bcast.c, clnt_dg.c, clnt_generic.c, clnt_perror.c, clnt_vc.c, debug.c, getnetconfig.c, getnetpath.c, mt_misc.c, rpc_callmsg.c, rpc_commondata.c, rpc_generic.c, rpc_prot.c, rpcb_clnt.c, rpcb_prot.c, xdr.c, xdr_array.c, xdr_float.c, xdr_reference.c,   xdr_rec.c, xdr_mem.c
>
> I had to change the following (already mentioned earlier):
>
> clnt_bcast.c
> (because POLLRDNORM, POLLRDBAND are already defined in /usr/include/sys/poll.h) :
>
> --> #ifndef POLLRDNORM
> # define POLLRDNORM     0x040           /* Normal data may be read.  */
> # define POLLRDBAND     0x080           /* Priority data may be read.  */
> --> #endif

Not only does /usr/include/sys/poll.h already define them, they're defined with different values.  Could be a red flag or a red herring; don't know yet.

> xdr.h (typedefs where missed):
>
> #include <rpc/types.h>
>
> --> typedef __uint64_t      u_quad_t;
> --> typedef __int64_t       quad_t;
>
>
> My compile-flags: -D_GNU_SOURCE -D__GLIBC__ -D__linux__

Just so I understand more fully: you are compiling with those options on Cygwin?

> If i build my test program with the sources above, the behavior is the same as with the Cygwin-libtirpc: the first RPC-connection is ok, the next call to clnt_create() delivers the error EADDRINUSE.
>
> The test program on a Linux-box works perfectly. With "netstat -n" you can see each connection from the client to the server creates a new pair of ports. On the server-side the pair is always the same (ports 111, 907). On the client-side however the ports differ from one connection to the other. When the connections are closed the ports remain in state TIME_WAIT. After a while they disappear.
>
> If the RPC-client (Cygwin) is running on a Windows-box, "netstat -n" shows the pair of ports for the first connection. The next connection attempt (clnt_create()) delivers: EADDRINUSE.
>
> In the sources you can see the following:
>
> clnt_create() calls the function bindresvport_sa() (in bindresvport.c). Here the port for the connection is determined:
>
> if (port == 0) {
>                 port = (getpid() % NPORTS) + STARTPORT; }

The idea there is to use a different port on each run of the program.  If you're looping around and reaching that code again with port==0, then yes you'll get the same port number each time because getpid() is not changing.

>
> and the socket is bound to the port:
>
> res = bind(sd, sa, salen);
>
> If this part is executed multiple times (loop) the port is always the same. The function bind() doesn't deliver an error (res = 0, maybe bind() doesn't work correctly?). Therefore the error is not detected until the call of  connect() in function clnt_vc_create() (in clnt_vc.c). But here you can not correct the port.
>
> I tried the following:
>
> The call of setsockopt(., SO_REUSEADDR, .) before bind() doesn't change anything. The error still occurs.
>
> I found 2 approaches to avoid the error:
>
> 1. Before the call to bind() set the port to 0 (port = 0). The following bind() searches an unused port.
>
> 2. Set the port to 0 (port = 0). Do not call bind(). Later the call to connect() searches an unused port.
>
> I tested the two methods and the behavior was the same as the Linux-client. It obviously works correctly.  But i'm not a network specialist and i don't know, if it's the best solution for the problem.
>
>
> Subsequently I downloaded the Cygwin-sources for libtirpc-0.3.2 and libtirpc-1.0.1. And here again, with the changes mentioned in the two approaches the RPC-connection works perfectly. The only difference her is: a connection does not create a pair of ports but only a single port (netstat -n).

I must compliment you; you are being very specific and detailed about what 
you're doing on the client end of the RPC connection.  But as I said in my last 
email, I don't know specifics about RPC implementation; I sort of know generally 
what RPC can do.

I don't know enough to set up an RPC test bed to try your client program 
locally.  *Unless* it's sufficient to run your client program and aim at the 
local machine even though it has no RPC server listening.  If that would 
demonstrate your issue, let me know and I'll have another look.  Otherwise, I 
would need to set up an RPC server on my Cygwin machine.  If you've done that on 
your Cygwin machine, what are the steps?  I think I can follow Unix-style 
instructions but will have to allow for Windows Firewall so if you've done this 
already on Cygwin your procedure would help a lot.
Thanks,

..mark


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: AW: RPC clnt_create() adress already in use

[Mark Geisert]

Hi Raimund,
I think I've now got a working RPC test-bed on my Cygwin machine.  I did have to 
deal with Windows Firewall but it was easy.  I installed rpcbind, started it up, 
compiled following your recipe and ran './server' and './client localhost' in 
two separate windows.  What I see is this...

./client localhost
Loop 0: time on host localhost = 1514712920 == Sun Dec 31 01:35:20 2017
can't establish connection with host localhost

Am I correct that what you expect to see is multiple lines each starting with a 
different "Loop" number?  And that if I use a vanilla (non-Cygwin) libtirpc I 
should see those multiple lines?
Thanks,

..mark

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: AW: RPC clnt_create() adress already in use

[Mark Geisert]

Hi Raimund,
I think I might have an idea where the root cause is.  But first I want to 
revisit your very first email on this subject, from last September 22.  You 
mentioned test results from various Cygwin versions vs various Windows versions.

Are you absolutely sure your test program ran correctly on Cygwin 1.5.18 on 
*both* Windows XP and Windows 7?

The reason I ask is that Windows' handling of socket option SO_REUSEADDR has 
changed over time and Cygwin has had to make accommodations to keep up.  There 
might possibly be a need to revisit this within Cygwin.

I want to test a possible solution within the Cygwin DLL on my test machine but 
it has another two days to go on a factorization problem it's running (under 
Cygwin).  So I will respond again after I test.
Thank you,

..mark

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: AW: RPC clnt_create() adress already in use

[PAULUS, Raimund, TI-ABN]

Hi Mark,

that's it. Maybe you have to add a linfeed in this line (rdate.c) for a better output:

  printf("Loop %d: time on host %s = %ld == %s \n",
         i, server, *lres, ctime(lres));


But indeed, that is the error. The output of my test (libtirpc without changes):

$ ./client ttil75
Loop 0: time on host ttil75 = 1514880129 == Tue Jan  2 09:02:09 2018
can't establish connection with host ttil75


And the test, if i patch the libtirpc (port = 0 before bind()):

$ ./client ttil75
Loop 0: time on host ttil75 = 1514879460 == Tue Jan  2 08:51:00 2018
Loop 1: time on host ttil75 = 1514879461 == Tue Jan  2 08:51:01 2018
Loop 2: time on host ttil75 = 1514879462 == Tue Jan  2 08:51:02 2018
Loop 3: time on host ttil75 = 1514879463 == Tue Jan  2 08:51:03 2018
Loop 4: time on host ttil75 = 1514879464 == Tue Jan  2 08:51:04 2018
Loop 5: time on host ttil75 = 1514879465 == Tue Jan  2 08:51:05 2018
Loop 6: time on host ttil75 = 1514879466 == Tue Jan  2 08:51:06 2018
Loop 7: time on host ttil75 = 1514879467 == Tue Jan  2 08:51:07 2018
Loop 8: time on host ttil75 = 1514879468 == Tue Jan  2 08:51:08 2018
Loop 9: time on host ttil75 = 1514879469 == Tue Jan  2 08:51:09 2018
Loop 10: time on host ttil75 = 1514879470 == Tue Jan  2 08:51:10 2018
Loop 11: time on host ttil75 = 1514879471 == Tue Jan  2 08:51:11 2018
Loop 12: time on host ttil75 = 1514879472 == Tue Jan  2 08:51:12 2018
Loop 13: time on host ttil75 = 1514879473 == Tue Jan  2 08:51:13 2018
..
..
..
Loop 96: time on host ttil75 = 1514879556 == Tue Jan  2 08:52:36 2018
Loop 97: time on host ttil75 = 1514879557 == Tue Jan  2 08:52:37 2018
Loop 98: time on host ttil75 = 1514879558 == Tue Jan  2 08:52:38 2018
Loop 99: time on host ttil75 = 1514879559 == Tue Jan  2 08:52:39 2018


I think, the cause for this error is a failure in the function bind(). But I don't know the sources of bind().

Greetings and a happy new year

Raimund


-----Ursprüngliche Nachricht-----
Von: cygwin-owner@cygwin.com [mailto:cygwin-owner@cygwin.com] Im Auftrag von Mark Geisert
Gesendet: Sonntag, 31. Dezember 2017 10:50
An: cygwin@cygwin.com
Betreff: Re: AW: RPC clnt_create() adress already in use

Hi Raimund,
I think I've now got a working RPC test-bed on my Cygwin machine.  I did have to deal with Windows Firewall but it was easy.  I installed rpcbind, started it up, compiled following your recipe and ran './server' and './client localhost' in two separate windows.  What I see is this...

./client localhost
Loop 0: time on host localhost = 1514712920 == Sun Dec 31 01:35:20 2017 can't establish connection with host localhost

Am I correct that what you expect to see is multiple lines each starting with a different "Loop" number?  And that if I use a vanilla (non-Cygwin) libtirpc I should see those multiple lines?
Thanks,

..mark

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

AW: RPC clnt_create() adress already in use

[PAULUS, Raimund, TI-ABN]

Hallo Mark Geisert,

many thanks for your answer. I supposed this too.

I included in my source code the following function calls after clnt_create():

int fd = 0;
bool bool_ret = clnt_control(cl, CLGET_FD, &fd); if(bool_ret == true) {
  printf("fd: %d\n", fd);

  int enable = 1;
  retval = setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(int));
  if(retval < 0)
    fprintf(stderr, "Fehler setsockopt(): %s\n", strerror(errno)); }

The function clnt_control() delivers the socket of the RPC-Client-Handle.
The result is the same as before. Moreover i think, the effect of setsockopt() would be valid only during the process is running (my test scenarios 1 and 2). 
But it wouldn't change anything regarding my test scenario 3 (several restarts).

Raimund


-----Ursprüngliche Nachricht-----
Von: cygwin-owner@cygwin.com [mailto:cygwin-owner@cygwin.com] Im Auftrag von Mark Geisert
Gesendet: Sonntag, 24. September 2017 11:49
An: cygwin@cygwin.com
Betreff: Re: RPC clnt_create() adress already in use

PAULUS, Raimund, TI-ABN wrote:
> In our environment we have a Linux-Server and several Windows-PCs (Windows XP SP3, in the past Windows NT too). On the Linux-Server RPC-Services (Remote Procedure Call) are running, one service for one Windows-PC each. To build the RPC-clients on the Windows-boxes, I used Cygwin 1.5.18 and actually Cygwin 2.5.1 (because we now have Windows 7 too).
>
> The RPC-Client (*.exe) on the Windows-box is started at any time by hand, establishes a connection to the server (clnt_create), executes a few calls to the server (clnt_call), closes the connection (clnt_destroy) and exits. The time-interval between the program starts can be 10 seconds up to 2 minutes.
>
> This worked over many years without any error (Cygwin 1.5.18).
> Now I upgraded to Cygwin 2_5_1 (libtirpc instead librpc) and there are problems. Sometimes the RPC-Client cannot establish a connection to the server. The error message from clnt_spcreateerror():
>
> Remote system error - Address already in use
>
> I think it is "EADDRINUSE".

I can only answer generally as I haven't tried your testcase and don't know anything about either RPC library on Cygwin.  If EADDRINUSE is the error you're getting, it might be due to a difference between libtirpc and the older librpc. 
When a program binds a specific address and port to a socket, uses the socket then later closes it, the system keeps the <address, port> tuple in a "locked" 
state until enough time has passed for the other end of the connection to notice the connection's been closed.  During that time, attempting to bind() with the same address and port will result in EADDRINUSE.  It's a TCP/IP safety mechanism.

If you wish to override that behavior, you set a specific option SO_REUSEADDR on the socket with setsockopt() before you issue the bind().  Perhaps libtirpc is not doing that, though librpc was doing that.  Only way to know is to examine the source to both libraries.  It's possible an strace of a broken session compared to an strace of a working session might shed some light.

..mark


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple