Building sunrpc from glibc source

Building sunrpc from glibc source

[Binello, Severino]

Hello

  As of RedHat 8, the sunrpc is no longer included with glibc shared object library.
Unfortunately, our communications software would require extensive redesign in order to use tirpc.
As such, we are looking into an alternative approach where we just build the sunrpc portion from the glibc source tar file.
However, running into difficulties separating it out.
Can you recommend a method for just building the sunrpc code ?

Thanks Much
-Sev

ps: Below is the reason why our code is incompatible with the tirpc design
with old glibc every RPC server runs in its own thread,
with tirpc library there can be only one RPC server per program.
See:
from svc.c of tirpc library:

static struct svc_callout /* removed declaration */ *svc_head;

from svc.c of glibc-2.25:

#ifdef _RPC_THREAD_SAFE_
#define svc_head RPC_THREAD_VARIABLE(svc_head_s)
#else
static struct svc_callout *svc_head;
#endif

As you can see, if RPC_THREAD_SAFE_ is defined,
svc_head is per thread variable.


--

Sev Binello
Brookhaven National Laboratory
Upton, New York
631-344-5647
sev@bnl.gov

Re: Building sunrpc from glibc source

[Florian Weimer]

* Severino via Libc-alpha Binello:

> As of RedHat 8, the sunrpc is no longer included with glibc shared
> object library.  Unfortunately, our communications software would
> require extensive redesign in order to use tirpc.  As such, we are
> looking into an alternative approach where we just build the sunrpc
> portion from the glibc source tar file.

You can build your software on Red Hat Enterprise Linux 7 and run it on
Red Hat Enterprise Linux 8.  The interfaces are there, they are just not
exposed for linking new applications.

Have you tried to copy the src/svc.c out of libtirpc, rename the
function, and the kind of per-thread data management you need, and use
that?  I haven't checked in detail, but it looks like the code does not
depend on libtirpc internals, so this should be feasible.  It may even
make sense to contribute this service framework to upstream libtirpc.

(I strongly suggest to start with libtirpc because the separate building
issue has already been solved there.  Also, IPv6.)

In any case, this sounds more like an enhancement request for libtirpc
than for glibc.  We will not bring back the old, IPv4-only and DES-only
RPC code.

Thanks,
Florian

Re: Building sunrpc from glibc source

[Ian Kent]

On 6/8/21 7:12 am, Binello, Severino via Libc-alpha wrote:
> Hello
>
>    As of RedHat 8, the sunrpc is no longer included with glibc shared object library.
> Unfortunately, our communications software would require extensive redesign in order to use tirpc.

For example?

Can you describe the sort of challenges you have doing this please.


> As such, we are looking into an alternative approach where we just build the sunrpc portion from the glibc source tar file.
> However, running into difficulties separating it out.
> Can you recommend a method for just building the sunrpc code ?

It's worth understanding what might be needed in order to use libtirpc

first.


>
> Thanks Much
> -Sev
>
> ps: Below is the reason why our code is incompatible with the tirpc design
> with old glibc every RPC server runs in its own thread,
> with tirpc library there can be only one RPC server per program.
> See:
> from svc.c of tirpc library:
>
> static struct svc_callout /* removed declaration */ *svc_head;
>
> from svc.c of glibc-2.25:
>
> #ifdef _RPC_THREAD_SAFE_
> #define svc_head RPC_THREAD_VARIABLE(svc_head_s)
> #else
> static struct svc_callout *svc_head;
> #endif
>
> As you can see, if RPC_THREAD_SAFE_ is defined,
> svc_head is per thread variable.

I think I have some quick and nasty multi-thread libtirpc svc code

kicking around somewhere (if I can find it now). It might be worth

cleaning that up and maybe enhancing it a little, or maybe it's broken

I don't know, but I'd recommend looking at that first, if there's not

to many other problems to deal with.


If that works out we could take this discussion to the libtirpc list.


Ian

Re: Building sunrpc from glibc source

[Ian Kent]

On 6/8/21 8:07 pm, Ian Kent wrote:
> On 6/8/21 7:12 am, Binello, Severino via Libc-alpha wrote:
>> Hello
>>
>>    As of RedHat 8, the sunrpc is no longer included with glibc shared 
>> object library.
>> Unfortunately, our communications software would require extensive 
>> redesign in order to use tirpc.
>
> For example?
>
> Can you describe the sort of challenges you have doing this please.
>
>
>> As such, we are looking into an alternative approach where we just 
>> build the sunrpc portion from the glibc source tar file.
>> However, running into difficulties separating it out.
>> Can you recommend a method for just building the sunrpc code ?
>
> It's worth understanding what might be needed in order to use libtirpc
>
> first.
>
>
>>
>> Thanks Much
>> -Sev
>>
>> ps: Below is the reason why our code is incompatible with the tirpc 
>> design
>> with old glibc every RPC server runs in its own thread,
>> with tirpc library there can be only one RPC server per program.
>> See:
>> from svc.c of tirpc library:
>>
>> static struct svc_callout /* removed declaration */ *svc_head;
>>
>> from svc.c of glibc-2.25:
>>
>> #ifdef _RPC_THREAD_SAFE_
>> #define svc_head RPC_THREAD_VARIABLE(svc_head_s)
>> #else
>> static struct svc_callout *svc_head;
>> #endif
>>
>> As you can see, if RPC_THREAD_SAFE_ is defined,
>> svc_head is per thread variable.
>
> I think I have some quick and nasty multi-thread libtirpc svc code
>
> kicking around somewhere (if I can find it now). It might be worth
>
> cleaning that up and maybe enhancing it a little, or maybe it's broken
>
> I don't know, but I'd recommend looking at that first, if there's not
>
> to many other problems to deal with.

Actually it looks like this was multi-threaded io not multi-threaded 
servers.


But I'm not sure that you can't register multiple services in both glibc

and libtirpc, it's just that it's not thread safe to do so in glibc.


Maybe I don't understand what your doing, explain it please.


Why do you need a separate services list for each thread rather than a

library global lock protected services list as in libtirpc?


Ian

Re: Building sunrpc from glibc source

[Binello, Severino]

Hello Ian and Florian

Thanks for the feedback, its much appreciated
Unfortunately Im not the best person to be asking the questions or providing the info
My role with our communication code right now is to get it to build

I am reaching out to the developer of our rpc based code to see if he can provide
a better picture and provide  further information.
So I am including Al Marusic in this email thread.

Thanks again!
-Sev

________________________________
From: Ian Kent <ikent@redhat.com>
Sent: Friday, August 6, 2021 8:27 PM
To: Binello, Severino <sev@bnl.gov>; libc-alpha@sourceware.org <libc-alpha@sourceware.org>
Subject: Re: Building sunrpc from glibc source


On 6/8/21 8:07 pm, Ian Kent wrote:
> On 6/8/21 7:12 am, Binello, Severino via Libc-alpha wrote:
>> Hello
>>
>>    As of RedHat 8, the sunrpc is no longer included with glibc shared
>> object library.
>> Unfortunately, our communications software would require extensive
>> redesign in order to use tirpc.
>
> For example?
>
> Can you describe the sort of challenges you have doing this please.
>
>
>> As such, we are looking into an alternative approach where we just
>> build the sunrpc portion from the glibc source tar file.
>> However, running into difficulties separating it out.
>> Can you recommend a method for just building the sunrpc code ?
>
> It's worth understanding what might be needed in order to use libtirpc
>
> first.
>
>
>>
>> Thanks Much
>> -Sev
>>
>> ps: Below is the reason why our code is incompatible with the tirpc
>> design
>> with old glibc every RPC server runs in its own thread,
>> with tirpc library there can be only one RPC server per program.
>> See:
>> from svc.c of tirpc library:
>>
>> static struct svc_callout /* removed declaration */ *svc_head;
>>
>> from svc.c of glibc-2.25:
>>
>> #ifdef _RPC_THREAD_SAFE_
>> #define svc_head RPC_THREAD_VARIABLE(svc_head_s)
>> #else
>> static struct svc_callout *svc_head;
>> #endif
>>
>> As you can see, if RPC_THREAD_SAFE_ is defined,
>> svc_head is per thread variable.
>
> I think I have some quick and nasty multi-thread libtirpc svc code
>
> kicking around somewhere (if I can find it now). It might be worth
>
> cleaning that up and maybe enhancing it a little, or maybe it's broken
>
> I don't know, but I'd recommend looking at that first, if there's not
>
> to many other problems to deal with.

Actually it looks like this was multi-threaded io not multi-threaded
servers.


But I'm not sure that you can't register multiple services in both glibc

and libtirpc, it's just that it's not thread safe to do so in glibc.


Maybe I don't understand what your doing, explain it please.


Why do you need a separate services list for each thread rather than a

library global lock protected services list as in libtirpc?


Ian

Re: Building sunrpc from glibc source

[Ian Kent]

On 11/8/21 1:47 am, Binello, Severino wrote:
> Hello Ian and Florian
>
> Thanks for the feedback, its much appreciated
> Unfortunately Im not the best person to be asking the questions or 
> providing the info
> My role with our communication code right now is to get it to build
>
> I am reaching out to the developer of our rpc based code to see if he 
> can provide
> a better picture and provide  further information.
> So I am including Al Marusic in this email thread.


Ok.


Don't get me wrong, I do understand the potential difficulty changing

to libtirpc.


But glibc rpc has been on the chopping block for years so there's been

plenty of time to investigate and change.


At the very least some analysis needs to be done about the application

rpc usage to understand where the difficulties are.


Certainly, if you use low level rpc calls (in order to take control of

timeouts and such) the work would be substantial but if your using higher

level rpc calls it might not be so bad.


TBH the application I maintain does need to use lower level rpc calls so

using libtirpc was a significant change for me, but that was a long time

ago now.


Bottom line is that libtirpc is thread safe but it depends on the rpc

calls that are used.


Ian

>
> Thanks again!
> -Sev
>
> ------------------------------------------------------------------------
> *From:* Ian Kent <ikent@redhat.com>
> *Sent:* Friday, August 6, 2021 8:27 PM
> *To:* Binello, Severino <sev@bnl.gov>; libc-alpha@sourceware.org 
> <libc-alpha@sourceware.org>
> *Subject:* Re: Building sunrpc from glibc source
>
> On 6/8/21 8:07 pm, Ian Kent wrote:
> > On 6/8/21 7:12 am, Binello, Severino via Libc-alpha wrote:
> >> Hello
> >>
> >>    As of RedHat 8, the sunrpc is no longer included with glibc shared
> >> object library.
> >> Unfortunately, our communications software would require extensive
> >> redesign in order to use tirpc.
> >
> > For example?
> >
> > Can you describe the sort of challenges you have doing this please.
> >
> >
> >> As such, we are looking into an alternative approach where we just
> >> build the sunrpc portion from the glibc source tar file.
> >> However, running into difficulties separating it out.
> >> Can you recommend a method for just building the sunrpc code ?
> >
> > It's worth understanding what might be needed in order to use libtirpc
> >
> > first.
> >
> >
> >>
> >> Thanks Much
> >> -Sev
> >>
> >> ps: Below is the reason why our code is incompatible with the tirpc
> >> design
> >> with old glibc every RPC server runs in its own thread,
> >> with tirpc library there can be only one RPC server per program.
> >> See:
> >> from svc.c of tirpc library:
> >>
> >> static struct svc_callout /* removed declaration */ *svc_head;
> >>
> >> from svc.c of glibc-2.25:
> >>
> >> #ifdef _RPC_THREAD_SAFE_
> >> #define svc_head RPC_THREAD_VARIABLE(svc_head_s)
> >> #else
> >> static struct svc_callout *svc_head;
> >> #endif
> >>
> >> As you can see, if RPC_THREAD_SAFE_ is defined,
> >> svc_head is per thread variable.
> >
> > I think I have some quick and nasty multi-thread libtirpc svc code
> >
> > kicking around somewhere (if I can find it now). It might be worth
> >
> > cleaning that up and maybe enhancing it a little, or maybe it's broken
> >
> > I don't know, but I'd recommend looking at that first, if there's not
> >
> > to many other problems to deal with.
>
> Actually it looks like this was multi-threaded io not multi-threaded
> servers.
>
>
> But I'm not sure that you can't register multiple services in both glibc
>
> and libtirpc, it's just that it's not thread safe to do so in glibc.
>
>
> Maybe I don't understand what your doing, explain it please.
>
>
> Why do you need a separate services list for each thread rather than a
>
> library global lock protected services list as in libtirpc?
>
>
> Ian
>

Re: Building sunrpc from glibc source

[Marusic, Aljosa]

One example how we use SUNRPC:
Let's assume a program needs to collect data from various data servers.
To collect the data, the program creates a thread for each data server it needs data from,
and in that thread, it creates a RPC server to receive data.
Then the program tells each data server to start delivering data to its own RPC server.
Then each data server creates the connection to its own RPC server and
starts delivering data using void RPC calls (RPC calls which do not require response).
That way a problem with any data server (for example crash or slowness)
does not affect data delivery from other data servers and
by having many threads simultaneously receiving data,
much greater throughput can be achieved.
With SUNRPC which was in glibc (derived from SUNRPC version 4)
these numerous RPC servers in different threads are independent,
with TI-RPC they are not.

On top of that, we have RPC servers for different purposes (not just data delivery) and
we use threads to separate those RPC servers from each other also.
This system was created on SunOS machines about 25 years ago to control
Relativistic Heavy Ion Collider and that is why SUNRPC is used for everything.

Al


________________________________
From: Ian Kent <ikent@redhat.com>
Sent: Tuesday, August 10, 2021 8:53 PM
To: Binello, Severino <sev@bnl.gov>; libc-alpha@sourceware.org <libc-alpha@sourceware.org>; fweimer@redhat.com <fweimer@redhat.com>; Marusic, Aljosa <marusic@bnl.gov>
Subject: Re: Building sunrpc from glibc source



On 11/8/21 1:47 am, Binello, Severino wrote:
Hello Ian and Florian

Thanks for the feedback, its much appreciated
Unfortunately Im not the best person to be asking the questions or providing the info
My role with our communication code right now is to get it to build

I am reaching out to the developer of our rpc based code to see if he can provide
a better picture and provide  further information.
So I am including Al Marusic in this email thread.


Ok.


Don't get me wrong, I do understand the potential difficulty changing

to libtirpc.


But glibc rpc has been on the chopping block for years so there's been

plenty of time to investigate and change.


At the very least some analysis needs to be done about the application

rpc usage to understand where the difficulties are.


Certainly, if you use low level rpc calls (in order to take control of

timeouts and such) the work would be substantial but if your using higher

level rpc calls it might not be so bad.


TBH the application I maintain does need to use lower level rpc calls so

using libtirpc was a significant change for me, but that was a long time

ago now.


Bottom line is that libtirpc is thread safe but it depends on the rpc

calls that are used.


Ian

Thanks again!
-Sev

________________________________
From: Ian Kent <ikent@redhat.com><mailto:ikent@redhat.com>
Sent: Friday, August 6, 2021 8:27 PM
To: Binello, Severino <sev@bnl.gov><mailto:sev@bnl.gov>; libc-alpha@sourceware.org<mailto:libc-alpha@sourceware.org> <libc-alpha@sourceware.org><mailto:libc-alpha@sourceware.org>
Subject: Re: Building sunrpc from glibc source


On 6/8/21 8:07 pm, Ian Kent wrote:
> On 6/8/21 7:12 am, Binello, Severino via Libc-alpha wrote:
>> Hello
>>
>>    As of RedHat 8, the sunrpc is no longer included with glibc shared
>> object library.
>> Unfortunately, our communications software would require extensive
>> redesign in order to use tirpc.
>
> For example?
>
> Can you describe the sort of challenges you have doing this please.
>
>
>> As such, we are looking into an alternative approach where we just
>> build the sunrpc portion from the glibc source tar file.
>> However, running into difficulties separating it out.
>> Can you recommend a method for just building the sunrpc code ?
>
> It's worth understanding what might be needed in order to use libtirpc
>
> first.
>
>
>>
>> Thanks Much
>> -Sev
>>
>> ps: Below is the reason why our code is incompatible with the tirpc
>> design
>> with old glibc every RPC server runs in its own thread,
>> with tirpc library there can be only one RPC server per program.
>> See:
>> from svc.c of tirpc library:
>>
>> static struct svc_callout /* removed declaration */ *svc_head;
>>
>> from svc.c of glibc-2.25:
>>
>> #ifdef _RPC_THREAD_SAFE_
>> #define svc_head RPC_THREAD_VARIABLE(svc_head_s)
>> #else
>> static struct svc_callout *svc_head;
>> #endif
>>
>> As you can see, if RPC_THREAD_SAFE_ is defined,
>> svc_head is per thread variable.
>
> I think I have some quick and nasty multi-thread libtirpc svc code
>
> kicking around somewhere (if I can find it now). It might be worth
>
> cleaning that up and maybe enhancing it a little, or maybe it's broken
>
> I don't know, but I'd recommend looking at that first, if there's not
>
> to many other problems to deal with.

Actually it looks like this was multi-threaded io not multi-threaded
servers.


But I'm not sure that you can't register multiple services in both glibc

and libtirpc, it's just that it's not thread safe to do so in glibc.


Maybe I don't understand what your doing, explain it please.


Why do you need a separate services list for each thread rather than a

library global lock protected services list as in libtirpc?


Ian

Re: Building sunrpc from glibc source

[Ian Kent]

On 11/8/21 10:37 am, Marusic, Aljosa wrote:
> One example how we use SUNRPC:
> Let's assume a program needs to collect data from various data servers.
> To collect the data, the program creates a thread for each data server 
> it needs data from,
> and in that thread, it creates a RPC server to receive data.
> Then the program tells each data server to start delivering data to 
> its own RPC server.
> Then each data server creates the connection to its own RPC server and
> starts delivering data using void RPC calls (RPC calls which do not 
> require response).
> That way a problem with any data server (for example crash or slowness)
> does not affect data delivery from other data servers and
> by having many threads simultaneously receiving data,
> much greater throughput can be achieved.
> With SUNRPC which was in glibc (derived from SUNRPC version 4)
> these numerous RPC servers in different threads are independent,
> with TI-RPC they are not.
>
> On top of that, we have RPC servers for different purposes (not just 
> data delivery) and
> we use threads to separate those RPC servers from each other also.
> This system was created on SunOS machines about 25 years ago to control
> Relativistic Heavy Ion Collider and that is why SUNRPC is used for 
> everything.

Sure I got that you need to create independent server instances.


I'm not asking why you need to do that, I try not to suggest people do

things differently, that's a decision I should not be involved with. My

interest is only in why it can't be achieved with libtirpc.


For example (note, I haven't looked very far into libtirpc yet) you

would call svc_register() for each service. But svc_register() in

libtirpc will add that service to the list of services without a lock,

same as glibc (hence the need for a per thread list variable) so that's

no good.


But in libtirpc svc_create() can create the service and mostly has the

same arguments and it calls svc_reg() instead of svc_register(), after

doing all the netconfig stuff libtirpc needs, and it does use a lock

when modifying the services list.


So, at first glace, if your not doing things that are lower level RPC

you might be able to use libtirpc without too much pain.


My question is what are the things that make it so hard to change?


Ian

Re: Building sunrpc from glibc source

[Ian Kent]

On 11/8/21 1:14 pm, Ian Kent wrote:
>
>
> On 11/8/21 10:37 am, Marusic, Aljosa wrote:
>> One example how we use SUNRPC:
>> Let's assume a program needs to collect data from various data servers.
>> To collect the data, the program creates a thread for each data 
>> server it needs data from,
>> and in that thread, it creates a RPC server to receive data.
>> Then the program tells each data server to start delivering data to 
>> its own RPC server.
>> Then each data server creates the connection to its own RPC server and
>> starts delivering data using void RPC calls (RPC calls which do not 
>> require response).
>> That way a problem with any data server (for example crash or slowness)
>> does not affect data delivery from other data servers and
>> by having many threads simultaneously receiving data,
>> much greater throughput can be achieved.
>> With SUNRPC which was in glibc (derived from SUNRPC version 4)
>> these numerous RPC servers in different threads are independent,
>> with TI-RPC they are not.
>>
>> On top of that, we have RPC servers for different purposes (not just 
>> data delivery) and
>> we use threads to separate those RPC servers from each other also.
>> This system was created on SunOS machines about 25 years ago to control
>> Relativistic Heavy Ion Collider and that is why SUNRPC is used for 
>> everything.
>
> Sure I got that you need to create independent server instances.
>
>
> I'm not asking why you need to do that, I try not to suggest people do
>
> things differently, that's a decision I should not be involved with. My
>
> interest is only in why it can't be achieved with libtirpc.
>
>
> For example (note, I haven't looked very far into libtirpc yet) you
>
> would call svc_register() for each service. But svc_register() in
>
> libtirpc will add that service to the list of services without a lock,
>
> same as glibc (hence the need for a per thread list variable) so that's
>
> no good.
>
>
> But in libtirpc svc_create() can create the service and mostly has the
>
> same arguments and it calls svc_reg() instead of svc_register(), after
>
> doing all the netconfig stuff libtirpc needs, and it does use a lock
>
> when modifying the services list.
>
>
> So, at first glace, if your not doing things that are lower level RPC
>
> you might be able to use libtirpc without too much pain.
>
Umm ... it looks like you could even continue using svc_register() with

something like:


/* From /usr/include/tirpc */

#include <reentrant.h>


extern rwlock_t svc_lock;


...


rwlock_wrlock(&svc_lock);

ret = svc_register(...);

rwlock_unlock(&svc_lock);


...


assuming libtirpc is built with PORTMAP defined in your distro, and the

upstream source does do this by default.


But for sure if you have multiple threads concurrently using the same

service (ie. transport) you would have more work to do but then that's

also the case with glibc rpc.


>
> My question is what are the things that make it so hard to change?
>
>
> Ian
>

Re: Building sunrpc from glibc source

[Marusic, Aljosa]

The problem is that svc_reg() (just like svc_register()) adds
the service to global svc_head list:

static struct svc_callout
{
  struct svc_callout *sc_next;
  rpcprog_t sc_prog;
  rpcvers_t sc_vers;
  char *sc_netid;
  void (*sc_dispatch) (struct svc_req *, SVCXPRT *);
} *svc_head;

When svc_getreq_common() is called in any thread,
it tries to check if any RPC socket needs to be serviced.
This means that in our code we have many threads trying to read / write the same sockets.

In glibc RPC, svc_head is thread variable:

#define svc_head RPC_THREAD_VARIABLE(svc_head_s)

so that does not happen.
If there were another version of svc_getreq_poll() function
with SVCXPRT* as additional argument, our code could be made to work.

Al


________________________________
From: Ian Kent <ikent@redhat.com>
Sent: Wednesday, August 11, 2021 1:14 AM
To: Marusic, Aljosa <marusic@bnl.gov>; Binello, Severino <sev@bnl.gov>; libc-alpha@sourceware.org <libc-alpha@sourceware.org>; fweimer@redhat.com <fweimer@redhat.com>
Subject: Re: Building sunrpc from glibc source



On 11/8/21 10:37 am, Marusic, Aljosa wrote:
One example how we use SUNRPC:
Let's assume a program needs to collect data from various data servers.
To collect the data, the program creates a thread for each data server it needs data from,
and in that thread, it creates a RPC server to receive data.
Then the program tells each data server to start delivering data to its own RPC server.
Then each data server creates the connection to its own RPC server and
starts delivering data using void RPC calls (RPC calls which do not require response).
That way a problem with any data server (for example crash or slowness)
does not affect data delivery from other data servers and
by having many threads simultaneously receiving data,
much greater throughput can be achieved.
With SUNRPC which was in glibc (derived from SUNRPC version 4)
these numerous RPC servers in different threads are independent,
with TI-RPC they are not.

On top of that, we have RPC servers for different purposes (not just data delivery) and
we use threads to separate those RPC servers from each other also.
This system was created on SunOS machines about 25 years ago to control
Relativistic Heavy Ion Collider and that is why SUNRPC is used for everything.

Sure I got that you need to create independent server instances.


I'm not asking why you need to do that, I try not to suggest people do

things differently, that's a decision I should not be involved with. My

interest is only in why it can't be achieved with libtirpc.


For example (note, I haven't looked very far into libtirpc yet) you

would call svc_register() for each service. But svc_register() in

libtirpc will add that service to the list of services without a lock,

same as glibc (hence the need for a per thread list variable) so that's

no good.


But in libtirpc svc_create() can create the service and mostly has the

same arguments and it calls svc_reg() instead of svc_register(), after

doing all the netconfig stuff libtirpc needs, and it does use a lock

when modifying the services list.


So, at first glace, if your not doing things that are lower level RPC

you might be able to use libtirpc without too much pain.


My question is what are the things that make it so hard to change?


Ian

Re: Building sunrpc from glibc source

[Ian Kent]

On 11/8/21 3:30 pm, Marusic, Aljosa wrote:
> The problem is that svc_reg() (just like svc_register()) adds
> the service to global svc_head list:
>
> static struct svc_callout
> {
>   struct svc_callout *sc_next;
>   rpcprog_t sc_prog;
>   rpcvers_t sc_vers;
>   char *sc_netid;
>   void (*sc_dispatch) (struct svc_req *, SVCXPRT *);
> } *svc_head;
>
> When svc_getreq_common() is called in any thread,
> it tries to check if any RPC socket needs to be serviced.
> This means that in our code we have many threads trying to read / 
> write the same sockets.

Yes, I'm familiar with that code.


So that select() gets in the road then.


I think your saying that, with the glibc per thread implementation, 
svc_fdset

also ends up essentially a per thread fd set so the sockets are also 
isolated.


I suspect that amounts to needing a multi-threaded svc_run() function (along

with a couple of it's friends). That's not all that hard to do (I have some

quickly put together example code that does that) but, by the sound of your

case, there might be other side effects ... and I'm not familiar with your

code either.


Oh well, it was worth exploring ... let me know if I can do anything to 
help.


>
> In glibc RPC, svc_head is thread variable:
>
> #define svc_head RPC_THREAD_VARIABLE(svc_head_s)
>
> so that does not happen.
> If there were another version of svc_getreq_poll() function
> with SVCXPRT* as additional argument, our code could be made to work.
>
> Al
>
>
> ------------------------------------------------------------------------
> *From:* Ian Kent <ikent@redhat.com>
> *Sent:* Wednesday, August 11, 2021 1:14 AM
> *To:* Marusic, Aljosa <marusic@bnl.gov>; Binello, Severino 
> <sev@bnl.gov>; libc-alpha@sourceware.org <libc-alpha@sourceware.org>; 
> fweimer@redhat.com <fweimer@redhat.com>
> *Subject:* Re: Building sunrpc from glibc source
>
>
> On 11/8/21 10:37 am, Marusic, Aljosa wrote:
>> One example how we use SUNRPC:
>> Let's assume a program needs to collect data from various data servers.
>> To collect the data, the program creates a thread for each data 
>> server it needs data from,
>> and in that thread, it creates a RPC server to receive data.
>> Then the program tells each data server to start delivering data to 
>> its own RPC server.
>> Then each data server creates the connection to its own RPC server and
>> starts delivering data using void RPC calls (RPC calls which do not 
>> require response).
>> That way a problem with any data server (for example crash or slowness)
>> does not affect data delivery from other data servers and
>> by having many threads simultaneously receiving data,
>> much greater throughput can be achieved.
>> With SUNRPC which was in glibc (derived from SUNRPC version 4)
>> these numerous RPC servers in different threads are independent,
>> with TI-RPC they are not.
>>
>> On top of that, we have RPC servers for different purposes (not just 
>> data delivery) and
>> we use threads to separate those RPC servers from each other also.
>> This system was created on SunOS machines about 25 years ago to control
>> Relativistic Heavy Ion Collider and that is why SUNRPC is used for 
>> everything.
>
> Sure I got that you need to create independent server instances.
>
>
> I'm not asking why you need to do that, I try not to suggest people do
>
> things differently, that's a decision I should not be involved with. My
>
> interest is only in why it can't be achieved with libtirpc.
>
>
> For example (note, I haven't looked very far into libtirpc yet) you
>
> would call svc_register() for each service. But svc_register() in
>
> libtirpc will add that service to the list of services without a lock,
>
> same as glibc (hence the need for a per thread list variable) so that's
>
> no good.
>
>
> But in libtirpc svc_create() can create the service and mostly has the
>
> same arguments and it calls svc_reg() instead of svc_register(), after
>
> doing all the netconfig stuff libtirpc needs, and it does use a lock
>
> when modifying the services list.
>
>
> So, at first glace, if your not doing things that are lower level RPC
>
> you might be able to use libtirpc without too much pain.
>
>
> My question is what are the things that make it so hard to change?
>
>
> Ian
>

Re: Building sunrpc from glibc source

[Ian Kent]

On 11/8/21 3:30 pm, Marusic, Aljosa wrote:
> If there were another version of svc_getreq_poll() function
> with SVCXPRT* as additional argument, our code could be made to work.

Right, I didn't look at svc_getreq_poll() when I was playing around

with this, sorry.


Ian