[newlib-cygwin] Filter TCP connections to SO_REUSEPORT_LB listen sockets by NUMA domain

[newlib-cygwin] Filter TCP connections to SO_REUSEPORT_LB listen sockets by NUMA domain

[Sebastian Huber]

https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;h=c76896074b31125c69534e25754646c3655c2187

commit c76896074b31125c69534e25754646c3655c2187
Author: Andrew Gallatin <gallatin@FreeBSD.org>
Date:   Sat Dec 19 22:04:46 2020 +0000

    Filter TCP connections to SO_REUSEPORT_LB listen sockets by NUMA domain
    
    In order to efficiently serve web traffic on a NUMA
    machine, one must avoid as many NUMA domain crossings as
    possible. With SO_REUSEPORT_LB, a number of workers can share a
    listen socket. However, even if a worker sets affinity to a core
    or set of cores on a NUMA domain, it will receive connections
    associated with all NUMA domains in the system. This will lead to
    cross-domain traffic when the server writes to the socket or
    calls sendfile(), and memory is allocated on the server's local
    NUMA node, but transmitted on the NUMA node associated with the
    TCP connection. Similarly, when the server reads from the socket,
    he will likely be reading memory allocated on the NUMA domain
    associated with the TCP connection.
    
    This change provides a new socket ioctl, TCP_REUSPORT_LB_NUMA. A
    server can now tell the kernel to filter traffic so that only
    incoming connections associated with the desired NUMA domain are
    given to the server. (Of course, in the case where there are no
    servers sharing the listen socket on some domain, then as a
    fallback, traffic will be hashed as normal to all servers sharing
    the listen socket regardless of domain). This allows a server to
    deal only with traffic that is local to its NUMA domain, and
    avoids cross-domain traffic in most cases.
    
    This patch, and a corresponding small patch to nginx to use
    TCP_REUSPORT_LB_NUMA allows us to serve 190Gb/s of kTLS encrypted
    https media content from dual-socket Xeons with only 13% (as
    measured by pcm.x) cross domain traffic on the memory controller.
    
    Reviewed by:    jhb, bz (earlier version), bcr (man page)
    Tested by: gonzo
    Sponsored by:   Netfix
    Differential Revision:  https://reviews.freebsd.org/D21636

Diff:
---
 newlib/libc/sys/rtems/include/netinet/tcp.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/newlib/libc/sys/rtems/include/netinet/tcp.h b/newlib/libc/sys/rtems/include/netinet/tcp.h
index faf142959..0b71bd465 100644
--- a/newlib/libc/sys/rtems/include/netinet/tcp.h
+++ b/newlib/libc/sys/rtems/include/netinet/tcp.h
@@ -196,6 +196,7 @@ struct tcphdr {
 #define	TCP_PCAP_IN	4096	/* number of input packets to keep */
 #define TCP_FUNCTION_BLK 8192	/* Set the tcp function pointers to the specified stack */
 /* Options for Rack and BBR */
+#define	TCP_REUSPORT_LB_NUMA   1026	/* set listen socket numa domain */
 #define TCP_RACK_MBUF_QUEUE   1050 /* Do we allow mbuf queuing if supported */
 #define TCP_RACK_PROP	      1051 /* RACK proportional rate reduction (bool) */
 #define TCP_RACK_TLP_REDUCE   1052 /* RACK TLP cwnd reduction (bool) */
@@ -406,4 +407,7 @@ struct tcp_function_set {
 #define	VOI_TCP_GPUT_ND		8 /* Goodput normalised delta */
 #define	VOI_TCP_ACKLEN		9 /* Average ACKed bytes per ACK */
 
+#define TCP_REUSPORT_LB_NUMA_NODOM	(-2) /* remove numa binding */
+#define TCP_REUSPORT_LB_NUMA_CURDOM	(-1) /* bind to current domain */
+
 #endif /* !_NETINET_TCP_H_ */