* Bug(s) with creating large numbers of sockets
@ 2017-11-03 14:39 Erik Bray
2017-11-03 20:36 ` Corinna Vinschen
0 siblings, 1 reply; 3+ messages in thread
From: Erik Bray @ 2017-11-03 14:39 UTC (permalink / raw)
To: cygwin
[-- Attachment #1: Type: text/plain, Size: 2498 bytes --]
Hi all,
I found a few bugs in Cygwin w.r.t. creating large numbers of sockets.
For example, Cygwin will gladly let you create up to RLIMIT_NOFILE
sockets (examples in Python, where I found this problem):
>>> import resource
>>> import socket
>>> resource.getrlimit(resource.RLIMIT_NOFILE)
(256, 3200)
>>> resource.setrlimit(resource.RLIMIT_NOFILE, (3200, 3200))
>>> socks = [socket.socket() for _ in range(3000)] # A bit fewer than the max but it doesn't matter
However, if I try to do anything interesting with those sockets, such
as poll on them, I get a rather unexpected error:
>>> import select
>>> poll = select.poll()
>>> for sock in socks:
... poll.register(sock, select.POLLOUT)
...
>>> poll.poll()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 14] Bad address
After some playing around I found that I could make up to exactly 1365
sockets and use them without error. At 1366 I get the error. A very
strange and arbitrary number. It turns out this is limited in Cygwin
by the array in fhandler_socket.cc:
496 /* Maximum number of concurrently opened sockets from all Cygwin processes
497 per session. Note that shared sockets (through dup/fork/exec) are
498 counted as one socket. */
499 #define NUM_SOCKS (32768 / sizeof (wsa_event))
...
510 static wsa_event wsa_events[NUM_SOCKS] __attribute__((section
(".cygwin_dll _common"), shared));
This choice for NUM_SOCKS is still seemingly small and pretty
arbitrary, but at least it's a choice, and probably well-motivated.
However, I think it's a problem that it's defined in terms of
sizeof(wsa_event). On 32-bit Cygwin this is 16, so NUM_SOCKS is 2048
(a less strange number), whereas on 64-bit Cygwin sizeof(wsa_event) ==
24 (due to sizeof(long) == 8, plus alignment), so we are limited
to...1365 sockets.
If we have to set a limit I would just hard-code it to 2048 exactly.
I understand that the overhead associated with sockets in Cygwin
probably limits us from having 10s of thousands (much less millions)
and that's OK--I'm not trying to run some kind of C10K challenge on
Cygwin :)
The other problem, then, seems to be a bug in
fhandler_socket::init_events(). It doesn't check the return value of
search_wsa_event_slot(), which returns NULL if the wsa_events array is
full (and the socket is not a shared socket). There's not a great
choice here for error code, but setting ENOBUF seems like the best
option.
Please see attached patch.
Best,
Erik
[-- Attachment #2: 0001-Fix-two-bugs-in-the-limit-of-large-numbers-of-socket.patch --]
[-- Type: application/octet-stream, Size: 1633 bytes --]
From 960e402c14db43130e6054d0fe6a4d0af71b4acf Mon Sep 17 00:00:00 2001
From: "Erik M. Bray" <erik.m.bray@gmail.com>
Date: Fri, 3 Nov 2017 15:33:17 +0100
Subject: [PATCH] Fix two bugs in the limit of large numbers of sockets:
* Fix the maximum number of sockets allowed in the session to 2048,
instead of making it relative to sizeof(wsa_event).
* Return an error and set errno=ENOBUF if a socket can't be created
due to this limit being reached.
---
winsup/cygwin/fhandler_socket.cc | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/winsup/cygwin/fhandler_socket.cc b/winsup/cygwin/fhandler_socket.cc
index 7a6dbdc..b8eda57 100644
--- a/winsup/cygwin/fhandler_socket.cc
+++ b/winsup/cygwin/fhandler_socket.cc
@@ -496,7 +496,7 @@ fhandler_socket::af_local_set_secret (char *buf)
/* Maximum number of concurrently opened sockets from all Cygwin processes
per session. Note that shared sockets (through dup/fork/exec) are
counted as one socket. */
-#define NUM_SOCKS (32768 / sizeof (wsa_event))
+#define NUM_SOCKS ((unsigned int) 2048)
#define LOCK_EVENTS \
if (wsock_mtx && \
@@ -623,7 +623,14 @@ fhandler_socket::init_events ()
NtClose (wsock_mtx);
return false;
}
- wsock_events = search_wsa_event_slot (new_serial_number);
+ if (!(wsock_events = search_wsa_event_slot (new_serial_number)));
+ {
+ set_errno (ENOBUFS);
+ NtClose (wsock_evt);
+ NtClose (wsock_mtx);
+ return false;
+ }
+
/* sock type not yet set here. */
if (pc.dev == FH_UDP || pc.dev == FH_DGRAM)
wsock_events->events = FD_WRITE;
--
2.8.3
[-- Attachment #3: Type: text/plain, Size: 219 bytes --]
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Bug(s) with creating large numbers of sockets
2017-11-03 14:39 Bug(s) with creating large numbers of sockets Erik Bray
@ 2017-11-03 20:36 ` Corinna Vinschen
[not found] ` <CAOTD34ZjH-iwQhCMKgz4B8hRRzXMsbLPT3VEhV9YkjZoTj0oGg@mail.gmail.com>
0 siblings, 1 reply; 3+ messages in thread
From: Corinna Vinschen @ 2017-11-03 20:36 UTC (permalink / raw)
To: cygwin
[-- Attachment #1: Type: text/plain, Size: 3276 bytes --]
Hi Erik,
why didn't you send this to cygwin-patches? Not much of a problem,
just wondering...
On Nov 3 15:39, Erik Bray wrote:
> [...]
> After some playing around I found that I could make up to exactly 1365
> sockets and use them without error. At 1366 I get the error. A very
> strange and arbitrary number. It turns out this is limited in Cygwin
> by the array in fhandler_socket.cc:
>
> 496 /* Maximum number of concurrently opened sockets from all Cygwin processes
> 497 per session. Note that shared sockets (through dup/fork/exec) are
> 498 counted as one socket. */
> 499 #define NUM_SOCKS (32768 / sizeof (wsa_event))
> ...
> 510 static wsa_event wsa_events[NUM_SOCKS] __attribute__((section
> (".cygwin_dll _common"), shared));
>
> This choice for NUM_SOCKS is still seemingly small and pretty
> arbitrary, but at least it's a choice, and probably well-motivated.
This obviously dates back to pre-64 bit times. The idea at the time was
that the .cygwin_dll_common section fits into a single 64K allocation
chunk. Every section takes at least this 64K chunk, so we get away with
only one of them for this section.
Turns out, shared sections have a few issues so we even reduced the
usage and only kept stuff in which doesn't pose much of a problem.
Right now .cygwin_dll_common has a size of 0x8220 / 0x8200 of a max.
of 0x10000, so we have still lots of room and nothing really in need
of using a shared section.
> However, I think it's a problem that it's defined in terms of
> sizeof(wsa_event). On 32-bit Cygwin this is 16, so NUM_SOCKS is 2048
> (a less strange number), whereas on 64-bit Cygwin sizeof(wsa_event) ==
> 24 (due to sizeof(long) == 8, plus alignment), so we are limited
> to...1365 sockets.
>
> If we have to set a limit I would just hard-code it to 2048 exactly.
> I understand that the overhead associated with sockets in Cygwin
> probably limits us from having 10s of thousands (much less millions)
> and that's OK--I'm not trying to run some kind of C10K challenge on
> Cygwin :)
We only need another 0x220 bytes so we have a theoretical 0xfde0 / 64992
bytes or 2708 sockets for wsa_events on 64 bit.
So, yeah, I think you're right, 2048 is good number. As I wrote in the
comment preceeding NUM_SOCKS, these are 2048 independent sockets per
user session in parallel. Sockets created by dup or shared via
fork/exec only take one slot anyway. 2048 independent sockets per user
session should really suffice outside of testcases.
> The other problem, then, seems to be a bug in
> fhandler_socket::init_events(). It doesn't check the return value of
> search_wsa_event_slot(), which returns NULL if the wsa_events array is
> full (and the socket is not a shared socket). There's not a great
> choice here for error code, but setting ENOBUF seems like the best
> option.
Yep, bug. Thanks for catching.
> Please see attached patch.
Can you please send it to cygwin-patches with a bit of additional
comment (just a couple of words) why we choose 2048 here?
Thanks,
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Bug(s) with creating large numbers of sockets
[not found] ` <CAOTD34aGdwfRA2HXKTJGSQDo1sATN8Ji2kdNVR14CzgicFnC-w@mail.gmail.com>
@ 2017-11-03 22:11 ` Erik Bray
0 siblings, 0 replies; 3+ messages in thread
From: Erik Bray @ 2017-11-03 22:11 UTC (permalink / raw)
To: cygwin
On Nov 3, 2017 9:36 PM, "Corinna Vinschen" wrote:
Hi Erik,
why didn't you send this to cygwin-patches? Not much of a problem,
just wondering...
Will follow up more later, but just to clarify on this a bit I thought the
issue was just interesting, and that the question(s) of the upper limit of
sockets allowed by Cygwin was (maybe?) interesting for
discussion/clarification.
But I also meant to send the patch to cygwin-patches and just forgot before
I dashed out the door to go home!
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2017-11-03 22:11 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-11-03 14:39 Bug(s) with creating large numbers of sockets Erik Bray
2017-11-03 20:36 ` Corinna Vinschen
[not found] ` <CAOTD34ZjH-iwQhCMKgz4B8hRRzXMsbLPT3VEhV9YkjZoTj0oGg@mail.gmail.com>
[not found] ` <CAOTD34aGdwfRA2HXKTJGSQDo1sATN8Ji2kdNVR14CzgicFnC-w@mail.gmail.com>
2017-11-03 22:11 ` Erik Bray
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).