python select() is limited to fds < 64

python select() is limited to fds < 64

[Jon TURNEY]

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]


python seems to be built with the default value of FD_SETSIZE, which is only
64 on cygwin.

I noticed this as it causes numerous tests in the twisted test suite to fail
with "ValueError: filedescriptor out of range in select()" exceptions, but can
also be demonstrated with a simple test case:

$ cat select_test.py

from socket import *
from select import select

ins = []

for i in range(1024):
    s = socket(AF_INET, SOCK_STREAM)
    ins.append(s)
    print "socket opened with fd", s.fileno()
    select(ins, [], [], 0)

$ python select_test.py
[...]
socket opened with fd 64
Traceback (most recent call last):
  File "./select_test.py", line 11, in <module>
    select(ins, [], [], 0)
ValueError: filedescriptor out of range in select()


Looking at the source [1], note that steps are already taken to increase the
default value of FD_SETSIZE on Win32, and I'd suggest it's appropriate to do
the same on cygwin, patch attached.

Note some code motion is necessary as FD_SETSIZE must be defined before
sys/types.h is included if we are going to override the default value set there.

I don't believe this can cause any ABI issues as the interface is in terms of
python lists, rather than fd_set values.

[1] http://hg.python.org/cpython/file/3c0edb157ea2/Modules/selectmodule.c


[-- Attachment #2: 2.6.5-FD_SETSIZE.patch --]
[-- Type: text/plain, Size: 1361 bytes --]

--- origsrc/Python-2.6.5/Modules/selectmodule.c	2009-10-27 15:39:53.000000000 +0000
+++ src/Python-2.6.5/Modules/selectmodule.c	2011-03-14 18:25:48.859375000 +0000
@@ -6,6 +6,21 @@
    >= 0.
 */
 
+/* Windows #defines FD_SETSIZE to 64 if FD_SETSIZE isn't already defined.
+   64 is too small (too many people have bumped into that limit).
+   Here we boost it.
+
+   Cygwin also defines FD_SETSIZE to 64, so also increase the limit on
+   Cygwin.  We must do this before sys/types.h is included, which otherwise
+   sets FD_SETSIZE to the default.
+
+   Users who want even more than the boosted limit should #define
+   FD_SETSIZE higher before this; e.g., via compiler /D switch.
+*/
+#if (defined(MS_WINDOWS) || defined(__CYGWIN__)) && !defined(FD_SETSIZE)
+#define FD_SETSIZE 512
+#endif
+
 #include "Python.h"
 #include <structmember.h>
 
@@ -16,16 +31,6 @@
 #undef HAVE_BROKEN_POLL
 #endif
 
-/* Windows #defines FD_SETSIZE to 64 if FD_SETSIZE isn't already defined.
-   64 is too small (too many people have bumped into that limit).
-   Here we boost it.
-   Users who want even more than the boosted limit should #define
-   FD_SETSIZE higher before this; e.g., via compiler /D switch.
-*/
-#if defined(MS_WINDOWS) && !defined(FD_SETSIZE)
-#define FD_SETSIZE 512
-#endif 
-
 #if defined(HAVE_POLL_H)
 #include <poll.h>
 #elif defined(HAVE_SYS_POLL_H)



[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: python select() is limited to fds < 64

[Václav Haisman]

Jon TURNEY wrote, On 22.3.2011 20:29:
> 
> python seems to be built with the default value of FD_SETSIZE, which is only
> 64 on cygwin.
Is this not because of the inherent limitation of WaitForMultipleObjects() call?

> [...]

-- 
VH

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: python select() is limited to fds < 64

[Christopher Faylor]

On Tue, Mar 22, 2011 at 08:53:34PM +0100, V??clav Haisman wrote:
>Jon TURNEY wrote, On 22.3.2011 20:29:
>> 
>> python seems to be built with the default value of FD_SETSIZE, which is only
>> 64 on cygwin.
>Is this not because of the inherent limitation of WaitForMultipleObjects() call?

Yep.  Without a rewrite, it's a hard limit to Cygwin's select().

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: python select() is limited to fds < 64

[Jon TURNEY]

On 22/03/2011 20:08, Christopher Faylor wrote:
> On Tue, Mar 22, 2011 at 08:53:34PM +0100, V??clav Haisman wrote:
>> Jon TURNEY wrote, On 22.3.2011 20:29:
>>>
>>> python seems to be built with the default value of FD_SETSIZE, which is only
>>> 64 on cygwin.
>> Is this not because of the inherent limitation of WaitForMultipleObjects() call?
> 
> Yep.  Without a rewrite, it's a hard limit to Cygwin's select().

Please read my email more closely.  I am not saying "python select() is
limited to waiting on 64 fds or less", I am saying "python select() is limited
to waiting on fd which are less than 64"

Admittedly, the test case isn't the clearest demonstration that this is a
problem, so here's an attempt to clarify:

$ cat select_test_2.py

from socket import *
from select import select

fds = []

for i in range(64):
    s = socket(AF_INET, SOCK_STREAM)
    fds.append(s)
    print "selecting on fd", s.fileno()
    select([s], [], [], 0)

$ python select_test_2.py
selecting on fd 3
[...]
selecting on fd 64
Traceback (most recent call last):
  File "select_test_2.py", line 11, in <module>
    select([s], [], [], 0)
ValueError: filedescriptor out of range in select()

The exception occurs on the attempt to wait on a single fd greater than 64.

Note that the python "filedescriptor out of range in select()" exception is
not an error being propagated from select().  Before python calls select() it
converts the sequence to a fd_set, verifying that each fd is less than
FD_SETSIZE, and raising this exception if it is not.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: python select() is limited to fds < 64

[Christopher Faylor]

On Wed, Mar 23, 2011 at 11:31:46AM +0000, Jon TURNEY wrote:
>On 22/03/2011 20:08, Christopher Faylor wrote:
>>On Tue, Mar 22, 2011 at 08:53:34PM +0100, V??clav Haisman wrote:
>>>Jon TURNEY wrote, On 22.3.2011 20:29:
>>>>
>>>>python seems to be built with the default value of FD_SETSIZE, which is
>>>>only 64 on cygwin.
>>>Is this not because of the inherent limitation of
>>>WaitForMultipleObjects() call?
>>
>>Yep.  Without a rewrite, it's a hard limit to Cygwin's select().
>
>Please read my email more closely.  I am not saying "python select() is
>limited to waiting on 64 fds or less", I am saying "python select() is
>limited to waiting on fd which are less than 64"

I forgot to add one bit of data.  Unless you go out of your way to
change it, Cygwin's select can't wait for an fd > 63.  It's basically a
bit mask.  So, there are two limitations: 1) the number of handles that
you can wait for with WaitForMultipleObjects() and 2) the size of
Cygwin's fd_set.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: python select() is limited to fds < 64

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]

On 03/23/2011 09:00 AM, Christopher Faylor wrote:
> On Wed, Mar 23, 2011 at 11:31:46AM +0000, Jon TURNEY wrote:
>> On 22/03/2011 20:08, Christopher Faylor wrote:
>>> On Tue, Mar 22, 2011 at 08:53:34PM +0100, V??clav Haisman wrote:
>>>> Jon TURNEY wrote, On 22.3.2011 20:29:
>>>>>
>>>>> python seems to be built with the default value of FD_SETSIZE, which is
>>>>> only 64 on cygwin.
>>>> Is this not because of the inherent limitation of
>>>> WaitForMultipleObjects() call?
>>>
>>> Yep.  Without a rewrite, it's a hard limit to Cygwin's select().
>>
>> Please read my email more closely.  I am not saying "python select() is
>> limited to waiting on 64 fds or less", I am saying "python select() is
>> limited to waiting on fd which are less than 64"
> 
> I forgot to add one bit of data.  Unless you go out of your way to
> change it, Cygwin's select can't wait for an fd > 63.  It's basically a
> bit mask.  So, there are two limitations: 1) the number of handles that
> you can wait for with WaitForMultipleObjects() and 2) the size of
> Cygwin's fd_set.

Which is why poll() was invented - it lets you wait on fds > 63
(however, you are still limited by WaitForMultipleObjects that nfds of
poll() must be <= 63).

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: python select() is limited to fds < 64

[Jon TURNEY]

On 23/03/2011 15:00, Christopher Faylor wrote:
> On Wed, Mar 23, 2011 at 11:31:46AM +0000, Jon TURNEY wrote:
>> On 22/03/2011 20:08, Christopher Faylor wrote:
>>> On Tue, Mar 22, 2011 at 08:53:34PM +0100, V??clav Haisman wrote:
>>>> Jon TURNEY wrote, On 22.3.2011 20:29:
>>>>>
>>>>> python seems to be built with the default value of FD_SETSIZE, which is
>>>>> only 64 on cygwin.
>>>> Is this not because of the inherent limitation of
>>>> WaitForMultipleObjects() call?
>>>
>>> Yep.  Without a rewrite, it's a hard limit to Cygwin's select().
>>
>> Please read my email more closely.  I am not saying "python select() is
>> limited to waiting on 64 fds or less", I am saying "python select() is
>> limited to waiting on fd which are less than 64"
> 
> I forgot to add one bit of data.  Unless you go out of your way to
> change it, Cygwin's select can't wait for an fd > 63.  It's basically a
> bit mask.  So, there are two limitations: 1) the number of handles that
> you can wait for with WaitForMultipleObjects() and 2) the size of
> Cygwin's fd_set.

Yes, I know fd_set is implemented as a bitmap.  It's size is controlled by
FD_SETSIZE.  That is the whole point of the patch in the original mail.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: python select() is limited to fds < 64

[Christopher Faylor]

On Wed, Mar 23, 2011 at 04:38:15PM +0000, Jon TURNEY wrote:
>On 23/03/2011 15:00, Christopher Faylor wrote:
>> On Wed, Mar 23, 2011 at 11:31:46AM +0000, Jon TURNEY wrote:
>>> On 22/03/2011 20:08, Christopher Faylor wrote:
>>>> On Tue, Mar 22, 2011 at 08:53:34PM +0100, V??clav Haisman wrote:
>>>>> Jon TURNEY wrote, On 22.3.2011 20:29:
>>>>>>
>>>>>> python seems to be built with the default value of FD_SETSIZE, which is
>>>>>> only 64 on cygwin.
>>>>> Is this not because of the inherent limitation of
>>>>> WaitForMultipleObjects() call?
>>>>
>>>> Yep.  Without a rewrite, it's a hard limit to Cygwin's select().
>>>
>>> Please read my email more closely.  I am not saying "python select() is
>>> limited to waiting on 64 fds or less", I am saying "python select() is
>>> limited to waiting on fd which are less than 64"
>> 
>> I forgot to add one bit of data.  Unless you go out of your way to
>> change it, Cygwin's select can't wait for an fd > 63.  It's basically a
>> bit mask.  So, there are two limitations: 1) the number of handles that
>> you can wait for with WaitForMultipleObjects() and 2) the size of
>> Cygwin's fd_set.
>
>Yes, I know fd_set is implemented as a bitmap.  It's size is controlled by
>FD_SETSIZE.  That is the whole point of the patch in the original mail.

I actually did see that last night and then promptly forgot it.

I assume you know that this greately increases the likelihood that
select() will fail with an EINVAL.  Maybe python somehow deals
gracefully with that.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: python select() is limited to fds < 64

[Jon TURNEY]

On 23/03/2011 20:14, Christopher Faylor wrote:
> On Wed, Mar 23, 2011 at 04:38:15PM +0000, Jon TURNEY wrote:
>> On 23/03/2011 15:00, Christopher Faylor wrote:
>>> On Wed, Mar 23, 2011 at 11:31:46AM +0000, Jon TURNEY wrote:
>>>> On 22/03/2011 20:08, Christopher Faylor wrote:
>>>>> On Tue, Mar 22, 2011 at 08:53:34PM +0100, V??clav Haisman wrote:
>>>>>> Jon TURNEY wrote, On 22.3.2011 20:29:
>>>>>>>
>>>>>>> python seems to be built with the default value of FD_SETSIZE, which is
>>>>>>> only 64 on cygwin.
>>>>>> Is this not because of the inherent limitation of
>>>>>> WaitForMultipleObjects() call?
>>>>>
>>>>> Yep.  Without a rewrite, it's a hard limit to Cygwin's select().
>>>>
>>>> Please read my email more closely.  I am not saying "python select() is
>>>> limited to waiting on 64 fds or less", I am saying "python select() is
>>>> limited to waiting on fd which are less than 64"
>>>
>>> I forgot to add one bit of data.  Unless you go out of your way to
>>> change it, Cygwin's select can't wait for an fd > 63.  It's basically a
>>> bit mask.  So, there are two limitations: 1) the number of handles that
>>> you can wait for with WaitForMultipleObjects() and 2) the size of
>>> Cygwin's fd_set.
>>
>> Yes, I know fd_set is implemented as a bitmap.  It's size is controlled by
>> FD_SETSIZE.  That is the whole point of the patch in the original mail.
> 
> I actually did see that last night and then promptly forgot it.
> 
> I assume you know that this greately increases the likelihood that
> select() will fail with an EINVAL.  Maybe python somehow deals
> gracefully with that.

Well, I believe if select() returns an error, python will raise an exception,
which is no less graceful than current behaviour.

I don't quite follow your reasoning that this change will increase the
likelihood of such exceptions, though.

To select() on more than 64 fds requires using fd numbers 64 and greater.

For existing python programs which use less than 64 fds, this change should
have no effect.

For existing python program which use more than 64 fds, at the moment they
will throw an exception on any attempt to use select() on fd number 64 or
greater.  With this change, attempts to use select() on fd number 64 or
greater will succeed, provided fewer than 64 fds in total are selected on.

However... I did test my original patch, and didn't see an EINVAL error
occurring, which, now knowing that select() has this limitation, I should
have.  So, another test case:

$ cat select_MAXIMUM_WAIT_OBJECTS_test.c

#define FD_SETSIZE 256
#include <sys/select.h>
#include <sys/socket.h>
#include <stdio.h>
#include <errno.h>

//
// gcc select_MAXIMUM_WAIT_OBJECTS_test.c -o select_MAXIMUM_WAIT_OBJECTS_test.exe
//

int main()
{
  fd_set read;
  struct timeval t;
  int i, n;

  FD_ZERO(&read);

  for (i = 0; i < FD_SETSIZE; i++)
    {
      int s = socket(AF_INET, SOCK_STREAM, 0);
      printf("socket opened with fd %d\n", s);
      FD_SET(s, &read);
    }

  t.tv_sec = 1;
  t.tv_usec = 0;

  n = select(FD_SETSIZE, &read, 0, 0, &t);
  printf("select returned %d, errno %d\n", n, errno);

  return 0;

}

$ ./select_MAXIMUM_WAIT_OBJECTS_test
socket opened with fd 3
[...]
socket opened with fd 258
select returned 0, errno 0

Assuming this tests what I think it tests, EINVAL should be returned, but isn't.

Looking at select.cc in the cygwin source code, I can see that
select_stuff::wait() has code to return EINVAL if we exceed
MAXIMUM_WAIT_OBJECTS, so this puzzled me for a while.

Looking at this in a bit more detail, it seems that start_thread_socket()
guzzles all the sockets and starts a single thread, the handle of which it
returns to wait() to WMFO on for every socket fd, even though thread_socket()
will only wait on MAXIMUM_WAIT_OBJECTS at most.

I guess I need to write a patch for this problem, also :-)

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: python select() is limited to fds < 64

[Bert Belder]

Christopher Faylor wrote:
> I forgot to add one bit of data.  Unless you go out of your way to
> change it, Cygwin's select can't wait for an fd > 63.  It's basically a
> bit mask.  So, there are two limitations: 1) the number of handles that
> you can wait for with WaitForMultipleObjects() and 2) the size of
> Cygwin's fd_set.

I have trouble believing this for two reasons:

1. I use node.js on Cygwin, and there seem to be no problems waiting for
readiness of a large (>1000) number of sockets. By default node uses poll()
which calls cygwin_select under the hood, but even when I force it to use
select() directly everything works ok.

2. Inspecting src/winsup/cygwin/select.cc, it looks like Cygwin distributes
the sockets (and other selectable stuff) that it is interested in over
different threads and uses WaitForMultipleObjects in those threads. Then in
the main thread it uses another WaiForMultipleObjects call to synchronize
between those threads. As different interests (read/write/error) and
different fd types require their own thread the fd limit must be somewhere
between a few hundred and  4032 (=64*63).

What did I miss?

Thanks,
Bert




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple