* python fails asyncio tests (py 3.7 & 3.8)
@ 2020-11-21 11:59 Jim Bell
2020-11-22 0:13 ` Jim Bell
0 siblings, 1 reply; 6+ messages in thread
From: Jim Bell @ 2020-11-21 11:59 UTC (permalink / raw)
To: cygwin
The standard python asyncio tests hang.
cd /usr/lib/python3.8/test
python3.8 test_asyncore.py -v
Passes a bunch of tests, then hangs at:
test_connection_attributes (__main__.TestAPI_UseUnixSocketsPoll) ...
(A more focused test run: "python3.8 test_asyncore.py -c -f -k Unix")
Likewise,
python3.8 -m unittest -v test_asyncio
Passes a bunch of tests, then hangs at:
test_create_unix_server
(test.test_asyncio.test_events.PollEventLoopTests) ...
I tried enabling tracing, but then unittest doesn't want to run any
tests ("python3.8 -m trace -c test_asyncore.py" says "Ran 0 tests")
Same for python3.7. I only tested 3.7 & 3.8.
Glad to attach a run of "cygcheck -s -v -r" but it's 1.4M. Running
CygWin-64 under Windows 10 Professional Ver 10.0 Build 19041
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: python fails asyncio tests (py 3.7 & 3.8)
2020-11-21 11:59 python fails asyncio tests (py 3.7 & 3.8) Jim Bell
@ 2020-11-22 0:13 ` Jim Bell
2020-12-01 0:53 ` Mark Geisert
0 siblings, 1 reply; 6+ messages in thread
From: Jim Bell @ 2020-11-22 0:13 UTC (permalink / raw)
To: cygwin
On 2020-11-21 5:59 AM, Jim Bell wrote:
> The standard python asyncio tests hang.
>
> cd /usr/lib/python3.8/test
>
> python3.8 test_asyncore.py -v
>
> [...]
Using strace, stripping down this very repeatable test, and grabbing the
source-code snapshot, it looks like winsup/cygwin/select.cc
socket_cleanup() is waiting forever for the thread to go away.
strace:
121 6732185 [main] python3.8 13329 select_stuff::cleanup: calling
cleanup rout
ines
178 6732363 [main] python3.8 13329 socket_cleanup: si 0x800290E10
si->thread 0
x18023E758
-
But I don't see the select_printf("returning") at select.cc:1808, so
the si->thread->detach() call seems to be blocked forever. I don't see
that we make it here either:
select.cc:1685 select_printf ("leaving thread_socket");
I see a bool stop_thread field in select_info, inherited by
select_socket_info (select.h), used by other mechanisms. Seems that
could be set by socket_cleanup() and used by thread_socket() to break
out of its event loops (select.cc lines 1660 and 1667)
https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/select.cc;hb=HEAD#l1796
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: python fails asyncio tests (py 3.7 & 3.8)
2020-11-22 0:13 ` Jim Bell
@ 2020-12-01 0:53 ` Mark Geisert
2020-12-07 7:58 ` Mark Geisert
0 siblings, 1 reply; 6+ messages in thread
From: Mark Geisert @ 2020-12-01 0:53 UTC (permalink / raw)
To: Jim Bell, cygwin
Hi Jim,
Jim Bell wrote:
> On 2020-11-21 5:59 AM, Jim Bell wrote:
>> The standard python asyncio tests hang.
>>
>> cd /usr/lib/python3.8/test
>>
>> python3.8 test_asyncore.py -v
>>
>> [...]
>
>
> Using strace, stripping down this very repeatable test, and grabbing the
> source-code snapshot, it looks like winsup/cygwin/select.cc socket_cleanup() is
> waiting forever for the thread to go away.
>
> strace:
>
> 121 6732185 [main] python3.8 13329 select_stuff::cleanup: calling cleanup rout
> ines
> 178 6732363 [main] python3.8 13329 socket_cleanup: si 0x800290E10 si->thread 0
> x18023E758
> -
>
> But I don't see the select_printf("returning") at select.cc:1808, so the
> si->thread->detach() call seems to be blocked forever. I don't see that we make
> it here either:
>
> select.cc:1685 select_printf ("leaving thread_socket");
>
> I see a bool stop_thread field in select_info, inherited by select_socket_info
> (select.h), used by other mechanisms. Seems that could be set by socket_cleanup()
> and used by thread_socket() to break out of its event loops (select.cc lines 1660
> and 1667)
>
> https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/select.cc;hb=HEAD#l1796
Thanks for the report and especially for the initial debugging you've done. I've
reproduced the issue on my test machine. No need to supply 'cygcheck -svr' at
this point. I'll investigate this further and keep you posted (on the Cygwin
mailing list).
..mark
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: python fails asyncio tests (py 3.7 & 3.8)
2020-12-01 0:53 ` Mark Geisert
@ 2020-12-07 7:58 ` Mark Geisert
2020-12-07 18:01 ` Marco Atzeri
0 siblings, 1 reply; 6+ messages in thread
From: Mark Geisert @ 2020-12-07 7:58 UTC (permalink / raw)
To: cygwin
[Replying to myself...]
Mark Geisert wrote:
> Jim Bell wrote:
>> On 2020-11-21 5:59 AM, Jim Bell wrote:
>>> The standard python asyncio tests hang.
>>>
>>> cd /usr/lib/python3.8/test
>>>
>>> python3.8 test_asyncore.py -v
>>>
>>> [...]
>> Using strace, stripping down this very repeatable test, and grabbing the
>> source-code snapshot, it looks like winsup/cygwin/select.cc socket_cleanup() is
>> waiting forever for the thread to go away.
>>
>> strace:
>>
>> 121 6732185 [main] python3.8 13329 select_stuff::cleanup: calling cleanup rout
>> ines
>> 178 6732363 [main] python3.8 13329 socket_cleanup: si 0x800290E10 si->thread 0
>> x18023E758
>> -
[...]
> Thanks for the report and especially for the initial debugging you've done. I've
> reproduced the issue on my test machine. No need to supply 'cygcheck -svr' at
> this point. I'll investigate this further and keep you posted (on the Cygwin
> mailing list).
The problem is actually further down in the guts of Cygwin's Unix Domain Socket
handling. Specifically it has to do with the credential passing between client
and server ends of a connection as part of its setup. There is a workaround for
this problem that involves turning off socket option SO_PEERCRED on both the
connecting and listening sockets.
Unfortunately there's another problem. With the current Cygwin DLL build you'll
get an EINVAL error trying to do that setsockopt() operation. I have submitted a
patch that fixes this 2nd problem. A future Cygwin snapshot TBA will contain this
patch.
As for the test script errors you reported, I can submit a workaround patch that
would make its way into the Python3.8 tests. I am unsure at the moment which
Cygwin package contains those tests but I can figure that out. Otherwise, I could
tell you what needs to be patched in test_asyncore.py and you could edit the
script yourself to fix this locally for yourself.
Let me know how you'd like to proceed, when you have a chance.
..mark
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: python fails asyncio tests (py 3.7 & 3.8)
2020-12-07 7:58 ` Mark Geisert
@ 2020-12-07 18:01 ` Marco Atzeri
2020-12-08 3:14 ` Mark Geisert
0 siblings, 1 reply; 6+ messages in thread
From: Marco Atzeri @ 2020-12-07 18:01 UTC (permalink / raw)
To: cygwin
On 07.12.2020 08:58, Mark Geisert wrote:
> [Replying to myself...]
>
> Mark Geisert wrote:
>> Jim Bell wrote:
>>> On 2020-11-21 5:59 AM, Jim Bell wrote:
>>>> The standard python asyncio tests hang.
>>>>
>>>> cd /usr/lib/python3.8/test
>>>>
>>>> python3.8 test_asyncore.py -v
>>>>
> As for the test script errors you reported, I can submit a workaround
> patch that would make its way into the Python3.8 tests. I am unsure at
> the moment which Cygwin package contains those tests but I can figure
> that out. Otherwise, I could tell you what needs to be patched in
> test_asyncore.py and you could edit the script yourself to fix this
> locally for yourself.
>
> Let me know how you'd like to proceed, when you have a chance.
test_asyncore.py is part of the test suite of python itself.
I doubt patching the test will change the functionality of the program
itself
> ..mark
Regards
Marco
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: python fails asyncio tests (py 3.7 & 3.8)
2020-12-07 18:01 ` Marco Atzeri
@ 2020-12-08 3:14 ` Mark Geisert
0 siblings, 0 replies; 6+ messages in thread
From: Mark Geisert @ 2020-12-08 3:14 UTC (permalink / raw)
To: cygwin
Marco Atzeri via Cygwin wrote:
> On 07.12.2020 08:58, Mark Geisert wrote:
>> [Replying to myself...]
>>
>> Mark Geisert wrote:
>>> Jim Bell wrote:
>>>> On 2020-11-21 5:59 AM, Jim Bell wrote:
>>>>> The standard python asyncio tests hang.
>>>>>
>>>>> cd /usr/lib/python3.8/test
>>>>>
>>>>> python3.8 test_asyncore.py -v
>>>>>
>
>> As for the test script errors you reported, I can submit a workaround patch that
>> would make its way into the Python3.8 tests. I am unsure at the moment which
>> Cygwin package contains those tests but I can figure that out. Otherwise, I
>> could tell you what needs to be patched in test_asyncore.py and you could edit
>> the script yourself to fix this locally for yourself.
>>
>> Let me know how you'd like to proceed, when you have a chance.
>
> test_asyncore.py is part of the test suite of python itself.
> I doubt patching the test will change the functionality of the program itself
I agree. I guess I was looking for some input from OP on what direction would be
most useful to him. If he'd like a workaround patch directly in order to make
progress sooner on whatever direction he's going, I can provide it. Otherwise I
would engage with the Python maintainer /* pause, looks up maintainer, oh Hi
Marco! */ to get the patch into Cygwin's Python package(s) for future release in
due course.
Regards,
..mark
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-12-08 3:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-21 11:59 python fails asyncio tests (py 3.7 & 3.8) Jim Bell
2020-11-22 0:13 ` Jim Bell
2020-12-01 0:53 ` Mark Geisert
2020-12-07 7:58 ` Mark Geisert
2020-12-07 18:01 ` Marco Atzeri
2020-12-08 3:14 ` Mark Geisert
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).