Race condition hangs on multiple mintty/tcsh? Brad Wetmore

Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Brad Wetmore]

Hi,

I generally kick off multiple (10) mintty sessions, and place them around the screen.

C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
--position @3 --position 120,0 --size 80x71 /bin/tcsh &
C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
--position @3 --position 715,0 --size 80x45 /bin/tcsh &
C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
--position @3 --position 715,660 --size 80x24 /bin/tcsh &

Within the last 6 months or so, about 2-3 of them would hang and either mintty/tcsh would not start. I put a "sleep 1" in between each invocation and that seemed to take care of it.

With the latest cygwin update, about 8 of them just hang even with the sleep 1. I put in a "sleep 2", and now everything is coming up again.

Not sure if this is a mintty or tcsh issue, but just wondering if others are seeing this before I start trying to debug this.

Thanks,

Brad

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Thomas Wolff]

Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
> Hi,
>
> I generally kick off multiple (10) mintty sessions, and place them around the screen.
>
> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> --position @3 --position 120,0 --size 80x71 /bin/tcsh &
> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> --position @3 --position 715,0 --size 80x45 /bin/tcsh &
> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> --position @3 --position 715,660 --size 80x24 /bin/tcsh &
>
> Within the last 6 months or so, about 2-3 of them would hang and either mintty/tcsh would not start. I put a "sleep 1" in between each invocation and that seemed to take care of it.
>
> With the latest cygwin update, about 8 of them just hang even with the sleep 1. I put in a "sleep 2", and now everything is coming up again.
>
> Not sure if this is a mintty or tcsh issue, but just wondering if others are seeing this before I start trying to debug this.
I can reproduce such behaviour with /bin/bash (easy cross-check), and in 
fact the shell is running in that case (easy test via `echo > .log`), so 
I have a vague and unpleasant suspicion it might in fact be related to 
mintty although I have no idea how that would happen. To be analysed.
Thomas

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Thomas Wolff]

Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
>> Hi,
>>
>> I generally kick off multiple (10) mintty sessions, and place them 
>> around the screen.
>>
>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>> --position @3 --position 120,0 --size 80x71 /bin/tcsh &
>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>> --position @3 --position 715,0 --size 80x45 /bin/tcsh &
>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>> --position @3 --position 715,660 --size 80x24 /bin/tcsh &
>>
>> Within the last 6 months or so, about 2-3 of them would hang and 
>> either mintty/tcsh would not start. I put a "sleep 1" in between each 
>> invocation and that seemed to take care of it.
>>
>> With the latest cygwin update, about 8 of them just hang even with 
>> the sleep 1. I put in a "sleep 2", and now everything is coming up 
>> again.
>>
>> Not sure if this is a mintty or tcsh issue, but just wondering if 
>> others are seeing this before I start trying to debug this.
> I can reproduce such behaviour with /bin/bash (easy cross-check), and 
> in fact the shell is running in that case (easy test via `echo > 
> .log`), so I have a vague and unpleasant suspicion it might in fact be 
> related to mintty although I have no idea how that would happen. To be 
> analysed.
When in this state, on the pty from which mintty receives child process 
output, select() does not report a ready for reading condition; could it 
be related to the recent poll/select patch? I could not reproduce it in 
cygwin 3.0.7.

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Kevin Schnitzius]

On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff <towo@towo.net> wrote: 
>Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
>>> Hi,
>>>
>>> I generally kick off multiple (10) mintty sessions, and place them 
>>> around the screen.
>>>
>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>> --position @3 --position 120,0 --size 80x71 /bin/tcsh &
>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>> --position @3 --position 715,0 --size 80x45 /bin/tcsh &
>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>> --position @3 --position 715,660 --size 80x24 /bin/tcsh &
>>>
>>> Within the last 6 months or so, about 2-3 of them would hang and 
>>> either mintty/tcsh would not start. I put a "sleep 1" in between each 
>>> invocation and that seemed to take care of it.
>>>
>>> With the latest cygwin update, about 8 of them just hang even with 
>>> the sleep 1. I put in a "sleep 2", and now everything is coming up 
>>> again.
>>>
>>> Not sure if this is a mintty or tcsh issue, but just wondering if 
>>> others are seeing this before I start trying to debug this.
>> I can reproduce such behaviour with /bin/bash (easy cross-check), and 
>> in fact the shell is running in that case (easy test via `echo > 
>> .log`), so I have a vague and unpleasant suspicion it might in fact be 
>> related to mintty although I have no idea how that would happen. To be 
>> analysed.
> When in this state, on the pty from which mintty receives child process 
> output, select() does not report a ready for reading condition; could it 
> be related to the recent poll/select patch? I could not reproduce it in 
> cygwin 3.0.7.

for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ; do mintty  -i /Cygwin-Terminal.ico  --position @3 --position 120,$i --size 80x24 /bin/bash & done

This does not fail for me.  Nor does it fail from cmd or powershell using a script.

I tried 3.1.4 and 3.1.6 on Windows 10.  I tried up to 100 instances of of mintty...


Kevin

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Thomas Wolff]

Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff <towo@towo.net> wrote:
>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
>>>> Hi,
>>>>
>>>> I generally kick off multiple (10) mintty sessions, and place them
>>>> around the screen.
>>>>
>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>> --position @3 --position 120,0 --size 80x71 /bin/tcsh &
>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>>> --position @3 --position 715,0 --size 80x45 /bin/tcsh &
>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>>> --position @3 --position 715,660 --size 80x24 /bin/tcsh &
>>>>
>>>> Within the last 6 months or so, about 2-3 of them would hang and
>>>> either mintty/tcsh would not start. I put a "sleep 1" in between each
>>>> invocation and that seemed to take care of it.
>>>>
>>>> With the latest cygwin update, about 8 of them just hang even with
>>>> the sleep 1. I put in a "sleep 2", and now everything is coming up
>>>> again.
>>>>
>>>> Not sure if this is a mintty or tcsh issue, but just wondering if
>>>> others are seeing this before I start trying to debug this.
>>> I can reproduce such behaviour with /bin/bash (easy cross-check), and
>>> in fact the shell is running in that case (easy test via `echo >
>>> .log`), so I have a vague and unpleasant suspicion it might in fact be
>>> related to mintty although I have no idea how that would happen. To be
>>> analysed.
>> When in this state, on the pty from which mintty receives child process
>> output, select() does not report a ready for reading condition; could it
>> be related to the recent poll/select patch? I could not reproduce it in
>> cygwin 3.0.7.
> for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ; do mintty  -i /Cygwin-Terminal.ico  --position @3 --position 120,$i --size 80x24 /bin/bash & done
>
> This does not fail for me.  Nor does it fail from cmd or powershell using a script.
>
> I tried 3.1.4 and 3.1.6 on Windows 10.  I tried up to 100 instances of of mintty...
Thanks for testing. It is in fact hard to reproduce, maybe also 
depending on system load (speculating).
I could reproduce one case of one of three terminals being unresponsive 
also with xterm.
Does anybody familiar with pty/select or recent changes have any idea?
Thomas

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Thomas Wolff]

Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
> Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
>> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff 
>> <towo@towo.net> wrote:
>>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
>>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
>>>>> Hi,
>>>>>
>>>>> I generally kick off multiple (10) mintty sessions, and place them
>>>>> around the screen.
>>>>>
>>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>>> --position @3 --position 120,0 --size 80x71 /bin/tcsh &
>>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>>>> --position @3 --position 715,0 --size 80x45 /bin/tcsh &
>>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>>>> --position @3 --position 715,660 --size 80x24 /bin/tcsh &
>>>>>
>>>>> Within the last 6 months or so, about 2-3 of them would hang and
>>>>> either mintty/tcsh would not start. I put a "sleep 1" in between each
>>>>> invocation and that seemed to take care of it.
>>>>>
>>>>> With the latest cygwin update, about 8 of them just hang even with
>>>>> the sleep 1. I put in a "sleep 2", and now everything is coming up
>>>>> again.
>>>>>
>>>>> Not sure if this is a mintty or tcsh issue, but just wondering if
>>>>> others are seeing this before I start trying to debug this.
>>>> I can reproduce such behaviour with /bin/bash (easy cross-check), and
>>>> in fact the shell is running in that case (easy test via `echo >
>>>> .log`), so I have a vague and unpleasant suspicion it might in fact be
>>>> related to mintty although I have no idea how that would happen. To be
>>>> analysed.
>>> When in this state, on the pty from which mintty receives child process
>>> output, select() does not report a ready for reading condition; 
>>> could it
>>> be related to the recent poll/select patch? I could not reproduce it in
>>> cygwin 3.0.7.
>> for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ; do 
>> mintty  -i /Cygwin-Terminal.ico  --position @3 --position 120,$i 
>> --size 80x24 /bin/bash & done
>>
>> This does not fail for me.  Nor does it fail from cmd or powershell 
>> using a script.
>>
>> I tried 3.1.4 and 3.1.6 on Windows 10.  I tried up to 100 instances 
>> of of mintty...
> Thanks for testing. It is in fact hard to reproduce, maybe also 
> depending on system load (speculating).
> I could reproduce one case of one of three terminals being 
> unresponsive also with xterm.
And it also happens if I drop select() from mintty (and use just 
non-blocking read()).
> Does anybody familiar with pty/select or recent changes have any idea?
?
> Thomas

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Corinna Vinschen]

On Aug  6 15:31, Thomas Wolff wrote:
> Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
> > Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
> > > On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff
> > > <towo@towo.net> wrote:
> > > > Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
> > > > > Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
> > > > > > Hi,
> > > > > > 
> > > > > > I generally kick off multiple (10) mintty sessions, and place them
> > > > > > around the screen.
> > > > > > 
> > > > > > C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> > > > > --position @3 --position 120,0 --size 80x71 /bin/tcsh &
> > > > > > C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> > > > > > --position @3 --position 715,0 --size 80x45 /bin/tcsh &
> > > > > > C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> > > > > > --position @3 --position 715,660 --size 80x24 /bin/tcsh &
> > > > > > 
> > > > > > Within the last 6 months or so, about 2-3 of them would hang and
> > > > > > either mintty/tcsh would not start. I put a "sleep 1" in between each
> > > > > > invocation and that seemed to take care of it.
> > > > > > 
> > > > > > With the latest cygwin update, about 8 of them just hang even with
> > > > > > the sleep 1. I put in a "sleep 2", and now everything is coming up
> > > > > > again.
> > > > > > 
> > > > > > Not sure if this is a mintty or tcsh issue, but just wondering if
> > > > > > others are seeing this before I start trying to debug this.
> > > > > I can reproduce such behaviour with /bin/bash (easy cross-check), and
> > > > > in fact the shell is running in that case (easy test via `echo >
> > > > > .log`), so I have a vague and unpleasant suspicion it might in fact be
> > > > > related to mintty although I have no idea how that would happen. To be
> > > > > analysed.
> > > > When in this state, on the pty from which mintty receives child process
> > > > output, select() does not report a ready for reading condition;
> > > > could it
> > > > be related to the recent poll/select patch? I could not reproduce it in
> > > > cygwin 3.0.7.
> > > for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ; do
> > > mintty  -i /Cygwin-Terminal.ico  --position @3 --position 120,$i
> > > --size 80x24 /bin/bash & done
> > > 
> > > This does not fail for me.  Nor does it fail from cmd or powershell
> > > using a script.
> > > 
> > > I tried 3.1.4 and 3.1.6 on Windows 10.  I tried up to 100 instances
> > > of of mintty...
> > Thanks for testing. It is in fact hard to reproduce, maybe also
> > depending on system load (speculating).
> > I could reproduce one case of one of three terminals being unresponsive
> > also with xterm.
> And it also happens if I drop select() from mintty (and use just
> non-blocking read()).
> > Does anybody familiar with pty/select or recent changes have any idea?
> ?

Something with the pseudo console?  Can you test w/o pseudo console code,
just to check?

Other than that we probably need some bisecting to find the cause.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Thomas Wolff]

Am 06.08.2020 um 16:26 schrieb Corinna Vinschen:
> On Aug  6 15:31, Thomas Wolff wrote:
>> Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
>>> Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
>>>> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff
>>>> <towo@towo.net> wrote:
>>>>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
>>>>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I generally kick off multiple (10) mintty sessions, and place them
>>>>>>> around the screen.
>>>>>>>
>>>>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>>>>> --position @3 --position 120,0 --size 80x71 /bin/tcsh &
>>>>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>>>>>> --position @3 --position 715,0 --size 80x45 /bin/tcsh &
>>>>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
>>>>>>> --position @3 --position 715,660 --size 80x24 /bin/tcsh &
>>>>>>>
>>>>>>> Within the last 6 months or so, about 2-3 of them would hang and
>>>>>>> either mintty/tcsh would not start. I put a "sleep 1" in between each
>>>>>>> invocation and that seemed to take care of it.
>>>>>>>
>>>>>>> With the latest cygwin update, about 8 of them just hang even with
>>>>>>> the sleep 1. I put in a "sleep 2", and now everything is coming up
>>>>>>> again.
>>>>>>>
>>>>>>> Not sure if this is a mintty or tcsh issue, but just wondering if
>>>>>>> others are seeing this before I start trying to debug this.
>>>>>> I can reproduce such behaviour with /bin/bash (easy cross-check), and
>>>>>> in fact the shell is running in that case (easy test via `echo >
>>>>>> .log`), so I have a vague and unpleasant suspicion it might in fact be
>>>>>> related to mintty although I have no idea how that would happen. To be
>>>>>> analysed.
>>>>> When in this state, on the pty from which mintty receives child process
>>>>> output, select() does not report a ready for reading condition;
>>>>> could it
>>>>> be related to the recent poll/select patch? I could not reproduce it in
>>>>> cygwin 3.0.7.
>>>> for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ; do
>>>> mintty  -i /Cygwin-Terminal.ico  --position @3 --position 120,$i
>>>> --size 80x24 /bin/bash & done
>>>>
>>>> This does not fail for me.  Nor does it fail from cmd or powershell
>>>> using a script.
>>>>
>>>> I tried 3.1.4 and 3.1.6 on Windows 10.  I tried up to 100 instances
>>>> of of mintty...
>>> Thanks for testing. It is in fact hard to reproduce, maybe also
>>> depending on system load (speculating).
>>> I could reproduce one case of one of three terminals being unresponsive
>>> also with xterm.
>> And it also happens if I drop select() from mintty (and use just
>> non-blocking read()).
>>> Does anybody familiar with pty/select or recent changes have any idea?
>> ?
> Something with the pseudo console?  Can you test w/o pseudo console code, just to check?
Hmm, CYGWIN=disable_pcon does not seem to disable ConPTY support for me 
any more...
>
> Other than that we probably need some bisecting to find the cause.

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Brad Wetmore]

[I apologize if this doesn't thread correctly:  I've been getting the digest, and can't respond directly.  And I have no idea how my name ended up in the Subject line:  Rookie Move!]

More debug information, and a workaround/solution:

I am running:

    Windows 10 Version 1909, fully patched.
    Cygwin v3.1.6-1
    Openssh v8.3p1-1
    mintty v3.2.0-1
    tcsh v6.22.02-1

The issue seems to be a race condition setting up ssh-agent, and then immediately starting up the the mintty/tcsh terminals. My guess is that something in ssh-agent is not fully setup yet, and is causing read hiccups in mintty.

My startup script looks like:

     #!/bin/sh

    if [ -z "$SSH_AGENT_PID" ] ; then
         echo "Starting ssh-agent"
         eval $(ssh-agent)
         ssh-add
         #sleep 3
     fi

    C:/cygwin64/bin/mintty.exe -i /usr/share/mintty/icon/wsl.ico \
         --position 0,0 --size 80x24 --fg black --bg white /bin/tcsh &
     #sleep 1
     ...9 more invocations deleted...

Without the sleep 3 after starting ssh-agent, I see one of three conditions:

1. The mintty window is created, but immediately crashes printing a dozen or so "Use "exit" to leave tcsh" before it does. At least that I think it says, it happens fast! ;)

2. The mintty window is created, and is ready for input. On the first keypress, it prints a dozen or so "Use "exit" to leave tcsh" and crashes.

3. The mintty window is created, and hangs. No prompt.

I get conditions #1/2 when I don't pause between mintty invocations.
I get condition #2 when I pause for 1 second between mintty invocations.

If I pause for 3 seconds after starting ssh-agent, then everything works as expected.

Note this issue occurs only during the first setup of 10 terminals. Running it again does not tickle the issue, probably because ssh-agent is fully up.

Hope this helps.

Brad

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Thomas Wolff]

Am 06.08.2020 um 20:07 schrieb Brad Wetmore via Cygwin:
> [I apologize if this doesn't thread correctly:  I've been getting the digest, and can't respond directly.  And I have no idea how my name ended up in the Subject line:  Rookie Move!]
>
> More debug information, and a workaround/solution:
>
> I am running:
>
>      Windows 10 Version 1909, fully patched.
>      Cygwin v3.1.6-1
>      Openssh v8.3p1-1
>      mintty v3.2.0-1
>      tcsh v6.22.02-1
>
> The issue seems to be a race condition setting up ssh-agent, and then immediately starting up the the mintty/tcsh terminals. My guess is that something in ssh-agent is not fully setup yet, and is causing read hiccups in mintty.
>
> My startup script looks like:
>
>       #!/bin/sh
>
>      if [ -z "$SSH_AGENT_PID" ] ; then
>           echo "Starting ssh-agent"
>           eval $(ssh-agent)
>           ssh-add
>           #sleep 3
>       fi
>
>      C:/cygwin64/bin/mintty.exe -i /usr/share/mintty/icon/wsl.ico \
>           --position 0,0 --size 80x24 --fg black --bg white /bin/tcsh &
>       #sleep 1
>       ...9 more invocations deleted...
>
> Without the sleep 3 after starting ssh-agent, I see one of three conditions:
>
> 1. The mintty window is created, but immediately crashes printing a dozen or so "Use "exit" to leave tcsh" before it does. At least that I think it says, it happens fast! ;)
>
> 2. The mintty window is created, and is ready for input. On the first keypress, it prints a dozen or so "Use "exit" to leave tcsh" and crashes.
>
> 3. The mintty window is created, and hangs. No prompt.
>
> I get conditions #1/2 when I don't pause between mintty invocations.
> I get condition #2 when I pause for 1 second between mintty invocations.
>
> If I pause for 3 seconds after starting ssh-agent, then everything works as expected.
>
> Note this issue occurs only during the first setup of 10 terminals. Running it again does not tickle the issue, probably because ssh-agent is fully up.
>
> Hope this helps.
You had previously reported the issue without ssh involved. Does this 
make it reproducible more often? Otherwise let's not make things 
unnecessarily complicated.

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Brad Wetmore]

Thomas Wolff wrote:
> You had previously reported the issue without ssh involved. Does this
> make it reproducible more often? Otherwise let's not make things
> unnecessarily complicated.

I didn't mention it originally because I didn't expect that would have an impact, but then I successfully tried 4 times with the ssh-agent/sleep 3.  That's when I sent the email with what seemed to be a good workaround.

But after trying it the 5th time following your email, 1 of the 10 didn't pass.  🙁
I took out the ssh, and am seeing it there too, so that was not the issue.  Drat.

So the delay helped, but didn't completely solve the problem.  I have been stalling before starting the cygwin termincal/script for some minutes after a reboot, but that doesn't seem to make a difference either.

I can send you the script if you want to play with it.

Brad

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Takashi Yano]

Hi Thomas,

On Thu, 6 Aug 2020 16:33:36 +0200
Thomas Wolff wrote:
> Am 06.08.2020 um 16:26 schrieb Corinna Vinschen:
> > On Aug  6 15:31, Thomas Wolff wrote:
> >> Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
> >>> Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
> >>>> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff
> >>>> <towo@towo.net> wrote:
> >>>>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
> >>>>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I generally kick off multiple (10) mintty sessions, and place them
> >>>>>>> around the screen.
> >>>>>>>
> >>>>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> >>>>>> --position @3 --position 120,0 --size 80x71 /bin/tcsh &
> >>>>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> >>>>>>> --position @3 --position 715,0 --size 80x45 /bin/tcsh &
> >>>>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> >>>>>>> --position @3 --position 715,660 --size 80x24 /bin/tcsh &
> >>>>>>>
> >>>>>>> Within the last 6 months or so, about 2-3 of them would hang and
> >>>>>>> either mintty/tcsh would not start. I put a "sleep 1" in between each
> >>>>>>> invocation and that seemed to take care of it.
> >>>>>>>
> >>>>>>> With the latest cygwin update, about 8 of them just hang even with
> >>>>>>> the sleep 1. I put in a "sleep 2", and now everything is coming up
> >>>>>>> again.
> >>>>>>>
> >>>>>>> Not sure if this is a mintty or tcsh issue, but just wondering if
> >>>>>>> others are seeing this before I start trying to debug this.
> >>>>>> I can reproduce such behaviour with /bin/bash (easy cross-check), and
> >>>>>> in fact the shell is running in that case (easy test via `echo >
> >>>>>> .log`), so I have a vague and unpleasant suspicion it might in fact be
> >>>>>> related to mintty although I have no idea how that would happen. To be
> >>>>>> analysed.
> >>>>> When in this state, on the pty from which mintty receives child process
> >>>>> output, select() does not report a ready for reading condition;
> >>>>> could it
> >>>>> be related to the recent poll/select patch? I could not reproduce it in
> >>>>> cygwin 3.0.7.
> >>>> for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ; do
> >>>> mintty  -i /Cygwin-Terminal.ico  --position @3 --position 120,$i
> >>>> --size 80x24 /bin/bash & done
> >>>>
> >>>> This does not fail for me.  Nor does it fail from cmd or powershell
> >>>> using a script.
> >>>>
> >>>> I tried 3.1.4 and 3.1.6 on Windows 10.  I tried up to 100 instances
> >>>> of of mintty...
> >>> Thanks for testing. It is in fact hard to reproduce, maybe also
> >>> depending on system load (speculating).
> >>> I could reproduce one case of one of three terminals being unresponsive
> >>> also with xterm.
> >> And it also happens if I drop select() from mintty (and use just
> >> non-blocking read()).
> >>> Does anybody familiar with pty/select or recent changes have any idea?
> >> ?
> > Something with the pseudo console?  Can you test w/o pseudo console code, just to check?
> Hmm, CYGWIN=disable_pcon does not seem to disable ConPTY support for me 
> any more...

CYGWIN=disable_pcon needs to be set before opening pty, i.e.
before opening mintty. It works for me with cygwin 3.1.6.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Takashi Yano]

[-- Attachment #1: Type: text/plain, Size: 3034 bytes --]

Hi Thomas,

On Thu, 6 Aug 2020 15:31:24 +0200
Thomas Wolff wrote:
> Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
> > Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
> >> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff 
> >> <towo@towo.net> wrote:
> >>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
> >>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
> >>>>> Hi,
> >>>>>
> >>>>> I generally kick off multiple (10) mintty sessions, and place them
> >>>>> around the screen.
> >>>>>
> >>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> >>>> --position @3 --position 120,0 --size 80x71 /bin/tcsh &
> >>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> >>>>> --position @3 --position 715,0 --size 80x45 /bin/tcsh &
> >>>>> C:/cygwin64/bin/mintty.exe -i /Cygwin-Terminal.ico \
> >>>>> --position @3 --position 715,660 --size 80x24 /bin/tcsh &
> >>>>>
> >>>>> Within the last 6 months or so, about 2-3 of them would hang and
> >>>>> either mintty/tcsh would not start. I put a "sleep 1" in between each
> >>>>> invocation and that seemed to take care of it.
> >>>>>
> >>>>> With the latest cygwin update, about 8 of them just hang even with
> >>>>> the sleep 1. I put in a "sleep 2", and now everything is coming up
> >>>>> again.
> >>>>>
> >>>>> Not sure if this is a mintty or tcsh issue, but just wondering if
> >>>>> others are seeing this before I start trying to debug this.
> >>>> I can reproduce such behaviour with /bin/bash (easy cross-check), and
> >>>> in fact the shell is running in that case (easy test via `echo >
> >>>> .log`), so I have a vague and unpleasant suspicion it might in fact be
> >>>> related to mintty although I have no idea how that would happen. To be
> >>>> analysed.
> >>> When in this state, on the pty from which mintty receives child process
> >>> output, select() does not report a ready for reading condition; 
> >>> could it
> >>> be related to the recent poll/select patch? I could not reproduce it in
> >>> cygwin 3.0.7.
> >> for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ; do 
> >> mintty  -i /Cygwin-Terminal.ico  --position @3 --position 120,$i 
> >> --size 80x24 /bin/bash & done
> >>
> >> This does not fail for me.  Nor does it fail from cmd or powershell 
> >> using a script.
> >>
> >> I tried 3.1.4 and 3.1.6 on Windows 10.  I tried up to 100 instances 
> >> of of mintty...
> > Thanks for testing. It is in fact hard to reproduce, maybe also 
> > depending on system load (speculating).
> > I could reproduce one case of one of three terminals being 
> > unresponsive also with xterm.
> And it also happens if I drop select() from mintty (and use just 
> non-blocking read()).
> > Does anybody familiar with pty/select or recent changes have any idea?

I looked into this problem. After much struggle, I think
I have found a workaround for this issue.

I am not sure why this solves the issue at all, however,
this works for me.

Could you please test a patch attached?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pty-Add-a-workaround-for-issue-of-starting-a-.patch --]
[-- Type: application/octet-stream, Size: 2139 bytes --]

From cc451648bbc7f395fa00cb2b97351d1513827f82 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 11 Aug 2020 13:16:52 +0900
Subject: [PATCH] Cygwin: pty: Add a workaround for issue of starting a lot of
 mintty.

- If a lot of mintty are started in a short time from a mintty, some
  of them hang with empty screen, crash immediately or hang on exiting
  mintty. The following report seems to be related to this issue.
    https://cygwin.com/pipermail/cygwin/2020-August/245751.html
  The cause is not clear at all, but this patch seems to solve the
  issue.
---
 winsup/cygwin/spawn.cc | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/winsup/cygwin/spawn.cc b/winsup/cygwin/spawn.cc
index 840ec4a86..e70ceb86d 100644
--- a/winsup/cygwin/spawn.cc
+++ b/winsup/cygwin/spawn.cc
@@ -177,7 +177,7 @@ find_exec (const char *name, path_conv& buf, const char *search,
 /* Utility for child_info_spawn::worker.  */
 
 static HANDLE
-handle (int fd, bool writing)
+handle (int fd, bool writing, bool iscygwin)
 {
   HANDLE h;
   cygheap_fdget cfd (fd);
@@ -188,6 +188,11 @@ handle (int fd, bool writing)
     h = INVALID_HANDLE_VALUE;
   else if (!writing)
     h = cfd->get_handle ();
+  else if (cfd->get_major () == DEV_PTYS_MAJOR && iscygwin)
+    {
+      fhandler_pty_slave *ptys = (fhandler_pty_slave *)(fhandler_base *) cfd;
+      h = ptys->get_output_handle_cyg ();
+    }
   else
     h = cfd->get_output_handle ();
 
@@ -625,9 +630,11 @@ child_info_spawn::worker (const char *prog_arg, const char *const *argv,
 
       /* Set up needed handles for stdio */
       si.dwFlags = STARTF_USESTDHANDLES;
-      si.hStdInput = handle ((in__stdin < 0 ? 0 : in__stdin), false);
-      si.hStdOutput = handle ((in__stdout < 0 ? 1 : in__stdout), true);
-      si.hStdError = handle (2, true);
+      si.hStdInput = handle ((in__stdin < 0 ? 0 : in__stdin), false,
+			     iscygwin ());
+      si.hStdOutput = handle ((in__stdout < 0 ? 1 : in__stdout), true,
+			      iscygwin ());
+      si.hStdError = handle (2, true, iscygwin ());
 
       si.cb = sizeof (si);
 
-- 
2.27.0

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Corinna Vinschen]

On Aug 11 13:59, Takashi Yano via Cygwin wrote:
> Hi Thomas,
> 
> On Thu, 6 Aug 2020 15:31:24 +0200
> Thomas Wolff wrote:
> > Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
> > > Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
> > >> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff 
> > >> <towo@towo.net> wrote:
> > >>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
> > >>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
> > >>>>> Hi,
> > >>>>>
> > >>>>> I generally kick off multiple (10) mintty sessions, and place them
> > >>>>> around the screen.
> > >>>>> [...]
> > > I could reproduce one case of one of three terminals being 
> > > unresponsive also with xterm.
> > And it also happens if I drop select() from mintty (and use just 
> > non-blocking read()).
> > > Does anybody familiar with pty/select or recent changes have any idea?
> 
> I looked into this problem. After much struggle, I think
> I have found a workaround for this issue.
> 
> I am not sure why this solves the issue at all, however,
> this works for me.
> 
> Could you please test a patch attached?

Shall I commit the patch and create a developer snapshot for testing?


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Takashi Yano]

On Tue, 11 Aug 2020 11:37:58 +0200
Corinna Vinschen wrote:
> On Aug 11 13:59, Takashi Yano via Cygwin wrote:
> > Hi Thomas,
> > 
> > On Thu, 6 Aug 2020 15:31:24 +0200
> > Thomas Wolff wrote:
> > > Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
> > > > Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
> > > >> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff 
> > > >> <towo@towo.net> wrote:
> > > >>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
> > > >>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> I generally kick off multiple (10) mintty sessions, and place them
> > > >>>>> around the screen.
> > > >>>>> [...]
> > > > I could reproduce one case of one of three terminals being 
> > > > unresponsive also with xterm.
> > > And it also happens if I drop select() from mintty (and use just 
> > > non-blocking read()).
> > > > Does anybody familiar with pty/select or recent changes have any idea?
> > 
> > I looked into this problem. After much struggle, I think
> > I have found a workaround for this issue.
> > 
> > I am not sure why this solves the issue at all, however,
> > this works for me.
> > 
> > Could you please test a patch attached?
> 
> Shall I commit the patch and create a developer snapshot for testing?

Thanks so much. That would be more convenient for potential testers.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Corinna Vinschen]

On Aug 11 19:02, Takashi Yano via Cygwin wrote:
> On Tue, 11 Aug 2020 11:37:58 +0200
> Corinna Vinschen wrote:
> > On Aug 11 13:59, Takashi Yano via Cygwin wrote:
> > > Hi Thomas,
> > > 
> > > On Thu, 6 Aug 2020 15:31:24 +0200
> > > Thomas Wolff wrote:
> > > > Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
> > > > > Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
> > > > >> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff 
> > > > >> <towo@towo.net> wrote:
> > > > >>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
> > > > >>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
> > > > >>>>> Hi,
> > > > >>>>>
> > > > >>>>> I generally kick off multiple (10) mintty sessions, and place them
> > > > >>>>> around the screen.
> > > > >>>>> [...]
> > > > > I could reproduce one case of one of three terminals being 
> > > > > unresponsive also with xterm.
> > > > And it also happens if I drop select() from mintty (and use just 
> > > > non-blocking read()).
> > > > > Does anybody familiar with pty/select or recent changes have any idea?
> > > 
> > > I looked into this problem. After much struggle, I think
> > > I have found a workaround for this issue.
> > > 
> > > I am not sure why this solves the issue at all, however,
> > > this works for me.
> > > 
> > > Could you please test a patch attached?
> > 
> > Shall I commit the patch and create a developer snapshot for testing?
> 
> Thanks so much. That would be more convenient for potential testers.

Done.  Please try the latest snapshot from https://cygwin.com/snapshots/


Thanks,
Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Thomas Wolff]

On 11.08.2020 12:30, Corinna Vinschen wrote:
> On Aug 11 19:02, Takashi Yano via Cygwin wrote:
>> On Tue, 11 Aug 2020 11:37:58 +0200
>> Corinna Vinschen wrote:
>>> On Aug 11 13:59, Takashi Yano via Cygwin wrote:
>>>> Hi Thomas,
>>>>
>>>> On Thu, 6 Aug 2020 15:31:24 +0200
>>>> Thomas Wolff wrote:
>>>>> Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
>>>>>> Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
>>>>>>> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff
>>>>>>> <towo@towo.net> wrote:
>>>>>>>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
>>>>>>>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I generally kick off multiple (10) mintty sessions, and place them
>>>>>>>>>> around the screen.
>>>>>>>>>> [...]
>>>>>> I could reproduce one case of one of three terminals being
>>>>>> unresponsive also with xterm.
>>>>> And it also happens if I drop select() from mintty (and use just
>>>>> non-blocking read()).
>>>>>> Does anybody familiar with pty/select or recent changes have any idea?
>>>> I looked into this problem. After much struggle, I think
>>>> I have found a workaround for this issue.
>>>>
>>>> I am not sure why this solves the issue at all, however,
>>>> this works for me.
>>>>
>>>> Could you please test a patch attached?
>>> Shall I commit the patch and create a developer snapshot for testing?
>> Thanks so much. That would be more convenient for potential testers.
> Done.  Please try the latest snapshot from https://cygwin.com/snapshots/
Thanks. I could not reproduce the issue on one system in ~27 cases. I'd 
like to retest on another system where I could previously produce the 
issue more frequently, but that's currently broken...
Thomas

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Thomas Wolff]

Am 11.08.2020 um 19:37 schrieb Thomas Wolff:
>
> On 11.08.2020 12:30, Corinna Vinschen wrote:
>> On Aug 11 19:02, Takashi Yano via Cygwin wrote:
>>> On Tue, 11 Aug 2020 11:37:58 +0200
>>> Corinna Vinschen wrote:
>>>> On Aug 11 13:59, Takashi Yano via Cygwin wrote:
>>>>> Hi Thomas,
>>>>>
>>>>> On Thu, 6 Aug 2020 15:31:24 +0200
>>>>> Thomas Wolff wrote:
>>>>>> Am 06.08.2020 um 13:46 schrieb Thomas Wolff:
>>>>>>> Am 06.08.2020 um 01:23 schrieb Kevin Schnitzius via Cygwin:
>>>>>>>> On Wednesday, August 5, 2020, 06:56:48 PM EDT, Thomas Wolff
>>>>>>>> <towo@towo.net> wrote:
>>>>>>>>> Am 04.08.2020 um 12:02 schrieb Thomas Wolff:
>>>>>>>>>> Am 04.08.2020 um 00:13 schrieb Brad Wetmore via Cygwin:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I generally kick off multiple (10) mintty sessions, and 
>>>>>>>>>>> place them
>>>>>>>>>>> around the screen.
>>>>>>>>>>> [...]
>>>>>>> I could reproduce one case of one of three terminals being
>>>>>>> unresponsive also with xterm.
>>>>>> And it also happens if I drop select() from mintty (and use just
>>>>>> non-blocking read()).
>>>>>>> Does anybody familiar with pty/select or recent changes have any 
>>>>>>> idea?
>>>>> I looked into this problem. After much struggle, I think
>>>>> I have found a workaround for this issue.
>>>>>
>>>>> I am not sure why this solves the issue at all, however,
>>>>> this works for me.
>>>>>
>>>>> Could you please test a patch attached?
>>>> Shall I commit the patch and create a developer snapshot for testing?
>>> Thanks so much. That would be more convenient for potential testers.
>> Done.  Please try the latest snapshot from https://cygwin.com/snapshots/
> Thanks. I could not reproduce the issue on one system in ~27 cases. 
> I'd like to retest on another system where I could previously produce 
> the issue more frequently, but that's currently broken...
I gave it another 22 test instances on my other machine (the more 
sensitive one), no issue reproduced with the patch.

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Brad Wetmore]

Hi Carinna/Thomas/Takashi,

> > > > > > Does anybody familiar with pty/select or recent changes have any idea?
> > > >
> > > > I looked into this problem. After much struggle, I think
> > > > I have found a workaround for this issue.
> > > >
> > > > I am not sure why this solves the issue at all, however,
> > > > this works for me.
> > > >
> > > Could you please test a patch attached?
> > >
> > > Shall I commit the patch and create a developer snapshot for testing?
> >
> > Thanks so much. That would be more convenient for potential testers.
>
> Done.  Please try the latest snapshot from https://cygwin.com/snapshots/

Thanks all for looking into this.  I dropped the .dll into my c:\cygwin64\bin directory and wasn't able to reproduce over 5 reboots.  The minute I put back the original .dll, it failed again.

Let me know if there's anything else I can do to help.

Best to you all,

Brad


________________________________
From: Brad Wetmore <bradfordwetmore@hotmail.com>
Sent: Thursday, August 6, 2020 12:25 PM
To: cygwin@cygwin.com <cygwin@cygwin.com>
Subject: Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore


Thomas Wolff wrote:
> You had previously reported the issue without ssh involved. Does this
> make it reproducible more often? Otherwise let's not make things
> unnecessarily complicated.

I didn't mention it originally because I didn't expect that would have an impact, but then I successfully tried 4 times with the ssh-agent/sleep 3.  That's when I sent the email with what seemed to be a good workaround.

But after trying it the 5th time following your email, 1 of the 10 didn't pass.  🙁
I took out the ssh, and am seeing it there too, so that was not the issue.  Drat.

So the delay helped, but didn't completely solve the problem.  I have been stalling before starting the cygwin termincal/script for some minutes after a reboot, but that doesn't seem to make a difference either.

I can send you the script if you want to play with it.

Brad

Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

[Brad Wetmore]

Just installed the latest cygwin (3.1.7-1), and my issue seems to be solved.  Thanks to Thomas, Kevin, Corinna, and Takashi for looking in it.

Brad

________________________________
From: Brad Wetmore <bradfordwetmore@hotmail.com>
Sent: Wednesday, August 12, 2020 8:08 PM
To: cygwin@cygwin.com <cygwin@cygwin.com>
Subject: Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore

Hi Carinna/Thomas/Takashi,

> > > > > > Does anybody familiar with pty/select or recent changes have any idea?
> > > >
> > > > I looked into this problem. After much struggle, I think
> > > > I have found a workaround for this issue.
> > > >
> > > > I am not sure why this solves the issue at all, however,
> > > > this works for me.
> > > >
> > > Could you please test a patch attached?
> > >
> > > Shall I commit the patch and create a developer snapshot for testing?
> >
> > Thanks so much. That would be more convenient for potential testers.
>
> Done.  Please try the latest snapshot from https://cygwin.com/snapshots/

Thanks all for looking into this.  I dropped the .dll into my c:\cygwin64\bin directory and wasn't able to reproduce over 5 reboots.  The minute I put back the original .dll, it failed again.

Let me know if there's anything else I can do to help.

Best to you all,

Brad


________________________________
From: Brad Wetmore <bradfordwetmore@hotmail.com>
Sent: Thursday, August 6, 2020 12:25 PM
To: cygwin@cygwin.com <cygwin@cygwin.com>
Subject: Re: Race condition hangs on multiple mintty/tcsh? Brad Wetmore


Thomas Wolff wrote:
> You had previously reported the issue without ssh involved. Does this
> make it reproducible more often? Otherwise let's not make things
> unnecessarily complicated.

I didn't mention it originally because I didn't expect that would have an impact, but then I successfully tried 4 times with the ssh-agent/sleep 3.  That's when I sent the email with what seemed to be a good workaround.

But after trying it the 5th time following your email, 1 of the 10 didn't pass.  🙁
I took out the ssh, and am seeing it there too, so that was not the issue.  Drat.

So the delay helped, but didn't completely solve the problem.  I have been stalling before starting the cygwin termincal/script for some minutes after a reboot, but that doesn't seem to make a difference either.

I can send you the script if you want to play with it.

Brad