fixing BLODA-caused fork failures

fixing BLODA-caused fork failures

[Adam Kellas]

My company uses Cygwin and we experience fairly frequent fork
failures, believed to be BLODA-related. I say "believed to be" because
in this corporate environment, like many, we cannot uninstall the
virus scanner even long enough to see what happens without it. The
presumed culprit in our case is Microsoft Forefront Endpoint
Protection, by the way.

So we need Cygwin and we're stuck with Forefront, putting us between a
rock and a hard place. It's clear from the documentation and mailing
list that the official stance wrt BLODA is "sorry, can't help you" and
I understand and accept that. I'm looking for the answer to a related
question: are BLODA-caused fork failures a logically unsolvable
problem due to the way Windows works or is it just a matter of round
tuits? In other words, if we were (hypothetically) able to pay someone
to make MS Forefront and Cygwin play nicely together, would that have
a chance of success? And would the Cygwin maintainers allow such work
into the code base or consider it an unfortunate precedent?

Thanks,
AK

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fixing BLODA-caused fork failures

[Ryan Johnson]

On 03/10/2013 4:55 PM, Adam Kellas wrote:
> My company uses Cygwin and we experience fairly frequent fork
> failures, believed to be BLODA-related. I say "believed to be" because
> in this corporate environment, like many, we cannot uninstall the
> virus scanner even long enough to see what happens without it. The
> presumed culprit in our case is Microsoft Forefront Endpoint
> Protection, by the way.
>
> So we need Cygwin and we're stuck with Forefront, putting us between a
> rock and a hard place. It's clear from the documentation and mailing
> list that the official stance wrt BLODA is "sorry, can't help you" and
> I understand and accept that. I'm looking for the answer to a related
> question: are BLODA-caused fork failures a logically unsolvable
> problem due to the way Windows works or is it just a matter of round
> tuits? In other words, if we were (hypothetically) able to pay someone
> to make MS Forefront and Cygwin play nicely together, would that have
> a chance of success? And would the Cygwin maintainers allow such work
> into the code base or consider it an unfortunate precedent?
I can't comment on actual code changes, but having played around quite a 
bit in the fork code, I can at least speak a little to the technical part.

BLODA commonly causes two classes of symptoms:

1. Fork failures due to injecting dlls at inopportune locations or 
otherwise messing with address space layouts in a way that violates 
posix semantics (memory-mapping a file where some cygwin dll should have 
gone, for example). Cygwin can do nothing, because the damage is 
invariably done before cygwin1.dll loads. The problem can only be fixed 
if the BLODA were to be more circumspect about where it injects things, 
which is unlikely because Windows processes normally care very little 
(if at all) about address space layouts.

2. Various file-related errors due to the BLODA touching/locking files 
at inopportune times (e.g. right when a cygwin process tries to delete 
the file, leading to an "unable to delete" error). This is simple 
Windows locking at play, which Cygwin actually respects (unlike many 
Windows programs). Again, there's little cygwin can do to address the 
problem on its side, because the file really is locked...

... all of which is the long way of explaining why the official stance 
is "sorry, you're SOL" ...

Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fixing BLODA-caused fork failures

[Larry Hall (Cygwin)]

On 10/3/2013 5:50 PM, Ryan Johnson wrote:
> On 03/10/2013 4:55 PM, Adam Kellas wrote:
>> My company uses Cygwin and we experience fairly frequent fork
>> failures, believed to be BLODA-related. I say "believed to be" because
>> in this corporate environment, like many, we cannot uninstall the
>> virus scanner even long enough to see what happens without it. The
>> presumed culprit in our case is Microsoft Forefront Endpoint
>> Protection, by the way.
>>
>> So we need Cygwin and we're stuck with Forefront, putting us between a
>> rock and a hard place. It's clear from the documentation and mailing
>> list that the official stance wrt BLODA is "sorry, can't help you" and
>> I understand and accept that. I'm looking for the answer to a related
>> question: are BLODA-caused fork failures a logically unsolvable
>> problem due to the way Windows works or is it just a matter of round
>> tuits? In other words, if we were (hypothetically) able to pay someone
>> to make MS Forefront and Cygwin play nicely together, would that have
>> a chance of success? And would the Cygwin maintainers allow such work
>> into the code base or consider it an unfortunate precedent?
> I can't comment on actual code changes, but having played around quite a bit
> in the fork code, I can at least speak a little to the technical part.
>
> BLODA commonly causes two classes of symptoms:
>
> 1. Fork failures due to injecting dlls at inopportune locations or otherwise
> messing with address space layouts in a way that violates posix semantics
> (memory-mapping a file where some cygwin dll should have gone, for example).
> Cygwin can do nothing, because the damage is invariably done before
> cygwin1.dll loads. The problem can only be fixed if the BLODA were to be
> more circumspect about where it injects things, which is unlikely because
> Windows processes normally care very little (if at all) about address space
> layouts.
>
> 2. Various file-related errors due to the BLODA touching/locking files at
> inopportune times (e.g. right when a cygwin process tries to delete the
> file, leading to an "unable to delete" error). This is simple Windows
> locking at play, which Cygwin actually respects (unlike many Windows
> programs). Again, there's little cygwin can do to address the problem on its
> side, because the file really is locked...
>
> ... all of which is the long way of explaining why the official stance is
> "sorry, you're SOL" ...

Nothing I say here should be misconstrued to in any way contradict what Ryan
just said.  Its a very good summary of the difficulties caused by BLODA.
But as a point of practicality, 64-bit Cygwin can help with some cases of
DLL address space collisions.  So if you haven't experimented with
64-bit Cygwin in your environment, it may be worth your time.


-- 
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fixing BLODA-caused fork failures

[Christopher Faylor]

On Thu, Oct 03, 2013 at 04:55:19PM -0400, Adam Kellas wrote:
>My company uses Cygwin and we experience fairly frequent fork
>failures, believed to be BLODA-related. I say "believed to be" because
>in this corporate environment, like many, we cannot uninstall the
>virus scanner even long enough to see what happens without it. The
>presumed culprit in our case is Microsoft Forefront Endpoint
>Protection, by the way.
>
>So we need Cygwin and we're stuck with Forefront, putting us between a
>rock and a hard place. It's clear from the documentation and mailing
>list that the official stance wrt BLODA is "sorry, can't help you" and
>I understand and accept that. I'm looking for the answer to a related
>question: are BLODA-caused fork failures a logically unsolvable
>problem due to the way Windows works or is it just a matter of round
>tuits? In other words, if we were (hypothetically) able to pay someone
>to make MS Forefront and Cygwin play nicely together, would that have
>a chance of success? And would the Cygwin maintainers allow such work
>into the code base or consider it an unfortunate precedent?

It wouldn't be without precedent.  I've done work for companies before
and the work has made it into the code base.  But I can't predict if it
is even possible to make things work better in this case.

Contact me at "me at-sign cgf period cx" and we can discuss the
possibility of at least investigating what's involved.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fixing BLODA-caused fork failures

[Achim Gratz]

Am 03.10.2013 22:55, schrieb Adam Kellas:
> My company uses Cygwin and we experience fairly frequent fork
> failures, believed to be BLODA-related. I say "believed to be" because
> in this corporate environment, like many, we cannot uninstall the
> virus scanner even long enough to see what happens without it.

In my experience the culprit is most often the "real-time" or 
"behavioral" component of the anti virus program.  Uninstalling the 
virus program is a bad idea and it will certainly draw the ire of 
corporate IT, but you may persuade them to exclude the whole Cygwin 
installation tree from these two functions.  In any case I'd try to get 
a test machine from IT and try if that helps.

 > So we need Cygwin and we're stuck with Forefront, putting us
 > between a rock and a hard place.

AFAIK Forefront uses the same engine on the client as MSE.  MSE has not 
been a problem w.r.t. Cygwin for me, so either the rules used by 
Forefront have been sharpened or the fork problems may not be related to 
it after all.  Do you use other programs that are based on Cygwin, like 
NoMachine NX or other (perhaps in-house) programs that insist on putting 
their DLL at fixed address spaces?


Regards,
Achim.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fixing BLODA-caused fork failures

[David Boyce]

> But as a point of practicality, 64-bit Cygwin can help with some cases of
> DLL address space collisions.  So if you haven't experimented with
> 64-bit Cygwin in your environment, it may be worth your time.

This sounds very reasonable but I'm curious: is it to date just a
theory that makes sense or have there been actual reports of 64-bit
Cygwin solving address-space collisions?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fixing BLODA-caused fork failures

[Larry Hall (Cygwin)]

On 10/8/2013 3:25 PM, David Boyce wrote:
>> But as a point of practicality, 64-bit Cygwin can help with some cases of
>> DLL address space collisions.  So if you haven't experimented with
>> 64-bit Cygwin in your environment, it may be worth your time.
>
> This sounds very reasonable but I'm curious: is it to date just a
> theory that makes sense or have there been actual reports of 64-bit
> Cygwin solving address-space collisions?

Solving?  Well no.  It won't "solve" address-space collisions.  But the
larger address space means that there's less chance of the O/S picking
the address your DLL wants for another.  No guarentees of course. ;-)

-- 
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fixing BLODA-caused fork failures

[Ryan Johnson]

On 08/10/2013 3:34 PM, Larry Hall (Cygwin) wrote:
> On 10/8/2013 3:25 PM, David Boyce wrote:
>>> But as a point of practicality, 64-bit Cygwin can help with some 
>>> cases of
>>> DLL address space collisions.  So if you haven't experimented with
>>> 64-bit Cygwin in your environment, it may be worth your time.
>>
>> This sounds very reasonable but I'm curious: is it to date just a
>> theory that makes sense or have there been actual reports of 64-bit
>> Cygwin solving address-space collisions?
>
> Solving?  Well no.  It won't "solve" address-space collisions. But the
> larger address space means that there's less chance of the O/S picking
> the address your DLL wants for another.  No guarentees of course. ;-)
>
One data point: I managed to make it a few weeks from a clean 
win64/cygwin64 install to the first BLODA symptoms, even though Windows 
Defender was alive and well that whole time. Once it hit, though, there 
was no obvoius way out except to disable WD. Rebooting might also help, 
by resetting an especially unfriendly ASLR arrangement.

Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple