Cyg32 on Win64 fails with stack-protector

Cyg32 on Win64 fails with stack-protector

[Ben RUBSON]

Hi,

We are facing an issue with Cygwin 2.10 32 bits running on Windows 64 bits.

Issue was found running EncFS FUSE FS using WinFsp.

When forking, and calling StartServiceCtrlDispatcher, program fails with  
0xC0000028 / STATUS_BAD_STACK.

Testing several Cygwin versions reveals that issue was introduced between  
2017-11-14 and 2017-12-01.
And Bill (WinFsp author) found that compiling with -fno-stack-protector  
works around the issue.

There you can then find the full story and Bill's nice investigation :
https://github.com/billziss-gh/winfsp/issues/161

Could it be possible something went wrong with Cygwin ?

Thank you very much for your support !

Best regards,

Ben

Re: Cyg32 on Win64 fails with stack-protector

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

Hi Ben,

On Apr 27 19:12, Ben RUBSON wrote:
> Hi,
> 
> We are facing an issue with Cygwin 2.10 32 bits running on Windows 64 bits.
> 
> Issue was found running EncFS FUSE FS using WinFsp.
> 
> When forking, and calling StartServiceCtrlDispatcher, program fails with
> 0xC0000028 / STATUS_BAD_STACK.
> 
> Testing several Cygwin versions reveals that issue was introduced between
> 2017-11-14 and 2017-12-01.
> And Bill (WinFsp author) found that compiling with -fno-stack-protector
> works around the issue.
> 
> There you can then find the full story and Bill's nice investigation :
> https://github.com/billziss-gh/winfsp/issues/161
> 
> Could it be possible something went wrong with Cygwin ?
> 
> Thank you very much for your support !

Even with Bill's additionl info I have no idea what change in the
above time frame might be the culprit, especially since no change
has been made to stack handling.

I suggest to build your own Cygwin and bisect it.  Also, you
may want to run your stuff under GDB and set breakpoints to, e.g.,
Cygwin's exception::handle method.  Make sure the cygwin1.dbg
file can be found by GDB.


HTH,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Cyg32 on Win64 fails with stack-protector

[Bill Zissimopoulos]

Corinna, thanks for your response.

There appear to be some SSP related commits in the discussed time frame.
For example:

https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0a5dfdbd1ba3
663a54fa1a7de1a6c4a0a3316a6e

Another possibility would be a change in the gcc default options between
Cygwin 2.9 and Cygwin 2.10. I do not know if this is possible, but perhaps
gcc in 2.9 compiles with -fstack-protector disabled by default and gcc in
2.10 compiles with it enabled by default?

Bill


On 5/29/18, 9:59 AM, Corinna Vinschen wrote:

>Hi Ben,
>
>On Apr 27 19:12, Ben RUBSON wrote:
>> Hi,
>> 
>> We are facing an issue with Cygwin 2.10 32 bits running on Windows 64
>>bits.
>> 
>> Issue was found running EncFS FUSE FS using WinFsp.
>> 
>> When forking, and calling StartServiceCtrlDispatcher, program fails with
>> 0xC0000028 / STATUS_BAD_STACK.
>> 
>> Testing several Cygwin versions reveals that issue was introduced
>>between
>> 2017-11-14 and 2017-12-01.
>> And Bill (WinFsp author) found that compiling with -fno-stack-protector
>> works around the issue.
>> 
>> There you can then find the full story and Bill's nice investigation :
>> https://github.com/billziss-gh/winfsp/issues/161
>> 
>> Could it be possible something went wrong with Cygwin ?
>> 
>> Thank you very much for your support !
>
>Even with Bill's additionl info I have no idea what change in the
>above time frame might be the culprit, especially since no change
>has been made to stack handling.
>
>I suggest to build your own Cygwin and bisect it.  Also, you
>may want to run your stuff under GDB and set breakpoints to, e.g.,
>Cygwin's exception::handle method.  Make sure the cygwin1.dbg
>file can be found by GDB.
>
>
>HTH,
>Corinna
>
>-- 
>Corinna Vinschen                  Please, send mails regarding Cygwin to
>Cygwin Maintainer                 cygwin AT cygwin DOT com
>Red Hat

Re: Cyg32 on Win64 fails with stack-protector

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2167 bytes --]

On May 30 19:29, Bill Zissimopoulos wrote:
> Corinna, thanks for your response.
> 
> On 5/29/18, 9:59 AM, Corinna Vinschen wrote:
> 
> >Hi Ben,
> >
> >On Apr 27 19:12, Ben RUBSON wrote:
> >> Hi,
> >> 
> >> We are facing an issue with Cygwin 2.10 32 bits running on Windows 64
> >>bits.
> >> 
> >> Issue was found running EncFS FUSE FS using WinFsp.
> >> 
> >> When forking, and calling StartServiceCtrlDispatcher, program fails with
> >> 0xC0000028 / STATUS_BAD_STACK.
> >> 
> >> Testing several Cygwin versions reveals that issue was introduced
> >>between
> >> 2017-11-14 and 2017-12-01.
> >> And Bill (WinFsp author) found that compiling with -fno-stack-protector
> >> works around the issue.
> >> 
> >> There you can then find the full story and Bill's nice investigation :
> >> https://github.com/billziss-gh/winfsp/issues/161
> >> 
> >> Could it be possible something went wrong with Cygwin ?
> >> 
> >> Thank you very much for your support !
> >
> >Even with Bill's additionl info I have no idea what change in the
> >above time frame might be the culprit, especially since no change
> >has been made to stack handling.
> >
> >I suggest to build your own Cygwin and bisect it.  Also, you
> >may want to run your stuff under GDB and set breakpoints to, e.g.,
> >Cygwin's exception::handle method.  Make sure the cygwin1.dbg
> >file can be found by GDB.
> 
> There appear to be some SSP related commits in the discussed time frame.
> For example:
> 
> https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0a5dfdbd1ba3
> 663a54fa1a7de1a6c4a0a3316a6e
> 
> Another possibility would be a change in the gcc default options between
> Cygwin 2.9 and Cygwin 2.10. I do not know if this is possible, but perhaps
> gcc in 2.9 compiles with -fstack-protector disabled by default and gcc in
> 2.10 compiles with it enabled by default?
> 
> Bill

I CCed Ken (gcc) and Yaakov (SSP) to get more qualified input than
my own Cygwin DLL centric view.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Cyg32 on Win64 fails with stack-protector

[Ken Brown]

On 5/31/2018 4:18 AM, Corinna Vinschen wrote:
> On May 30 19:29, Bill Zissimopoulos wrote:
>> Corinna, thanks for your response.
>>
>> On 5/29/18, 9:59 AM, Corinna Vinschen wrote:
>>
>>> Hi Ben,
>>>
>>> On Apr 27 19:12, Ben RUBSON wrote:
>>>> Hi,
>>>>
>>>> We are facing an issue with Cygwin 2.10 32 bits running on Windows 64
>>>> bits.
>>>>
>>>> Issue was found running EncFS FUSE FS using WinFsp.
>>>>
>>>> When forking, and calling StartServiceCtrlDispatcher, program fails with
>>>> 0xC0000028 / STATUS_BAD_STACK.
>>>>
>>>> Testing several Cygwin versions reveals that issue was introduced
>>>> between
>>>> 2017-11-14 and 2017-12-01.
>>>> And Bill (WinFsp author) found that compiling with -fno-stack-protector
>>>> works around the issue.
>>>>
>>>> There you can then find the full story and Bill's nice investigation :
>>>> https://github.com/billziss-gh/winfsp/issues/161
>>>>
>>>> Could it be possible something went wrong with Cygwin ?
>>>>
>>>> Thank you very much for your support !
>>>
>>> Even with Bill's additionl info I have no idea what change in the
>>> above time frame might be the culprit, especially since no change
>>> has been made to stack handling.
>>>
>>> I suggest to build your own Cygwin and bisect it.  Also, you
>>> may want to run your stuff under GDB and set breakpoints to, e.g.,
>>> Cygwin's exception::handle method.  Make sure the cygwin1.dbg
>>> file can be found by GDB.
>>
>> There appear to be some SSP related commits in the discussed time frame.
>> For example:
>>
>> https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0a5dfdbd1ba3
>> 663a54fa1a7de1a6c4a0a3316a6e
>>
>> Another possibility would be a change in the gcc default options between
>> Cygwin 2.9 and Cygwin 2.10. I do not know if this is possible, but perhaps
>> gcc in 2.9 compiles with -fstack-protector disabled by default and gcc in
>> 2.10 compiles with it enabled by default?
>>
>> Bill
> 
> I CCed Ken (gcc) and Yaakov (SSP) to get more qualified input than
> my own Cygwin DLL centric view.

I think you meant to CC a different Ken.

Ken

Re: Cyg32 on Win64 fails with stack-protector

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2621 bytes --]

On May 31 09:13, Ken Brown wrote:
> On 5/31/2018 4:18 AM, Corinna Vinschen wrote:
> > On May 30 19:29, Bill Zissimopoulos wrote:
> > > Corinna, thanks for your response.
> > > 
> > > On 5/29/18, 9:59 AM, Corinna Vinschen wrote:
> > > 
> > > > Hi Ben,
> > > > 
> > > > On Apr 27 19:12, Ben RUBSON wrote:
> > > > > Hi,
> > > > > 
> > > > > We are facing an issue with Cygwin 2.10 32 bits running on Windows 64
> > > > > bits.
> > > > > 
> > > > > Issue was found running EncFS FUSE FS using WinFsp.
> > > > > 
> > > > > When forking, and calling StartServiceCtrlDispatcher, program fails with
> > > > > 0xC0000028 / STATUS_BAD_STACK.
> > > > > 
> > > > > Testing several Cygwin versions reveals that issue was introduced
> > > > > between
> > > > > 2017-11-14 and 2017-12-01.
> > > > > And Bill (WinFsp author) found that compiling with -fno-stack-protector
> > > > > works around the issue.
> > > > > 
> > > > > There you can then find the full story and Bill's nice investigation :
> > > > > https://github.com/billziss-gh/winfsp/issues/161
> > > > > 
> > > > > Could it be possible something went wrong with Cygwin ?
> > > > > 
> > > > > Thank you very much for your support !
> > > > 
> > > > Even with Bill's additionl info I have no idea what change in the
> > > > above time frame might be the culprit, especially since no change
> > > > has been made to stack handling.
> > > > 
> > > > I suggest to build your own Cygwin and bisect it.  Also, you
> > > > may want to run your stuff under GDB and set breakpoints to, e.g.,
> > > > Cygwin's exception::handle method.  Make sure the cygwin1.dbg
> > > > file can be found by GDB.
> > > 
> > > There appear to be some SSP related commits in the discussed time frame.
> > > For example:
> > > 
> > > https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0a5dfdbd1ba3
> > > 663a54fa1a7de1a6c4a0a3316a6e
> > > 
> > > Another possibility would be a change in the gcc default options between
> > > Cygwin 2.9 and Cygwin 2.10. I do not know if this is possible, but perhaps
> > > gcc in 2.9 compiles with -fstack-protector disabled by default and gcc in
> > > 2.10 compiles with it enabled by default?
> > > 
> > > Bill
> > 
> > I CCed Ken (gcc) and Yaakov (SSP) to get more qualified input than
> > my own Cygwin DLL centric view.
> 
> I think you meant to CC a different Ken.

Oops, I meant to CC a JonY, in fact, not a Ken at all.  Sorry!


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Cyg32 on Win64 fails with stack-protector

[JonY]

[-- Attachment #1.1: Type: text/plain, Size: 683 bytes --]

On 06/01/2018 10:06 AM, Corinna Vinschen wrote:
>>>>
>>>> There appear to be some SSP related commits in the discussed time frame.
>>>> For example:
>>>>
>>>> https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0a5dfdbd1ba3
>>>> 663a54fa1a7de1a6c4a0a3316a6e
>>>>
>>>> Another possibility would be a change in the gcc default options between
>>>> Cygwin 2.9 and Cygwin 2.10. I do not know if this is possible, but perhaps
>>>> gcc in 2.9 compiles with -fstack-protector disabled by default and gcc in
>>>> 2.10 compiles with it enabled by default?
>>>>

libssp in gcc was disabled in favor of newlib's, not sure if that has
something to do with it too.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Cyg32 on Win64 fails with stack-protector

[Ben RUBSON]

Bill, Corinna, Jony, thank you very much for your answers and details on  
this topic.
And sorry for my late answer, I was totally stuck on some critical subjects  
last days...

On 01 Jun 2018 12:49, JonY wrote:

> On 06/01/2018 10:06 AM, Corinna Vinschen wrote:
>>>>> There appear to be some SSP related commits in the discussed time  
>>>>> frame.
>>>>> For example:
>>>>>
>>>>> https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0a5dfdbd1ba3
>>>>> 663a54fa1a7de1a6c4a0a3316a6e
>>>>>
>>>>> Another possibility would be a change in the gcc default options  
>>>>> between
>>>>> Cygwin 2.9 and Cygwin 2.10. I do not know if this is possible, but  
>>>>> perhaps
>>>>> gcc in 2.9 compiles with -fstack-protector disabled by default and  
>>>>> gcc in
>>>>> 2.10 compiles with it enabled by default?
>
> libssp in gcc was disabled in favor of newlib's, not sure if that has
> something to do with it too.

Jony, do you mean that compilation options have changed, which could lead  
to our issue ?
Would you manage to find out which change provided this ?
By any chance, between 2017-11-14 and 2017-12-01 ?

Thank you again,

Best regards,

Ben

Re: Cyg32 on Win64 fails with stack-protector

[JonY]

[-- Attachment #1.1: Type: text/plain, Size: 535 bytes --]

On 06/20/2018 04:42 PM, Ben RUBSON wrote:
>>
>> libssp in gcc was disabled in favor of newlib's, not sure if that has
>> something to do with it too.
> 
> Jony, do you mean that compilation options have changed, which could
> lead to our issue ?
> Would you manage to find out which change provided this ?
> By any chance, between 2017-11-14 and 2017-12-01 ?
> 

The SSP implementation has changed, may be related to
https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;h=6a848db44242a24d6570ef7994918c66ce483fd2



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Cyg32 on Win64 fails with stack-protector

[Ben RUBSON]

On 29 May 2018 18:59, Corinna Vinschen wrote:

> Hi Ben,
>
> On Apr 27 19:12, Ben RUBSON wrote:
>> Hi,
>>
>> We are facing an issue with Cygwin 2.10 32 bits running on Windows 64  
>> bits.
>>
>> Issue was found running EncFS FUSE FS using WinFsp.
>>
>> When forking, and calling StartServiceCtrlDispatcher, program fails with
>> 0xC0000028 / STATUS_BAD_STACK.
>>
>> Testing several Cygwin versions reveals that issue was introduced between
>> 2017-11-14 and 2017-12-01.
>> And Bill (WinFsp author) found that compiling with -fno-stack-protector
>> works around the issue.
>>
>> There you can then find the full story and Bill's nice investigation :
>> https://github.com/billziss-gh/winfsp/issues/161
>>
>> Could it be possible something went wrong with Cygwin ?
>>
>> Thank you very much for your support !
>
> Even with Bill's additionl info I have no idea what change in the
> above time frame might be the culprit, especially since no change
> has been made to stack handling.
>
> I suggest to build your own Cygwin and bisect it.

Corinna, could you elaborate please ?
I'm not sure to clearly understand what you mean by bisecting Cygwin.

Thank you !

Ben

Re: Cyg32 on Win64 fails with stack-protector

[Ford, Brian]

She means a binary search of source revisions to isolate when the problem was first introduced. 
-- 
Brian Ford - Systems Architect
FlightSafety Visual Systems 

> On Jun 20, 2018, at 11:51 AM, Ben RUBSON <ben.rubson@gmail.com> wrote:
> 
>> On 29 May 2018 18:59, Corinna Vinschen wrote:
>> 
>> Hi Ben,
>> 
>>> On Apr 27 19:12, Ben RUBSON wrote:
>>> Hi,
>>> 
>>> We are facing an issue with Cygwin 2.10 32 bits running on Windows 64 bits.
>>> 
>>> Issue was found running EncFS FUSE FS using WinFsp.
>>> 
>>> When forking, and calling StartServiceCtrlDispatcher, program fails with
>>> 0xC0000028 / STATUS_BAD_STACK.
>>> 
>>> Testing several Cygwin versions reveals that issue was introduced between
>>> 2017-11-14 and 2017-12-01.
>>> And Bill (WinFsp author) found that compiling with -fno-stack-protector
>>> works around the issue.
>>> 
>>> There you can then find the full story and Bill's nice investigation :
>>> https://github.com/billziss-gh/winfsp/issues/161
>>> 
>>> Could it be possible something went wrong with Cygwin ?
>>> 
>>> Thank you very much for your support !
>> 
>> Even with Bill's additionl info I have no idea what change in the
>> above time frame might be the culprit, especially since no change
>> has been made to stack handling.
>> 
>> I suggest to build your own Cygwin and bisect it.
> 
> Corinna, could you elaborate please ?
> I'm not sure to clearly understand what you mean by bisecting Cygwin.
> 
> Thank you !
> 
> Ben
> 

Re: Cyg32 on Win64 fails with stack-protector

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1748 bytes --]

On Jun 20 17:21, Ford, Brian wrote:
> > On Jun 20, 2018, at 11:51 AM, Ben RUBSON <ben.rubson@gmail.com> wrote:
> > 
> >> On 29 May 2018 18:59, Corinna Vinschen wrote:
> >> 
> >> Hi Ben,
> >> 
> >>> On Apr 27 19:12, Ben RUBSON wrote:
> >>> Hi,
> >>> 
> >>> We are facing an issue with Cygwin 2.10 32 bits running on Windows 64 bits.
> >>> 
> >>> Issue was found running EncFS FUSE FS using WinFsp.
> >>> 
> >>> When forking, and calling StartServiceCtrlDispatcher, program fails with
> >>> 0xC0000028 / STATUS_BAD_STACK.
> >>> 
> >>> Testing several Cygwin versions reveals that issue was introduced between
> >>> 2017-11-14 and 2017-12-01.
> >>> And Bill (WinFsp author) found that compiling with -fno-stack-protector
> >>> works around the issue.
> >>> 
> >>> There you can then find the full story and Bill's nice investigation :
> >>> https://github.com/billziss-gh/winfsp/issues/161
> >>> 
> >>> Could it be possible something went wrong with Cygwin ?
> >>> 
> >>> Thank you very much for your support !
> >> 
> >> Even with Bill's additionl info I have no idea what change in the
> >> above time frame might be the culprit, especially since no change
> >> has been made to stack handling.
> >> 
> >> I suggest to build your own Cygwin and bisect it.
> > 
> > Corinna, could you elaborate please ?
> > I'm not sure to clearly understand what you mean by bisecting Cygwin.
> > 
> > Thank you !
> > 
> > Ben
> > 
> 
> She means a binary search of source revisions to isolate when the problem was first introduced. 

Right, `git bisect'.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Cyg32 on Win64 fails with stack-protector

[Ben RUBSON]

On 29 May 2018 18:59, Corinna Vinschen wrote:

> I suggest to build your own Cygwin and bisect it.

So I tried to bisect and had some difficulties.
I used a fresh 32 bits installation on a 64 bits system and followed this  
howto :
https://cygwin.com/faq/faq.html#faq.programming.building-cygwin

Prior to this commit : 6a848db44242a24d6570ef7994918c66ce483fd2
2017-11-29 - Yaakov Selkowitz - cygwin: export SSP functions
(which is the one JonY suspects)

With gcc 7.3, compilation fails with undefined __stack_chk_fail,  
__stack_chk_guard etc...
With gcc 6.4, compilation succeeds. Cygwin correctly starts with the new  
cygwin1.dll, but then, some programs, such as cmake (cmake --help) do not  
show any output anymore.
Same behavior using Cygwin snapshots. As it worked a few weeks ago, I tried  
to downgrade some runtime libraries, but with no luck.
This is easily reproductible, just install Cygwin32 on a 64-bits system,  
and select Cygwin 2.9 library instead of the 2.10 one.

So, unfortunately, I was not able to test any commit prior to the one given  
above.
This commit however correctly compiles with both gcc 6.4 and 7.3.
Cygwin correctly starts with it, program correctly show their output. And  
I'm able to reproduce the issue I opened this topic for.
So something important has changed between 2017-11-14 and 2017-11-29 which  
triggers this bug.

Perhaps the new SSP mechanism is more powerful than the previous one and  
triggers a long-time existing bug ?

Thank you again,

Ben

Re: Cyg32 on Win64 fails with stack-protector

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1842 bytes --]

Yaakov, can you take a look, please?


Thanks,
Corinna

On Jun 23 15:50, Ben RUBSON wrote:
> On 29 May 2018 18:59, Corinna Vinschen wrote:
> 
> > I suggest to build your own Cygwin and bisect it.
> 
> So I tried to bisect and had some difficulties.
> I used a fresh 32 bits installation on a 64 bits system and followed this
> howto :
> https://cygwin.com/faq/faq.html#faq.programming.building-cygwin
> 
> Prior to this commit : 6a848db44242a24d6570ef7994918c66ce483fd2
> 2017-11-29 - Yaakov Selkowitz - cygwin: export SSP functions
> (which is the one JonY suspects)
> 
> With gcc 7.3, compilation fails with undefined __stack_chk_fail,
> __stack_chk_guard etc...
> With gcc 6.4, compilation succeeds. Cygwin correctly starts with the new
> cygwin1.dll, but then, some programs, such as cmake (cmake --help) do not
> show any output anymore.
> Same behavior using Cygwin snapshots. As it worked a few weeks ago, I tried
> to downgrade some runtime libraries, but with no luck.
> This is easily reproductible, just install Cygwin32 on a 64-bits system, and
> select Cygwin 2.9 library instead of the 2.10 one.
> 
> So, unfortunately, I was not able to test any commit prior to the one given
> above.
> This commit however correctly compiles with both gcc 6.4 and 7.3.
> Cygwin correctly starts with it, program correctly show their output. And
> I'm able to reproduce the issue I opened this topic for.
> So something important has changed between 2017-11-14 and 2017-11-29 which
> triggers this bug.
> 
> Perhaps the new SSP mechanism is more powerful than the previous one and
> triggers a long-time existing bug ?
> 
> Thank you again,
> 
> Ben

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: Cyg32 on Win64 fails with stack-protector

[Carter, Mark Andrew (Andy)]

We use Cyg32 on a Win64 system and have encountered a gcc compiler bug causing stack corruption due to an invalid optimized stack allocation for a function call that return a large structure (40 bytes).  Possibly related to the long standing bug reported here "https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83116".  This bug could bite with compiler version / switch settings or otherwise valid code changes.

Andy Carter  -  Senior Staff Software Engineer
FVS -  FVS Engineering  -  5695 Campus Parkway  -  Hazelwood, MO, 63042  -  United States
Tel: 314.551.8573  
Andy.Carter@FlightSafety.com  -  flightsafety.com  -  A Berkshire Hathaway company

-----Original Message-----
From: cygwin-developers-owner@cygwin.com [mailto:cygwin-developers-owner@cygwin.com] On Behalf Of Corinna Vinschen
Sent: Monday, June 25, 2018 4:45 AM
To: cygwin-developers@cygwin.com
Cc: Yaakov Selkowitz <yselkowi@redhat.com>
Subject: Re: Cyg32 on Win64 fails with stack-protector

Yaakov, can you take a look, please?


Thanks,
Corinna

On Jun 23 15:50, Ben RUBSON wrote:
> On 29 May 2018 18:59, Corinna Vinschen wrote:
> 
> > I suggest to build your own Cygwin and bisect it.
> 
> So I tried to bisect and had some difficulties.
> I used a fresh 32 bits installation on a 64 bits system and followed 
> this howto :
> https://cygwin.com/faq/faq.html#faq.programming.building-cygwin
> 
> Prior to this commit : 6a848db44242a24d6570ef7994918c66ce483fd2
> 2017-11-29 - Yaakov Selkowitz - cygwin: export SSP functions (which is 
> the one JonY suspects)
> 
> With gcc 7.3, compilation fails with undefined __stack_chk_fail, 
> __stack_chk_guard etc...
> With gcc 6.4, compilation succeeds. Cygwin correctly starts with the 
> new cygwin1.dll, but then, some programs, such as cmake (cmake --help) 
> do not show any output anymore.
> Same behavior using Cygwin snapshots. As it worked a few weeks ago, I 
> tried to downgrade some runtime libraries, but with no luck.
> This is easily reproductible, just install Cygwin32 on a 64-bits 
> system, and select Cygwin 2.9 library instead of the 2.10 one.
> 
> So, unfortunately, I was not able to test any commit prior to the one 
> given above.
> This commit however correctly compiles with both gcc 6.4 and 7.3.
> Cygwin correctly starts with it, program correctly show their output. 
> And I'm able to reproduce the issue I opened this topic for.
> So something important has changed between 2017-11-14 and 2017-11-29 
> which triggers this bug.
> 
> Perhaps the new SSP mechanism is more powerful than the previous one 
> and triggers a long-time existing bug ?
> 
> Thank you again,
> 
> Ben

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: Cyg32 on Win64 fails with stack-protector

[Ben RUBSON]

On 25 Jun 2018 17:35, Carter, Mark Andrew (Andy) wrote:

> We use Cyg32 on a Win64 system and have encountered a gcc compiler bug  
> causing stack corruption due to an invalid optimized stack allocation for  
> a function call that return a large structure (40 bytes).  Possibly  
> related to the long standing bug reported here  
> "https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83116".  This bug could  
> bite with compiler version / switch settings or otherwise valid code  
> changes.

Thank you Andy for your help.
About the bug you point to, sounds like it has been corrected in GCC 7.3 :
https://gcc.gnu.org/viewcvs/gcc/tags/gcc_7_3_0_release/gcc/cp/ChangeLog?view=markup#l30
Unfortunately I also face the issue with this GCC version.

Ben

Re: Cyg32 on Win64 fails with stack-protector

[Bill Zissimopoulos]

Ben RUBSON wrote:

> We are facing an issue with Cygwin 2.10 32 bits running on Windows 64
>bits.
> 
> Issue was found running EncFS FUSE FS using WinFsp.
> 
> When forking, and calling StartServiceCtrlDispatcher, program fails with
> 0xC0000028 / STATUS_BAD_STACK.Testing several Cygwin versions reveals
>that issue 
> was introduced between  2017-11-14 and 2017-12-01. And Bill (WinFsp
>author) found > that compiling with -fno-stack-protector  works around
>the issue.There you can
> then find the full story and Bill's nice investigation :
> https://github.com/billziss-gh/winfsp/issues/161
>
> Could it be possible something went wrong with Cygwin ?

I wanted to add a few more details to Ben’s post, which may help us
further understand and resolve this issue. Ben has done most of the
troubleshooting work on this and I was just lucky enough to capture an
illuminating message in the debugger.


The components that participate in the discussed interaction are:

- EncFS: a POSIX/Cygwin program that implements a FUSE file system. This
program may be built with or without fstack-protector.

- Cygfuse: a Cygwin DLL that provides a FUSE binary interface (i.e. the
fuse_* symbols). This library was built with cygport under Cygwin 2.9
32-bit. I am unclear on whether this means that the library was built with
fstack-protector (likely not).

- Winfsp-x86.dll: a native Windows DLL (32-bit). Built with Visual Studio.

- WOW64: the Windows on Windows environment.

- Win64: the Windows 64-bit OS.

There are no kernel mode components (other than Windows' own) involved in
the discussed interaction.


The interactions are:

- EncFS calls fuse_main_real which resides in libcygfuse.

- Cygfuse loads winfsp-x86.dll and forwards the fuse_main_real call to it.

- Winfsp-x86.dll proceeds to bootstrap the file system. If the command
line does not include the -f switch, then daemon(3) is called. This is
done by calling back into the Cygfuse Cygwin DLL.

- Cygfuse calls daemon(3). On return it reloads/reinits winfsp-x86.dll,
because Cygwin's fork does not handle native DLL’s. It then returns
control to (the possibly reloaded!) winfsp-x86.dll.

- Winfsp-x86.dll proceeds to initialize the FUSE file system. It calls
StartServiceCtrlDispatcher in an attempt to convert the process into a
service (I wanted FUSE file systems on Windows to be easily converted into
Windows services. Failure of the StartServiceCtrlDispatcher is benign and
the WinFsp-FUSE infrastructure knows how to handle it.)

Now if EncFS is compiled under Cygwin 2.9 (32-bit or 64-bit) or it is
compiled with fno-stack-protector under Cygwin 2.10 everything works just
fine. If however EncFS is compiled with the default (presumably it
includes fstack-protector) then the EncFS daemonized process dies without
any other information.

Attaching the debugger to the daemonized process reveals that while
unwinding an (likely benign) exception deep inside
StartServiceCtrlDispatcher, RtlUnwindEx (or its friends) determines that
the stack looks wrong and raises STATUS_BAD_STACK, which is rather fatal.


Bill