Windows DEP problem with nested functions

Windows DEP problem with nested functions

[Nicholas Sherlock]

Hi all,

I'm using GCC 3.4.4 on Cygwin on Windows XP. The following program:

#include <pthread.h>

int main() {
        pthread_t thread;

    void * task(void * arg) {
      return NULL;
        }

        pthread_create(&thread, NULL, task, NULL);
        pthread_join(thread, NULL);

    return 0;
}

only runs if DEP (Prevention of code execution on data pages) is turned 
off in Windows settings. I get a segfault if DEP is turned on. Is this a 
GCC problem, or a Cygwin problem?

Cheers,
Nicholas Sherlock

Re: Windows DEP problem with nested functions

[Ian Lance Taylor]

Nicholas Sherlock <n.sherlock@gmail.com> writes:

> I'm using GCC 3.4.4 on Cygwin on Windows XP. The following program:
>
> #include <pthread.h>
>
> int main() {
>        pthread_t thread;
>
>    void * task(void * arg) {
>      return NULL;
>        }
>
>        pthread_create(&thread, NULL, task, NULL);
>        pthread_join(thread, NULL);
>
>    return 0;
> }
>
> only runs if DEP (Prevention of code execution on data pages) is
> turned off in Windows settings. I get a segfault if DEP is turned
> on. Is this a GCC problem, or a Cygwin problem?

On targets which require extra effort to enable stack execution, gcc
is supposed to enable it when it builds a trampoline for a nested
function.  So in that sense it is a gcc problem.

I think this problem is fixed in the mainline gcc sources.  At least,
I see support for calling VirtualProtect on a trampoline.

Ian