Fix to allow attempted run of non-Win32 EXEs

Fix to allow attempted run of non-Win32 EXEs

[David J. Fiddes]

Hi,

I have been trying to get RTEMS an embedded RTOS to build cleanly with
Cygwin. I ran into a nasty problem. When RTEMS is configuring itself for the
target processor it uses the standard autoconf process to test to see if the
compiler it has been asked to use works and is a cross compiler.

This test involves building a small EXE with the compiler and, if that
succeeds, attempting to run it. If the program runs then the compiler is
native(the normal route for most programs) if it fails to run then it is a
cross-compiler. Unfortunately due to the "flexibility" of NT's process
loader this causes interesting results when you try to load say a m68k
program... What happens is that NT figures out that it is not a PE, NE or
standard DOS EXE and concludes that it is a .COM and just loads the first
64K into memory and runs it. If you are lucky you get the "This program has
executed and invalid instruction" dialog...if you are unlucky it just hangs
in an infinte loop.

Of course on a unix/Linux box the OS is smart enough to reject the EXE.

Having tried the latest snapshot to see if that fixed it I have ended up
hacking the 20.1 source for cygwin so that it checks for the magic 'MZ' at
the start of each program passed to CreateProcess(). This assumes that each
EXE has a DOS stub. It is bound to slow down cygwin but it does allow RTEMS
to build properly.

I don't suppose that this(or a variant) could make it into the official
version... Even if the behaviour had to be turned on with CYGWIN= flag it
would be really useful.

thanks,
Dave

Fix to allow attempted run of non-Win32 EXEs

[David J. Fiddes]

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

Hi,

I have been trying to get RTEMS an embedded RTOS to build cleanly with
Cygwin. I ran into a nasty problem. When RTEMS is configuring itself for the
target processor it uses the standard autoconf process to test to see if the
compiler it has been asked to use works and is a cross compiler.

This test involves building a small EXE with the compiler and, if that
succeeds, attempting to run it. If the program runs then the compiler is
native(the normal route for most programs) if it fails to run then it is a
cross-compiler. Unfortunately due to the "flexibility" of NT's process
loader this causes interesting results when you try to load say a m68k
program... What happens is that NT figures out that it is not a PE, NE or
standard DOS EXE and concludes that it is a .COM and just loads the first
64K into memory and runs it. If you are lucky you get the "This program has
executed and invalid instruction" dialog...if you are unlucky it just hangs
in an infinte loop.

Of course on a unix/Linux box the OS is smart enough to reject the EXE.

Having tried the latest snapshot to see if that fixed it I have ended up
hacking the 20.1 source for cygwin so that it checks for the magic 'MZ' at
the start of each program passed to CreateProcess(). This assumes that each
EXE has a DOS stub. It is bound to slow down cygwin but it does allow RTEMS
to build properly.

I don't suppose that this(or a variant) could make it into the official
version... Even if the behaviour had to be turned on with CYGWIN= flag it
would be really useful.

thanks,
Dave

[-- Attachment #2: cygwin-hack-19990902.diff --]
[-- Type: text/x-diff, Size: 3288 bytes --]

*** spawn.cc.orig	Tue Dec 01 21:24:52 1998
--- spawn.cc	Thu Sep 02 22:09:47 1999
***************
*** 490,496 ****
        si.lpDesktop = wstname;
        /* force the new process to reread /etc/passwd and /etc/group */
        myself->uid = USHRT_MAX;
!       rc = CreateProcessAsUser (hToken,
  		       real_path,	/* image name - with full path */
  		       copy,	/* what was passed to exec */
  		       &sec_all_nih,	/* process security attrs */
--- 490,526 ----
        si.lpDesktop = wstname;
        /* force the new process to reread /etc/passwd and /etc/group */
        myself->uid = USHRT_MAX;
!       
!       /* carry out a quick and dirty hack to check that what we exec is an EXE */
!       HANDLE hnd = CreateFileA (real_path, 
! 				GENERIC_READ,
! 				FILE_SHARE_READ | FILE_SHARE_WRITE,
! 				&sec_none_nih,
! 				OPEN_EXISTING,
! 				FILE_ATTRIBUTE_NORMAL,
! 				0);
!       if (hnd == INVALID_HANDLE_VALUE)
! 	{
! 	  __seterrno ();
! 	  return -1;
! 	}
! 
!       DWORD done;
! 
!       char buf[3];
!       buf[0] = buf[1] = buf[2] = '\0';
!       if (! ReadFile (hnd, buf, sizeof (buf) - 1, &done, 0))
! 	{
! 	  CloseHandle (hnd);
! 	  __seterrno ();
! 	  return -1;
! 	}
! 
!       CloseHandle (hnd);
! 
!       if (buf[0] == 'M' && buf[1] == 'Z')
!         {
!           rc = CreateProcessAsUser (hToken,
  		       real_path,	/* image name - with full path */
  		       copy,	/* what was passed to exec */
  		       &sec_all_nih,	/* process security attrs */
***************
*** 501,509 ****
  		       0,	/* use current drive/directory */
  		       &si,
  		       &pi);
      }
    else
!     rc = CreateProcessA (real_path,	/* image name - with full path */
  		       copy,	/* what was passed to exec */
  		       &sec_all_nih,	/* process security attrs */
  		       &sec_all_nih,	/* thread security attrs */
--- 531,574 ----
  		       0,	/* use current drive/directory */
  		       &si,
  		       &pi);
+         }	       
+       else
+         {
+ 	  rc = 0; /* report an error */
+ 	}
      }
    else
!     {
!       /* carry out a quick and dirty hack to check that what we exec is an EXE */
!       HANDLE hnd = CreateFileA (real_path, 
! 				GENERIC_READ,
! 				FILE_SHARE_READ | FILE_SHARE_WRITE,
! 				&sec_none_nih,
! 				OPEN_EXISTING,
! 				FILE_ATTRIBUTE_NORMAL,
! 				0);
!       if (hnd == INVALID_HANDLE_VALUE)
! 	{
! 	  __seterrno ();
! 	  return -1;
! 	}
! 
!       DWORD done;
! 
!       char buf[3];
!       buf[0] = buf[1] = buf[2] = '\0';
!       if (! ReadFile (hnd, buf, sizeof (buf) - 1, &done, 0))
! 	{
! 	  CloseHandle (hnd);
! 	  __seterrno ();
! 	  return -1;
! 	}
! 
!       CloseHandle (hnd);
! 
!       if (buf[0] == 'M' && buf[1] == 'Z')
!         {
!           rc = CreateProcessA (real_path,	/* image name - with full path */
  		       copy,	/* what was passed to exec */
  		       &sec_all_nih,	/* process security attrs */
  		       &sec_all_nih,	/* thread security attrs */
***************
*** 513,519 ****
--- 578,591 ----
  		       0,	/* use current drive/directory */
  		       &si,
  		       &pi);
+         }	       
+       else
+         {
+ 	  rc = 0; /* report an error */
+ 	}
  
+     }
+     
    free (envblock);
  
    /* Set errno now so that debugging messages from it appear before our