RE: [cygwin] Re: Mandatory ASLR breaks Cygwin - Windows 10

["Jason Pyeron" <jpyeron@pdinc.us>]

> -----Original Message-----
> From: Brian Inglis
> Sent: Tuesday, August 25, 2020 11:13 AM
> 
> On 2020-08-25 08:36, Alexandria Cortez wrote:
> > On Tuesday, August 25, 2020 10:35 AM, Eliot Moss wrote:>> On Aug 25, 2020, at 10:17 AM, Alexandria
> Cortez wrote:
> >>> I was experimenting with security settings this morning on windows, and
> >>> after changing Mandatory ASLR (Windows Security -> App and Browser Control
> >>> -> Exploit Protection) to default on, no Cygwin programs that rely on the
> >>> Cygwin dll would start, stating that a resource was temporarily unavailable
> >>> and could not fork. Rebasell, bash, you name it crashed and would not start.
> >>> After some investigation, turning off that setting allows Cygwin to work.
> >>>
> >>> Now the next question: why does this not work? Is this intended behavior or
> >>> a bug? Having that setting turned on seems like a good idea from a security
> >>> standpoint, and who knows it  may eventually become default.
> 
> >> It’s intentional; too long to explain in detail on phone, but fork
> >> requires each dll to load in the child at the same address as in the
> >> parent, and ASLR interferes with achieving that.
> > Is there any plans to implement a workaround in the future? Seeing as Cygwin
> > is only one of two programs I've noticed that are broken with it on, it
> > would be nice to be able to have it on from a security perspective.
> Cygwin is an all-volunteer project - Someone Has To Do It!
> Feel free to submit patches to support that in Cygwin under Windows.
> A low level understanding of details of both is required.

Some relevant reading 

https://stackoverflow.com/questions/8171298/is-address-space-layout-randomization-applied-on-a-forked-process

Is Address Space Layout Randomization applied on a forked process

Q: Say I fork a process from another process. Will Address Space Layout Randomization (ASLR) be applied on it in an OS which has ASLR set?

Note that I am talking about the case where I don't call execve function after doing fork.

A: 12

Yes.

However note that after fork both parent and child have the same randomization applied to them (they are copies of each other after all!).

If the child and parent are to call mmap(NULL, ...), then their address maps will start to diverge.

Update:

> Isn't your statement contradictory?

Not at all. Immediately after fork, the parent and child address spaces are identical (that's the definition of what fork does). But the ASLR is still in effect for both the parent and the child. The randomization can't "go back in time" and randomize decisions that have already been made, but any future decisions (such as where to place next mmap) will be randomized, and will likely result in different outcome for parent and child.

> Does it have to do with basic mmap or OS writers introduce randomness in mmap as well for security?

Perhaps you don't understand what ASLR is?

In short, with ASLR on, the OS will randomize placement of main stack, and placement of any non-MAP_FIXED mmaps.

By the time you fork, the main stack placement has long been determined, so parent and child will have the same. The future mmap are the only things that can (and will be) affected by ASLR going forward.

https://cygwin.com/cygwin-ug-net/highlights.html

Process Creation
The fork call in Cygwin is particularly interesting because it does not map well on top of the Win32 API. This makes it very difficult to implement correctly. Currently, the Cygwin fork is a non-copy-on-write implementation similar to what was present in early flavors of UNIX.

As the child process is created as new process, both the main executable and all the dlls loaded either statically or dynamically have to be identical as to when the parent process has started or loaded a dll. While Windows does not allow to remove binaries in use from the file system, they still can be renamed or moved into the recycle bin, as outlined for unlink(2) in the section called “File Access related changes”. To allow an existing process to fork, the original binary files need to be available via their original file names, but they may reside in a different directory when using the DotLocal (.local) Dll Redirection feature. Since NTFS does support hardlinks, when the fork fails we try again, but create a private directory containing hardlinks to the original files as well as the .local file now. The base directory for the private hardlink directory is /var/run/cygfork/, which you have to create manually for now if you need to protect fork against updates to executables and dlls on your Cygwin instance. As hardlinks cannot be used across multiple NTFS file systems, please make sure your executable and dll replacing operations operate on the same single NTFS file system as your Cygwin instance and the /var/run/cygfork/ directory. Note that this private hardlink directory also does help for when a wrong dll is found in the parent process' current working directory. To enable creating the hardlinks, you need to stop all currently running Cygwin processes after creating this directory, once per Cygwin installation:

$ mkdir --mode=a=rwxt /var/run/cygfork

We create one hardlink directory per user, application and application age, and remove it when no more processes use that directory. To indicate whether a directory still is in use, we define a mutex name similar to the directory name. As mutexes are destroyed when no process holds a handle open any more, we can clean up even after power loss or similar: Both the parent and child process, at exit they lock the mutex with almost no timeout and close it, to get the closure promoted synchronously. If the lock succeeded before closing, directory cleanup is started: For each directory found, the corresponding mutex is created with lock. If that succeeds, the directory is removed, as it is unused now, and the corresponding mutex handle is closed.

Before fork, when about to create hardlinks for the first time, the mutex is opened and locked with infinite timeout, to wait for the cleanup that may run at the same time. Once locked, the mutex is unlocked immediately, but the mutex handle stays open until exit, and the hardlinks are created. It is fine for multiple processes to concurrently create the same hardlinks, as the result really should be identical. Once the mutex is open, we can create more hardlinks within this one directory without the need to lock the mutex again.

The first thing that happens when a parent process forks a child process is that the parent initializes a space in the Cygwin process table for the child. It then creates a suspended child process using the Win32 CreateProcess call. Next, the parent process calls setjmp to save its own context and sets a pointer to this in a Cygwin shared memory area (shared among all Cygwin tasks). It then fills in the child's .data and .bss sections by copying from its own address space into the suspended child's address space. After the child's address space is initialized, the child is run while the parent waits on a mutex. The child discovers it has been forked and longjumps using the saved jump buffer. The child then sets the mutex the parent is waiting on and blocks on another mutex. This is the signal for the parent to copy its stack and heap into the child, after which it releases the mutex the child is waiting on and returns from the fork call. Finally, the child wakes from blocking on the last mutex, recreates any memory-mapped areas passed to it via the shared area, and returns from fork itself.

While we have some ideas as to how to speed up our fork implementation by reducing the number of context switches between the parent and child process, fork will almost certainly always be inefficient under Win32. Fortunately, in most circumstances the spawn family of calls provided by Cygwin can be substituted for a fork/exec pair with only a little effort. These calls map cleanly on top of the Win32 API. As a result, they are much more efficient. Changing the compiler's driver program to call spawn instead of fork was a trivial change and increased compilation speeds by twenty to thirty percent in our tests.

However, spawn and exec present their own set of difficulties. Because there is no way to do an actual exec under Win32, Cygwin has to invent its own Process IDs (PIDs). As a result, when a process performs multiple exec calls, there will be multiple Windows PIDs associated with a single Cygwin PID. In some cases, stubs of each of these Win32 processes may linger, waiting for their exec'd Cygwin process to exit.

Problems with process creation
The semantics of fork require that a forked child process have exactly the same address space layout as its parent. However, Windows provides no native support for cloning address space between processes and several features actively undermine a reliable fork implementation. Three issues are especially prevalent:

DLL base address collisions. Unlike *nix shared libraries, which use "position-independent code", Windows shared libraries assume a fixed base address. Whenever the hard-wired address ranges of two DLLs collide (which occurs quite often), the Windows loader must "rebase" one of them to a different address. However, it may not resolve collisions consistently, and may rebase a different dll and/or move it to a different address every time. Cygwin can usually compensate for this effect when it involves libraries opened dynamically, but collisions among statically-linked dlls (dependencies known at compile time) are resolved before cygwin1.dll initializes and cannot be fixed afterward. This problem can only be solved by removing the base address conflicts which cause the problem, usually using the rebaseall tool.

Address space layout randomization (ASLR). Starting with Vista, Windows implements ASLR, which means that thread stacks, heap, memory-mapped files, and statically-linked dlls are placed at different (random) locations in each process. This behaviour interferes with a proper fork, and if an unmovable object (process heap or system dll) ends up at the wrong location, Cygwin can do nothing to compensate (though it will retry a few times automatically).

DLL injection by BLODA. Badly-behaved applications which inject dlls into other processes often manage to clobber important sections of the child's address space, leading to base address collisions which rebasing cannot fix. The only way to resolve this problem is to remove (usually uninstall) the offending app.

In summary, current Windows implementations make it impossible to implement a perfectly reliable fork, and occasional fork failures are inevitable.