Re: fast/native fork?

Re: fast/native fork?

[Kaz Kylheku]

On 2018-01-21 00:32, Jay K wrote:
> I have some desire to discuss fork.
> I know it is an old and difficult topic.
> 
> 
> I found this:
> 
>  "Cygwin fork and RtlCloneUserProcess"
>  https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/afdf1b68-1f3e-47f5-94cf-51e397afe073/cygwin-fork-and-rtlcloneuserprocess?forum=windowsgeneraldevelopmentissues
> 
> 
> 
> NT has had fork since v1.
> The Posix subsystem used it.
> 
> You didn't need Vista's introduction of RtlCloneUserProcess.
> 
> This from 2005 alludes to how to make it work:
>   
> https://www.winvistatips.com/threads/question-about-ntcreateprocess.186504/
> 
> but I have difficult questions for you -- anyone, but including 
> Corinna.
> 
> What do you expect it to do?
> 
> 
> I mean, consider that there is no pthread_atfork or an analog in Win32.

That's the last of your problems; pthread_atfork doesn't require kernel 
support.
   
> Dlls at all levels of the Win32 stack, might have
> process-specific state, that needs to be reinitialized.

That issue affects any implementation of fork, including the current 
Cygwin one.

https://www.cygwin.com/faq.html#faq.api.fork

> ntdll.dll is special. It somehow knows fork occured and can 
> reinitialize itself.

> "Somehow" probably being the fact that
> ntdll.dll is special

Or, rather "Somehow" being the fact that ntdll.dll is where the
RtlCloneUserProcess function lives, so of course ntdll.dll is notified
when a fork takes place: it's taking a direct function call.
You don't get any more more notified than that.

> But no other dll expects this.

But under the current implementation of fork in Cygwin, no regular 
Windows DLL
expects to be attached to a Cygwin process which forks by copying .bss
and .data and executing a longjmp in the child to recover the stack 
context.

Moreover, ntdll.dll does not know about Cygwin fork, unlike its own 
fork.

So it really just boils down to this forking mechanism just not being
documented.

An idea might be to have fork be switchable between the current 
implementation
(default) and the RtlCloneUserProcess (experimental, run-time 
switchable).
Just to get it to users to experiment with while keeping things stable,
and without requiring special builds.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fast/native fork?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2454 bytes --]

On Jan 21 12:50, Thomas Wolff wrote:
> Am 21.01.2018 um 11:56 schrieb Marco Atzeri:
> > On 21/01/2018 16:32, Jay K wrote:
> > > 
> > > I have some desire to discuss fork.
> > > I know it is an old and difficult topic.
> > > 
> > > I found this:
> > > 
> > >   "Cygwin fork and RtlCloneUserProcess"
> > >   https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/afdf1b68-1f3e-47f5-94cf-51e397afe073/cygwin-fork-and-rtlcloneuserprocess?forum=windowsgeneraldevelopmentissues
> > > 
> > > NT has had fork since v1.
> > > The Posix subsystem used it.
> > 
> > If you look at the several Corinna's tentatives of having any info
> > on the Microsoft forum the problem is the lack of official documentation
> > of the details.
> > 
> > Without details no solution can be implemented that is guaranteed
> > to work on all existing and hopefully future version of Windows.
> > 
> > Cygwin can only use official external API and not hidden internal one.
> There was this mail from Microsoft, offering support in case Windows 10
> would raise console problems with cygwin:
> https://cygwin.com/ml/cygwin/2015-04/msg00623.html
> While this was particularly about the console, maybe this offer could be
> used for a request to provide official documentation of that system call, so
> it could then be used?

BTDT.

There already was an internal discussion between us Cygwin devs and
Microsoft devs back in 2012 to handle various aspects of a POSIX fork
implementation(*), which was fairly detailed.  This was triggered by
Windows 8 previews breaking fork on Cygwin due to a change in ASLR
behaviour.  We also asked for some minor provisions in the official
Win32 API(**) but the discussion petered out during the Windows 8
release with no further reply from the Microsoft side.


Corinna

(*) E.g., when using RtlCloneUserProcess everything done on the
    kernel/ntdll level appears to work (reading/writing to file handles,
    shared memory is available, etc), but simple higher-level stuff like
    Windows console handles become non-functional.

(**) E.g., telling CreateProcess that, right now, we don't want ASLR,
     regardless of the executables dynamicbase flag, or allowing the
     parent to tell the created child exactly where to load DLLs.

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: fast/native fork?

[Thomas Wolff]

Am 21.01.2018 um 11:56 schrieb Marco Atzeri:
> On 21/01/2018 16:32, Jay K wrote:
>>
>> I have some desire to discuss fork.
>> I know it is an old and difficult topic.
>>
>> I found this:
>>
>>   "Cygwin fork and RtlCloneUserProcess"
>>   https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/afdf1b68-1f3e-47f5-94cf-51e397afe073/cygwin-fork-and-rtlcloneuserprocess?forum=windowsgeneraldevelopmentissues
>>
>> NT has had fork since v1.
>> The Posix subsystem used it.
>
> If you look at the several Corinna's tentatives of having any info
> on the Microsoft forum the problem is the lack of official documentation
> of the details.
>
> Without details no solution can be implemented that is guaranteed
> to work on all existing and hopefully future version of Windows.
>
> Cygwin can only use official external API and not hidden internal one.
There was this mail from Microsoft, offering support in case Windows 10 
would raise console problems with cygwin:
https://cygwin.com/ml/cygwin/2015-04/msg00623.html
While this was particularly about the console, maybe this offer could be 
used for a request to provide official documentation of that system 
call, so it could then be used?
Thomas

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fast/native fork?

[Marco Atzeri]

On 21/01/2018 16:32, Jay K wrote:
> 
> I have some desire to discuss fork.
> I know it is an old and difficult topic.
> 
> 
> I found this:
> 
>   "Cygwin fork and RtlCloneUserProcess"
>   https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/afdf1b68-1f3e-47f5-94cf-51e397afe073/cygwin-fork-and-rtlcloneuserprocess?forum=windowsgeneraldevelopmentissues
> 
> 
> 
> NT has had fork since v1.
> The Posix subsystem used it.

If you look at the several Corinna's tentatives of having any info
on the Microsoft forum the problem is the lack of official documentation
of the details.

Without details no solution can be implemented that is guaranteed
to work on all existing and hopefully future version of Windows.

Cygwin can only use official external API and not hidden internal one.

Regards
Marco




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

fast/native fork?

[Jay K]

I have some desire to discuss fork.
I know it is an old and difficult topic.


I found this:

 "Cygwin fork and RtlCloneUserProcess"
 https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/afdf1b68-1f3e-47f5-94cf-51e397afe073/cygwin-fork-and-rtlcloneuserprocess?forum=windowsgeneraldevelopmentissues



NT has had fork since v1.
The Posix subsystem used it.

You didn't need Vista's introduction of RtlCloneUserProcess.

This from 2005 alludes to how to make it work:
  https://www.winvistatips.com/threads/question-about-ntcreateprocess.186504/

but I have difficult questions for you -- anyone, but including Corinna.

What do you expect it to do?


I mean, consider that there is no pthread_atfork or an analog in Win32.
 
Dlls at all levels of the Win32 stack, might have
process-specific state, that needs to be reinitialized.
Maybe they were holding locks in a worker thread.
Maybe they had the pid cached. Dumb, but it works w/o fork.
The usual problems that pthread_atfork is meant to solve.
 
Which memory do you expect to be inherited copy on write, and which
memory do you expect to revert to whatever it is when a process starts (or a
dll is loaded)?

You could hope for something like calling DllMain(process attach) of all the
dlls could help, but it can't, at least because dlls just depend on the
default data in the image, they don't write the defaults to memory
at every load.


ntdll.dll is special. It somehow knows fork occured and can reinitialize itself.
ntdll.dll is special -- it used by the Posix subsystems (SFU/SUA/Interix/etc.) and always had to work with fork.
  (Up until Windows 10 and changes for WSL, ntdll.dll was loaded into
  all usermode processes: Win32, Posix, OS/2. This was specifically
  changed for WSL.)


But no other dll expects this.

  
Now, I have some wierd ideas. Let's brainstorm a little?


Can you somehow leave the child process in limbo waiting for exec, know
when you have waited too long (because *anything* else happened) and only
then do the expensive copying?

Or there is the very problem of getting to the exec call and having the exec parameters?

I mean, what if you actually knew 100% that exec would come very soon after fork?
 What would/could you do then?

 
And when exec does not follow, what do you do?
How much of the child process is inherited from the parent, vs. how much
is reinitialized such as for a new process?

If there a solution that optimizes the guaranteed to exec case,
can you almost just assume it? Breaking the rare (?) program that does not exec?
You could even omit fork.
You'd have fork_slow and fork_before_exec.
People would have to ifdef Cygwin and chose what they want.
Or the default could be fork_before_exec, breaking a small number
of programs, that could be easily ported.


Ok, how about this?
Can you implement exec using only ntdll.dll, avoiding kernel32.dll?
 And the small/zero number of other things valid/used between fork and exec?
And assume exec follows fork?
If so then that is a solution:
 learn how to use native fork
 and have exec only use ntdll.dll
 
 That will give you a fast fork + exec sequence.
 
 Or, can you in the new process, just reinitialize kernel32.dll and kernelbase.dll,
 and only use them for exec?
 
 
It doesn't do anything for fork without exec but I still don't understand
how that is supposed to work in Win32.
 
 
 Or how about this:
 Again, if you assume exec is coming, and you just need fork 
 to do the minimum -- basically to get a pid.
 
 fork calls CreateProcess, with a helper .exe, suspended
 fork calls CreateThread, passing it the register context of the creating thread
 the helper thread suspends the creating thread, and takes over its
 register context (including rip and rsp, approximately), shortly thereafter
 the helper thread in the creating process reaches exec.
 
 At this point, somehow, it adjusts everything..hand waving.
 
 How do you implement exec today? Does fork actually get the pid
 of the new child, and exec in the child somehow "replaces" the executable, or
 does exec create a second child, with another pid, and the original
 child just waits for it, and returns its exit code as its own?
 
 Or do you have indirection on pids, and cygwin pids are not win32 pids?
 
Oh, that's right, setjmp/longjmp.
fork calls setjmp in the parent.
The first return continues until exec in the parent and then
returns the new pid the second time?
  Given an arrival at exec, in the parent instead of the child,
  the usual child part of fork need never run at all.


To repeat: To what extent, if any, can we assume exec follows fork?
And what can be done with this idea?
I understand the more general model, where exec does not follow fork.
But how common is it? How would Cygwin fair if by default fork+exec
was fast, fork w/o exec didn't work, and people ported those somehow?


Fyi:  
 https://mikeash.com/pyblog/friday-qa-2012-01-20-fork-safety.html
 
 
 - Jay

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple