Atomic mmap replacement

Atomic mmap replacement

[Ken Brown]

Some code in emacs wants to reserve a chunk of address space with a big 
PROT_NONE anonymous mapping, and then carve it up into separate mappings 
associated to segments of a file.  This fails on Cygwin.  Here's a test 
case that illustrates the problem:

$ truncate -s 64k foo

$ cat mmap_test.c
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/mman.h>

const size_t page_size = 64 * 1024;

int
main ()
{
   void *mem = mmap (NULL, 2 * page_size,
                     PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
   if (mem == MAP_FAILED)
     {
       perror ("mmap");
       exit (1);
     }
   int fd = open ("foo", O_RDONLY);
   void *res = mmap (mem, page_size, PROT_READ | PROT_WRITE,
                     MAP_PRIVATE | MAP_FIXED, fd, 0);
   if (res == MAP_FAILED)
     {
       perror ("mmap");
       exit (2);
     }
}

$ gcc mmap_test.c

$ ./a
mmap: Invalid argument

$ echo $?
2

Is this a bug, or is it simply a limitation of Cygwin's mmap?  If the 
latter, is there a simple workaround?

Ken

P.S. For context, you can start reading here, but it might not make a 
lot of sense:

   https://lists.gnu.org/archive/html/emacs-devel/2018-02/msg00440.html

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Atomic mmap replacement

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2645 bytes --]

On Feb 17 22:37, Ken Brown wrote:
> Some code in emacs wants to reserve a chunk of address space with a big
> PROT_NONE anonymous mapping, and then carve it up into separate mappings
> associated to segments of a file.  This fails on Cygwin.  Here's a test case
> that illustrates the problem:
> 
> $ truncate -s 64k foo
> 
> $ cat mmap_test.c
> #include <stdio.h>
> #include <stdlib.h>
> #include <fcntl.h>
> #include <sys/mman.h>
> 
> const size_t page_size = 64 * 1024;
> 
> int
> main ()
> {
>   void *mem = mmap (NULL, 2 * page_size,
>                     PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>   if (mem == MAP_FAILED)
>     {
>       perror ("mmap");
>       exit (1);
>     }
>   int fd = open ("foo", O_RDONLY);
>   void *res = mmap (mem, page_size, PROT_READ | PROT_WRITE,
>                     MAP_PRIVATE | MAP_FIXED, fd, 0);
>   if (res == MAP_FAILED)
>     {
>       perror ("mmap");
>       exit (2);
>     }
> }
> 
> $ gcc mmap_test.c
> 
> $ ./a
> mmap: Invalid argument
> 
> $ echo $?
> 2
> 
> Is this a bug, or is it simply a limitation of Cygwin's mmap?  If the
> latter, is there a simple workaround?

Several limitations in the Windows kernel disallow this:

- It doesn't allow to unmap parts of a map, only the entire map as a
  whole.
  
  Cygwin has a workaround: If you unmap parts of a map it just keeps
  track of this and sets the protection of the affected pages to
  PAGE_NOACCESS.  In case of anonymous mappings, it even recycles them
  potentially for other mappings.

- It also disallows to re-map any allocated or mapped mamory for another
  purpose.

So this part of the POSIX specs for mmap:

  "The mapping established by mmap() shall replace any previous mappings
   for those whole pages containing any part of the address space of the
   process starting at pa and continuing for len bytes"

can't be implemented with Windows means.

The only workaround possible would be to handle this *exact* scenario as
a special case in Cygwin's mmap:  If the new mapping falls in the middle
of an existing mapping and if the original mapping was an anonymous
mapping with PROT_NONE page protection, then

- unmap the old mapping
- remap the unaffected parts as separate anonymous mapping
- map the affected parts for the requested file mapping

This is pretty complicated and I'm not hot on implementing it.  If it's
really required we can take a look of course.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Atomic mmap replacement

[Ken Brown]

On 2/19/2018 4:00 AM, Corinna Vinschen wrote:
> On Feb 17 22:37, Ken Brown wrote:
>> Some code in emacs wants to reserve a chunk of address space with a big
>> PROT_NONE anonymous mapping, and then carve it up into separate mappings
>> associated to segments of a file.  This fails on Cygwin.  Here's a test case
>> that illustrates the problem:
>>
>> $ truncate -s 64k foo
>>
>> $ cat mmap_test.c
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <fcntl.h>
>> #include <sys/mman.h>
>>
>> const size_t page_size = 64 * 1024;
>>
>> int
>> main ()
>> {
>>    void *mem = mmap (NULL, 2 * page_size,
>>                      PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>>    if (mem == MAP_FAILED)
>>      {
>>        perror ("mmap");
>>        exit (1);
>>      }
>>    int fd = open ("foo", O_RDONLY);
>>    void *res = mmap (mem, page_size, PROT_READ | PROT_WRITE,
>>                      MAP_PRIVATE | MAP_FIXED, fd, 0);
>>    if (res == MAP_FAILED)
>>      {
>>        perror ("mmap");
>>        exit (2);
>>      }
>> }
>>
>> $ gcc mmap_test.c
>>
>> $ ./a
>> mmap: Invalid argument
>>
>> $ echo $?
>> 2
>>
>> Is this a bug, or is it simply a limitation of Cygwin's mmap?  If the
>> latter, is there a simple workaround?
> 
> Several limitations in the Windows kernel disallow this:
> 
> - It doesn't allow to unmap parts of a map, only the entire map as a
>    whole.
>    
>    Cygwin has a workaround: If you unmap parts of a map it just keeps
>    track of this and sets the protection of the affected pages to
>    PAGE_NOACCESS.  In case of anonymous mappings, it even recycles them
>    potentially for other mappings.
> 
> - It also disallows to re-map any allocated or mapped mamory for another
>    purpose.
> 
> So this part of the POSIX specs for mmap:
> 
>    "The mapping established by mmap() shall replace any previous mappings
>     for those whole pages containing any part of the address space of the
>     process starting at pa and continuing for len bytes"
> 
> can't be implemented with Windows means.
> 
> The only workaround possible would be to handle this *exact* scenario as
> a special case in Cygwin's mmap:  If the new mapping falls in the middle
> of an existing mapping and if the original mapping was an anonymous
> mapping with PROT_NONE page protection, then
> 
> - unmap the old mapping
> - remap the unaffected parts as separate anonymous mapping
> - map the affected parts for the requested file mapping
> 
> This is pretty complicated and I'm not hot on implementing it.  If it's
> really required we can take a look of course.

Thanks, Corinna.  I'll take this information back to the emacs list.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Atomic mmap replacement

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2931 bytes --]

On Feb 19 08:22, Ken Brown wrote:
> On 2/19/2018 4:00 AM, Corinna Vinschen wrote:
> > On Feb 17 22:37, Ken Brown wrote:
> > > Some code in emacs wants to reserve a chunk of address space with a big
> > > PROT_NONE anonymous mapping, and then carve it up into separate mappings
> > > associated to segments of a file.  This fails on Cygwin.  Here's a test case
> > > that illustrates the problem:
> > > [...]
> > Several limitations in the Windows kernel disallow this:
> > 
> > - It doesn't allow to unmap parts of a map, only the entire map as a
> >    whole.
> >    Cygwin has a workaround: If you unmap parts of a map it just keeps
> >    track of this and sets the protection of the affected pages to
> >    PAGE_NOACCESS.  In case of anonymous mappings, it even recycles them
> >    potentially for other mappings.
> > 
> > - It also disallows to re-map any allocated or mapped mamory for another
> >    purpose.
> > 
> > So this part of the POSIX specs for mmap:
> > 
> >    "The mapping established by mmap() shall replace any previous mappings
> >     for those whole pages containing any part of the address space of the
> >     process starting at pa and continuing for len bytes"
> > 
> > can't be implemented with Windows means.
> > 
> > The only workaround possible would be to handle this *exact* scenario as
> > a special case in Cygwin's mmap:  If the new mapping falls in the middle
> > of an existing mapping and if the original mapping was an anonymous
> > mapping with PROT_NONE page protection, then

On second thought, we *could* do this, if the pages have been mmapped
before(*).  Unfortunately this would require a *major* revamp of the
page handling in mmap.  We would have to keep the mapping of every
single 64K page separate.

I.e., requesting a file mapping of 256K at offset 0 on the POSIX level
would have to be handled as four Windows file mappings under the hood:

1. a 64K file mapping at offset 0
2. a 64K file mapping at offset 65536
3. a 64K file mapping at offset 131072
4. a 64K file mapping at offset 196608

A request to mmap another 64K page to the third mapping in this example
could then be done by unmapping the third mapping and replace it with
the requested mapping.

I'm not sure this is feasible.  It would complicate and slow down the
code especially for big mappings; one call to NtCreateSection and one to
NtMapViewOfSection per 64K page, plus the overhead of making sure that
all mappings are in the right, sequential order in memory.  Plus the
overhead of having to remap a lot more mappings in forked children.  The
"Cygwin is slow" meme would get another interesting facet :}

(*) Meaning: We still couldn't call mmap on addresses allocated on the heap.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Atomic mmap replacement

[Ken Brown]

On 2/19/2018 12:19 PM, Corinna Vinschen wrote:
> On Feb 19 08:22, Ken Brown wrote:
>> On 2/19/2018 4:00 AM, Corinna Vinschen wrote:
>>> On Feb 17 22:37, Ken Brown wrote:
>>>> Some code in emacs wants to reserve a chunk of address space with a big
>>>> PROT_NONE anonymous mapping, and then carve it up into separate mappings
>>>> associated to segments of a file.  This fails on Cygwin.  Here's a test case
>>>> that illustrates the problem:
>>>> [...]
>>> Several limitations in the Windows kernel disallow this:
>>>
>>> - It doesn't allow to unmap parts of a map, only the entire map as a
>>>     whole.
>>>     Cygwin has a workaround: If you unmap parts of a map it just keeps
>>>     track of this and sets the protection of the affected pages to
>>>     PAGE_NOACCESS.  In case of anonymous mappings, it even recycles them
>>>     potentially for other mappings.
>>>
>>> - It also disallows to re-map any allocated or mapped mamory for another
>>>     purpose.
>>>
>>> So this part of the POSIX specs for mmap:
>>>
>>>     "The mapping established by mmap() shall replace any previous mappings
>>>      for those whole pages containing any part of the address space of the
>>>      process starting at pa and continuing for len bytes"
>>>
>>> can't be implemented with Windows means.
>>>
>>> The only workaround possible would be to handle this *exact* scenario as
>>> a special case in Cygwin's mmap:  If the new mapping falls in the middle
>>> of an existing mapping and if the original mapping was an anonymous
>>> mapping with PROT_NONE page protection, then
> 
> On second thought, we *could* do this, if the pages have been mmapped
> before(*).  Unfortunately this would require a *major* revamp of the
> page handling in mmap.  We would have to keep the mapping of every
> single 64K page separate.
> 
> I.e., requesting a file mapping of 256K at offset 0 on the POSIX level
> would have to be handled as four Windows file mappings under the hood:
> 
> 1. a 64K file mapping at offset 0
> 2. a 64K file mapping at offset 65536
> 3. a 64K file mapping at offset 131072
> 4. a 64K file mapping at offset 196608
> 
> A request to mmap another 64K page to the third mapping in this example
> could then be done by unmapping the third mapping and replace it with
> the requested mapping.
> 
> I'm not sure this is feasible.  It would complicate and slow down the
> code especially for big mappings; one call to NtCreateSection and one to
> NtMapViewOfSection per 64K page, plus the overhead of making sure that
> all mappings are in the right, sequential order in memory.  Plus the
> overhead of having to remap a lot more mappings in forked children.  The
> "Cygwin is slow" meme would get another interesting facet :}

That doesn't sound great.

In the meantime, the problem was solved on the emacs side by doing what 
you suggested in your previous email:

 > - unmap the old mapping
 > - remap the unaffected parts as separate anonymous mapping
 > - map the affected parts for the requested file mapping

But in the emacs application it's simpler, because there are no 
unaffected parts, so step 2 can be skipped.

Thanks.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Atomic mmap replacement

[John Hood]

On 02/19/2018 12:19, Corinna Vinschen wrote:
> On second thought, we *could* do this, if the pages have been mmapped
> before(*).  Unfortunately this would require a *major* revamp of the
> page handling in mmap.  We would have to keep the mapping of every
> single 64K page separate.
>
> I.e., requesting a file mapping of 256K at offset 0 on the POSIX level
> would have to be handled as four Windows file mappings under the hood:
>
> 1. a 64K file mapping at offset 0
> 2. a 64K file mapping at offset 65536
> 3. a 64K file mapping at offset 131072
> 4. a 64K file mapping at offset 196608
>
> A request to mmap another 64K page to the third mapping in this example
> could then be done by unmapping the third mapping and replace it with
> the requested mapping.
>
> I'm not sure this is feasible.  It would complicate and slow down the
> code especially for big mappings; one call to NtCreateSection and one to
> NtMapViewOfSection per 64K page, plus the overhead of making sure that
> all mappings are in the right, sequential order in memory.  Plus the
> overhead of having to remap a lot more mappings in forked children.  The
> "Cygwin is slow" meme would get another interesting facet :}
>
I work on an app that does something like this (but for other reasons, 
and it'll never be ported to Cygwin).

I'm also dubious, but I'll point out that it'd probably be reasonable to 
do this only on regions that are mapped PROT_NONE initially, other 
regions could work as they do now.  That'd help performance in the 
common case.  Also, if Windows has a way to prevent use of a region 
other than creating a mapping, Cygwin could perhaps emulate PROT_NONE 
mappings without actually creating a mapping (at the cost of even more 
code complexity, probably).

regards,

   --jh


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Atomic mmap replacement

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

On Feb 19 17:33, Ken Brown wrote:
> On 2/19/2018 12:19 PM, Corinna Vinschen wrote:
> In the meantime, the problem was solved on the emacs side by doing what you
> suggested in your previous email:
> 
> > - unmap the old mapping
> > - remap the unaffected parts as separate anonymous mapping
> > - map the affected parts for the requested file mapping
> 
> But in the emacs application it's simpler, because there are no unaffected
> parts, so step 2 can be skipped.

Thanks, that's obviously the simpler approach :)

Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Atomic mmap replacement

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2911 bytes --]

On Feb 19 20:03, John Hood wrote:
> 
> 
> On 02/19/2018 12:19, Corinna Vinschen wrote:
> > On second thought, we *could* do this, if the pages have been mmapped
> > before(*).  Unfortunately this would require a *major* revamp of the
> > page handling in mmap.  We would have to keep the mapping of every
> > single 64K page separate.
> > 
> > I.e., requesting a file mapping of 256K at offset 0 on the POSIX level
> > would have to be handled as four Windows file mappings under the hood:
> > 
> > 1. a 64K file mapping at offset 0
> > 2. a 64K file mapping at offset 65536
> > 3. a 64K file mapping at offset 131072
> > 4. a 64K file mapping at offset 196608
> > 
> > A request to mmap another 64K page to the third mapping in this example
> > could then be done by unmapping the third mapping and replace it with
> > the requested mapping.
> > 
> > I'm not sure this is feasible.  It would complicate and slow down the
> > code especially for big mappings; one call to NtCreateSection and one to
> > NtMapViewOfSection per 64K page, plus the overhead of making sure that
> > all mappings are in the right, sequential order in memory.  Plus the
> > overhead of having to remap a lot more mappings in forked children.  The
> > "Cygwin is slow" meme would get another interesting facet :}
> > 
> I work on an app that does something like this (but for other reasons, and
> it'll never be ported to Cygwin).
> 
> I'm also dubious, but I'll point out that it'd probably be reasonable to do
> this only on regions that are mapped PROT_NONE initially, other regions
> could work as they do now.

As far as I'm concerned, the problem is that we collect special cases
which raise the number of lines of code, getting more complicated and
less well debuggable, but still only work in some cases.

Assuming we'd do this at all, doing this per page *all* the time would
certainly lead to simpler code.  The big change affects the bookkeeping
code in the first place.  Right now we have a list of files (anonympous
mappings are one file) with a list of mappings per file attached.  In
theory, to support scenarios like this, the bookkeeping structure should
be changed to a hash or list of per-page structs.  So we would look at
maps starting at the page address, rather than starting at the file
descriptor the map is attached to.

I'm not opposed to change Cygwin's mmap this way, but it's a loooot of
work and there's so much to do in so limited a time...

If somebody would like to take a stab at this, I'd be pretty excited.
Fortunately the mmap code is collected in a single source file
winsup/cygwin/mmap.cc, so you only need limited insight in the Cygwin
code.  Help from my side is guaranteed.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]