possible problem with memory allocation using calloc/mmap/munmap

possible problem with memory allocation using calloc/mmap/munmap

[Stanislav Kascak]

Hello cygwin team,

I came across a problem with memory allocation/deallocation when
trying to compile and run tests of openldap under cygwin.

I created a test program to simulate sequence of actions. First a
bigger chunk of memory (>~262kB) is allocated using calloc(), then
mmap() is called with requested memory size much bigger than actual
file size. Subsequently,  munmap() to free previous mmap(), and at the
end an attempt to access memory allocated with calloc(). That last
memory access is causing Segmentation fault.

It seems that when mmap() is called with length argument exceeding
size of file, only memory to fit that file is allocated. munmap()
however frees the full specified length. Since (at least on my
computer) big chunk of memory allocated by calloc() is located after
mmap() allocation, munmap() frees even memory of that calloc().

I'm not sure if this is only a problem of my environment, or there is
something I'm missing.

Below is the test (notice that memory allocated by mmap() is only 64k
bytes in front of the memory allocated by calloc(), not 32MB as
requested).

My system is Windows 7 64 bit, cygwin64 3.0.7.

Best regards,

Stanislav Kascak

$ cat a.c
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>

int main(int argc, char** argv){
  char* x;
  char* y;
  int fd;
  size_t s1 = 0x40000;
  size_t s2 = 0x2000000;
  x = (char*)calloc(1, s1);
  if(x == NULL){
    printf("calloc failed\n");
  }
  x[0] = 'a';
  fd = open("testfile", O_RDONLY);
  if(fd == -1){
    printf("open failed: %d\n", errno);
  }
  y = mmap(NULL, s2, PROT_READ, MAP_SHARED, fd, 0);
  if(y == MAP_FAILED){
    printf("mmap failed: %d\n", errno);
  }
  close(fd);
  printf("%lx -> %c\n", x, x[0]);
  printf("%lx\n", y);
  if(munmap(y, s2)){
    printf("munmap failed: %d\n", errno);
  }
  printf("%lx -> %c\n", x, x[0]);
  free(x);
}

$ gcc a.c

$ echo "testdata" > testfile

$ ./a.exe
6fffffb0010 -> a
6fffffa0000
Segmentation fault (core dumped)

$ uname -a
CYGWIN_NT-6.1 myhost 3.0.7(0.338/5/3) 2019-04-30 18:08 x86_64 Cygwin

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: possible problem with memory allocation using calloc/mmap/munmap

[Ken Brown]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2257 bytes --]

On 5/3/2019 7:33 AM, Stanislav Kascak wrote:
> I came across a problem with memory allocation/deallocation when
> trying to compile and run tests of openldap under cygwin.

Corinna can give you a definitive response when she returns from vacation, but I 
looked at the code and have a few comments.

> I created a test program to simulate sequence of actions. First a
> bigger chunk of memory (>~262kB) is allocated using calloc(), then
> mmap() is called with requested memory size much bigger than actual
> file size. Subsequently,  munmap() to free previous mmap(), and at the
> end an attempt to access memory allocated with calloc(). That last
> memory access is causing Segmentation fault.
> 
> It seems that when mmap() is called with length argument exceeding
> size of file, only memory to fit that file is allocated.

That's correct.  The following comment in mmap.cc:1100 explains why: "don't map 
beyond EOF, since Windows would change the file to the new length, in contrast 
to POSIX."

> munmap()
 > however frees the full specified length. Since (at least on my
 > computer) big chunk of memory allocated by calloc() is located after
 > mmap() allocation, munmap() frees even memory of that calloc().

I think munmap does exactly what it's supposed to do according to Posix, which 
is "remove any mappings for those entire pages containing any part of the 
address space of the process starting at addr and continuing for len bytes."  In 
your case, that includes the memory allocated by calloc, which uses mmap.

I'm not sure what openldap is doing in your actual use case, but is there really 
a good reason to call mmap with requested size much bigger than the actual file 
size?  After all, you can never access that extra memory.  Posix says, 
"References within the address range starting at pa and continuing for len bytes 
to whole pages following the end of an object shall result in delivery of a 
SIGBUS signal."

But let's wait and see what Corinna says.

Ken
\0ТÒÐÐ¥\a&ö&ÆVÒ\a&W\x06÷'G3¢\x02\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒ÷\a&ö&ÆV×2æ‡FÖÀФd\x15\x13¢\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒöf\x17\x12ðФFö7VÖVçF\x17F–öã¢\x02\x02\x02\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒöFö72æ‡FÖÀÐ¥Vç7V'67&–&R\x06–æfó¢\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒöÖÂò7Vç7V'67&–&R×6–×\x06ÆPРÐ

Re: possible problem with memory allocation using calloc/mmap/munmap

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1981 bytes --]

On May  3 13:33, Stanislav Kascak wrote:
> Hello cygwin team,
> 
> I came across a problem with memory allocation/deallocation when
> trying to compile and run tests of openldap under cygwin.
> 
> I created a test program to simulate sequence of actions. First a
> bigger chunk of memory (>~262kB) is allocated using calloc(), then
> mmap() is called with requested memory size much bigger than actual
> file size. Subsequently,  munmap() to free previous mmap(), and at the
> end an attempt to access memory allocated with calloc(). That last
> memory access is causing Segmentation fault.
> 
> It seems that when mmap() is called with length argument exceeding
> size of file, only memory to fit that file is allocated. munmap()
> however frees the full specified length. Since (at least on my
> computer) big chunk of memory allocated by calloc() is located after
> mmap() allocation, munmap() frees even memory of that calloc().

Ken's right.  Due to the differences between mapping files on Windows
vs. Unix, Cygwin can't map beyond the file size + the remainder of the
last page.  Cygwin tries to workaround that on 32 bit by allocating
an anonymous mapping following the file mapping to keep the range free
from other mappings.  But on 64 bit this workaround doesn't work anymore
because the OS is missing an (undocumented) flag which allows to
create mappings on 4K boundaries, rather than just on 64K boundaries.

I know this situation is unsatisfying, but I have no easy workaround
to allow this.  Cygwin could add the anonymous mapping on the next
64K boundary on 64 bit, but that would result in a hole in the mapping
which seemed like a rather bad idea when porting mmap to 64 bit.

Ken's also right that munmap is doing the right thing here.  If
anything's wrong, it's mmap's workaround for mappings beyond the file
length.  If only 64 bit would allow 4K-aligned mappings :(


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: possible problem with memory allocation using calloc/mmap/munmap

[Stanislav Kascak]

> > It seems that when mmap() is called with length argument exceeding
> > size of file, only memory to fit that file is allocated. munmap()
> > however frees the full specified length. Since (at least on my
> > computer) big chunk of memory allocated by calloc() is located after
> > mmap() allocation, munmap() frees even memory of that calloc().
>
> Ken's right.  Due to the differences between mapping files on Windows
> vs. Unix, Cygwin can't map beyond the file size + the remainder of the
> last page.  Cygwin tries to workaround that on 32 bit by allocating
> an anonymous mapping following the file mapping to keep the range free
> from other mappings.  But on 64 bit this workaround doesn't work anymore
> because the OS is missing an (undocumented) flag which allows to
> create mappings on 4K boundaries, rather than just on 64K boundaries.
>
> I know this situation is unsatisfying, but I have no easy workaround
> to allow this.  Cygwin could add the anonymous mapping on the next
> 64K boundary on 64 bit, but that would result in a hole in the mapping
> which seemed like a rather bad idea when porting mmap to 64 bit.
>
> Ken's also right that munmap is doing the right thing here.  If
> anything's wrong, it's mmap's workaround for mappings beyond the file
> length.  If only 64 bit would allow 4K-aligned mappings :(

Thanks for the answer. It is appreciated.
I understand the problem and difficulty to resolve it. Maybe returning
an error from mmap (and putting a comment to code for its reason)
would be sufficient. mmap caller could just adjust requested
allocation size to file size. Without error, caller has no way of
knowing memory was not allocated and segfault is then thrown in an
unrelated memory segment which makes the root cause hard to track
down. But, I do not know all the implication that could result from
that, so evaluation of this approach is up to you.
Thanks again.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: possible problem with memory allocation using calloc/mmap/munmap

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3722 bytes --]

On Jun  4 11:38, Stanislav Kascak wrote:
> > > It seems that when mmap() is called with length argument exceeding
> > > size of file, only memory to fit that file is allocated. munmap()
> > > however frees the full specified length. Since (at least on my
> > > computer) big chunk of memory allocated by calloc() is located after
> > > mmap() allocation, munmap() frees even memory of that calloc().
> >
> > Ken's right.  Due to the differences between mapping files on Windows
> > vs. Unix, Cygwin can't map beyond the file size + the remainder of the
> > last page.  Cygwin tries to workaround that on 32 bit by allocating
> > an anonymous mapping following the file mapping to keep the range free
> > from other mappings.  But on 64 bit this workaround doesn't work anymore
> > because the OS is missing an (undocumented) flag which allows to
> > create mappings on 4K boundaries, rather than just on 64K boundaries.
> >
> > I know this situation is unsatisfying, but I have no easy workaround
> > to allow this.  Cygwin could add the anonymous mapping on the next
> > 64K boundary on 64 bit, but that would result in a hole in the mapping
> > which seemed like a rather bad idea when porting mmap to 64 bit.
> >
> > Ken's also right that munmap is doing the right thing here.  If
> > anything's wrong, it's mmap's workaround for mappings beyond the file
> > length.  If only 64 bit would allow 4K-aligned mappings :(
> 
> Thanks for the answer. It is appreciated.
> I understand the problem and difficulty to resolve it. Maybe returning
> an error from mmap (and putting a comment to code for its reason)
> would be sufficient. mmap caller could just adjust requested
> allocation size to file size. Without error, caller has no way of
> knowing memory was not allocated and segfault is then thrown in an
> unrelated memory segment which makes the root cause hard to track
> down. But, I do not know all the implication that could result from
> that, so evaluation of this approach is up to you.

Given that most of the required code already exists for 32 bit systems
(except under WOW64, suffering the same problem as the 64 bit WIndows
environment), I hacked a bit on this code this morning and I got your
testcase running fine.  The idea being that after a successful mmap the
expectation that a matching munmap does *not* unmap unrelated mappings
is valid.

In more depth, here's what Cygwin does on 32 bit, assuming a file size
of 100 bytes and a mapping request of 256K:

First Cygwin mmaps the file.  This results in a 4K mapping in Windows:

 file:    |-- 100b --|

 mapping: |-- 4K --....--|

Next Cygwin adds another mapping to fill up the range up to the next
64K allocation granularity boundary:

 |-- file 4K --|-- filler 60K --|

Eventually Cygwin adds another mapping to fullfill the entire mapping
request:

 |-- file 4K --|-- filler 60K --|-- filler 192K --|

The problem on WOW64 and real 64 bit is that it's impossible to map
the first filler.  However, this area in the VM will *never* be
allocated by other application functions due to the allocation
granularity of 64K!

So my workaround for 64 bit and WOW64 is to just skip allocating the
first filler:

 |-- file 4K --|-- THE VOID 60K --|-- filler 192K --|

The advantage is now that the following munmap of 256K will only
unmap the map for the file and the filler, but not the region you
calloced before, which formerly was accidentally mapped to the
filler region.  This just can't happen anymore now.

Would that be feasible?  If so I can push my patch and create a
developer snapshot for testing.


Thanks,
Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: possible problem with memory allocation using calloc/mmap/munmap

[Stanislav Kascak]

> > > > It seems that when mmap() is called with length argument exceeding
> > > > size of file, only memory to fit that file is allocated. munmap()
> > > > however frees the full specified length. Since (at least on my
> > > > computer) big chunk of memory allocated by calloc() is located after
> > > > mmap() allocation, munmap() frees even memory of that calloc().
> > >
> > > Ken's right.  Due to the differences between mapping files on Windows
> > > vs. Unix, Cygwin can't map beyond the file size + the remainder of the
> > > last page.  Cygwin tries to workaround that on 32 bit by allocating
> > > an anonymous mapping following the file mapping to keep the range free
> > > from other mappings.  But on 64 bit this workaround doesn't work anymore
> > > because the OS is missing an (undocumented) flag which allows to
> > > create mappings on 4K boundaries, rather than just on 64K boundaries.
> > >
> > > I know this situation is unsatisfying, but I have no easy workaround
> > > to allow this.  Cygwin could add the anonymous mapping on the next
> > > 64K boundary on 64 bit, but that would result in a hole in the mapping
> > > which seemed like a rather bad idea when porting mmap to 64 bit.
> > >
> > > Ken's also right that munmap is doing the right thing here.  If
> > > anything's wrong, it's mmap's workaround for mappings beyond the file
> > > length.  If only 64 bit would allow 4K-aligned mappings :(
> >
> > Thanks for the answer. It is appreciated.
> > I understand the problem and difficulty to resolve it. Maybe returning
> > an error from mmap (and putting a comment to code for its reason)
> > would be sufficient. mmap caller could just adjust requested
> > allocation size to file size. Without error, caller has no way of
> > knowing memory was not allocated and segfault is then thrown in an
> > unrelated memory segment which makes the root cause hard to track
> > down. But, I do not know all the implication that could result from
> > that, so evaluation of this approach is up to you.
>
> Given that most of the required code already exists for 32 bit systems
> (except under WOW64, suffering the same problem as the 64 bit WIndows
> environment), I hacked a bit on this code this morning and I got your
> testcase running fine.  The idea being that after a successful mmap the
> expectation that a matching munmap does *not* unmap unrelated mappings
> is valid.
>
> In more depth, here's what Cygwin does on 32 bit, assuming a file size
> of 100 bytes and a mapping request of 256K:
>
> First Cygwin mmaps the file.  This results in a 4K mapping in Windows:
>
>  file:    |-- 100b --|
>
>  mapping: |-- 4K --....--|
>
> Next Cygwin adds another mapping to fill up the range up to the next
> 64K allocation granularity boundary:
>
>  |-- file 4K --|-- filler 60K --|
>
> Eventually Cygwin adds another mapping to fullfill the entire mapping
> request:
>
>  |-- file 4K --|-- filler 60K --|-- filler 192K --|
>
> The problem on WOW64 and real 64 bit is that it's impossible to map
> the first filler.  However, this area in the VM will *never* be
> allocated by other application functions due to the allocation
> granularity of 64K!
>
> So my workaround for 64 bit and WOW64 is to just skip allocating the
> first filler:
>
>  |-- file 4K --|-- THE VOID 60K --|-- filler 192K --|
>
> The advantage is now that the following munmap of 256K will only
> unmap the map for the file and the filler, but not the region you
> calloced before, which formerly was accidentally mapped to the
> filler region.  This just can't happen anymore now.
>
> Would that be feasible?  If so I can push my patch and create a
> developer snapshot for testing.

Two questions arise when I'm thinking about workaround solution:
- what happens if caller tries to write to |-- THE VOID 60K --|. Since
this is unallocated, would there be a segfault?
- is it possible that some subsequent mem alloc request would return
region from |-- THE VOID 60K --| which could again cause segfault
after munmap?

Stanislav Kascak

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: possible problem with memory allocation using calloc/mmap/munmap

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3058 bytes --]

On Jun  4 15:48, Stanislav Kascak wrote:
> > > > > It seems that when mmap() is called with length argument exceeding
> > > > > size of file, only memory to fit that file is allocated. munmap()
> > > > > however frees the full specified length. [...]
> > > > [...]
> > > > I know this situation is unsatisfying, but I have no easy workaround
> > > > to allow this.  Cygwin could add the anonymous mapping on the next
> > > > 64K boundary on 64 bit, but that would result in a hole in the mapping
> > > > which seemed like a rather bad idea when porting mmap to 64 bit.
> > > >
> > > > Ken's also right that munmap is doing the right thing here.  If
> > > > anything's wrong, it's mmap's workaround for mappings beyond the file
> > > > length.  If only 64 bit would allow 4K-aligned mappings :(
> > >
> > > Thanks for the answer. It is appreciated.
> > > I understand the problem and difficulty to resolve it. Maybe returning
> > > an error from mmap (and putting a comment to code for its reason)
> > > would be sufficient. mmap caller could just adjust requested
> > > allocation size to file size. Without error, caller has no way of
> > > knowing memory was not allocated and segfault is then thrown in an
> > > unrelated memory segment which makes the root cause hard to track
> > > down. But, I do not know all the implication that could result from
> > > that, so evaluation of this approach is up to you.
> > [...]
> > Eventually Cygwin adds another mapping to fullfill the entire mapping
> > request:
> >
> >  |-- file 4K --|-- filler 60K --|-- filler 192K --|
> >
> > The problem on WOW64 and real 64 bit is that it's impossible to map
> > the first filler.  However, this area in the VM will *never* be
> > allocated by other application functions due to the allocation
> > granularity of 64K!
> >
> > So my workaround for 64 bit and WOW64 is to just skip allocating the
> > first filler:
> >
> >  |-- file 4K --|-- THE VOID 60K --|-- filler 192K --|
> >
> > The advantage is now that the following munmap of 256K will only
> > unmap the map for the file and the filler, but not the region you
> > calloced before, which formerly was accidentally mapped to the
> > filler region.  This just can't happen anymore now.
> >
> > Would that be feasible?  If so I can push my patch and create a
> > developer snapshot for testing.
> 
> Two questions arise when I'm thinking about workaround solution:
> - what happens if caller tries to write to |-- THE VOID 60K --|. Since
> this is unallocated, would there be a segfault?

Accessing the VOID would raise SIGSEGV, while accessing the filler
raises SIGBUS.  The latter is also used to implement MAP_NORESERVE,
which the VOID can't support.

> - is it possible that some subsequent mem alloc request would return
> region from |-- THE VOID 60K --| which could again cause segfault
> after munmap?

No, as stated above.  Allocations are restricted to Windows' 64K
allocation granularity.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: possible problem with memory allocation using calloc/mmap/munmap

[Stanislav Kascak]

> > > > > > It seems that when mmap() is called with length argument exceeding
> > > > > > size of file, only memory to fit that file is allocated. munmap()
> > > > > > however frees the full specified length. [...]
> > > > > [...]
> > > > > I know this situation is unsatisfying, but I have no easy workaround
> > > > > to allow this.  Cygwin could add the anonymous mapping on the next
> > > > > 64K boundary on 64 bit, but that would result in a hole in the mapping
> > > > > which seemed like a rather bad idea when porting mmap to 64 bit.
> > > > >
> > > > > Ken's also right that munmap is doing the right thing here.  If
> > > > > anything's wrong, it's mmap's workaround for mappings beyond the file
> > > > > length.  If only 64 bit would allow 4K-aligned mappings :(
> > > >
> > > > Thanks for the answer. It is appreciated.
> > > > I understand the problem and difficulty to resolve it. Maybe returning
> > > > an error from mmap (and putting a comment to code for its reason)
> > > > would be sufficient. mmap caller could just adjust requested
> > > > allocation size to file size. Without error, caller has no way of
> > > > knowing memory was not allocated and segfault is then thrown in an
> > > > unrelated memory segment which makes the root cause hard to track
> > > > down. But, I do not know all the implication that could result from
> > > > that, so evaluation of this approach is up to you.
> > > [...]
> > > Eventually Cygwin adds another mapping to fullfill the entire mapping
> > > request:
> > >
> > >  |-- file 4K --|-- filler 60K --|-- filler 192K --|
> > >
> > > The problem on WOW64 and real 64 bit is that it's impossible to map
> > > the first filler.  However, this area in the VM will *never* be
> > > allocated by other application functions due to the allocation
> > > granularity of 64K!
> > >
> > > So my workaround for 64 bit and WOW64 is to just skip allocating the
> > > first filler:
> > >
> > >  |-- file 4K --|-- THE VOID 60K --|-- filler 192K --|
> > >
> > > The advantage is now that the following munmap of 256K will only
> > > unmap the map for the file and the filler, but not the region you
> > > calloced before, which formerly was accidentally mapped to the
> > > filler region.  This just can't happen anymore now.
> > >
> > > Would that be feasible?  If so I can push my patch and create a
> > > developer snapshot for testing.
> >
> > Two questions arise when I'm thinking about workaround solution:
> > - what happens if caller tries to write to |-- THE VOID 60K --|. Since
> > this is unallocated, would there be a segfault?
>
> Accessing the VOID would raise SIGSEGV, while accessing the filler
> raises SIGBUS.  The latter is also used to implement MAP_NORESERVE,
> which the VOID can't support.

I played around a bit and I can confirm it would be consistent with
current behavior:
memwrite <0 - filesize) - no error, written to file
memwrite <filesize - 4k) - no error, no file change
memwrite <4k, 64k) - SIGSEGV
memwrite <64k, mmap alloc size) - SIGSEGV or another mem alloc
overwrite (depending on whether there is another allocation)
With workaround last line would be fixed to SIGBUS (along with proper
allocation length). I believe this is completely OK.

>
> > - is it possible that some subsequent mem alloc request would return
> > region from |-- THE VOID 60K --| which could again cause segfault
> > after munmap?
>
> No, as stated above.  Allocations are restricted to Windows' 64K
> allocation granularity.

I apologize. I missed that sentence. So, your workaround seems fine.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: possible problem with memory allocation using calloc/mmap/munmap

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3995 bytes --]

On Jun  4 18:01, Stanislav Kascak wrote:
> > > > > > > It seems that when mmap() is called with length argument exceeding
> > > > > > > size of file, only memory to fit that file is allocated. munmap()
> > > > > > > however frees the full specified length. [...]
> > > > > > [...]
> > > > > > I know this situation is unsatisfying, but I have no easy workaround
> > > > > > to allow this.  Cygwin could add the anonymous mapping on the next
> > > > > > 64K boundary on 64 bit, but that would result in a hole in the mapping
> > > > > > which seemed like a rather bad idea when porting mmap to 64 bit.
> > > > > >
> > > > > > Ken's also right that munmap is doing the right thing here.  If
> > > > > > anything's wrong, it's mmap's workaround for mappings beyond the file
> > > > > > length.  If only 64 bit would allow 4K-aligned mappings :(
> > > > >
> > > > > Thanks for the answer. It is appreciated.
> > > > > I understand the problem and difficulty to resolve it. Maybe returning
> > > > > an error from mmap (and putting a comment to code for its reason)
> > > > > would be sufficient. mmap caller could just adjust requested
> > > > > allocation size to file size. Without error, caller has no way of
> > > > > knowing memory was not allocated and segfault is then thrown in an
> > > > > unrelated memory segment which makes the root cause hard to track
> > > > > down. But, I do not know all the implication that could result from
> > > > > that, so evaluation of this approach is up to you.
> > > > [...]
> > > > Eventually Cygwin adds another mapping to fullfill the entire mapping
> > > > request:
> > > >
> > > >  |-- file 4K --|-- filler 60K --|-- filler 192K --|
> > > >
> > > > The problem on WOW64 and real 64 bit is that it's impossible to map
> > > > the first filler.  However, this area in the VM will *never* be
> > > > allocated by other application functions due to the allocation
> > > > granularity of 64K!
> > > >
> > > > So my workaround for 64 bit and WOW64 is to just skip allocating the
> > > > first filler:
> > > >
> > > >  |-- file 4K --|-- THE VOID 60K --|-- filler 192K --|
> > > >
> > > > The advantage is now that the following munmap of 256K will only
> > > > unmap the map for the file and the filler, but not the region you
> > > > calloced before, which formerly was accidentally mapped to the
> > > > filler region.  This just can't happen anymore now.
> > > >
> > > > Would that be feasible?  If so I can push my patch and create a
> > > > developer snapshot for testing.
> > >
> > > Two questions arise when I'm thinking about workaround solution:
> > > - what happens if caller tries to write to |-- THE VOID 60K --|. Since
> > > this is unallocated, would there be a segfault?
> >
> > Accessing the VOID would raise SIGSEGV, while accessing the filler
> > raises SIGBUS.  The latter is also used to implement MAP_NORESERVE,
> > which the VOID can't support.
> 
> I played around a bit and I can confirm it would be consistent with
> current behavior:
> memwrite <0 - filesize) - no error, written to file
> memwrite <filesize - 4k) - no error, no file change
> memwrite <4k, 64k) - SIGSEGV
> memwrite <64k, mmap alloc size) - SIGSEGV or another mem alloc
> overwrite (depending on whether there is another allocation)
> With workaround last line would be fixed to SIGBUS (along with proper
> allocation length). I believe this is completely OK.
> 
> >
> > > - is it possible that some subsequent mem alloc request would return
> > > region from |-- THE VOID 60K --| which could again cause segfault
> > > after munmap?
> >
> > No, as stated above.  Allocations are restricted to Windows' 64K
> > allocation granularity.
> 
> I apologize. I missed that sentence. So, your workaround seems fine.

Please try the latest snapshot from https://cygwin.com/snapshots/
Just replacing the Cygwin DLL is sufficient.


Thanks,
Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: possible problem with memory allocation using calloc/mmap/munmap

[Stanislav Kascak]

> > > > > > > > It seems that when mmap() is called with length argument exceeding
> > > > > > > > size of file, only memory to fit that file is allocated. munmap()
> > > > > > > > however frees the full specified length. [...]
> > > > > > > [...]
> > > > > > > I know this situation is unsatisfying, but I have no easy workaround
> > > > > > > to allow this.  Cygwin could add the anonymous mapping on the next
> > > > > > > 64K boundary on 64 bit, but that would result in a hole in the mapping
> > > > > > > which seemed like a rather bad idea when porting mmap to 64 bit.
> > > > > > >
> > > > > > > Ken's also right that munmap is doing the right thing here.  If
> > > > > > > anything's wrong, it's mmap's workaround for mappings beyond the file
> > > > > > > length.  If only 64 bit would allow 4K-aligned mappings :(
> > > > > >
> > > > > > Thanks for the answer. It is appreciated.
> > > > > > I understand the problem and difficulty to resolve it. Maybe returning
> > > > > > an error from mmap (and putting a comment to code for its reason)
> > > > > > would be sufficient. mmap caller could just adjust requested
> > > > > > allocation size to file size. Without error, caller has no way of
> > > > > > knowing memory was not allocated and segfault is then thrown in an
> > > > > > unrelated memory segment which makes the root cause hard to track
> > > > > > down. But, I do not know all the implication that could result from
> > > > > > that, so evaluation of this approach is up to you.
> > > > > [...]
> > > > > Eventually Cygwin adds another mapping to fullfill the entire mapping
> > > > > request:
> > > > >
> > > > >  |-- file 4K --|-- filler 60K --|-- filler 192K --|
> > > > >
> > > > > The problem on WOW64 and real 64 bit is that it's impossible to map
> > > > > the first filler.  However, this area in the VM will *never* be
> > > > > allocated by other application functions due to the allocation
> > > > > granularity of 64K!
> > > > >
> > > > > So my workaround for 64 bit and WOW64 is to just skip allocating the
> > > > > first filler:
> > > > >
> > > > >  |-- file 4K --|-- THE VOID 60K --|-- filler 192K --|
> > > > >
> > > > > The advantage is now that the following munmap of 256K will only
> > > > > unmap the map for the file and the filler, but not the region you
> > > > > calloced before, which formerly was accidentally mapped to the
> > > > > filler region.  This just can't happen anymore now.
> > > > >
> > > > > Would that be feasible?  If so I can push my patch and create a
> > > > > developer snapshot for testing.
> > > >
> > > > Two questions arise when I'm thinking about workaround solution:
> > > > - what happens if caller tries to write to |-- THE VOID 60K --|. Since
> > > > this is unallocated, would there be a segfault?
> > >
> > > Accessing the VOID would raise SIGSEGV, while accessing the filler
> > > raises SIGBUS.  The latter is also used to implement MAP_NORESERVE,
> > > which the VOID can't support.
> >
> > I played around a bit and I can confirm it would be consistent with
> > current behavior:
> > memwrite <0 - filesize) - no error, written to file
> > memwrite <filesize - 4k) - no error, no file change
> > memwrite <4k, 64k) - SIGSEGV
> > memwrite <64k, mmap alloc size) - SIGSEGV or another mem alloc
> > overwrite (depending on whether there is another allocation)
> > With workaround last line would be fixed to SIGBUS (along with proper
> > allocation length). I believe this is completely OK.
> >
> > >
> > > > - is it possible that some subsequent mem alloc request would return
> > > > region from |-- THE VOID 60K --| which could again cause segfault
> > > > after munmap?
> > >
> > > No, as stated above.  Allocations are restricted to Windows' 64K
> > > allocation granularity.
> >
> > I apologize. I missed that sentence. So, your workaround seems fine.
>
> Please try the latest snapshot from https://cygwin.com/snapshots/
> Just replacing the Cygwin DLL is sufficient.

My test programs work fine and also openldap crash that led me to this
issue is gone (openldap tests still fail, but on a different place
after the one above and now with a SIGBUS which already gives a good
hint as to where to look).
I think your patch works quite well. Thanks a lot.

Stanislav Kascak

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: possible problem with memory allocation using calloc/mmap/munmap

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]

On Jun  6 15:13, Stanislav Kascak wrote:
> > > [...]
> > > I played around a bit and I can confirm it would be consistent with
> > > current behavior:
> > > memwrite <0 - filesize) - no error, written to file
> > > memwrite <filesize - 4k) - no error, no file change
> > > memwrite <4k, 64k) - SIGSEGV
> > > memwrite <64k, mmap alloc size) - SIGSEGV or another mem alloc
> > > overwrite (depending on whether there is another allocation)
> > > With workaround last line would be fixed to SIGBUS (along with proper
> > > allocation length). I believe this is completely OK.
> > >
> > > >
> > > > > - is it possible that some subsequent mem alloc request would return
> > > > > region from |-- THE VOID 60K --| which could again cause segfault
> > > > > after munmap?
> > > >
> > > > No, as stated above.  Allocations are restricted to Windows' 64K
> > > > allocation granularity.
> > >
> > > I apologize. I missed that sentence. So, your workaround seems fine.
> >
> > Please try the latest snapshot from https://cygwin.com/snapshots/
> > Just replacing the Cygwin DLL is sufficient.
> 
> My test programs work fine and also openldap crash that led me to this
> issue is gone (openldap tests still fail, but on a different place
> after the one above and now with a SIGBUS which already gives a good
> hint as to where to look).
> I think your patch works quite well. Thanks a lot.

Thanks for testing!


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]