Broken autoconf mmap test

Broken autoconf mmap test

[Ken Brown]

What's the status of the broken autoconf mmap test, which always fails 
on Cygwin even though Cygwin has a working mmap?  The last message I can 
find on the cygwin list is from November 2009:

   http://cygwin.com/ml/cygwin/2009-11/msg00368.html

Was this ever resolved?  I see that cygport is still using the workaround

   export ac_cv_func_mmap_fixed_mapped=yes

in autotools.cygclass.  And I still have to do something similar when 
building texlive, even though the latter uses autoconf 2.68.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken autoconf mmap test

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

On 03/20/2011 02:51 PM, Ken Brown wrote:
> What's the status of the broken autoconf mmap test, which always fails
> on Cygwin even though Cygwin has a working mmap?

Autoconf 2.65 and newer do not have the bug.  But packages still exist
where configure was generated with < 2.65, so these packages still have
the bug unless you either rerun newer autoconf, or pre-seed the cache.

> Was this ever resolved?  I see that cygport is still using the workaround
> 
>   export ac_cv_func_mmap_fixed_mapped=yes

This workaround is still necessary, as long as there are still packages
that don't run autoreconf and where configure was generated with a
too-old autoconf.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Broken autoconf mmap test

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]

On 3/22/2011 4:59 PM, Eric Blake wrote:
> On 03/20/2011 02:51 PM, Ken Brown wrote:
>> What's the status of the broken autoconf mmap test, which always fails
>> on Cygwin even though Cygwin has a working mmap?
>
> Autoconf 2.65 and newer do not have the bug.  But packages still exist
> where configure was generated with<  2.65, so these packages still have
> the bug unless you either rerun newer autoconf, or pre-seed the cache.
>
>> Was this ever resolved?  I see that cygport is still using the workaround
>>
>>    export ac_cv_func_mmap_fixed_mapped=yes
>
> This workaround is still necessary, as long as there are still packages
> that don't run autoreconf and where configure was generated with a
> too-old autoconf.

I'm still seeing failures when building both emacs and texlive, even 
after autoreconf with the current (2.68) version installed.  Attached is 
the relevant portion of config.log from an emacs build.  This is a build 
of the current emacs trunk, after running 'autoreconf -I m4'.  The same 
thing happens if I build emacs-23.3.  If you want to try to reproduce 
it, here's the source:

   wget http://ftp.gnu.org/pub/gnu/emacs/emacs-23.3.tar.gz

In case it's relevant, my system is 64-bit Windows 7, with the most 
recent cygwin snapshot:

$ uname -a
CYGWIN_NT-6.1-WOW64 moufang 1.7.9s(0.236/5/3) 20110318 10:29:21 i686 Cygwin

Ken

[-- Attachment #2: config.log --]
[-- Type: text/plain, Size: 7556 bytes --]

configure:9314: checking for working mmap
configure:9461: gcc -o conftest.exe -g -O2     conftest.c  >&5 
configure:9461: $? = 0
configure:9461: ./conftest.exe
./configure: line 2146:  6404 Segmentation fault      (core dumped) ./conftest$ac_exeext
configure:9461: $? = 139
configure: program exited with status 139
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "emacs"
| #define PACKAGE_TARNAME "emacs"
| #define PACKAGE_VERSION "24.0.50"
| #define PACKAGE_STRING "emacs 24.0.50"
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE_URL ""
| #define PACKAGE "emacs"
| #define VERSION "24.0.50"
| #define MAIL_USE_POP 1
| #define SYNC_INPUT 1
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define __EXTENSIONS__ 1
| #define _ALL_SOURCE 1
| #define _GNU_SOURCE 1
| #define _POSIX_PTHREAD_SEMANTICS 1
| #define _TANDEM_SOURCE 1
| #define HAVE_SYS_SOUNDCARD_H 1
| #define HAVE_SYS_SELECT_H 1
| #define HAVE_SYS_TIME_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_UTIME_H 1
| #define HAVE_LIMITS_H 1
| #define HAVE_FCNTL_H 1
| #define HAVE_PTY_H 1
| #define HAVE_SYS_MMAN_H 1
| #define HAVE_SYS_RESOURCE_H 1
| #define HAVE_LOCALE_H 1
| #define HAVE_SYS_UTSNAME_H 1
| #define HAVE_PWD_H 1
| #define HAVE_UTMP_H 1
| #define HAVE_DIRENT_H 1
| #define STDC_HEADERS 1
| #define TIME_WITH_SYS_TIME 1
| #define HAVE_DECL_SYS_SIGLIST 0
| #define HAVE_DECL___SYS_SIGLIST 0
| #define HAVE_SYS_WAIT_H 1
| #define HAVE_STRUCT_UTIMBUF 1
| #define RETSIGTYPE void
| #define HAVE_SPEED_T 1
| #define HAVE_TIMEVAL 1
| #define HAVE_SYS_SOCKET_H 1
| #define HAVE_NET_IF_H 1
| #define HAVE_DECL_TZNAME 1
| #define HAVE_TZNAME 1
| #define HAVE_STRUCT_IFREQ_IFR_FLAGS 1
| #define HAVE_STRUCT_IFREQ_IFR_HWADDR 1
| #define HAVE_STRUCT_IFREQ_IFR_NETMASK 1
| #define HAVE_STRUCT_IFREQ_IFR_BROADADDR 1
| #define HAVE_STRUCT_IFREQ_IFR_ADDR 1
| #define PROTOTYPES 1
| #define __PROTOTYPES 1
| #define POINTER_TYPE void
| #define HAVE_ATTRIBUTE_ALIGNED 1
| #define HAVE_LONG_FILE_NAMES 1
| #define HAVE_STDLIB_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_SYS_PARAM_H 1
| #define HAVE_GETOPT_H 1
| #define HAVE_WCHAR_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_SYS_TIME_H 1
| #define HAVE_GETPAGESIZE 1
| /* end confdefs.h.  */
| #include <stdio.h>
| #ifdef HAVE_SYS_TYPES_H
| # include <sys/types.h>
| #endif
| #ifdef HAVE_SYS_STAT_H
| # include <sys/stat.h>
| #endif
| #ifdef STDC_HEADERS
| # include <stdlib.h>
| # include <stddef.h>
| #else
| # ifdef HAVE_STDLIB_H
| #  include <stdlib.h>
| # endif
| #endif
| #ifdef HAVE_STRING_H
| # if !defined STDC_HEADERS && defined HAVE_MEMORY_H
| #  include <memory.h>
| # endif
| # include <string.h>
| #endif
| #ifdef HAVE_STRINGS_H
| # include <strings.h>
| #endif
| #ifdef HAVE_INTTYPES_H
| # include <inttypes.h>
| #endif
| #ifdef HAVE_STDINT_H
| # include <stdint.h>
| #endif
| #ifdef HAVE_UNISTD_H
| # include <unistd.h>
| #endif
| /* malloc might have been renamed as rpl_malloc. */
| #undef malloc
| 
| /* Thanks to Mike Haertel and Jim Avera for this test.
|    Here is a matrix of mmap possibilities:
| 	mmap private not fixed
| 	mmap private fixed at somewhere currently unmapped
| 	mmap private fixed at somewhere already mapped
| 	mmap shared not fixed
| 	mmap shared fixed at somewhere currently unmapped
| 	mmap shared fixed at somewhere already mapped
|    For private mappings, we should verify that changes cannot be read()
|    back from the file, nor mmap's back from the file at a different
|    address.  (There have been systems where private was not correctly
|    implemented like the infamous i386 svr4.0, and systems where the
|    VM page cache was not coherent with the file system buffer cache
|    like early versions of FreeBSD and possibly contemporary NetBSD.)
|    For shared mappings, we should conversely verify that changes get
|    propagated back to all the places they're supposed to be.
| 
|    Grep wants private fixed already mapped.
|    The main things grep needs to know about mmap are:
|    * does it exist and is it safe to write into the mmap'd area
|    * how to use it (BSD variants)  */
| 
| #include <fcntl.h>
| #include <sys/mman.h>
| 
| #if !defined STDC_HEADERS && !defined HAVE_STDLIB_H
| char *malloc ();
| #endif
| 
| /* This mess was copied from the GNU getpagesize.h.  */
| #ifndef HAVE_GETPAGESIZE
| # ifdef _SC_PAGESIZE
| #  define getpagesize() sysconf(_SC_PAGESIZE)
| # else /* no _SC_PAGESIZE */
| #  ifdef HAVE_SYS_PARAM_H
| #   include <sys/param.h>
| #   ifdef EXEC_PAGESIZE
| #    define getpagesize() EXEC_PAGESIZE
| #   else /* no EXEC_PAGESIZE */
| #    ifdef NBPG
| #     define getpagesize() NBPG * CLSIZE
| #     ifndef CLSIZE
| #      define CLSIZE 1
| #     endif /* no CLSIZE */
| #    else /* no NBPG */
| #     ifdef NBPC
| #      define getpagesize() NBPC
| #     else /* no NBPC */
| #      ifdef PAGESIZE
| #       define getpagesize() PAGESIZE
| #      endif /* PAGESIZE */
| #     endif /* no NBPC */
| #    endif /* no NBPG */
| #   endif /* no EXEC_PAGESIZE */
| #  else /* no HAVE_SYS_PARAM_H */
| #   define getpagesize() 8192	/* punt totally */
| #  endif /* no HAVE_SYS_PARAM_H */
| # endif /* no _SC_PAGESIZE */
| 
| #endif /* no HAVE_GETPAGESIZE */
| 
| int
| main ()
| {
|   char *data, *data2, *data3;
|   const char *cdata2;
|   int i, pagesize;
|   int fd, fd2;
| 
|   pagesize = getpagesize ();
| 
|   /* First, make a file with some known garbage in it. */
|   data = (char *) malloc (pagesize);
|   if (!data)
|     return 1;
|   for (i = 0; i < pagesize; ++i)
|     *(data + i) = rand ();
|   umask (0);
|   fd = creat ("conftest.mmap", 0600);
|   if (fd < 0)
|     return 2;
|   if (write (fd, data, pagesize) != pagesize)
|     return 3;
|   close (fd);
| 
|   /* Next, check that the tail of a page is zero-filled.  File must have
|      non-zero length, otherwise we risk SIGBUS for entire page.  */
|   fd2 = open ("conftest.txt", O_RDWR | O_CREAT | O_TRUNC, 0600);
|   if (fd2 < 0)
|     return 4;
|   cdata2 = "";
|   if (write (fd2, cdata2, 1) != 1)
|     return 5;
|   data2 = (char *) mmap (0, pagesize, PROT_READ | PROT_WRITE, MAP_SHARED, fd2, 0L);
|   if (data2 == MAP_FAILED)
|     return 6;
|   for (i = 0; i < pagesize; ++i)
|     if (*(data2 + i))
|       return 7;
|   close (fd2);
|   if (munmap (data2, pagesize))
|     return 8;
| 
|   /* Next, try to mmap the file at a fixed address which already has
|      something else allocated at it.  If we can, also make sure that
|      we see the same garbage.  */
|   fd = open ("conftest.mmap", O_RDWR);
|   if (fd < 0)
|     return 9;
|   if (data2 != mmap (data2, pagesize, PROT_READ | PROT_WRITE,
| 		     MAP_PRIVATE | MAP_FIXED, fd, 0L))
|     return 10;
|   for (i = 0; i < pagesize; ++i)
|     if (*(data + i) != *(data2 + i))
|       return 11;
| 
|   /* Finally, make sure that changes to the mapped area do not
|      percolate back to the file as seen by read().  (This is a bug on
|      some variants of i386 svr4.0.)  */
|   for (i = 0; i < pagesize; ++i)
|     *(data2 + i) = *(data2 + i) + 1;
|   data3 = (char *) malloc (pagesize);
|   if (!data3)
|     return 12;
|   if (read (fd, data3, pagesize) != pagesize)
|     return 13;
|   for (i = 0; i < pagesize; ++i)
|     if (*(data + i) != *(data3 + i))
|       return 14;
|   close (fd);
|   return 0;
| }
configure:9471: result: no


[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin fails autoconf mmap test under Win 7 (was: Broken autoconf mmap test)

[Ken Brown]

On 3/22/2011 5:53 PM, Ken Brown wrote:
> On 3/22/2011 4:59 PM, Eric Blake wrote:
>> On 03/20/2011 02:51 PM, Ken Brown wrote:
>>> What's the status of the broken autoconf mmap test, which always fails
>>> on Cygwin even though Cygwin has a working mmap?
>>
>> Autoconf 2.65 and newer do not have the bug.  But packages still exist
>> where configure was generated with<   2.65, so these packages still have
>> the bug unless you either rerun newer autoconf, or pre-seed the cache.
>>
>>> Was this ever resolved?  I see that cygport is still using the workaround
>>>
>>>     export ac_cv_func_mmap_fixed_mapped=yes
>>
>> This workaround is still necessary, as long as there are still packages
>> that don't run autoreconf and where configure was generated with a
>> too-old autoconf.
>
> I'm still seeing failures when building both emacs and texlive, even
> after autoreconf with the current (2.68) version installed.  Attached is
> the relevant portion of config.log from an emacs build.  This is a build
> of the current emacs trunk, after running 'autoreconf -I m4'.  The same
> thing happens if I build emacs-23.3.  If you want to try to reproduce
> it, here's the source:
>
>     wget http://ftp.gnu.org/pub/gnu/emacs/emacs-23.3.tar.gz
>
> In case it's relevant, my system is 64-bit Windows 7, with the most
> recent cygwin snapshot:
>
> $ uname -a
> CYGWIN_NT-6.1-WOW64 moufang 1.7.9s(0.236/5/3) 20110318 10:29:21 i686 Cygwin

I've just checked that the mmap test succeeds (again when configuring 
emacs-23.3) on an XP SP3 system with the same snapshot installed:

$ uname -a
CYGWIN_NT-5.1 Desktop 1.7.9s(0.236/5/3) 20110318 10:29:21 i686 Cygwin

So maybe this is really a problem with Cygwin's mmap (under Win 7) and 
not an autoconf problem after all.

Ken


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken autoconf mmap test

[Corinna Vinschen]

On Mar 22 17:53, Ken Brown wrote:
> On 3/22/2011 4:59 PM, Eric Blake wrote:
> >On 03/20/2011 02:51 PM, Ken Brown wrote:
> >>What's the status of the broken autoconf mmap test, which always fails
> >>on Cygwin even though Cygwin has a working mmap?
> >
> >Autoconf 2.65 and newer do not have the bug.  But packages still exist
> >where configure was generated with<  2.65, so these packages still have
> >the bug unless you either rerun newer autoconf, or pre-seed the cache.
> >
> >>Was this ever resolved?  I see that cygport is still using the workaround
> >>
> >>   export ac_cv_func_mmap_fixed_mapped=yes
> >
> >This workaround is still necessary, as long as there are still packages
> >that don't run autoreconf and where configure was generated with a
> >too-old autoconf.
> 
> I'm still seeing failures when building both emacs and texlive, even
> after autoreconf with the current (2.68) version installed.
> Attached is the relevant portion of config.log from an emacs build.
> This is a build of the current emacs trunk, after running
> 'autoreconf -I m4'.  The same thing happens if I build emacs-23.3.
> If you want to try to reproduce it, here's the source:
> 
>   wget http://ftp.gnu.org/pub/gnu/emacs/emacs-23.3.tar.gz
> 
> In case it's relevant, my system is 64-bit Windows 7, with the most
> recent cygwin snapshot:

It is relevant, but it has nothing to do with Windows 7.  Actually, the
autoconf mmap test is testing a scenario which does not work on 64 bit
Windows systems.  Here's what happens in a nutshell.

The actual pagesize on Windows is 4K.  But there's something called
"allocation granularity" which is 64K.  If you request virtual memory on
Windows, the memory address is always aligned to the allocation
granularity boundary.  Even if you request 4K chunks, they are 64K
aligned.

If you mmap anonymous memory, this is no big problem.  Cygwin always
requests memory in 64K chunks since Cygwin's pagesize is set to 64K.

However, if you mmap a file, there's a problem.  The mapping always
starts on a 64K boundary, but it end on the next 4K boundary beyond EOF.
So, for a file of size 1 byte, the memory section will be 4K.  The other
60K following in the virtual address space are lost.  And they are not
allocated.  And i you try to access them you get a SEGV.

Fortunately, Cygwin uses the native NT calls in mmap.  The native calls
support a flag called AT_ROUND_TO_PAGE, which allows to allocate memory
on a 4K boundary.  So what Cygwin does is to mmap the file, and then to
request virtual memory rounded to the next 4K page to fill the rest of
the 64K allocation area.  Now we have the full 64K allocated.

Unfortunately this doesn't work on 64 bit Windows.  The reason is that
the WOW64 layer does not support the AT_ROUND_TO_PAGE flag.  If you try
to use it, the OS call returns STTAUS_INVALID_PARAMETER.  So you can't
allocate the remainder of the 64K allocation slot at all.

Back to the autoconf mmap test.  What it does is to mmap a file of size
1 byte.  Then it tests each byte in the mapped page.  And since the
pagesize in Cygwin is 64K it tests all of 64K.

Given the above description, you can now imagine this works fine on 32
bit Windows, but it crashes on 64 bit Windows.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken autoconf mmap test

[Ken Brown]

On 3/23/2011 5:19 AM, Corinna Vinschen wrote:
> On Mar 22 17:53, Ken Brown wrote:
>> On 3/22/2011 4:59 PM, Eric Blake wrote:
>>> On 03/20/2011 02:51 PM, Ken Brown wrote:
>>>> What's the status of the broken autoconf mmap test, which always fails
>>>> on Cygwin even though Cygwin has a working mmap?
>>>
>>> Autoconf 2.65 and newer do not have the bug.  But packages still exist
>>> where configure was generated with<   2.65, so these packages still have
>>> the bug unless you either rerun newer autoconf, or pre-seed the cache.
>>>
>>>> Was this ever resolved?  I see that cygport is still using the workaround
>>>>
>>>>    export ac_cv_func_mmap_fixed_mapped=yes
>>>
>>> This workaround is still necessary, as long as there are still packages
>>> that don't run autoreconf and where configure was generated with a
>>> too-old autoconf.
>>
>> I'm still seeing failures when building both emacs and texlive, even
>> after autoreconf with the current (2.68) version installed.
>> Attached is the relevant portion of config.log from an emacs build.
>> This is a build of the current emacs trunk, after running
>> 'autoreconf -I m4'.  The same thing happens if I build emacs-23.3.
>> If you want to try to reproduce it, here's the source:
>>
>>    wget http://ftp.gnu.org/pub/gnu/emacs/emacs-23.3.tar.gz
>>
>> In case it's relevant, my system is 64-bit Windows 7, with the most
>> recent cygwin snapshot:
>
> It is relevant, but it has nothing to do with Windows 7.  Actually, the
> autoconf mmap test is testing a scenario which does not work on 64 bit
> Windows systems.  Here's what happens in a nutshell.
>
> The actual pagesize on Windows is 4K.  But there's something called
> "allocation granularity" which is 64K.  If you request virtual memory on
> Windows, the memory address is always aligned to the allocation
> granularity boundary.  Even if you request 4K chunks, they are 64K
> aligned.
>
> If you mmap anonymous memory, this is no big problem.  Cygwin always
> requests memory in 64K chunks since Cygwin's pagesize is set to 64K.
>
> However, if you mmap a file, there's a problem.  The mapping always
> starts on a 64K boundary, but it end on the next 4K boundary beyond EOF.
> So, for a file of size 1 byte, the memory section will be 4K.  The other
> 60K following in the virtual address space are lost.  And they are not
> allocated.  And i you try to access them you get a SEGV.
>
> Fortunately, Cygwin uses the native NT calls in mmap.  The native calls
> support a flag called AT_ROUND_TO_PAGE, which allows to allocate memory
> on a 4K boundary.  So what Cygwin does is to mmap the file, and then to
> request virtual memory rounded to the next 4K page to fill the rest of
> the 64K allocation area.  Now we have the full 64K allocated.
>
> Unfortunately this doesn't work on 64 bit Windows.  The reason is that
> the WOW64 layer does not support the AT_ROUND_TO_PAGE flag.  If you try
> to use it, the OS call returns STTAUS_INVALID_PARAMETER.  So you can't
> allocate the remainder of the 64K allocation slot at all.
>
> Back to the autoconf mmap test.  What it does is to mmap a file of size
> 1 byte.  Then it tests each byte in the mapped page.  And since the
> pagesize in Cygwin is 64K it tests all of 64K.
>
> Given the above description, you can now imagine this works fine on 32
> bit Windows, but it crashes on 64 bit Windows.

Thanks for the explanation, Corinna.  I've reported this to the autoconf 
list.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken autoconf mmap test

[Corinna Vinschen]

On Mar 23 07:08, Ken Brown wrote:
> On 3/23/2011 5:19 AM, Corinna Vinschen wrote:
> >On Mar 22 17:53, Ken Brown wrote:
> >>In case it's relevant, my system is 64-bit Windows 7, with the most
> >>recent cygwin snapshot:
> >
> >It is relevant, but it has nothing to do with Windows 7.  Actually, the
> >autoconf mmap test is testing a scenario which does not work on 64 bit
> >Windows systems.  Here's what happens in a nutshell.
> >
> >The actual pagesize on Windows is 4K.  But there's something called
> >"allocation granularity" which is 64K.  If you request virtual memory on
> >Windows, the memory address is always aligned to the allocation
> >granularity boundary.  Even if you request 4K chunks, they are 64K
> >aligned.
> >
> >If you mmap anonymous memory, this is no big problem.  Cygwin always
> >requests memory in 64K chunks since Cygwin's pagesize is set to 64K.
> >
> >However, if you mmap a file, there's a problem.  The mapping always
> >starts on a 64K boundary, but it end on the next 4K boundary beyond EOF.
> >So, for a file of size 1 byte, the memory section will be 4K.  The other
> >60K following in the virtual address space are lost.  And they are not
> >allocated.  And i you try to access them you get a SEGV.
> >
> >Fortunately, Cygwin uses the native NT calls in mmap.  The native calls
> >support a flag called AT_ROUND_TO_PAGE, which allows to allocate memory
> >on a 4K boundary.  So what Cygwin does is to mmap the file, and then to
> >request virtual memory rounded to the next 4K page to fill the rest of
> >the 64K allocation area.  Now we have the full 64K allocated.
> >
> >Unfortunately this doesn't work on 64 bit Windows.  The reason is that
> >the WOW64 layer does not support the AT_ROUND_TO_PAGE flag.  If you try
> >to use it, the OS call returns STTAUS_INVALID_PARAMETER.  So you can't
> >allocate the remainder of the 64K allocation slot at all.
> >
> >Back to the autoconf mmap test.  What it does is to mmap a file of size
> >1 byte.  Then it tests each byte in the mapped page.  And since the
> >pagesize in Cygwin is 64K it tests all of 64K.
> >
> >Given the above description, you can now imagine this works fine on 32
> >bit Windows, but it crashes on 64 bit Windows.
> 
> Thanks for the explanation, Corinna.  I've reported this to the
> autoconf list.

Btw., I just experimented with another flag which has not been exposed
to the Win32 API, and which is not officially documented.  It's called
AT_EXTENDABLE_FILE and allows to specify a view which is bigger than
the filesize.  I tried this on W7-64, and it works.  Just not exactly
as we need it.  The trailing pages are not commited but only reserved,
so accessing them still results in a SEGV.  And while commiting them
allows to run the autoconf testcase, it also changes the filesize of
the file on disk.  Oh well.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken autoconf mmap test

[Ken Brown]

On 3/23/2011 9:26 AM, Corinna Vinschen wrote:
> On Mar 23 07:08, Ken Brown wrote:
>> On 3/23/2011 5:19 AM, Corinna Vinschen wrote:
>>> On Mar 22 17:53, Ken Brown wrote:
>>>> In case it's relevant, my system is 64-bit Windows 7, with the most
>>>> recent cygwin snapshot:
>>>
>>> It is relevant, but it has nothing to do with Windows 7.  Actually, the
>>> autoconf mmap test is testing a scenario which does not work on 64 bit
>>> Windows systems.  Here's what happens in a nutshell.
>>>
>>> The actual pagesize on Windows is 4K.  But there's something called
>>> "allocation granularity" which is 64K.  If you request virtual memory on
>>> Windows, the memory address is always aligned to the allocation
>>> granularity boundary.  Even if you request 4K chunks, they are 64K
>>> aligned.
>>>
>>> If you mmap anonymous memory, this is no big problem.  Cygwin always
>>> requests memory in 64K chunks since Cygwin's pagesize is set to 64K.
>>>
>>> However, if you mmap a file, there's a problem.  The mapping always
>>> starts on a 64K boundary, but it end on the next 4K boundary beyond EOF.
>>> So, for a file of size 1 byte, the memory section will be 4K.  The other
>>> 60K following in the virtual address space are lost.  And they are not
>>> allocated.  And i you try to access them you get a SEGV.
>>>
>>> Fortunately, Cygwin uses the native NT calls in mmap.  The native calls
>>> support a flag called AT_ROUND_TO_PAGE, which allows to allocate memory
>>> on a 4K boundary.  So what Cygwin does is to mmap the file, and then to
>>> request virtual memory rounded to the next 4K page to fill the rest of
>>> the 64K allocation area.  Now we have the full 64K allocated.
>>>
>>> Unfortunately this doesn't work on 64 bit Windows.  The reason is that
>>> the WOW64 layer does not support the AT_ROUND_TO_PAGE flag.  If you try
>>> to use it, the OS call returns STTAUS_INVALID_PARAMETER.  So you can't
>>> allocate the remainder of the 64K allocation slot at all.
>>>
>>> Back to the autoconf mmap test.  What it does is to mmap a file of size
>>> 1 byte.  Then it tests each byte in the mapped page.  And since the
>>> pagesize in Cygwin is 64K it tests all of 64K.
>>>
>>> Given the above description, you can now imagine this works fine on 32
>>> bit Windows, but it crashes on 64 bit Windows.
>>
>> Thanks for the explanation, Corinna.  I've reported this to the
>> autoconf list.
>
> Btw., I just experimented with another flag which has not been exposed
> to the Win32 API, and which is not officially documented.  It's called
> AT_EXTENDABLE_FILE and allows to specify a view which is bigger than
> the filesize.  I tried this on W7-64, and it works.  Just not exactly
> as we need it.  The trailing pages are not commited but only reserved,
> so accessing them still results in a SEGV.  And while commiting them
> allows to run the autoconf testcase, it also changes the filesize of
> the file on disk.  Oh well.

Is the scenario in which the autoconf test fails likely to occur in 
practice?  I wonder if it could be responsible for some of the 
mysterious crashes that people have reported on 64-bit systems.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken autoconf mmap test

[Corinna Vinschen]

On Mar 23 10:08, Ken Brown wrote:
> On 3/23/2011 9:26 AM, Corinna Vinschen wrote:
> >Btw., I just experimented with another flag which has not been exposed
> >to the Win32 API, and which is not officially documented.  It's called
> >AT_EXTENDABLE_FILE and allows to specify a view which is bigger than
> >the filesize.  I tried this on W7-64, and it works.  Just not exactly
> >as we need it.  The trailing pages are not commited but only reserved,
> >so accessing them still results in a SEGV.  And while commiting them
> >allows to run the autoconf testcase, it also changes the filesize of
> >the file on disk.  Oh well.
> 
> Is the scenario in which the autoconf test fails likely to occur in
> practice?  I wonder if it could be responsible for some of the
> mysterious crashes that people have reported on 64-bit systems.

Usually it shouldn't.  While you can write into a mmaped area beyond
EOF up to the end of the last page, it doesn't make a lot of sense.
The data will never show up in the mapped file.

I have no idea if there are applications out there which rely on this
crude behaviour.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken autoconf mmap test

[Ken Brown]

On 3/23/2011 11:44 AM, Corinna Vinschen wrote:
> On Mar 23 10:08, Ken Brown wrote:
>> On 3/23/2011 9:26 AM, Corinna Vinschen wrote:
>>> Btw., I just experimented with another flag which has not been exposed
>>> to the Win32 API, and which is not officially documented.  It's called
>>> AT_EXTENDABLE_FILE and allows to specify a view which is bigger than
>>> the filesize.  I tried this on W7-64, and it works.  Just not exactly
>>> as we need it.  The trailing pages are not commited but only reserved,
>>> so accessing them still results in a SEGV.  And while commiting them
>>> allows to run the autoconf testcase, it also changes the filesize of
>>> the file on disk.  Oh well.
>>
>> Is the scenario in which the autoconf test fails likely to occur in
>> practice?  I wonder if it could be responsible for some of the
>> mysterious crashes that people have reported on 64-bit systems.
>
> Usually it shouldn't.  While you can write into a mmaped area beyond
> EOF up to the end of the last page, it doesn't make a lot of sense.
> The data will never show up in the mapped file.
>
> I have no idea if there are applications out there which rely on this
> crude behaviour.

I was mainly thinking of this from the point of view of a Cygwin package 
maintainer.  In view of what you said, it seems that it's probably safe 
to continue using the workaround

   export ac_cv_func_mmap_fixed_mapped=yes

that's built into cygport.  Maintainers who don't use cygport and who 
build on a 64-bit system will have to manually use a similar workaround, 
or else their builds won't use mmap.

It doesn't look like there's going to be an autoconf change to resolve this:

   http://lists.gnu.org/archive/html/autoconf/2011-03/msg00041.html

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken autoconf mmap test

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

On 03/23/2011 11:40 AM, Ken Brown wrote:
>> Usually it shouldn't.  While you can write into a mmaped area beyond
>> EOF up to the end of the last page, it doesn't make a lot of sense.
>> The data will never show up in the mapped file.
>>
>> I have no idea if there are applications out there which rely on this
>> crude behaviour.
> 
> It doesn't look like there's going to be an autoconf change to resolve
> this:
> 
>   http://lists.gnu.org/archive/html/autoconf/2011-03/msg00041.html

Unless the right thing to do in autoconf is to separate the test into
two levels; one of whether most mmap works but you can't extend files
(cygwin always passes) and one whether writing beyond EOF works as
required by POSIX (cygwin currently fails on W64, but hopefully fewer
packages rely on this stricter behavior).  Or figure out a way to make
cygwin work around the W64 limitations.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Broken autoconf mmap test

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1233 bytes --]

On 03/23/2011 11:46 AM, Eric Blake wrote:
> On 03/23/2011 11:40 AM, Ken Brown wrote:
>>> Usually it shouldn't.  While you can write into a mmaped area beyond
>>> EOF up to the end of the last page, it doesn't make a lot of sense.
>>> The data will never show up in the mapped file.

Hmm, rereading POSIX:

If the size of the mapped file changes after the call to mmap( ) as a
result of some other operation
on the mapped file, the effect of references to portions of the mapped
region that correspond to
added or removed portions of the file is unspecified.

> 
> Unless the right thing to do in autoconf is to separate the test into
> two levels; one of whether most mmap works but you can't extend files
> (cygwin always passes) and one whether writing beyond EOF works as
> required by POSIX (cygwin currently fails on W64, but hopefully fewer
> packages rely on this stricter behavior).  Or figure out a way to make
> cygwin work around the W64 limitations.

I'm still debating if cygwin is compliant and the autoconf test is
exploiting undefined behavior... this is tricky stuff to get right.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Broken autoconf mmap test

[Corinna Vinschen]

On Mar 23 11:51, Eric Blake wrote:
> On 03/23/2011 11:46 AM, Eric Blake wrote:
> > On 03/23/2011 11:40 AM, Ken Brown wrote:
> >>> Usually it shouldn't.  While you can write into a mmaped area beyond
> >>> EOF up to the end of the last page, it doesn't make a lot of sense.
> >>> The data will never show up in the mapped file.
> 
> Hmm, rereading POSIX:
> 
> If the size of the mapped file changes after the call to mmap( ) as a
> result of some other operation
> on the mapped file, the effect of references to portions of the mapped
> region that correspond to
> added or removed portions of the file is unspecified.

That has nothing to do with what the autoconf test tests.  Consider a
filesize of 1 bytes and a pagesize of 4K.  Since mmap always returns
full pages, the file is mapped into a single 4K page.

The autoconf test only test if you can access the entire page in memory,
and if this page is filled with 0-bytes.

It does not test what happens if you change the file after calling mmap.

> > Unless the right thing to do in autoconf is to separate the test into
> > two levels; one of whether most mmap works but you can't extend files
> > (cygwin always passes) and one whether writing beyond EOF works as
> > required by POSIX (cygwin currently fails on W64, but hopefully fewer
> > packages rely on this stricter behavior).  Or figure out a way to make
> > cygwin work around the W64 limitations.
> 
> I'm still debating if cygwin is compliant and the autoconf test is
> exploiting undefined behavior... this is tricky stuff to get right.

Cygwin tries to be as compliant as possible, and it tries to implement
expectations which are typical for some Unix systems even if they are
not covered by the standards.

And yes, the autoconf test tests defined behaviour, see the SUSv4 mmap
man page:

 "The address range starting at pa and continuing for len bytes shall be
  legitimate for the possible (not necessarily current) address space of
  the process.
  [...]
  The system performs mapping operations over whole pages. Thus, while
  the parameter len need not meet a size or alignment constraint, the
  system shall include, in any mapping operation, any partial page
  specified by the address range starting at pa and continuing for len
  bytes.
  
  The system shall always zero-fill any partial page at the end of an
  object. Further, the system shall never write out any modified
  portions of the last page of an object which are beyond its end."


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken autoconf mmap test

[Charles Wilson]

Corinna Vinschen wrote:
> I had hoped that you, as the autoconf maintainer, would put this
> upstream...

Well, I would have done so...but you guys beat me to it.  I go off-net
for a day or so, and lookit what happens...I think I need to go off-net
more often!

--
Chuck

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple