Question about flock - potential memory corruption?

Question about flock - potential memory corruption?

[Qian Hong]

[-- Attachment #1: Type: text/plain, Size: 2704 bytes --]

Dear list,

When testing Cygwin/MSYS2 on Wine, I found randomly failure of flock():
https://bugs.wine-staging.com/show_bug.cgi?id=466#c13

I ran MSYS2 with Wine+Valgrind, and found warnings like below when
calling flock():

  7 ==19315== Conditional jump or move depends on uninitialised value(s)
  8 ==19315==    at 0x7BC82750: RtlGetOwnerSecurityDescriptor (sec.c:740)
  9 ==19315==    by 0x7BC9222A: NTDLL_create_struct_sd (sync.c:96)
 10 ==19315==    by 0x7BC92CE4: NtCreateEvent (sync.c:294)
 11 ==19315==    by 0x6107B687: ???
 12 ==19315==    by 0x612FC347: ???

Then I read Cygwin/MSYS2 source code, and found:

--- snip ---
extern PSECURITY_DESCRIPTOR _everyone_sd (void *buf, ACCESS_MASK access);
#define everyone_sd(access) (_everyone_sd (alloca (SD_MIN_SIZE), (access)))
--- snip ---

man alloca says:
       The alloca() function allocates size bytes of space in the
stack frame of the caller.  This temporary space is automatically
freed when
       the function that called alloca() returns to its caller.

However, Cygwin/MSYS2 seems passed a freed pointer to NtCreateEvent:
https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/flock.cc;h=2332f5467e37d124acfd12c0f85a30281f10a952;hb=HEAD#l773

 638 POBJECT_ATTRIBUTES
 639 lockf_t::create_lock_obj_attr (lockfattr_t *attr, ULONG flags)
 640 {
 641   __small_swprintf (attr->name, LOCK_OBJ_NAME_FMT,
 642                     lf_flags & (F_POSIX | F_FLOCK), lf_type,
lf_start, lf_end,
 643                     lf_id, lf_wid, lf_ver);
 644   RtlInitCountedUnicodeString (&attr->uname, attr->name,
 645                                LOCK_OBJ_NAME_LEN * sizeof (WCHAR));
 646   InitializeObjectAttributes (&attr->attr, &attr->uname, flags,
lf_inode->i_dir,
 647                               everyone_sd (FLOCK_EVENT_ACCESS));
 648   return &attr->attr;
 649 }

 772       status = NtCreateEvent (&lf_obj, CYG_EVENT_ACCESS,
 773                               create_lock_obj_attr (&attr, OBJ_INHERIT),
 774                               NotificationEvent, FALSE);

It seems flock() works very stable on Windows according to my previous
testing, however, I have feeling that as a kernel function,
NtCreateEvent on Windows doesn't have terrible affects to the user
space stack of the process, while Wine implements NtCreateEvent as a
user space function, so the old stack was easier to be destroyed.

I write a hack as attachment 0001-cygwin-flock-user-static-buffer.txt
and recompile MSYS2, then the bug seems go away.

Could someone confirm whether there is a potential Cygwin bug? If true
I'd love to leave the bug for Cygwin devs to write a fix.

Thanks very much!



-- 
Regards,
Qian Hong

-
http://www.winehq.org

[-- Attachment #2: 0001-cygwin-flock-use-static-buffer.txt --]
[-- Type: text/plain, Size: 783 bytes --]

diff --git a/winsup/cygwin/flock.cc b/winsup/cygwin/flock.cc
index 2332f54..ac3f772 100644
--- a/winsup/cygwin/flock.cc
+++ b/winsup/cygwin/flock.cc
@@ -638,13 +638,14 @@ inode_t::get_all_locks_list ()
 POBJECT_ATTRIBUTES
 lockf_t::create_lock_obj_attr (lockfattr_t *attr, ULONG flags)
 {
+  static char buf[SD_MIN_SIZE];
   __small_swprintf (attr->name, LOCK_OBJ_NAME_FMT,
 		    lf_flags & (F_POSIX | F_FLOCK), lf_type, lf_start, lf_end,
 		    lf_id, lf_wid, lf_ver);
   RtlInitCountedUnicodeString (&attr->uname, attr->name,
 			       LOCK_OBJ_NAME_LEN * sizeof (WCHAR));
   InitializeObjectAttributes (&attr->attr, &attr->uname, flags, lf_inode->i_dir,
-			      everyone_sd (FLOCK_EVENT_ACCESS));
+			      _everyone_sd (buf, FLOCK_EVENT_ACCESS));
   return &attr->attr;
 }
 

[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Question about flock - potential memory corruption?

[Qian Hong]

Hi,

I was still not able to make valgrind display Cygwin symbols, so I
manually translate the address to line this time.

Tested with
$ uname -a
CYGWIN_NT-5.1  2.2.1(0.289/5/3) 2015-08-20 11:40 i686 Cygwin

==29863== Conditional jump or move depends on uninitialised value(s)
==29863==    at 0x7BC82768: RtlGetOwnerSecurityDescriptor (sec.c:740)
==29863==    by 0x7BC92356: NTDLL_create_struct_sd (sync.c:96)
==29863==    by 0x7BC92E10: NtCreateEvent (sync.c:294)
==29863==    by 0x6107B937: ???
==29863==    by 0x61304697: ???

Here 0x6107b937 is the call to NtCreateEvent() inside
lockf_t::create_lock_obj() at:
https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/flock.cc;h=2332f5467e37d124acfd12c0f85a30281f10a952;hb=HEAD#l772

(I can't explain what the address 0x61304697 means, it seems a bit
weird to me, not sure if it is valgrind related)

Related Wine source code:
https://github.com/wine-compholio/wine-patched/blob/8b3a785e97a7e28ff58731b58d19237a59239acc/dlls/ntdll/sync.c#L294
https://github.com/wine-compholio/wine-patched/blob/8b3a785e97a7e28ff58731b58d19237a59239acc/dlls/ntdll/sync.c#L96
https://github.com/wine-compholio/wine-patched/blob/8b3a785e97a7e28ff58731b58d19237a59239acc/dlls/ntdll/sec.c#L740

According to valgrind, lpsd->Control in
RtlGetOwnerSecurityDescriptor() is uninitialized, which means
((SECURITY_DESCRIPTOR*)attr->SecurityDescriptor)->Control in
NtCreateEvent(,attr,) is not initialized, that's why I'm looking at
create_lock_obj_attr() and everyone_sd(). It seems after
create_lock_obj_attr returned the memory allocated by alloca() is
gone, which cause the valgrind warnings, also cause the random
failures.

If there is any further information required I'm glad to test and provide.

Any comments are great appreciated!

Thanks again!

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Question about flock - potential memory corruption?

[Qian Hong]

[-- Attachment #1: Type: text/plain, Size: 198 bytes --]

Hi,

Attach is the simple test case I use to reproduce the valgrind warning
and random failures.

Compile the test case in Cygwin with below command line:
$ gcc -g -O0 flock.c -o flock.exe

Thanks!

[-- Attachment #2: flock.c --]
[-- Type: text/x-csrc, Size: 1205 bytes --]

/***********************************************************************
 * This is a STC that causes the following error on my test machine:
 *   NtCreateEvent(lock): 0xC0000035
 *
 * It tries to use flock() for file locking. It creates a temporary
 * file, the uses fork to spawn a number of children. Each child opens
 * the file, then repeatedly uses flock to lock and unlock it.
 *
 * This test was extracted from the APR test suite.
 *
 * Compile: gcc -Wall -o stc-flock-fork stc-flock-fork.c
 ***********************************************************************/

#include <sys/types.h>
#include <sys/file.h>
#include <sys/wait.h>

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>

#define MAX_ITER 2000
#define CHILDREN 6

/* A temporary file used for flock. */
char tmpfilename[] = "/tmp/flocktstXXXXXX";

int main(int argc, const char * const * argv, const char * const *env)
{
    int fd;
 
    /* Create the temporary file. */
    fd = mkstemp(tmpfilename);
    if (fd < 0) {
        perror("open failed");
        exit(1);
    }
    close(fd);
	int fd2 = open(tmpfilename, O_RDONLY);
	int rc;
	int i;
	rc = flock(fd2, LOCK_EX);
	return 0;
}

[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Question about flock - potential memory corruption?

[Sam Edge]

On 07/09/2015 19:45, Qian Hong wrote:
> Hi,
>
> Attach is the simple test case I use to reproduce the valgrind warning
> and random failures.
>
> Compile the test case in Cygwin with below command line:
> $ gcc -g -O0 flock.c -o flock.exe
>
> Thanks!
>
Hi,

Erm ... slight technical hitch? Your example flock.c doesn't call
fork(), nor does it use your two macros MAX_ITER & CHILDREN.

-- 
Sam Edge

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Question about flock - potential memory corruption?

[Qian Hong]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Hi Sam,

Thank you very much for your reply!

On Tue, Sep 8, 2015 at 12:41 PM, Sam Edge <sam.edge@dwalin.fsnet.co.uk> wrote:
> Erm ... slight technical hitch? Your example flock.c doesn't call
> fork(), nor does it use your two macros MAX_ITER & CHILDREN.

Good question. This problem was originally found when running autom4te on Wine.

In autom4te-2.69, we have:
 68 my $flock_implemented = 'yes';
 982 $icache_file = new Autom4te::XFile $icache, O_RDWR|O_CREAT;
 983 $icache_file->lock (LOCK_EX)
 984   if ($flock_implemented eq "yes");

This cause problem on Wine on some machine, while it works fine on
some other machine.

After tracking down, I successfully reproduce the problem with Cygwin
using the attached test case stc-flock-fork-2.c, which is changed
based on another test case from an old cygwin mailing list archive
[1].

Later then, I found that even without fork(), we can still reproduce
the same problem, Valgrind + Wine shows same warning either with or
without fork, so I simply the test case a bit further, that's why you
didn't see fork in my test case.

Thank for your comment, it's good to make things clearer to everybody.

Any further comment is welcome!

[1] https://www.cygwin.com/ml/cygwin/2011-08/txt00012.txt

-- 
Regards,
Qian Hong

-
http://www.winehq.org

[-- Attachment #2: stc-flock-fork-2.c --]
[-- Type: text/x-csrc, Size: 2505 bytes --]

/***********************************************************************
 * This is a STC that causes the following error on my test machine:
 *   NtCreateEvent(lock): 0xC0000035
 *
 * It tries to use flock() for file locking. It creates a temporary
 * file, the uses fork to spawn a number of children. Each child opens
 * the file, then repeatedly uses flock to lock and unlock it.
 *
 * This test was extracted from the APR test suite.
 *
 * Compile: gcc -Wall -o stc-flock-fork stc-flock-fork.c
 ***********************************************************************/

#include <sys/types.h>
#include <sys/file.h>
#include <sys/wait.h>

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>

#define MAX_ITER 3
#define CHILDREN 2

/* A temporary file used for flock. */
char tmpfilename[] = "/tmp/flocktstXXXXXX";

/* Fork and use flock to lock and unlock the file repeatedly in the child. */
void make_child(int trylock, pid_t *pid)
{
    if ((*pid = fork()) < 0) {
        perror("fork failed");
        exit(1);
    }
    else if (*pid == 0) {
        int fd2 = open(tmpfilename, O_RDONLY);
        if (fd2 < 0) {
            perror("child open");
            exit(1);
        }

        int rc;
        int i;
        for (i=0; i<MAX_ITER; ++i) {
            do {
                rc = flock(fd2, LOCK_EX);
            } while (rc < 0 && errno == EINTR);
            if (rc < 0) {
                perror("lock");
                exit(1);
            }
            do {
                rc = flock(fd2, LOCK_UN);
            } while (rc < 0 && errno == EINTR);
            if (rc < 0) {
                perror("unlock");
                exit(1);
            }
        }
        exit(0);
    }
}

/* Wait for the child to finish. */
void await_child(pid_t pid)
{
    pid_t pstatus;
    int exit_int;

    do {
        pstatus = waitpid(pid, &exit_int, WUNTRACED);
    } while (pstatus < 0 && errno == EINTR);
}

int main(int argc, const char * const * argv, const char * const *env)
{
    pid_t child[CHILDREN];
    int n;
    int fd;
 
    /* Create the temporary file. */
    fd = mkstemp(tmpfilename);
    if (fd < 0) {
        perror("open failed");
        exit(1);
    }
    close(fd);

    /* Create the children. */
    for (n = 0; n < CHILDREN; n++)
        make_child(0, &child[n]);

    /* Wait for them to finish. */
    for (n = 0; n < CHILDREN; n++)
        await_child(child[n]);

    /* Clean up. */
    unlink(tmpfilename);
    return 0;
}

[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Question about flock - potential memory corruption?

[Corinna Vinschen]

[-- Attachment #1.1: Type: text/plain, Size: 3288 bytes --]

Hi Qian,


sorry for the late reply, somehow I missed this mail.


On Sep  5 04:22, Qian Hong wrote:
> Dear list,
> 
> When testing Cygwin/MSYS2 on Wine, I found randomly failure of flock():
> https://bugs.wine-staging.com/show_bug.cgi?id=466#c13
> 
> I ran MSYS2 with Wine+Valgrind, and found warnings like below when
> calling flock():

I'd still be glad to stick to upstream Cygwin, but, oh well.

>   7 ==19315== Conditional jump or move depends on uninitialised value(s)
>   8 ==19315==    at 0x7BC82750: RtlGetOwnerSecurityDescriptor (sec.c:740)
>   9 ==19315==    by 0x7BC9222A: NTDLL_create_struct_sd (sync.c:96)
>  10 ==19315==    by 0x7BC92CE4: NtCreateEvent (sync.c:294)
>  11 ==19315==    by 0x6107B687: ???
>  12 ==19315==    by 0x612FC347: ???
> 
> Then I read Cygwin/MSYS2 source code, and found:
> 
> --- snip ---
> extern PSECURITY_DESCRIPTOR _everyone_sd (void *buf, ACCESS_MASK access);
> #define everyone_sd(access) (_everyone_sd (alloca (SD_MIN_SIZE), (access)))
> --- snip ---
> 
> man alloca says:
>        The alloca() function allocates size bytes of space in the
> stack frame of the caller.  This temporary space is automatically
> freed when
>        the function that called alloca() returns to its caller.
> 
> However, Cygwin/MSYS2 seems passed a freed pointer to NtCreateEvent:
> https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/flock.cc;h=2332f5467e37d124acfd12c0f85a30281f10a952;hb=HEAD#l773
> 
>  638 POBJECT_ATTRIBUTES
>  639 lockf_t::create_lock_obj_attr (lockfattr_t *attr, ULONG flags)
>  640 {
>  641   __small_swprintf (attr->name, LOCK_OBJ_NAME_FMT,
>  642                     lf_flags & (F_POSIX | F_FLOCK), lf_type,
> lf_start, lf_end,
>  643                     lf_id, lf_wid, lf_ver);
>  644   RtlInitCountedUnicodeString (&attr->uname, attr->name,
>  645                                LOCK_OBJ_NAME_LEN * sizeof (WCHAR));
>  646   InitializeObjectAttributes (&attr->attr, &attr->uname, flags,
> lf_inode->i_dir,
>  647                               everyone_sd (FLOCK_EVENT_ACCESS));
>  648   return &attr->attr;
>  649 }
> 
>  772       status = NtCreateEvent (&lf_obj, CYG_EVENT_ACCESS,
>  773                               create_lock_obj_attr (&attr, OBJ_INHERIT),
>  774                               NotificationEvent, FALSE);

Yuk!  You're right.  This is certainly a bug.

> It seems flock() works very stable on Windows according to my previous
> testing, however, I have feeling that as a kernel function,
> NtCreateEvent on Windows doesn't have terrible affects to the user
> space stack of the process, while Wine implements NtCreateEvent as a
> user space function, so the old stack was easier to be destroyed.
> 
> I write a hack as attachment 0001-cygwin-flock-user-static-buffer.txt
> and recompile MSYS2, then the bug seems go away.

Right, but you can't use a static buffer here.

> Could someone confirm whether there is a potential Cygwin bug? If true
> I'd love to leave the bug for Cygwin devs to write a fix.

Please try the attached patch.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #1.2: 0001-flock.cc-Fix-stack-allocation-from-callee-used-in-ca.patch --]
[-- Type: text/plain, Size: 2868 bytes --]

From 51d38004b2f51ac659f7ccc663c222f5ffe24b80 Mon Sep 17 00:00:00 2001
From: Corinna Vinschen <corinna@vinschen.de>
Date: Tue, 8 Sep 2015 10:57:54 +0200
Subject: [PATCH] flock.cc: Fix stack allocation from callee used in caller

	* flock.cc (lockf_t::create_lock_obj_attr): Add buffer parameter.
	Call _everyone_sd with buffer argument from caller rather than
	everyone_sd with locally allocated stack buffer.
	(lockf_t::create_lock_obj): Call create_lock_obj_attr only once
	outside the loop and with additional buffer argument.
	(lockf_t::open_lock_obj): Call create_lock_obj_attr with additional
	buffer argument.
---
 winsup/cygwin/flock.cc | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/winsup/cygwin/flock.cc b/winsup/cygwin/flock.cc
index 2332f54..f26a76a 100644
--- a/winsup/cygwin/flock.cc
+++ b/winsup/cygwin/flock.cc
@@ -290,7 +290,7 @@ class lockf_t
     { cfree (p); }
 
     POBJECT_ATTRIBUTES create_lock_obj_attr (lockfattr_t *attr,
-					     ULONG flags);
+					     ULONG flags, void *sd_buf);
 
     void create_lock_obj ();
     bool open_lock_obj ();
@@ -636,7 +636,7 @@ inode_t::get_all_locks_list ()
 /* Create the lock object name.  The name is constructed from the lock
    properties which identify it uniquely, all values in hex. */
 POBJECT_ATTRIBUTES
-lockf_t::create_lock_obj_attr (lockfattr_t *attr, ULONG flags)
+lockf_t::create_lock_obj_attr (lockfattr_t *attr, ULONG flags, void *sd_buf)
 {
   __small_swprintf (attr->name, LOCK_OBJ_NAME_FMT,
 		    lf_flags & (F_POSIX | F_FLOCK), lf_type, lf_start, lf_end,
@@ -644,7 +644,7 @@ lockf_t::create_lock_obj_attr (lockfattr_t *attr, ULONG flags)
   RtlInitCountedUnicodeString (&attr->uname, attr->name,
 			       LOCK_OBJ_NAME_LEN * sizeof (WCHAR));
   InitializeObjectAttributes (&attr->attr, &attr->uname, flags, lf_inode->i_dir,
-			      everyone_sd (FLOCK_EVENT_ACCESS));
+			      _everyone_sd (sd_buf, FLOCK_EVENT_ACCESS));
   return &attr->attr;
 }
 
@@ -766,11 +766,13 @@ lockf_t::create_lock_obj ()
 {
   lockfattr_t attr;
   NTSTATUS status;
+  POBJECT_ATTRIBUTES lock_obj_attr;
 
+  lock_obj_attr = create_lock_obj_attr (&attr, OBJ_INHERIT,
+					alloca (SD_MIN_SIZE));
   do
     {
-      status = NtCreateEvent (&lf_obj, CYG_EVENT_ACCESS,
-			      create_lock_obj_attr (&attr, OBJ_INHERIT),
+      status = NtCreateEvent (&lf_obj, CYG_EVENT_ACCESS, lock_obj_attr,
 			      NotificationEvent, FALSE);
       if (!NT_SUCCESS (status))
 	{
@@ -852,7 +854,7 @@ lockf_t::open_lock_obj ()
   NTSTATUS status;
 
   status = NtOpenEvent (&lf_obj, FLOCK_EVENT_ACCESS,
-			create_lock_obj_attr (&attr, 0));
+			create_lock_obj_attr (&attr, 0, alloca (SD_MIN_SIZE)));
   if (!NT_SUCCESS (status))
     {
       SetLastError (RtlNtStatusToDosError (status));
-- 
2.1.0


[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Question about flock - potential memory corruption?

[Qian Hong]

Hi Corinna,

Thanks a lot for your time.

On Tue, Sep 8, 2015 at 4:58 PM, Corinna Vinschen
<corinna-cygwin@cygwin.com> wrote:

> I'd still be glad to stick to upstream Cygwin, but, oh well.
Understand, will take care of that.


> Please try the attached patch.
Thanks a lot for the patch, I can confirm it works for me, flock()
doesn't fail on Wine anymore. Also, valgrind doesn't report the
previous warnings.

Once you have snapshot build I'm glad to retest.

Thanks!



-- 
Regards,
Qian Hong

-
http://www.winehq.org

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Question about flock - potential memory corruption?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

On Sep  8 18:35, Qian Hong wrote:
> Hi Corinna,
> 
> Thanks a lot for your time.
> 
> On Tue, Sep 8, 2015 at 4:58 PM, Corinna Vinschen
> <corinna-cygwin@cygwin.com> wrote:
> 
> > I'd still be glad to stick to upstream Cygwin, but, oh well.
> Understand, will take care of that.
> 
> 
> > Please try the attached patch.
> Thanks a lot for the patch, I can confirm it works for me, flock()
> doesn't fail on Wine anymore. Also, valgrind doesn't report the
> previous warnings.
> 
> Once you have snapshot build I'm glad to retest.

Snapshot is up.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Question about flock - potential memory corruption?

[Qian Hong]

On Tue, Sep 8, 2015 at 11:31 PM, Corinna Vinschen
<corinna-cygwin@cygwin.com> wrote:
> Snapshot is up.

Thanks! Tested with and without valgrind, confirming fixed.

$ uname -a
CYGWIN_NT-5.1  2.3.0(0.290/5/3) 2015-09-08 14:46 i686 Cygwin

-- 
Regards,
Qian Hong

-
http://www.winehq.org

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple