Problems with native Unix domain sockets on Win 10/2019

Problems with native Unix domain sockets on Win 10/2019

[Michael McMahon]

[-- Attachment #1: Type: text/plain, Size: 1887 bytes --]

Hi,

I searched for related issues but haven't found anything.

I am having some trouble with Windows native Unix domain sockets
(a recent feature in Windows 10 and 2019 server) and Cygwin.
I think I possibly know the cause since I had to investigate a similar
looking issue on another platform built on Windows.

The problem is that cygwin commands don't seem to recognise native Unix
domain sockets correctly. For example, the socket "foo.sock" should
have the same ownership and similar permissions to other files
in the example below:

$ ls -lrt
total 2181303

-rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 test.c
-rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 test.obj
-rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 test.exe
-rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 foo.sock
-rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 check.ot

A bigger problem is that foo.sock can't be deleted with the cygwin "rm"
command.

$ rm -f foo.sock
rm: cannot remove 'foo.sock': Permission denied

$ chmod 777 foo.sock
chmod: changing permissions of 'foo.sock': Permission denied

$ cmd /c del foo.sock

But, native Windows commands are okay, as the third example shows.

I think the problem may relate to the way native Unix domain sockets are
implemented in Windows and the resulting special handling required.
They are implemented as NTFS reparse points and when opening them
with CreateFile, you need to specify the FILE_FLAG_OPEN_REPARSE_POINT
flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
complications unfortunately, which I'd be happy to discuss further.

But, to reproduce it, you can compile the attached code snippet
which creates foo.sock in the current directory. Obviously, this
only works on recent versions of Windows 10 and 2019 server.

All the best,
Michael McMahon


[-- Attachment #2: test.c --]
[-- Type: text/plain, Size: 1316 bytes --]

#define WIN32_LEAN_AND_MEAN

#include <windows.h>
#include <winsock2.h>
#include <WS2tcpip.h>
#include <stddef.h>
#include <stdio.h>
#include <afunix.h>

#define WIN32_LEAN_AND_MEAN


int __cdecl main(int argc, char *argv[])
{
    WSADATA wsadata;
    struct sockaddr_un addr;
    socklen_t len;
    int z = AF_UNIX;
    SOCKET s, s0;

    if (WSAStartup(MAKEWORD(2,2), &wsadata) != 0) {
        printf("STartup failed\n");
        return 0;
    }
    s0 = socket(AF_UNIX, SOCK_STREAM, 0);
    memset(&addr, 0, sizeof(addr));

    addr.sun_family = AF_UNIX;
    //strcpy(addr.sun_path, argv[1]);
    strcpy(addr.sun_path, "foo.sock");

    z = bind(s0, (const struct sockaddr *) &addr, strlen(addr.sun_path) + sizeof (addr.sun_family));
    if (z != 0) {
        printf("bind failed %ld\n", WSAGetLastError());
    }
    len = sizeof(addr);
    z = getsockname(s0, (struct sockaddr *)&addr, &len);
    if (z != 0) {
        printf("getsockname failed %ld\n", WSAGetLastError());
    } else {
        printf("getsockname works\n");
        printf("fam = %d, len = %d\n", addr.sun_family, len);
        int clen = len - offsetof(struct sockaddr_un, sun_path);
        printf("offsetof clen = %d\n", clen);
        printf("strlen = %zd\n", strlen(addr.sun_path));
        printf("name = %s\n", addr.sun_path);
    }
}

Re: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
> Hi,
> 
> I searched for related issues but haven't found anything.
> 
> I am having some trouble with Windows native Unix domain sockets
> (a recent feature in Windows 10 and 2019 server) and Cygwin.
> I think I possibly know the cause since I had to investigate a similar
> looking issue on another platform built on Windows.
> 
> The problem is that cygwin commands don't seem to recognise native Unix
> domain sockets correctly. For example, the socket "foo.sock" should
> have the same ownership and similar permissions to other files
> in the example below:
> 
> $ ls -lrt
> total 2181303
> 
> -rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 test.c
> -rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 test.obj
> -rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 test.exe
> -rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 foo.sock
> -rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 check.ot
> 
> A bigger problem is that foo.sock can't be deleted with the cygwin "rm"
> command.
> 
> $ rm -f foo.sock
> rm: cannot remove 'foo.sock': Permission denied
> 
> $ chmod 777 foo.sock
> chmod: changing permissions of 'foo.sock': Permission denied
> 
> $ cmd /c del foo.sock
> 
> But, native Windows commands are okay, as the third example shows.
> 
> I think the problem may relate to the way native Unix domain sockets are
> implemented in Windows and the resulting special handling required.
> They are implemented as NTFS reparse points and when opening them
> with CreateFile, you need to specify the FILE_FLAG_OPEN_REPARSE_POINT
> flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
> complications unfortunately, which I'd be happy to discuss further.
> 
> But, to reproduce it, you can compile the attached code snippet
> which creates foo.sock in the current directory. Obviously, this
> only works on recent versions of Windows 10 and 2019 server.

Cygwin doesn't currently support native Windows AF_UNIX sockets, as you've 
discovered.  See

   https://cygwin.com/pipermail/cygwin/2020-June/245088.html

for the current state of AF_UNIX sockets on Cygwin, including the possibility of 
using native Windows AF_UNIX sockets on systems that support them.

If all you want is for Cygwin to recognize such sockets and allow you to apply 
rm, chmod, etc., I don't think it would be hard to add that capability.  But I 
doubt if that's all you want.

Further discussion of this will have to wait until Corinna is available.

Ken

Re: Problems with native Unix domain sockets on Win 10/2019

[Michael McMahon]

On 24/09/2020 12:26, Ken Brown wrote:
> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>> Hi,
>>
>> I searched for related issues but haven't found anything.
>>
>> I am having some trouble with Windows native Unix domain sockets
>> (a recent feature in Windows 10 and 2019 server) and Cygwin.
>> I think I possibly know the cause since I had to investigate a similar
>> looking issue on another platform built on Windows.
>>
>> The problem is that cygwin commands don't seem to recognise native Unix
>> domain sockets correctly. For example, the socket "foo.sock" should
>> have the same ownership and similar permissions to other files
>> in the example below:
>>
>> $ ls -lrt
>> total 2181303
>>
>> -rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 test.c
>> -rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 test.obj
>> -rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 test.exe
>> -rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 foo.sock
>> -rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 check.ot
>>
>> A bigger problem is that foo.sock can't be deleted with the cygwin "rm"
>> command.
>>
>> $ rm -f foo.sock
>> rm: cannot remove 'foo.sock': Permission denied
>>
>> $ chmod 777 foo.sock
>> chmod: changing permissions of 'foo.sock': Permission denied
>>
>> $ cmd /c del foo.sock
>>
>> But, native Windows commands are okay, as the third example shows.
>>
>> I think the problem may relate to the way native Unix domain sockets are
>> implemented in Windows and the resulting special handling required.
>> They are implemented as NTFS reparse points and when opening them
>> with CreateFile, you need to specify the FILE_FLAG_OPEN_REPARSE_POINT
>> flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
>> complications unfortunately, which I'd be happy to discuss further.
>>
>> But, to reproduce it, you can compile the attached code snippet
>> which creates foo.sock in the current directory. Obviously, this
>> only works on recent versions of Windows 10 and 2019 server.
> 
> Cygwin doesn't currently support native Windows AF_UNIX sockets, as 
> you've discovered.  See
> 
>    
> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$ 
> 
> for the current state of AF_UNIX sockets on Cygwin, including the 
> possibility of using native Windows AF_UNIX sockets on systems that 
> support them.
> 
> If all you want is for Cygwin to recognize such sockets and allow you to 
> apply rm, chmod, etc., I don't think it would be hard to add that 
> capability.  But I doubt if that's all you want.
> 
> Further discussion of this will have to wait until Corinna is available.
> 

Thanks for the info. It's mainly about recognition of sockets for
regular commands. Since these objects can exist on Windows filesystems
now, potentially created by any kind of Windows application,
it would be great if Cygwin could handle them, irrespective of whether
the Cygwin development environment does. Though that sounds like a
good idea too.

- Michael

Re: Problems with native Unix domain sockets on Win 10/2019

[Brian Inglis]

On 2020-09-24 06:01, Michael McMahon via Cygwin wrote:
> 
> 
> On 24/09/2020 12:26, Ken Brown wrote:
>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>> Hi,
>>>
>>> I searched for related issues but haven't found anything.
>>>
>>> I am having some trouble with Windows native Unix domain sockets
>>> (a recent feature in Windows 10 and 2019 server) and Cygwin.
>>> I think I possibly know the cause since I had to investigate a similar
>>> looking issue on another platform built on Windows.
>>>
>>> The problem is that cygwin commands don't seem to recognise native Unix
>>> domain sockets correctly. For example, the socket "foo.sock" should
>>> have the same ownership and similar permissions to other files
>>> in the example below:
>>>
>>> $ ls -lrt
>>> total 2181303
>>>
>>> -rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 test.c
>>> -rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 test.obj
>>> -rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 test.exe
>>> -rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 foo.sock
>>> -rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 check.ot
>>>
>>> A bigger problem is that foo.sock can't be deleted with the cygwin "rm"
>>> command.
>>>
>>> $ rm -f foo.sock
>>> rm: cannot remove 'foo.sock': Permission denied
>>>
>>> $ chmod 777 foo.sock
>>> chmod: changing permissions of 'foo.sock': Permission denied
>>>
>>> $ cmd /c del foo.sock
>>>
>>> But, native Windows commands are okay, as the third example shows.
>>>
>>> I think the problem may relate to the way native Unix domain sockets are
>>> implemented in Windows and the resulting special handling required.
>>> They are implemented as NTFS reparse points and when opening them
>>> with CreateFile, you need to specify the FILE_FLAG_OPEN_REPARSE_POINT
>>> flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
>>> complications unfortunately, which I'd be happy to discuss further.
>>>
>>> But, to reproduce it, you can compile the attached code snippet
>>> which creates foo.sock in the current directory. Obviously, this
>>> only works on recent versions of Windows 10 and 2019 server.
>>
>> Cygwin doesn't currently support native Windows AF_UNIX sockets, as you've
>> discovered.  See
>>
>>   
>> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$
>>
>> for the current state of AF_UNIX sockets on Cygwin, including the possibility
>> of using native Windows AF_UNIX sockets on systems that support them.
>>
>> If all you want is for Cygwin to recognize such sockets and allow you to apply
>> rm, chmod, etc., I don't think it would be hard to add that capability.  But I
>> doubt if that's all you want.
>>
>> Further discussion of this will have to wait until Corinna is available.
>>
> 
> Thanks for the info. It's mainly about recognition of sockets for
> regular commands. Since these objects can exist on Windows filesystems
> now, potentially created by any kind of Windows application,
> it would be great if Cygwin could handle them, irrespective of whether
> the Cygwin development environment does. Though that sounds like a
> good idea too.

See recent discussion:

https://cygwin.com/pipermail/cygwin/2020-June/245077.html

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 5261 bytes --]

On 9/24/2020 8:01 AM, Michael McMahon wrote:
> 
> 
> On 24/09/2020 12:26, Ken Brown wrote:
>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>> Hi,
>>>
>>> I searched for related issues but haven't found anything.
>>>
>>> I am having some trouble with Windows native Unix domain sockets
>>> (a recent feature in Windows 10 and 2019 server) and Cygwin.
>>> I think I possibly know the cause since I had to investigate a similar
>>> looking issue on another platform built on Windows.
>>>
>>> The problem is that cygwin commands don't seem to recognise native Unix
>>> domain sockets correctly. For example, the socket "foo.sock" should
>>> have the same ownership and similar permissions to other files
>>> in the example below:
>>>
>>> $ ls -lrt
>>> total 2181303
>>>
>>> -rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 test.c
>>> -rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 test.obj
>>> -rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 test.exe
>>> -rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 foo.sock
>>> -rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 check.ot
>>>
>>> A bigger problem is that foo.sock can't be deleted with the cygwin "rm"
>>> command.
>>>
>>> $ rm -f foo.sock
>>> rm: cannot remove 'foo.sock': Permission denied
>>>
>>> $ chmod 777 foo.sock
>>> chmod: changing permissions of 'foo.sock': Permission denied
>>>
>>> $ cmd /c del foo.sock
>>>
>>> But, native Windows commands are okay, as the third example shows.
>>>
>>> I think the problem may relate to the way native Unix domain sockets are
>>> implemented in Windows and the resulting special handling required.
>>> They are implemented as NTFS reparse points and when opening them
>>> with CreateFile, you need to specify the FILE_FLAG_OPEN_REPARSE_POINT
>>> flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
>>> complications unfortunately, which I'd be happy to discuss further.
>>>
>>> But, to reproduce it, you can compile the attached code snippet
>>> which creates foo.sock in the current directory. Obviously, this
>>> only works on recent versions of Windows 10 and 2019 server.
>>
>> Cygwin doesn't currently support native Windows AF_UNIX sockets, as you've 
>> discovered.  See
>>
>> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$ 
>>
>> for the current state of AF_UNIX sockets on Cygwin, including the possibility 
>> of using native Windows AF_UNIX sockets on systems that support them.
>>
>> If all you want is for Cygwin to recognize such sockets and allow you to apply 
>> rm, chmod, etc., I don't think it would be hard to add that capability.  But I 
>> doubt if that's all you want.
>>
>> Further discussion of this will have to wait until Corinna is available.
>>
> 
> Thanks for the info. It's mainly about recognition of sockets for
> regular commands. Since these objects can exist on Windows filesystems
> now, potentially created by any kind of Windows application,
> it would be great if Cygwin could handle them, irrespective of whether
> the Cygwin development environment does. Though that sounds like a
> good idea too.

I think this has a simple fix (attached), but I can't easily test it because 
your test program doesn't compile for me.  First, I got

$ gcc -o native_unix_socket native_unix_socket.c
native_unix_socket.c:5:10: fatal error: WS2tcpip.h: No such file or directory
     5 | #include <WS2tcpip.h>
       |          ^~~~~~~~~~~~
compilation terminated.

I fixed this by making the include file name lower case.  (My system is case 
sensitive, so it matters.)

Next:

$ gcc -o native_unix_socket native_unix_socket.c
native_unix_socket.c:8:10: fatal error: afunix.h: No such file or directory
     8 | #include <afunix.h>
       |          ^~~~~~~~~~
compilation terminated.

There's no file afunix.h in the Cygwin distribution, but I located it online and 
pasted in the contents.  The program now compiles but fails to link:

$ gcc -o native_unix_socket native_unix_socket.c
/usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
/tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): undefined reference to 
`__imp_WSAStartup'
/tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): relocation truncated to fit: 
R_X86_64_PC32 against undefined symbol `__imp_WSAStartup'
/usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
/tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): undefined reference to 
`__imp_WSAGetLastError'
/tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): relocation truncated to fit: 
R_X86_64_PC32 against undefined symbol `__imp_WSAGetLastError'
/usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
/tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): undefined reference to 
`__imp_WSAGetLastError'
/tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): relocation truncated to fit: 
R_X86_64_PC32 against undefined symbol `__imp_WSAGetLastError'
collect2: error: ld returned 1 exit status

This is probably easy to fix too, but I don't feel like tracking it down. 
Please send compilation instructions (that use Cygwin tools).

Ken

[-- Attachment #2: 0001-Cygwin-recognize-native-Windows-AF_UNIX-sockets.patch --]
[-- Type: text/plain, Size: 1620 bytes --]

From 5549952dba88d506b8b28425f8dcb8348446905d Mon Sep 17 00:00:00 2001
From: Ken Brown <kbrown@cornell.edu>
Date: Fri, 25 Sep 2020 08:30:26 -0400
Subject: [PATCH] Cygwin: recognize native Windows AF_UNIX sockets

These are represented by reparse points with reparse tag
IO_REPARSE_TAG_AF_UNIX.  check_reparse_point_target now returns
PATH_REP on files of this type, so that they are treated as known
reparse points (but not as sockets).  This allows tools like 'rm',
'ls', etc. to operate on these files.

Also fix the comment preceding check_reparse_point_target to more
accurately describe the return value.

Addresses: https://cygwin.com/pipermail/cygwin/2020-September/246362.html
---
 winsup/cygwin/path.cc | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/winsup/cygwin/path.cc b/winsup/cygwin/path.cc
index 1d0c38a20..dd33e2847 100644
--- a/winsup/cygwin/path.cc
+++ b/winsup/cygwin/path.cc
@@ -2475,8 +2475,8 @@ check_reparse_point_string (PUNICODE_STRING subst)
 
 /* Return values:
     <0: Negative errno.
-     0: No symlink.
-     1: Symlink.
+     0: Not a recognized reparse point.
+    >0: path_flags for known reparse point.
 */
 int
 check_reparse_point_target (HANDLE h, bool remote, PREPARSE_DATA_BUFFER rp,
@@ -2617,6 +2617,9 @@ check_reparse_point_target (HANDLE h, bool remote, PREPARSE_DATA_BUFFER rp,
 	}
       return -EIO;
     }
+  else if (rp->ReparseTag == IO_REPARSE_TAG_AF_UNIX)
+    /* Native Windows AF_UNIX socket; treat as ordinary file. */
+    return PATH_REP;
 #ifdef __WITH_AF_UNIX
   else if (rp->ReparseTag == IO_REPARSE_TAG_CYGUNIX)
     {
-- 
2.28.0

Re: Problems with native Unix domain sockets on Win 10/2019

[Michael McMahon]

On 25/09/2020 14:19, Ken Brown wrote:
> On 9/24/2020 8:01 AM, Michael McMahon wrote:
>>
>>
>> On 24/09/2020 12:26, Ken Brown wrote:
>>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>>> Hi,
>>>>
>>>> I searched for related issues but haven't found anything.
>>>>
>>>> I am having some trouble with Windows native Unix domain sockets
>>>> (a recent feature in Windows 10 and 2019 server) and Cygwin.
>>>> I think I possibly know the cause since I had to investigate a similar
>>>> looking issue on another platform built on Windows.
>>>>
>>>> The problem is that cygwin commands don't seem to recognise native Unix
>>>> domain sockets correctly. For example, the socket "foo.sock" should
>>>> have the same ownership and similar permissions to other files
>>>> in the example below:
>>>>
>>>> $ ls -lrt
>>>> total 2181303
>>>>
>>>> -rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 test.c
>>>> -rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 
>>>> test.obj
>>>> -rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 
>>>> test.exe
>>>> -rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 
>>>> foo.sock
>>>> -rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 
>>>> check.ot
>>>>
>>>> A bigger problem is that foo.sock can't be deleted with the cygwin "rm"
>>>> command.
>>>>
>>>> $ rm -f foo.sock
>>>> rm: cannot remove 'foo.sock': Permission denied
>>>>
>>>> $ chmod 777 foo.sock
>>>> chmod: changing permissions of 'foo.sock': Permission denied
>>>>
>>>> $ cmd /c del foo.sock
>>>>
>>>> But, native Windows commands are okay, as the third example shows.
>>>>
>>>> I think the problem may relate to the way native Unix domain sockets 
>>>> are
>>>> implemented in Windows and the resulting special handling required.
>>>> They are implemented as NTFS reparse points and when opening them
>>>> with CreateFile, you need to specify the FILE_FLAG_OPEN_REPARSE_POINT
>>>> flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
>>>> complications unfortunately, which I'd be happy to discuss further.
>>>>
>>>> But, to reproduce it, you can compile the attached code snippet
>>>> which creates foo.sock in the current directory. Obviously, this
>>>> only works on recent versions of Windows 10 and 2019 server.
>>>
>>> Cygwin doesn't currently support native Windows AF_UNIX sockets, as 
>>> you've discovered.  See
>>>
>>> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$ 
>>>
>>> for the current state of AF_UNIX sockets on Cygwin, including the 
>>> possibility of using native Windows AF_UNIX sockets on systems that 
>>> support them.
>>>
>>> If all you want is for Cygwin to recognize such sockets and allow you 
>>> to apply rm, chmod, etc., I don't think it would be hard to add that 
>>> capability.  But I doubt if that's all you want.
>>>
>>> Further discussion of this will have to wait until Corinna is available.
>>>
>>
>> Thanks for the info. It's mainly about recognition of sockets for
>> regular commands. Since these objects can exist on Windows filesystems
>> now, potentially created by any kind of Windows application,
>> it would be great if Cygwin could handle them, irrespective of whether
>> the Cygwin development environment does. Though that sounds like a
>> good idea too.
> 
> I think this has a simple fix (attached), but I can't easily test it 
> because your test program doesn't compile for me.  First, I got
> 
> $ gcc -o native_unix_socket native_unix_socket.c
> native_unix_socket.c:5:10: fatal error: WS2tcpip.h: No such file or 
> directory
>      5 | #include <WS2tcpip.h>
>        |          ^~~~~~~~~~~~
> compilation terminated.
> 
> I fixed this by making the include file name lower case.  (My system is 
> case sensitive, so it matters.)
> 
> Next:
> 
> $ gcc -o native_unix_socket native_unix_socket.c
> native_unix_socket.c:8:10: fatal error: afunix.h: No such file or directory
>      8 | #include <afunix.h>
>        |          ^~~~~~~~~~
> compilation terminated.
> 
> There's no file afunix.h in the Cygwin distribution, but I located it 
> online and pasted in the contents.  The program now compiles but fails 
> to link:
> 
> $ gcc -o native_unix_socket native_unix_socket.c
> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): undefined reference 
> to `__imp_WSAStartup'
> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): relocation truncated 
> to fit: R_X86_64_PC32 against undefined symbol `__imp_WSAStartup'
> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): undefined reference 
> to `__imp_WSAGetLastError'
> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): relocation truncated 
> to fit: R_X86_64_PC32 against undefined symbol `__imp_WSAGetLastError'
> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): undefined reference 
> to `__imp_WSAGetLastError'
> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): relocation truncated 
> to fit: R_X86_64_PC32 against undefined symbol `__imp_WSAGetLastError'
> collect2: error: ld returned 1 exit status
> 
> This is probably easy to fix too, but I don't feel like tracking it 
> down. Please send compilation instructions (that use Cygwin tools).
> 
> Ken

Hi

Sorry, I had compiled it in a native Visual C environment.

Assuming you have afunix.h in the current directory.

gcc -o native_unix_socket -I. native_unix_socket.c -lws2_32

should do it.

Michael.

Re: Problems with native Unix domain sockets on Win 10/2019

[Eliot Moss]

Native Windows is not Cygwin ...

If you have a problem in the native Windows environment, that would be with Microsoft, not Cygwin. 
If you're compiling for Cygwin, you need to use the CYgwin tool chain.

Eliot Moss

Re: Problems with native Unix domain sockets on Win 10/2019

[Michael McMahon]

We're just trying to compile a test case and I checked that the
command line I sent does work. So, I think that it okay.
But, the original problem is with the Cygwin command line tools.

Michael.


On 25/09/2020 15:37, Eliot Moss wrote:
> Native Windows is not Cygwin ...
> 
> If you have a problem in the native Windows environment, that would be 
> with Microsoft, not Cygwin. If you're compiling for Cygwin, you need to 
> use the CYgwin tool chain.
> 
> Eliot Moss

Re: Problems with native Unix domain sockets on Win 10/2019

[Eliot Moss]

On 9/25/2020 12:13 PM, Michael McMahon wrote:
> We're just trying to compile a test case and I checked that the
> command line I sent does work. So, I think that it okay.
> But, the original problem is with the Cygwin command line tools.

Well, it's not a problem with the tools (IMO), it's that the particular Linux program you're 
interested in has not been ported to Cygwin.  It's not unusual that Linux programs need at least 
some adjustment to their build process in order to work - sometimes the adjustment is larger than 
others, and some things make such deep assumptions about Linux that it is not practical to port 
them.  That may be the case here.

Someone who knows more about Samba, how it is supported on Windows, and the limitations of Cygwin 
w.r.t. Samba may be able to say more.  If there's a Windows command line tool, you can invoke it 
from Cygwin (as I previously pointed out).

Just trying to help get you to the place where you can run a command to shutdown / restart a Samba 
remote ... Does it have to be exactly the "net" program from Linux?

Best wishes - EM

Re: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

On 9/25/2020 10:29 AM, Michael McMahon wrote:
> 
> 
> On 25/09/2020 14:19, Ken Brown wrote:
>> On 9/24/2020 8:01 AM, Michael McMahon wrote:
>>>
>>>
>>> On 24/09/2020 12:26, Ken Brown wrote:
>>>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>>>> Hi,
>>>>>
>>>>> I searched for related issues but haven't found anything.
>>>>>
>>>>> I am having some trouble with Windows native Unix domain sockets
>>>>> (a recent feature in Windows 10 and 2019 server) and Cygwin.
>>>>> I think I possibly know the cause since I had to investigate a similar
>>>>> looking issue on another platform built on Windows.
>>>>>
>>>>> The problem is that cygwin commands don't seem to recognise native Unix
>>>>> domain sockets correctly. For example, the socket "foo.sock" should
>>>>> have the same ownership and similar permissions to other files
>>>>> in the example below:
>>>>>
>>>>> $ ls -lrt
>>>>> total 2181303
>>>>>
>>>>> -rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 test.c
>>>>> -rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 test.obj
>>>>> -rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 test.exe
>>>>> -rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 foo.sock
>>>>> -rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 check.ot
>>>>>
>>>>> A bigger problem is that foo.sock can't be deleted with the cygwin "rm"
>>>>> command.
>>>>>
>>>>> $ rm -f foo.sock
>>>>> rm: cannot remove 'foo.sock': Permission denied
>>>>>
>>>>> $ chmod 777 foo.sock
>>>>> chmod: changing permissions of 'foo.sock': Permission denied
>>>>>
>>>>> $ cmd /c del foo.sock
>>>>>
>>>>> But, native Windows commands are okay, as the third example shows.
>>>>>
>>>>> I think the problem may relate to the way native Unix domain sockets are
>>>>> implemented in Windows and the resulting special handling required.
>>>>> They are implemented as NTFS reparse points and when opening them
>>>>> with CreateFile, you need to specify the FILE_FLAG_OPEN_REPARSE_POINT
>>>>> flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
>>>>> complications unfortunately, which I'd be happy to discuss further.
>>>>>
>>>>> But, to reproduce it, you can compile the attached code snippet
>>>>> which creates foo.sock in the current directory. Obviously, this
>>>>> only works on recent versions of Windows 10 and 2019 server.
>>>>
>>>> Cygwin doesn't currently support native Windows AF_UNIX sockets, as you've 
>>>> discovered.  See
>>>>
>>>> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$ 
>>>>
>>>> for the current state of AF_UNIX sockets on Cygwin, including the 
>>>> possibility of using native Windows AF_UNIX sockets on systems that support 
>>>> them.
>>>>
>>>> If all you want is for Cygwin to recognize such sockets and allow you to 
>>>> apply rm, chmod, etc., I don't think it would be hard to add that 
>>>> capability.  But I doubt if that's all you want.
>>>>
>>>> Further discussion of this will have to wait until Corinna is available.
>>>>
>>>
>>> Thanks for the info. It's mainly about recognition of sockets for
>>> regular commands. Since these objects can exist on Windows filesystems
>>> now, potentially created by any kind of Windows application,
>>> it would be great if Cygwin could handle them, irrespective of whether
>>> the Cygwin development environment does. Though that sounds like a
>>> good idea too.
>>
>> I think this has a simple fix (attached), but I can't easily test it because 
>> your test program doesn't compile for me.  First, I got
>>
>> $ gcc -o native_unix_socket native_unix_socket.c
>> native_unix_socket.c:5:10: fatal error: WS2tcpip.h: No such file or directory
>>      5 | #include <WS2tcpip.h>
>>        |          ^~~~~~~~~~~~
>> compilation terminated.
>>
>> I fixed this by making the include file name lower case.  (My system is case 
>> sensitive, so it matters.)
>>
>> Next:
>>
>> $ gcc -o native_unix_socket native_unix_socket.c
>> native_unix_socket.c:8:10: fatal error: afunix.h: No such file or directory
>>      8 | #include <afunix.h>
>>        |          ^~~~~~~~~~
>> compilation terminated.
>>
>> There's no file afunix.h in the Cygwin distribution, but I located it online 
>> and pasted in the contents.  The program now compiles but fails to link:
>>
>> $ gcc -o native_unix_socket native_unix_socket.c
>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): undefined reference to 
>> `__imp_WSAStartup'
>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): relocation truncated to 
>> fit: R_X86_64_PC32 against undefined symbol `__imp_WSAStartup'
>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): undefined reference to 
>> `__imp_WSAGetLastError'
>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): relocation truncated to 
>> fit: R_X86_64_PC32 against undefined symbol `__imp_WSAGetLastError'
>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): undefined reference to 
>> `__imp_WSAGetLastError'
>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): relocation truncated to 
>> fit: R_X86_64_PC32 against undefined symbol `__imp_WSAGetLastError'
>> collect2: error: ld returned 1 exit status
>>
>> This is probably easy to fix too, but I don't feel like tracking it down. 
>> Please send compilation instructions (that use Cygwin tools).
>>
>> Ken
> 
> Hi
> 
> Sorry, I had compiled it in a native Visual C environment.
> 
> Assuming you have afunix.h in the current directory.
> 
> gcc -o native_unix_socket -I. native_unix_socket.c -lws2_32
> 
> should do it.

Thanks, that works.  But now I can't reproduce your problem.  Here's what I see, 
using Cygwin 3.1.7 without applying my patch:

$ ./native_unix_socket.exe
getsockname works
fam = 1, len = 11
offsetof clen = 9
strlen = 8
name = foo.sock

$ ls -l foo.sock
-rwxr-xr-x 1 kbrown None 0 2020-09-25 14:39 foo.sock*

$ chmod 644 foo.sock

$ ls -l foo.sock
-rw-r--r-- 1 kbrown None 0 2020-09-25 14:39 foo.sock

$ rm foo.sock

$ ls -l foo.sock
ls: cannot access 'foo.sock': No such file or directory

I'm running 64-bit Cygwin on Windows 10 1909.

Ken

Re: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

On 9/25/2020 2:50 PM, Ken Brown via Cygwin wrote:
> On 9/25/2020 10:29 AM, Michael McMahon wrote:
>>
>>
>> On 25/09/2020 14:19, Ken Brown wrote:
>>> On 9/24/2020 8:01 AM, Michael McMahon wrote:
>>>>
>>>>
>>>> On 24/09/2020 12:26, Ken Brown wrote:
>>>>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I searched for related issues but haven't found anything.
>>>>>>
>>>>>> I am having some trouble with Windows native Unix domain sockets
>>>>>> (a recent feature in Windows 10 and 2019 server) and Cygwin.
>>>>>> I think I possibly know the cause since I had to investigate a similar
>>>>>> looking issue on another platform built on Windows.
>>>>>>
>>>>>> The problem is that cygwin commands don't seem to recognise native Unix
>>>>>> domain sockets correctly. For example, the socket "foo.sock" should
>>>>>> have the same ownership and similar permissions to other files
>>>>>> in the example below:
>>>>>>
>>>>>> $ ls -lrt
>>>>>> total 2181303
>>>>>>
>>>>>> -rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 test.c
>>>>>> -rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 test.obj
>>>>>> -rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 test.exe
>>>>>> -rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 foo.sock
>>>>>> -rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 check.ot
>>>>>>
>>>>>> A bigger problem is that foo.sock can't be deleted with the cygwin "rm"
>>>>>> command.
>>>>>>
>>>>>> $ rm -f foo.sock
>>>>>> rm: cannot remove 'foo.sock': Permission denied
>>>>>>
>>>>>> $ chmod 777 foo.sock
>>>>>> chmod: changing permissions of 'foo.sock': Permission denied
>>>>>>
>>>>>> $ cmd /c del foo.sock
>>>>>>
>>>>>> But, native Windows commands are okay, as the third example shows.
>>>>>>
>>>>>> I think the problem may relate to the way native Unix domain sockets are
>>>>>> implemented in Windows and the resulting special handling required.
>>>>>> They are implemented as NTFS reparse points and when opening them
>>>>>> with CreateFile, you need to specify the FILE_FLAG_OPEN_REPARSE_POINT
>>>>>> flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
>>>>>> complications unfortunately, which I'd be happy to discuss further.
>>>>>>
>>>>>> But, to reproduce it, you can compile the attached code snippet
>>>>>> which creates foo.sock in the current directory. Obviously, this
>>>>>> only works on recent versions of Windows 10 and 2019 server.
>>>>>
>>>>> Cygwin doesn't currently support native Windows AF_UNIX sockets, as you've 
>>>>> discovered.  See
>>>>>
>>>>> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$ 
>>>>>
>>>>> for the current state of AF_UNIX sockets on Cygwin, including the 
>>>>> possibility of using native Windows AF_UNIX sockets on systems that support 
>>>>> them.
>>>>>
>>>>> If all you want is for Cygwin to recognize such sockets and allow you to 
>>>>> apply rm, chmod, etc., I don't think it would be hard to add that 
>>>>> capability.  But I doubt if that's all you want.
>>>>>
>>>>> Further discussion of this will have to wait until Corinna is available.
>>>>>
>>>>
>>>> Thanks for the info. It's mainly about recognition of sockets for
>>>> regular commands. Since these objects can exist on Windows filesystems
>>>> now, potentially created by any kind of Windows application,
>>>> it would be great if Cygwin could handle them, irrespective of whether
>>>> the Cygwin development environment does. Though that sounds like a
>>>> good idea too.
>>>
>>> I think this has a simple fix (attached), but I can't easily test it because 
>>> your test program doesn't compile for me.  First, I got
>>>
>>> $ gcc -o native_unix_socket native_unix_socket.c
>>> native_unix_socket.c:5:10: fatal error: WS2tcpip.h: No such file or directory
>>>      5 | #include <WS2tcpip.h>
>>>        |          ^~~~~~~~~~~~
>>> compilation terminated.
>>>
>>> I fixed this by making the include file name lower case.  (My system is case 
>>> sensitive, so it matters.)
>>>
>>> Next:
>>>
>>> $ gcc -o native_unix_socket native_unix_socket.c
>>> native_unix_socket.c:8:10: fatal error: afunix.h: No such file or directory
>>>      8 | #include <afunix.h>
>>>        |          ^~~~~~~~~~
>>> compilation terminated.
>>>
>>> There's no file afunix.h in the Cygwin distribution, but I located it online 
>>> and pasted in the contents.  The program now compiles but fails to link:
>>>
>>> $ gcc -o native_unix_socket native_unix_socket.c
>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): undefined reference to 
>>> `__imp_WSAStartup'
>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): relocation truncated to 
>>> fit: R_X86_64_PC32 against undefined symbol `__imp_WSAStartup'
>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): undefined reference to 
>>> `__imp_WSAGetLastError'
>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): relocation truncated to 
>>> fit: R_X86_64_PC32 against undefined symbol `__imp_WSAGetLastError'
>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): undefined reference to 
>>> `__imp_WSAGetLastError'
>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): relocation truncated to 
>>> fit: R_X86_64_PC32 against undefined symbol `__imp_WSAGetLastError'
>>> collect2: error: ld returned 1 exit status
>>>
>>> This is probably easy to fix too, but I don't feel like tracking it down. 
>>> Please send compilation instructions (that use Cygwin tools).
>>>
>>> Ken
>>
>> Hi
>>
>> Sorry, I had compiled it in a native Visual C environment.
>>
>> Assuming you have afunix.h in the current directory.
>>
>> gcc -o native_unix_socket -I. native_unix_socket.c -lws2_32
>>
>> should do it.
> 
> Thanks, that works.  But now I can't reproduce your problem.  Here's what I see, 
> using Cygwin 3.1.7 without applying my patch:
> 
> $ ./native_unix_socket.exe
> getsockname works
> fam = 1, len = 11
> offsetof clen = 9
> strlen = 8
> name = foo.sock
> 
> $ ls -l foo.sock
> -rwxr-xr-x 1 kbrown None 0 2020-09-25 14:39 foo.sock*
> 
> $ chmod 644 foo.sock
> 
> $ ls -l foo.sock
> -rw-r--r-- 1 kbrown None 0 2020-09-25 14:39 foo.sock
> 
> $ rm foo.sock
> 
> $ ls -l foo.sock
> ls: cannot access 'foo.sock': No such file or directory
> 
> I'm running 64-bit Cygwin on Windows 10 1909.

I just ran the 'rm' command under gdb to see what's going on, and it seems that 
foo.sock is not being recognized as a reparse point.  So maybe your test 
program, when compiled and run under Cygwin, doesn't actually produce a native 
Windows AF_UNIX socket.  And when I try to run it in a Windows Command Prompt, I get

bind failed 10050
getsockname failed 10022

Can you make your version of the test executable available for me to try?  Or 
tell me some other way to create a native Windows AF_UNIX socket?

Ken

Re: Problems with native Unix domain sockets on Win 10/2019

[Duncan Roe]

On Fri, Sep 25, 2020 at 04:30:45PM -0400, cygwin wrote:
> On 9/25/2020 2:50 PM, Ken Brown via Cygwin wrote:
> > On 9/25/2020 10:29 AM, Michael McMahon wrote:
> > >
> > >
> > > On 25/09/2020 14:19, Ken Brown wrote:
> > > > On 9/24/2020 8:01 AM, Michael McMahon wrote:
> > > > >
> > > > >
> > > > > On 24/09/2020 12:26, Ken Brown wrote:
> > > > > > On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I searched for related issues but haven't found anything.
> > > > > > >
> > > > > > > I am having some trouble with Windows native Unix domain sockets
> > > > > > > (a recent feature in Windows 10 and 2019 server) and Cygwin.
> > > > > > > I think I possibly know the cause since I had to investigate a
> > > > > > > similar looking issue on another platform built on Windows.
[SNIP}
Perhaps the question here is:

 "Is it a bug that Cygwin rm can't remove a file system entry that
  no Cygwin program can create?"

HTH

Cheers ... Duncan.

Re: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

On 9/25/2020 8:31 PM, Duncan Roe wrote:
> On Fri, Sep 25, 2020 at 04:30:45PM -0400, cygwin wrote:
>> On 9/25/2020 2:50 PM, Ken Brown via Cygwin wrote:
>>> On 9/25/2020 10:29 AM, Michael McMahon wrote:
>>>>
>>>>
>>>> On 25/09/2020 14:19, Ken Brown wrote:
>>>>> On 9/24/2020 8:01 AM, Michael McMahon wrote:
>>>>>>
>>>>>>
>>>>>> On 24/09/2020 12:26, Ken Brown wrote:
>>>>>>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I searched for related issues but haven't found anything.
>>>>>>>>
>>>>>>>> I am having some trouble with Windows native Unix domain sockets
>>>>>>>> (a recent feature in Windows 10 and 2019 server) and Cygwin.
>>>>>>>> I think I possibly know the cause since I had to investigate a
>>>>>>>> similar looking issue on another platform built on Windows.
> [SNIP}
> Perhaps the question here is:
> 
>   "Is it a bug that Cygwin rm can't remove a file system entry that
>    no Cygwin program can create?"

No, it's not a bug.  But we can certainly try to add a feature to Cygwin that 
allows it to recognize and work with such files, especially since it appears at 
first glance to be quite easy to do so.

Ken

Re: Problems with native Unix domain sockets on Win 10/2019

[Michael McMahon]

On 25/09/2020 21:30, Ken Brown wrote:
> On 9/25/2020 2:50 PM, Ken Brown via Cygwin wrote:
>> On 9/25/2020 10:29 AM, Michael McMahon wrote:
>>>
>>>
>>> On 25/09/2020 14:19, Ken Brown wrote:
>>>> On 9/24/2020 8:01 AM, Michael McMahon wrote:
>>>>>
>>>>>
>>>>> On 24/09/2020 12:26, Ken Brown wrote:
>>>>>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I searched for related issues but haven't found anything.
>>>>>>>
>>>>>>> I am having some trouble with Windows native Unix domain sockets
>>>>>>> (a recent feature in Windows 10 and 2019 server) and Cygwin.
>>>>>>> I think I possibly know the cause since I had to investigate a 
>>>>>>> similar
>>>>>>> looking issue on another platform built on Windows.
>>>>>>>
>>>>>>> The problem is that cygwin commands don't seem to recognise 
>>>>>>> native Unix
>>>>>>> domain sockets correctly. For example, the socket "foo.sock" should
>>>>>>> have the same ownership and similar permissions to other files
>>>>>>> in the example below:
>>>>>>>
>>>>>>> $ ls -lrt
>>>>>>> total 2181303
>>>>>>>
>>>>>>> -rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 
>>>>>>> test.c
>>>>>>> -rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 
>>>>>>> test.obj
>>>>>>> -rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 
>>>>>>> test.exe
>>>>>>> -rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 
>>>>>>> foo.sock
>>>>>>> -rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 
>>>>>>> check.ot
>>>>>>>
>>>>>>> A bigger problem is that foo.sock can't be deleted with the 
>>>>>>> cygwin "rm"
>>>>>>> command.
>>>>>>>
>>>>>>> $ rm -f foo.sock
>>>>>>> rm: cannot remove 'foo.sock': Permission denied
>>>>>>>
>>>>>>> $ chmod 777 foo.sock
>>>>>>> chmod: changing permissions of 'foo.sock': Permission denied
>>>>>>>
>>>>>>> $ cmd /c del foo.sock
>>>>>>>
>>>>>>> But, native Windows commands are okay, as the third example shows.
>>>>>>>
>>>>>>> I think the problem may relate to the way native Unix domain 
>>>>>>> sockets are
>>>>>>> implemented in Windows and the resulting special handling required.
>>>>>>> They are implemented as NTFS reparse points and when opening them
>>>>>>> with CreateFile, you need to specify the 
>>>>>>> FILE_FLAG_OPEN_REPARSE_POINT
>>>>>>> flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
>>>>>>> complications unfortunately, which I'd be happy to discuss further.
>>>>>>>
>>>>>>> But, to reproduce it, you can compile the attached code snippet
>>>>>>> which creates foo.sock in the current directory. Obviously, this
>>>>>>> only works on recent versions of Windows 10 and 2019 server.
>>>>>>
>>>>>> Cygwin doesn't currently support native Windows AF_UNIX sockets, 
>>>>>> as you've discovered.  See
>>>>>>
>>>>>> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$ 
>>>>>>
>>>>>> for the current state of AF_UNIX sockets on Cygwin, including the 
>>>>>> possibility of using native Windows AF_UNIX sockets on systems 
>>>>>> that support them.
>>>>>>
>>>>>> If all you want is for Cygwin to recognize such sockets and allow 
>>>>>> you to apply rm, chmod, etc., I don't think it would be hard to 
>>>>>> add that capability.  But I doubt if that's all you want.
>>>>>>
>>>>>> Further discussion of this will have to wait until Corinna is 
>>>>>> available.
>>>>>>
>>>>>
>>>>> Thanks for the info. It's mainly about recognition of sockets for
>>>>> regular commands. Since these objects can exist on Windows filesystems
>>>>> now, potentially created by any kind of Windows application,
>>>>> it would be great if Cygwin could handle them, irrespective of whether
>>>>> the Cygwin development environment does. Though that sounds like a
>>>>> good idea too.
>>>>
>>>> I think this has a simple fix (attached), but I can't easily test it 
>>>> because your test program doesn't compile for me.  First, I got
>>>>
>>>> $ gcc -o native_unix_socket native_unix_socket.c
>>>> native_unix_socket.c:5:10: fatal error: WS2tcpip.h: No such file or 
>>>> directory
>>>>      5 | #include <WS2tcpip.h>
>>>>        |          ^~~~~~~~~~~~
>>>> compilation terminated.
>>>>
>>>> I fixed this by making the include file name lower case.  (My system 
>>>> is case sensitive, so it matters.)
>>>>
>>>> Next:
>>>>
>>>> $ gcc -o native_unix_socket native_unix_socket.c
>>>> native_unix_socket.c:8:10: fatal error: afunix.h: No such file or 
>>>> directory
>>>>      8 | #include <afunix.h>
>>>>        |          ^~~~~~~~~~
>>>> compilation terminated.
>>>>
>>>> There's no file afunix.h in the Cygwin distribution, but I located 
>>>> it online and pasted in the contents.  The program now compiles but 
>>>> fails to link:
>>>>
>>>> $ gcc -o native_unix_socket native_unix_socket.c
>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): 
>>>> undefined reference to `__imp_WSAStartup'
>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): relocation 
>>>> truncated to fit: R_X86_64_PC32 against undefined symbol 
>>>> `__imp_WSAStartup'
>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): 
>>>> undefined reference to `__imp_WSAGetLastError'
>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): relocation 
>>>> truncated to fit: R_X86_64_PC32 against undefined symbol 
>>>> `__imp_WSAGetLastError'
>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): 
>>>> undefined reference to `__imp_WSAGetLastError'
>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): relocation 
>>>> truncated to fit: R_X86_64_PC32 against undefined symbol 
>>>> `__imp_WSAGetLastError'
>>>> collect2: error: ld returned 1 exit status
>>>>
>>>> This is probably easy to fix too, but I don't feel like tracking it 
>>>> down. Please send compilation instructions (that use Cygwin tools).
>>>>
>>>> Ken
>>>
>>> Hi
>>>
>>> Sorry, I had compiled it in a native Visual C environment.
>>>
>>> Assuming you have afunix.h in the current directory.
>>>
>>> gcc -o native_unix_socket -I. native_unix_socket.c -lws2_32
>>>
>>> should do it.
>>
>> Thanks, that works.  But now I can't reproduce your problem.  Here's 
>> what I see, using Cygwin 3.1.7 without applying my patch:
>>
>> $ ./native_unix_socket.exe
>> getsockname works
>> fam = 1, len = 11
>> offsetof clen = 9
>> strlen = 8
>> name = foo.sock
>>
>> $ ls -l foo.sock
>> -rwxr-xr-x 1 kbrown None 0 2020-09-25 14:39 foo.sock*
>>
>> $ chmod 644 foo.sock
>>
>> $ ls -l foo.sock
>> -rw-r--r-- 1 kbrown None 0 2020-09-25 14:39 foo.sock
>>
>> $ rm foo.sock
>>
>> $ ls -l foo.sock
>> ls: cannot access 'foo.sock': No such file or directory
>>
>> I'm running 64-bit Cygwin on Windows 10 1909.
> 
> I just ran the 'rm' command under gdb to see what's going on, and it 
> seems that foo.sock is not being recognized as a reparse point.  So 
> maybe your test program, when compiled and run under Cygwin, doesn't 
> actually produce a native Windows AF_UNIX socket.  And when I try to run 
> it in a Windows Command Prompt, I get
> 
> bind failed 10050
> getsockname failed 10022
> 
> Can you make your version of the test executable available for me to 
> try?  Or tell me some other way to create a native Windows AF_UNIX socket?
> 
> Ken

That is all very strange. I have checked both the gcc compiled and MS
compiled executables on my system (2019 server) and they are both
definitely producing native AF_UNIX sockets.

I can email you the two exe files. They are both quite small. But, first
I want to check the patch status of my test system.

Thanks,
Michael.

Re: Problems with native Unix domain sockets on Win 10/2019

[Michael McMahon]

On 26/09/2020 08:30, Michael McMahon via Cygwin wrote:
> 
> 
> On 25/09/2020 21:30, Ken Brown wrote:
>> On 9/25/2020 2:50 PM, Ken Brown via Cygwin wrote:
>>> On 9/25/2020 10:29 AM, Michael McMahon wrote:
>>>>
>>>>
>>>> On 25/09/2020 14:19, Ken Brown wrote:
>>>>> On 9/24/2020 8:01 AM, Michael McMahon wrote:
>>>>>>
>>>>>>
>>>>>> On 24/09/2020 12:26, Ken Brown wrote:
>>>>>>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I searched for related issues but haven't found anything.
>>>>>>>>
>>>>>>>> I am having some trouble with Windows native Unix domain sockets
>>>>>>>> (a recent feature in Windows 10 and 2019 server) and Cygwin.
>>>>>>>> I think I possibly know the cause since I had to investigate a 
>>>>>>>> similar
>>>>>>>> looking issue on another platform built on Windows.
>>>>>>>>
>>>>>>>> The problem is that cygwin commands don't seem to recognise 
>>>>>>>> native Unix
>>>>>>>> domain sockets correctly. For example, the socket "foo.sock" should
>>>>>>>> have the same ownership and similar permissions to other files
>>>>>>>> in the example below:
>>>>>>>>
>>>>>>>> $ ls -lrt
>>>>>>>> total 2181303
>>>>>>>>
>>>>>>>> -rw-r--r--  1 mimcmah      None             1259   Sep 23 10:22 
>>>>>>>> test.c
>>>>>>>> -rwxr-xr-x  1 mimcmah      None             3680   Sep 23 10:22 
>>>>>>>> test.obj
>>>>>>>> -rwxr-xr-x  1 mimcmah      None             121344 Sep 23 10:22 
>>>>>>>> test.exe
>>>>>>>> -rw-r-----  1 Unknown+User Unknown+Group         0 Sep 23 10:23 
>>>>>>>> foo.sock
>>>>>>>> -rw-r--r--  1 mimcmah      None             144356 Sep 23 10:27 
>>>>>>>> check.ot
>>>>>>>>
>>>>>>>> A bigger problem is that foo.sock can't be deleted with the 
>>>>>>>> cygwin "rm"
>>>>>>>> command.
>>>>>>>>
>>>>>>>> $ rm -f foo.sock
>>>>>>>> rm: cannot remove 'foo.sock': Permission denied
>>>>>>>>
>>>>>>>> $ chmod 777 foo.sock
>>>>>>>> chmod: changing permissions of 'foo.sock': Permission denied
>>>>>>>>
>>>>>>>> $ cmd /c del foo.sock
>>>>>>>>
>>>>>>>> But, native Windows commands are okay, as the third example shows.
>>>>>>>>
>>>>>>>> I think the problem may relate to the way native Unix domain 
>>>>>>>> sockets are
>>>>>>>> implemented in Windows and the resulting special handling required.
>>>>>>>> They are implemented as NTFS reparse points and when opening them
>>>>>>>> with CreateFile, you need to specify the 
>>>>>>>> FILE_FLAG_OPEN_REPARSE_POINT
>>>>>>>> flag. Otherwise, you get an ERROR_CANT_ACCESS_FILE. There are other
>>>>>>>> complications unfortunately, which I'd be happy to discuss further.
>>>>>>>>
>>>>>>>> But, to reproduce it, you can compile the attached code snippet
>>>>>>>> which creates foo.sock in the current directory. Obviously, this
>>>>>>>> only works on recent versions of Windows 10 and 2019 server.
>>>>>>>
>>>>>>> Cygwin doesn't currently support native Windows AF_UNIX sockets, 
>>>>>>> as you've discovered.  See
>>>>>>>
>>>>>>> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$ 
>>>>>>>
>>>>>>> for the current state of AF_UNIX sockets on Cygwin, including the 
>>>>>>> possibility of using native Windows AF_UNIX sockets on systems 
>>>>>>> that support them.
>>>>>>>
>>>>>>> If all you want is for Cygwin to recognize such sockets and allow 
>>>>>>> you to apply rm, chmod, etc., I don't think it would be hard to 
>>>>>>> add that capability.  But I doubt if that's all you want.
>>>>>>>
>>>>>>> Further discussion of this will have to wait until Corinna is 
>>>>>>> available.
>>>>>>>
>>>>>>
>>>>>> Thanks for the info. It's mainly about recognition of sockets for
>>>>>> regular commands. Since these objects can exist on Windows 
>>>>>> filesystems
>>>>>> now, potentially created by any kind of Windows application,
>>>>>> it would be great if Cygwin could handle them, irrespective of 
>>>>>> whether
>>>>>> the Cygwin development environment does. Though that sounds like a
>>>>>> good idea too.
>>>>>
>>>>> I think this has a simple fix (attached), but I can't easily test 
>>>>> it because your test program doesn't compile for me.  First, I got
>>>>>
>>>>> $ gcc -o native_unix_socket native_unix_socket.c
>>>>> native_unix_socket.c:5:10: fatal error: WS2tcpip.h: No such file or 
>>>>> directory
>>>>>      5 | #include <WS2tcpip.h>
>>>>>        |          ^~~~~~~~~~~~
>>>>> compilation terminated.
>>>>>
>>>>> I fixed this by making the include file name lower case.  (My 
>>>>> system is case sensitive, so it matters.)
>>>>>
>>>>> Next:
>>>>>
>>>>> $ gcc -o native_unix_socket native_unix_socket.c
>>>>> native_unix_socket.c:8:10: fatal error: afunix.h: No such file or 
>>>>> directory
>>>>>      8 | #include <afunix.h>
>>>>>        |          ^~~~~~~~~~
>>>>> compilation terminated.
>>>>>
>>>>> There's no file afunix.h in the Cygwin distribution, but I located 
>>>>> it online and pasted in the contents.  The program now compiles but 
>>>>> fails to link:
>>>>>
>>>>> $ gcc -o native_unix_socket native_unix_socket.c
>>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): undefined 
>>>>> reference to `__imp_WSAStartup'
>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): relocation 
>>>>> truncated to fit: R_X86_64_PC32 against undefined symbol 
>>>>> `__imp_WSAStartup'
>>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): undefined 
>>>>> reference to `__imp_WSAGetLastError'
>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): relocation 
>>>>> truncated to fit: R_X86_64_PC32 against undefined symbol 
>>>>> `__imp_WSAGetLastError'
>>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): undefined 
>>>>> reference to `__imp_WSAGetLastError'
>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): relocation 
>>>>> truncated to fit: R_X86_64_PC32 against undefined symbol 
>>>>> `__imp_WSAGetLastError'
>>>>> collect2: error: ld returned 1 exit status
>>>>>
>>>>> This is probably easy to fix too, but I don't feel like tracking it 
>>>>> down. Please send compilation instructions (that use Cygwin tools).
>>>>>
>>>>> Ken
>>>>
>>>> Hi
>>>>
>>>> Sorry, I had compiled it in a native Visual C environment.
>>>>
>>>> Assuming you have afunix.h in the current directory.
>>>>
>>>> gcc -o native_unix_socket -I. native_unix_socket.c -lws2_32
>>>>
>>>> should do it.
>>>
>>> Thanks, that works.  But now I can't reproduce your problem.  Here's 
>>> what I see, using Cygwin 3.1.7 without applying my patch:
>>>
>>> $ ./native_unix_socket.exe
>>> getsockname works
>>> fam = 1, len = 11
>>> offsetof clen = 9
>>> strlen = 8
>>> name = foo.sock
>>>
>>> $ ls -l foo.sock
>>> -rwxr-xr-x 1 kbrown None 0 2020-09-25 14:39 foo.sock*
>>>
>>> $ chmod 644 foo.sock
>>>
>>> $ ls -l foo.sock
>>> -rw-r--r-- 1 kbrown None 0 2020-09-25 14:39 foo.sock
>>>
>>> $ rm foo.sock
>>>
>>> $ ls -l foo.sock
>>> ls: cannot access 'foo.sock': No such file or directory
>>>
>>> I'm running 64-bit Cygwin on Windows 10 1909.
>>
>> I just ran the 'rm' command under gdb to see what's going on, and it 
>> seems that foo.sock is not being recognized as a reparse point.  So 
>> maybe your test program, when compiled and run under Cygwin, doesn't 
>> actually produce a native Windows AF_UNIX socket.  And when I try to 
>> run it in a Windows Command Prompt, I get
>>
>> bind failed 10050
>> getsockname failed 10022
>>
>> Can you make your version of the test executable available for me to 
>> try?  Or tell me some other way to create a native Windows AF_UNIX 
>> socket?
>>
>> Ken
> 
> That is all very strange. I have checked both the gcc compiled and MS
> compiled executables on my system (2019 server) and they are both
> definitely producing native AF_UNIX sockets.
> 
> I can email you the two exe files. They are both quite small. But, first
> I want to check the patch status of my test system.
> 

So, it turns out that this issue only happens on some of our test
systems. It does not happen on a personal copy of Windows 10 on my
laptop either.

I also noticed that some native Windows commands don't work properly on
any affected system (eg 'attrib' or 'fsutil'). Though 'fsutil' can be
used to verify that the reparse point is created correctly.

Possibly, this was a Windows bug that has been fixed. It never made
sense that you had to open socket files using the 
FILE_FLAG_OPEN_REPARSE_POINT flag, because you would have to know in
advance that the file is a socket to be able to do this (or else be
prepared to have to open the file twice). But, I don't fully
understand yet, why some systems are affected and others not.
All seem to be patched up to date.

In any case, I think it's clear this isn't a Cygwin issue.
So, apologies for the noise, and thanks for the assistance!

Regards,
Michael.

Re: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

On 9/28/2020 7:03 AM, Michael McMahon wrote:
> 
> 
> On 26/09/2020 08:30, Michael McMahon via Cygwin wrote:
>> 
>> 
>> On 25/09/2020 21:30, Ken Brown wrote:
>>> On 9/25/2020 2:50 PM, Ken Brown via Cygwin wrote:
>>>> On 9/25/2020 10:29 AM, Michael McMahon wrote:
>>>>> 
>>>>> 
>>>>> On 25/09/2020 14:19, Ken Brown wrote:
>>>>>> On 9/24/2020 8:01 AM, Michael McMahon wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> On 24/09/2020 12:26, Ken Brown wrote:
>>>>>>>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I searched for related issues but haven't found anything.
>>>>>>>>> 
>>>>>>>>> I am having some trouble with Windows native Unix domain
>>>>>>>>> sockets (a recent feature in Windows 10 and 2019 server) and
>>>>>>>>> Cygwin. I think I possibly know the cause since I had to
>>>>>>>>> investigate a similar looking issue on another platform built
>>>>>>>>> on Windows.
>>>>>>>>> 
>>>>>>>>> The problem is that cygwin commands don't seem to recognise
>>>>>>>>> native Unix domain sockets correctly. For example, the socket
>>>>>>>>> "foo.sock" should have the same ownership and similar
>>>>>>>>> permissions to other files in the example below:
>>>>>>>>> 
>>>>>>>>> $ ls -lrt total 2181303
>>>>>>>>> 
>>>>>>>>> -rw-r--r--  1 mimcmah      None             1259   Sep 23
>>>>>>>>> 10:22 test.c -rwxr-xr-x  1 mimcmah      None             3680
>>>>>>>>> Sep 23 10:22 test.obj -rwxr-xr-x  1 mimcmah      None
>>>>>>>>> 121344 Sep 23 10:22 test.exe -rw-r-----  1 Unknown+User
>>>>>>>>> Unknown+Group         0 Sep 23 10:23 foo.sock -rw-r--r--  1
>>>>>>>>> mimcmah      None             144356 Sep 23 10:27 check.ot
>>>>>>>>> 
>>>>>>>>> A bigger problem is that foo.sock can't be deleted with the
>>>>>>>>> cygwin "rm" command.
>>>>>>>>> 
>>>>>>>>> $ rm -f foo.sock rm: cannot remove 'foo.sock': Permission
>>>>>>>>> denied
>>>>>>>>> 
>>>>>>>>> $ chmod 777 foo.sock chmod: changing permissions of
>>>>>>>>> 'foo.sock': Permission denied
>>>>>>>>> 
>>>>>>>>> $ cmd /c del foo.sock
>>>>>>>>> 
>>>>>>>>> But, native Windows commands are okay, as the third example
>>>>>>>>> shows.
>>>>>>>>> 
>>>>>>>>> I think the problem may relate to the way native Unix domain
>>>>>>>>> sockets are implemented in Windows and the resulting special
>>>>>>>>> handling required. They are implemented as NTFS reparse
>>>>>>>>> points and when opening them with CreateFile, you need to
>>>>>>>>> specify the FILE_FLAG_OPEN_REPARSE_POINT flag. Otherwise, you
>>>>>>>>> get an ERROR_CANT_ACCESS_FILE. There are other complications
>>>>>>>>> unfortunately, which I'd be happy to discuss further.
>>>>>>>>> 
>>>>>>>>> But, to reproduce it, you can compile the attached code
>>>>>>>>> snippet which creates foo.sock in the current directory.
>>>>>>>>> Obviously, this only works on recent versions of Windows 10
>>>>>>>>> and 2019 server.
>>>>>>>> 
>>>>>>>> Cygwin doesn't currently support native Windows AF_UNIX
>>>>>>>> sockets, as you've discovered.  See
>>>>>>>> 
>>>>>>>> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$
>>>>>>>> 
>>>>>>>> 
>>>>>>>> for the current state of AF_UNIX sockets on Cygwin, including
>>>>>>>> the possibility of using native Windows AF_UNIX sockets on
>>>>>>>> systems that support them.
>>>>>>>> 
>>>>>>>> If all you want is for Cygwin to recognize such sockets and
>>>>>>>> allow you to apply rm, chmod, etc., I don't think it would be
>>>>>>>> hard to add that capability.  But I doubt if that's all you
>>>>>>>> want.
>>>>>>>> 
>>>>>>>> Further discussion of this will have to wait until Corinna is
>>>>>>>> available.
>>>>>>>> 
>>>>>>> 
>>>>>>> Thanks for the info. It's mainly about recognition of sockets
>>>>>>> for regular commands. Since these objects can exist on Windows
>>>>>>> filesystems now, potentially created by any kind of Windows
>>>>>>> application, it would be great if Cygwin could handle them,
>>>>>>> irrespective of whether the Cygwin development environment does.
>>>>>>> Though that sounds like a good idea too.
>>>>>> 
>>>>>> I think this has a simple fix (attached), but I can't easily test
>>>>>> it because your test program doesn't compile for me.  First, I got
>>>>>> 
>>>>>> $ gcc -o native_unix_socket native_unix_socket.c 
>>>>>> native_unix_socket.c:5:10: fatal error: WS2tcpip.h: No such file or
>>>>>> directory 5 | #include <WS2tcpip.h> |          ^~~~~~~~~~~~ 
>>>>>> compilation terminated.
>>>>>> 
>>>>>> I fixed this by making the include file name lower case.  (My
>>>>>> system is case sensitive, so it matters.)
>>>>>> 
>>>>>> Next:
>>>>>> 
>>>>>> $ gcc -o native_unix_socket native_unix_socket.c 
>>>>>> native_unix_socket.c:8:10: fatal error: afunix.h: No such file or
>>>>>> directory 8 | #include <afunix.h> |          ^~~~~~~~~~ compilation
>>>>>> terminated.
>>>>>> 
>>>>>> There's no file afunix.h in the Cygwin distribution, but I located
>>>>>> it online and pasted in the contents.  The program now compiles but
>>>>>> fails to link:
>>>>>> 
>>>>>> $ gcc -o native_unix_socket native_unix_socket.c 
>>>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld:
>>>>>>  /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): undefined
>>>>>> reference to `__imp_WSAStartup' 
>>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): relocation
>>>>>> truncated to fit: R_X86_64_PC32 against undefined symbol
>>>>>> `__imp_WSAStartup' 
>>>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld:
>>>>>>  /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): undefined
>>>>>> reference to `__imp_WSAGetLastError' 
>>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): relocation
>>>>>> truncated to fit: R_X86_64_PC32 against undefined symbol
>>>>>> `__imp_WSAGetLastError' 
>>>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld:
>>>>>>  /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): undefined
>>>>>> reference to `__imp_WSAGetLastError' 
>>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): relocation
>>>>>> truncated to fit: R_X86_64_PC32 against undefined symbol
>>>>>> `__imp_WSAGetLastError' collect2: error: ld returned 1 exit status
>>>>>> 
>>>>>> This is probably easy to fix too, but I don't feel like tracking it
>>>>>> down. Please send compilation instructions (that use Cygwin
>>>>>> tools).
>>>>>> 
>>>>>> Ken
>>>>> 
>>>>> Hi
>>>>> 
>>>>> Sorry, I had compiled it in a native Visual C environment.
>>>>> 
>>>>> Assuming you have afunix.h in the current directory.
>>>>> 
>>>>> gcc -o native_unix_socket -I. native_unix_socket.c -lws2_32
>>>>> 
>>>>> should do it.
>>>> 
>>>> Thanks, that works.  But now I can't reproduce your problem.  Here's
>>>> what I see, using Cygwin 3.1.7 without applying my patch:
>>>> 
>>>> $ ./native_unix_socket.exe getsockname works fam = 1, len = 11 offsetof
>>>> clen = 9 strlen = 8 name = foo.sock
>>>> 
>>>> $ ls -l foo.sock -rwxr-xr-x 1 kbrown None 0 2020-09-25 14:39 foo.sock*
>>>> 
>>>> $ chmod 644 foo.sock
>>>> 
>>>> $ ls -l foo.sock -rw-r--r-- 1 kbrown None 0 2020-09-25 14:39 foo.sock
>>>> 
>>>> $ rm foo.sock
>>>> 
>>>> $ ls -l foo.sock ls: cannot access 'foo.sock': No such file or
>>>> directory
>>>> 
>>>> I'm running 64-bit Cygwin on Windows 10 1909.
>>> 
>>> I just ran the 'rm' command under gdb to see what's going on, and it
>>> seems that foo.sock is not being recognized as a reparse point.  So maybe
>>> your test program, when compiled and run under Cygwin, doesn't actually
>>> produce a native Windows AF_UNIX socket.  And when I try to run it in a
>>> Windows Command Prompt, I get
>>> 
>>> bind failed 10050 getsockname failed 10022
>>> 
>>> Can you make your version of the test executable available for me to try?
>>> Or tell me some other way to create a native Windows AF_UNIX socket?
>>> 
>>> Ken
>> 
>> That is all very strange. I have checked both the gcc compiled and MS 
>> compiled executables on my system (2019 server) and they are both 
>> definitely producing native AF_UNIX sockets.
>> 
>> I can email you the two exe files. They are both quite small. But, first I
>> want to check the patch status of my test system.
>> 
> 
> So, it turns out that this issue only happens on some of our test systems. It
> does not happen on a personal copy of Windows 10 on my laptop either.
> 
> I also noticed that some native Windows commands don't work properly on any
> affected system (eg 'attrib' or 'fsutil'). Though 'fsutil' can be used to
> verify that the reparse point is created correctly.
> 
> Possibly, this was a Windows bug that has been fixed. It never made sense
> that you had to open socket files using the FILE_FLAG_OPEN_REPARSE_POINT 
> flag, because you would have to know in advance that the file is a socket to
> be able to do this (or else be prepared to have to open the file twice). But,
> I don't fully understand yet, why some systems are affected and others not. 
> All seem to be patched up to date.
> 
> In any case, I think it's clear this isn't a Cygwin issue.

It turns out that this is a Cygwin issue after all.  In a private message
Michael has said:

> From what I can see, the only versions that are *not* affected by the problem
> are 1903 and 1909 (which you tested with).  Versions I have tested with since
> then (2004, and 20H2) all show the problem.

I can't immediately test it myself because I'm still on 1909.  But I'll send a 
patch to cygwin-patches that I think should fix it, along with Michael's test 
program.

Ken

Re: Problems with native Unix domain sockets on Win 10/2019

[Michael McMahon]

On 30/01/2021 16:00, Ken Brown wrote:
> On 9/28/2020 7:03 AM, Michael McMahon wrote:
>>
>>
>> On 26/09/2020 08:30, Michael McMahon via Cygwin wrote:
>>>
>>>
>>> On 25/09/2020 21:30, Ken Brown wrote:
>>>> On 9/25/2020 2:50 PM, Ken Brown via Cygwin wrote:
>>>>> On 9/25/2020 10:29 AM, Michael McMahon wrote:
>>>>>>
>>>>>>
>>>>>> On 25/09/2020 14:19, Ken Brown wrote:
>>>>>>> On 9/24/2020 8:01 AM, Michael McMahon wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 24/09/2020 12:26, Ken Brown wrote:
>>>>>>>>> On 9/23/2020 7:25 AM, Michael McMahon via Cygwin wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I searched for related issues but haven't found anything.
>>>>>>>>>>
>>>>>>>>>> I am having some trouble with Windows native Unix domain
>>>>>>>>>> sockets (a recent feature in Windows 10 and 2019 server) and
>>>>>>>>>> Cygwin. I think I possibly know the cause since I had to
>>>>>>>>>> investigate a similar looking issue on another platform built
>>>>>>>>>> on Windows.
>>>>>>>>>>
>>>>>>>>>> The problem is that cygwin commands don't seem to recognise
>>>>>>>>>> native Unix domain sockets correctly. For example, the socket
>>>>>>>>>> "foo.sock" should have the same ownership and similar
>>>>>>>>>> permissions to other files in the example below:
>>>>>>>>>>
>>>>>>>>>> $ ls -lrt total 2181303
>>>>>>>>>>
>>>>>>>>>> -rw-r--r--  1 mimcmah      None 1259   Sep 23
>>>>>>>>>> 10:22 test.c -rwxr-xr-x  1 mimcmah None             3680
>>>>>>>>>> Sep 23 10:22 test.obj -rwxr-xr-x  1 mimcmah None
>>>>>>>>>> 121344 Sep 23 10:22 test.exe -rw-r-----  1 Unknown+User
>>>>>>>>>> Unknown+Group         0 Sep 23 10:23 foo.sock -rw-r--r--  1
>>>>>>>>>> mimcmah      None             144356 Sep 23 10:27 check.ot
>>>>>>>>>>
>>>>>>>>>> A bigger problem is that foo.sock can't be deleted with the
>>>>>>>>>> cygwin "rm" command.
>>>>>>>>>>
>>>>>>>>>> $ rm -f foo.sock rm: cannot remove 'foo.sock': Permission
>>>>>>>>>> denied
>>>>>>>>>>
>>>>>>>>>> $ chmod 777 foo.sock chmod: changing permissions of
>>>>>>>>>> 'foo.sock': Permission denied
>>>>>>>>>>
>>>>>>>>>> $ cmd /c del foo.sock
>>>>>>>>>>
>>>>>>>>>> But, native Windows commands are okay, as the third example
>>>>>>>>>> shows.
>>>>>>>>>>
>>>>>>>>>> I think the problem may relate to the way native Unix domain
>>>>>>>>>> sockets are implemented in Windows and the resulting special
>>>>>>>>>> handling required. They are implemented as NTFS reparse
>>>>>>>>>> points and when opening them with CreateFile, you need to
>>>>>>>>>> specify the FILE_FLAG_OPEN_REPARSE_POINT flag. Otherwise, you
>>>>>>>>>> get an ERROR_CANT_ACCESS_FILE. There are other complications
>>>>>>>>>> unfortunately, which I'd be happy to discuss further.
>>>>>>>>>>
>>>>>>>>>> But, to reproduce it, you can compile the attached code
>>>>>>>>>> snippet which creates foo.sock in the current directory.
>>>>>>>>>> Obviously, this only works on recent versions of Windows 10
>>>>>>>>>> and 2019 server.
>>>>>>>>>
>>>>>>>>> Cygwin doesn't currently support native Windows AF_UNIX
>>>>>>>>> sockets, as you've discovered.  See
>>>>>>>>>
>>>>>>>>> https://urldefense.com/v3/__https://cygwin.com/pipermail/cygwin/2020-June/245088.html__;!!GqivPVa7Brio!P7lIFI4rYAtWh8_DtCbRCxT-M_E4vwQ0qwzQ0p656T73BpJ0jbUkLI_bXdA6mmSL9lJcSQ$ 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> for the current state of AF_UNIX sockets on Cygwin, including
>>>>>>>>> the possibility of using native Windows AF_UNIX sockets on
>>>>>>>>> systems that support them.
>>>>>>>>>
>>>>>>>>> If all you want is for Cygwin to recognize such sockets and
>>>>>>>>> allow you to apply rm, chmod, etc., I don't think it would be
>>>>>>>>> hard to add that capability.  But I doubt if that's all you
>>>>>>>>> want.
>>>>>>>>>
>>>>>>>>> Further discussion of this will have to wait until Corinna is
>>>>>>>>> available.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for the info. It's mainly about recognition of sockets
>>>>>>>> for regular commands. Since these objects can exist on Windows
>>>>>>>> filesystems now, potentially created by any kind of Windows
>>>>>>>> application, it would be great if Cygwin could handle them,
>>>>>>>> irrespective of whether the Cygwin development environment does.
>>>>>>>> Though that sounds like a good idea too.
>>>>>>>
>>>>>>> I think this has a simple fix (attached), but I can't easily test
>>>>>>> it because your test program doesn't compile for me. First, I got
>>>>>>>
>>>>>>> $ gcc -o native_unix_socket native_unix_socket.c 
>>>>>>> native_unix_socket.c:5:10: fatal error: WS2tcpip.h: No such file or
>>>>>>> directory 5 | #include <WS2tcpip.h> | ^~~~~~~~~~~~ compilation 
>>>>>>> terminated.
>>>>>>>
>>>>>>> I fixed this by making the include file name lower case.  (My
>>>>>>> system is case sensitive, so it matters.)
>>>>>>>
>>>>>>> Next:
>>>>>>>
>>>>>>> $ gcc -o native_unix_socket native_unix_socket.c 
>>>>>>> native_unix_socket.c:8:10: fatal error: afunix.h: No such file or
>>>>>>> directory 8 | #include <afunix.h> | ^~~~~~~~~~ compilation
>>>>>>> terminated.
>>>>>>>
>>>>>>> There's no file afunix.h in the Cygwin distribution, but I located
>>>>>>> it online and pasted in the contents.  The program now compiles but
>>>>>>> fails to link:
>>>>>>>
>>>>>>> $ gcc -o native_unix_socket native_unix_socket.c 
>>>>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>>>>>>>
>>>>>>>  /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): undefined
>>>>>>> reference to `__imp_WSAStartup' 
>>>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x3b): relocation
>>>>>>> truncated to fit: R_X86_64_PC32 against undefined symbol
>>>>>>> `__imp_WSAStartup' 
>>>>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>>>>>>>
>>>>>>>  /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): undefined
>>>>>>> reference to `__imp_WSAGetLastError' 
>>>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0xf2): relocation
>>>>>>> truncated to fit: R_X86_64_PC32 against undefined symbol
>>>>>>> `__imp_WSAGetLastError' 
>>>>>>> /usr/lib/gcc/x86_64-pc-cygwin/10/../../../../x86_64-pc-cygwin/bin/ld: 
>>>>>>>
>>>>>>>  /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): undefined
>>>>>>> reference to `__imp_WSAGetLastError' 
>>>>>>> /tmp/cc74urPr.o:native_unix_socket.c:(.text+0x13d): relocation
>>>>>>> truncated to fit: R_X86_64_PC32 against undefined symbol
>>>>>>> `__imp_WSAGetLastError' collect2: error: ld returned 1 exit status
>>>>>>>
>>>>>>> This is probably easy to fix too, but I don't feel like tracking it
>>>>>>> down. Please send compilation instructions (that use Cygwin
>>>>>>> tools).
>>>>>>>
>>>>>>> Ken
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Sorry, I had compiled it in a native Visual C environment.
>>>>>>
>>>>>> Assuming you have afunix.h in the current directory.
>>>>>>
>>>>>> gcc -o native_unix_socket -I. native_unix_socket.c -lws2_32
>>>>>>
>>>>>> should do it.
>>>>>
>>>>> Thanks, that works.  But now I can't reproduce your problem.  Here's
>>>>> what I see, using Cygwin 3.1.7 without applying my patch:
>>>>>
>>>>> $ ./native_unix_socket.exe getsockname works fam = 1, len = 11 
>>>>> offsetof
>>>>> clen = 9 strlen = 8 name = foo.sock
>>>>>
>>>>> $ ls -l foo.sock -rwxr-xr-x 1 kbrown None 0 2020-09-25 14:39 
>>>>> foo.sock*
>>>>>
>>>>> $ chmod 644 foo.sock
>>>>>
>>>>> $ ls -l foo.sock -rw-r--r-- 1 kbrown None 0 2020-09-25 14:39 foo.sock
>>>>>
>>>>> $ rm foo.sock
>>>>>
>>>>> $ ls -l foo.sock ls: cannot access 'foo.sock': No such file or
>>>>> directory
>>>>>
>>>>> I'm running 64-bit Cygwin on Windows 10 1909.
>>>>
>>>> I just ran the 'rm' command under gdb to see what's going on, and it
>>>> seems that foo.sock is not being recognized as a reparse point.  So 
>>>> maybe
>>>> your test program, when compiled and run under Cygwin, doesn't 
>>>> actually
>>>> produce a native Windows AF_UNIX socket.  And when I try to run it 
>>>> in a
>>>> Windows Command Prompt, I get
>>>>
>>>> bind failed 10050 getsockname failed 10022
>>>>
>>>> Can you make your version of the test executable available for me 
>>>> to try?
>>>> Or tell me some other way to create a native Windows AF_UNIX socket?
>>>>
>>>> Ken
>>>
>>> That is all very strange. I have checked both the gcc compiled and 
>>> MS compiled executables on my system (2019 server) and they are both 
>>> definitely producing native AF_UNIX sockets.
>>>
>>> I can email you the two exe files. They are both quite small. But, 
>>> first I
>>> want to check the patch status of my test system.
>>>
>>
>> So, it turns out that this issue only happens on some of our test 
>> systems. It
>> does not happen on a personal copy of Windows 10 on my laptop either.
>>
>> I also noticed that some native Windows commands don't work properly 
>> on any
>> affected system (eg 'attrib' or 'fsutil'). Though 'fsutil' can be 
>> used to
>> verify that the reparse point is created correctly.
>>
>> Possibly, this was a Windows bug that has been fixed. It never made 
>> sense
>> that you had to open socket files using the 
>> FILE_FLAG_OPEN_REPARSE_POINT flag, because you would have to know in 
>> advance that the file is a socket to
>> be able to do this (or else be prepared to have to open the file 
>> twice). But,
>> I don't fully understand yet, why some systems are affected and 
>> others not. All seem to be patched up to date.
>>
>> In any case, I think it's clear this isn't a Cygwin issue.
>
> It turns out that this is a Cygwin issue after all.  In a private message
> Michael has said:
>
>> From what I can see, the only versions that are *not* affected by the 
>> problem
>> are 1903 and 1909 (which you tested with).  Versions I have tested 
>> with since
>> then (2004, and 20H2) all show the problem.
>
> I can't immediately test it myself because I'm still on 1909.  But 
> I'll send a patch to cygwin-patches that I think should fix it, along 
> with Michael's test program.
>
> Ken

Thanks for taking this up again. While I had thought it was a Windows 
bug, and it is arguable, but at least there is a reasonable workaround 
for it. I'd be happy to test an update you have for it.

Michael.

Re: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

On 1/31/2021 6:30 PM, Michael McMahon wrote:
> Thanks for taking this up again. While I had thought it was a Windows bug, and 
> it is arguable, but at least there is a reasonable workaround for it. I'd be 
> happy to test an update you have for it.

This should now be fixed.  As soon as Corinna has a chance to build new 
snapshots, you'll be able to test it by installing a snapshot of cygwin1.dll from

   https://cygwin.com/snapshots/

Ken

Re: Problems with native Unix domain sockets on Win 10/2019

[Corinna Vinschen]

On Feb  1 10:04, Ken Brown via Cygwin wrote:
> On 1/31/2021 6:30 PM, Michael McMahon wrote:
> > Thanks for taking this up again. While I had thought it was a Windows
> > bug, and it is arguable, but at least there is a reasonable workaround
> > for it. I'd be happy to test an update you have for it.
> 
> This should now be fixed.  As soon as Corinna has a chance to build new
> snapshots, you'll be able to test it by installing a snapshot of cygwin1.dll
> from
> 
>   https://cygwin.com/snapshots/

Available now.


Corinna

Re: Problems with native Unix domain sockets on Win 10/2019

[Michael McMahon]

Hi Ken, Corinna

Sorry for the delay in responding. I have just tested the fix and it 
works great. Thank you very much for looking into it.

When do you think it would be available in a form that regular users 
would be able to update/install?

Also, one observation FYI, is that the "ls -l" command on Unix normally 
indicates sockets with an "s" as the first character of the output: eg

srwxr-xr-x  1 Michael None         0 Feb  7 11:19 foo.sock

Currently, what is shown on Cygwin is

-rwxr-xr-x  1 Michael None         0 Feb  7 11:19 foo.sock

indicating a regular file.

All the best,

Michael.

On 01/02/2021 15:10, Corinna Vinschen via Cygwin wrote:
> On Feb  1 10:04, Ken Brown via Cygwin wrote:
>> On 1/31/2021 6:30 PM, Michael McMahon wrote:
>>> Thanks for taking this up again. While I had thought it was a Windows
>>> bug, and it is arguable, but at least there is a reasonable workaround
>>> for it. I'd be happy to test an update you have for it.
>> This should now be fixed.  As soon as Corinna has a chance to build new
>> snapshots, you'll be able to test it by installing a snapshot of cygwin1.dll
>> from
>>
>>    https://urldefense.com/v3/__https://cygwin.com/snapshots/__;!!GqivPVa7Brio!O47H_hw3AhLkrYlQii6UDqGK080PqYYal57b0lCpdErLSUCxWxepfI6N4dDuRAwTH0K4xA$
> Available now.
>
>
> Corinna
> --
> Problem reports:      https://urldefense.com/v3/__https://cygwin.com/problems.html__;!!GqivPVa7Brio!O47H_hw3AhLkrYlQii6UDqGK080PqYYal57b0lCpdErLSUCxWxepfI6N4dDuRAwIKtlIwg$
> FAQ:                  https://urldefense.com/v3/__https://cygwin.com/faq/__;!!GqivPVa7Brio!O47H_hw3AhLkrYlQii6UDqGK080PqYYal57b0lCpdErLSUCxWxepfI6N4dDuRAwMz06BeA$
> Documentation:        https://urldefense.com/v3/__https://cygwin.com/docs.html__;!!GqivPVa7Brio!O47H_hw3AhLkrYlQii6UDqGK080PqYYal57b0lCpdErLSUCxWxepfI6N4dDuRAxsFrAxfg$
> Unsubscribe info:     https://urldefense.com/v3/__https://cygwin.com/ml/*unsubscribe-simple__;Iw!!GqivPVa7Brio!O47H_hw3AhLkrYlQii6UDqGK080PqYYal57b0lCpdErLSUCxWxepfI6N4dDuRAyiCIzNxQ$

Re: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

On 2/7/2021 2:35 PM, Michael McMahon via Cygwin wrote:
> Hi Ken, Corinna
> 
> Sorry for the delay in responding. I have just tested the fix and it works 
> great. Thank you very much for looking into it.
> 
> When do you think it would be available in a form that regular users would be 
> able to update/install?

Corinna decides when to make a new Cygwin release.  My best guess is "pretty soon".

> Also, one observation FYI, is that the "ls -l" command on Unix normally 
> indicates sockets with an "s" as the first character of the output: eg
> 
> srwxr-xr-x  1 Michael None         0 Feb  7 11:19 foo.sock
> 
> Currently, what is shown on Cygwin is
> 
> -rwxr-xr-x  1 Michael None         0 Feb  7 11:19 foo.sock
> 
> indicating a regular file.

Right.  From Cygwin's point of view, it is a regular file.  There might come a 
time when Cygwin's AF_UNIX implementation makes use of native Windows AF_UNIX 
sockets on systems that support them.  But that time is not currently in sight.

Ken

Sv: Problems with native Unix domain sockets on Win 10/2019

[sten.kristian.ivarsson]

Hi all

Does anyone know the status of these fixes ?

I saw an announcement for cygwin-3.2.0-0.1 that seemed to contain some
AF_UNIX-related fixes but I fail to find out where that distribution exists
(if it is supposed to be publicly accessible?), but I tried out the
2021-03-01 snapshot and perhaps they are similar ?

We bumped into some AF_UNIX-related issues when trying the 2021-03-01
-snapshot though. It might be some flaws in our code base but the same code
works in quite a few Linux-distros

We're more than willing to help out with testing this and/or trying to
narrow down any possible cygwin issues, but first we'd be glad if someone
could give us some kind of status report of this (so no one is doing any
unnecessary work)


Best regards,
Kristian

p.s.
   I tried to build the topic/af_unix -branch according to the FAQ, but
there was some issues
d.s.



> > Hi Ken, Corinna
> >
> > Sorry for the delay in responding. I have just tested the fix and it
> > works great. Thank you very much for looking into it.
> >
> > When do you think it would be available in a form that regular users
> > would be able to update/install?
> 
> Corinna decides when to make a new Cygwin release.  My best guess is
> "pretty soon".
> 
> > Also, one observation FYI, is that the "ls -l" command on Unix
> > normally indicates sockets with an "s" as the first character of the
> > output: eg
> >
> > srwxr-xr-x  1 Michael None         0 Feb  7 11:19 foo.sock
> >
> > Currently, what is shown on Cygwin is
> >
> > -rwxr-xr-x  1 Michael None         0 Feb  7 11:19 foo.sock
> >
> > indicating a regular file.
> 
> Right.  From Cygwin's point of view, it is a regular file.  There might
> come a time when Cygwin's AF_UNIX implementation makes use of native
> Windows AF_UNIX sockets on systems that support them.  But that time is
> not currently in sight.
> 
> Ken
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: Sv: Problems with native Unix domain sockets on Win 10/2019

[Michael McMahon]

I think you can download pre-released builds from:

https://cygwin.com/snapshots/

As mentioned already, the fix here is not actually support of AF_UNIX 
sockets natively. My understanding is that they are implemented over 
loopback TCP in cygwin, but the fix allows native AF_UNIX sockets to be 
handled (eg deleted) as regular files through regular cygwin commands.

Michael.


On 16/03/2021 11:06, Kristian Ivarsson via Cygwin wrote:
> Hi all
>
> Does anyone know the status of these fixes ?
>
> I saw an announcement for cygwin-3.2.0-0.1 that seemed to contain some
> AF_UNIX-related fixes but I fail to find out where that distribution exists
> (if it is supposed to be publicly accessible?), but I tried out the
> 2021-03-01 snapshot and perhaps they are similar ?
>
> We bumped into some AF_UNIX-related issues when trying the 2021-03-01
> -snapshot though. It might be some flaws in our code base but the same code
> works in quite a few Linux-distros
>
> We're more than willing to help out with testing this and/or trying to
> narrow down any possible cygwin issues, but first we'd be glad if someone
> could give us some kind of status report of this (so no one is doing any
> unnecessary work)
>
>
> Best regards,
> Kristian
>
> p.s.
>     I tried to build the topic/af_unix -branch according to the FAQ, but
> there was some issues
> d.s.
>
>
>
>>> Hi Ken, Corinna
>>>
>>> Sorry for the delay in responding. I have just tested the fix and it
>>> works great. Thank you very much for looking into it.
>>>
>>> When do you think it would be available in a form that regular users
>>> would be able to update/install?
>> Corinna decides when to make a new Cygwin release.  My best guess is
>> "pretty soon".
>>
>>> Also, one observation FYI, is that the "ls -l" command on Unix
>>> normally indicates sockets with an "s" as the first character of the
>>> output: eg
>>>
>>> srwxr-xr-x  1 Michael None         0 Feb  7 11:19 foo.sock
>>>
>>> Currently, what is shown on Cygwin is
>>>
>>> -rwxr-xr-x  1 Michael None         0 Feb  7 11:19 foo.sock
>>>
>>> indicating a regular file.
>> Right.  From Cygwin's point of view, it is a regular file.  There might
>> come a time when Cygwin's AF_UNIX implementation makes use of native
>> Windows AF_UNIX sockets on systems that support them.  But that time is
>> not currently in sight.
>>
>> Ken
>> --
>> Problem reports:      https://urldefense.com/v3/__https://cygwin.com/problems.html__;!!GqivPVa7Brio!OrhjUwBgvLe4_I38uHpiucq3-_xiQDxazv5s7XuBkOmzXTgGhz9Vb3GrB2RYCJM8jmPtbA$
>> FAQ:                  https://urldefense.com/v3/__https://cygwin.com/faq/__;!!GqivPVa7Brio!OrhjUwBgvLe4_I38uHpiucq3-_xiQDxazv5s7XuBkOmzXTgGhz9Vb3GrB2RYCJMQE63Mlg$
>> Documentation:        https://urldefense.com/v3/__https://cygwin.com/docs.html__;!!GqivPVa7Brio!OrhjUwBgvLe4_I38uHpiucq3-_xiQDxazv5s7XuBkOmzXTgGhz9Vb3GrB2RYCJMRgFcrHA$
>> Unsubscribe info:     https://urldefense.com/v3/__https://cygwin.com/ml/*unsubscribe-simple__;Iw!!GqivPVa7Brio!OrhjUwBgvLe4_I38uHpiucq3-_xiQDxazv5s7XuBkOmzXTgGhz9Vb3GrB2RYCJPOK0x2gw$
> --
> Problem reports:      https://urldefense.com/v3/__https://cygwin.com/problems.html__;!!GqivPVa7Brio!OrhjUwBgvLe4_I38uHpiucq3-_xiQDxazv5s7XuBkOmzXTgGhz9Vb3GrB2RYCJM8jmPtbA$
> FAQ:                  https://urldefense.com/v3/__https://cygwin.com/faq/__;!!GqivPVa7Brio!OrhjUwBgvLe4_I38uHpiucq3-_xiQDxazv5s7XuBkOmzXTgGhz9Vb3GrB2RYCJMQE63Mlg$
> Documentation:        https://urldefense.com/v3/__https://cygwin.com/docs.html__;!!GqivPVa7Brio!OrhjUwBgvLe4_I38uHpiucq3-_xiQDxazv5s7XuBkOmzXTgGhz9Vb3GrB2RYCJMRgFcrHA$
> Unsubscribe info:     https://urldefense.com/v3/__https://cygwin.com/ml/*unsubscribe-simple__;Iw!!GqivPVa7Brio!OrhjUwBgvLe4_I38uHpiucq3-_xiQDxazv5s7XuBkOmzXTgGhz9Vb3GrB2RYCJPOK0x2gw$

Re: Sv: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

On 3/16/2021 9:00 AM, Michael McMahon via Cygwin wrote:
> I think you can download pre-released builds from:
> 
> https://cygwin.com/snapshots/

Right, but see below for cygwin-3.2.0-0.1.

> As mentioned already, the fix here is not actually support of AF_UNIX sockets 
> natively. My understanding is that they are implemented over loopback TCP in 
> cygwin, but the fix allows native AF_UNIX sockets to be handled (eg deleted) as 
> regular files through regular cygwin commands.
> 
> Michael.
> 
> 
> On 16/03/2021 11:06, Kristian Ivarsson via Cygwin wrote:
>> Hi all
>>
>> Does anyone know the status of these fixes ?
>>
>> I saw an announcement for cygwin-3.2.0-0.1 that seemed to contain some
>> AF_UNIX-related fixes but I fail to find out where that distribution exists
>> (if it is supposed to be publicly accessible?), but I tried out the
>> 2021-03-01 snapshot and perhaps they are similar ?

You can install cygwin-3.2.0-0.1 in the usual way, through Cygwin's setup 
program.  Since it's a test release, you'll have to explicitly select that 
release; setup won't just offer it to you.

>> We bumped into some AF_UNIX-related issues when trying the 2021-03-01
>> -snapshot though. It might be some flaws in our code base but the same code
>> works in quite a few Linux-distros
>>
>> We're more than willing to help out with testing this and/or trying to
>> narrow down any possible cygwin issues, but first we'd be glad if someone
>> could give us some kind of status report of this (so no one is doing any
>> unnecessary work)
>>
>>
>> Best regards,
>> Kristian
>>
>> p.s.
>>     I tried to build the topic/af_unix -branch according to the FAQ, but
>> there was some issues

I'm still in the middle of some things on that branch, and I haven't had much 
time to work on it recently.  I hope to get back to it very soon.  In order to 
build it, you need to add -D__WITH_AF_UNIX to CXXFLAGS.  In order to test it, 
you need

#undef AF_UNIX
#define AF_UNIX 31

after including <sys/socket.h>

Ken

Sv: Sv: Problems with native Unix domain sockets on Win 10/2019

[sten.kristian.ivarsson]

[Snip]
> >> Hi all
> >>
> >> Does anyone know the status of these fixes ?
> >>
> >> I saw an announcement for cygwin-3.2.0-0.1 that seemed to contain
> >> some AF_UNIX-related fixes but I fail to find out where that
> >> distribution exists (if it is supposed to be publicly accessible?),
> >> but I tried out the
> >> 2021-03-01 snapshot and perhaps they are similar ?
> 
> You can install cygwin-3.2.0-0.1 in the usual way, through Cygwin's setup
> program.  Since it's a test release, you'll have to explicitly select that
> release; setup won't just offer it to you.

Ahh thanks, it took a while before I noticed that Text-checkbox ;-)

See more below

> >> We bumped into some AF_UNIX-related issues when trying the 2021-03-01
> >> -snapshot though. It might be some flaws in our code base but the
> >> same code works in quite a few Linux-distros
> >>
> >> We're more than willing to help out with testing this and/or trying
> >> to narrow down any possible cygwin issues, but first we'd be glad if
> >> someone could give us some kind of status report of this (so no one
> >> is doing any unnecessary work)
> >>
> >>
> >> Best regards,
> >> Kristian
> >>
> >> p.s.
> >>     I tried to build the topic/af_unix -branch according to the FAQ,
> >> but there was some issues
> 
> I'm still in the middle of some things on that branch, and I haven't had
> much time to work on it recently.  I hope to get back to it very soon.  In
> order to build it, you need to add -D__WITH_AF_UNIX to CXXFLAGS. 

Is this when building newlib-cygwin ? I discovered now that I had wrong
MinGW installed and apparently other things are missing as well

See more below

> In order to test it, you need
> 
> #undef AF_UNIX
> #define AF_UNIX 31
> 
> after including <sys/socket.h>
> 
> Ken


The issues we're experiencing is that messages are lost during heavy load.
We essentially do have one thread writing a bunch of messages with a fixed
buffer chunk size and one thread consuming and sometimes messages disappear
or at least end up in the wrong order. When playing around with the size of
the buffer (we sometimes get -1 but no errno) from write and/or receive

Another observation is that for smaller number of, it is faster than named
pipes but the more messages that are written/consumed the performance
derogates a in the end throughput is getting exponentially slower

Hopefully, this is a defect in our code base, but I will try to narrow it
down with and try to reproduce the behaviour and I will then possibly make
another issue at this mailing list

Keep up the good work

Best regards,
Kristian

Re: Sv: Sv: Problems with native Unix domain sockets on Win 10/2019

[Ken Brown]

On 3/17/2021 8:47 AM, sten.kristian.ivarsson@gmail.com wrote:
> [Snip]
>>>> Hi all
>>>>
>>>> Does anyone know the status of these fixes ?
>>>>
>>>> I saw an announcement for cygwin-3.2.0-0.1 that seemed to contain
>>>> some AF_UNIX-related fixes but I fail to find out where that
>>>> distribution exists (if it is supposed to be publicly accessible?),
>>>> but I tried out the
>>>> 2021-03-01 snapshot and perhaps they are similar ?
>>
>> You can install cygwin-3.2.0-0.1 in the usual way, through Cygwin's setup
>> program.  Since it's a test release, you'll have to explicitly select that
>> release; setup won't just offer it to you.
> 
> Ahh thanks, it took a while before I noticed that Text-checkbox ;-)
> 
> See more below
> 
>>>> We bumped into some AF_UNIX-related issues when trying the 2021-03-01
>>>> -snapshot though. It might be some flaws in our code base but the
>>>> same code works in quite a few Linux-distros
>>>>
>>>> We're more than willing to help out with testing this and/or trying
>>>> to narrow down any possible cygwin issues, but first we'd be glad if
>>>> someone could give us some kind of status report of this (so no one
>>>> is doing any unnecessary work)
>>>>
>>>>
>>>> Best regards,
>>>> Kristian
>>>>
>>>> p.s.
>>>>      I tried to build the topic/af_unix -branch according to the FAQ,
>>>> but there was some issues
>>
>> I'm still in the middle of some things on that branch, and I haven't had
>> much time to work on it recently.  I hope to get back to it very soon.  In
>> order to build it, you need to add -D__WITH_AF_UNIX to CXXFLAGS.
> 
> Is this when building newlib-cygwin ?

It's for building the topic/af_unix branch of the newlib-cygwin repo.  If you 
try to build the master branch with that flag, it should still build, but the 
resulting DLL will be missing a lot of the AF_UNIX functionality.

> I discovered now that I had wrong
> MinGW installed and apparently other things are missing as well
> 
> See more below
> 
>> In order to test it, you need
>>
>> #undef AF_UNIX
>> #define AF_UNIX 31
>>
>> after including <sys/socket.h>
>>
>> Ken
> 
> 
> The issues we're experiencing is that messages are lost during heavy load.
> We essentially do have one thread writing a bunch of messages with a fixed
> buffer chunk size and one thread consuming and sometimes messages disappear
> or at least end up in the wrong order. When playing around with the size of
> the buffer (we sometimes get -1 but no errno) from write and/or receive
> 
> Another observation is that for smaller number of, it is faster than named
> pipes but the more messages that are written/consumed the performance
> derogates a in the end throughput is getting exponentially slower
> 
> Hopefully, this is a defect in our code base, but I will try to narrow it
> down with and try to reproduce the behaviour and I will then possibly make
> another issue at this mailing list

A reproducible test case would be good.

Thanks.

Ken