Re: FUSE for Cygwin - was: Re: Fork and Windows Heap

Re: FUSE for Cygwin - was: Re: Fork and Windows Heap

[Bill Zissimopoulos]

Hi, Corinna:

On Jun 17 07:25, Bill Zissimopoulos wrote:
> > Windows hard links are rather un-POSIX like and rarely used on Windows.
> > After considering the required changes on the FSD for a feature that is
> > so rarely used I opted against supporting them.
> 
> I disagree here.  Windows hardlinks work fine and pretty much as on
> POSIX with the exception of DOS mode bits.  Those are not per file but
> per file entry as far as my experiecen goes.  One of the reasons we try
> to ignore them as much as possible.

I no longer remember all the details now, because it was a few months ago
that I looked into this and determined that hard links are "un-POSIX like".
As far as I recall there was the FileAttributes issue (which I now think
may have been my incorrect understanding of the documentation), but there
was also the issue of FindFirstFileName/FindNextFileName, which is not
something that POSIX supports out of the box (AFAIK).

Regarding the FileAttributes issue. The Windows documentation seems to
suggest that they are stored with the file and not the directory entry.
I have not experimented though to confirm whether this is true.

From MSDN "Hard Links and Junctions" [1]:
<<
Note that the attributes on the file are reflected in every hard link
to that file, and changes to that file's attributes propagate to all
the hard links. For example if you reset the READONLY attribute on a
hard link to delete that particular hard link, and there are multiple
hard links to the actual file, then you will need to reset the READONLY
bit on the file from one of the remaining hard links to bring the file
and all remaining hard links back to the READONLY state.
>>

And then of course there is this ambiguous bit:
<<
However, the directory entry size and attribute information is updated
only for the link through which the change was made.
>>

I think that attribute information in this case means "NTFS attribute"
and not FILE_ATTRIBUTE_*. NTFS attributes are simply the method that
NTFS uses to store information about files.

Taken together this information suggests that NTFS hard links are more
"POSIX like" than I thought. In fact they may be "POSIX like with extra
capabilities".

The bigger issue is that the FSD now maintains an internal table of open
files that is indexed by file name. It actually started as a table of open
files indexed by IndexNumber (inode number Windows equivalent), but I
eventually had to change it for a number of issues (notably Rename
support). I am not saying that it would not be possible to change this
part of WinFsp, I just believe that it is a non-trivial change at this
moment.

> > > You write that you are mapping
> > >  - characters not supported by Win32 but by POSIX via the Unicode
> > >    private use page
> > >  - Security apspects (SID vs. uid/gid, ACL)
> > > between POSIX and Windows and that you do it like Cygwin/SFU/SFM is
> > > doing it.
>
> Uhm... I don't understand.  Cygwin is using the private unicode range as
> well to map characters invalid in a Windows filename (e.g. colon 0x3a is
> mapped to 0xf03a).  This is how such filenames are given to the OS
>functions.
> 
> This means, WinFsp would get these filenames from a Cygwin process
> already in the transposed form, with invalid FS chars transposed into the
> private area.
> 
> If WinFsp would support that, it would be transparent to applications
> with very minor effort.

I think I may have inadequately explained what WinFsp does with this
mapping. It basically does the opposite of what Cygwin does on any given
code path.

Let's examine the lifetime of a call to creat(). Suppose a Cygwin process
does creat("/cygdrive/z/foo*bar"). In the following OP is the "originating
process", CW is the "Cygwin runtime", NT is NTOS, WD is the "WinFsp FSD",
WL is the "WinFsp DLL", FL is the "FUSE layer", and FS is the "user mode
FUSE file system".

OP: creat("/cygdrive/z/foo*bar")
CW: NtCreateFile(L"<DEVICE>\\foo\xf02abar")     <--- Cygwin translation
NT: IRP_MJ_CREATE L"\\foo\xf02abar"
WD: FspFsctlTransactCreateKind L"\\foo\xf02abar"
WL: FSP_FILE_SYSTEM_INTERFACE::Create L"\\foo\xf02abar"
FL: fuse_operations::create "/foo*bar"          <--- WinFsp/FUSE
translation
FS: somehow satisfies fuse_operations::create
[snip return path]

The opposite happens when the file system communicates a file name back
to the originating process, as in readdir().

OP: readdir("/cygdrive/z/")
[snip call path]
FS: fuse_operations::readdir response: "foo*bar"
FL: FSP_FILE_SYSTEM_INTERFACE::ReadDirectory response: L"foo\xf02abar"
WL: FspFsctlTransactQueryDirectoryKind response: L"foo\xf02abar"
WD: IRP_MN_QUERY_DIRECTORY response: L"foo\xf02abar"
NT: NtQueryDirectoryFile response: L"foo\xf02abar"
CW: readdir response: "foo*bar"
OP: receives "foo*bar"

Now also recall that native Windows is my primary target environment and
that the originating process will likely not be a Cygwin process. Suppose
that SSHFS is used to mount a directory that has a "foo*bar" file. How to
present that to a Win32 process? I hope that this makes the need for the
WinFsp mapping more apparent.

Bill

[1] 
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).
aspx

Re: FUSE for Cygwin - was: Re: Fork and Windows Heap

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3874 bytes --]

Hi Bill,

On Jun 17 19:48, Bill Zissimopoulos wrote:
> Hi, Corinna:
> 
> On Jun 17 07:25, Bill Zissimopoulos wrote:
> > > Windows hard links are rather un-POSIX like and rarely used on Windows.
> > > After considering the required changes on the FSD for a feature that is
> > > so rarely used I opted against supporting them.
> > 
> > I disagree here.  Windows hardlinks work fine and pretty much as on
> > POSIX with the exception of DOS mode bits.  Those are not per file but
> > per file entry as far as my experiecen goes.  One of the reasons we try
> > to ignore them as much as possible.
> 
> I no longer remember all the details now, because it was a few months ago
> that I looked into this and determined that hard links are "un-POSIX like".
> As far as I recall there was the FileAttributes issue (which I now think
> may have been my incorrect understanding of the documentation), but there
> was also the issue of FindFirstFileName/FindNextFileName, which is not
> something that POSIX supports out of the box (AFAIK).
> 
> Regarding the FileAttributes issue. The Windows documentation seems to
> suggest that they are stored with the file and not the directory entry.

Yes, you're right.  Apparently I didn't really think when writing.  The
*real* issue with DOS attributes is that they are per file :) That has
implications if you have multiple hardlinks to symlinks of the "Windows
shortcut" type, but that shouldn't affect your WinFSD.

> The bigger issue is that the FSD now maintains an internal table of open
> files that is indexed by file name. It actually started as a table of open
> files indexed by IndexNumber (inode number Windows equivalent),

Yeah, Cygwin is presenting them as i-node numbers.

> but I
> eventually had to change it for a number of issues (notably Rename
> support).

For rename support you can use the index number as well, usually,
since you can open a file by index number.  At least on NTFS.

> I am not saying that it would not be possible to change this
> part of WinFsp, I just believe that it is a non-trivial change at this
> moment.

Ok.

> [...]
> Let's examine the lifetime of a call to creat(). Suppose a Cygwin process
> does creat("/cygdrive/z/foo*bar"). In the following OP is the "originating
> process", CW is the "Cygwin runtime", NT is NTOS, WD is the "WinFsp FSD",
> WL is the "WinFsp DLL", FL is the "FUSE layer", and FS is the "user mode
> FUSE file system".
> 
> OP: creat("/cygdrive/z/foo*bar")
> CW: NtCreateFile(L"<DEVICE>\\foo\xf02abar")     <--- Cygwin translation
> NT: IRP_MJ_CREATE L"\\foo\xf02abar"
> WD: FspFsctlTransactCreateKind L"\\foo\xf02abar"
> WL: FSP_FILE_SYSTEM_INTERFACE::Create L"\\foo\xf02abar"
> FL: fuse_operations::create "/foo*bar"          <--- WinFsp/FUSE
> translation
> FS: somehow satisfies fuse_operations::create
> [snip return path]
> 
> The opposite happens when the file system communicates a file name back
> to the originating process, as in readdir().
> 
> OP: readdir("/cygdrive/z/")
> [snip call path]
> FS: fuse_operations::readdir response: "foo*bar"
> FL: FSP_FILE_SYSTEM_INTERFACE::ReadDirectory response: L"foo\xf02abar"
> WL: FspFsctlTransactQueryDirectoryKind response: L"foo\xf02abar"
> WD: IRP_MN_QUERY_DIRECTORY response: L"foo\xf02abar"
> NT: NtQueryDirectoryFile response: L"foo\xf02abar"
> CW: readdir response: "foo*bar"
> OP: receives "foo*bar"

If I'm not missing anything crucial, that's pretty much exactly what we
need.  The Cygwin process reads and writes the file as if the '*' is
a valid character and the entire process of mapping is completely
transparent to both sides.  Looks perfect to me.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: FUSE for Cygwin - was: Re: Fork and Windows Heap

[Bill Zissimopoulos]

On 6/18/16, 1:02 AM, "Corinna Vinschen" <cygwin-owner@cygwin.com on behalf
of corinna-cygwin@cygwin.com> wrote:


>>but I eventually had to change it for a number of issues (notably Rename
>> support).
>
>For rename support you can use the index number as well, usually,
>since you can open a file by index number.  At least on NTFS.

Unfortunately it is not as simple as that. A proper Rename implementation
needs to conform to rules that effectively require the FSD to maintain
filename/path information about open files.

From "FILE_RENAME_INFORMATION structure” [1]:
<<
Special rules for renaming open files:

* A file cannot be renamed if it has any open handles, unless it is only
open because of a batch opportunistic lock (oplock) and the batch oplock
can be broken immediately.

* A file cannot be renamed if a file with the same name exists and has
open handles (except in the batch-oplock case described earlier).

* A directory cannot be renamed if it or any of its subdirectories
contains a file that has open handles (except in the batch-oplock case
described earlier).

>>

In particular the third bullet point mandates that the FSD keeps
information not only about files that are open, but also of their
hierarchical relationships. The easiest way to do this on Windows is to
maintain a mapping of file names to open files.

>>I am not saying that it would not be possible to change this
>> part of WinFsp, I just believe that it is a non-trivial change at this
>> moment.
>
>Ok.

I have thought more about this.

As mentioned, originally the FSD maintained a mapping of IndexNumber to
“FileNode” somewhat similar to UNIX. This is because it is important for
an FSD to be able to identify when the same file is being reopened; for
example, to properly implement file deletion (a file gets deleted only
when its last handle is closed) or sharing or caching.

When I had to implement Rename, bullet point (3) above complicated
matters. So I ended up maintaining two tables for a while, one for
IndexNumbers and one for file names. Then I ended up scrapping the
IndexNumber table when I decided that I would not implement hard links
(for a variety of reasons).

Perhaps what I should have done instead is to maintain IndexNumber
relationships (parent/child) and do my rename tests based on that. That’s
what a VFS system would do, although I am not sure that it is the cleanest
solution within a Windows FSD.

Compounding this the user mode file system may not even send me back real
IndexNumber’s, so then I would have to fake them (e.g. file name hash).

In any case I am going to shelve this for a while and come back to it at a
later time as there are lots of higher priority (for me) things to do on
WinFsp.

--

In other news I made a new release yesterday and I am happy to say that
this release supports Cygwin. I am able to compile SSHFS with a minimal
patch and it runs correctly under Cygwin.

The SSHFS-Win repo is here [2].

I may actually go the extra mile and create a libfuse.dll so that things
like FUSEPY can work out of the box.

Bill

[1] 
https://msdn.microsoft.com/en-us/library/windows/hardware/ff540344(v=vs.85)
.aspx
[2] https://bitbucket.org/billziss/sshfs-win

Re: FUSE for Cygwin - was: Re: Fork and Windows Heap

[Jeffrey Altman]

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]

On 6/18/2016 4:03 PM, Bill Zissimopoulos wrote:
> * A directory cannot be renamed if it or any of its subdirectories
> contains a file that has open handles (except in the batch-oplock case
> described earlier).
> 
> 
> In particular the third bullet point mandates that the FSD keeps
> information not only about files that are open, but also of their
> hierarchical relationships. The easiest way to do this on Windows is to
> maintain a mapping of file names to open files.

This is not how my file system redirector enforces the rule.  The file
control block (representing the handle) for an open file maintains a
reference to the directory object through which it was opened.  As long
as the directory object has outstanding references the redirector fails
the rename request.

File path maps to specific files in fact do not work because the file
can be hard linked into more than one directory.  Only the directory
that was used to create a file handle is restricted from renaming.

Jeffrey Altman



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4349 bytes --]

Re: FUSE for Cygwin - was: Re: Fork and Windows Heap

[Bill Zissimopoulos]

Hello, Jeffrey:

On 6/18/16, 1:19 PM, "Jeffrey Altman" <cygwin-owner@cygwin.com on behalf
of jaltman@secure-endpoints.com> wrote:


>On 6/18/2016 4:03 PM, Bill Zissimopoulos wrote:
>> * A directory cannot be renamed if it or any of its subdirectories
>> contains a file that has open handles (except in the batch-oplock case
>> described earlier).
>> 
>> 
>> In particular the third bullet point mandates that the FSD keeps
>> information not only about files that are open, but also of their
>> hierarchical relationships. The easiest way to do this on Windows is to
>> maintain a mapping of file names to open files.
>
>This is not how my file system redirector enforces the rule.  The file
>control block (representing the handle) for an open file maintains a
>reference to the directory object through which it was opened.  As long
>as the directory object has outstanding references the redirector fails
>the rename request.

Thank you for your very useful pointer. I have a question. In your file
system redirector if the file "\comp1\comp2\name.ext” gets opened, do you
also open (internally) the directories "\comp1" and "\comp1\comp2”? It
sounds like you do.

I understand that this is what UNIX VFS does, but it is my understanding
that Windows does not require this. WinFsp avoids it so that it will not
send extraneous requests to the user mode file system. WinFsp only knows
about files that have been explicitly opened using IRP_MJ_CREATE.

But perhaps I misunderstand what you are suggesting. Apologies for being
thick, but can you please elaborate?

>File path maps to specific files in fact do not work because the file
>can be hard linked into more than one directory.  Only the directory
>that was used to create a file handle is restricted from renaming.

I agree. But my FSD does not currently support hard links. If/when it does
I will have to revisit this design.

Many thanks for your insight. BTW, if your redirector is public/opensource
I would love to take a look.

Bill

Re: FUSE for Cygwin - was: Re: Fork and Windows Heap

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3653 bytes --]

Hi Bill,

On Jun 17 07:25, Bill Zissimopoulos wrote:
> > How are you dealing with limitations of the Windows file system as seen
> > from a POSIX perspective? You say you can't support hard links.
> 
> Windows hard links are rather un-POSIX like and rarely used on Windows.
> After considering the required changes on the FSD for a feature that is
> so rarely used I opted against supporting them.

I disagree here.  Windows hardlinks work fine and pretty much as on
POSIX with the exception of DOS mode bits.  Those are not per file but
per file entry as far as my experiecen goes.  One of the reasons we try
to ignore them as much as possible.

Hardlinks are pretty essential I think.  The API from user space is
rather simple as well (NtSetInformationFile ([...], FileLinkInformation)
and supported in Cygwin for the link(2) call practically since the last
century :)

> > How will a Cygwin program see symlinks exported by the FUSE file system?
> 
> WinFsp will present symlinks as reparse points. Cygwin already knows how
> to deal with NTFS reparse points and it will present them as POSIX
> symlinks.
> I may even relax the whole “you have to be admin to create a symlink”
> nonsense.

Full ACK!

> Please note that WinFsp does (not) yet support reparse points. There is no
> big obstacle in the way, just a question of time.
> 
> > And pipes?
> 
> Pipes in Windows are not stored in the file system, rather they are
> implemented by a special Windows file system called NPFS. OTOH POSIX
> FIFO's are special files. I am unsure at this time how Cygwin represents
> FIFO's.
> 
> [Quick experiment:
> 
> $ mkfifo foo; cmd /c dir 'foo*' | grep foo; rm foo
> 06/16/2016  11:02 PM               130 foo.lnk
> ]
> 
> Ok, so they are shortcuts. Naturally they are supported.

...just the FIFO implementation itself... let's just say, it could be
improved.

> > You write that you are mapping
> >  - characters not supported by Win32 but by POSIX via the Unicode
> >    private use page
> >  - Security apspects (SID vs. uid/gid, ACL)
> > between POSIX and Windows and that you do it like Cygwin/SFU/SFM is
> > doing it.
> >
> > But if that's done by your code, a Cygwin process may see slightly
> > different mappings through WinFsp and through Cygwin. Won't that be a
> > potential for Bugs (misbehaving apps) or even for security issues?
> 
> I agree that the potential is there. But I did not want WinFsp to depend
> on Cygwin so reimplementation was the best option for me. Additionally
> Cygwin does not expose this functionality to third party apps AFAIK.

Uhm... I don't understand.  Cygwin is using the private unicode range as
well to map characters invalid in a Windows filename (e.g. colon 0x3a is
mapped to 0xf03a).  This is how such filenames are given to the OS functions.

This means, WinFsp would get these filenames from a Cygwin process
already in the transposed form, with invalid FS chars transposed into the
private area.

If WinFsp would support that, it would be transparent to applications
with very minor effort.

> Try this from a cmd.exe prompt:
>     type \\?\GLOBALROOT\Device\HarddiskVolume2\Cygwin64\Cygwin.bat

Or thhis from Cygwin:

  /proc/sys/Device/HarddiskVolume2/Cygwin64/Cygwin.bat

:)

> For this reason alone, I expect that most user mode file systems
> will be packaged to work as network file systems.

That would make a lot of sense, I think.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

FUSE for Cygwin - was: Re: Fork and Windows Heap

[Bill Zissimopoulos]

[I apologize if my responses to the list appear to break the mailing list's
threading model. I am not actually subscribed to the list and I respond to
individual messages using my mail app.]

Hello, Herbert:

Herbert Stocker wrote:
> > On 16.06.2016 08:37, Bill Zissimopoulos wrote:
> >
> > I have a Windows solution developed using Visual Studio
> > and the Windows Driver Kit that I am building a FUSE
> > compatibility layer for.
> 
> this is a GREAT piece of work.
> Although i did not dive into kernel development, i do understand that
> writing a file system driver for Windows is surely lots of painful work,
> with its undocumented and grown API.
>
> You should write a book documenting all the knowledge you gathered
> through your research, if the only one available is from '97 as you say.
>
> Really, great work.
> Also that you add the FUSE API.

Thank you for your very kind words.

> So, how is your architecture of the FUSE part of WinFsp?
> Are you porting libfuse to Cygwin, so that a Cygwin process can link to
> it and instead of receiving requests via /dev/fuse fro the Linux kernel,
> your libfuse will receive IRPs through your user mode DLL from your
> Windows kernel driver FSD?

WinFsp consists of a kernel-mode file system driver (FSD) and a user mode
dynamic link library (DLL). The FSD registers two devices in the NTOS
(Windows
kernel) namespace with names: \Device\WinFsp.Disk and \Device\WinFsp.Net.
The
DLL interacts with these devices using CreateFile and DeviceIoControl.

The FSD receives IRP's by the kernel and if it cannot handle them itself,
it
posts them to an I/O queue. At a later time the user mode file system
process
receives the IRP's from the queue using DeviceIoControl. The user mode file
system handles the IRP's and then send the responses back to the FSD, which
then completes the IRP's.

All this is of course nicely wrapped by an API so that the user-mode file
system developer never sees IRP's. Instead IRP's are translated to callback
calls with familiar names and semantics such as Create, Open, Read, Write,
etc.

The FUSE layer is a thin layer on top of this api. For example, the WinFsp
Create callback maps to FUSE mkdir or create, the Open callback maps to
FUSE
opendir or open, etc.

I am not porting libfuse to Windows. This is a complete reimplementation.

For the full details on the WinFsp design please see this document:
    http://www.secfs.net/winfsp/develop/design/

For details on the FUSE port please see the WinFsp blog:
    http://www.secfs.net/winfsp/blog/

> Are you planning to have FUSE file systems be ported to native Windows
> apps, i.e. don't link with cygwin1.dll, or will they run in a Cygwin
> environment?

WinFsp provides three (3) different modes of integration:

(1) Its own native API, which allows a user mode file system to do almost
anything NTFS can do (with a few exceptions). It also allows for the
Read, Write and ReadDirectory callbacks to be asynchronous (return
STATUS_PENDING). I recommend this API for file systems that want maximum
performance and integration with Windows.

(2) A FUSE (high-level) API for Win32 (i.e. non-Cygwin) applications. This
is useful if one has a FUSE file system that they can easily port to
Windows or they do not want any Cygwin dependencies.

(3) A FUSE (high-level) API for Cygwin. As you correctly note many FUSE
file systems are too POSIX specific (as they were born on Linux/OSX) and
they only make sense in a Cygwin environment.

I expect that most FUSE file systems will probably go for option (3)
initially. Then they may want to consider going to (2) if they can
easily port their core file system code to Windows. Then they may even
consider (1) if they have needs that the FUSE API cannot support (e.g.
full support for Windows ACL's).

> If porting to native app, this would be easier for the user to install
> them, but:
>  - FUSE has bindings to many languages like Perl, PHP, etc. Will these
>    work then?

I expect that parts of the bindings will have to be rewritten. WinFsp does
not pretend to be FUSE, it only has a FUSE API for someone that writes C
or C++ and has a compiler that understands #include <fuse.h>

> How are you dealing with limitations of the Windows file system as seen
> from a POSIX perspective? You say you can't support hard links.

Windows hard links are rather un-POSIX like and rarely used on Windows.
After considering the required changes on the FSD for a feature that is
so rarely used I opted against supporting them.

I expect that POSIX file systems with hard link support need to take some
care when exposed through WinFsp. But I admit that I have not thought too
hard about this problem and I do not have any solid recommendations or
solutions at this point.

> How will a Cygwin program see symlinks exported by the FUSE file system?

WinFsp will present symlinks as reparse points. Cygwin already knows how
to deal with NTFS reparse points and it will present them as POSIX
symlinks.
I may even relax the whole “you have to be admin to create a symlink”
nonsense.

Please note that WinFsp does (not) yet support reparse points. There is no
big obstacle in the way, just a question of time.

> And pipes?

Pipes in Windows are not stored in the file system, rather they are
implemented by a special Windows file system called NPFS. OTOH POSIX
FIFO's are special files. I am unsure at this time how Cygwin represents
FIFO's.

[Quick experiment:

$ mkfifo foo; cmd /c dir 'foo*' | grep foo; rm foo
06/16/2016  11:02 PM               130 foo.lnk
]

Ok, so they are shortcuts. Naturally they are supported.

> You write that you are mapping
>  - characters not supported by Win32 but by POSIX via the Unicode
>    private use page
>  - Security apspects (SID vs. uid/gid, ACL)
> between POSIX and Windows and that you do it like Cygwin/SFU/SFM is
> doing it.
>
> But if that's done by your code, a Cygwin process may see slightly
> different mappings through WinFsp and through Cygwin. Won't that be a
> potential for Bugs (misbehaving apps) or even for security issues?

I agree that the potential is there. But I did not want WinFsp to depend
on Cygwin so reimplementation was the best option for me. Additionally
Cygwin does not expose this functionality to third party apps AFAIK.

> i'm looking forward to trying it out this weekend and to see it go
> from alpha state to stable.

Thank you for your support :)

> (And i'm curious about the differences of disk based and network based
> file systems. Is it just the handling as seen from the user or also
> the semantics.)

The main difference is how "mounting" happens. Also the Windows explorer
provides different behavior for a disk and network file system. The actual
user mode file system sees no difference whether it is mounted as a disk
or a network file system.

Ok, a quick primer on "mounting" on Windows.

A file system is implemented as a special device that is unnamed (i.e.
does not have a name in the NTOS namespace). Nevertheless this file system
device must somehow receive file system related requests. This is done by
associating (mounting) this device with either a disk device (e.g.
a hard drive) or with the MUP (Multiple UNC Provider).

When the file system device gets associated with a disk device it becomes
a disk file system. When it gets associated with the MUP it becomes a
network file system.

Usually we assign a drive letter X: to a new file system and that is how
we access files on it. But with MUP there is another option: using the
\\Server\Share notation. This is the first important difference between
a disk and network file system.

[There is actually another option for accessing files on a file system.

Try this from a cmd.exe prompt:
    type \\?\GLOBALROOT\Device\HarddiskVolume2\Cygwin64\Cygwin.bat

This assumes that your C: drive points to \Device\HarddiskVolume2 and
that you have Cygwin installed under C:\Cygwin64.]

The second important difference between disk and network file systems
is Windows Explorer support. Windows explorer allows you to map
network drives, list them, etc.

I am using this with WinFsp to spawn file system instances straight
from explorer. For example mapping the network drive Z: to
\\sshfs\billziss@myhost will spawn a new instance of SSHFS that
connects to billziss@myhost. [Please note that while WinFsp already
implements this support, my SSHFS port is not there yet.]

For this reason alone, I expect that most user mode file systems
will be packaged to work as network file systems.

Bill

FUSE for Cygwin - was: Re: Fork and Windows Heap

[Herbert Stocker]

On 16.06.2016 08:37, Bill Zissimopoulos wrote:
>
> I am not porting anything from UNIX. I have a Windows solution developed
> using Visual Studio and the Windows Driver Kit that I am building a FUSE
> compatibility layer for. My DLL is not a Cygwin DLL. It is a native
> Windows DLL that also has a FUSE compatibility layer. I am taking pains to
> make that FUSE compatibility layer available to both Win32 and Cygwin apps.
>

Hey Bill,

this is a GREAT piece of work.
Although i did not dive into kernel development, i do understand that
writing a file system driver for Windows is surely lots of painful work,
with its undocumented and grown API.

You should write a book documenting all the knowledge you gathered
through your research, if the only one available is from '97 as you say.

Really, great work.
Also that you add the FUSE API.

Having FUSE available for Cygwin and maybe for Windows apps is something
i longed for, too. And i thought about adding it to Cygwin. Seariously.
But i'm too constrained to manage that amount of work. :-(

What is FUSE:
   Architecturally a FUSE file system is a (usermode) process that regis-
   ters with the kernel for a certain path in the file system and then
   responds to requests like open file, read/write data, rename file,
   stat file.

So, how is your architecture of the FUSE part of WinFsp?
Are you porting libfuse to Cygwin, so that a Cygwin process can link to
it and instead of receiving requests via /dev/fuse fro the Linux kernel,
your libfuse will receive IRPs through your user mode DLL from your
Windows kernel driver FSD?

Are you planning to have FUSE file systems be ported to native Windows
apps, i.e. don't link with cygwin1.dll, or will they run in a Cygwin
environment?
If porting to native app, this would be easier for the user to install
them, but:
  - FUSE has bindings to many languages like Perl, PHP, etc. Will these
    work then?
  - Aren't the FUSE programs relying on Posix features too much?
    Can they be easily ported with e.g. MinGW?

How are you dealing with limitations of the Windows file system as seen
from a POSIX perspective? You say you can't support hard links.
How will a Cygwin program see symlinks exported by the FUSE file system?
And pipes?

You write that you are mapping
  - characters not supported by Win32 but by POSIX via the Unicode
    private use page
  - Security apspects (SID vs. uid/gid, ACL)
between POSIX and Windows and that you do it like Cygwin/SFU/SFM is
doing it.

But if that's done by your code, a Cygwin process may see slightly
different mappings through WinFsp and through Cygwin. Won't that be a
potential for Bugs (misbehaving apps) or even for security issues?

i'm looking forward to trying it out this weekend and to see it go
from alpha state to stable.

(And i'm curious about the differences of disk based and network based
file systems. Is it just the handling as seen from the user or also
the semantics.)


Herbert



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple