Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

[Cedric Blancher]

Good morning!

Small comment about 3.5.0 commit "Cygwin: sparse support: enable
automatic sparsifying of files on SSDs"
https://cygwin.com/git/?p=newlib-cygwin.git;a=commit;h=832e91422c4ae9b2dcd0c307779f3dd53ee9c0ac

Can this functionality be extended to VMware and QEMU virtual disks, please?

Also, does Cygwin have a syslog where people can check whether the
feature is on or off?

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

[Corinna Vinschen]

On Dec  1 11:12, Cedric Blancher via Cygwin wrote:
> Good morning!
> 
> Small comment about 3.5.0 commit "Cygwin: sparse support: enable
> automatic sparsifying of files on SSDs"
> https://cygwin.com/git/?p=newlib-cygwin.git;a=commit;h=832e91422c4ae9b2dcd0c307779f3dd53ee9c0ac
> 
> Can this functionality be extended to VMware and QEMU virtual disks, please?

Not without a complete rewrite and lots of slowing down code.

  VMware and QEMU virtual disks are not exposed to the OS as VMware and
  QEMU virtual disks as far as the volume and filesystem information is
  going.  They are just some arbitrary harddisks managed by some
  arbitrary driver.

  What Cygwin gets to see is that you want to access a file on some
  disk.  It opens the file and fetches volume information.  So it finds:

  - The filesystem returns the FILE_SUPPORTS_SPARSE_FILES

    So we can create sparse files.

  - The volume returns the SSINFO_FLAGS_NO_SEEK_PENALTY flag.

    So we're on a drive having no seek penalty.  This is *probably*
    an SSD, but may be some other type of drive.

That's it!

So the decision made is now:

- Does the current file action (fallocate, write after lseek)
  create a hole in the file which is big enough to fit a sparse
  block into it?

- If yes, does the filesystem support FILE_SUPPORTS_SPARSE_FILES?

  - If yes, is it on a volume having no seek penalty?

    - If yes, make the file sparse.

    - If no, is the "sparse" mount option set?

      - If yes, make the file sparse.

> Also, does Cygwin have a syslog where people can check whether the
> feature is on or off?

We have a syslog API for applications, like services, which utilze it.
The Cygwin DLL itself doesn't use it.  That would flood either a syslog
daemon, or the Windows logs with 99.9% unnecessary info.


Corinna

Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

[Cedric Blancher]

On Fri, 1 Dec 2023 at 11:43, Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Dec  1 11:12, Cedric Blancher via Cygwin wrote:
> > Good morning!
> >
> > Small comment about 3.5.0 commit "Cygwin: sparse support: enable
> > automatic sparsifying of files on SSDs"
> > https://cygwin.com/git/?p=newlib-cygwin.git;a=commit;h=832e91422c4ae9b2dcd0c307779f3dd53ee9c0ac
> >
> > Can this functionality be extended to VMware and QEMU virtual disks, please?
>
> Not without a complete rewrite and lots of slowing down code.
>
>   VMware and QEMU virtual disks are not exposed to the OS as VMware and
>   QEMU virtual disks as far as the volume and filesystem information is
>   going.  They are just some arbitrary harddisks managed by some
>   arbitrary driver.

VMware emulates NVME SSD these days, so this should work, yes?

Primary concern is how to detect whether sparse file support is ON in
Cygwin for a specific filesystem? Maybe /sbin/mount without options
should have a sparse/nosparse mount flag to reflect what is used?

>
>   What Cygwin gets to see is that you want to access a file on some
>   disk.  It opens the file and fetches volume information.  So it finds:
>
>   - The filesystem returns the FILE_SUPPORTS_SPARSE_FILES
>
>     So we can create sparse files.
>
>   - The volume returns the SSINFO_FLAGS_NO_SEEK_PENALTY flag.
>
>     So we're on a drive having no seek penalty.  This is *probably*
>     an SSD, but may be some other type of drive.

What about SMB? SMB supports sparse files, and should certainly use
this by DEFAULT, as it makes many HPC apps operate much faster.
BUT: SSINFO_FLAGS_NO_SEEK_PENALTY is only for filesystems with a
drive, and not for network filesystem.

Same applies to the NFSv4.1 filesystem driver, I'm going to pester
Roland Mainz to add FILE_SUPPORTS_SPARSE_FILES support, so same issue
as SMB here.

Ced

Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

[Corinna Vinschen]

On Dec  6 09:49, Cedric Blancher via Cygwin wrote:
> On Fri, 1 Dec 2023 at 11:43, Corinna Vinschen via Cygwin
> VMware emulates NVME SSD these days, so this should work, yes?

It doesn't matter, see below.

> Primary concern is how to detect whether sparse file support is ON in
> Cygwin for a specific filesystem? Maybe /sbin/mount without options
> should have a sparse/nosparse mount flag to reflect what is used?

What concern?  I explained exactly what happens, and it all revolves
around the fact that a filesystem sets the FILE_SUPPORTS_SPARSE_FILES
flag or not.  This is the only interesting fact.  Nothing else
matters.

If you want to see if a filesystem supports that flag, just use the
/usr/lib/csih/getVolInfo tool:

  $ /usr/lib/csih/getVolInfo .
  Device Type        : 7
  Characteristics    : 20020
  Volume Name        : <root>
  Serial Number      : 2292001085
  Max Filenamelength : 255
  Filesystemname     : <NTFS>
  Flags              : 3e72eff
    FILE_CASE_SENSITIVE_SEARCH        : TRUE
    FILE_CASE_PRESERVED_NAMES         : TRUE
    FILE_UNICODE_ON_DISK              : TRUE
    FILE_PERSISTENT_ACLS              : TRUE
    FILE_FILE_COMPRESSION             : TRUE
    FILE_VOLUME_QUOTAS                : TRUE
    FILE_SUPPORTS_SPARSE_FILES        : TRUE
    FILE_SUPPORTS_REPARSE_POINTS      : TRUE
    FILE_SUPPORTS_REMOTE_STORAGE      : FALSE
    FILE_RETURNS_CLEANUP_RESULT_INFO  : TRUE
    FILE_SUPPORTS_POSIX_UNLINK_RENAME : TRUE
    FILE_VOLUME_IS_COMPRESSED         : FALSE
    FILE_SUPPORTS_OBJECT_IDS          : TRUE
    FILE_SUPPORTS_ENCRYPTION          : TRUE
    FILE_NAMED_STREAMS                : TRUE
    FILE_READ_ONLY_VOLUME             : FALSE
    FILE_SEQUENTIAL_WRITE_ONCE        : FALSE
    FILE_SUPPORTS_TRANSACTIONS        : TRUE
    FILE_SUPPORTS_HARD_LINKS          : TRUE
    FILE_SUPPORTS_EXTENDED_ATTRIBUTES : TRUE
    FILE_SUPPORTS_OPEN_BY_FILE_ID     : TRUE
    FILE_SUPPORTS_USN_JOURNAL         : TRUE
    FILE_SUPPORTS_INTEGRITY_STREAMS   : FALSE
    FILE_SUPPORTS_BLOCK_REFCOUNTING   : FALSE
    FILE_SUPPORTS_SPARSE_VDL          : FALSE
    FILE_DAX_VOLUME                   : FALSE
    FILE_SUPPORTS_GHOSTING            : FALSE

> >   What Cygwin gets to see is that you want to access a file on some
> >   disk.  It opens the file and fetches volume information.  So it finds:
> >
> >   - The filesystem returns the FILE_SUPPORTS_SPARSE_FILES
> >
> >     So we can create sparse files.
> >
> >   - The volume returns the SSINFO_FLAGS_NO_SEEK_PENALTY flag.
> >
> >     So we're on a drive having no seek penalty.  This is *probably*
> >     an SSD, but may be some other type of drive.
> 
> What about SMB? SMB supports sparse files, and should certainly use
> this by DEFAULT, as it makes many HPC apps operate much faster.
> BUT: SSINFO_FLAGS_NO_SEEK_PENALTY is only for filesystems with a
> drive, and not for network filesystem.
> 
> Same applies to the NFSv4.1 filesystem driver, I'm going to pester
> Roland Mainz to add FILE_SUPPORTS_SPARSE_FILES support, so same issue
> as SMB here.

Again, the filesystem doesn't matter.  It either sets the
FILE_SUPPORTS_SPARSE_FILES flag or not, as simple as that.

If it does, you can create sparse files with chattr +S, or you can rely
on the lseek/ftruncate/posix_fallocate automatism, or you stamp a hole
into the file with fallocate(FALLOC_FL_PUNCH_HOLE).

The *only* difference is if you have to use the "sparse" mount option or
not.

Basically, with 3.4, you always have to set the "sparse" mount option,
with 3.5, on local SSDs you don't.  I don't see a problem here.


Corinna

Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

[Corinna Vinschen]

On Dec  6 10:38, Corinna Vinschen via Cygwin wrote:
> Again, the filesystem doesn't matter.  It either sets the
> FILE_SUPPORTS_SPARSE_FILES flag or not, as simple as that.
> 
> If it does, you can create sparse files with chattr +S, or you can rely
> on the lseek/ftruncate/posix_fallocate automatism, or you stamp a hole
> into the file with fallocate(FALLOC_FL_PUNCH_HOLE).
> 
> The *only* difference is if you have to use the "sparse" mount option or
> not.
> 
> Basically, with 3.4, you always have to set the "sparse" mount option,
> with 3.5, on local SSDs you don't.  I don't see a problem here.

Oh, and to be very clear here:

This *only* affects new files or files which are not yet sparse.

As soon as a file is sparse, it stays sparse.

Subsequently, the way sparse blocks are created or converted to
allocated blocks during write, lseek, ftruncate/posix_fallocate depends
solely on how the OS and the filesystem driver perform in this matter.

Cygwin is out of the picture then.  After all, it's still just a user
space DLL.


Corinna

Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

[Shaddy Baddah]

Hi Corinna,

On 6/12/2023 8:38 pm, Corinna Vinschen via Cygwin wrote:

> If you want to see if a filesystem supports that flag, just use the
> /usr/lib/csih/getVolInfo tool:
>
>    $ /usr/lib/csih/getVolInfo .
>    Device Type        : 7
>    Characteristics    : 20020
>    Volume Name        : <root>
>    Serial Number      : 2292001085
>    Max Filenamelength : 255
>    Filesystemname     : <NTFS>
>    Flags              : 3e72eff
>      FILE_CASE_SENSITIVE_SEARCH        : TRUE
...
>      FILE_SUPPORTS_GHOSTING            : FALSE

Is is worth it/possible to extend this getVolInfo to also include the
SSINFO_FLAGS_NO_SEEK_PENALTY flag in the output?-- Regards, Shaddy

Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

[Corinna Vinschen]

On Dec  7 17:42, Shaddy Baddah via Cygwin wrote:
> Hi Corinna,
> 
> On 6/12/2023 8:38 pm, Corinna Vinschen via Cygwin wrote:
> 
> > If you want to see if a filesystem supports that flag, just use the
> > /usr/lib/csih/getVolInfo tool:
> > 
> >    $ /usr/lib/csih/getVolInfo .
> >    Device Type        : 7
> >    Characteristics    : 20020
> >    Volume Name        : <root>
> >    Serial Number      : 2292001085
> >    Max Filenamelength : 255
> >    Filesystemname     : <NTFS>
> >    Flags              : 3e72eff
> >      FILE_CASE_SENSITIVE_SEARCH        : TRUE
> ...
> >      FILE_SUPPORTS_GHOSTING            : FALSE
> 
> Is is worth it/possible to extend this getVolInfo to also include the
> SSINFO_FLAGS_NO_SEEK_PENALTY flag in the output?-- Regards, Shaddy

Good idea.  I created a patch and a new version of the csih package
(0.9.14) will be available shortly.


Thanks,
Corinna

Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

[Cedric Blancher]

On Wed, 6 Dec 2023 at 10:39, Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Dec  6 09:49, Cedric Blancher via Cygwin wrote:
> > On Fri, 1 Dec 2023 at 11:43, Corinna Vinschen via Cygwin
> > VMware emulates NVME SSD these days, so this should work, yes?
>
> It doesn't matter, see below.
>
> > Primary concern is how to detect whether sparse file support is ON in
> > Cygwin for a specific filesystem? Maybe /sbin/mount without options
> > should have a sparse/nosparse mount flag to reflect what is used?
>
> What concern?  I explained exactly what happens, and it all revolves
> around the fact that a filesystem sets the FILE_SUPPORTS_SPARSE_FILES
> flag or not.  This is the only interesting fact.  Nothing else
> matters.
>
> If you want to see if a filesystem supports that flag, just use the
> /usr/lib/csih/getVolInfo tool:
>
>   $ /usr/lib/csih/getVolInfo .
>   Device Type        : 7
>   Characteristics    : 20020
>   Volume Name        : <root>
>   Serial Number      : 2292001085
>   Max Filenamelength : 255
>   Filesystemname     : <NTFS>
>   Flags              : 3e72eff
>     FILE_CASE_SENSITIVE_SEARCH        : TRUE
>     FILE_CASE_PRESERVED_NAMES         : TRUE
>     FILE_UNICODE_ON_DISK              : TRUE
>     FILE_PERSISTENT_ACLS              : TRUE
>     FILE_FILE_COMPRESSION             : TRUE
>     FILE_VOLUME_QUOTAS                : TRUE
>     FILE_SUPPORTS_SPARSE_FILES        : TRUE
>     FILE_SUPPORTS_REPARSE_POINTS      : TRUE
>     FILE_SUPPORTS_REMOTE_STORAGE      : FALSE
>     FILE_RETURNS_CLEANUP_RESULT_INFO  : TRUE
>     FILE_SUPPORTS_POSIX_UNLINK_RENAME : TRUE
>     FILE_VOLUME_IS_COMPRESSED         : FALSE
>     FILE_SUPPORTS_OBJECT_IDS          : TRUE
>     FILE_SUPPORTS_ENCRYPTION          : TRUE
>     FILE_NAMED_STREAMS                : TRUE
>     FILE_READ_ONLY_VOLUME             : FALSE
>     FILE_SEQUENTIAL_WRITE_ONCE        : FALSE
>     FILE_SUPPORTS_TRANSACTIONS        : TRUE
>     FILE_SUPPORTS_HARD_LINKS          : TRUE
>     FILE_SUPPORTS_EXTENDED_ATTRIBUTES : TRUE
>     FILE_SUPPORTS_OPEN_BY_FILE_ID     : TRUE
>     FILE_SUPPORTS_USN_JOURNAL         : TRUE
>     FILE_SUPPORTS_INTEGRITY_STREAMS   : FALSE
>     FILE_SUPPORTS_BLOCK_REFCOUNTING   : FALSE
>     FILE_SUPPORTS_SPARSE_VDL          : FALSE
>     FILE_DAX_VOLUME                   : FALSE
>     FILE_SUPPORTS_GHOSTING            : FALSE
>
> > >   What Cygwin gets to see is that you want to access a file on some
> > >   disk.  It opens the file and fetches volume information.  So it finds:
> > >
> > >   - The filesystem returns the FILE_SUPPORTS_SPARSE_FILES
> > >
> > >     So we can create sparse files.
> > >
> > >   - The volume returns the SSINFO_FLAGS_NO_SEEK_PENALTY flag.
> > >
> > >     So we're on a drive having no seek penalty.  This is *probably*
> > >     an SSD, but may be some other type of drive.
> >
> > What about SMB? SMB supports sparse files, and should certainly use
> > this by DEFAULT, as it makes many HPC apps operate much faster.
> > BUT: SSINFO_FLAGS_NO_SEEK_PENALTY is only for filesystems with a
> > drive, and not for network filesystem.
> >
> > Same applies to the NFSv4.1 filesystem driver, I'm going to pester
> > Roland Mainz to add FILE_SUPPORTS_SPARSE_FILES support, so same issue
> > as SMB here.
>
> Again, the filesystem doesn't matter.  It either sets the
> FILE_SUPPORTS_SPARSE_FILES flag or not, as simple as that.
>
> If it does, you can create sparse files with chattr +S, or you can rely
> on the lseek/ftruncate/posix_fallocate automatism, or you stamp a hole
> into the file with fallocate(FALLOC_FL_PUNCH_HOLE).
>
> The *only* difference is if you have to use the "sparse" mount option or
> not.
>
> Basically, with 3.4, you always have to set the "sparse" mount option,
> with 3.5, on local SSDs you don't.  I don't see a problem here.

How can I remount the existing C: mount with the "sparse" option in
Cygwin 3.4? I cannot get it working, or I am too stupid.

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?

[Corinna Vinschen]

On Dec 18 12:10, Cedric Blancher via Cygwin wrote:
> On Wed, 6 Dec 2023 at 10:39, Corinna Vinschen via Cygwin
> <cygwin@cygwin.com> wrote:
> > Basically, with 3.4, you always have to set the "sparse" mount option,
> > with 3.5, on local SSDs you don't.  I don't see a problem here.
> 
> How can I remount the existing C: mount with the "sparse" option in
> Cygwin 3.4? I cannot get it working, or I am too stupid.

How about you tell us what you did, so we can check?  I know that it
works, but apparently something crucial is missing.

One thing, for instance, is, that changes in fstab and fstab.d only
start working after stopping all Cygwin processes of the user or after
calling `mount -a'.

Another is, of course, how your fstab line actually looked like.


Corinna