SEEK_HOLE Support in Cygwin?

SEEK_HOLE Support in Cygwin?

[Cedric Blancher]

Good evening!

Are there plans for a SEEK_HOLE support to Cygwin, e.g. on top of the
FSCTL_QUERY_ALLOCATED_RANGES api (if appropriate)?

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: SEEK_HOLE Support in Cygwin?

[Corinna Vinschen]

Wrong mailing list, use cygwin AT cygwin DOT com for user questions.

On Aug  6 19:14, Cedric Blancher wrote:
> Good evening!
> 
> Are there plans for a SEEK_HOLE support to Cygwin, e.g. on top of the
> FSCTL_QUERY_ALLOCATED_RANGES api (if appropriate)?

No plans, but patches are welcome.  This looks like a fun task for a
rainy weekend.


Corinna

Re: SEEK_HOLE Support in Cygwin?

[Corinna Vinschen]

On Aug  7 10:33, Corinna Vinschen wrote:
> Wrong mailing list, use cygwin AT cygwin DOT com for user questions.
> 
> On Aug  6 19:14, Cedric Blancher wrote:
> > Good evening!
> > 
> > Are there plans for a SEEK_HOLE support to Cygwin, e.g. on top of the
> > FSCTL_QUERY_ALLOCATED_RANGES api (if appropriate)?
> 
> No plans, but patches are welcome.  This looks like a fun task for a
> rainy weekend.

Today was a really rainy day and I had to wait for a test system to
come up, so I hacked on SEEK_DATA/SEEK_HOLE support.

Check out the next Cygwin test release 3.5.0-0.384.g9939aa7d0945.

Note that this defaults to the default behaviour described in the
Linux man page, see https://man7.org/linux/man-pages/man2/lseek.2.html

I.e., "In the simplest implementation, a filesystem can support the
operations by making SEEK_HOLE always return the offset of the end of
the file, and making SEEK_DATA always return offset"

The real deal obviously requires a filesystem supporting sparse files,
*and* it requires that you access the file via a mount point with the
"sparse" option set!  The mount option "sparse" is not set by default.


Corinna

Re: SEEK_HOLE Support in Cygwin?

[Cedric Blancher]

On Wed, 9 Aug 2023 at 17:30, Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:
>
> On Aug  7 10:33, Corinna Vinschen wrote:
> > Wrong mailing list, use cygwin AT cygwin DOT com for user questions.
> >
> > On Aug  6 19:14, Cedric Blancher wrote:
> > > Good evening!
> > >
> > > Are there plans for a SEEK_HOLE support to Cygwin, e.g. on top of the
> > > FSCTL_QUERY_ALLOCATED_RANGES api (if appropriate)?
> >
> > No plans, but patches are welcome.  This looks like a fun task for a
> > rainy weekend.
>
> Today was a really rainy day and I had to wait for a test system to
> come up, so I hacked on SEEK_DATA/SEEK_HOLE support.
>
> Check out the next Cygwin test release 3.5.0-0.384.g9939aa7d0945.

Thank you!
Or should I thank the weather gods? ;)

How can I download this version?

>
> Note that this defaults to the default behaviour described in the
> Linux man page, see https://man7.org/linux/man-pages/man2/lseek.2.html
>
> I.e., "In the simplest implementation, a filesystem can support the
> operations by making SEEK_HOLE always return the offset of the end of
> the file, and making SEEK_DATA always return offset"

So this supports multiple holes in one file, right?

>
> The real deal obviously requires a filesystem supporting sparse files,
> *and* it requires that you access the file via a mount point with the
> "sparse" option set!  The mount option "sparse" is not set by default.

Why is "sparse" not the default mount option?

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: SEEK_HOLE Support in Cygwin?

[Corinna Vinschen]

On Aug 10 04:43, Cedric Blancher wrote:
> On Wed, 9 Aug 2023 at 17:30, Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:
> >
> > On Aug  7 10:33, Corinna Vinschen wrote:
> > > Wrong mailing list, use cygwin AT cygwin DOT com for user questions.
> > >
> > > On Aug  6 19:14, Cedric Blancher wrote:
> > > > Good evening!
> > > >
> > > > Are there plans for a SEEK_HOLE support to Cygwin, e.g. on top of the
> > > > FSCTL_QUERY_ALLOCATED_RANGES api (if appropriate)?
> > >
> > > No plans, but patches are welcome.  This looks like a fun task for a
> > > rainy weekend.
> >
> > Today was a really rainy day and I had to wait for a test system to
> > come up, so I hacked on SEEK_DATA/SEEK_HOLE support.
> >
> > Check out the next Cygwin test release 3.5.0-0.384.g9939aa7d0945.
> 
> Thank you!
> Or should I thank the weather gods? ;)
> 
> How can I download this version?

Cygwin setup, as usual.  Just install the above version of the packages
cygwin and cygwin-devel.

> > Note that this defaults to the default behaviour described in the
> > Linux man page, see https://man7.org/linux/man-pages/man2/lseek.2.html
> >
> > I.e., "In the simplest implementation, a filesystem can support the
> > operations by making SEEK_HOLE always return the offset of the end of
> > the file, and making SEEK_DATA always return offset"
> 
> So this supports multiple holes in one file, right?

Curious question.  Cygwin supports sparse files for ages.  Only the
lseek options SEEK_HOLE and SEEK_DATA are new.

But, yes, it supports what Windows supports.  But note that Cygwin
*only* switches files to sparse, if you seek by more than 128K beyond
current EOF and then wrote something, or if you call truncate/ftruncate,
raising the filesize by more than 128K.

In contrast to Linux, one has to sparsify a file explicitely on WIndows,
and that's Cygwin's strategy.

> > The real deal obviously requires a filesystem supporting sparse files,
> > *and* it requires that you access the file via a mount point with the
> > "sparse" option set!  The mount option "sparse" is not set by default.
> 
> Why is "sparse" not the default mount option?

It was, at one point long ago.  People complained that sparse file
access is slower than standard file access, so the "sparse" mount option
was added.


Corinna

Re: SEEK_HOLE Support in Cygwin?

[Cedric Blancher]

On Thu, 10 Aug 2023 at 10:28, Corinna Vinschen
<corinna-cygwin@cygwin.com> wrote:
>
> On Aug 10 04:43, Cedric Blancher wrote:
> > On Wed, 9 Aug 2023 at 17:30, Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:
> > >
> > > On Aug  7 10:33, Corinna Vinschen wrote:
> > > > Wrong mailing list, use cygwin AT cygwin DOT com for user questions.
> > > >
> > > > On Aug  6 19:14, Cedric Blancher wrote:
> > > > > Good evening!
> > > > >
> > > > > Are there plans for a SEEK_HOLE support to Cygwin, e.g. on top of the
> > > > > FSCTL_QUERY_ALLOCATED_RANGES api (if appropriate)?
> > > >
> > > > No plans, but patches are welcome.  This looks like a fun task for a
> > > > rainy weekend.
> > >
> > > Today was a really rainy day and I had to wait for a test system to
> > > come up, so I hacked on SEEK_DATA/SEEK_HOLE support.
> > >
> > > Check out the next Cygwin test release 3.5.0-0.384.g9939aa7d0945.
> >
> > Thank you!
> > Or should I thank the weather gods? ;)
> >
> > How can I download this version?
>
> Cygwin setup, as usual.  Just install the above version of the packages
> cygwin and cygwin-devel.

Corinna, Thank you!

>
> > > Note that this defaults to the default behaviour described in the
> > > Linux man page, see https://man7.org/linux/man-pages/man2/lseek.2.html
> > >
> > > I.e., "In the simplest implementation, a filesystem can support the
> > > operations by making SEEK_HOLE always return the offset of the end of
> > > the file, and making SEEK_DATA always return offset"
> >
> > So this supports multiple holes in one file, right?
>
> Curious question.  Cygwin supports sparse files for ages.  Only the
> lseek options SEEK_HOLE and SEEK_DATA are new.

Is SEEK_HOLE support enabled for MSFT NFSv3 client?

>
> But, yes, it supports what Windows supports.  But note that Cygwin
> *only* switches files to sparse, if you seek by more than 128K beyond
> current EOF and then wrote something, or if you call truncate/ftruncate,
> raising the filesize by more than 128K.

Where does the value of 128K come from? Is this a Windows limit,
kernel tunable, or something from Cygwin?

>
> In contrast to Linux, one has to sparsify a file explicitely on WIndows,
> and that's Cygwin's strategy.

Maybe add a O_SPARSEFILE to open/openat()?

>
> > > The real deal obviously requires a filesystem supporting sparse files,
> > > *and* it requires that you access the file via a mount point with the
> > > "sparse" option set!  The mount option "sparse" is not set by default.
> >
> > Why is "sparse" not the default mount option?
>
> It was, at one point long ago.  People complained that sparse file
> access is slower than standard file access, so the "sparse" mount option
> was added.

Do you have any refs in the mailing list?

Ced

Re: SEEK_HOLE Support in Cygwin?

[Corinna Vinschen]

On Aug 21 06:33, Cedric Blancher wrote:
> On Thu, 10 Aug 2023 at 10:28, Corinna Vinschen
> <corinna-cygwin@cygwin.com> wrote:
> > On Aug 10 04:43, Cedric Blancher wrote:
> > > So this supports multiple holes in one file, right?
> >
> > Curious question.  Cygwin supports sparse files for ages.  Only the
> > lseek options SEEK_HOLE and SEEK_DATA are new.
> 
> Is SEEK_HOLE support enabled for MSFT NFSv3 client?

NFSv3 doesn't return the FILE_SUPPORTS_SPARSE_FILES capability and
doesn't support Windows sparse file functions.  Sparsification is
(correctly) the responsibility of the remote file system.

> > But, yes, it supports what Windows supports.  But note that Cygwin
> > *only* switches files to sparse, if you seek by more than 128K beyond
> > current EOF and then wrote something, or if you call truncate/ftruncate,
> > raising the filesize by more than 128K.
> 
> Where does the value of 128K come from? Is this a Windows limit,
> kernel tunable, or something from Cygwin?

Something from Cygwin.

> > In contrast to Linux, one has to sparsify a file explicitely on WIndows,
> > and that's Cygwin's strategy.
> 
> Maybe add a O_SPARSEFILE to open/openat()?

We won't add any non-standard or non-Linux open flags.  You can use
the chattr/lsattr tools.

> > > Why is "sparse" not the default mount option?
> >
> > It was, at one point long ago.  People complained that sparse file
> > access is slower than standard file access, so the "sparse" mount option
> > was added.
> 
> Do you have any refs in the mailing list?

Not without searching myself.


Corinna

Re: SEEK_HOLE Support in Cygwin?

[Roland Mainz]

On Mon, Aug 21, 2023 at 6:33 AM Cedric Blancher
<cedric.blancher@gmail.com> wrote:
> On Thu, 10 Aug 2023 at 10:28, Corinna Vinschen
> <corinna-cygwin@cygwin.com> wrote:
> > On Aug 10 04:43, Cedric Blancher wrote:
[snip]
> > > > Note that this defaults to the default behaviour described in the
> > > > Linux man page, see https://man7.org/linux/man-pages/man2/lseek.2.html
> > > >
> > > > I.e., "In the simplest implementation, a filesystem can support the
> > > > operations by making SEEK_HOLE always return the offset of the end of
> > > > the file, and making SEEK_DATA always return offset"
> > >
> > > So this supports multiple holes in one file, right?
> >
> > Curious question.  Cygwin supports sparse files for ages.  Only the
> > lseek options SEEK_HOLE and SEEK_DATA are new.
>
> Is SEEK_HOLE support enabled for MSFT NFSv3 client?

The NFSv3 protocol does not support sparse files.
Support for sparse files, including SEEK_HOLE/SEEK_DATA, and
allocating/deallocating blocks was added in NFSv4.2 (see
https://datatracker.ietf.org/doc/html/rfc7862#page-32).

What needs to be done now is to add NFSv4.2 READ_PLUS support to the
CITI NFSv4.[12] driver, and hook up |FILE_SUPPORTS_SPARSE_FILES|
support - then sparse files should work.

Question for Corinna:
Does Cygwin support allocation/deallocation of filesystem blocks via
Linux's |fallocate()|-API ?

----

Bye,
Roland
-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz@nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Re: SEEK_HOLE Support in Cygwin?

[Corinna Vinschen]

On Aug 24 14:03, Roland Mainz wrote:
> Question for Corinna:

Everybody involved in the project could answer this question, but...

> Does Cygwin support allocation/deallocation of filesystem blocks via
> Linux's |fallocate()|-API ?

...just try this:

$ cat > a.c <<EOF
extern void fallocate();
extern void posix_fallocate();

int main()
{
  fallocate();
  posix_fallocate();
}
EOF
$ gcc -o a a.c

In fact, we have a list of supported APIs: https://cygwin.com/cygwin-api/


Corinna

Re: SEEK_HOLE Support in Cygwin?

[Roland Mainz]

On Fri, Aug 25, 2023 at 11:05 AM Corinna Vinschen
<corinna-cygwin@cygwin.com> wrote:
>
> On Aug 24 14:03, Roland Mainz wrote:
> > Question for Corinna:
>
> Everybody involved in the project could answer this question, but...
>
> > Does Cygwin support allocation/deallocation of filesystem blocks via
> > Linux's |fallocate()|-API ?
>
> ...just try this:
>
> $ cat > a.c <<EOF
> extern void fallocate();
> extern void posix_fallocate();
>
> int main()
> {
>   fallocate();
>   posix_fallocate();
> }
> EOF
> $ gcc -o a a.c

Thanks...
... I saw that the functions are there, but my question was more like
"supports"=="calls native Windows API".

|fallocate()| support seems to be in
./newlib-cygwin/winsup/cygwin/fhandler/disk_file.cc, and supports the
allocation of blocks - but I couldn't find support for
|FALLOC_FL_PUNCH_HOLE|, e.g. deallocation of blocks (to "punch" a hole
into the file).
Is this an oversight, or does Window's native API not support that ?

> In fact, we have a list of supported APIs: https://cygwin.com/cygwin-api/

Thanks

----

Bye,
Roland
--
  __ .  . __
 (o.\ \/ /.o) roland.mainz@nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Re: SEEK_HOLE Support in Cygwin?

[Corinna Vinschen]

On Aug 25 13:09, Roland Mainz wrote:
> On Fri, Aug 25, 2023 at 11:05 AM Corinna Vinschen
> <corinna-cygwin@cygwin.com> wrote:
> >
> > On Aug 24 14:03, Roland Mainz wrote:
> > > Question for Corinna:
> >
> > Everybody involved in the project could answer this question, but...
> >
> > > Does Cygwin support allocation/deallocation of filesystem blocks via
> > > Linux's |fallocate()|-API ?
> >
> > ...just try this:
> >
> > $ cat > a.c <<EOF
> > extern void fallocate();
> > extern void posix_fallocate();
> >
> > int main()
> > {
> >   fallocate();
> >   posix_fallocate();
> > }
> > EOF
> > $ gcc -o a a.c
> 
> Thanks...
> ... I saw that the functions are there,

So you didn't actually run the test?

> but my question was more like
> "supports"=="calls native Windows API".
> 
> |fallocate()| support seems to be in
> ./newlib-cygwin/winsup/cygwin/fhandler/disk_file.cc, and supports the
> allocation of blocks - but I couldn't find support for
> |FALLOC_FL_PUNCH_HOLE|, e.g. deallocation of blocks (to "punch" a hole
> into the file).
> Is this an oversight, or does Window's native API not support that ?

If you found fallocate() in fhandler/disk_file.cc, I need new glasses.


Corinna

Re: SEEK_HOLE Support in Cygwin?

[Corinna Vinschen]

On Aug 25 13:23, Corinna Vinschen wrote:
> On Aug 25 13:09, Roland Mainz wrote:
> > On Fri, Aug 25, 2023 at 11:05 AM Corinna Vinschen
> > <corinna-cygwin@cygwin.com> wrote:
> > >
> > > On Aug 24 14:03, Roland Mainz wrote:
> > > > Question for Corinna:
> > >
> > > Everybody involved in the project could answer this question, but...
> > >
> > > > Does Cygwin support allocation/deallocation of filesystem blocks via
> > > > Linux's |fallocate()|-API ?
> > >
> > > ...just try this:
> > >
> > > $ cat > a.c <<EOF
> > > extern void fallocate();
> > > extern void posix_fallocate();
> > >
> > > int main()
> > > {
> > >   fallocate();
> > >   posix_fallocate();
> > > }
> > > EOF
> > > $ gcc -o a a.c
> > 
> > Thanks...
> > ... I saw that the functions are there,
> 
> So you didn't actually run the test?
> 
> > but my question was more like
> > "supports"=="calls native Windows API".
> > 
> > |fallocate()| support seems to be in
> > ./newlib-cygwin/winsup/cygwin/fhandler/disk_file.cc, and supports the
> > allocation of blocks - but I couldn't find support for
> > |FALLOC_FL_PUNCH_HOLE|, e.g. deallocation of blocks (to "punch" a hole
> > into the file).
> > Is this an oversight, or does Window's native API not support that ?
> 
> If you found fallocate() in fhandler/disk_file.cc, I need new glasses.

No, we don't support fallocate, we only support posix_fallocate.
The flags to fallocate can be supported like this:

  0				== posix_fallocate
  FALLOC_FL_KEEP_SIZE		posix_fallocate with forcing sparse
  FALLOC_FL_UNSHARE_RANGE	no-op
  FALLOC_FL_PUNCH_HOLE		use Windows ioctl FSCTL_SET_ZERO_DATA

https://cygwin.com/acronyms/#SHTDI


Corinna