Cygwin api to punch a hole into a file?

Cygwin api to punch a hole into a file?

[Cedric Blancher]

Linux has fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, ...)
to punch a hole into a file, i.e. deallocate the blocks given and make
the file a "sparse file".

But how can I do that with the Cygwin API? Does anyone have an example
for Cygwin, which is the Win equivalent to Linux fallocate(fd,
FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, 1048576, 2097152);?

I tried with Cygwin 3.5, but the file was not sparse after that.

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: Cygwin api to punch a hole into a file?

[Corinna Vinschen]

On Nov 23 23:36, Cedric Blancher via Cygwin wrote:
> Linux has fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, ...)
> to punch a hole into a file, i.e. deallocate the blocks given and make
> the file a "sparse file".

We don't support the Linux-specific fallocate(2) call, only ftruncate(2)
and posix_fallocate(3).  Patches, as usual, thoughtfully considered.

> But how can I do that with the Cygwin API? Does anyone have an example
> for Cygwin, which is the Win equivalent to Linux fallocate(fd,
> FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, 1048576, 2097152);?
> 
> I tried with Cygwin 3.5, but the file was not sparse after that.

Do you remember the discussion in August, starting here:
https://cygwin.com/pipermail/cygwin-developers/2023-August/012664.html
especially:
https://cygwin.com/pipermail/cygwin-developers/2023-August/012679.html

So, did you set the sparse mount option per
https://cygwin.com/cygwin-ug-net/using.html#mount-table
?

Also, chattr -S, which is independent from the mount option "sparse":
https://cygwin.com/cygwin-ug-net/chattr.html


Corinna

Re: Cygwin api to punch a hole into a file?

[Corinna Vinschen]

On Nov 24 12:01, Corinna Vinschen via Cygwin wrote:
> On Nov 23 23:36, Cedric Blancher via Cygwin wrote:
> > Linux has fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, ...)
> > to punch a hole into a file, i.e. deallocate the blocks given and make
> > the file a "sparse file".
> 
> We don't support the Linux-specific fallocate(2) call, only ftruncate(2)
> and posix_fallocate(3).  Patches, as usual, thoughtfully considered.

The next test release cygwin-3.5.0-0.485.g65831f88d6c4 introduces
the Linux-specific fallocate(2) call.  Naturally we can't support
all flags, but the following flag combinations are allowed:

- 0				same as posix_fallocate(3)
- FALLOC_FL_KEEP_SIZE
- FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE
- FALLOC_FL_ZERO_RANGE
- FALLOC_FL_ZERO_RANGE | FALLOC_FL_KEEP_SIZE

A few comments:

- With 0 and FALLOC_FL_KEEP_SIZE, sparse blocks in the given range
  are re-allocated as per the POSIX requirements.  For that, it uses
  the same code as FALLOC_FL_ZERO_RANGE, but only for the holes within
  the range.

- With FALLOC_FL_KEEP_SIZE, over-allocation is done by setting
  the allocation size of a file while keeping EOF the same.
  However, in contrast to your typical Linux filesystem, over-
  allocation on Windows filesystems is only temporary.  The
  over-allocated blocks are returned to free blocks as soon as
  the last HANDLE to the file is closed.

- With FALLOC_FL_KEEP_SIZE, no over-allocation is performed on sparse
  files.  The reason is that over-allocation on sparse files has no
  effect on Windows.

- FALLOC_FL_ZERO_RANGE is implemented as writing zeros to the
  given range.  For parts of the range which are already allocated
  data blocks, as much zeros are written as necessary.  For holes
  in sparse files, only a single zero byte is written to the
  hole per 64K chunk, which is the allocation granularity of
  sparse files.

- FALLOC_FL_ZERO_RANGE is NOT atomic.

Please give it a try.


Corinna

Re: Cygwin api to punch a hole into a file?

[Cedric Blancher]

On Tue, 28 Nov 2023 at 11:30, Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Nov 24 12:01, Corinna Vinschen via Cygwin wrote:
> > On Nov 23 23:36, Cedric Blancher via Cygwin wrote:
> > > Linux has fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, ...)
> > > to punch a hole into a file, i.e. deallocate the blocks given and make
> > > the file a "sparse file".
> >
> > We don't support the Linux-specific fallocate(2) call, only ftruncate(2)
> > and posix_fallocate(3).  Patches, as usual, thoughtfully considered.
>
> The next test release cygwin-3.5.0-0.485.g65831f88d6c4 introduces
> the Linux-specific fallocate(2) call.  Naturally we can't support
> all flags, but the following flag combinations are allowed:
>
> - 0                             same as posix_fallocate(3)
> - FALLOC_FL_KEEP_SIZE
> - FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE
> - FALLOC_FL_ZERO_RANGE
> - FALLOC_FL_ZERO_RANGE | FALLOC_FL_KEEP_SIZE
>
> A few comments:
>
> - With 0 and FALLOC_FL_KEEP_SIZE, sparse blocks in the given range
>   are re-allocated as per the POSIX requirements.  For that, it uses
>   the same code as FALLOC_FL_ZERO_RANGE, but only for the holes within
>   the range.
>
> - With FALLOC_FL_KEEP_SIZE, over-allocation is done by setting
>   the allocation size of a file while keeping EOF the same.
>   However, in contrast to your typical Linux filesystem, over-
>   allocation on Windows filesystems is only temporary.  The
>   over-allocated blocks are returned to free blocks as soon as
>   the last HANDLE to the file is closed.
>
> - With FALLOC_FL_KEEP_SIZE, no over-allocation is performed on sparse
>   files.  The reason is that over-allocation on sparse files has no
>   effect on Windows.
>
> - FALLOC_FL_ZERO_RANGE is implemented as writing zeros to the
>   given range.  For parts of the range which are already allocated
>   data blocks, as much zeros are written as necessary.  For holes
>   in sparse files, only a single zero byte is written to the
>   hole per 64K chunk, which is the allocation granularity of
>   sparse files.
>
> - FALLOC_FL_ZERO_RANGE is NOT atomic.
>
> Please give it a try.

Corinna, THANK you!

Related question about commit "Cygwin: pwrite(2): sparsify file"
https://cygwin.com/git/?p=newlib-cygwin.git;a=blobdiff;f=winsup/cygwin/fhandler/disk_file.cc;h=c70afed49f1ecdb11812d31a9663d18c0a5f03f7;hp=b49b25c71ad030a9391db89a157cadcd095d4f36;hb=f64f3eced8e0f51753bc199f3ba96fab06a54848;hpb=114f89caff7b9b62b0b12bc2c6d143daf47b8042

I see the value of 128k (128*1024 bytes) quite often in your sparse
file commits. Can you please make this value a per filesystem tunable?
Not all filesystems have a 128k block/stripe size, and certainly most
filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes
is common, ref pathconf _PC_MIN_HOLE_SIZE).

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: Cygwin api to punch a hole into a file?

[Corinna Vinschen]

On Dec  1 11:22, Cedric Blancher via Cygwin wrote:
> I see the value of 128k (128*1024 bytes) quite often in your sparse
> file commits.

Yes, but they have been removed.  Read the patches again, especially commit
65831f88d6c4.

> Can you please make this value a per filesystem tunable?

No, because we don't have a facility for that.

> Not all filesystems have a 128k block/stripe size, and certainly most
> filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes
> is common, ref pathconf _PC_MIN_HOLE_SIZE).

There's no _PC_MIN_HOLE_SIZE in Linux or POSIX.  In Windows, a sparse
file uses chunks of 64K.  You can see this even with a file of just
a single block.  Try this:

  $ touch x
  $ chattr +S x

  $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
  $ ls -ls x
  1 -rw-r--r-- 1 corinna vinschen 80 Dec  1 11:56 x

  [repeat echo and ls -ls until...]

  $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
  $ ls -ls x
  1 -rw-r--r-- 1 corinna vinschen 720 Dec  1 11:56 x
  $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
  $ ls -ls x
  64 -rw-r--r-- 1 corinna vinschen 800 Dec  1 11:56 x2
  ^^
  This

I don't know how Windows stores short files, maybe they are part of the
metadata up to a point or some such.  However, as soon as you raise the
size over a given point, the *allocation size* will be rounded up to 64K
and from that point on, it will be in 64K chunks.  Also sparsifying and
desparsifying of blocks only works in 64K chunks.


Corinna

Re: Cygwin api to punch a hole into a file?

[Andrey Repin]

Greetings, Corinna Vinschen via Cygwin!

> I don't know how Windows stores short files,

By Windows I Think you did mean NTFS.
Then yes, file data for certain short values is stored directly in $MFT.
This is not specifically file contents, and even if a file content is just 1
byte, not necessarily it will be stored in $MFT. It all depends on the entire
size of file metadata. If it does not exceed some-under-1K in total, then file
data may be stored in $MFT as well.

> maybe they are part of the metadata up to a point or some such.  However, as
> soon as you raise the size over a given point, the *allocation size* will be
> rounded up to 64K and from that point on, it will be in 64K chunks.  Also
> sparsifying and desparsifying of blocks only works in 64K chunks.


-- 
With best regards,
Andrey Repin
Monday, December 4, 2023 18:56:37

Sorry for my terrible english...

Re: Cygwin api to punch a hole into a file?

[Thomas Wolff]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]



Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin:
> On Dec  1 11:22, Cedric Blancher via Cygwin wrote:
>> Not all filesystems have a 128k block/stripe size, and certainly most
>> filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes
>> is common, ref pathconf _PC_MIN_HOLE_SIZE).
> There's no _PC_MIN_HOLE_SIZE in Linux or POSIX.  In Windows, a sparse
> file uses chunks of 64K.  You can see this even with a file of just
> a single block.  Try this:
>
>    $ touch x
>    $ chattr +S x
>
>    $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
>    $ ls -ls x
>    1 -rw-r--r-- 1 corinna vinschen 80 Dec  1 11:56 x
>
>    [repeat echo and ls -ls until...]
>
>    $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
>    $ ls -ls x
>    1 -rw-r--r-- 1 corinna vinschen 720 Dec  1 11:56 x
>    $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
>    $ ls -ls x
>    64 -rw-r--r-- 1 corinna vinschen 800 Dec  1 11:56 x2
>    ^^
>    This
For me, it goes up from 1 to 4, then in steps of 4KB.

> I don't know how Windows stores short files, maybe they are part of the
> metadata up to a point or some such.  However, as soon as you raise the
> size over a given point, the *allocation size* will be rounded up to 64K
> and from that point on, it will be in 64K chunks.  Also sparsifying and
> desparsifying of blocks only works in 64K chunks.
>
>
> Corinna
>

Re: Cygwin api to punch a hole into a file?

[Corinna Vinschen]

On Dec  5 07:36, Thomas Wolff via Cygwin wrote:
> 
> 
> Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin:
> > On Dec  1 11:22, Cedric Blancher via Cygwin wrote:
> > > Not all filesystems have a 128k block/stripe size, and certainly most
> > > filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes
> > > is common, ref pathconf _PC_MIN_HOLE_SIZE).
> > There's no _PC_MIN_HOLE_SIZE in Linux or POSIX.  In Windows, a sparse
> > file uses chunks of 64K.  You can see this even with a file of just
> > a single block.  Try this:
> > 
> >    $ touch x
> >    $ chattr +S x
> > 
> >    $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
> >    $ ls -ls x
> >    1 -rw-r--r-- 1 corinna vinschen 80 Dec  1 11:56 x
> > 
> >    [repeat echo and ls -ls until...]
> > 
> >    $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
> >    $ ls -ls x
> >    1 -rw-r--r-- 1 corinna vinschen 720 Dec  1 11:56 x
> >    $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
> >    $ ls -ls x
> >    64 -rw-r--r-- 1 corinna vinschen 800 Dec  1 11:56 x2
> >    ^^
> >    This
> For me, it goes up from 1 to 4, then in steps of 4KB.

Is that a local NTFS, did you actually call

  chattr +S x

after touching the file, and did you check with

  lsattr x

that x is actually sparse?


Corinna

Re: Cygwin api to punch a hole into a file?

[Thomas Wolff]

Am 05/12/2023 um 14:53 schrieb Corinna Vinschen via Cygwin:
> On Dec  5 07:36, Thomas Wolff via Cygwin wrote:
>>
>> Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin:
>>> On Dec  1 11:22, Cedric Blancher via Cygwin wrote:
>>>> Not all filesystems have a 128k block/stripe size, and certainly most
>>>> filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes
>>>> is common, ref pathconf _PC_MIN_HOLE_SIZE).
>>> There's no _PC_MIN_HOLE_SIZE in Linux or POSIX.  In Windows, a sparse
>>> file uses chunks of 64K.  You can see this even with a file of just
>>> a single block.  Try this:
>>>
>>>     $ touch x
>>>     $ chattr +S x
>>>
>>>     $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
>>>     $ ls -ls x
>>>     1 -rw-r--r-- 1 corinna vinschen 80 Dec  1 11:56 x
>>>
>>>     [repeat echo and ls -ls until...]
>>>
>>>     $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
>>>     $ ls -ls x
>>>     1 -rw-r--r-- 1 corinna vinschen 720 Dec  1 11:56 x
>>>     $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x
>>>     $ ls -ls x
>>>     64 -rw-r--r-- 1 corinna vinschen 800 Dec  1 11:56 x2
>>>     ^^
>>>     This
>> For me, it goes up from 1 to 4, then in steps of 4KB.
> Is that a local NTFS, did you actually call
>
>    chattr +S x
>
> after touching the file, and did you check with
>
>    lsattr x
>
> that x is actually sparse?
Ups, sorry, I was just throwing my 2p into something. Yes, on a local 
NTFS; setting chattr +s does not change it but lsattr says:
---a-------- .ls-s

So why does chattr not seem to work?
Thomas

> Corinna
>

Re: Cygwin api to punch a hole into a file?

[Thomas Wolff]

Am 05/12/2023 um 15:40 schrieb Thomas Wolff via Cygwin:
> Am 05/12/2023 um 14:53 schrieb Corinna Vinschen via Cygwin:
>> On Dec  5 07:36, Thomas Wolff via Cygwin wrote:
>>>
>>> Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin:
>>>> On Dec  1 11:22, Cedric Blancher via Cygwin wrote:
>>>>> Not all filesystems have a 128k block/stripe size, and certainly most
>>>>> filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes
>>>>> is common, ref pathconf _PC_MIN_HOLE_SIZE).
>>>> There's no _PC_MIN_HOLE_SIZE in Linux or POSIX.  In Windows, a sparse
>>>> file uses chunks of 64K.  You can see this even with a file of just
>>>> a single block.  Try this:
>>>>
>>>>     $ touch x
>>>>     $ chattr +S x
>>>>
>>>>     $ echo 
>>>> 1234567890123456789012345678901234567890123456789012345678901234567890123456789 
>>>> >> x
>>>>     $ ls -ls x
>>>>     1 -rw-r--r-- 1 corinna vinschen 80 Dec  1 11:56 x
>>>>
>>>>     [repeat echo and ls -ls until...]
>>>>
>>>>     $ echo 
>>>> 1234567890123456789012345678901234567890123456789012345678901234567890123456789 
>>>> >> x
>>>>     $ ls -ls x
>>>>     1 -rw-r--r-- 1 corinna vinschen 720 Dec  1 11:56 x
>>>>     $ echo 
>>>> 1234567890123456789012345678901234567890123456789012345678901234567890123456789 
>>>> >> x
>>>>     $ ls -ls x
>>>>     64 -rw-r--r-- 1 corinna vinschen 800 Dec  1 11:56 x2
>>>>     ^^
>>>>     This
>>> For me, it goes up from 1 to 4, then in steps of 4KB.
>> Is that a local NTFS, did you actually call
>>
>>    chattr +S x
>>
>> after touching the file, and did you check with
>>
>>    lsattr x
>>
>> that x is actually sparse?
> Ups, sorry, I was just throwing my 2p into something. Yes, on a local 
> NTFS; setting chattr +s does not change it but lsattr says:
> ---a-------- .ls-s
>
> So why does chattr not seem to work?
Sorry again, I failed to test properly (I had copied your `chattr +S x` 
which is not the filename I tested... blush).
With chattr +S, I reproduce your observation.

> Thomas
>
>> Corinna
>>
>
>

Re: Cygwin api to punch a hole into a file?

[Corinna Vinschen]

On Dec  5 15:47, Thomas Wolff via Cygwin wrote:
> 
> Am 05/12/2023 um 15:40 schrieb Thomas Wolff via Cygwin:
> > Am 05/12/2023 um 14:53 schrieb Corinna Vinschen via Cygwin:
> > > On Dec  5 07:36, Thomas Wolff via Cygwin wrote:
> > > > 
> > > > Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin:
> > > > > On Dec  1 11:22, Cedric Blancher via Cygwin wrote:
> > > > > > Not all filesystems have a 128k block/stripe size, and certainly most
> > > > > > filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes
> > > > > > is common, ref pathconf _PC_MIN_HOLE_SIZE).
> > > > > There's no _PC_MIN_HOLE_SIZE in Linux or POSIX.  In Windows, a sparse
> > > > > file uses chunks of 64K.  You can see this even with a file of just
> > > > > a single block.  Try this:
> > > > > 
> > > > >     $ touch x
> > > > >     $ chattr +S x
> > > > > 
> > > > >     $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789
> > > > > >> x
> > > > >     $ ls -ls x
> > > > >     1 -rw-r--r-- 1 corinna vinschen 80 Dec  1 11:56 x
> > > > > 
> > > > >     [repeat echo and ls -ls until...]
> > > > > 
> > > > >     $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789
> > > > > >> x
> > > > >     $ ls -ls x
> > > > >     1 -rw-r--r-- 1 corinna vinschen 720 Dec  1 11:56 x
> > > > >     $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789
> > > > > >> x
> > > > >     $ ls -ls x
> > > > >     64 -rw-r--r-- 1 corinna vinschen 800 Dec  1 11:56 x2
> > > > >     ^^
> > > > >     This
> > > > For me, it goes up from 1 to 4, then in steps of 4KB.
> > > Is that a local NTFS, did you actually call
> > > 
> > >    chattr +S x
> > > 
> > > after touching the file, and did you check with
> > > 
> > >    lsattr x
> > > 
> > > that x is actually sparse?
> > Ups, sorry, I was just throwing my 2p into something. Yes, on a local
> > NTFS; setting chattr +s does not change it but lsattr says:
> > ---a-------- .ls-s
> > 
> > So why does chattr not seem to work?
> Sorry again, I failed to test properly (I had copied your `chattr +S x`
> which is not the filename I tested... blush).
> With chattr +S, I reproduce your observation.

No worries. Thanks for testing and confirming!


Corinna

_PC_MIN_HOLE_SIZE pathconf() Re: Cygwin api to punch a hole into a file?

[Cedric Blancher]

On Tue, 28 Nov 2023 at 11:30, Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Nov 24 12:01, Corinna Vinschen via Cygwin wrote:
> > On Nov 23 23:36, Cedric Blancher via Cygwin wrote:
> > > Linux has fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, ...)
> > > to punch a hole into a file, i.e. deallocate the blocks given and make
> > > the file a "sparse file".
> >
> > We don't support the Linux-specific fallocate(2) call, only ftruncate(2)
> > and posix_fallocate(3).  Patches, as usual, thoughtfully considered.
>
> The next test release cygwin-3.5.0-0.485.g65831f88d6c4 introduces
> the Linux-specific fallocate(2) call.  Naturally we can't support
> all flags, but the following flag combinations are allowed:
>
> - 0                             same as posix_fallocate(3)
> - FALLOC_FL_KEEP_SIZE
> - FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE
> - FALLOC_FL_ZERO_RANGE
> - FALLOC_FL_ZERO_RANGE | FALLOC_FL_KEEP_SIZE
>
> A few comments:
>
> - With 0 and FALLOC_FL_KEEP_SIZE, sparse blocks in the given range
>   are re-allocated as per the POSIX requirements.  For that, it uses
>   the same code as FALLOC_FL_ZERO_RANGE, but only for the holes within
>   the range.
>
> - With FALLOC_FL_KEEP_SIZE, over-allocation is done by setting
>   the allocation size of a file while keeping EOF the same.
>   However, in contrast to your typical Linux filesystem, over-
>   allocation on Windows filesystems is only temporary.  The
>   over-allocated blocks are returned to free blocks as soon as
>   the last HANDLE to the file is closed.
>
> - With FALLOC_FL_KEEP_SIZE, no over-allocation is performed on sparse
>   files.  The reason is that over-allocation on sparse files has no
>   effect on Windows.
>
> - FALLOC_FL_ZERO_RANGE is implemented as writing zeros to the
>   given range.  For parts of the range which are already allocated
>   data blocks, as much zeros are written as necessary.  For holes
>   in sparse files, only a single zero byte is written to the
>   hole per 64K chunk, which is the allocation granularity of
>   sparse files.
>
> - FALLOC_FL_ZERO_RANGE is NOT atomic.
>
> Please give it a try.

Could you please implement _PC_MIN_HOLE_SIZE in pathconf(), so that
people can query the minimum hole size for a (path in a) filesystem?
Should return nothing for FAT and NFSv3, should work on NTFS and
NFSv4.2

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur