Wish list: Cygwin `mv` to move OneDrive files without downloading them

Wish list: Cygwin `mv` to move OneDrive files without downloading them

[Adam Dinwoodie]

Microsoft OneDrive has a "Files On-Demand" function, where it will
synchronise file metadata to a local system, but won't actually download
the file content until an application attempts to read the content.
When moving a file within Cygwin using `mv`, the file always gets
downloaded, which seems like it shouldn't be necessary.  Is there any
way to have `mv` (and presumably the underlying rename call) work
without downloading the file content in this circumstance?  It'd
definitely make some of my life easier, but I'm not sure if it's a
trivial issue, one that would require years of work, or somwhere in
between...

As best I can tell, `mv` doesn't need to know the content of the file,
at least as long as it's not moving the file outside of OneDrive; it
feels very similar to me to moving a file within a partition on *nix:
generally it's just a case of updating the directory records, with no
need to look at the file content.

This does work as desired from within PowerShell: PowerShell's
`Move-Item` won't download a file that's not currently stored on the
local system.  So it's presumably possible to achieve this, although I
don't know if it'd rely on non-public Microsoft APIs, and/or lots more
complexity in the Cygwin code.

Simple demonstration below, showing the different behaviours of
PowerShell versus Cygwin.  I've not included a cygcheck.out, as I'm
pretty sure this is much closer to a feature request than a problem
report, but let me know if I've got that wrong.

    PS C:\Users\adinwoodie\OneDrive> # Create a directory and some files to experiment with
    PS C:\Users\adinwoodie\OneDrive> New-Item -Type directory -Name temp


        Directory: C:\Users\adinwoodie\OneDrive


    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    d-----        14/06/2022     12:09                temp


    PS C:\Users\adinwoodie\OneDrive> cd temp
    PS C:\Users\adinwoodie\OneDrive\temp> New-Item -Type file -Name move-with-pwsh -Value "Some arbitrary file content"


        Directory: C:\Users\adinwoodie\OneDrive\temp


    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    -a----        14/06/2022     12:10             27 move-with-pwsh


    PS C:\Users\adinwoodie\OneDrive\temp> New-Item -Type file -Name move-with-cygwin -Value "Some arbitrary file content"


        Directory: C:\Users\adinwoodie\OneDrive\temp


    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    -a----        14/06/2022     12:10             27 move-with-cygwin


    PS C:\Users\adinwoodie\OneDrive\temp> # Look at the directory listing prior to recovering the disk space
    PS C:\Users\adinwoodie\OneDrive\temp> dir


        Directory: C:\Users\adinwoodie\OneDrive\temp


    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    -a---l        14/06/2022     12:10             27 move-with-cygwin
    -a---l        14/06/2022     12:10             27 move-with-pwsh


    PS C:\Users\adinwoodie\OneDrive\temp> # Mark the files as to be stored online and the local disk space to be recovered
    PS C:\Users\adinwoodie\OneDrive\temp> attrib.exe +O +U *
    PS C:\Users\adinwoodie\OneDrive\temp> # Check the disk space is now marked as recovered; note the brackets around the file length
    PS C:\Users\adinwoodie\OneDrive\temp> dir


        Directory: C:\Users\adinwoodie\OneDrive\temp


    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    -a---l        14/06/2022     12:10           (27) move-with-cygwin
    -a---l        14/06/2022     12:10           (27) move-with-pwsh


    PS C:\Users\adinwoodie\OneDrive\temp> # Move the files
    PS C:\Users\adinwoodie\OneDrive\temp> Move-Item move-with-pwsh moved-with-pwsh
    PS C:\Users\adinwoodie\OneDrive\temp> C:\cygwin64\bin\mv.exe move-with-cygwin moved-with-cygwin
    PS C:\Users\adinwoodie\OneDrive\temp> # Note the version moved with Cygwin is now consuming local disk space again, while the version moved with PowerShell doesn't
    PS C:\Users\adinwoodie\OneDrive\temp> dir


        Directory: C:\Users\adinwoodie\OneDrive\temp


    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    -a---l        14/06/2022     12:10             27 moved-with-cygwin
    -a---l        14/06/2022     12:10           (27) moved-with-pwsh

Re: Wish list: Cygwin `mv` to move OneDrive files without downloading them

[Sam Edge]

On 14/06/2022 12:25, Adam Dinwoodie wrote:
 > Microsoft OneDrive has a "Files On-Demand" function, where it will
 > synchronise file metadata to a local system, but won't actually download
 > the file content until an application attempts to read the content.
 > When moving a file within Cygwin using `mv`, the file always gets
 > downloaded, which seems like it shouldn't be necessary.  Is there any
 > way to have `mv` (and presumably the underlying rename call) work
 > without downloading the file content in this circumstance? It'd
 > definitely make some of my life easier, but I'm not sure if it's a
 > trivial issue, one that would require years of work, or somwhere in
 > between...
 >
 > As best I can tell, `mv` doesn't need to know the content of the file,
 > at least as long as it's not moving the file outside of OneDrive; it
 > feels very similar to me to moving a file within a partition on *nix:
 > generally it's just a case of updating the directory records, with no
 > need to look at the file content.
 >
 > This does work as desired from within PowerShell: PowerShell's
 > `Move-Item` won't download a file that's not currently stored on the
 > local system.  So it's presumably possible to achieve this, although I
 > don't know if it'd rely on non-public Microsoft APIs, and/or lots more
 > complexity in the Cygwin code.
 >
 > Simple demonstration below, showing the different behaviours of
 > PowerShell versus Cygwin.  I've not included a cygcheck.out, as I'm
 > pretty sure this is much closer to a feature request than a problem
 > report, but let me know if I've got that wrong.

Greetings Adam.

I wasn't even aware you could do that with OneDrive - there's still no
Explorer
way of doing it in Windows 10 21H2 as far as I can see. Thanks for the
info. :-)

I imagine Cygwin doesn't know about the extended attributes being used
within
the local copy of your OneDrive directory so it is doing the move using the
underlying POSIX 'rename within mount' API which hopefully uses the Win32
'rename' but not in a way that carries the new 'O' attribute.

Presumably PowerShell's Move-Item does use the right flags to the Win32 API
call.

One of the problems the Cygwin maintainers have is that Microsoft introduces
'enhancements' to NTFS & the Win32 API arbitrarily in Windows updates
without
announcing them so it's a game of whack'a'mole.

--
Sam Edge

Re: Wish list: Cygwin `mv` to move OneDrive files without downloading them

[Adam Dinwoodie]

On Tue, Jun 14, 2022 at 04:32:21PM +0100, Sam Edge wrote:
> On 14/06/2022 12:25, Adam Dinwoodie wrote:
> > Microsoft OneDrive has a "Files On-Demand" function, where it will
> > synchronise file metadata to a local system, but won't actually download
> > the file content until an application attempts to read the content.
> > When moving a file within Cygwin using `mv`, the file always gets
> > downloaded, which seems like it shouldn't be necessary.  Is there any
> > way to have `mv` (and presumably the underlying rename call) work
> > without downloading the file content in this circumstance? It'd
> > definitely make some of my life easier, but I'm not sure if it's a
> > trivial issue, one that would require years of work, or somwhere in
> > between...
> >
> > As best I can tell, `mv` doesn't need to know the content of the file,
> > at least as long as it's not moving the file outside of OneDrive; it
> > feels very similar to me to moving a file within a partition on *nix:
> > generally it's just a case of updating the directory records, with no
> > need to look at the file content.
> >
> > This does work as desired from within PowerShell: PowerShell's
> > `Move-Item` won't download a file that's not currently stored on the
> > local system.  So it's presumably possible to achieve this, although I
> > don't know if it'd rely on non-public Microsoft APIs, and/or lots more
> > complexity in the Cygwin code.
> >
> > Simple demonstration below, showing the different behaviours of
> > PowerShell versus Cygwin.  I've not included a cygcheck.out, as I'm
> > pretty sure this is much closer to a feature request than a problem
> > report, but let me know if I've got that wrong.
> 
> Greetings Adam.
> 
> I wasn't even aware you could do that with OneDrive - there's still no
> Explorer
> way of doing it in Windows 10 21H2 as far as I can see. Thanks for the
> info. :-)

FWIW, this also works exactly the same using click-and-drag or Ctrl+C /
Ctrl+V in File Explorer for me, on all my Windows systems (currently
running a mix of Win11 Enterprise and Win11 Pro, both the 22H2
releases), and I *think* I remember it working on Win10 too, but I'm
less confident there.

> I imagine Cygwin doesn't know about the extended attributes being used
> within
> the local copy of your OneDrive directory so it is doing the move using the
> underlying POSIX 'rename within mount' API which hopefully uses the Win32
> 'rename' but not in a way that carries the new 'O' attribute.

That seems very plausible; I know Cygwin's mv/rename seems to keep at
least some of the Windows file attributes, but I definitely don't
profess any expertise here.

> Presumably PowerShell's Move-Item does use the right flags to the Win32 API
> call.
> 
> One of the problems the Cygwin maintainers have is that Microsoft introduces
> 'enhancements' to NTFS & the Win32 API arbitrarily in Windows updates
> without
> announcing them so it's a game of whack'a'mole.

Yeah, I am painfully familiar with that sort of problem from Microsoft,
not just with Windows!