Renaming (with 'mv') very large files is SLOW

Renaming (with 'mv') very large files is SLOW

[cygwin]

I tried renaming some very large files (20-40 GB) using:
   mv <oldname> <newname>
without changing the directory of course.

The process took about 10-20 minutes with Task Manager showing disk
activity of 100+ MB/s.

Is there something about such large 'renaming' that actually results
in the file being really moved (aka copied) rather than just renamed?

Re: Renaming (with 'mv') very large files is SLOW

[Eliot Moss]

On 1/31/2022 9:52 AM, cygwin@kosowsky.org wrote:
 > I tried renaming some very large files (20-40 GB) using:
 >     mv <oldname> <newname>
 > without changing the directory of course.
 >
 > The process took about 10-20 minutes with Task Manager showing disk
 > activity of 100+ MB/s.
 >
 > Is there something about such large 'renaming' that actually results
 > in the file being really moved (aka copied) rather than just renamed?

The two places are probably on different volumes (loosely, different disks).
That requires a physical move, even under Linux.  Your volumes seem a bit slow
to access - is one perhaps across a slow network?  The rates you cite suggest
movement of 50Mb/s (50Mb read + 50Mb write = 100Mb overall).  For 40 Gb that
should take 40Gb / 50Mb = about 820 secs = a little under 14 mins.

(When I say your volumes are slow, I speak from the luxury of having a 2Tb
solid state drive!  Actually, those speeds may be reasonable depending on the
nature of your system.)

If the two locations are on different drives, there's no real avoiding this.

Regards - Eliot Moss

Re: Renaming (with 'mv') very large files is SLOW

[René Berber]

On 1/31/2022 8:59 AM, Eliot Moss wrote:

> On 1/31/2022 9:52 AM, cygwin@kosowsky.org wrote:

>> I tried renaming some very large files (20-40 GB) using: mv
>> <oldname> <newname> without changing the directory of course.
>> 
>> The process took about 10-20 minutes with Task Manager showing
>> disk activity of 100+ MB/s.
>> 
>> Is there something about such large 'renaming' that actually
>> results in the file being really moved (aka copied) rather than
>> just renamed?
> 
> The two places are probably on different volumes (loosely, different
>  disks). That requires a physical move, even under Linux.  Your
> volumes seem a bit slow to access - is one perhaps across a slow
> network?  The rates you cite suggest movement of 50Mb/s (50Mb read +
> 50Mb write = 100Mb overall).  For 40 Gb that should take 40Gb / 50Mb
> = about 820 secs = a little under 14 mins.
> 
> (When I say your volumes are slow, I speak from the luxury of having
> a 2Tb solid state drive!  Actually, those speeds may be reasonable
> depending on the nature of your system.)
> 
> If the two locations are on different drives, there's no real
> avoiding this.

Nope, I've also complained about this (long ago), if the two locations
are the same remote drive... Cygwin moves the entire file over the network.

I ended up writing my own Samba mv command.
-- 
R. Berber

Re: Renaming (with 'mv') very large files is SLOW

[cygwin]

René Berber wrote at about 09:13:59 -0600 on Monday, January 31, 2022:
 > On 1/31/2022 8:59 AM, Eliot Moss wrote:
 > 
 > > On 1/31/2022 9:52 AM, cygwin@kosowsky.org wrote:
 > 
 > >> I tried renaming some very large files (20-40 GB) using: mv
 > >> <oldname> <newname> without changing the directory of course.
 > >> 
 > >> The process took about 10-20 minutes with Task Manager showing
 > >> disk activity of 100+ MB/s.
 > >> 
 > >> Is there something about such large 'renaming' that actually
 > >> results in the file being really moved (aka copied) rather than
 > >> just renamed?
 > > 
 > > The two places are probably on different volumes (loosely, different
 > >  disks). That requires a physical move, even under Linux.  Your
 > > volumes seem a bit slow to access - is one perhaps across a slow
 > > network?  The rates you cite suggest movement of 50Mb/s (50Mb read +
 > > 50Mb write = 100Mb overall).  For 40 Gb that should take 40Gb / 50Mb
 > > = about 820 secs = a little under 14 mins.
 > > 
 > > (When I say your volumes are slow, I speak from the luxury of having
 > > a 2Tb solid state drive!  Actually, those speeds may be reasonable
 > > depending on the nature of your system.)
 > > 
 > > If the two locations are on different drives, there's no real
 > > avoiding this.
 > 
 > Nope, I've also complained about this (long ago), if the two locations
 > are the same remote drive... Cygwin moves the entire file over the network.
 > 
 > I ended up writing my own Samba mv command.
 > -- 

In my case the directory (where I renamed the file) is on my C-drive.

RE: [EXTERNAL] Re: Renaming (with 'mv') very large files is SLOW

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> 
> In my case the directory (where I renamed the file) is on my C-drive.
>

If the file is currently open for access, the move can turn out to be a copy, actually.

Check open files to make sure whatever you are moving is not being opened / accessed from any other program.

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: Renaming (with 'mv') very large files is SLOW

[cygwin]

Eliot Moss wrote at about 09:59:17 -0500 on Monday, January 31, 2022:
 > On 1/31/2022 9:52 AM, cygwin@kosowsky.org wrote:
 >  > I tried renaming some very large files (20-40 GB) using:
 >  >     mv <oldname> <newname>
 >  > without changing the directory of course.
 >  >
 >  > The process took about 10-20 minutes with Task Manager showing disk
 >  > activity of 100+ MB/s.
 >  >
 >  > Is there something about such large 'renaming' that actually results
 >  > in the file being really moved (aka copied) rather than just renamed?
 > 
 > The two places are probably on different volumes (loosely, different disks).
 > That requires a physical move, even under Linux.

No my point is I am just *renaming*, not physically moving the file!!
i.e., I am not changing the directory location of the file, let alone
the volume/disk location.
(I am well aware that 'mv' does a copy when changing volumes/disks).

I literally am typing something like:
  mv foo bar

In Linux, that just edits the file system table & inode...

UPDATE...
I just tried a second 'mv' and it was near instantaneous.
(and similarly with subsequent renaming of the same file)
So perhaps not a 'Cygwin' thing but something going on within Windows.

Could it be that the first 'mv' triggered an anti-virus read of the file since
perhaps it detects it as a new/changed file?

But if so, would 'mv' (under Task Manager) be showing the 100+ MB/s disk activity?

Re: Renaming (with 'mv') very large files is SLOW

[Adam Dinwoodie]

On Mon, Jan 31, 2022 at 10:17:59AM -0500, cygwin@kosowsky.org wrote:
> Eliot Moss wrote at about 09:59:17 -0500 on Monday, January 31, 2022:
>  > On 1/31/2022 9:52 AM, cygwin@kosowsky.org wrote:
>  >  > I tried renaming some very large files (20-40 GB) using:
>  >  >     mv <oldname> <newname>
>  >  > without changing the directory of course.
>  >  >
>  >  > The process took about 10-20 minutes with Task Manager showing disk
>  >  > activity of 100+ MB/s.
>  >  >
>  >  > Is there something about such large 'renaming' that actually results
>  >  > in the file being really moved (aka copied) rather than just renamed?
>  > 
>  > The two places are probably on different volumes (loosely, different disks).
>  > That requires a physical move, even under Linux.
> 
> No my point is I am just *renaming*, not physically moving the file!!
> i.e., I am not changing the directory location of the file, let alone
> the volume/disk location.
> (I am well aware that 'mv' does a copy when changing volumes/disks).
> 
> I literally am typing something like:
>   mv foo bar
> 
> In Linux, that just edits the file system table & inode...
> 
> UPDATE...
> I just tried a second 'mv' and it was near instantaneous.
> (and similarly with subsequent renaming of the same file)
> So perhaps not a 'Cygwin' thing but something going on within Windows.
> 
> Could it be that the first 'mv' triggered an anti-virus read of the file since
> perhaps it detects it as a new/changed file?
> 
> But if so, would 'mv' (under Task Manager) be showing the 100+ MB/s disk activity?

That definitely seems plausible; there's a reason a significant number
of the applications that are known to interfere with Cygwin operation
(see [0]) are antivirus applications.  But what would trigger your
antivirus to want to scan a file, and how much work is required to do
that, is something you'll need to take up with your antivirus vendor,
I'm afraid.

[0]: https://cygwin.com/faq/faq.html#faq.using.bloda

Re: Renaming (with 'mv') very large files is SLOW

[L A Walsh]

On 2022/01/31 13:36, Adam Dinwoodie wrote:
>
>> Could it be that the first 'mv' triggered an anti-virus read of the file since
>> perhaps it detects it as a new/changed file?
>>
>> But if so, would 'mv' (under Task Manager) be showing the 100+ MB/s disk activity?
>>     
>
> That definitely seems plausible; there's a reason a significant number
> of the applications that are known to interfere with Cygwin operation
> (see [0]) are antivirus applications.  But what would trigger your
> antivirus to want to scan a file, and how much work is required to do
> that, is something you'll need to take up with your antivirus vendor,
> I'm afraid.
>   
----
    Something that most people don't realize, is that windows always
puts a lock on a file when it is going to READ it.  It's an advisory lock,
and usually, on a local file access, it can be removed by the user who
started the read  and it's not noticed.
    But if cygwin is accessing the file through some virtual table, Windows
might think it is on a separate virtual device -- like an indexing 
scanner that
indexes content, or anti-vir.

This becomes real noticeable if it is a real remote file on a remote fs.

    In samba there's a setting to allow breaking advisory locks -- and 
if you
are the only person who can be accessing that file, its best to allow 
them to
be broken -- only real useful if you have multiple users (or programs) 
trying to
modify the same file at the same time.  If the oplocks are held by 
another process
windows may return a 'file busy' so cygwin uses a file-copy method to 'move'
the file.  I usually only run into this locally when the file is opened 
by a system
process when I try to modify it, like deleting a thumbs.db when explorer 
is updating
it.

The param in samba is "fake oplocks = yes", tells samba to fake oplocks 
and not
really enforce them.  It's a per-share parameter, so you need to set it 
on every
share.  But only on shares where you are the only 'modifier'.  For actual
shared-access w/other users -- only read-only access should be used.

    If you ever want to have local file caching of remote content work 
-- need to set
the oplocks to fake or have the files be read-only.



> [0]: https://cygwin.com/faq/faq.html#faq.using.bloda
>
>   

Re: Renaming (with 'mv') very large files is SLOW

[Andrey Repin]

Greetings, cygwin@kosowsky.org!

> I literally am typing something like:
>   mv foo bar

> In Linux, that just edits the file system table & inode...

> UPDATE...
> I just tried a second 'mv' and it was near instantaneous.
> (and similarly with subsequent renaming of the same file)
> So perhaps not a 'Cygwin' thing but something going on within Windows.

> Could it be that the first 'mv' triggered an anti-virus read of the file since
> perhaps it detects it as a new/changed file?

> But if so, would 'mv' (under Task Manager) be showing the 100+ MB/s disk activity?

Install processmanager and check what actual activity is happening on that
file. Is it reading or writing? Who is doing that?


-- 
With best regards,
Andrey Repin
Tuesday, February 1, 2022 11:47:15

Sorry for my terrible english...