find command seems to lock files

find command seems to lock files

[Morten Kjærulff]

Hi,

I have an application which constantly:
creates a file
do some processing
deletes the file

One way to monitor if the application has crashed, is to check the age of
the file, so I made a script that:

find //$server/d$/dir/subdir*/subsubdir -name 'thefile' -printf '%A+\n'

subdir* will be subdir1 subdir2 ...
under subsubdir there will be dirA, dirB, ... and under those, thefile may
exist.

Problem is that it seems this command locks thefile, as the application
sometimes can't delete it.

Could this be true?

/Morten

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: find command seems to lock files

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

On Aug 19 14:33, Morten Kjærulff wrote:
> Hi,
> 
> I have an application which constantly:
> creates a file
> do some processing
> deletes the file
> 
> One way to monitor if the application has crashed, is to check the age of
> the file, so I made a script that:
> 
> find //$server/d$/dir/subdir*/subsubdir -name 'thefile' -printf '%A+\n'
> 
> subdir* will be subdir1 subdir2 ...
> under subsubdir there will be dirA, dirB, ... and under those, thefile may
> exist.
> 
> Problem is that it seems this command locks thefile, as the application
> sometimes can't delete it.
> 
> Could this be true?

Cygwin does not actually lock anything except in very rare
circumstances.  Your problem is more likely triggered by a realtime
virus scanner.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: find command seems to lock files

[Eliot Moss]

On 8/19/2019 10:03 AM, Corinna Vinschen wrote:
> On Aug 19 14:33, Morten Kjærulff wrote:
>> Hi,
>>
>> I have an application which constantly:
>> creates a file
>> do some processing
>> deletes the file
>>
>> One way to monitor if the application has crashed, is to check the age of
>> the file, so I made a script that:
>>
>> find //$server/d$/dir/subdir*/subsubdir -name 'thefile' -printf '%A+\n'
>>
>> subdir* will be subdir1 subdir2 ...
>> under subsubdir there will be dirA, dirB, ... and under those, thefile may
>> exist.
>>
>> Problem is that it seems this command locks thefile, as the application
>> sometimes can't delete it.
>>
>> Could this be true?
> 
> Cygwin does not actually lock anything except in very rare
> circumstances.  Your problem is more likely triggered by a realtime
> virus scanner.

I was wondering, though, whether the parent directory would
be non-delete-able while find has the directory open for scanning.
If the application in question creates and deletes the parent
directory, as well as the leaf file, then things would be left
around unexpectedly.

So would use of find trigger a virus scanner, which in turn might
hold on to the file and prevent its deletion?

Regards - Eliot

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: find command seems to lock files

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1854 bytes --]

On Aug 19 10:06, Eliot Moss wrote:
> On 8/19/2019 10:03 AM, Corinna Vinschen wrote:
> > On Aug 19 14:33, Morten Kjærulff wrote:
> > > Hi,
> > > 
> > > I have an application which constantly:
> > > creates a file
> > > do some processing
> > > deletes the file
> > > 
> > > One way to monitor if the application has crashed, is to check the age of
> > > the file, so I made a script that:
> > > 
> > > find //$server/d$/dir/subdir*/subsubdir -name 'thefile' -printf '%A+\n'
> > > 
> > > subdir* will be subdir1 subdir2 ...
> > > under subsubdir there will be dirA, dirB, ... and under those, thefile may
> > > exist.
> > > 
> > > Problem is that it seems this command locks thefile, as the application
> > > sometimes can't delete it.
> > > 
> > > Could this be true?
> > 
> > Cygwin does not actually lock anything except in very rare
> > circumstances.  Your problem is more likely triggered by a realtime
> > virus scanner.
> 
> I was wondering, though, whether the parent directory would
> be non-delete-able while find has the directory open for scanning.

Usually yes.  Cygwin moves the entire directory into the recycler in
case it's a local dir.  That works even if a file is blocking the
dir from deletion.

> If the application in question creates and deletes the parent
> directory, as well as the leaf file, then things would be left
> around unexpectedly.

The question was just if the file is locked.

> So would use of find trigger a virus scanner, which in turn might
> hold on to the file and prevent its deletion?

That's how some realtime scanners work.  They have hooks in the file API
and if some other process opens a file these scanners open the file as
well, typically without FILE_SHARE_DELETE, which Cygwin uses by default.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: find command seems to lock files

[Morten Kjærulff]

On Mon, Aug 19, 2019 at 4:13 PM Corinna Vinschen
<corinna-cygwin@cygwin.com> wrote:
>
> On Aug 19 10:06, Eliot Moss wrote:
> > On 8/19/2019 10:03 AM, Corinna Vinschen wrote:
> > > On Aug 19 14:33, Morten Kjærulff wrote:
> > > > Hi,
> > > >
> > > > I have an application which constantly:
> > > > creates a file
> > > > do some processing
> > > > deletes the file
> > > >
> > > > One way to monitor if the application has crashed, is to check the age of
> > > > the file, so I made a script that:
> > > >
> > > > find //$server/d$/dir/subdir*/subsubdir -name 'thefile' -printf '%A+\n'
> > > >
> > > > subdir* will be subdir1 subdir2 ...
> > > > under subsubdir there will be dirA, dirB, ... and under those, thefile may
> > > > exist.
> > > >
> > > > Problem is that it seems this command locks thefile, as the application
> > > > sometimes can't delete it.
> > > >
> > > > Could this be true?
> > >
> > > Cygwin does not actually lock anything except in very rare
> > > circumstances.  Your problem is more likely triggered by a realtime
> > > virus scanner.
> >
> > I was wondering, though, whether the parent directory would
> > be non-delete-able while find has the directory open for scanning.
>
> Usually yes.  Cygwin moves the entire directory into the recycler in
> case it's a local dir.  That works even if a file is blocking the
> dir from deletion.
>
> > If the application in question creates and deletes the parent
> > directory, as well as the leaf file, then things would be left
> > around unexpectedly.
>
> The question was just if the file is locked.
>
> > So would use of find trigger a virus scanner, which in turn might
> > hold on to the file and prevent its deletion?
>
> That's how some realtime scanners work.  They have hooks in the file API
> and if some other process opens a file these scanners open the file as
> well, typically without FILE_SHARE_DELETE, which Cygwin uses by default.
>
>
> Corinna
>
> --
> Corinna Vinschen
> Cygwin Maintainer

I forgot to say that I run the find command on my own PC, and the
application runs on a server, which I have 'net use' its disk.

Would it be the virus scanner on my PC or on the server?
Any idea of a different way to get the age of the file? (I am sure I
cannot change the virus scanner).

/Morten

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: find command seems to lock files

[Eliot Moss]

On 8/19/2019 10:26 AM, Morten Kjærulff wrote:

> I forgot to say that I run the find command on my own PC, and the
> application runs on a server, which I have 'net use' its disk.
> 
> Would it be the virus scanner on my PC or on the server?
> Any idea of a different way to get the age of the file? (I am sure I
> cannot change the virus scanner).
> 
> /Morten

I wonder if you would get different / better behavior by
writing a script / app to run on the server, which you
request from your PC.  For example, I had an issue with
my server holding too many open IMAP connections and
then my PC's Thunderbird would block.  I invoke a remote
command sequence with ssh, that finds the imap processes
in question and kills them off.

Regards - EM

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: find command seems to lock files

[Andrey Repin]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 2021 bytes --]

Greetings, Morten Kjærulff!

>>
>> > If the application in question creates and deletes the parent
>> > directory, as well as the leaf file, then things would be left
>> > around unexpectedly.
>>
>> The question was just if the file is locked.
>>
>> > So would use of find trigger a virus scanner, which in turn might
>> > hold on to the file and prevent its deletion?
>>
>> That's how some realtime scanners work.  They have hooks in the file API
>> and if some other process opens a file these scanners open the file as
>> well, typically without FILE_SHARE_DELETE, which Cygwin uses by default.
>>
>>
>> Corinna
>>
>> --
>> Corinna Vinschen
>> Cygwin Maintainer

> I forgot to say that I run the find command on my own PC, and the
> application runs on a server, which I have 'net use' its disk.

> Would it be the virus scanner on my PC or on the server?
> Any idea of a different way to get the age of the file? (I am sure I
> cannot change the virus scanner).

To begin with, the results of your `find` calls will be at least 5 seconds
stale over CIFS share with default settings.
Said that, you could safely write something like

find /xx -type а -iname "zzz" -mtime +10s -execdir 'msg * "Achtung programme crash boom!"'

In regard to antivirus,

1. first make sure your local AV does not scan network directories by default.
This is a gigantic usability issue for multiple reasons and normally is never done.
2. if possible, check what exactly happens when your program loсks up.
Yes, as said above, find will lock /directories/ it is scanning.
This is how Windows filesystem API works.
But it should not lock files by itself.


-- 
With best regards,
Andrey Repin
Monday, August 19, 2019 19:36:25

Sorry for my terrible english...\x03B‹KCB”\x1c›Ø›\x19[H\x1c™\^[ܝ\x1cΈ\b\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÜ\x1c›Ø›\x19[\Ëš\x1d^[[\x03B‘TNˆ\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÙ˜\KÃB‘^[ØÝ[Y[\x18]\x1a[ÛŽˆ\b\b\b\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÙ^[ØÜËš\x1d^[[\x03B•[œÝXœØÜšX™H\x1a[™›Îˆ\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÛ[\vÈÝ[œÝXœØÜšX™K\Ú[\^[\x19CBƒB

Re: find command seems to lock files

[Andrey Repin]

Greetings, Andrey Repin!


> find /xx -type а -iname "zzz" -mtime +10s -execdir 'msg * "Achtung programme crash boom!"'

-type f

//me sorry


-- 
With best regards,
Andrey Repin
Monday, August 19, 2019 19:51:10

Sorry for my terrible english...
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: find command seems to lock files

[Morten Kjærulff]

On Mon, Aug 19, 2019 at 6:50 PM Andrey Repin <anrdaemon@yandex.ru> wrote:
>
> Greetings, Morten Kjærulff!
>
> >>
> >> > If the application in question creates and deletes the parent
> >> > directory, as well as the leaf file, then things would be left
> >> > around unexpectedly.
> >>
> >> The question was just if the file is locked.
> >>
> >> > So would use of find trigger a virus scanner, which in turn might
> >> > hold on to the file and prevent its deletion?
> >>
> >> That's how some realtime scanners work.  They have hooks in the file API
> >> and if some other process opens a file these scanners open the file as
> >> well, typically without FILE_SHARE_DELETE, which Cygwin uses by default.
> >>
> >>
> >> Corinna
> >>
> >> --
> >> Corinna Vinschen
> >> Cygwin Maintainer
>
> > I forgot to say that I run the find command on my own PC, and the
> > application runs on a server, which I have 'net use' its disk.
>
> > Would it be the virus scanner on my PC or on the server?
> > Any idea of a different way to get the age of the file? (I am sure I
> > cannot change the virus scanner).
>
> To begin with, the results of your `find` calls will be at least 5 seconds
> stale over CIFS share with default settings.
> Said that, you could safely write something like
>
> find /xx -type а -iname "zzz" -mtime +10s -execdir 'msg * "Achtung programme crash boom!"'
>
> In regard to antivirus,
>
> 1. first make sure your local AV does not scan network directories by default.
> This is a gigantic usability issue for multiple reasons and normally is never done.
> 2. if possible, check what exactly happens when your program loсks up.
> Yes, as said above, find will lock /directories/ it is scanning.
> This is how Windows filesystem API works.
> But it should not lock files by itself.
>
>
> --
> With best regards,
> Andrey Repin
> Monday, August 19, 2019 19:36:25
>
> Sorry for my terrible english...

Thanks.

I guess that the reason find opens the file (and thereby trigger
antivirus) is because I print the files timestamp (-printf '%A+\n'),
right?
If I just printed the filename, the file would not be opened, right?

Will
find /xx -type f -iname "zzz" -mtime +10s -execdir 'msg * "Achtung
programme crash boom!"'
not open the file to get the timestamp?

/Morten

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: find command seems to lock files

[Eliot Moss]

On 8/19/2019 1:21 PM, Morten Kjærulff wrote:
 > I guess that the reason find opens the file (and thereby trigger
 > antivirus) is because I print the files timestamp (-printf '%A+\n'),
 > right?
 > If I just printed the filename, the file would not be opened, right?
I wouldn't say it _opens_ the file, but I read through the Posix man
page for stat/lstat/fstatat and it reveals that those calls update
the last access time in the inode, so that access the file "more"
than just getting some information from the directory.

 > Will
 > find /xx -type f -iname "zzz" -mtime +10s -execdir 'msg * "Achtung
 > programme crash boom!"'
 > not open the file to get the timestamp?

It still needs to look at the timestamps, and presumably
needs stat to do that.  All you can get from the directory
proper is the inode number and the name, I believe.  That is
not even enough for find to know whether it has to recurse
into the thing (is it a directory or not?), so I suspect
that find always does a stat call on every entry.

Regards - EM

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: find command seems to lock files

[Greywolf]

Greets, Andrey!

> To begin with, the results of your `find` calls will be at least 5 seconds
> stale over CIFS share with default settings.
> Said that, you could safely write something like
> 
> find /xx -type а -iname "zzz" -mtime +10s -execdir 'msg * "Achtung programme crash boom!"'

Thank you for the kaffe spit-take.  You just made my day!

				--*greywolf;


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: find command seems to lock files

[L A Walsh]

On 2019/08/19 09:42, Andrey Repin wrote:
> In regard to antivirus,
> 1. first make sure your local AV does not scan network directories by default.
> This is a gigantic usability issue for multiple reasons and normally is never done.
>   
Except by microsoft, of course.

Microsoft's malware scanner (home essentials) scans network
drives as well as local by *default*.  I had to make sure to list
any network drives in the exclusions section (which it seems to honor).
Sorta odd how their builtin OS 'search' command cannot index remote
drives**, but their virus scanner, by default, is able to do so.

**-can only use the remote server's index via its 'federated search
capability' which delegates the search of the remote file system to
the remote search program -- of course its only available on MS systems,
so indices of non-MS, network attached storage (NAS) isn't available.





--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: find command seems to lock files

[Chris Wagner]

On 2019-08-19 8:33 am, Morten Kjærulff wrote:
> find //$server/d$/dir/subdir*/subsubdir -name 'thefile' -printf '%A+\n'
> 
> Problem is that it seems this command locks thefile, as the application
> sometimes can't delete it.

Unfortunately, yes.  On Windows, a "full stat" requires that the file be 
opened.

This is why Perl has the ${^WIN32_SLOPPY_STAT} flag variable.

https://metacpan.org/pod/distribution/perl/pod/perlvar.pod#${^WIN32_SLOPPY_STAT}
> ${^WIN32_SLOPPY_STAT}
> 
> If this variable is set to a true value, then stat() on Windows will 
> not try to open the file. This means that the link count cannot be 
> determined and file attributes may be out of date if additional 
> hardlinks to the file exist. On the other hand, not opening the file is 
> considerably faster, especially for files on network drives.
> 


-Chris

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple