Unreliable flock

Unreliable flock

[Andrey Repin]

Greetings, All!

The script (let's call it test.sh):

#!/bin/dash -x
_lock="./console-session.lock"
{
  flock -n 9 || {
      echo "The $(cat "$_lock") command is running already."
      exit 3
    } >&2

  printf "$1" >&9

  trap 'rm "$_lock";' EXIT HUP INT ABRT TERM

  sleep 20

} 9>> "$_lock"

Very simple in essence.
The one-liner:

{ flock -n 9 && sleep 10 || echo Fail;} 9>> ./console-session.lock

Also extremely simple.

Open two terminals. mintty or native, bash or dash, doesn't matter.
chdir to the directory with the script.

Start script in one terminal. ./test.sh
Start oneliner in another terminal. Fail. How predictable!

Wait till both expire.
Now start oneliner first.
Then script.
Both succeed. Not 100% but highly, highly probable. Sometimes it even work
both ways.

At the same time, I can't reproduce it on Linux at all.
But I do can reproduce it on Cygwin. Both 32 and 64 bit. With both dash and
bash. With both local and remote FS.

CYGWIN_NT-5.1 foserver 2.4.1(0.293/5/3) 2016-01-24 11:24 i686 Cygwin
CYGWIN_NT-6.1 daemon2 2.4.1(0.293/5/3) 2016-01-24 11:26 x86_64 Cygwin
CYGWIN_NT-6.1 buildbotw7 2.4.1(0.293/5/3) 2016-01-24 11:26 x86_64 Cygwin


-- 
With best regards,
Andrey Repin
Saturday, April 2, 2016 23:22:25

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Unreliable flock

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

On Apr  3 00:22, Andrey Repin wrote:
> Greetings, All!
> 
> The script (let's call it test.sh):
> 
> #!/bin/dash -x
> _lock="./console-session.lock"
> {
>   flock -n 9 || {
>       echo "The $(cat "$_lock") command is running already."
>       exit 3
>     } >&2
> 
>   printf "$1" >&9
> 
>   trap 'rm "$_lock";' EXIT HUP INT ABRT TERM
> 
>   sleep 20
> 
> } 9>> "$_lock"
> 
> Very simple in essence.
> The one-liner:
> 
> { flock -n 9 && sleep 10 || echo Fail;} 9>> ./console-session.lock
> 
> Also extremely simple.
> 
> Open two terminals. mintty or native, bash or dash, doesn't matter.
> chdir to the directory with the script.
> 
> Start script in one terminal. ./test.sh
> Start oneliner in another terminal. Fail. How predictable!

I don't know what I'm doing wrong, but this WJFFM with Cygwin 2.4.1
as well as 2.5.0.  Additionally I tried to put the oneliner into a
script but it still WFM.

Apart from that, did you notice the flock restrictions outlined in
https://cygwin.com/cygwin-api/std-notes.html ?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Unreliable flock

[Andrey Repin]

Greetings, Corinna Vinschen!

>> The script (let's call it test.sh):
>> 
>> #!/bin/dash -x
>> _lock="./console-session.lock"
>> {
>>   flock -n 9 || {
>>       echo "The $(cat "$_lock") command is running already."
>>       exit 3
>>     } >&2
>> 
>>   printf "$1" >&9
>> 
>>   trap 'rm "$_lock";' EXIT HUP INT ABRT TERM
>> 
>>   sleep 20
>> 
>> } 9>> "$_lock"
>> 
>> Very simple in essence.
>> The one-liner:
>> 
>> { flock -n 9 && sleep 10 || echo Fail;} 9>> ./console-session.lock
>> 
>> Also extremely simple.
>> 
>> Open two terminals. mintty or native, bash or dash, doesn't matter.
>> chdir to the directory with the script.
>> 
>> Start script in one terminal. ./test.sh
>> Start oneliner in another terminal. Fail. How predictable!

> I don't know what I'm doing wrong, but this WJFFM with Cygwin 2.4.1
> as well as 2.5.0.  Additionally I tried to put the oneliner into a
> script but it still WFM.

It works for me, too. From time to time. But this work is completely
unreliable. It may work five times, then fail ten times in a row.
Most often it fails, if I run oneliner first. Almost 100%.

> Apart from that, did you notice the flock restrictions outlined in
> https://cygwin.com/cygwin-api/std-notes.html ?

If you mean the part about

> BSD file locks created via flock are only propagated to the direct parent
> process, not to grand parents or sibling processes. The locks are only valid
> in the creating process, its parent process, and subsequently started child
> processes sharing the same file descriptor.

then that's a showstopper. In short, it makes the function literally useless.
I can work around it in a given script, but... *sad panda*

Why they aren't real locks? What's use for "advisory locks"? "I think I may
have a use for this file, but you are free to delete it, if you wish" ?


-- 
With best regards,
Andrey Repin
Monday, April 4, 2016 19:27:11

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Unreliable flock

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1748 bytes --]

On Apr  4 19:51, Andrey Repin wrote:
> If you mean the part about
> 
> > BSD file locks created via flock are only propagated to the direct parent
> > process, not to grand parents or sibling processes. The locks are only valid
> > in the creating process, its parent process, and subsequently started child
> > processes sharing the same file descriptor.
> 
> then that's a showstopper. In short, it makes the function literally useless.
> I can work around it in a given script, but... *sad panda*

The problem with BSD locks is that the way they are implemented requires
a common shared datastructure in all child processes of a common
ancestor process which opened the file.  This isn't available in Cygwin.
Implementing this is tricky and potentially costly so I only got around
to implement this for the direct ancestor process.  I'm open to
suggestions or even patches to make this work in full.  Have a look into
Cygwin's flock.cc to see how locks are implemented.  Feel free to discuss
implementation details on the cygwin-developers ML.

Note that the same restrictions don't apply to POSIX locks since they
are working quite differently.

> Why they aren't real locks? What's use for "advisory locks"? "I think I may
> have a use for this file, but you are free to delete it, if you wish" ?

Unix systems usually implement file locking advisory, not mandatory.
Windows only implements mandatory locks which are incompatible in
behaviour with POSIX as well as BSD locks.  I guess you read that
there's a way to use Windows mandatory locks, too.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Unreliable flock

[Warren Young]

On Apr 4, 2016, at 10:51 AM, Andrey Repin <anrdaemon@yandex.ru> wrote:
> 
>> BSD file locks created via flock are only propagated to the direct parent
> 
> that's a showstopper. In short, it makes the function literally useless.

Nonsense.  That’s only true if “literally” every program that uses BSD locks creates grandchildren that also need to use those same locks.

I know of one program for certain that uses BSD locks under Cygwin that doesn’t create grandchildren, and its extensive test suite passed with Cygwin/BSD locks the last time I ran it.

> Why they aren't real locks? What's use for "advisory locks”?

“Real” locks are generally not what you want when running purely Unix/Linux software under Cygwin, because that’s not what you get by default on Linux or Unix.  You have to go out of your way to get mandatory locking on POSIX systems, as a rule.

Mandatory locks aren’t purely positive.  They’re the single biggest reason Windows still needs a reboot for many kinds of upgrades, while Linux generally only has to be rebooted for kernel or glibc upgrades, and the latter is, strictly speaking, optional.

> "I think I may
> have a use for this file, but you are free to delete it, if you wish” ?

Among cooperating processes, that’s perfectly fine.

Consider SQLite.  On a system where SQLite decides to use some form of advisory locking, the primary risk of damage isn’t rm or dd, it’s another copy of SQLite coming along and writing to the DB while the first is in the middle of a transaction,.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Unreliable flock

[Andrey Repin]

Greetings, Corinna Vinschen!

> I guess you read that
> there's a way to use Windows mandatory locks, too.

Yes, if I compile my own program. But I'm not. I'm writing a shell script.
And /usr/bin/flock fails to lock a file. Even though the man claims that it
"uses an exclusive lock by default".


-- 
With best regards,
Andrey Repin
Tuesday, April 5, 2016 05:06:19

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Unreliable flock

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

On Apr  5 05:08, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> > I guess you read that
> > there's a way to use Windows mandatory locks, too.
> 
> Yes, if I compile my own program. But I'm not. I'm writing a shell script.
> And /usr/bin/flock fails to lock a file. Even though the man claims that it
> "uses an exclusive lock by default".

exclusive != mandatory

See, e.g, http://man7.org/linux/man-pages/man2/fcntl.2.html


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]