Re: sqlite defect

Re: sqlite defect

[David Rothenberger]

These kinds of questions belong on the main mailing list. Responding
there.

fger555@gmx.de wrote:
> I never had problems with sqlite before.
>  
> But since 3.7.17-3 I cannot read my databases any more
> "unable to open datase file". I remember there was a discussion
> on the cygwin general mailing list about this problem
> (backslashes in path?).

Please provide more specifics about the error and how to cause it. I
use SVN on a daily basis in both 32-bit and 64-bit Cygwin and have
no troubles whatsoever.

You should also read and follow the instructions here:
http://cygwin.com/problems.html.

> Is there a solution for this bug.
> Or could you switch back to the working version again,
> with all the dependencies, as a workaround?

I will not be re-rolling SVN against a prior SQLite release. We'll
have to figure out what's causing the problem in your environment.

-- 
David Rothenberger  ----  daveroth@acm.org

English literature's performing flea.
                -- Sean O'Casey on P. G. Wodehouse

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[Warren Young]

On 7/17/2013 17:15, fger555@gmx.de wrote:
> Von: "Warren Young" <warren@etr-usa.com>
>> try setting the new CYGWIN_SQLITE_LOCKING environment
>> variable to "posix".
>
> Why isn't it the default?

Because there are more native Windows programs based on SQLite than 
there are Cygwin programs based on SQLite.

Since we cannot interoperate with both sets of programs at the same 
time, we chose to interoperate with the larger set by default.

> I copied the mozilla places.sqlite bookmarks file to a backup directory

This at least gives me a way to reproduce it.

For anyone else who cares:

$ cp ~/AppData/Roaming/Mozilla/Firefox/Profiles/*/places.sqlite /tmp

(~/AppData is a symlink to the obvious location on my machine.)

$ sqlite3 /tmp/places.sqlite .schema

The query string can be anything that causes it to access the DB file. 
So, it can't just be ".quit", for example.

I've done some strace diffing, and the problem seems to happen outside 
the Cygwin DLL, immediately after the new locking code is called:

> read: 32768 = read(3, 0x80054620, 32768)
> flock: 0 = flock(3, 8)
> fhandler_pty_slave::write: pty0, write(0x40A502, 7)
> fhandler_pty_slave::write: (654): pty output_mutex (0xBC): waiting -1 ms
> fhandler_pty_slave::write: (654): pty output_mutex: acquired
> fhandler_pty_slave::write: (669): pty output_mutex(0xBC) released
> write: 7 = write(2, 0x40A502, 7)
> fhandler_pty_slave::write: pty0, write(0x80053FF0, 28)
> fhandler_pty_slave::write: (654): pty output_mutex (0xBC): waiting -1 ms
> fhandler_pty_slave::write: (654): pty output_mutex: acquired
> fhandler_pty_slave::write: (669): pty output_mutex(0xBC) released
> write: 28 = write(2, 0x80053FF0, 28)
> fhandler_pty_slave::write: pty0, write(0x40A50B, 1)
> fhandler_pty_slave::write: (654): pty output_mutex (0xBC): waiting -1 ms
> fhandler_pty_slave::write: (654): pty output_mutex: acquired
> fhandler_pty_slave::write: (669): pty output_mutex(0xBC) released
> write: 1 = write(2, 0x40A50B, 1)

Distilled, what this says is that it successfully read()s from the DB 
file it has open (fd=3), then it successfully flock()s it, and then for 
reasons I don't understand, does 3 write()s to stderr.

The mystery is what happens between lines 2 and 3, and for that I don't 
see anything for it but to dive back into the SQLite code.

> Does the sqlite file contain a locking flag inside?

Nothing so simple.  Locking is handled at the OS and/or Cygwin DLL 
level.  The build change between 3.7.16.2 and 3.7.17-3 is that we're now 
relying on new features in the Cygwin DLL to do Windows-style locking by 
default.

Older versions of Cygwin SQLite bypassed the Cygwin DLL entirely for 
this, going straight to the Win32 API, thereby preventing the DLL from 
interposing itself for the "posix" case.

> With the old version
> I never had a problem to access the files.

Yes, well, other people did have trouble with the old version, which is 
why we tried giving locking responsibility to the Cygwin DLL.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[jojelino]

On 2013-07-18 AM 10:53, Warren Young wrote:
> Nothing so simple.  Locking is handled at the OS and/or Cygwin DLL
> level.  The build change between 3.7.16.2 and 3.7.17-3 is that we're now
> relying on new features in the Cygwin DLL to do Windows-style locking by
> default.
>
> Older versions of Cygwin SQLite bypassed the Cygwin DLL entirely for
> this, going straight to the Win32 API, thereby preventing the DLL from
> interposing itself for the "posix" case.
>
Mandatory locking feature of cygwin used in sqlite is broken.
Just build plain sqlite build without the cygport patch that led the 
disaster. unless you have familiar with exotic adventure like finding 
undocumented behavior of windows wasting many man-hours to workaround 
the problem.

-- 
Regards.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[Corinna Vinschen]

On Jul 18 17:11, jojelino wrote:
> On 2013-07-18 AM 10:53, Warren Young wrote:
> >Nothing so simple.  Locking is handled at the OS and/or Cygwin DLL
> >level.  The build change between 3.7.16.2 and 3.7.17-3 is that we're now
> >relying on new features in the Cygwin DLL to do Windows-style locking by
> >default.
> >
> >Older versions of Cygwin SQLite bypassed the Cygwin DLL entirely for
> >this, going straight to the Win32 API, thereby preventing the DLL from
> >interposing itself for the "posix" case.
> >
> Mandatory locking feature of cygwin used in sqlite is broken.

Simple testcase in plain C?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[jojelino]

On 2013-07-18 PM 5:59, Corinna Vinschen wrote:> On Jul 18 17:11, 
jojelino wrote:
 >> On 2013-07-18 AM 10:53, Warren Young wrote:
 >>> Nothing so simple.  Locking is handled at the OS and/or Cygwin DLL
 >>> level.  The build change between 3.7.16.2 and 3.7.17-3 is that 
we're now
 >>> relying on new features in the Cygwin DLL to do Windows-style 
locking by
 >>> default.
 >>>
 >>> Older versions of Cygwin SQLite bypassed the Cygwin DLL entirely for
 >>> this, going straight to the Win32 API, thereby preventing the DLL from
 >>> interposing itself for the "posix" case.
 >>>
 >> Mandatory locking feature of cygwin used in sqlite is broken.
 >
 > Simple testcase in plain C?
 >
 >
 > Corinna
 >
#include <stdio.h>
#include <fcntl.h>
#include <assert.h>
#include <string.h>
#include <sys/file.h>
int
main()
{
   int fd = open("asdf.txt", O_BINARY | O_RDWR | O_NOINHERIT | O_CREAT);
   const char* buf = 
"KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK\0";
   int i=0;
   write(fd, buf, strlen(buf));
   fcntl(fd,F_LCK_MANDATORY,0);
   assert(flock(fd,F_SETOWN)==0);
   assert(flock(fd,F_SETOWN)==0);
   assert(flock(fd,LOCK_NB|LOCK_UN)==0);
   fcntl(fd,F_LCK_MANDATORY,1);
   assert(flock(fd,F_SETOWN)==0);
   assert(flock(fd,LOCK_NB|LOCK_UN)==0);
   assert(flock(fd,F_SETOWN)==0);
   assert(flock(fd,F_SETOWN)==0); /* assertion "flock(fd,F_SETOWN)==0" 
failed: file "f_lck.c", line 28, function: main */
   return 0;
}

/**/
-- 
Regards.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[Corinna Vinschen]

On Jul 19 10:56, jojelino wrote:
> On 2013-07-18 PM 5:59, Corinna Vinschen wrote:> On Jul 18 17:11,
> jojelino wrote:
> >> On 2013-07-18 AM 10:53, Warren Young wrote:
> >>> Nothing so simple.  Locking is handled at the OS and/or Cygwin DLL
> >>> level.  The build change between 3.7.16.2 and 3.7.17-3 is that
> we're now
> >>> relying on new features in the Cygwin DLL to do Windows-style
> locking by
> >>> default.
> >>>
> >>> Older versions of Cygwin SQLite bypassed the Cygwin DLL entirely for
> >>> this, going straight to the Win32 API, thereby preventing the DLL from
> >>> interposing itself for the "posix" case.
> >>>
> >> Mandatory locking feature of cygwin used in sqlite is broken.
> >
> > Simple testcase in plain C?

Thanks, but...

> #include <stdio.h>
> #include <fcntl.h>
> #include <assert.h>
> #include <string.h>
> #include <sys/file.h>
> int
> main()
> {
>   int fd = open("asdf.txt", O_BINARY | O_RDWR | O_NOINHERIT | O_CREAT);
>   const char* buf =
> "KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK\0";
>   int i=0;
>   write(fd, buf, strlen(buf));
>   fcntl(fd,F_LCK_MANDATORY,0);
>   assert(flock(fd,F_SETOWN)==0);
                    ^^^^^^^^
                    F_SETOWN is no valid flag for flock(2).

Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[Corinna Vinschen]

On Jul 19 12:03, Corinna Vinschen wrote:
> On Jul 19 10:56, jojelino wrote:
> > On 2013-07-18 PM 5:59, Corinna Vinschen wrote:> On Jul 18 17:11,
> > jojelino wrote:
> > >> On 2013-07-18 AM 10:53, Warren Young wrote:
> > >>> Nothing so simple.  Locking is handled at the OS and/or Cygwin DLL
> > >>> level.  The build change between 3.7.16.2 and 3.7.17-3 is that
> > we're now
> > >>> relying on new features in the Cygwin DLL to do Windows-style
> > locking by
> > >>> default.
> > >>>
> > >>> Older versions of Cygwin SQLite bypassed the Cygwin DLL entirely for
> > >>> this, going straight to the Win32 API, thereby preventing the DLL from
> > >>> interposing itself for the "posix" case.
> > >>>
> > >> Mandatory locking feature of cygwin used in sqlite is broken.
> > >
> > > Simple testcase in plain C?
> 
> Thanks, but...
> 
> > #include <stdio.h>
> > #include <fcntl.h>
> > #include <assert.h>
> > #include <string.h>
> > #include <sys/file.h>
> > int
> > main()
> > {
> >   int fd = open("asdf.txt", O_BINARY | O_RDWR | O_NOINHERIT | O_CREAT);
> >   const char* buf =
> > "KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK\0";
> >   int i=0;
> >   write(fd, buf, strlen(buf));
> >   fcntl(fd,F_LCK_MANDATORY,0);
> >   assert(flock(fd,F_SETOWN)==0);
>                     ^^^^^^^^
>                     F_SETOWN is no valid flag for flock(2).

...and, btw., fcntl(fd, F_SETOWN) is only supported on sockets.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[jojelino]

On 2013-07-19 PM 7:08, Corinna Vinschen wrote:
 > ...and, btw., fcntl(fd, F_SETOWN) is only supported on sockets.
 >
 >
 > Corinna
 >
28 55977555 [main] python 5616 open: 9 = 
open(/home/Administrator/....../..............sqlite, 0x58202)
..
..
  4680 55982813 [main] python 5616 flock: 0 = flock(9, 6)
..
..
    35 55983687 [main] python 5616 seterrno_from_nt_status: 
/netrel/src/cygwin-snapshot-20130619-1/winsup/cygwin/flock.cc:2020 
status 0xC0000055 -> windows error 33
    26 55983713 [main] python 5616 geterrno_from_win_error: windows 
error 33 == errno 16
    24 55983737 [main] python 5616 flock: -1 = flock(9, 6), errno 16

I tried reproducing above failure.
-- 
Regards.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[Corinna Vinschen]

On Jul 19 19:28, jojelino wrote:
> On 2013-07-19 PM 7:08, Corinna Vinschen wrote:
> > ...and, btw., fcntl(fd, F_SETOWN) is only supported on sockets.
> >
> >
> > Corinna
> >
> 28 55977555 [main] python 5616 open: 9 =
> open(/home/Administrator/....../..............sqlite, 0x58202)
> ..
> ..
>  4680 55982813 [main] python 5616 flock: 0 = flock(9, 6)
> ..
> ..
>    35 55983687 [main] python 5616 seterrno_from_nt_status:
> /netrel/src/cygwin-snapshot-20130619-1/winsup/cygwin/flock.cc:2020
> status 0xC0000055 -> windows error 33
>    26 55983713 [main] python 5616 geterrno_from_win_error: windows
> error 33 == errno 16
>    24 55983737 [main] python 5616 flock: -1 = flock(9, 6), errno 16
> 
> I tried reproducing above failure.

A valid testcase would help a lot, though.  What you meant to do
was calling flock with LOCK_EX|LOCK_NB.

And then again, your testcase works as designed.  Not by me, but by
Microsoft.  You can't overwrite an existing lock, even if hold by the
same file handle.  See http://cygwin.com/cygwin-api/std-notes.html


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[jojelino]

On 2013-07-19 PM 8:30, Corinna Vinschen wrote:
> A valid testcase would help a lot, though.  What you meant to do
> was calling flock with LOCK_EX|LOCK_NB.
>
that's what exactly sqlite3 that uses the mandatory-locking did. 
reproducing the behavior was i meant to do.
> And then again, your testcase works as designed.  Not by me, but by
> Microsoft.  You can't overwrite an existing lock, even if hold by the
> same file handle.  See http://cygwin.com/cygwin-api/std-notes.html
>
Yes. the testcase works without mandatory locking. so i hope next 
sqlite3 release doesn't use mandatory locking feature of cygwin. someone 
who have plenty of time to waste digging into sqlite3 source code would 
come with workaround to the problem.
>
> Corinna
>


-- 
Regards.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[Corinna Vinschen]

On Jul 19 20:53, jojelino wrote:
> On 2013-07-19 PM 8:30, Corinna Vinschen wrote:
> >A valid testcase would help a lot, though.  What you meant to do
> >was calling flock with LOCK_EX|LOCK_NB.
> >
> that's what exactly sqlite3 that uses the mandatory-locking did.
> reproducing the behavior was i meant to do.
> >And then again, your testcase works as designed.  Not by me, but by
> >Microsoft.  You can't overwrite an existing lock, even if hold by the
> >same file handle.  See http://cygwin.com/cygwin-api/std-notes.html
> >
> Yes. the testcase works without mandatory locking. so i hope next
> sqlite3 release doesn't use mandatory locking feature of cygwin.
> someone who have plenty of time to waste digging into sqlite3 source
> code would come with workaround to the problem.

There *is* a workaround:

  export CYGWIN_SQLITE_LOCKING=posix

See http://cygwin.com/ml/cygwin-announce/2013-06/msg00014.html


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[jojelino]

On 2013-07-19 PM 9:02, Corinna Vinschen wrote:
> There *is* a workaround:
>
>    export CYGWIN_SQLITE_LOCKING=posix
>
> See http://cygwin.com/ml/cygwin-announce/2013-06/msg00014.html
>
It's very good workaround instead of rebuild sqlite3. thanks for the point.
-- 
Regards.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[jojelino]

On 2013-07-19 PM 9:08, jojelino wrote:
> On 2013-07-19 PM 9:02, Corinna Vinschen wrote:
>> There *is* a workaround:
>>
>>    export CYGWIN_SQLITE_LOCKING=posix
>>
>> See http://cygwin.com/ml/cygwin-announce/2013-06/msg00014.html
>>
> It's very good workaround instead of rebuild sqlite3. thanks for the point.
Also, I commented out F_LCK_MANDATORY in fcntl.h to keep from 
confronting the disastrous situation introduced by this experimental 
feature. I expect it would save many hours instead of digging into 
unmaintained source codes. especially for projects that uses the sort of 
autotools which just checks the existence of F_LCK_MANDATORY macro. and 
is optimistic about the experimental feature.
-- 
Regards.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sqlite defect

[Corinna Vinschen]

On Jul 19 21:36, jojelino wrote:
> On 2013-07-19 PM 9:08, jojelino wrote:
> >On 2013-07-19 PM 9:02, Corinna Vinschen wrote:
> >>There *is* a workaround:
> >>
> >>   export CYGWIN_SQLITE_LOCKING=posix
> >>
> >>See http://cygwin.com/ml/cygwin-announce/2013-06/msg00014.html
> >>
> >It's very good workaround instead of rebuild sqlite3. thanks for the point.
> Also, I commented out F_LCK_MANDATORY in fcntl.h to keep from
> confronting the disastrous situation introduced by this experimental
> feature. I expect it would save many hours instead of digging into
> unmaintained source codes. especially for projects that uses the
> sort of autotools which just checks the existence of F_LCK_MANDATORY
> macro. and is optimistic about the experimental feature.

This is over the top.  F_LCK_MANDATORY is a Cygwin invention not known
to the outer world, *and* the behaviour of F_LCK_MANDATORY is
documented.  If any project actually chooses to use it, it will do so
with the full information about the semantics given.  Also, since
mandatory locking is not standard conformant anyway, the project
has to know exactly what it's doing.  So what's your point?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple