Creating the sqlite3.exe stand-alone w/o cygwin dependency

Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Jose Isaias Cabrera]

Greetings!

Using these commands,

./configure --host=i686-w64-mingw32 CFLAGS=-shared CFLAGS=-static-libgcc
make sqlite3.exe sqlite3.dll

I can create the sqlite3 DOS application and the sqlite3 Windows 10 DLL. I have no problem with the DLL. But with the sqlite3.exe CLI tool, I can run it in the Cygwin environment, and I can also double click on the app from the Windows 10 environment and run it, and that also works perfectly. The problem happens after I move that sqlite3.exe to C:\bin directory. When I run it from the DOS Command Prompt, I get this error:

DOS|16:19:00.58>sqlite3
DOS|sqlite3: FATAL: Couldn't find sqlite3.

If I add .exe to the end,

DOS|16:19:57.46>sqlite3.exe
DOS|
DOS|16:21:33.36>

It appears as if it starts and does nothing. I tried chatting with SQLite folks, but, apparently, I am the only one that uses Cygwin. I have been using Cygwin for a long time, and I like it. Thoughts? Suggestions? Any help would be greatly appreciated.

PS: This is the make sqlite.exe run:

E608313@HOR711318E ~/b/sqlite/SQLite-78723a9a
$ make sqlite3.exe
./libtool --mode=link i686-w64-mingw32-gcc -shared -static-libgcc -DSQLITE_OS_WIN=1 -I. -I/home/E608313/b/sqlite/SQLite-78723a9a/src -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/rtree -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/icu -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/fts3 -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/async -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/session -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/userauth -D_HAVE_SQLITE_CONFIG_H -DBUILD_sqlite -DNDEBUG -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_MATH_FUNCTIONS     -DHAVE_READLINE=0  -DHAVE_EDITLINE=0 -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_EXPLAIN_COMMENTS -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION -DSQLITE_ENABLE_STMTVTAB -DSQLITE_ENABLE_DBPAGE_VTAB -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_BYTECODE_VTAB -DSQLITE_ENABLE_OFFSET_SQL_FUNC -o sqlite3.exe \
        shell.c sqlite3.c \
          -rpath "/usr/local/lib"
libtool: link: i686-w64-mingw32-gcc -shared -static-libgcc -DSQLITE_OS_WIN=1 -I. -I/home/E608313/b/sqlite/SQLite-78723a9a/src -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/rtree -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/icu -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/fts3 -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/async -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/session -I/home/E608313/b/sqlite/SQLite-78723a9a/ext/userauth -D_HAVE_SQLITE_CONFIG_H -DBUILD_sqlite -DNDEBUG -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_MATH_FUNCTIONS -DHAVE_READLINE=0 -DHAVE_EDITLINE=0 -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_EXPLAIN_COMMENTS -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION -DSQLITE_ENABLE_STMTVTAB -DSQLITE_ENABLE_DBPAGE_VTAB -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_BYTECODE_VTAB -DSQLITE_ENABLE_OFFSET_SQL_FUNC -o .libs/sqlite3.exe shell.c sqlite3.c  -L/usr/local/lib

Re: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Eliot Moss]

On 12/9/2022 4:34 PM, Jose Isaias Cabrera via Cygwin wrote:

This is a bit of a shot in the dark, but I wonder about
search paths and find the DLL when trying to start the
program.  Is the necessary directory on the search path
available to Windows and/or passed to the sqlite3
executable as an environment variable?

Regards - Eliot Moss

RE: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Jose Isaias Cabrera]

On Friday, December 9, 2022 6:33 PM, Eliot Moss  expressed:
> 
> On 12/9/2022 4:34 PM, Jose Isaias Cabrera via Cygwin wrote:
> 
> This is a bit of a shot in the dark, but I wonder about search paths and find
> the DLL when trying to start the program.  Is the necessary directory on the
> search path available to Windows and/or passed to the sqlite3 executable as
> an environment variable?

Thanks, Eliot. I have placed the sqlite3.dll in the same folder of the executable, but, it does not work either. I don't think is the sqlite3.dll that is looking for. It's probably some of the Cygwin libraries.

Any thoughts where I can get help?
 

Re: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Eliot Moss]

On 12/9/2022 9:43 PM, Jose Isaias Cabrera wrote:
> 
> On Friday, December 9, 2022 6:33 PM, Eliot Moss  expressed:
>>
>> On 12/9/2022 4:34 PM, Jose Isaias Cabrera via Cygwin wrote:
>>
>> This is a bit of a shot in the dark, but I wonder about search paths and find
>> the DLL when trying to start the program.  Is the necessary directory on the
>> search path available to Windows and/or passed to the sqlite3 executable as
>> an environment variable?
> 
> Thanks, Eliot. I have placed the sqlite3.dll in the same folder of the executable, but, it does not work either. I don't think is the sqlite3.dll that is looking for. It's probably some of the Cygwin libraries.
> 
> Any thoughts where I can get help?

This list continue to be a place.

I still wonder if there is some path issue.  Just because the dll is in
the same directory does not mean that Windows will look there if . (the
current directory) is not on your path.  As I recall (and I hope someone
will correct me!) the relevant environment variable is PATH, but it may
be a different one for finding dlls.  (I'm sure someone on the list can
clarify that point!)

Best - Eliot Moss

Re: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Cristobal Escamilla Cavazos]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

Actually if the dlls is in the same directory it does mean that windows
will look there even if the current directory is not in your path (always
true if the executable is in the same location, only true if safe dll
search is off) given that windows has not found the dll registered in the
system in any other place and there are no overrides anywhere else.

The search goes in this order:
If the dll is already in memory, this is used.
If the dll is in the list of known dlls versions it is loaded from there
(list of dlls is at HKLM\SYSTEM\CurrentControlSet\Control\Session
Manager\KnownDLLs)
If it is not found in any of those locations then it is searched
Finally dependencies are resolved using this steps as well

If dll has not been found and it is not overridden by a full path
definition nor a manifest then the search goes as follow
Directory from which the application was loaded
If safe dll search mode is disabled then the current directory is searched
The system directory - starting with the current arch, then fallbacks to
previous archs
The windows directory
And finally all the path environment variable

Hopes this helps a little bit.

Reference:
https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order

Best regards - Cristobal E

RE: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Jose Isaias Cabrera]

On December 10, 2022 9:48 AM, Eliot Moss  expressed:
> 
> On 12/9/2022 9:43 PM, Jose Isaias Cabrera wrote:
> >
> > On Friday, December 9, 2022 6:33 PM, Eliot Moss  expressed:
> >>
> >> On 12/9/2022 4:34 PM, Jose Isaias Cabrera via Cygwin wrote:
> >>
> >> This is a bit of a shot in the dark, but I wonder about search paths
> >> and find the DLL when trying to start the program.  Is the necessary
> >> directory on the search path available to Windows and/or passed to
> >> the sqlite3 executable as an environment variable?
> >
> > Thanks, Eliot. I have placed the sqlite3.dll in the same folder of the
> > executable, but, it does not work either. I don't think is the
> > sqlite3.dll that is looking for. It's probably some of the Cygwin
> > libraries.
> >
> > Any thoughts where I can get help?
> 
> This list continue to be a place.
> 
> I still wonder if there is some path issue.  Just because the dll is in
> the same directory does not mean that Windows will look there if . (the 
> current directory) is not on your path.  As I recall (and I hope someone
> will correct me!) the relevant environment variable is PATH, but it may
> be a different one for finding dlls.  (I'm sure someone on the list can
> clarify that point!)

I can tell you that in Windows 10 the search path is the live directory first (.), then the path. But, again, I don't think it's the DLL, because this application does not use the sqlite3.dll or needs it. It's a stand-alone application. It has all it needs inside. For example, if I download the Windows Tool from the Sqlite page, I can place it anywhere in the computer and it will run without anything else.

Re: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Ken Brown]

On 12/9/2022 4:34 PM, Jose Isaias Cabrera via Cygwin wrote:
> 
> Greetings!
> 
> Using these commands,
> 
> ./configure --host=i686-w64-mingw32 CFLAGS=-shared CFLAGS=-static-libgcc
> make sqlite3.exe sqlite3.dll
> 
> I can create the sqlite3 DOS application and the sqlite3 Windows 10 DLL. I have no problem with the DLL. But with the sqlite3.exe CLI tool, I can run it in the Cygwin environment, and I can also double click on the app from the Windows 10 environment and run it, and that also works perfectly. The problem happens after I move that sqlite3.exe to C:\bin directory. When I run it from the DOS Command Prompt, I get this error:
> 
> DOS|16:19:00.58>sqlite3
> DOS|sqlite3: FATAL: Couldn't find sqlite3.
> 
> If I add .exe to the end,
> 
> DOS|16:19:57.46>sqlite3.exe
> DOS|
> DOS|16:21:33.36>
> 
> It appears as if it starts and does nothing. I tried chatting with SQLite folks, but, apparently, I am the only one that uses Cygwin. I have been using Cygwin for a long time, and I like it. Thoughts? Suggestions? Any help would be greatly appreciated.
> 
> PS: This is the make sqlite.exe run:
> 
> E608313@HOR711318E ~/b/sqlite/SQLite-78723a9a
> $ make sqlite3.exe
> ./libtool --mode=link i686-w64-mingw32-gcc -shared -static-libgcc -DSQLITE_OS_WIN=1 -I. -I/home/E608313

I don't know if this is the problem, but do you really want to be using 
'-shared' when building an executable?  I thought that was for building shared 
libraries.

Ken

Re: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Michael Soegtrop]

 > The problem happens after I move that sqlite3.exe to C:\bin directory.

very likely quite a few of the MinGW shared libraries supplied by cygwin 
are missing - you need to copy these as well. You can use the `ldd` tool 
on the .exe and the .dll to find out which these are. You need to copy 
sqlite.dll and all MinGW shared libraries required by both to the bin 
folder. A typical example is the C runtime shared library-

I have no idea about sqlite, but I maintain a project which compiles 
MinGW executables using shared libraries and create relocatable 
installers from that. The shell script which finds the shared library 
dependencies is here:

https://github.com/coq/platform/blob/cb1646593beb42629d660e22323a3fe9e3779435/windows/create_installer_windows.sh#L107

It should explain how to do this.

Best regards,

Michael

RE: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Jose Isaias Cabrera]

On December 10, 2022 10:20 AM, Michael Soegtrop expressed:
> 
> > The problem happens after I move that sqlite3.exe to C:\bin directory.
> 
> very likely quite a few of the MinGW shared libraries supplied by cygwin are
> missing - you need to copy these as well. You can use the `ldd` tool on the

Very interesting:
$ ldd sqlite3.exe
        ntdll.dll => /cygdrive/c/Windows/SYSTEM32/ntdll.dll (0x7ffc1d6f0000)
        ntdll.dll => /cygdrive/c/Windows/SysWOW64/ntdll.dll (0x778c0000)
        wow64.dll => /cygdrive/c/Windows/System32/wow64.dll (0x7ffc1c830000)
        wow64win.dll => /cygdrive/c/Windows/System32/wow64win.dll (0x7ffc1d540000)

Why are there two ntdll.dll files? Never mind. :-)

These reported DLL are in the search path, and also moving these to the executable folder still gives the same error. I also moved the sqlite3.exe file from the built folder to the previous folder in the Cygwin environment, and the same problem happens: 

jcabrera@JIC ~/build/sqlite/SQLite-060eb284
$ mv sqlite3.exe ..
jcabrera@JIC ~/build/sqlite/SQLite-060eb284
$ cd ..
jcabrera@JIC ~/build/sqlite
$ ./sqlite3

jcabrera@JIC ~/build/sqlite
$ ./sqlite3.exe

jcabrera@JIC ~/build/sqlite


Well, let's say that nothing happens. So, it's something to do with the built command that only allows the sqlite3.exe to be run in the folder built folder and no-where else.

RE: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Jose Isaias Cabrera]

On Saturday, December 10, 2022 1:01 PM, Jose Isaias Cabrera expressed:
>
> These reported DLL are in the search path, and also moving these to the executable 
> folder still gives the same error. I also moved the sqlite3.exe file from the built
> folder to the previous folder in the Cygwin environment, and the same problem
> happens:

Just to put this to rest, and I am sure that there is a way of doing this using i686-w64-mingw32, but the only way that it appears to work is just doing plain ./configure and make. 

./configure
make

And then doing these commands:

To create the DLL:
i686-w64-mingw32-gcc -shared -static-libgcc sqlite3.c -o sqlite3.dll

To create the CLI tool:
i686-w64-mingw32-gcc -static-libgcc shell.c -o sqlite3.exe sqlite3.c

Both sqlite3.dll and sqlite3.exe CLI tool will work outside the cygwin environment. Just in case someone ever needs this. Thanks for all the support.

josé

Re: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Michael Soegtrop]

Hi José,

did you try copying the MinGW DLLs as I suggested?

You find them in folder 
"<cygroot>\usr\x86_64-w64-mingw32\sys-root\mingw\bin" and as I said 
"ldd" will tell you which ones you need. As a test you can also simply 
copy all of them to your bin folder, but it might be many (I have 85 
DLLs in this folder, it depends on which MinGW libraries you installed 
in cygwin).

I didn't do this for sqlite, but for quite a number of open source 
executables and this always worked since years. I would say this is the 
recommended way of doing it.

Static linking is also an option, but typically not well supported by 
the build systems. Also note that the distribution of a statically 
linked executable in many cases violates open source licenses, but doing 
this just for yourself is OK.

Best regards,

Michael

RE: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Jose Isaias Cabrera]

On December 11, 2022 5:23 AM, Michael Soegtrop  expressed:
> 
> Hi José,
> 
> did you try copying the MinGW DLLs as I suggested?

Yes, I did. I think you missed one of my emails. But, here is what I did:
$ ldd sqlite3.exe
        ntdll.dll => /cygdrive/c/Windows/SYSTEM32/ntdll.dll (0x7ffc1d6f0000)
        ntdll.dll => /cygdrive/c/Windows/SysWOW64/ntdll.dll (0x778c0000)
        wow64.dll => /cygdrive/c/Windows/System32/wow64.dll (0x7ffc1c830000)
        wow64win.dll => /cygdrive/c/Windows/System32/wow64win.dll (0x7ffc1d540000)

These are in the path, but also copying these to the executatble folder still gives the same error. I am able to get what I need by doing configure and make without any input. Then running the two commands on the previous email. The idea was to create stand-alone CLI tool and DLL without having any dependency of cygwin. I am able to do that with extra steps, but, for now, that is ok.

Grabbing on a very week branch, I think Corinna's changes for v3.4 is "breaking"/"fixing" some things with i686-w64-mingw32. Some of this used to work 3-4 weeks ago. I think I still have a machine with some old software. I am going to try it there and see if that old Cygwin version works.

Re: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Michael Soegtrop]

Hi José,

Yes, I did. I think you missed one of my emails. But, here is what I did:
> $ ldd sqlite3.exe
>          ntdll.dll => /cygdrive/c/Windows/SYSTEM32/ntdll.dll (0x7ffc1d6f0000)
>          ntdll.dll => /cygdrive/c/Windows/SysWOW64/ntdll.dll (0x778c0000)
>          wow64.dll => /cygdrive/c/Windows/System32/wow64.dll (0x7ffc1c830000)
>          wow64win.dll => /cygdrive/c/Windows/System32/wow64win.dll (0x7ffc1d540000)

Ah, you compiled a 32 bit executable - I guess on a 64 bit install of 
cygwin.

Try

./configure --host=x86_64-w64-mingw32 CFLAGS=-shared CFLAGS=-static-libgcc

instead of

./configure --host=i686-w64-mingw32 CFLAGS=-shared CFLAGS=-static-libgcc

Of course you need to install the corresponding tool chain in cygwin.

As far as I can tell compiling 32 bit apps on 64 bit cygwin did never 
work (easily). Afair the reason is that certain DLLs like 
SYSTEM32/ntdll.dll have 2 copies under the same file name, a 32 bit and 
a 64 bit variant (a Windows file system hack). Which one you get depends 
on if the calling process is 32 bit or 64 bit. Now if you try to link a 
32 bit executable with a 64 bit linker, it gets the wrong DLL, so your 
32 bit app ends up being linked to a 64 bit DLL.

One could only compile 32 bit Windows apps with 32 bit cygwin - since 
this is no longer supported, I would say 32 bit MinGW is neither. There 
are hacks around this (use a 32 bit executable to copy the DLLs from 
System32 somewhere else and redirect the linker to these files). But the 
better choice is to stop compiling for 32 bit.

Best regards,

Michael

RE: Creating the sqlite3.exe stand-alone w/o cygwin dependency

[Jose Isaias Cabrera]

On December 11, 2022 3:42 PM, Michael Soegtrop  expressed:
> 
> Hi José,
> 
> Yes, I did. I think you missed one of my emails. But, here is what I did:
> > $ ldd sqlite3.exe
> >          ntdll.dll => /cygdrive/c/Windows/SYSTEM32/ntdll.dll (0x7ffc1d6f0000)
> >          ntdll.dll => /cygdrive/c/Windows/SysWOW64/ntdll.dll (0x778c0000)
> >          wow64.dll => /cygdrive/c/Windows/System32/wow64.dll (0x7ffc1c830000)
> >          wow64win.dll => /cygdrive/c/Windows/System32/wow64win.dll
> > (0x7ffc1d540000)
> 
> Ah, you compiled a 32 bit executable - I guess on a 64 bit install of cygwin.
> 
> Try
> 
> ./configure --host=x86_64-w64-mingw32 CFLAGS=-shared CFLAGS=-static-libgcc

Yes, I need the 32 bit DLL because the application that I have is 32 bit application. However, now that you mention it below, and thanks, I will build the 64 bit CLI. Thanks a lot.

> As far as I can tell compiling 32 bit apps on 64 bit cygwin did never work 
> (easily). 

It actually works for me with a command. But, thanks for your support. You have gotten me much farther.

josé