Python C Extension Module loading issue on Cygwin

Python C Extension Module loading issue on Cygwin

[Mesibo Technical]

This issue is about Python on Cygwin not using the recommended module extension.

We have a real-time messaging Python module
(https://pypi.org/project/mesibo/) available on various platforms,
including Linux, macOS, Windows, and Raspberry Pi. It is written in
C/C++ and has been working correctly on these platforms.

Recently, one of our users encountered the "ModuleNotFoundError issue"
when trying to use it on Cygwin.

ModuleNotFoundError: No module named '_mesibo'

Upon investigation, we found that the Python version running on Cygwin
is not using the correct DLL extension as recommended in Python's
official documentation. According to Python's documentation here
https://docs.python.org/3/extending/building.html:

"A C extension for CPython is a shared library (e.g. a .so file on
Linux, .pyd on Windows), which exports an initialization function. To
be importable, the shared library must be available on PYTHONPATH, and
must be named after the module name, with an appropriate extension."

We followed this documentation and used the .pyd extension as
suggested for Windows. It works fine with the Python version bundled
with Windows, and also when downloaded from the Python website.
However, Python on Cygwin seems to be using the .dll extension instead
of .pyd and hence not able to find the module resulting in
ModuleNotFoundError, as you can see from the following logs:

Following are the logs when running the Cygwin version, as you can
see, it is trying to find the module with extension .dll (_mesibo.dll)
instead of .pyd (_mesibo.pyd)

[/cygdrive/c/python] $ python -vvvv test.py
...
# trying /usr/local/lib/python3.9/site-packages/mesibo/_mesibo.cpython-39-x86_64-cygwin.dll
# trying /usr/local/lib/python3.9/site-packages/mesibo/_mesibo.abi3.dll
# trying /usr/local/lib/python3.9/site-packages/mesibo/_mesibo.dll
...

On the contrary, on Python bundle with Windows or when we download and
install from the Python website correctly looks for and loads
_mesibo.pyd

C:\mesibo > python -vvvv test.py
...
# trying
C:\Users\Grace\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\mesibo\_mesibo.cp311-win_amd64.pyd
# trying C:\Users\Grace\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\mesibo\_mesibo.pyd

While we were able to temporarily address this issue by manually
renaming .pyd to .dll for Cygwin, we are skeptical whether this is the
correct and future-proof solution.

Any idea why Cygwin is using the .dll extension instead of the .pyd
extension as recommended by Python's official documentation?
Additionally, is there a way to know during runtime the specific
extension used by a particular version of Python, without having to
make assumptions based on the system type?

Any insights or solutions will be greatly appreciated.

Thanks a lot!

Re: Python C Extension Module loading issue on Cygwin

[jojelino]

On 9/22/2023 3:39 PM, Mesibo Technical via Cygwin wrote:
> Any idea why Cygwin is using the .dll extension instead of the .pyd
> extension as recommended by Python's official documentation?

$ grep "_SUFFIX=" /usr/lib/python3.9/config-3.9-x86_64-cygwin/Makefile
SHLIB_SUFFIX=   .dll
EXT_SUFFIX=     .cpython-39-x86_64-cygwin.dll

Your speculation is quite weird, Normally you don't have no problem 
loading extension module without suffix in python. and paragraph like 
(eg,) is regarded as neither exhaustive or authoritative.

How about look through source code of distutils,setuptools,cpython 
instead of complaining about software without any warranty(according to 
PSF license)?

Re: Python C Extension Module loading issue on Cygwin

[Marco Atzeri]

On 22.09.2023 08:39, Mesibo Technical via Cygwin wrote:
> This issue is about Python on Cygwin not using the recommended module extension.
[cut]
> Any idea why Cygwin is using the .dll extension instead of the .pyd
> extension as recommended by Python's official documentation?
> Additionally, is there a way to know during runtime the specific
> extension used by a particular version of Python, without having to
> make assumptions based on the system type?

Cygwin is Unix on Windows, so the dll extension is equivalent
to the so extension on Linux/Unix.
I am almost sure that we are not modifying the source code for this 
specific issue. But I could be wrong


> Any insights or solutions will be greatly appreciated.
> 
> Thanks a lot!


Regards
Marco

Re: Python C Extension Module loading issue on Cygwin

[Eliot Moss]

On 11/27/2023 12:49 PM, Marco Atzeri via Cygwin wrote:
> On 22.09.2023 08:39, Mesibo Technical via Cygwin wrote:
>> This issue is about Python on Cygwin not using the recommended module extension.
> [cut]
>> Any idea why Cygwin is using the .dll extension instead of the .pyd
>> extension as recommended by Python's official documentation?
>> Additionally, is there a way to know during runtime the specific
>> extension used by a particular version of Python, without having to
>> make assumptions based on the system type?
> 
> Cygwin is Unix on Windows, so the dll extension is equivalent
> to the so extension on Linux/Unix.
> I am almost sure that we are not modifying the source code for this specific issue. But I could be wrong

Allow me to add that Cygwin is not a virtual machine in the sense of
WSL (Windows Subsystem for Linux), VirtualBox, etc.  Rather, it consists
of a dll library that does a pretty good job mapping POSIX library
functions to Windows ones, and large collections of programs and libraries
that have been ported to the Cygwin environment, including X windows.

In any case, this means that it follows Windows linking conventions and
object formats.

HTH understand why tings are as they are.

Regards - Eliot Moss