python-numpy (1.22.0-1) can't be imported

python-numpy (1.22.0-1) can't be imported

[ggl329]

Hi Marco,

I upgraded python39-numpy to 1.22.0-1, and failed to import numpy.
It seems that mtrand.cpython-39-x86_64-cygwin.dll does not have PyInit_mtrand.
Could you check if numpy can be imported in your environment?


$ python3.9
Python 3.9.9 (main, Dec  5 2021, 13:29:55)
[GCC 11.2.0] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python3.9/site-packages/numpy/__init__.py", line 155, in <module>
     from . import random
   File "/usr/lib/python3.9/site-packages/numpy/random/__init__.py", line 180, in <module>
     from . import _pickle
   File "/usr/lib/python3.9/site-packages/numpy/random/_pickle.py", line 1, in <module>
     from .mtrand import RandomState
ImportError: dynamic module does not define module export function (PyInit_mtrand)
>>>


== python39-numpy 1.22.0-1 ==
$ objdump.exe -x /usr/lib/python3.9/site-packages/numpy/random/mtrand.cpython-39-x86_64-cygwin.dll
(snip)
[Ordinal/Name Pointer] Table
         [   0] random_beta
         [   1] random_binomial
         [   2] random_bounded_bool_fill
         [   3] random_bounded_uint16_fill
         [   4] random_bounded_uint32_fill
         ...


== python39-numpy 1.21.4-1 ==
$ objdump.exe -x /usr/lib/python3.9/site-packages/numpy/random/mtrand.cpython-39-x86_64-cygwin.dll
(snip)
[Ordinal/Name Pointer] Table
         [   0] PyInit_mtrand
         [   1] __gcc_deregister_frame
         [   2] __gcc_register_frame
         [   3] __pyx_module_is_main_numpy__random__mtrand
         [   4] legacy_beta
         ...
## 1.21.4-1 has PyInit_mtrand.


Regards,

-- 
   ggl329

Re: python-numpy (1.22.0-1) can't be imported

[Marco Atzeri]

On 12.01.2022 12:47, ggl329 wrote:
> Hi Marco,
> 
> I upgraded python39-numpy to 1.22.0-1, and failed to import numpy.
> It seems that mtrand.cpython-39-x86_64-cygwin.dll does not have 
> PyInit_mtrand.
> Could you check if numpy can be imported in your environment?
> 
> 

working on it.
In theory I have not changed the build between the two versions
but there was a patch in 1.19.4-1 for similar issue.

Regards
Marco

Re: python-numpy (1.22.0-1) can't be imported

[airplanemath]

On Wed, Jan 12 2022, Marco Atzeri wrote:

> On 12.01.2022 12:47, ggl329 wrote:
>> Hi Marco,
>> I upgraded python39-numpy to 1.22.0-1, and failed to import numpy.
>> It seems that mtrand.cpython-39-x86_64-cygwin.dll does not have
>> PyInit_mtrand.
>> Could you check if numpy can be imported in your environment?
>> 
>
> working on it.
> In theory I have not changed the build between the two versions
> but there was a patch in 1.19.4-1 for similar issue.
>
> Regards
> Marco

I have the same problems with the distribution NumPy package, but I can
use a locally-compiled NumPy with no patches.  It doesn't look like your
build has any patches either:
https://cygwin.com/cgi-bin2/package-cat.cgi?file=x86_64%2Fpython-numpy-src%2Fpython-numpy-1.22.0-1-src&grep=numpy
so that's fun.

Out of curiousity, what options are you passing for cpu-baseline and
cpu-dispatch?
https://numpy.org/devdocs/reference/simd/build-options.html
I don't see the newest cygport in the Cygwin package repository.

Re: python-numpy (1.22.0-1) can't be imported

[Marco Atzeri]

On 14.01.2022 03:04, airplanemath via Cygwin wrote:
> On Wed, Jan 12 2022, Marco Atzeri wrote:
> 
>> On 12.01.2022 12:47, ggl329 wrote:
>>> Hi Marco,
>>> I upgraded python39-numpy to 1.22.0-1, and failed to import numpy.
>>> It seems that mtrand.cpython-39-x86_64-cygwin.dll does not have
>>> PyInit_mtrand.
>>> Could you check if numpy can be imported in your environment?
>>>
>>
>> working on it.
>> In theory I have not changed the build between the two versions
>> but there was a patch in 1.19.4-1 for similar issue.
>>
>> Regards
>> Marco
> 
> I have the same problems with the distribution NumPy package, but I can
> use a locally-compiled NumPy with no patches.  It doesn't look like your
> build has any patches either:
> https://cygwin.com/cgi-bin2/package-cat.cgi?file=x86_64%2Fpython-numpy-src%2Fpython-numpy-1.22.0-1-src&grep=numpy
> so that's fun.
> 
> Out of curiousity, what options are you passing for cpu-baseline and
> cpu-dispatch?
> https://numpy.org/devdocs/reference/simd/build-options.html
> I don't see the newest cygport in the Cygwin package repository.

I have not found the root cause yet.
As the  1.21.4-1 imports correctly I removed the 1.22.0-1 until I solve 
the issue.

I do not see anything obvious in upstream source between 1.21.4 and 
1.22.0 that gives me any hint on root cause.

Also 1.22.1 shows the same problem.
I excluded the build chain as rebuilding 1.19.4 worked fine
for all 3.6 to 3.9

Regards
Marco

Re: python-numpy (1.22.0-1) can't be imported

[Masamichi Hosoda]

> I have not found the root cause yet.
> As the  1.21.4-1 imports correctly I removed the 1.22.0-1 until I solve 
> the issue.
> 
> I do not see anything obvious in upstream source between 1.21.4 and 
> 1.22.0 that gives me any hint on root cause.
> 
> Also 1.22.1 shows the same problem.
> I excluded the build chain as rebuilding 1.19.4 worked fine
> for all 3.6 to 3.9

If I understand correctly,
the patch below is just a quick hack, but it solves the problem.

```
--- a/numpy/random/setup.py
+++ b/numpy/random/setup.py
@@ -147,7 +147,8 @@
                          include_dirs=['.', 'src', 'src/legacy'],
                          libraries=mtrand_libs,
                          extra_compile_args=EXTRA_COMPILE_ARGS,
-                         extra_link_args=EXTRA_LINK_ARGS,
+                         extra_link_args=(EXTRA_LINK_ARGS +
+                                          ['-Wl,--export-all-symbols']),
                          depends=depends + ['mtrand.pyx'],
                          define_macros=defs + LEGACY_DEFS,
                          )
```

Re: python-numpy (1.22.0-1) can't be imported

[Marco Atzeri]

On 22.01.2022 02:04, Masamichi Hosoda wrote:
>> I have not found the root cause yet.
>> As the  1.21.4-1 imports correctly I removed the 1.22.0-1 until I solve
>> the issue.
>>
>> I do not see anything obvious in upstream source between 1.21.4 and
>> 1.22.0 that gives me any hint on root cause.
>>
>> Also 1.22.1 shows the same problem.
>> I excluded the build chain as rebuilding 1.19.4 worked fine
>> for all 3.6 to 3.9
> 
> If I understand correctly,
> the patch below is just a quick hack, but it solves the problem.
> 
> ```
> --- a/numpy/random/setup.py
> +++ b/numpy/random/setup.py
> @@ -147,7 +147,8 @@
>                            include_dirs=['.', 'src', 'src/legacy'],
>                            libraries=mtrand_libs,
>                            extra_compile_args=EXTRA_COMPILE_ARGS,
> -                         extra_link_args=EXTRA_LINK_ARGS,
> +                         extra_link_args=(EXTRA_LINK_ARGS +
> +                                          ['-Wl,--export-all-symbols']),
>                            depends=depends + ['mtrand.pyx'],
>                            define_macros=defs + LEGACY_DEFS,
>                            )
> ```


Thanks

It is something like that, but "-Wl,--export-all-symbols"
is not used on 1.21.4 and is not needed for most of the other
modules on 1.22.x

so I am looking for a less extreme action.
Also to understand how it can impact other python subpackages


Regards
Marco

Re: python-numpy (1.22.0-1) can't be imported

[Masamichi Hosoda]

> It is something like that, but "-Wl,--export-all-symbols"
> is not used on 1.21.4 and is not needed for most of the other
> modules on 1.22.x
> 
> so I am looking for a less extreme action.
> Also to understand how it can impact other python subpackages

If I understand correctly, I've found the root cause of the issue.
I've sent a pull request to numpy.
https://github.com/numpy/numpy/pull/20874

Re: python-numpy (1.22.0-1) can't be imported

[Masamichi Hosoda]

> If I understand correctly, I've found the root cause of the issue.
> I've sent a pull request to numpy.
> https://github.com/numpy/numpy/pull/20874

A question is asked by NumPy if there is documentation
on the Cygwin package's Cython behavior.
https://github.com/numpy/numpy/pull/20874#issuecomment-1019442058

Does such a document exist?

One of the causes of this issue is a patch
in Cygwin's python-cython package like the one below.
It removes the `__declspec(dllexport)` attribute
from the symbols to be exported.

On the other hand, if you install Cython
by pip without using the Cygwin package,
the exported symbols have the `__declspec(dllexport)` attribute
because the patch has not been applied.

What is the purpose of removing the `__declspec(dllexport)` attribute
in this patch?
Is this to make the modules using Cython
without considering the Cygwin environment,
the module does not have the `__declspec(dllexport)` attribute
like Linux environment, works correctly?

```
--- origsrc/Cython-0.29.21/Cython/Utility/ModuleSetupCode.c	2020-07-08 23:44:39.000000000 +0200
+++ Cython/Utility/ModuleSetupCode.c	2021-01-30 08:34:37.402649500 +0100
@@ -709,7 +709,11 @@ static CYTHON_INLINE void * PyThread_tss
 /////////////// PyModInitFuncType.proto ///////////////
 
 #ifndef CYTHON_NO_PYINIT_EXPORT
-#define __Pyx_PyMODINIT_FUNC PyMODINIT_FUNC
+#ifdef __cplusplus
+#define __Pyx_PyMODINIT_FUNC extern "C" PyObject *
+#else
+#define __Pyx_PyMODINIT_FUNC PyObject *
+#endif
 
 #elif PY_MAJOR_VERSION < 3
 // Py2: define this to void manually because PyMODINIT_FUNC adds __declspec(dllexport) to it's definition.
```

Re: python-numpy (1.22.0-1) can't be imported

[airplanemath]

On Sat, Jan 22 2022, Masamichi Hosoda wrote:

>> It is something like that, but "-Wl,--export-all-symbols"
>> is not used on 1.21.4 and is not needed for most of the other
>> modules on 1.22.x
>> 
>> so I am looking for a less extreme action.
>> Also to understand how it can impact other python subpackages
>
> If I understand correctly, I've found the root cause of the issue.
> I've sent a pull request to numpy.
> https://github.com/numpy/numpy/pull/20874

Based on that pull request, it looks like it's related to the patches
used to avoid python C-extension modules with only one exported function
mentioned a year ago:
https://cygwin.com/pipermail/cygwin/2021-January/247211.html
https://cygwin.com/pipermail/cygwin/2021-January/247216.html
From a glance at the Python C API docs, it looks like Python C extension
modules are supposed to have only one exported function:
https://docs.python.org/3/extending/extending.html#the-module-s-method-table-and-initialization-function
https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module

The NumPy CFFI tests seem to want to use a few C extension modules as
normal shared libraries, which requires that the functions the tests
want to use to be exported from the DLL, but this seems to be a somewhat
unusual case.

I tend to compile my own versions of Cython (without patches), which may
have been picked up when I compiled the new version of NumPy.