Re: Failure during build of Python 3.8 via cygport

Re: Failure during build of Python 3.8 via cygport

[Mark Geisert]

[Bringing this thread over from cygwin-apps...]

Mark Geisert wrote:
> Mark Geisert wrote:
>> This seems to be a problem setting up a platform-specific build directory. The 
>> sysconfig.py script wants to use "lib." + platform + pythonversion but the 
>> platform string somehow gets corrupted into non-utf8 bytes.  For instance, 
>> building Python 3.8 comes up with:
>>      lib.cygwin-\365\377\377o-\377o-3.8
>> as the directory name.  Broken, but could work.  The build failure happens 
>> because the script tries to write this directory name into a file but it's not a 
>> valid utf8 string.  The directory name should have been:
>>      lib.cygwin-3.2.0-x86_64-3.8
> 
> And the corruption is due to something about a recent change to the operation of 
> Cygwin's uname() function.  The change was introduced in Cygwin API version 335; 
> I'm running 340 on my test machine.  This being a fairly recent change might 
> possibly explain why nobody else has run into this issue yet.
> 
> Basically, os.uname within Python is calling Cygwin's uname() passing the address 
> of a buffer declared to be 'struct utsname'.  The structure layout changed in API 
> 335.  What I've hit is a mismatch between what Python expects and Cygwin delivers.

Brian Inglis pointed out the API 335 change was made a couple years ago, so strike 
the quoted conjecture about why nobody else has hit this.  A new conjecture follows...

Debugging Python's os.uname showed Cygwin's uname() being called with a 'struct 
uname' as defined in /usr/include/sys/utsname.h, which is fine.  But it's the 
"old" pre-335 uname() interface being called, not the "new" 335+ interface 
uname_x().  Note that the famous 'mkimport' script, used when building the Cygwin 
DLL, has an arg "--replace=uname=uname_x" which I believe is supposed to equate 
the two names so the code in uname_x() is called whether the interface is 
uname_x() or uname().  That's not happening.

My first thought was, dang it, my "optimizations" of mkimport broke it.  But 
that's not the case.  Restoring a previous version of mkimport doesn't help.

'nm' shows that both uname and uname_x exist in libcygwin.a and also cygdll.a. 
And a newly-created Cygwin DLL has both functions, with different addresses.
That's wrong, isn't it?

..mark

Re: Failure during build of Python 3.8 via cygport

[Corinna Vinschen]

On Dec 14 02:42, Mark Geisert wrote:
> [Bringing this thread over from cygwin-apps...]
> 
> Mark Geisert wrote:
> > Mark Geisert wrote:
> > > This seems to be a problem setting up a platform-specific build
> > > directory. The sysconfig.py script wants to use "lib." + platform +
> > > pythonversion but the platform string somehow gets corrupted into
> > > non-utf8 bytes.  For instance, building Python 3.8 comes up with:
> > >      lib.cygwin-\365\377\377o-\377o-3.8
> > > as the directory name.  Broken, but could work.  The build failure
> > > happens because the script tries to write this directory name into a
> > > file but it's not a valid utf8 string.  The directory name should
> > > have been:
> > >      lib.cygwin-3.2.0-x86_64-3.8
> > 
> > And the corruption is due to something about a recent change to the
> > operation of Cygwin's uname() function.  The change was introduced in
> > Cygwin API version 335; I'm running 340 on my test machine.  This being
> > a fairly recent change might possibly explain why nobody else has run
> > into this issue yet.
> > 
> > Basically, os.uname within Python is calling Cygwin's uname() passing
> > the address of a buffer declared to be 'struct utsname'.  The structure
> > layout changed in API 335.  What I've hit is a mismatch between what
> > Python expects and Cygwin delivers.
> 
> Brian Inglis pointed out the API 335 change was made a couple years ago, so
> strike the quoted conjecture about why nobody else has hit this.  A new
> conjecture follows...
> 
> Debugging Python's os.uname showed Cygwin's uname() being called with a
> 'struct uname' as defined in /usr/include/sys/utsname.h, which is fine.  But
> it's the "old" pre-335 uname() interface being called, not the "new" 335+
> interface uname_x().  Note that the famous 'mkimport' script, used when
> building the Cygwin DLL, has an arg "--replace=uname=uname_x" which I
> believe is supposed to equate the two names so the code in uname_x() is
> called whether the interface is uname_x() or uname().  That's not happening.
> 
> My first thought was, dang it, my "optimizations" of mkimport broke it.  But
> that's not the case.  Restoring a previous version of mkimport doesn't help.
> 
> 'nm' shows that both uname and uname_x exist in libcygwin.a and also
> cygdll.a. And a newly-created Cygwin DLL has both functions, with different
> addresses.
> That's wrong, isn't it?

No, it isn't.  The old uname entry point is still required for old
applications built prior to API version 335.  If you build a new
application it will actually call uname_x when it tries to call uname
since that's how the libcygwin.a layer translates it.

It's a bit unfortunate that uname_x is exported by libcygwin.a
as well (in theory it should only export uname, pointing to uname_x
in the DLL), but this is how it always was, as you can see in
the 32 bit build with symbols only available there, e. g. aclcheck:

  $ nm libcygwin.a | grep aclcheck
  00000000 I __imp___aclcheck
  00000000 I __imp___aclcheck32
  00000000 I __imp__aclcheck
  00000000 T __aclcheck
	   U __imp___aclcheck
  00000000 T __aclcheck32
	   U __imp___aclcheck32
  00000000 T _aclcheck
	   U __imp___aclcheck32

In retrospect, uname_x should be named _uname_x or so, with a leading
underscore, so as not to pollute the namespace, but either way, that
isn;t your problem.

The problem here might be that you get the old uname function if
you dlopen the cygwin dll and dlsym(hdl, "uname").  Is that the
case in python?

If so, I have a simple, dirty workaround below.  Can you check if that's
the problem, please?


Corinna


From 45519fc1faa616e18fc5e2d78c483699e0f1f475 Mon Sep 17 00:00:00 2001
From: Corinna Vinschen <corinna@vinschen.de>
Date: Mon, 14 Dec 2020 12:29:23 +0100
Subject: [PATCH] Cygwin: Make sure newer apps get uname_x even when loading
 uname dynamically

if an application built after API version 334 loads uname dynamically,
it actually gets the old uname, rather than the new uname_x.  Fix this by
checking the apps API version in uname and call uname_x instead, if it's
a newer app.

Signed-off-by: Corinna Vinschen <corinna@vinschen.de>
---
 winsup/cygwin/include/cygwin/version.h | 3 +++
 winsup/cygwin/uname.cc                 | 7 +++++++
 2 files changed, 10 insertions(+)

diff --git a/winsup/cygwin/include/cygwin/version.h b/winsup/cygwin/include/cygwin/version.h
index f21aba72669a..b39d0dbe95bb 100644
--- a/winsup/cygwin/include/cygwin/version.h
+++ b/winsup/cygwin/include/cygwin/version.h
@@ -74,6 +74,9 @@ details. */
 #define CYGWIN_VERSION_CHECK_FOR_EXTRA_TM_MEMBERS \
   (CYGWIN_VERSION_USER_API_VERSION_COMBINED >= 272)
 
+#define CYGWIN_VERSION_CHECK_FOR_UNAME_X \
+  (CYGWIN_VERSION_USER_API_VERSION_COMBINED >= 335)
+
 #define CYGWIN_VERSION_CYGWIN_CONV 181
 
 /* API_MAJOR 0.0: Initial version.  API_MINOR changes:
diff --git a/winsup/cygwin/uname.cc b/winsup/cygwin/uname.cc
index 350216681fa1..48f7dda6029b 100644
--- a/winsup/cygwin/uname.cc
+++ b/winsup/cygwin/uname.cc
@@ -92,7 +92,14 @@ struct old_utsname
 extern "C" int
 uname (struct utsname *in_name)
 {
+  /* This occurs if the application fetches the uname symbol dynamically.
+     We must call uname_x for newer API versions, otherwise the idea of
+     struct utsname doesn't match. */
+  if (CYGWIN_VERSION_CHECK_FOR_UNAME_X)
+    return uname_x (in_name);
+
   struct old_utsname *name = (struct old_utsname *) in_name;
+
   __try
     {
       char *snp = strstr  (cygwin_version.dll_build_date, "SNP");
-- 
2.29.2

Re: Failure during build of Python 3.8 via cygport

[Mark Geisert]

Hi Corinna,

Corinna Vinschen via Cygwin-developers wrote:
> On Dec 14 02:42, Mark Geisert wrote:
[...]
>> Debugging Python's os.uname showed Cygwin's uname() being called with a
>> 'struct uname' as defined in /usr/include/sys/utsname.h, which is fine.  But
>> it's the "old" pre-335 uname() interface being called, not the "new" 335+
>> interface uname_x().  Note that the famous 'mkimport' script, used when
>> building the Cygwin DLL, has an arg "--replace=uname=uname_x" which I
>> believe is supposed to equate the two names so the code in uname_x() is
>> called whether the interface is uname_x() or uname().  That's not happening.
[...]
>> 'nm' shows that both uname and uname_x exist in libcygwin.a and also
>> cygdll.a. And a newly-created Cygwin DLL has both functions, with different
>> addresses.
>> That's wrong, isn't it?
> 
> No, it isn't.  The old uname entry point is still required for old
> applications built prior to API version 335.  If you build a new
> application it will actually call uname_x when it tries to call uname
> since that's how the libcygwin.a layer translates it.
> 
> It's a bit unfortunate that uname_x is exported by libcygwin.a
> as well (in theory it should only export uname, pointing to uname_x
> in the DLL), but this is how it always was, as you can see in
> the 32 bit build with symbols only available there, e. g. aclcheck:
> 
>    $ nm libcygwin.a | grep aclcheck
>    00000000 I __imp___aclcheck
>    00000000 I __imp___aclcheck32
>    00000000 I __imp__aclcheck
>    00000000 T __aclcheck
> 	   U __imp___aclcheck
>    00000000 T __aclcheck32
> 	   U __imp___aclcheck32
>    00000000 T _aclcheck
> 	   U __imp___aclcheck32
> 
> In retrospect, uname_x should be named _uname_x or so, with a leading
> underscore, so as not to pollute the namespace, but either way, that
> isn;t your problem.

OK, I see.

> The problem here might be that you get the old uname function if
> you dlopen the cygwin dll and dlsym(hdl, "uname").  Is that the
> case in python?

Yes it is.

> If so, I have a simple, dirty workaround below.  Can you check if that's
> the problem, please?

A new Cygwin DLL built with your patch does correct this 'uname' issue when 
building Python.  Wonderful!
Thank you very much!

..mark

Re: Failure during build of Python 3.8 via cygport

[Marco Atzeri]

On 15.12.2020 09:06, Mark Geisert wrote:
> Hi Corinna,
> 
> Corinna Vinschen via Cygwin-developers wrote:
>> On Dec 14 02:42, Mark Geisert wrote:
> [...]
>>
>> In retrospect, uname_x should be named _uname_x or so, with a leading
>> underscore, so as not to pollute the namespace, but either way, that
>> isn;t your problem.
> 
> OK, I see.
> 
>> The problem here might be that you get the old uname function if
>> you dlopen the cygwin dll and dlsym(hdl, "uname").  Is that the
>> case in python?
> 
> Yes it is.
> 
>> If so, I have a simple, dirty workaround below.  Can you check if that's
>> the problem, please?
> 
> A new Cygwin DLL built with your patch does correct this 'uname' issue 
> when building Python.  Wonderful!
> Thank you very much!
> 
> ..mark

It seems I do not need to rebuild python for this reason. Correct ?

Re: Failure during build of Python 3.8 via cygport

[Mark Geisert]

Marco Atzeri via Cygwin-developers wrote:
> On 15.12.2020 09:06, Mark Geisert wrote:
>> Hi Corinna,
>>
>> Corinna Vinschen via Cygwin-developers wrote:
>>> On Dec 14 02:42, Mark Geisert wrote:
>> [...]
>>>
>>> In retrospect, uname_x should be named _uname_x or so, with a leading
>>> underscore, so as not to pollute the namespace, but either way, that
>>> isn;t your problem.
>>
>> OK, I see.
>>
>>> The problem here might be that you get the old uname function if
>>> you dlopen the cygwin dll and dlsym(hdl, "uname").  Is that the
>>> case in python?
>>
>> Yes it is.
>>
>>> If so, I have a simple, dirty workaround below.  Can you check if that's
>>> the problem, please?
>>
>> A new Cygwin DLL built with your patch does correct this 'uname' issue when 
>> building Python.  Wonderful!
>> Thank you very much!
>>
>> ..mark
> 
> It seems I do not need to rebuild python for this reason. Correct ?

That's correct Marco.  But it allows me to build Python locally so I can produce 
patch(es) for the test_asyncore.py issue involving AF_UNIX sockets.  When I've got 
a patch I'll follow up on the cygwin-apps list, I guess.  Thanks for keeping an 
eye on this.

..mark

Re: Failure during build of Python 3.8 via cygport

[Corinna Vinschen]

On Dec 15 00:06, Mark Geisert wrote:
> Hi Corinna,
> 
> Corinna Vinschen via Cygwin-developers wrote:
> > On Dec 14 02:42, Mark Geisert wrote:
> [...]
> > > Debugging Python's os.uname showed Cygwin's uname() being called with a
> > > 'struct uname' as defined in /usr/include/sys/utsname.h, which is fine.  But
> > > it's the "old" pre-335 uname() interface being called, not the "new" 335+
> > > interface uname_x().  Note that the famous 'mkimport' script, used when
> > > building the Cygwin DLL, has an arg "--replace=uname=uname_x" which I
> > > believe is supposed to equate the two names so the code in uname_x() is
> > > called whether the interface is uname_x() or uname().  That's not happening.
> [...]
> > > 'nm' shows that both uname and uname_x exist in libcygwin.a and also
> > > cygdll.a. And a newly-created Cygwin DLL has both functions, with different
> > > addresses.
> > > That's wrong, isn't it?
> > 
> > No, it isn't.  The old uname entry point is still required for old
> > applications built prior to API version 335.  If you build a new
> > application it will actually call uname_x when it tries to call uname
> > since that's how the libcygwin.a layer translates it.
> > 
> > It's a bit unfortunate that uname_x is exported by libcygwin.a
> > as well (in theory it should only export uname, pointing to uname_x
> > in the DLL), but this is how it always was, as you can see in
> > the 32 bit build with symbols only available there, e. g. aclcheck:
> > 
> >    $ nm libcygwin.a | grep aclcheck
> >    00000000 I __imp___aclcheck
> >    00000000 I __imp___aclcheck32
> >    00000000 I __imp__aclcheck
> >    00000000 T __aclcheck
> > 	   U __imp___aclcheck
> >    00000000 T __aclcheck32
> > 	   U __imp___aclcheck32
> >    00000000 T _aclcheck
> > 	   U __imp___aclcheck32
> > 
> > In retrospect, uname_x should be named _uname_x or so, with a leading
> > underscore, so as not to pollute the namespace, but either way, that
> > isn;t your problem.
> 
> OK, I see.
> 
> > The problem here might be that you get the old uname function if
> > you dlopen the cygwin dll and dlsym(hdl, "uname").  Is that the
> > case in python?
> 
> Yes it is.
> 
> > If so, I have a simple, dirty workaround below.  Can you check if that's
> > the problem, please?
> 
> A new Cygwin DLL built with your patch does correct this 'uname' issue when
> building Python.  Wonderful!
> Thank you very much!

Great, thanks for testing and confirming!  I pushed the patch.


Corinna

Re: Failure during build of Python 3.8 via cygport

[Corinna Vinschen]

On Dec 15 13:06, Corinna Vinschen via Cygwin-developers wrote:
> On Dec 15 00:06, Mark Geisert wrote:
> > Hi Corinna,
> > 
> > Corinna Vinschen via Cygwin-developers wrote:
> > > On Dec 14 02:42, Mark Geisert wrote:
> > [...]
> > > > Debugging Python's os.uname showed Cygwin's uname() being called with a
> > > > 'struct uname' as defined in /usr/include/sys/utsname.h, which is fine.  But
> > > > it's the "old" pre-335 uname() interface being called, not the "new" 335+
> > > > interface uname_x().  Note that the famous 'mkimport' script, used when
> > > > building the Cygwin DLL, has an arg "--replace=uname=uname_x" which I
> > > > believe is supposed to equate the two names so the code in uname_x() is
> > > > called whether the interface is uname_x() or uname().  That's not happening.
> > [...]
> > > > 'nm' shows that both uname and uname_x exist in libcygwin.a and also
> > > > cygdll.a. And a newly-created Cygwin DLL has both functions, with different
> > > > addresses.
> > > > That's wrong, isn't it?
> > > 
> > > No, it isn't.  The old uname entry point is still required for old
> > > applications built prior to API version 335.  If you build a new
> > > application it will actually call uname_x when it tries to call uname
> > > since that's how the libcygwin.a layer translates it.
> > > 
> > > It's a bit unfortunate that uname_x is exported by libcygwin.a
> > > as well (in theory it should only export uname, pointing to uname_x
> > > in the DLL), but this is how it always was, as you can see in
> > > the 32 bit build with symbols only available there, e. g. aclcheck:
> > > 
> > >    $ nm libcygwin.a | grep aclcheck
> > >    00000000 I __imp___aclcheck
> > >    00000000 I __imp___aclcheck32
> > >    00000000 I __imp__aclcheck
> > >    00000000 T __aclcheck
> > > 	   U __imp___aclcheck
> > >    00000000 T __aclcheck32
> > > 	   U __imp___aclcheck32
> > >    00000000 T _aclcheck
> > > 	   U __imp___aclcheck32
> > > 
> > > In retrospect, uname_x should be named _uname_x or so, with a leading
> > > underscore, so as not to pollute the namespace, but either way, that
> > > isn;t your problem.
> > 
> > OK, I see.
> > 
> > > The problem here might be that you get the old uname function if
> > > you dlopen the cygwin dll and dlsym(hdl, "uname").  Is that the
> > > case in python?
> > 
> > Yes it is.
> > 
> > > If so, I have a simple, dirty workaround below.  Can you check if that's
> > > the problem, please?
> > 
> > A new Cygwin DLL built with your patch does correct this 'uname' issue when
> > building Python.  Wonderful!
> > Thank you very much!
> 
> Great, thanks for testing and confirming!  I pushed the patch.

I created new developer snapshots.


Corinna