* Re: Failure during build of Python 3.8 via cygport [not found] ` <758d2138-587b-2970-6c35-69d5c655a598@maxrnd.com> @ 2020-12-14 10:42 ` Mark Geisert 2020-12-14 11:33 ` Corinna Vinschen 0 siblings, 1 reply; 7+ messages in thread From: Mark Geisert @ 2020-12-14 10:42 UTC (permalink / raw) To: cygwin-developers [Bringing this thread over from cygwin-apps...] Mark Geisert wrote: > Mark Geisert wrote: >> This seems to be a problem setting up a platform-specific build directory. The >> sysconfig.py script wants to use "lib." + platform + pythonversion but the >> platform string somehow gets corrupted into non-utf8 bytes. For instance, >> building Python 3.8 comes up with: >> lib.cygwin-\365\377\377o-\377o-3.8 >> as the directory name. Broken, but could work. The build failure happens >> because the script tries to write this directory name into a file but it's not a >> valid utf8 string. The directory name should have been: >> lib.cygwin-3.2.0-x86_64-3.8 > > And the corruption is due to something about a recent change to the operation of > Cygwin's uname() function. The change was introduced in Cygwin API version 335; > I'm running 340 on my test machine. This being a fairly recent change might > possibly explain why nobody else has run into this issue yet. > > Basically, os.uname within Python is calling Cygwin's uname() passing the address > of a buffer declared to be 'struct utsname'. The structure layout changed in API > 335. What I've hit is a mismatch between what Python expects and Cygwin delivers. Brian Inglis pointed out the API 335 change was made a couple years ago, so strike the quoted conjecture about why nobody else has hit this. A new conjecture follows... Debugging Python's os.uname showed Cygwin's uname() being called with a 'struct uname' as defined in /usr/include/sys/utsname.h, which is fine. But it's the "old" pre-335 uname() interface being called, not the "new" 335+ interface uname_x(). Note that the famous 'mkimport' script, used when building the Cygwin DLL, has an arg "--replace=uname=uname_x" which I believe is supposed to equate the two names so the code in uname_x() is called whether the interface is uname_x() or uname(). That's not happening. My first thought was, dang it, my "optimizations" of mkimport broke it. But that's not the case. Restoring a previous version of mkimport doesn't help. 'nm' shows that both uname and uname_x exist in libcygwin.a and also cygdll.a. And a newly-created Cygwin DLL has both functions, with different addresses. That's wrong, isn't it? ..mark ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Failure during build of Python 3.8 via cygport 2020-12-14 10:42 ` Failure during build of Python 3.8 via cygport Mark Geisert @ 2020-12-14 11:33 ` Corinna Vinschen 2020-12-15 8:06 ` Mark Geisert 0 siblings, 1 reply; 7+ messages in thread From: Corinna Vinschen @ 2020-12-14 11:33 UTC (permalink / raw) To: cygwin-developers On Dec 14 02:42, Mark Geisert wrote: > [Bringing this thread over from cygwin-apps...] > > Mark Geisert wrote: > > Mark Geisert wrote: > > > This seems to be a problem setting up a platform-specific build > > > directory. The sysconfig.py script wants to use "lib." + platform + > > > pythonversion but the platform string somehow gets corrupted into > > > non-utf8 bytes. For instance, building Python 3.8 comes up with: > > > lib.cygwin-\365\377\377o-\377o-3.8 > > > as the directory name. Broken, but could work. The build failure > > > happens because the script tries to write this directory name into a > > > file but it's not a valid utf8 string. The directory name should > > > have been: > > > lib.cygwin-3.2.0-x86_64-3.8 > > > > And the corruption is due to something about a recent change to the > > operation of Cygwin's uname() function. The change was introduced in > > Cygwin API version 335; I'm running 340 on my test machine. This being > > a fairly recent change might possibly explain why nobody else has run > > into this issue yet. > > > > Basically, os.uname within Python is calling Cygwin's uname() passing > > the address of a buffer declared to be 'struct utsname'. The structure > > layout changed in API 335. What I've hit is a mismatch between what > > Python expects and Cygwin delivers. > > Brian Inglis pointed out the API 335 change was made a couple years ago, so > strike the quoted conjecture about why nobody else has hit this. A new > conjecture follows... > > Debugging Python's os.uname showed Cygwin's uname() being called with a > 'struct uname' as defined in /usr/include/sys/utsname.h, which is fine. But > it's the "old" pre-335 uname() interface being called, not the "new" 335+ > interface uname_x(). Note that the famous 'mkimport' script, used when > building the Cygwin DLL, has an arg "--replace=uname=uname_x" which I > believe is supposed to equate the two names so the code in uname_x() is > called whether the interface is uname_x() or uname(). That's not happening. > > My first thought was, dang it, my "optimizations" of mkimport broke it. But > that's not the case. Restoring a previous version of mkimport doesn't help. > > 'nm' shows that both uname and uname_x exist in libcygwin.a and also > cygdll.a. And a newly-created Cygwin DLL has both functions, with different > addresses. > That's wrong, isn't it? No, it isn't. The old uname entry point is still required for old applications built prior to API version 335. If you build a new application it will actually call uname_x when it tries to call uname since that's how the libcygwin.a layer translates it. It's a bit unfortunate that uname_x is exported by libcygwin.a as well (in theory it should only export uname, pointing to uname_x in the DLL), but this is how it always was, as you can see in the 32 bit build with symbols only available there, e. g. aclcheck: $ nm libcygwin.a | grep aclcheck 00000000 I __imp___aclcheck 00000000 I __imp___aclcheck32 00000000 I __imp__aclcheck 00000000 T __aclcheck U __imp___aclcheck 00000000 T __aclcheck32 U __imp___aclcheck32 00000000 T _aclcheck U __imp___aclcheck32 In retrospect, uname_x should be named _uname_x or so, with a leading underscore, so as not to pollute the namespace, but either way, that isn;t your problem. The problem here might be that you get the old uname function if you dlopen the cygwin dll and dlsym(hdl, "uname"). Is that the case in python? If so, I have a simple, dirty workaround below. Can you check if that's the problem, please? Corinna From 45519fc1faa616e18fc5e2d78c483699e0f1f475 Mon Sep 17 00:00:00 2001 From: Corinna Vinschen <corinna@vinschen.de> Date: Mon, 14 Dec 2020 12:29:23 +0100 Subject: [PATCH] Cygwin: Make sure newer apps get uname_x even when loading uname dynamically if an application built after API version 334 loads uname dynamically, it actually gets the old uname, rather than the new uname_x. Fix this by checking the apps API version in uname and call uname_x instead, if it's a newer app. Signed-off-by: Corinna Vinschen <corinna@vinschen.de> --- winsup/cygwin/include/cygwin/version.h | 3 +++ winsup/cygwin/uname.cc | 7 +++++++ 2 files changed, 10 insertions(+) diff --git a/winsup/cygwin/include/cygwin/version.h b/winsup/cygwin/include/cygwin/version.h index f21aba72669a..b39d0dbe95bb 100644 --- a/winsup/cygwin/include/cygwin/version.h +++ b/winsup/cygwin/include/cygwin/version.h @@ -74,6 +74,9 @@ details. */ #define CYGWIN_VERSION_CHECK_FOR_EXTRA_TM_MEMBERS \ (CYGWIN_VERSION_USER_API_VERSION_COMBINED >= 272) +#define CYGWIN_VERSION_CHECK_FOR_UNAME_X \ + (CYGWIN_VERSION_USER_API_VERSION_COMBINED >= 335) + #define CYGWIN_VERSION_CYGWIN_CONV 181 /* API_MAJOR 0.0: Initial version. API_MINOR changes: diff --git a/winsup/cygwin/uname.cc b/winsup/cygwin/uname.cc index 350216681fa1..48f7dda6029b 100644 --- a/winsup/cygwin/uname.cc +++ b/winsup/cygwin/uname.cc @@ -92,7 +92,14 @@ struct old_utsname extern "C" int uname (struct utsname *in_name) { + /* This occurs if the application fetches the uname symbol dynamically. + We must call uname_x for newer API versions, otherwise the idea of + struct utsname doesn't match. */ + if (CYGWIN_VERSION_CHECK_FOR_UNAME_X) + return uname_x (in_name); + struct old_utsname *name = (struct old_utsname *) in_name; + __try { char *snp = strstr (cygwin_version.dll_build_date, "SNP"); -- 2.29.2 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Failure during build of Python 3.8 via cygport 2020-12-14 11:33 ` Corinna Vinschen @ 2020-12-15 8:06 ` Mark Geisert 2020-12-15 8:52 ` Marco Atzeri 2020-12-15 12:06 ` Corinna Vinschen 0 siblings, 2 replies; 7+ messages in thread From: Mark Geisert @ 2020-12-15 8:06 UTC (permalink / raw) To: cygwin-developers Hi Corinna, Corinna Vinschen via Cygwin-developers wrote: > On Dec 14 02:42, Mark Geisert wrote: [...] >> Debugging Python's os.uname showed Cygwin's uname() being called with a >> 'struct uname' as defined in /usr/include/sys/utsname.h, which is fine. But >> it's the "old" pre-335 uname() interface being called, not the "new" 335+ >> interface uname_x(). Note that the famous 'mkimport' script, used when >> building the Cygwin DLL, has an arg "--replace=uname=uname_x" which I >> believe is supposed to equate the two names so the code in uname_x() is >> called whether the interface is uname_x() or uname(). That's not happening. [...] >> 'nm' shows that both uname and uname_x exist in libcygwin.a and also >> cygdll.a. And a newly-created Cygwin DLL has both functions, with different >> addresses. >> That's wrong, isn't it? > > No, it isn't. The old uname entry point is still required for old > applications built prior to API version 335. If you build a new > application it will actually call uname_x when it tries to call uname > since that's how the libcygwin.a layer translates it. > > It's a bit unfortunate that uname_x is exported by libcygwin.a > as well (in theory it should only export uname, pointing to uname_x > in the DLL), but this is how it always was, as you can see in > the 32 bit build with symbols only available there, e. g. aclcheck: > > $ nm libcygwin.a | grep aclcheck > 00000000 I __imp___aclcheck > 00000000 I __imp___aclcheck32 > 00000000 I __imp__aclcheck > 00000000 T __aclcheck > U __imp___aclcheck > 00000000 T __aclcheck32 > U __imp___aclcheck32 > 00000000 T _aclcheck > U __imp___aclcheck32 > > In retrospect, uname_x should be named _uname_x or so, with a leading > underscore, so as not to pollute the namespace, but either way, that > isn;t your problem. OK, I see. > The problem here might be that you get the old uname function if > you dlopen the cygwin dll and dlsym(hdl, "uname"). Is that the > case in python? Yes it is. > If so, I have a simple, dirty workaround below. Can you check if that's > the problem, please? A new Cygwin DLL built with your patch does correct this 'uname' issue when building Python. Wonderful! Thank you very much! ..mark ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Failure during build of Python 3.8 via cygport 2020-12-15 8:06 ` Mark Geisert @ 2020-12-15 8:52 ` Marco Atzeri 2020-12-15 9:07 ` Mark Geisert 2020-12-15 12:06 ` Corinna Vinschen 1 sibling, 1 reply; 7+ messages in thread From: Marco Atzeri @ 2020-12-15 8:52 UTC (permalink / raw) To: cygwin-developers On 15.12.2020 09:06, Mark Geisert wrote: > Hi Corinna, > > Corinna Vinschen via Cygwin-developers wrote: >> On Dec 14 02:42, Mark Geisert wrote: > [...] >> >> In retrospect, uname_x should be named _uname_x or so, with a leading >> underscore, so as not to pollute the namespace, but either way, that >> isn;t your problem. > > OK, I see. > >> The problem here might be that you get the old uname function if >> you dlopen the cygwin dll and dlsym(hdl, "uname"). Is that the >> case in python? > > Yes it is. > >> If so, I have a simple, dirty workaround below. Can you check if that's >> the problem, please? > > A new Cygwin DLL built with your patch does correct this 'uname' issue > when building Python. Wonderful! > Thank you very much! > > ..mark It seems I do not need to rebuild python for this reason. Correct ? ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Failure during build of Python 3.8 via cygport 2020-12-15 8:52 ` Marco Atzeri @ 2020-12-15 9:07 ` Mark Geisert 0 siblings, 0 replies; 7+ messages in thread From: Mark Geisert @ 2020-12-15 9:07 UTC (permalink / raw) To: cygwin-developers Marco Atzeri via Cygwin-developers wrote: > On 15.12.2020 09:06, Mark Geisert wrote: >> Hi Corinna, >> >> Corinna Vinschen via Cygwin-developers wrote: >>> On Dec 14 02:42, Mark Geisert wrote: >> [...] >>> >>> In retrospect, uname_x should be named _uname_x or so, with a leading >>> underscore, so as not to pollute the namespace, but either way, that >>> isn;t your problem. >> >> OK, I see. >> >>> The problem here might be that you get the old uname function if >>> you dlopen the cygwin dll and dlsym(hdl, "uname"). Is that the >>> case in python? >> >> Yes it is. >> >>> If so, I have a simple, dirty workaround below. Can you check if that's >>> the problem, please? >> >> A new Cygwin DLL built with your patch does correct this 'uname' issue when >> building Python. Wonderful! >> Thank you very much! >> >> ..mark > > It seems I do not need to rebuild python for this reason. Correct ? That's correct Marco. But it allows me to build Python locally so I can produce patch(es) for the test_asyncore.py issue involving AF_UNIX sockets. When I've got a patch I'll follow up on the cygwin-apps list, I guess. Thanks for keeping an eye on this. ..mark ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Failure during build of Python 3.8 via cygport 2020-12-15 8:06 ` Mark Geisert 2020-12-15 8:52 ` Marco Atzeri @ 2020-12-15 12:06 ` Corinna Vinschen 2020-12-15 12:14 ` Corinna Vinschen 1 sibling, 1 reply; 7+ messages in thread From: Corinna Vinschen @ 2020-12-15 12:06 UTC (permalink / raw) To: cygwin-developers On Dec 15 00:06, Mark Geisert wrote: > Hi Corinna, > > Corinna Vinschen via Cygwin-developers wrote: > > On Dec 14 02:42, Mark Geisert wrote: > [...] > > > Debugging Python's os.uname showed Cygwin's uname() being called with a > > > 'struct uname' as defined in /usr/include/sys/utsname.h, which is fine. But > > > it's the "old" pre-335 uname() interface being called, not the "new" 335+ > > > interface uname_x(). Note that the famous 'mkimport' script, used when > > > building the Cygwin DLL, has an arg "--replace=uname=uname_x" which I > > > believe is supposed to equate the two names so the code in uname_x() is > > > called whether the interface is uname_x() or uname(). That's not happening. > [...] > > > 'nm' shows that both uname and uname_x exist in libcygwin.a and also > > > cygdll.a. And a newly-created Cygwin DLL has both functions, with different > > > addresses. > > > That's wrong, isn't it? > > > > No, it isn't. The old uname entry point is still required for old > > applications built prior to API version 335. If you build a new > > application it will actually call uname_x when it tries to call uname > > since that's how the libcygwin.a layer translates it. > > > > It's a bit unfortunate that uname_x is exported by libcygwin.a > > as well (in theory it should only export uname, pointing to uname_x > > in the DLL), but this is how it always was, as you can see in > > the 32 bit build with symbols only available there, e. g. aclcheck: > > > > $ nm libcygwin.a | grep aclcheck > > 00000000 I __imp___aclcheck > > 00000000 I __imp___aclcheck32 > > 00000000 I __imp__aclcheck > > 00000000 T __aclcheck > > U __imp___aclcheck > > 00000000 T __aclcheck32 > > U __imp___aclcheck32 > > 00000000 T _aclcheck > > U __imp___aclcheck32 > > > > In retrospect, uname_x should be named _uname_x or so, with a leading > > underscore, so as not to pollute the namespace, but either way, that > > isn;t your problem. > > OK, I see. > > > The problem here might be that you get the old uname function if > > you dlopen the cygwin dll and dlsym(hdl, "uname"). Is that the > > case in python? > > Yes it is. > > > If so, I have a simple, dirty workaround below. Can you check if that's > > the problem, please? > > A new Cygwin DLL built with your patch does correct this 'uname' issue when > building Python. Wonderful! > Thank you very much! Great, thanks for testing and confirming! I pushed the patch. Corinna ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Failure during build of Python 3.8 via cygport 2020-12-15 12:06 ` Corinna Vinschen @ 2020-12-15 12:14 ` Corinna Vinschen 0 siblings, 0 replies; 7+ messages in thread From: Corinna Vinschen @ 2020-12-15 12:14 UTC (permalink / raw) To: cygwin-developers On Dec 15 13:06, Corinna Vinschen via Cygwin-developers wrote: > On Dec 15 00:06, Mark Geisert wrote: > > Hi Corinna, > > > > Corinna Vinschen via Cygwin-developers wrote: > > > On Dec 14 02:42, Mark Geisert wrote: > > [...] > > > > Debugging Python's os.uname showed Cygwin's uname() being called with a > > > > 'struct uname' as defined in /usr/include/sys/utsname.h, which is fine. But > > > > it's the "old" pre-335 uname() interface being called, not the "new" 335+ > > > > interface uname_x(). Note that the famous 'mkimport' script, used when > > > > building the Cygwin DLL, has an arg "--replace=uname=uname_x" which I > > > > believe is supposed to equate the two names so the code in uname_x() is > > > > called whether the interface is uname_x() or uname(). That's not happening. > > [...] > > > > 'nm' shows that both uname and uname_x exist in libcygwin.a and also > > > > cygdll.a. And a newly-created Cygwin DLL has both functions, with different > > > > addresses. > > > > That's wrong, isn't it? > > > > > > No, it isn't. The old uname entry point is still required for old > > > applications built prior to API version 335. If you build a new > > > application it will actually call uname_x when it tries to call uname > > > since that's how the libcygwin.a layer translates it. > > > > > > It's a bit unfortunate that uname_x is exported by libcygwin.a > > > as well (in theory it should only export uname, pointing to uname_x > > > in the DLL), but this is how it always was, as you can see in > > > the 32 bit build with symbols only available there, e. g. aclcheck: > > > > > > $ nm libcygwin.a | grep aclcheck > > > 00000000 I __imp___aclcheck > > > 00000000 I __imp___aclcheck32 > > > 00000000 I __imp__aclcheck > > > 00000000 T __aclcheck > > > U __imp___aclcheck > > > 00000000 T __aclcheck32 > > > U __imp___aclcheck32 > > > 00000000 T _aclcheck > > > U __imp___aclcheck32 > > > > > > In retrospect, uname_x should be named _uname_x or so, with a leading > > > underscore, so as not to pollute the namespace, but either way, that > > > isn;t your problem. > > > > OK, I see. > > > > > The problem here might be that you get the old uname function if > > > you dlopen the cygwin dll and dlsym(hdl, "uname"). Is that the > > > case in python? > > > > Yes it is. > > > > > If so, I have a simple, dirty workaround below. Can you check if that's > > > the problem, please? > > > > A new Cygwin DLL built with your patch does correct this 'uname' issue when > > building Python. Wonderful! > > Thank you very much! > > Great, thanks for testing and confirming! I pushed the patch. I created new developer snapshots. Corinna ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-12-15 12:14 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <a7bcec5f-b01b-c1e4-81e6-1990f11e351e@maxrnd.com> [not found] ` <Pine.BSF.4.63.2012112223120.53260@m0.truegem.net> [not found] ` <758d2138-587b-2970-6c35-69d5c655a598@maxrnd.com> 2020-12-14 10:42 ` Failure during build of Python 3.8 via cygport Mark Geisert 2020-12-14 11:33 ` Corinna Vinschen 2020-12-15 8:06 ` Mark Geisert 2020-12-15 8:52 ` Marco Atzeri 2020-12-15 9:07 ` Mark Geisert 2020-12-15 12:06 ` Corinna Vinschen 2020-12-15 12:14 ` Corinna Vinschen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).