* Failure during build of Python 3.8 via cygport
@ 2020-12-09 8:14 Mark Geisert
2020-12-12 6:33 ` Mark Geisert
0 siblings, 1 reply; 5+ messages in thread
From: Mark Geisert @ 2020-12-09 8:14 UTC (permalink / raw)
To: Cygwin-Apps
Hi Marco,
I was building Python locally so I can later submit a patch against it. It
appears the local python.exe was built successfully, but a later step failed with:
> ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
> if test $? -ne 0 ; then \
> echo "generate-posix-vars failed" ; \
> rm -f ./pybuilddir.txt ; \
> exit 1 ; \
> fi
> Traceback (most recent call last):
> File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py", line 194, in _run_module_as_main
> return _run_code(code, main_globals, None,
> File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py", line 87, in _run_code
> exec(code, run_globals)
> File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", line 711, in <module>
> _main()
> File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", line 699, in _main
> _generate_posix_vars()
> File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", line 416, in _generate_posix_vars
> f.write(pybuilddir)
> UnicodeEncodeError: 'utf-8' codec can't encode characters in position 17-19: surrogates not allowed
> generate-posix-vars failed
> make: *** [Makefile:592: pybuilddir.txt] Error 1
> *** ERROR: make failed
I've seen UnicodeEncodeError before and searched and found how to fix it.. but
hitting the issue while building Python itself seems more fraught. Is this a
known issue with known fix?
Thank you,
..mark
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Failure during build of Python 3.8 via cygport
2020-12-09 8:14 Failure during build of Python 3.8 via cygport Mark Geisert
@ 2020-12-12 6:33 ` Mark Geisert
2020-12-13 8:52 ` Mark Geisert
0 siblings, 1 reply; 5+ messages in thread
From: Mark Geisert @ 2020-12-12 6:33 UTC (permalink / raw)
To: Cygwin-Apps
[replying to myself again...]
A similar problem happens when building 3.6 and 3.7 too. Details at end.
On Wed, 9 Dec 2020, Mark Geisert wrote:
> Hi Marco,
> I was building Python locally so I can later submit a patch against it. It
> appears the local python.exe was built successfully, but a later step failed
> with:
>
>> ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
>> if test $? -ne 0 ; then \
>> echo "generate-posix-vars failed" ; \
>> rm -f ./pybuilddir.txt ; \
>> exit 1 ; \
>> fi
>> Traceback (most recent call last):
>> File
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py",
>> line 194, in _run_module_as_main
>> return _run_code(code, main_globals, None,
>> File
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py",
>> line 87, in _run_code
>> exec(code, run_globals)
>> File
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py",
>> line 711, in <module>
>> _main()
>> File
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py",
>> line 699, in _main
>> _generate_posix_vars()
>> File
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py",
>> line 416, in _generate_posix_vars
>> f.write(pybuilddir)
>> UnicodeEncodeError: 'utf-8' codec can't encode characters in position
>> 17-19: surrogates not allowed
>> generate-posix-vars failed
>> make: *** [Makefile:592: pybuilddir.txt] Error 1
>> *** ERROR: make failed
>
> I've seen UnicodeEncodeError before and searched and found how to fix it..
> but hitting the issue while building Python itself seems more fraught. Is
> this a known issue with known fix?
This seems to be a problem setting up a platform-specific build directory.
The sysconfig.py script wants to use "lib." + platform + pythonversion but
the platform string somehow gets corrupted into non-utf8 bytes. For
instance, building Python 3.8 comes up with:
lib.cygwin-\365\377\377o-\377o-3.8
as the directory name. Broken, but could work. The build failure happens
because the script tries to write this directory name into a file but it's
not a valid utf8 string. The directory name should have been:
lib.cygwin-3.2.0-x86_64-3.8
I'm trying to debug further, learning Python as I go. Whee....
..mark
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Failure during build of Python 3.8 via cygport
2020-12-12 6:33 ` Mark Geisert
@ 2020-12-13 8:52 ` Mark Geisert
2020-12-13 14:34 ` Marco Atzeri
0 siblings, 1 reply; 5+ messages in thread
From: Mark Geisert @ 2020-12-13 8:52 UTC (permalink / raw)
To: Cygwin-Apps
Mark Geisert wrote:
> This seems to be a problem setting up a platform-specific build directory. The
> sysconfig.py script wants to use "lib." + platform + pythonversion but the
> platform string somehow gets corrupted into non-utf8 bytes. For instance,
> building Python 3.8 comes up with:
> lib.cygwin-\365\377\377o-\377o-3.8
> as the directory name. Broken, but could work. The build failure happens because
> the script tries to write this directory name into a file but it's not a valid
> utf8 string. The directory name should have been:
> lib.cygwin-3.2.0-x86_64-3.8
And the corruption is due to something about a recent change to the operation of
Cygwin's uname() function. The change was introduced in Cygwin API version 335;
I'm running 340 on my test machine. This being a fairly recent change might
possibly explain why nobody else has run into this issue yet.
Basically, os.uname within Python is calling Cygwin's uname() passing the address
of a buffer declared to be 'struct utsname'. The structure layout changed in API
335. What I've hit is a mismatch between what Python expects and Cygwin delivers.
I'll move this discussion over to the developers list tomorrow.
..mark
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Failure during build of Python 3.8 via cygport
2020-12-13 8:52 ` Mark Geisert
@ 2020-12-13 14:34 ` Marco Atzeri
2020-12-13 21:59 ` Brian Inglis
0 siblings, 1 reply; 5+ messages in thread
From: Marco Atzeri @ 2020-12-13 14:34 UTC (permalink / raw)
To: cygwin-apps
On 13.12.2020 09:52, Mark Geisert wrote:
> Mark Geisert wrote:
>> This seems to be a problem setting up a platform-specific build
>> directory. The sysconfig.py script wants to use "lib." + platform +
>> pythonversion but the platform string somehow gets corrupted into
>> non-utf8 bytes. For instance, building Python 3.8 comes up with:
>> lib.cygwin-\365\377\377o-\377o-3.8
>> as the directory name. Broken, but could work. The build failure
>> happens because the script tries to write this directory name into a
>> file but it's not a valid utf8 string. The directory name should have
>> been:
>> lib.cygwin-3.2.0-x86_64-3.8
>
> And the corruption is due to something about a recent change to the
> operation of Cygwin's uname() function. The change was introduced in
> Cygwin API version 335; I'm running 340 on my test machine. This being
> a fairly recent change might possibly explain why nobody else has run
> into this issue yet.
>
> Basically, os.uname within Python is calling Cygwin's uname() passing
> the address of a buffer declared to be 'struct utsname'. The structure
> layout changed in API 335. What I've hit is a mismatch between what
> Python expects and Cygwin delivers.
>
> I'll move this discussion over to the developers list tomorrow.
>
> ..mark
thanks for the analysis
let me know if you think we should correct the python build in any way
Regards
Marco
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Failure during build of Python 3.8 via cygport
2020-12-13 14:34 ` Marco Atzeri
@ 2020-12-13 21:59 ` Brian Inglis
0 siblings, 0 replies; 5+ messages in thread
From: Brian Inglis @ 2020-12-13 21:59 UTC (permalink / raw)
To: cygwin-apps
On 2020-12-13 07:34, Marco Atzeri via Cygwin-apps wrote:
> On 13.12.2020 09:52, Mark Geisert wrote:
>> Mark Geisert wrote:
>>> This seems to be a problem setting up a platform-specific build directory.
>>> The sysconfig.py script wants to use "lib." + platform + pythonversion but
>>> the platform string somehow gets corrupted into non-utf8 bytes. For
>>> instance, building Python 3.8 comes up with:
>>> lib.cygwin-\365\377\377o-\377o-3.8
>>> as the directory name. Broken, but could work. The build failure happens
>>> because the script tries to write this directory name into a file but it's
>>> not a valid utf8 string. The directory name should have been:
>>> lib.cygwin-3.2.0-x86_64-3.8
>>
>> And the corruption is due to something about a recent change to the operation
>> of Cygwin's uname() function. The change was introduced in Cygwin API version
>> 335; I'm running 340 on my test machine. This being a fairly recent change
>> might possibly explain why nobody else has run into this issue yet.
That was nearly two years ago.
>> Basically, os.uname within Python is calling Cygwin's uname() passing the
>> address of a buffer declared to be 'struct utsname'. The structure layout
>> changed in API 335. What I've hit is a mismatch between what Python expects
>> and Cygwin delivers.
>>
>> I'll move this discussion over to the developers list tomorrow.
uname(2):
NOTES
...
The length of the fields in the struct varies. Some operating
systems or libraries use a hardcoded 9 or 33 or 65 or 257. Other
systems use *SYS_NMLN* or *_SYS_NMLN* or *UTSLEN* or *_UTSNAME_LENGTH*.
Clearly, it is a bad idea to use any of these constants; just use
*sizeof*(...). Often 257 is chosen in order to have room for an
internet hostname.
...
C library/kernel differences
Over time, increases in the size of the /utsname/ structure have led to
three successive versions of *uname*():
/sys_olduname/() (slot /__NR_oldolduname/),
/sys_uname/() (slot /__NR_olduname/), and
/sys_newuname/() (slot /__NR_uname/).
The first one used length 9 for all fields;
the second used 65;
the third also uses 65 but adds the domainname field.
The glibc *uname*() wrapper function hides these details from
applications, invoking the most recent version of the system call
provided by the kernel.
> thanks for the analysis
>
> let me know if you think we should correct the python build in any way
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-12-13 22:00 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-09 8:14 Failure during build of Python 3.8 via cygport Mark Geisert
2020-12-12 6:33 ` Mark Geisert
2020-12-13 8:52 ` Mark Geisert
2020-12-13 14:34 ` Marco Atzeri
2020-12-13 21:59 ` Brian Inglis
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).