Failure during build of Python 3.8 via cygport

Failure during build of Python 3.8 via cygport

[Mark Geisert]

Hi Marco,
I was building Python locally so I can later submit a patch against it.  It 
appears the local python.exe was built successfully, but a later step failed with:

> ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
> if test $? -ne 0 ; then \
>         echo "generate-posix-vars failed" ; \
>         rm -f ./pybuilddir.txt ; \
>         exit 1 ; \
> fi
> Traceback (most recent call last):
>   File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py", line 194, in _run_module_as_main
>     return _run_code(code, main_globals, None,
>   File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py", line 87, in _run_code
>     exec(code, run_globals)
>   File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", line 711, in <module>
>     _main()
>   File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", line 699, in _main
>     _generate_posix_vars()
>   File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", line 416, in _generate_posix_vars
>     f.write(pybuilddir)
> UnicodeEncodeError: 'utf-8' codec can't encode characters in position 17-19: surrogates not allowed
> generate-posix-vars failed
> make: *** [Makefile:592: pybuilddir.txt] Error 1
> *** ERROR: make failed

I've seen UnicodeEncodeError before and searched and found how to fix it.. but 
hitting the issue while building Python itself seems more fraught.  Is this a 
known issue with known fix?
Thank you,

..mark

Re: Failure during build of Python 3.8 via cygport

[Mark Geisert]

[replying to myself again...]

A similar problem happens when building 3.6 and 3.7 too.  Details at end.

On Wed, 9 Dec 2020, Mark Geisert wrote:
> Hi Marco,
> I was building Python locally so I can later submit a patch against it.  It 
> appears the local python.exe was built successfully, but a later step failed 
> with:
>
>> ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
>> if test $? -ne 0 ; then \
>>         echo "generate-posix-vars failed" ; \
>>         rm -f ./pybuilddir.txt ; \
>>         exit 1 ; \
>> fi
>> Traceback (most recent call last):
>>   File 
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py", 
>> line 194, in _run_module_as_main
>>     return _run_code(code, main_globals, None,
>>   File 
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py", 
>> line 87, in _run_code
>>     exec(code, run_globals)
>>   File 
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", 
>> line 711, in <module>
>>     _main()
>>   File 
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", 
>> line 699, in _main
>>     _generate_posix_vars()
>>   File 
>> "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", 
>> line 416, in _generate_posix_vars
>>     f.write(pybuilddir)
>> UnicodeEncodeError: 'utf-8' codec can't encode characters in position 
>> 17-19: surrogates not allowed
>> generate-posix-vars failed
>> make: *** [Makefile:592: pybuilddir.txt] Error 1
>> *** ERROR: make failed
>
> I've seen UnicodeEncodeError before and searched and found how to fix it.. 
> but hitting the issue while building Python itself seems more fraught.  Is 
> this a known issue with known fix?

This seems to be a problem setting up a platform-specific build directory. 
The sysconfig.py script wants to use "lib." + platform + pythonversion but 
the platform string somehow gets corrupted into non-utf8 bytes.  For 
instance, building Python 3.8 comes up with:
     lib.cygwin-\365\377\377o-\377o-3.8
as the directory name.  Broken, but could work.  The build failure happens 
because the script tries to write this directory name into a file but it's 
not a valid utf8 string.  The directory name should have been:
     lib.cygwin-3.2.0-x86_64-3.8

I'm trying to debug further, learning Python as I go.  Whee....

..mark

Re: Failure during build of Python 3.8 via cygport

[Mark Geisert]

Mark Geisert wrote:
> This seems to be a problem setting up a platform-specific build directory. The 
> sysconfig.py script wants to use "lib." + platform + pythonversion but the 
> platform string somehow gets corrupted into non-utf8 bytes.  For instance, 
> building Python 3.8 comes up with:
>      lib.cygwin-\365\377\377o-\377o-3.8
> as the directory name.  Broken, but could work.  The build failure happens because 
> the script tries to write this directory name into a file but it's not a valid 
> utf8 string.  The directory name should have been:
>      lib.cygwin-3.2.0-x86_64-3.8

And the corruption is due to something about a recent change to the operation of 
Cygwin's uname() function.  The change was introduced in Cygwin API version 335; 
I'm running 340 on my test machine.  This being a fairly recent change might 
possibly explain why nobody else has run into this issue yet.

Basically, os.uname within Python is calling Cygwin's uname() passing the address 
of a buffer declared to be 'struct utsname'.  The structure layout changed in API 
335.  What I've hit is a mismatch between what Python expects and Cygwin delivers.

I'll move this discussion over to the developers list tomorrow.

..mark

Re: Failure during build of Python 3.8 via cygport

[Marco Atzeri]

On 13.12.2020 09:52, Mark Geisert wrote:
> Mark Geisert wrote:
>> This seems to be a problem setting up a platform-specific build 
>> directory. The sysconfig.py script wants to use "lib." + platform + 
>> pythonversion but the platform string somehow gets corrupted into 
>> non-utf8 bytes.  For instance, building Python 3.8 comes up with:
>>      lib.cygwin-\365\377\377o-\377o-3.8
>> as the directory name.  Broken, but could work.  The build failure 
>> happens because the script tries to write this directory name into a 
>> file but it's not a valid utf8 string.  The directory name should have 
>> been:
>>      lib.cygwin-3.2.0-x86_64-3.8
> 
> And the corruption is due to something about a recent change to the 
> operation of Cygwin's uname() function.  The change was introduced in 
> Cygwin API version 335; I'm running 340 on my test machine.  This being 
> a fairly recent change might possibly explain why nobody else has run 
> into this issue yet.
> 
> Basically, os.uname within Python is calling Cygwin's uname() passing 
> the address of a buffer declared to be 'struct utsname'.  The structure 
> layout changed in API 335.  What I've hit is a mismatch between what 
> Python expects and Cygwin delivers.
> 
> I'll move this discussion over to the developers list tomorrow.
> 
> ..mark

thanks for the analysis

let me know if you think we should correct the python build in any way

Regards
Marco

Re: Failure during build of Python 3.8 via cygport

[Brian Inglis]

On 2020-12-13 07:34, Marco Atzeri via Cygwin-apps wrote:
> On 13.12.2020 09:52, Mark Geisert wrote:
>> Mark Geisert wrote:
>>> This seems to be a problem setting up a platform-specific build directory. 
>>> The sysconfig.py script wants to use "lib." + platform + pythonversion but 
>>> the platform string somehow gets corrupted into non-utf8 bytes.  For 
>>> instance, building Python 3.8 comes up with:
>>>      lib.cygwin-\365\377\377o-\377o-3.8
>>> as the directory name.  Broken, but could work.  The build failure happens 
>>> because the script tries to write this directory name into a file but it's 
>>> not a valid utf8 string.  The directory name should have been:
>>>      lib.cygwin-3.2.0-x86_64-3.8
>>
>> And the corruption is due to something about a recent change to the operation 
>> of Cygwin's uname() function.  The change was introduced in Cygwin API version 
>> 335; I'm running 340 on my test machine.  This being a fairly recent change 
>> might possibly explain why nobody else has run into this issue yet.

That was nearly two years ago.

>> Basically, os.uname within Python is calling Cygwin's uname() passing the 
>> address of a buffer declared to be 'struct utsname'.  The structure layout 
>> changed in API 335.  What I've hit is a mismatch between what Python expects 
>> and Cygwin delivers.
>>
>> I'll move this discussion over to the developers list tomorrow.

uname(2):
NOTES
...
        The length of the fields in the struct varies.  Some operating
        systems or libraries use a hardcoded 9 or 33 or 65 or 257.  Other
        systems use *SYS_NMLN* or *_SYS_NMLN* or *UTSLEN* or *_UTSNAME_LENGTH*.
        Clearly, it is a bad idea to use any of these constants; just use
        *sizeof*(...). Often 257 is chosen in order to have room for an
        internet hostname.
...
    C library/kernel differences
        Over time, increases in the size of the /utsname/ structure have led to
        three successive versions of *uname*():
		/sys_olduname/()	(slot /__NR_oldolduname/),
		/sys_uname/()		(slot /__NR_olduname/), and
        		/sys_newuname/()	(slot /__NR_uname/).
        The first one used length 9 for all fields;
        the second used 65;
        the third also uses 65 but adds the domainname field.
        The glibc *uname*() wrapper function hides these details from
        applications, invoking the most recent version of the system call
        provided by the kernel.

> thanks for the analysis
> 
> let me know if you think we should correct the python build in any way

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]