Moving from python2 to python3

Moving from python2 to python3

[Ken Brown]

This is a followup to a thread with subject "[PATCH] doc: Various fixes to 
makedocbook for python3.8" that began on the newlib list and then moved to

   https://cygwin.com/pipermail/cygwin-apps/2020-August/040442.html,

where it evolved into a discussion of moving from python2 to python3.  A recent 
post on the cygwin list reminded me that we never finished that discussion.  We 
left it with the question of what to do about scripts with a shebang that names 
python but might not work with python3.

With python2 reaching EOL in a month, it seems that we ought to finish that 
discussion and make the move.

Ken

Re: Moving from python2 to python3

[Marco Atzeri]

On 28.11.2020 19:11, Ken Brown via Cygwin-apps wrote:
> This is a followup to a thread with subject "[PATCH] doc: Various fixes 
> to makedocbook for python3.8" that began on the newlib list and then 
> moved to
> 
>    https://cygwin.com/pipermail/cygwin-apps/2020-August/040442.html,
> 
> where it evolved into a discussion of moving from python2 to python3.  A 
> recent post on the cygwin list reminded me that we never finished that 
> discussion.  We left it with the question of what to do about scripts 
> with a shebang that names python but might not work with python3.
> 
> With python2 reaching EOL in a month, it seems that we ought to finish 
> that discussion and make the move.
> 
> Ken

working slowing on it

I plan to use alternatives for both python and python3,
so that by default they will both point to latest python 3.x installed,
but you could use alternatives to reverse to any version

probably I will release a test version of all

python-2.18
python-3.6
python-3.7
python-3.8

before going in full

Regards
Marco

Re: Moving from python2 to python3

[Marco Atzeri]

On 04.12.2020 22:04, Marco Atzeri wrote:
> On 28.11.2020 19:11, Ken Brown via Cygwin-apps wrote:

> probably I will release a test version of all
> 
> python-2.18
> python-3.6
> python-3.7
> python-3.8
> 
> before going in full
> 
> Regards
> Marco

all python packages have been updated.

/usr/bin/python
/usr/bin/python3

both point now to python3.8 as default using alternatives.

I will send a separate announcement shortly

Regards
Marco

Re: Moving from python2 to python3

[Achim Gratz]

Marco Atzeri via Cygwin-apps writes:
> all python packages have been updated.
>
> /usr/bin/python
> /usr/bin/python3
>
> both point now to python3.8 as default using alternatives.

You broke the CI:

Traceback (most recent call last):
  File "./scallywag", line 69, in <module>
    response = urllib.request.urlopen(req, json.dumps(data).encode('utf-8'))
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1379, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/lib/python3.8/http/client.py", line 923, in connect
    self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
AttributeError: module 'socket' has no attribute 'TCP_NODELAY'

> I will send a separate announcement shortly

You'd better fix that error first.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Moving from python2 to python3

[Marco Atzeri]

On 29.12.2020 13:04, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>> all python packages have been updated.
>>
>> /usr/bin/python
>> /usr/bin/python3
>>
>> both point now to python3.8 as default using alternatives.
> 
> You broke the CI:
> 

>      self.connect()
>    File "/usr/lib/python3.8/http/client.py", line 923, in connect
>      self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
> AttributeError: module 'socket' has no attribute 'TCP_NODELAY'
> 

In reality I have not changed anything, except Mark's patch,
that seems not the cause of it, so this problem is caused by
something sneaking.

 >> I will send a separate announcement shortly
> 
> You'd better fix that error first.

I can agree

> Regards,
> Achim.

Thanks
Marco

Re: Moving from python2 to python3

[Achim Gratz]

Marco Atzeri via Cygwin-apps writes:
>> You'd better fix that error first.
>
> I can agree

AFAICS this is still the current package version.  You might want top
move it to test whilem you build the new packages, then replace that
version with the new release later on.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Moving from python2 to python3

[Marco Atzeri]

On 29.12.2020 17:42, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>>> You'd better fix that error first.
>>
>> I can agree
> 
> AFAICS this is still the current package version.  You might want top
> move it to test whilem you build the new packages, then replace that
> version with the new release later on.

they are in test now


> Regards,
> Achim.

Re: Moving from python2 to python3

[Marco Atzeri]

On 29.12.2020 18:02, Marco Atzeri wrote:
> On 29.12.2020 17:42, Achim Gratz wrote:
>> Marco Atzeri via Cygwin-apps writes:
>>>> You'd better fix that error first.
>>>
>>> I can agree
>>
>> AFAICS this is still the current package version.  You might want top
>> move it to test whilem you build the new packages, then replace that
>> version with the new release later on.
> 
> they are in test now
> 
> 
>> Regards,
>> Achim.
>  

Removed now, and new versions are up as test for
python36, 37, 38 and pip

I think they are fine, but I will wait your feedback

Regards
Marco

Re: Moving from python2 to python3

[Achim Gratz]

Marco Atzeri via Cygwin-apps writes:
> Removed now, and new versions are up as test for
> python36, 37, 38 and pip

That's not what I'm seeing at the moment.  Package python3 wants to
update to 3.8.6-1 and python38 tries to install 3.8.3-1, that doesn't
sound right.  I'm not sure I am seeing the latest, though (I've updated
from kernel.org).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Moving from python2 to python3

[Marco Atzeri]

On 29.12.2020 21:57, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>> Removed now, and new versions are up as test for
>> python36, 37, 38 and pip
> 
> That's not what I'm seeing at the moment.  Package python3 wants to
> update to 3.8.6-1 and python38 tries to install 3.8.3-1, that doesn't
> sound right.  I'm not sure I am seeing the latest, though (I've updated
> from kernel.org).

python3 is almost empty and pulls python38

python38 3.8.6-2 should be available as test
3.8.3-1 is last stable

> 
> Regards,
> Achim.
> 

Re: Moving from python2 to python3

[Achim Gratz]

Marco Atzeri via Cygwin-apps writes:
> On 29.12.2020 21:57, Achim Gratz wrote:
>> Marco Atzeri via Cygwin-apps writes:
>>> Removed now, and new versions are up as test for
>>> python36, 37, 38 and pip
>> That's not what I'm seeing at the moment.  Package python3 wants to
>> update to 3.8.6-1 and python38 tries to install 3.8.3-1, that doesn't
>> sound right.  I'm not sure I am seeing the latest, though (I've updated
>> from kernel.org).
>
> python3 is almost empty and pulls python38

Yes, but it is at version 3.8.6…

> python38 3.8.6-2 should be available as test
> 3.8.3-1 is last stable

I was concerned about that apparent mixing of Python releases and
unfortunately it seems I was right:

https://ci.appveyor.com/project/cygwin/scallywag/builds/37043802


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Moving from python2 to python3

[Marco Atzeri]

On 30.12.2020 08:21, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>> On 29.12.2020 21:57, Achim Gratz wrote:
>>> Marco Atzeri via Cygwin-apps writes:
>>>> Removed now, and new versions are up as test for
>>>> python36, 37, 38 and pip
>>> That's not what I'm seeing at the moment.  Package python3 wants to
>>> update to 3.8.6-1 and python38 tries to install 3.8.3-1, that doesn't
>>> sound right.  I'm not sure I am seeing the latest, though (I've updated
>>> from kernel.org).
>>
>> python3 is almost empty and pulls python38
> 
> Yes, but it is at version 3.8.6…
> 
>> python38 3.8.6-2 should be available as test
>> 3.8.3-1 is last stable
> 
> I was concerned about that apparent mixing of Python releases and
> unfortunately it seems I was right:
> 
> https://ci.appveyor.com/project/cygwin/scallywag/builds/37043802

it seems the appveyor is not running the postinstall scripts

$ ls -l /usr/bin/python3
lrwxrwxrwx 1 Marco Kein 25 Dec 29 20:49 /usr/bin/python3 -> 
/etc/alternatives/python3

$ alternatives --display python3
python3 - status is auto.
  link currently points to /usr/bin/python3.8
/usr/bin/python3.8 - priority 38
/usr/bin/python3.6 - priority 36
/usr/bin/python3.7 - priority 37
Current `best' version is /usr/bin/python3.8.


> Regards,
> Achim.
> 

Re: Moving from python2 to python3

[ASSI]

Marco Atzeri via Cygwin-apps writes:
> it seems the appveyor is not running the postinstall scripts

It does run all the other ones, though… so there's something amiss with
the logic that triggers that script.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Moving from python2 to python3

[Ken Brown]

On 12/30/2020 9:10 AM, ASSI wrote:
> Marco Atzeri via Cygwin-apps writes:
>> it seems the appveyor is not running the postinstall scripts
> 
> It does run all the other ones, though… so there's something amiss with
> the logic that triggers that script.

python38-3.8.3-1 is missing its postinstall script.  All I see is a 
cygport-generated /etc/postinstall/python38.sh, packaged with idle38-3.8.3-1.

Ken

Re: Moving from python2 to python3

[Marco Atzeri]

On 30.12.2020 16:41, Ken Brown via Cygwin-apps wrote:
> On 12/30/2020 9:10 AM, ASSI wrote:
>> Marco Atzeri via Cygwin-apps writes:
>>> it seems the appveyor is not running the postinstall scripts
>>
>> It does run all the other ones, though… so there's something amiss with
>> the logic that triggers that script.
> 
> python38-3.8.3-1 is missing its postinstall script.  All I see is a 
> cygport-generated /etc/postinstall/python38.sh, packaged with 
> idle38-3.8.3-1.
> 
> Ken

I was referring to python38-3.8.6-2

$ cygcheck -l python38 |grep "^/etc"
/etc/postinstall/python38.sh
/etc/preremove/python38.sh

$ cygcheck -l idle38 |grep "^/etc"
/etc/postinstall/idle38.sh
/etc/preremove/idle38.sh

python27-2.7.18-2 has it wrongly placed. I am uploading a version -3

Re: Moving from python2 to python3

[Ken Brown]

On 12/29/2020 7:04 AM, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>> all python packages have been updated.
>>
>> /usr/bin/python
>> /usr/bin/python3
>>
>> both point now to python3.8 as default using alternatives.
> 
> You broke the CI:
> 
> Traceback (most recent call last):
>    File "./scallywag", line 69, in <module>
>      response = urllib.request.urlopen(req, json.dumps(data).encode('utf-8'))
>    File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
>      return opener.open(url, data, timeout)
>    File "/usr/lib/python3.8/urllib/request.py", line 525, in open
>      response = self._open(req, data)
>    File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
>      result = self._call_chain(self.handle_open, protocol, protocol +
>    File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
>      result = func(*args)
>    File "/usr/lib/python3.8/urllib/request.py", line 1379, in http_open
>      return self.do_open(http.client.HTTPConnection, req)
>    File "/usr/lib/python3.8/urllib/request.py", line 1350, in do_open
>      h.request(req.get_method(), req.selector, req.data, headers,
>    File "/usr/lib/python3.8/http/client.py", line 1255, in request
>      self._send_request(method, url, body, headers, encode_chunked)
>    File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
>      self.endheaders(body, encode_chunked=encode_chunked)
>    File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
>      self._send_output(message_body, encode_chunked=encode_chunked)
>    File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
>      self.send(msg)
>    File "/usr/lib/python3.8/http/client.py", line 950, in send
>      self.connect()
>    File "/usr/lib/python3.8/http/client.py", line 923, in connect
>      self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
> AttributeError: module 'socket' has no attribute 'TCP_NODELAY'

An internet search led me to this:

   https://bugs.python.org/issue41374

As explained in that report, the TCP_* macros weren't defined in the build of 
python on Cygwin because <netinet/tcp.h> was not included (see 
https://github.com/python/cpython/blob/592527f3ee59616eca2bd1da771f7c14cee808d5/Modules/socketmodule.h#L11). 
  Is that still the case in the current python sources?

Ken

Re: Moving from python2 to python3

[Marco Atzeri]

On 29.12.2020 16:04, Ken Brown via Cygwin-apps wrote:
> On 12/29/2020 7:04 AM, Achim Gratz wrote:
>> Marco Atzeri via Cygwin-apps writes:
>>> all python packages have been updated.
>>>
>>> /usr/bin/python
>>> /usr/bin/python3
>>>
>>> both point now to python3.8 as default using alternatives.
>>
>> You broke the CI:
>>

> 
> An internet search led me to this:
> 
>    https://bugs.python.org/issue41374
> 
> As explained in that report, the TCP_* macros weren't defined in the 
> build of python on Cygwin because <netinet/tcp.h> was not included (see 
> https://github.com/python/cpython/blob/592527f3ee59616eca2bd1da771f7c14cee808d5/Modules/socketmodule.h#L11). 
>   Is that still the case in the current python sources?
> 
> Ken

for what I see it is still true, and looked strange.
I will update and rebuild

Thanks
Marco