rebuilding python pip from source

rebuilding python pip from source

[Doug Henderson]

I am trying to rebuild the python37-pip package from source.
Checking source in setup gives me /usr/src/python-pip-19.0.2-1.src/
When I try to build the package I get an error:

    $ cygport python-pip.cygport all
    *** ERROR: python27-pip is required to build this package

Initially I got errors saying python36-pip, etc. were required, so I
installed the binary and source packages for them, but there is no
python27-pip or python2-pip package available to install.

Ultimately I intend to completely rebuild from source both Python 2.7 and 3.7
and all the other packages named "^python.*".

Doug
--
Doug Henderson, Calgary, Alberta, Canada - from gmail.com

Re: rebuilding python pip from source

[Yaakov Selkowitz]

On Mon, 2019-03-04 at 04:57 -0700, Doug Henderson wrote:
> I am trying to rebuild the python37-pip package from source.
> Checking source in setup gives me /usr/src/python-pip-19.0.2-1.src/
> When I try to build the package I get an error:
> 
>     $ cygport python-pip.cygport all
>     *** ERROR: python27-pip is required to build this package
> 
> Initially I got errors saying python36-pip, etc. were required, so I
> installed the binary and source packages for them, but there is no
> python27-pip or python2-pip package available to install.

There is: https://cygwin.com/packages/x86_64/python27-pip/

> Ultimately I intend to completely rebuild from source both Python 2.7 and 3.7
> and all the other packages named "^python.*".

To what end?

--
Yaakov

Re: rebuilding python pip from source

[Doug Henderson]

On Mon, 4 Mar 2019 at 09:39, Yaakov Selkowitz <> wrote:
>
> On Mon, 2019-03-04 at 04:57 -0700, Doug Henderson wrote:
> > I am trying to rebuild the python37-pip package from source.
> > Checking source in setup gives me /usr/src/python-pip-19.0.2-1.src/
> > When I try to build the package I get an error:
> >
> >     $ cygport python-pip.cygport all
> >     *** ERROR: python27-pip is required to build this package
> >
> > Initially I got errors saying python36-pip, etc. were required, so I
> > installed the binary and source packages for them, but there is no
> > python27-pip or python2-pip package available to install.
>
> There is: https://cygwin.com/packages/x86_64/python27-pip/

I have that package installed. I want to be able to compile it from
source, as a small step towards my ultimate goal.

> > Ultimately I intend to completely rebuild from source both Python 2.7 and 3.7
> > and all the other packages named "^python.*".
>
> To what end?

Because that is a right granted to me by the open source license under
which some source code is released. Perhaps not explicitly by the
Python and cygwin licences, but as an implied ability.

What is the point of having a source package but to be able to rebuild
the binary package. For me, it is more a learning process than
anything else.

I understand that you are the maintainer of the main Python packages,
as well as many other related packages. Thank you for your
contributions. I must keep you very busy, so automating the process
using cygport makes a lot of sense. There are other python related
packages maintained by others, or orphaned. Starting with pip seemed a
good idea.

Previously, before python3 was available on cygwin, I rebuilt python2
from source, and also attempted to port python3. I don't think I used
cygport back then. Back in the early 80's I received gcc, emacs, X11,
sun-rpc, etc. on 9 track tape, and built them on the wide variety of
Unix workstations and servers in our office. And over the years I
built Python every where I had a C compiler but no Python. Building
software from a source distribution is not a problem for me. Currently
I maintain the cygwin expat package.

I suspect that the Python build process requires some manual steps in
addition to perhaps running cygport in small step, or even editing
the.cygport file between those steps, that are not obvious from
examining the source package. I suspect there is a recipe that you use
every time you build a new Python release. I hope you can share that
with me.

I am thinking about porting some other packages to cygwin, and Python
represents a complex system of packages that appears to make good use
of cygport. Along the way, I need a better understanding of cygport,
and how to wrap a provided build system within it.

Thanks,
Doug

-- 
Doug Henderson, Calgary, Alberta, Canada - from gmail.com

Re: rebuilding python pip from source

[Yaakov Selkowitz]

On Mon, 2019-03-04 at 11:09 -0700, Doug Henderson wrote:
> On Mon, 4 Mar 2019 at 09:39, Yaakov Selkowitz <> wrote:
> > On Mon, 2019-03-04 at 04:57 -0700, Doug Henderson wrote:
> > > I am trying to rebuild the python37-pip package from source.
> > > Checking source in setup gives me /usr/src/python-pip-19.0.2-1.src/
> > > When I try to build the package I get an error:
> > > 
> > >     $ cygport python-pip.cygport all
> > >     *** ERROR: python27-pip is required to build this package
> > > 
> > > Initially I got errors saying python36-pip, etc. were required, so I
> > > installed the binary and source packages for them, but there is no
> > > python27-pip or python2-pip package available to install.
> > 
> > There is: https://cygwin.com/packages/x86_64/python27-pip/
> 
> I have that package installed. I want to be able to compile it from
> source, as a small step towards my ultimate goal.

To be clear, to build the python-pip package from source with the
included .cygport file, you still need to have pythonXY-pip installed.

> > > Ultimately I intend to completely rebuild from source both Python 2.7 and 3.7
> > > and all the other packages named "^python.*".
> > 
> > To what end?
> 
> Because that is a right granted to me by the open source license under
> which some source code is released. Perhaps not explicitly by the
> Python and cygwin licences, but as an implied ability.
> 
> What is the point of having a source package but to be able to rebuild
> the binary package. For me, it is more a learning process than
> anything else.

Fair enough, but keep in mind that Cygwin is a binary distribution, not
a BLFS manual.

> I suspect that the Python build process requires some manual steps in
> addition to perhaps running cygport in small step, or even editing
> the.cygport file between those steps, that are not obvious from
> examining the source package. I suspect there is a recipe that you use
> every time you build a new Python release. I hope you can share that
> with me.

The .cygport file together with the cygport program in whatever state
it is at the time of building (keep in mind, as the primary author
thereof, I always use git master) is exactly how the package is built,
and how it can be rebuilt.

Now, if you're asking about how I bootstrap python-pip's build-time
requirement on pip for a new X.Y version of Python (wrt the brand new
scheme where each is supported concurrently), that would be:

1. build pythonXY with cygport and install locally
2. pythonX.Y -m ensurepip --altinstall
3. rebuild/update python-wheel with cygport and install pythonXY-wheel
locally
4. rebuild/update python-setuptools with cygport and install pythonXY-
setuptools locally
5. rebuild/update python-pip with cygport and install pythonXY-pip
locally
6. remove any cruft from step 2.
7. rebuild/update python-virtualenv with cygport

This isn't reflected in the .cygport files because exactly how pip
became present on the system it is irrelevant to the building of the
packages themselves.

> I am thinking about porting some other packages to cygwin, and Python
> represents a complex system of packages that appears to make good use
> of cygport. Along the way, I need a better understanding of cygport,
> and how to wrap a provided build system within it.

My Ports repos cover a wide variety of packages from which you may be
able to learn more:

https://github.com/cygwinports/
https://github.com/cygwinports-extras/

Also, the manual is included with the cygport package and is also
available online:

https://cygwinports.github.io/cygport/

--
Yaakov

Re: rebuilding python pip from source

[Doug Henderson]

tOn Mon, 4 Mar 2019 at 12:00, Yaakov Selkowitz <> wrote:
>
> On Mon, 2019-03-04 at 11:09 -0700, Doug Henderson wrote:
> > On Mon, 4 Mar 2019 at 09:39, Yaakov Selkowitz <> wrote:
> > > On Mon, 2019-03-04 at 04:57 -0700, Doug Henderson wrote:

> The .cygport file together with the cygport program in whatever state
> it is at the time of building (keep in mind, as the primary author
> thereof, I always use git master) is exactly how the package is built,
> and how it can be rebuilt.

Hopefully I can get by with your releases, rather than tracking the
HEAD. Do you have tags that correspond to the releases? I don't know
how to tell Github to show tags, if it can.

> Now, if you're asking about how I bootstrap python-pip's build-time
> requirement on pip for a new X.Y version of Python (wrt the brand new
> scheme where each is supported concurrently), that would be:

Yes, this is what I want to understand.

> 1. build pythonXY with cygport and install locally
> 2. pythonX.Y -m ensurepip --altinstall
> 3. rebuild/update python-wheel with cygport and install pythonXY-wheel
> locally
> 4. rebuild/update python-setuptools with cygport and install pythonXY-
> setuptools locally
> 5. rebuild/update python-pip with cygport and install pythonXY-pip
> locally
> 6. remove any cruft from step 2.
> 7. rebuild/update python-virtualenv with cygport
>
> This isn't reflected in the .cygport files because exactly how pip
> became present on the system it is irrelevant to the building of the
> packages themselves.

Thanks. This will help a lot, if I understand it correctly.

It looks like you do not clear your build directories using 'cygport
finish' after uploading new/changed packages, so a lot of
inter-package state is retained in your /usr/src tree. Is this
correct? Or am I misunderstanding the recipe you show above.

In step 2, should I be executing the just built executable:
../inst/usr/bin/python3.7m.exe ? That seems to work.


In step 3, I get:

$ pwd
/usr/src/python-wheel-0.32.3-1.src
$ cygport python-wheel.cygport prep
*** ERROR: python27-pip is required to build this package

I don't understand how to get past this point.
Am I calling cygport correctly at this point?
Do I need any environment variables?
What does this mean, besides the obvious? What kind of thing is
`python27-pip' and where should it be? Is it looking in a wheel file
somewhere? In the /usr/src/python37-3.7.2-1.src tree? In the
/usr/src/python2-2.7.15-1.src tree? Do I need to cygport pe & compile
& install and then install pip2 in that local install tree? And
likewise for each of Python 3.4, 3.5, 3.6? Or do these versions of pip
need to be installed into my system via setup.exe?

Do I need to study Gentoo's portage to get a better understanding of
how cygport works?

Sorry about all the questions. Perhaps I'm trying to move ahead too
quickly, without pouring through the source for cygport. I've found
the document at https://cygwinports.github.io/cygport/cygport_in.html
to be a bit sparse in some important places (and I know exactly how
this kind of documentation gets created, as I've done it myself a time
or two).

Thanks
Doug
-- 
Doug Henderson, Calgary, Alberta, Canada - from gmail.com

Re: rebuilding python pip from source

[Brian Inglis]

On 2019-03-04 16:07, Doug Henderson wrote:
> On Mon, 4 Mar 2019 at 12:00, Yaakov Selkowitz wrote:
>> On Mon, 2019-03-04 at 11:09 -0700, Doug Henderson wrote:
>>> On Mon, 4 Mar 2019 at 09:39, Yaakov Selkowitz wrote:
>>>> On Mon, 2019-03-04 at 04:57 -0700, Doug Henderson wrote:
>> The .cygport file together with the cygport program in whatever state
>> it is at the time of building (keep in mind, as the primary author
>> thereof, I always use git master) is exactly how the package is built,
>> and how it can be rebuilt.
> Hopefully I can get by with your releases, rather than tracking the
> HEAD. Do you have tags that correspond to the releases? I don't know
> how to tell Github to show tags, if it can.

Left button dropdowns for branches or tags or git tag --list command.

>> Now, if you're asking about how I bootstrap python-pip's build-time
>> requirement on pip for a new X.Y version of Python (wrt the brand new
>> scheme where each is supported concurrently), that would be:
> Yes, this is what I want to understand.
>> 1. build pythonXY with cygport and install locally
>> 2. pythonX.Y -m ensurepip --altinstall
>> 3. rebuild/update python-wheel with cygport and install pythonXY-wheel
>> locally
>> 4. rebuild/update python-setuptools with cygport and install pythonXY-
>> setuptools locally
>> 5. rebuild/update python-pip with cygport and install pythonXY-pip
>> locally
>> 6. remove any cruft from step 2.
>> 7. rebuild/update python-virtualenv with cygport
>> This isn't reflected in the .cygport files because exactly how pip
>> became present on the system it is irrelevant to the building of the
>> packages themselves.
> Thanks. This will help a lot, if I understand it correctly.
> It looks like you do not clear your build directories using 'cygport
> finish' after uploading new/changed packages, so a lot of
> inter-package state is retained in your /usr/src tree. Is this
> correct? Or am I misunderstanding the recipe you show above.
> In step 2, should I be executing the just built executable:
> ../inst/usr/bin/python3.7m.exe ? That seems to work.
> In step 3, I get:
> $ pwd
> /usr/src/python-wheel-0.32.3-1.src
> $ cygport python-wheel.cygport prep
> *** ERROR: python27-pip is required to build this package
> I don't understand how to get past this point.
> Am I calling cygport correctly at this point?
> Do I need any environment variables?
> What does this mean, besides the obvious? What kind of thing is
> `python27-pip' and where should it be? Is it looking in a wheel file
> somewhere? In the /usr/src/python37-3.7.2-1.src tree? In the
> /usr/src/python2-2.7.15-1.src tree? Do I need to cygport pe & compile
> & install and then install pip2 in that local install tree? And
> likewise for each of Python 3.4, 3.5, 3.6? Or do these versions of pip
> need to be installed into my system via setup.exe?
> Do I need to study Gentoo's portage to get a better understanding of
> how cygport works?
> Sorry about all the questions. Perhaps I'm trying to move ahead too
> quickly, without poring through the source for cygport. I've found
> the document at https://cygwinports.github.io/cygport/cygport_in.html
> to be a bit sparse in some important places (and I know exactly how
> this kind of documentation gets created, as I've done it myself a time
> or two).

Symlink /usr/share/doc/cygport/html/manual/index.html -> cyport-doc, also alias
opening with cygstart, also bookmark in your browser, and start reading; browse
around in /usr/share/cygport/cygclass and that neighbourhood.

When third party builds have problems under Cygwin, try using cygport to solve
some of the problems, often more easily than hacking the build directly,
although that can't always be avoided, in which case those patches are
candidates for submitting upstream.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.