Re: setup -g ???

Re: setup -g ???

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 3206 bytes --]

[Redirected to cygwin-apps from 
https://cygwin.com/ml/cygwin/2018-03/msg00365.html.]

On 3/22/2018 6:46 PM, Jon Turney wrote:
> On 14/03/2018 15:26, David Allsopp wrote:
>> [reformatted for top-posting]
>>
>> Lee wrote:
>>>> ---------- Forwarded message ----------
>>>> From: Jon Turney
>>>> Date: Fri, 3 Nov 2017 15:26:27 +0000
>>>> Subject: Re: Problem running the latest python2-2.7.14-1 under AppVeyor
>>>> To: The Cygwin Mailing List <cygwin@cygwin.com>
>>>>
>>>> On 03/11/2017 14:45, Vadim Zeitlin wrote:
>>>>> Our build has started on AppVeyor, a continuous integration provider,
>>>>> started failing since a couple of days as a makefile command running a
>>>>> Python script started failing with exit code 127 without any more
>>>>> information. This is a strange situation as I can't reproduce the
>>>>> problem locally, but something definitely seems to be wrong with this
>>>>> package on the AppVeyor machine as Python just doesn't seem to be
>>>>> executable, e.g. the output of these commands in our batch file
>>>> driving the build:
>>>>
>>>> Perhaps you need to provide the -g (--upgrade-also) flag to cygwin's
>>>> setup.
>>>>
>>>> Due to setup terribleness, without this flag, it will install the
>>>> requested packages, and any missing dependencies of them, but not
>>>> upgrade any of the dependencies which are already installed to the
>>>> current (and perhaps needed) version...
>>>>
>>>> See also [1].
>>>>
>>>> [1] https://sourceware.org/ml/cygwin/2017-03/msg00365.html
>>>
>>> Should we still be using the -g (--upgrade-also) flag on setup?
>>
>> I believe so (or at least hope so). I think it's the case that setup 
>> should now know to upgrade a dependency if you install a new package 
>> which requires a newer version of it, but I hope that's not become the 
>> same as setup effectively acting with --upgrade-also every time you 
>> run it (that would be a real nuisance, unless the entire Cygwin 
>> package universe is going to be recompiled on every new Cygwin release).
> 
> This is basically correct.
> 
> setup is now capable of being told about dependencies where upgrading an 
> already installed package is required, but this information isn't 
> currently collected
> 
> (For example, some packages now exist (e.g. vim [1]), which were built 
> with gcc 6.4.0-5 and cygport 0.31.0-1.  These packages almost certainly 
> use ssp/fortify functions in the cygwin DLL, and so require a cygwin 
> package >=2.10.0-1 (technically, the requirement is cygwin API >=0.320), 
> but the dependency recorded is only on the cygwin package at any version)
> 
> That's something someone could usefully work on, if they were so inclined.

The attached cygport patch attempts to address this by requiring, for 
each dependency of a package, a version >= the version installed at the 
time the package was built.  It treats only dependencies found by 
cygport, not those added via REQUIRES in the cygport file.  My thinking 
is that the cygport user should be in control of added dependencies; 
s/he can add version numbers if desired.

mksetupini would need to be updated to parse versioned dependencies. 
(Or maybe it's expecting a different syntax; I didn't check.)

Ken

[-- Attachment #2: 0001-__pkg_dist-record-package-versions-in-requires.patch --]
[-- Type: text/plain, Size: 1092 bytes --]

From 591f8ec8cf62382cb1a24feef7603080a44b4021 Mon Sep 17 00:00:00 2001
From: Ken Brown <kbrown@cornell.edu>
Date: Mon, 2 Apr 2018 12:44:42 -0400
Subject: [PATCH cygport] __pkg_dist: record package versions in "requires:"

This makes it less likely that a user will update a package and fail
to update its dependencies.

https://cygwin.com/ml/cygwin/2018-03/msg00365.html
---
 lib/pkg_pkg.cygpart | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
index a7624ff..651d412 100644
--- a/lib/pkg_pkg.cygpart
+++ b/lib/pkg_pkg.cygpart
@@ -690,7 +690,7 @@ __pkg_dist() {
 			if [ "${CBUILD##*-}" = "cygwin" ]
 			then
 				pkg_bin_requires=$(__list_deps $(sort -fu ${T}/.${pkg_name[${n}]}.lst | sed -e '\|/$|d;s| |^_^|g') \
-						   | sort -fu | sed -e "/^${pkg_name[${n}]}-[0-9].*$/d;s/-[0-9].*$//g" \
+						   | sort -fu | sed -e "/^${pkg_name[${n}]}-[0-9].*$/d;s/-\([0-9].*\)$/ (>=\1)/g" \
 						   | sed -e ':a;N;$!ba;s/\n/ /g')
 				__step "${pkg_name[${n}]} requires: ${pkg_bin_requires} ${!pkg_requires_var}"
 			else
-- 
2.16.2

Re: setup -g ???

[Ken Brown]

On 4/2/2018 1:03 PM, Ken Brown wrote:
> [Redirected to cygwin-apps from 
> https://cygwin.com/ml/cygwin/2018-03/msg00365.html.]
> 
> On 3/22/2018 6:46 PM, Jon Turney wrote:
>> On 14/03/2018 15:26, David Allsopp wrote:
>>> [reformatted for top-posting]
>>>
>>> Lee wrote:
>>>>> ---------- Forwarded message ----------
>>>>> From: Jon Turney
>>>>> Date: Fri, 3 Nov 2017 15:26:27 +0000
>>>>> Subject: Re: Problem running the latest python2-2.7.14-1 under 
>>>>> AppVeyor
>>>>> To: The Cygwin Mailing List <cygwin@cygwin.com>
>>>>>
>>>>> On 03/11/2017 14:45, Vadim Zeitlin wrote:
>>>>>> Our build has started on AppVeyor, a continuous integration provider,
>>>>>> started failing since a couple of days as a makefile command 
>>>>>> running a
>>>>>> Python script started failing with exit code 127 without any more
>>>>>> information. This is a strange situation as I can't reproduce the
>>>>>> problem locally, but something definitely seems to be wrong with this
>>>>>> package on the AppVeyor machine as Python just doesn't seem to be
>>>>>> executable, e.g. the output of these commands in our batch file
>>>>> driving the build:
>>>>>
>>>>> Perhaps you need to provide the -g (--upgrade-also) flag to cygwin's
>>>>> setup.
>>>>>
>>>>> Due to setup terribleness, without this flag, it will install the
>>>>> requested packages, and any missing dependencies of them, but not
>>>>> upgrade any of the dependencies which are already installed to the
>>>>> current (and perhaps needed) version...
>>>>>
>>>>> See also [1].
>>>>>
>>>>> [1] https://sourceware.org/ml/cygwin/2017-03/msg00365.html
>>>>
>>>> Should we still be using the -g (--upgrade-also) flag on setup?
>>>
>>> I believe so (or at least hope so). I think it's the case that setup 
>>> should now know to upgrade a dependency if you install a new package 
>>> which requires a newer version of it, but I hope that's not become 
>>> the same as setup effectively acting with --upgrade-also every time 
>>> you run it (that would be a real nuisance, unless the entire Cygwin 
>>> package universe is going to be recompiled on every new Cygwin release).
>>
>> This is basically correct.
>>
>> setup is now capable of being told about dependencies where upgrading 
>> an already installed package is required, but this information isn't 
>> currently collected
>>
>> (For example, some packages now exist (e.g. vim [1]), which were built 
>> with gcc 6.4.0-5 and cygport 0.31.0-1.  These packages almost 
>> certainly use ssp/fortify functions in the cygwin DLL, and so require 
>> a cygwin package >=2.10.0-1 (technically, the requirement is cygwin 
>> API >=0.320), but the dependency recorded is only on the cygwin 
>> package at any version)
>>
>> That's something someone could usefully work on, if they were so 
>> inclined.
> 
> The attached cygport patch attempts to address this by requiring, for 
> each dependency of a package, a version >= the version installed at the 
> time the package was built.  It treats only dependencies found by 
> cygport, not those added via REQUIRES in the cygport file.  My thinking 
> is that the cygport user should be in control of added dependencies; 
> s/he can add version numbers if desired.
> 
> mksetupini would need to be updated to parse versioned dependencies. (Or 
> maybe it's expecting a different syntax; I didn't check.)

I finally got around to checking this, and it seems that mksetupini 
already recognizes dependencies with version relations, provided that 
the hint file uses "depends:" instead of "requires:".

Unfortunately, changing cygport to use "depends:" instead of "requires:" 
might confuse maintainers, given that cygport files currently use 
"DEPEND" for build dependencies.

An alternative would be for cygport to start using commas rather than 
spaces to separate dependencies.  mksetupini could then parse 
"requires:" the same way it now parses "depends:" if it finds commas in 
the "requires:" list.

Ken

Re: setup -g ???

[Ken Brown]

On 4/23/2018 4:45 PM, Ken Brown wrote:
> On 4/2/2018 1:03 PM, Ken Brown wrote:
>> [Redirected to cygwin-apps from 
>> https://cygwin.com/ml/cygwin/2018-03/msg00365.html.]
>>
>> On 3/22/2018 6:46 PM, Jon Turney wrote:
>>> On 14/03/2018 15:26, David Allsopp wrote:
>>>> [reformatted for top-posting]
>>>>
>>>> Lee wrote:
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Jon Turney
>>>>>> Date: Fri, 3 Nov 2017 15:26:27 +0000
>>>>>> Subject: Re: Problem running the latest python2-2.7.14-1 under 
>>>>>> AppVeyor
>>>>>> To: The Cygwin Mailing List <cygwin@cygwin.com>
>>>>>>
>>>>>> On 03/11/2017 14:45, Vadim Zeitlin wrote:
>>>>>>> Our build has started on AppVeyor, a continuous integration 
>>>>>>> provider,
>>>>>>> started failing since a couple of days as a makefile command 
>>>>>>> running a
>>>>>>> Python script started failing with exit code 127 without any more
>>>>>>> information. This is a strange situation as I can't reproduce the
>>>>>>> problem locally, but something definitely seems to be wrong with 
>>>>>>> this
>>>>>>> package on the AppVeyor machine as Python just doesn't seem to be
>>>>>>> executable, e.g. the output of these commands in our batch file
>>>>>> driving the build:
>>>>>>
>>>>>> Perhaps you need to provide the -g (--upgrade-also) flag to cygwin's
>>>>>> setup.
>>>>>>
>>>>>> Due to setup terribleness, without this flag, it will install the
>>>>>> requested packages, and any missing dependencies of them, but not
>>>>>> upgrade any of the dependencies which are already installed to the
>>>>>> current (and perhaps needed) version...
>>>>>>
>>>>>> See also [1].
>>>>>>
>>>>>> [1] https://sourceware.org/ml/cygwin/2017-03/msg00365.html
>>>>>
>>>>> Should we still be using the -g (--upgrade-also) flag on setup?
>>>>
>>>> I believe so (or at least hope so). I think it's the case that setup 
>>>> should now know to upgrade a dependency if you install a new package 
>>>> which requires a newer version of it, but I hope that's not become 
>>>> the same as setup effectively acting with --upgrade-also every time 
>>>> you run it (that would be a real nuisance, unless the entire Cygwin 
>>>> package universe is going to be recompiled on every new Cygwin 
>>>> release).
>>>
>>> This is basically correct.
>>>
>>> setup is now capable of being told about dependencies where upgrading 
>>> an already installed package is required, but this information isn't 
>>> currently collected
>>>
>>> (For example, some packages now exist (e.g. vim [1]), which were 
>>> built with gcc 6.4.0-5 and cygport 0.31.0-1.  These packages almost 
>>> certainly use ssp/fortify functions in the cygwin DLL, and so require 
>>> a cygwin package >=2.10.0-1 (technically, the requirement is cygwin 
>>> API >=0.320), but the dependency recorded is only on the cygwin 
>>> package at any version)
>>>
>>> That's something someone could usefully work on, if they were so 
>>> inclined.
>>
>> The attached cygport patch attempts to address this by requiring, for 
>> each dependency of a package, a version >= the version installed at 
>> the time the package was built.  It treats only dependencies found by 
>> cygport, not those added via REQUIRES in the cygport file.  My 
>> thinking is that the cygport user should be in control of added 
>> dependencies; s/he can add version numbers if desired.
>>
>> mksetupini would need to be updated to parse versioned dependencies. 
>> (Or maybe it's expecting a different syntax; I didn't check.)
> 
> I finally got around to checking this, and it seems that mksetupini 
> already recognizes dependencies with version relations, provided that 
> the hint file uses "depends:" instead of "requires:".

I've just sent a patch series that does this, among other things.

Ken

collecting information for versioned dependencies (was Re: setup -g ???)

[Jon Turney]

On 02/04/2018 18:03, Ken Brown wrote:
> [Redirected to cygwin-apps from 
> https://cygwin.com/ml/cygwin/2018-03/msg00365.html.]
> 
> On 3/22/2018 6:46 PM, Jon Turney wrote:
[...]
>>
>> This is basically correct.
>>
>> setup is now capable of being told about dependencies where upgrading 
>> an already installed package is required, but this information isn't 
>> currently collected
>>
>> (For example, some packages now exist (e.g. vim [1]), which were built 
>> with gcc 6.4.0-5 and cygport 0.31.0-1.  These packages almost 
>> certainly use ssp/fortify functions in the cygwin DLL, and so require 
>> a cygwin package >=2.10.0-1 (technically, the requirement is cygwin 
>> API >=0.320), but the dependency recorded is only on the cygwin 
>> package at any version)

(The example given is kind of incomplete, in that we could define an 
additional provides: to record that API version.)

>> That's something someone could usefully work on, if they were so 
>> inclined.
> 
> The attached cygport patch attempts to address this by requiring, for 
> each dependency of a package, a version >= the version installed at the 
> time the package was built.  It treats only dependencies found by 

Thanks.  This is an interesting approach to automatically collecting 
this information I hadn't considered.

Problems are that (i) it makes package builds less reproducible (as 
these dependencies will depend on the currently installed version of 
them, which might well change over time), and (ii) it will tend of 
over-estimate the version required.

I don't know how these problems are solved in other distros, but that's 
probably worth looking at...

> cygport, not those added via REQUIRES in the cygport file.  My thinking 
> is that the cygport user should be in control of added dependencies; 
> s/he can add version numbers if desired.
> 
> mksetupini would need to be updated to parse versioned dependencies. (Or 
> maybe it's expecting a different syntax; I didn't check.)

Re: collecting information for versioned dependencies (was Re: setup -g ???)

[Achim Gratz]

Jon Turney writes:
>> The attached cygport patch attempts to address this by requiring,
>> for each dependency of a package, a version >= the version installed
>> at the time the package was built.  It treats only dependencies
>> found by 
>
> Thanks.  This is an interesting approach to automatically collecting
> this information I hadn't considered.

Recording that information is a good thing.  Making these a (hard)
dependency is probably not very helpful.

> Problems are that (i) it makes package builds less reproducible (as
> these dependencies will depend on the currently installed version of
> them, which might well change over time), and (ii) it will tend of
> over-estimate the version required.

Yes.  I'd rather not go there.

> I don't know how these problems are solved in other distros, but
> that's probably worth looking at...

I can only (barely) tell how openSUSE Tumbleweed does it.  The first
part is that the dependencies are using capabilities, not versions
(i.e. required API level) and those requirements are part of the spec
files.  The second part is that the package dependencies are organized
in "rings" so that outer rings can only start building when the inner
ones are finished.  The third part is that the build is fully automated
via OBS, so if anything changes that somebody else depends on, those
builds are automatically triggered.  There is some extensive scripted
and sometimes manual intervention to prevent certain known false
positives from re-building the whole distro.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada