Dependency issues in setup.ini.

Dependency issues in setup.ini.

[Sam Edge (Cygwin)]

Hi all.

I've been developing a Python package that can interrogate and
manipulate local package caches (the directories where setupXXX.exe
keeps its downloads) and installation databases (from Cygwin
/etc/setup/installed.db files) with a mind to pruning, merging and
reporting in the spiript of Michael A. Chase's 'clean_setup' utility but
as a scriptable tool set rather than a stand-alone utility.

It's not production ready yet but it's already flagged up some issues.

For example we have lots of dependency loops in the 'requires' fields in
setup.ini - even to the point that some packages depend upon themselves!

And also we have some dependency omissions. For example, mintty doesn't
depend upon anything - it has no requires field. Surely, every binary
package should depend at least upon 'cygwin'?

Is this a known issue or should I report in more detail?

-- 
Sam Edge


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Steven Penny]

On Fri, 29 Sep 2017 21:16:17, "Sam Edge (Cygwin)" wrote:
> For example we have lots of dependency loops in the 'requires' fields in
> setup.ini - even to the point that some packages depend upon themselves!

It is the job of the package manager to detect and avoid such loops. No database
is perfect - it is up to the good programmer to account for an deal with problem
data


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Doug Henderson]

On 29 September 2017 at 14:16, Sam Edge (Cygwin) wrote:

> … mintty doesn't
> depend upon anything - it has no requires field. …


mintty is a windows, non-cygwin, executable which does not depend on cygwin1.dll

There are a few other such programs, which (I guess) are built using
the mingw32 tools chains. Such programs have no execution dependencies
on cygwin packages, but they may have build dependencies.

HTH
Doug

-- 
Doug Henderson, Calgary, Alberta, Canada - from gmail.com

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Steven Penny]

On Fri, 29 Sep 2017 17:53:52, Doug Henderson wrote:
> mintty is a windows, non-cygwin, executable

No, its not:

$ cygcheck mintty
Found: C:\cygwin64\bin\mintty.exe
C:\cygwin64\bin\mintty.exe
C:\cygwin64\bin\cygwin1.dll

> which does not depend on cygwin1.dll

Yes, it does:

$ ldd /bin/mintty.exe
cygwin1.dll => /usr/bin/cygwin1.dll (0x180040000)

> HTH

It doesnt. Really bad to spread false info in such a public setting, and such
easily checkable info. Shame on you.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Sam Edge (Cygwin)]

On 29/09/2017 23:39, Steven Penny wrote:
> On Fri, 29 Sep 2017 21:16:17, "Sam Edge (Cygwin)" wrote:
>> For example we have lots of dependency loops in the 'requires' fields in
>> setup.ini - even to the point that some packages depend upon themselves!
>
> It is the job of the package manager to detect and avoid such loops.
> No database
> is perfect - it is up to the good programmer to account for an deal
> with problem
> data
>
>

Thanks for the clarification. Okay. Once I've written some code to
unpick which packages are erroneously creating the loops I'll report back.

Next question - :-) - should I report here or in cygwin-apps?

At the risk of getting a SHTDI response, perhaps part of the release
process for setup.ini should include a topological sort on '@
package'/'requires: package ...' pairs to flag up such issues? :-D

-- 
Sam Edge


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Marco Atzeri]

On 30/09/2017 00:39, Steven Penny wrote:
> On Fri, 29 Sep 2017 21:16:17, "Sam Edge (Cygwin)" wrote:
>> For example we have lots of dependency loops in the 'requires' fields in
>> setup.ini - even to the point that some packages depend upon themselves!
> 
> It is the job of the package manager to detect and avoid such loops. No 
> database
> is perfect - it is up to the good programmer to account for an deal with 
> problem
> data
> 

some dependency loop are real as some packages
have cross dependency:

$ cygcheck cygMagickCore-6.Q16-5.dll | grep auto
   E:\cygwin64\bin\cygautotrace-3.dll

$ cygcheck cygautotrace-3.dll |grep Q16
   E:\cygwin64\bin\cygMagickCore-6.Q16-5.dll


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Andrey Repin]

Greetings, Sam Edge (Cygwin)!

> I've been developing a Python package that can interrogate and
> manipulate local package caches (the directories where setupXXX.exe
> keeps its downloads) and installation databases (from Cygwin
> /etc/setup/installed.db files) with a mind to pruning, merging and
> reporting in the spiript of Michael A. Chase's 'clean_setup' utility but
> as a scriptable tool set rather than a stand-alone utility.

I'm eager to see the fruits of your labor.

> It's not production ready yet but it's already flagged up some issues.

> For example we have lots of dependency loops in the 'requires' fields in
> setup.ini - even to the point that some packages depend upon themselves!

Dependency upon itself is curious, but other than that, this is a normal
situation for a package manager. Some packages are split for easier
maintenance of each, but are interlocked in their typical usage pattern.

> And also we have some dependency omissions. For example, mintty doesn't
> depend upon anything - it has no requires field. Surely, every binary
> package should depend at least upon 'cygwin'?

While this is "not good", this is also not particularly bad for packages in
base - this group is always installed.

> Is this a known issue or should I report in more detail?

Nonetheless, such issues are best kept highlighted, unless it is clearly
seen/documented as intended.


-- 
With best regards,
Andrey Repin
Saturday, September 30, 2017 11:54:56

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Sam Edge]

Hi Andrey.

Nice to be back in a thread with such esteemed folk. ;-)

On 30/09/2017 10:00, Andrey Repin wrote:
> Greetings, Sam Edge (Cygwin)!
>
>> I've been developing a Python package that can interrogate and
>> manipulate local package caches (the directories where setupXXX.exe
>> keeps its downloads) and installation databases (from Cygwin
>> /etc/setup/installed.db files) with a mind to pruning, merging and
>> reporting in the spiript of Michael A. Chase's 'clean_setup' utility but
>> as a scriptable tool set rather than a stand-alone utility.
> I'm eager to see the fruits of your labor.

Don't hold your breath! :-) I'm doing it partly to teach myself the
subtleties of Python classes and I've not yet got to the process of
turning it into an installable import module. Also, its re-based parser
is still a bit naive. It works okay with the current setup.ini but it
has vulnerabilities I'd like to eliminate.(Hopefully I'll have something
I won't be embarrassed to share before Yuletide.)

>> It's not production ready yet but it's already flagged up some issues.
>> For example we have lots of dependency loops in the 'requires' fields in
>> setup.ini - even to the point that some packages depend upon themselves!
> Dependency upon itself is curious, but other than that, this is a normal
> situation for a package manager. Some packages are split for easier
> maintenance of each, but are interlocked in their typical usage pattern.

Ah, okay. Fair enough. It can be difficult to keep things layered purely
up & down I know. Although often it can be resolved by introducing a
third module that acts as the muxer between the other two to avoid cross
API dependencies. But that's a discussion for another mailing list.

But I'm also seeing loops deeper that X->Y->X. More like X->Y->Z->W->X.

(The self-dependency is cygwin-debuginfo by the way.)

>> And also we have some dependency omissions. For example, mintty doesn't
>> depend upon anything - it has no requires field. Surely, every binary
>> package should depend at least upon 'cygwin'?
> While this is "not good", this is also not particularly bad for packages in
> base - this group is always installed.

Indeed. However, while off label usage of Cygwin is anathema to me but
sometimes I wish 'base' wasn't quite so big and have to pare things down
a little once installed, e.g. as part of a makefile- and/or
Eclipse-based build tree in source code control.(Which was also one of
my motivations for the Python stuff.)

>> Is this a known issue or should I report in more detail?
> Nonetheless, such issues are best kept highlighted, unless it is clearly
> seen/documented as intended.
>

Will try to collate a list as soon as I have a free (as in not too
bushed) weekday evening.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Andrey Repin]

Greetings, Sam Edge!

>>> It's not production ready yet but it's already flagged up some issues.
>>> For example we have lots of dependency loops in the 'requires' fields in
>>> setup.ini - even to the point that some packages depend upon themselves!
>> Dependency upon itself is curious, but other than that, this is a normal
>> situation for a package manager. Some packages are split for easier
>> maintenance of each, but are interlocked in their typical usage pattern.

> Ah, okay. Fair enough. It can be difficult to keep things layered purely
> up & down I know.

More than that, naive assumption of no circular dependency is the most common
cause for infinite recursions.

> Although often it can be resolved by introducing a
> third module that acts as the muxer between the other two to avoid cross
> API dependencies. But that's a discussion for another mailing list.

> But I'm also seeing loops deeper that X->Y->X. More like X->Y->Z->W->X.

A list indexed by package name is necessary when you resolve package
dependencies. Then you always know when to avoid dependency rescans.

> (The self-dependency is cygwin-debuginfo by the way.)

:)

>>> And also we have some dependency omissions. For example, mintty doesn't
>>> depend upon anything - it has no requires field. Surely, every binary
>>> package should depend at least upon 'cygwin'?
>> While this is "not good", this is also not particularly bad for packages in
>> base - this group is always installed.

> Indeed. However, while off label usage of Cygwin is anathema to me but
> sometimes I wish 'base' wasn't quite so big and have to pare things down
> a little once installed, e.g. as part of a makefile- and/or
> Eclipse-based build tree in source code control.(Which was also one of
> my motivations for the Python stuff.)

Rational suggestions are always welcome, I suppose.
While my own usage of Cygwin is prone to spread thin across all aspects of my
daily work, I can see situations, where a much smaller subset of packages
(let's name it "core" or something) would be beneficial. I.e. when packaging
Cygwin as part of your own application.


-- 
With best regards,
Andrey Repin
Saturday, September 30, 2017 14:16:20

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Sam Edge]

On 30/09/2017 12:23, Andrey Repin wrote:
>
>> Indeed. However, while off label usage of Cygwin is anathema to me but
>> sometimes I wish 'base' wasn't quite so big and have to pare things down
>> a little once installed, e.g. as part of a makefile- and/or
>> Eclipse-based build tree in source code control.(Which was also one of
>> my motivations for the Python stuff.)
> Rational suggestions are always welcome, I suppose.
> While my own usage of Cygwin is prone to spread thin across all aspects of my
> daily work, I can see situations, where a much smaller subset of packages
> (let's name it "core" or something) would be beneficial. I.e. when packaging
> Cygwin as part of your own application.
>
>
Again, agree. A 'core' package that has enough to get a shell up
(preferably dash not bash) would be very useful.

I don't believe in packaging Cygwin with an application - I'd rather say
"as a prerequisite, you need to have a Cygwin installation newer than
v.X" - but for build systems where you're trying (somewhat in vain
sometimes I know) to attempt deterministic behaviour it would be nice to
have a smaller subset upon which you could add just what's needed,
usually make, grep, sed, cp, mv & rm & maybe a scripter such as awk,
perl, tcl (or python!) depending upon.

Anyway, it's the weekend so I'm off.

Enjoy,
-- 
Sam

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Dependency issues in setup.ini.

[Jon Turney]

On 29/09/2017 21:16, Sam Edge (Cygwin) wrote:
> Hi all.
> 
> I've been developing a Python package that can interrogate and
> manipulate local package caches (the directories where setupXXX.exe
> keeps its downloads) and installation databases (from Cygwin
> /etc/setup/installed.db files) with a mind to pruning, merging and
> reporting in the spiript of Michael A. Chase's 'clean_setup' utility but
> as a scriptable tool set rather than a stand-alone utility.
> 
> It's not production ready yet but it's already flagged up some issues.
> 
> For example we have lots of dependency loops in the 'requires' fields in
> setup.ini - even to the point that some packages depend upon themselves!

Cycles in the dependency graph are unfortunately a real thing (although 
all cycles which exist may not be correct or unavoidable)

cygwin-debuginfo depending on itself is simply a bug in cygport [1], as 
it makes all debuginfo packages unconditionally depend on cygwin-debuginfo

calm permits this as a historical exception, any other package with a 
self-dependency should be rejected.

[1] 
https://github.com/cygwinports/cygport/commit/65cc501f3bea06bcd69326649646568fa5a94092

> And also we have some dependency omissions. For example, mintty doesn't
> depend upon anything - it has no requires field. Surely, every binary
> package should depend at least upon 'cygwin'?

Again, this is because of historical reasons.

For reasons lost in the mists of time, packaging used to suppress any 
dependency on 'cygwin' from requires: in the setup.hint, and then upset 
would add it back for every single package in setup.ini.

Nowadays, we treat the 'cygwin' dependency the same as any other (i.e. 
any package containing an exe or dll linked to the cygwin DLL should 
have a 'cygwin' dependency)

I believe the maintainer of mintty is still using an old version of 
cygport, which has the historical behaviour, hence this dependency is 
missing.

Practically, this shouldn't cause problems, as the cygwin package is in 
the base category, and so should always be present.

> Is this a known issue or should I report in more detail?

Specific problems should be reported to the cygwin-apps list.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple