[RFC] 1.7 Packaging: Obsolete packages

[RFC] 1.7 Packaging: Obsolete packages

[Yaakov (Cygwin Ports)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

As we start preparing for 1.7, I know that until now the focus has been
on the development of Cygwin itself.  I'll admit that I know little
about Cygwin's internals, and hence have contributed very little to the
DLL itself, but I do very much appreciate the work of those that do.

What I *do* know about by now is porting and packaging for Cygwin,
having done it for almost five years now.  And with the transition to
1.7 coming up, there are a few general packaging issues that I think it
would be timely to address.

The first thing is that I think a mass rebuild is in order.  There are
too many packages that are outdated, dependent on old libraries, or that
would greatly benefit from the new features in 1.7 (or 1.5 for that matter).

Over the years a lot of cruft has accumulated within the distro, namely:

a) old versions of libraries (e.g. libintl{1,2,3});
b) empty _obsolete packages to smoothly handle renamings and such;
c) packages which are of little use and unmaintained.

At the same time, there could be some major transitions coming up:

d) some consistency in library package naming;
e) monolithic X11R6 in /usr/X11R6, to modular X11R7 in /usr.

More details on these later.

One of the key factors on how to handle all this, which I mentioned
briefly before, is that we need to make a decision on how we will expect
the user base to upgrade to 1.7.  The options are:

1) try to make 1.5->1.7 like 1.5.x->1.5.y.

*If* it's possible, then this would theoretically make it easier for
users to upgrade.  When the necessary packages are rebuilt, then the old
libraries could be removed from the distro, but the empty compatibility
packages may need to remain.

2) start over with a new install.

This may be more difficult for users, depending on how they use Cygwin,
and depending on what will ultimately be necessary for handling a
straight upgrade.  As for the packaging, it would give us the chance to
dump a lot of old baggage and make some additional changes without
worrying about now-ancient history.

There are obviously pros and cons to both; we need to make a decision
that will best serve the project and its users in the long term.


Yaakov
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREIAAYFAkiICHkACgkQpiWmPGlmQSPFsACgt283Vny05vFrNkb4H0VnyYPo
/JwAn0/xVpvt5quRTTQA5uciucImoKNh
=SZTY
-----END PGP SIGNATURE-----

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Jul 23 23:43, Yaakov (Cygwin Ports) wrote:
> What I *do* know about by now is porting and packaging for Cygwin,
> having done it for almost five years now.  And with the transition to
> 1.7 coming up, there are a few general packaging issues that I think it
> would be timely to address.

Can't contribute much to your points but...

> The first thing is that I think a mass rebuild is in order.  There are
> too many packages that are outdated, dependent on old libraries, or that
> would greatly benefit from the new features in 1.7 (or 1.5 for that 
> matter).

To reiterate, the most important features of Cygwin 1.7 to consider when
rebuilding packages, because of how they influence building applications
are:

  - IPv6 support, new functions getaddrinfo, getnameinfo.

  - minires and libresolv.a are now part of Cygwin.

  - PATH_MAX has changed from 260 to 4096.  getcwd() can potentially
    return paths of up to 32K.

  - Potentially case sensitivity file operations (on OSes and FSes
    supporting it).

  - New file conversion API for conversion from Win32 to POSIX path and
    vice versa (cygwin_conv_path, cygwin_create_path, cygwin_conv_path_list).

> 1) try to make 1.5->1.7 like 1.5.x->1.5.y.
>
> *If* it's possible, then this would theoretically make it easier for
> users to upgrade.  When the necessary packages are rebuilt, then the old
> libraries could be removed from the distro, but the empty compatibility
> packages may need to remain.
>
> 2) start over with a new install.
>
> This may be more difficult for users, depending on how they use Cygwin,
> and depending on what will ultimately be necessary for handling a
> straight upgrade.  As for the packaging, it would give us the chance to
> dump a lot of old baggage and make some additional changes without
> worrying about now-ancient history.

I'm not sure if there's really a big difference between these two points.
Since we're using two different installation directories, we can get rid
of old cruft, if we just look carefully what's still used and what not.

The release-2 directory has already no obsolete packages anymore.  Stuff
like minires, which is now a part of Cygwin, can entirely go away as
soon as all packages relying on resolver functions have been rebuilt.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Yaakov (Cygwin Ports)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Corinna Vinschen wrote:
| I'm not sure if there's really a big difference between these two points.
| Since we're using two different installation directories, we can get rid
| of old cruft, if we just look carefully what's still used and what not.
|
| The release-2 directory has already no obsolete packages anymore.  Stuff
| like minires, which is now a part of Cygwin, can entirely go away as
| soon as all packages relying on resolver functions have been rebuilt.

The difference is if I want to reorganize a package when I rebuild it
for 1.7, e.g. right now I have:

pcre => pcre, libpcre0, pcre-devel, pcre-doc

If I want to rename pcre-devel to libpcre-devel, then normally I would
need an empty _obsolete pcre-devel package which deps libpcre-devel to
make the upgrade smooth.  That wouldn't be necessary if we don't support
upgrading to 1.7 in the same installation.

That may seem like a trivial example, but a transition like X11R6 to
X11R7 would be a lot bigger.


Yaakov
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREIAAYFAkiJE9kACgkQpiWmPGlmQSN2xQCeOih7EzNLgL4HsDjAPFeyfPMW
RxgAoJEjoxN10tfbginETYl9BJRDjgoh
=m1Bm
-----END PGP SIGNATURE-----

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Jul 24 18:44, Yaakov (Cygwin Ports) wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Corinna Vinschen wrote:
> | I'm not sure if there's really a big difference between these two points.
> | Since we're using two different installation directories, we can get rid
> | of old cruft, if we just look carefully what's still used and what not.
> |
> | The release-2 directory has already no obsolete packages anymore.  Stuff
> | like minires, which is now a part of Cygwin, can entirely go away as
> | soon as all packages relying on resolver functions have been rebuilt.
>
> The difference is if I want to reorganize a package when I rebuild it
> for 1.7, e.g. right now I have:
>
> pcre => pcre, libpcre0, pcre-devel, pcre-doc
>
> If I want to rename pcre-devel to libpcre-devel, then normally I would
> need an empty _obsolete pcre-devel package which deps libpcre-devel to
> make the upgrade smooth.  That wouldn't be necessary if we don't support
> upgrading to 1.7 in the same installation.
>
> That may seem like a trivial example, but a transition like X11R6 to
> X11R7 would be a lot bigger.

Nice example.  Still, for now we should assume that we go the upgrade
path.  I'm going to investigate the impact of a clean cut in the next
couple of days.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Christopher Faylor]

On Fri, Jul 25, 2008 at 11:40:34AM +0200, Corinna Vinschen wrote:
>Nice example.  Still, for now we should assume that we go the upgrade
>path.  I'm going to investigate the impact of a clean cut in the next
>couple of days.

It seems like maybe another change to setup.exe would be in order here
or maybe there could be a base package which did any cleanup required to
purge unneeded packages.

cgf

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Jul 25 10:52, Christopher Faylor wrote:
> On Fri, Jul 25, 2008 at 11:40:34AM +0200, Corinna Vinschen wrote:
> >Nice example.  Still, for now we should assume that we go the upgrade
> >path.  I'm going to investigate the impact of a clean cut in the next
> >couple of days.
> 
> It seems like maybe another change to setup.exe would be in order here
> or maybe there could be a base package which did any cleanup required to
> purge unneeded packages.

Another base package sounds like an interesting idea.  OTOH, setup would
use the existing local setup database and might get something wrong
before the new base package has a chance to do the cleanup.  Baah.
Maybe we could provide an upgrade helper script which should run *before*
running setup for the ultimate upgrade.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Christopher Faylor]

On Fri, Jul 25, 2008 at 05:16:30PM +0200, Corinna Vinschen wrote:
>On Jul 25 10:52, Christopher Faylor wrote:
>> On Fri, Jul 25, 2008 at 11:40:34AM +0200, Corinna Vinschen wrote:
>> >Nice example.  Still, for now we should assume that we go the upgrade
>> >path.  I'm going to investigate the impact of a clean cut in the next
>> >couple of days.
>> 
>> It seems like maybe another change to setup.exe would be in order here
>> or maybe there could be a base package which did any cleanup required to
>> purge unneeded packages.
>
>Another base package sounds like an interesting idea.  OTOH, setup would
>use the existing local setup database and might get something wrong
>before the new base package has a chance to do the cleanup.  Baah.
>Maybe we could provide an upgrade helper script which should run *before*
>running setup for the ultimate upgrade.

I was thinking of a variation of _update_info which waited until everything
was done  before it started doing anything.  You could even write a mingw
app which didn't rely on cygwin to do some of the work.

cgf

Re: [RFC] 1.7 Packaging: Obsolete packages

[Nicholas Wourms]

Corinna Vinschen wrote:
>
>    - Potentially case sensitivity file operations (on OSes and FSes
>      supporting it).
>
Have managed mounts been fixed yet? Sorry, I've been out of the loop for quite 
awhile. Anyway, I noticed that Yakkov was using this feature in Cygwin Ports, so 
I thought I might inquire, since I'm curious as well. ;-)

Cheers,
Nicholas

Re: [RFC] 1.7 Packaging: Obsolete packages

[Yaakov (Cygwin Ports)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Christopher Faylor wrote:
| It seems like maybe another change to setup.exe would be in order here
| or maybe there could be a base package which did any cleanup required to
| purge unneeded packages.

Could setup-1.7 know if the last installation was 1.5, and simply ignore
the installed.db?


Yaakov

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREIAAYFAkiNLOMACgkQpiWmPGlmQSNiiACgm5cuEvmd1fuuIIumBu+pW4OT
7X8AoPmF6BtWcxtMp2C7+Dvz3cUe0BYA
=4fyB
-----END PGP SIGNATURE-----

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Jul 27 21:20, Yaakov (Cygwin Ports) wrote:
> Christopher Faylor wrote:
> | It seems like maybe another change to setup.exe would be in order here
> | or maybe there could be a base package which did any cleanup required to
> | purge unneeded packages.
>
> Could setup-1.7 know if the last installation was 1.5, and simply ignore
> the installed.db?

Sounds good.  It could check the last installed Cygwin version in
installed.db, or it could check if the new
"{HKLM,HKCU}/Software/Cygwin/root" registry key exists or not.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Brian Dessent]

"Yaakov (Cygwin Ports)" wrote:

> Could setup-1.7 know if the last installation was 1.5, and simply ignore
> the installed.db?

As outlined in <http://cygwin.com/ml/cygwin-apps/2008-04/msg00169.html>
the way that it will work is entirely dependent on what you enter for
the root location.  It will start up with the value from your existing
1.5 install; if you leave it the same it will read the installed.db from
that location and attempt to upgrade.  If you change the root location
to something else it will forget everything it ever knew about that
previous value (like anything it read from the installed.db there) and
do a new install to that location.  In either case it will write the
value to the new registry key so that it will start with that value the
next time it's run.

Brian

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Jul 25 11:40, Corinna Vinschen wrote:
> > Corinna Vinschen wrote:
> > | I'm not sure if there's really a big difference between these two points.
> > | Since we're using two different installation directories, we can get rid
> > | of old cruft, if we just look carefully what's still used and what not.
> >
> > The difference is if I want to reorganize a package when I rebuild it
> > for 1.7, e.g. right now I have:
> >
> > pcre => pcre, libpcre0, pcre-devel, pcre-doc
> >
> > If I want to rename pcre-devel to libpcre-devel, then normally I would
> > need an empty _obsolete pcre-devel package which deps libpcre-devel to
> > make the upgrade smooth.  That wouldn't be necessary if we don't support
> > upgrading to 1.7 in the same installation.
> >
> > That may seem like a trivial example, but a transition like X11R6 to
> > X11R7 would be a lot bigger.
> 
> Nice example.  Still, for now we should assume that we go the upgrade
> path.  I'm going to investigate the impact of a clean cut in the next
> couple of days.

What if we add an "obsoletes:" line to setup.{ini,hint}?

The idea is that you don't have to provide empty replacement packages
for the old packages anymore, and setup removes all packages noted in
the "obsoletes:" line before installing the new package, in the same
step as when replaing old versions of the same package.

Would that work?  Wouldn't that ease the transition to new package
schemes a lot?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Yaakov (Cygwin Ports)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Corinna Vinschen wrote:
| What if we add an "obsoletes:" line to setup.{ini,hint}?
|
| The idea is that you don't have to provide empty replacement packages
| for the old packages anymore, and setup removes all packages noted in
| the "obsoletes:" line before installing the new package, in the same
| step as when replaing old versions of the same package.
|
| Would that work?  Wouldn't that ease the transition to new package
| schemes a lot?

YES!!

The trick is to make sure that there can be a many-to-many correlation;
IOW a given package could specify plural packages in obsoletes:, and
setup could handle several packages which obsoletes: it.


Yaakov
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREIAAYFAkiQgkgACgkQpiWmPGlmQSMGuACgyk2YRHrc55z1c8iuXXzAmFRr
pUkAoO/k0Tyw/cWRyTgRJZfz+LWj30wq
=3Cb5
-----END PGP SIGNATURE-----

Re: [RFC] 1.7 Packaging: Obsolete packages

[Christopher Faylor]

On Wed, Jul 30, 2008 at 02:18:15PM +0200, Corinna Vinschen wrote:
>On Jul 25 11:40, Corinna Vinschen wrote:
>> > Corinna Vinschen wrote:
>> > | I'm not sure if there's really a big difference between these two points.
>> > | Since we're using two different installation directories, we can get rid
>> > | of old cruft, if we just look carefully what's still used and what not.
>> >
>> > The difference is if I want to reorganize a package when I rebuild it
>> > for 1.7, e.g. right now I have:
>> >
>> > pcre => pcre, libpcre0, pcre-devel, pcre-doc
>> >
>> > If I want to rename pcre-devel to libpcre-devel, then normally I would
>> > need an empty _obsolete pcre-devel package which deps libpcre-devel to
>> > make the upgrade smooth.  That wouldn't be necessary if we don't support
>> > upgrading to 1.7 in the same installation.
>> >
>> > That may seem like a trivial example, but a transition like X11R6 to
>> > X11R7 would be a lot bigger.
>> 
>> Nice example.  Still, for now we should assume that we go the upgrade
>> path.  I'm going to investigate the impact of a clean cut in the next
>> couple of days.
>
>What if we add an "obsoletes:" line to setup.{ini,hint}?
>
>The idea is that you don't have to provide empty replacement packages
>for the old packages anymore, and setup removes all packages noted in
>the "obsoletes:" line before installing the new package, in the same
>step as when replaing old versions of the same package.
>
>Would that work?  Wouldn't that ease the transition to new package
>schemes a lot?

Yes, of course.  It's been suggested before.  It's how other package
managers do it.

Like so many other setup.exe suggestions, it is not likely that anyone
is going to step up to implement it though.  I suggested a post-install
script because those are easy enough to write.

If we're talking about enhancements to setup.exe then another obvious one
is to have it offer to display documentation or at least offer helpful
per-package advice about where to display documentation so that we don't
have to haughtily assume that people should know about the existence of
/usr/share/doc/Cygwin.

cgf

Re: [RFC] 1.7 Packaging: Obsolete packages

[Yaakov (Cygwin Ports)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Corinna Vinschen wrote:
> What if we add an "obsoletes:" line to setup.{ini,hint}?
> 
> The idea is that you don't have to provide empty replacement packages
> for the old packages anymore, and setup removes all packages noted in
> the "obsoletes:" line before installing the new package, in the same
> step as when replaing old versions of the same package.
> 
> Would that work?  Wouldn't that ease the transition to new package
> schemes a lot?

Any progress yet on this one?


Yaakov
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREIAAYFAkis7UQACgkQpiWmPGlmQSM4VwCgpGQKwhSzDfB1sGsFFLDCFiOC
KcMAoKPmRNZ4PQlRPA74jgKEShLn8/S6
=MaIg
-----END PGP SIGNATURE-----

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Aug 20 23:21, Yaakov (Cygwin Ports) wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Corinna Vinschen wrote:
> > What if we add an "obsoletes:" line to setup.{ini,hint}?
> > 
> > The idea is that you don't have to provide empty replacement packages
> > for the old packages anymore, and setup removes all packages noted in
> > the "obsoletes:" line before installing the new package, in the same
> > step as when replaing old versions of the same package.
> > 
> > Would that work?  Wouldn't that ease the transition to new package
> > schemes a lot?
> 
> Any progress yet on this one?

Unfortunately not.  The package management in setup is written in some
sort of martial-arts C++, which I have trouble to follow through.  For
this and for the problem I describe in

http://cygwin.com/ml/cygwin-apps/2008-08/msg00162.html

I would need help.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Aug 21 10:26, Corinna Vinschen wrote:
> On Aug 20 23:21, Yaakov (Cygwin Ports) wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> > 
> > Corinna Vinschen wrote:
> > > What if we add an "obsoletes:" line to setup.{ini,hint}?
> > > 
> > > The idea is that you don't have to provide empty replacement packages
> > > for the old packages anymore, and setup removes all packages noted in
> > > the "obsoletes:" line before installing the new package, in the same
> > > step as when replaing old versions of the same package.
> > > 
> > > Would that work?  Wouldn't that ease the transition to new package
> > > schemes a lot?
> > 
> > Any progress yet on this one?
> 
> Unfortunately not.  The package management in setup is written in some
> sort of martial-arts C++, which I have trouble to follow through.  For
> this and for the problem I describe in
> 
> http://cygwin.com/ml/cygwin-apps/2008-08/msg00162.html
> 
> I would need help.

Setup.exe apparently already understands package lists in setup.ini
which start with "Replaces:" (upper case R), collects the information
and stores it in an internal list, but then doesn't seem to use it
later on.  Too bad.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Aug 21 12:35, Corinna Vinschen wrote:
> On Aug 21 10:26, Corinna Vinschen wrote:
> > On Aug 20 23:21, Yaakov (Cygwin Ports) wrote:
> > > Corinna Vinschen wrote:
> > > > What if we add an "obsoletes:" line to setup.{ini,hint}?
> > > > [...]
> > > Any progress yet on this one?
> > 
> > Unfortunately not.  The package management in setup is written in some
> > sort of martial-arts C++, which I have trouble to follow through.  For
> > this and for the problem I describe in
> > 
> > http://cygwin.com/ml/cygwin-apps/2008-08/msg00162.html
> > 
> > I would need help.
> 
> Setup.exe apparently already understands package lists in setup.ini
> which start with "Replaces:" (upper case R), collects the information
> and stores it in an internal list, but then doesn't seem to use it
> later on.  Too bad.

IIUC, the ConnectedLoopFinder::visit() function is the core function
which creates the dependency order.  What I don't get is this: The
function starts with:

  if (!db.packages[nodeToVisit]->installed)
    /* Can't visit this node, and it is not less than any visted node */
    return db.packages.size() + 1;

So, it skips packages which are not installed?  What about packages
which are about to be installed the first time?  Isn't their "installed"
packageversion NULL and only "desired" is a vaild version?

And then, assuming a package is about to be uninstalled.  Doesn't that
package have a "desired" of NULL?  And shouldn't such a package be
skipped in the dependency tracking?

I don't see where "installed" is set to the "desired" packageversion
before the dependency tracking is called.  Can anybody help me to
understand this?


Corinna


-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Yaakov (Cygwin Ports)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Corinna Vinschen wrote:
> Setup.exe apparently already understands package lists in setup.ini
> which start with "Replaces:" (upper case R), collects the information
> and stores it in an internal list, but then doesn't seem to use it
> later on.  Too bad.

I admit that I don't grok setup's code either.  But if the parser
already handles such a tag (which IMO should be replaces: with a
lower-case r, just like other tags), then it should be straightforward.
 The goal is for an install on such a package to trigger an uninstall on
all packages listed in replaces:, similar to how requires: triggers an
install, except that the former is not recursive.

Unfortunately that's as much as I can contribute at the moment, with my
lack of understanding of the code involved.


Yaakov

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREIAAYFAkit/kYACgkQpiWmPGlmQSPI7wCg7//2WPnmq2NX8Y7ybvaNENKq
jckAoLdCQoZmF40574ML9Fn7dp1YgjM5
=aiPP
-----END PGP SIGNATURE-----

Re: [RFC] 1.7 Packaging: Obsolete packages

[Brian Dessent]

Corinna Vinschen wrote:

> IIUC, the ConnectedLoopFinder::visit() function is the core function
> which creates the dependency order.  What I don't get is this: The

That code is concerned with creating a topological ordering for the
specific purpose of running postinstall scripts.  This isn't called
until after the "remove old and unpack new" install phase is finished
and so the 'installed' member has already been set (e.g. the end of
Installer::installOne.)  Packages that are not selected or were earlier
uninstalled are irrelevant here; this is only about figuring out what
order to do postinstalls.

If you want to see a general example of dependency handling when a
package is selected, see ChooserPage::changeTrust (what is called when
you change everything at once, e.g. Curr to Exp) or
PickPackageLine::click (what is called when you click on a package.)  In
both cases the main workhorse is packagemeta::set_requirements and in
turn packageversion::set_requirements.  The logic for Replaces would
probably have to be wedged in there.

But it's a lot harder than just adding some code to deselect the
packages -- consider if setup followed the "Replaces:" advice and
unselected some packages because the user selected a new package, but
then they changed their mind and deselected that same package.  If we
don't go back and re-enable those packages then we potentially leave
them with a broken system.  We could delay this processing until later
in the process when the user no longer can change their mind, but then
the package selection pane becomes a lie.  We will potentially be
removing stuff that isn't marked as remove, and the "Selected" view no
longer accurately serves as a "here's a list of everything I'm about to
do" summary which I think is a valuable feature.

Brian

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Aug 21 18:29, Brian Dessent wrote:
> Corinna Vinschen wrote:
> 
> > IIUC, the ConnectedLoopFinder::visit() function is the core function
> > which creates the dependency order.  What I don't get is this: The
> 
> That code is concerned with creating a topological ordering for the
> specific purpose of running postinstall scripts.  This isn't called
> until after the "remove old and unpack new" install phase is finished
> [...]

Oh, cool, I was looking into the wrong code all the time.  No surprise
I coudn't figure out how it works.

> If you want to see a general example of dependency handling when a
> package is selected, see ChooserPage::changeTrust (what is called when
> you change everything at once, e.g. Curr to Exp) or
> PickPackageLine::click (what is called when you click on a package.)  In
> both cases the main workhorse is packagemeta::set_requirements and in
> turn packageversion::set_requirements.  The logic for Replaces would
> probably have to be wedged in there.

Ok, I'll have a look.  Any idea about my other question?  How to remove
the entire installed.db package DB when the user goes back to the root
dir selection dialog so we can reload from another location, should the
user choose one?

> But it's a lot harder than just adding some code to deselect the
> packages -- consider if setup followed the "Replaces:" advice and
> unselected some packages because the user selected a new package, but
> then they changed their mind and deselected that same package.  If we
> don't go back and re-enable those packages then we potentially leave
> them with a broken system.  We could delay this processing until later
> in the process when the user no longer can change their mind, but then
> the package selection pane becomes a lie.  We will potentially be
> removing stuff that isn't marked as remove, and the "Selected" view no
> longer accurately serves as a "here's a list of everything I'm about to
> do" summary which I think is a valuable feature.

Is that really a big point?  The "replaces" stuff is meant to seemlessly
replace one package with (usually) a functionally at least equivalent
package.  There shouldn't be much of a difference for the user.

OTOH, a "replaces" rule does not allow to replace an obsolete package
automatically.  So, if we want to upgrade the user automatically to the
new packages, we would *still* have to provide empty obsolete packages
with a setup.hint requiring the new packages.

What would the "replaces" rule buy us again?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Brian Dessent]

Corinna Vinschen wrote:

> Ok, I'll have a look.  Any idea about my other question?  How to remove
> the entire installed.db package DB when the user goes back to the root
> dir selection dialog so we can reload from another location, should the
> user choose one?

My aborted unfinished attempt at a 1.7 setup also suffered from this. 
The plan of attack that I had was to move the code that reads
installed.db out of the packagedb ctor and into its own private or
static function.  Then write a new member function that first frees the
in-memory copy (if it exists) and then calls that reader function.  Then
hook this function into the wizard dialog so that it's called every time
you move off the rootpage.

As far as the actual freeing of memory, I think it would go something
like

for (vector <packagemeta *>::iterator i = packages.begin ();
     i != packages.end (); ++i)
  {
    delete *i;
  }
packages.clear();

(And likewise for sourcePackages.)  Yes, it royally sucks that setup has
this horrific mix of containers as well as explicit pointer management. 
Ideally it would be great to eliminate all explicit new/delete
housekeeping and instead make them smart pointers, but that's a big
project.

> Is that really a big point?  The "replaces" stuff is meant to seemlessly
> replace one package with (usually) a functionally at least equivalent
> package.  There shouldn't be much of a difference for the user.

You mean is it not a big point that the uninstallation of the packages
listed in "Replaces:" wouldn't be indicated anywhre?  Yes, I suppose
it's somewhat of a minor point.  But I really do think that setup has
enough odd and inexplicable behavior already that I hesitate to add more
-- being able to always double check the precise set of
install/remove/upgrade operations that are about to take place with the
"Partial" view is a useful debugging tool.

> OTOH, a "replaces" rule does not allow to replace an obsolete package
> automatically.  So, if we want to upgrade the user automatically to the
> new packages, we would *still* have to provide empty obsolete packages
> with a setup.hint requiring the new packages.
> 
> What would the "replaces" rule buy us again?

I don't know.  Maybe what is needed is a key that works in the other
direction, "Replaced-by:" with the semantics of: if this package is
currently installed then de-install it and install everything in the
specified list.  So to rename/split a package you would upload the new
package(s) under the new name(s), move the old package to category
_obsolete (so that it's not shown) and add a "Replaced-by: newname"
line.  This would cause a silent upgrade for everyone the next time they
ran setup, the same way that bumping the version and uploading an empty
file does today.

Off the top of my head, complication with this idea:

- the Replaced-by would have to be transitive in the dependency
computation code as well.  So if a maintainer renames package OLDNAME to
NEWNAME, then a package that "requires:" OLDNAME would have to have
OLDNAME replaced with NEWNAME in the internal representation of the
dependency list.  It's tempting to say that the maintainer should just
add NEWNAME to the "requires:" of OLDNAME, and let the requirement
chain.  But I don't think that would fly because what we're saying with
this keyword is that OLDNAME won't be installed any more.  This is in
opposition to the current method where OLDNAME does actually stay
installed (in the mind of setup) but contains an empty payload so has no
footprint.

- the Replaced-by method would not allow a determined user to continue
using an old version of a package without upgrading.  With the current
scheme they can just mark the existing package as "Keep" (or select a
"Prev" version) which has the effect of blocking the upgrade since it
works on the principle of the bumped version triggering the action.  But
with a Replaced-by keyword doing the action instead of the presence of a
bumped version, there would be no choice.

Brian

Re: [RFC] 1.7 Packaging: Obsolete packages

[Brian Dessent]

Brian Dessent wrote:

> - the Replaced-by would have to be transitive in the dependency
> computation code as well.  So if a maintainer renames package OLDNAME to

And, as a corollary to that: Replaced-by should accept only a single
packagename as predicate, since we have this requirement of replacing
any occurance of OLDNAME with NEWNAME in the internal dependency
database.  I suppose it could in theory take a list of multiple packages
but then we'd have to deal with replacing one thing with multiple things
and that just seems overly complicated.  If a one-to-many relationship
is required it can be achieved by setting the "requires:" of NEWNAME to
pull in the extra ones.

Brian

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Aug 22 01:52, Brian Dessent wrote:
> Corinna Vinschen wrote:
> 
> > Ok, I'll have a look.  Any idea about my other question?  How to remove
> > the entire installed.db package DB when the user goes back to the root
> > dir selection dialog so we can reload from another location, should the
> > user choose one?
> 
> As far as the actual freeing of memory, I think it would go something
> like
> 
> for (vector <packagemeta *>::iterator i = packages.begin ();
>      i != packages.end (); ++i)
>   {
>     delete *i;
>   }
> packages.clear();
> 
> (And likewise for sourcePackages.)

If that works, it should be fairly simple.  Since the installed.db is
the first time read when pressing the "Next" button in the root dir
dialog, it might be best to place the cleanup above in the same
place, right before initializing the packagedb.

> > What would the "replaces" rule buy us again?
> 
> I don't know.  Maybe what is needed is a key that works in the other
> direction, "Replaced-by:" with the semantics of: if this package is
> currently installed then de-install it and install everything in the
> specified list.  So to rename/split a package you would upload the new
> package(s) under the new name(s), move the old package to category
> _obsolete (so that it's not shown) and add a "Replaced-by: newname"
> line.  This would cause a silent upgrade for everyone the next time they
> ran setup, the same way that bumping the version and uploading an empty
> file does today.
> 
> Off the top of my head, complication with this idea:
> 
> - the Replaced-by would have to be transitive in the dependency
> computation code as well.  So if a maintainer renames package OLDNAME to
> NEWNAME, then a package that "requires:" OLDNAME would have to have
> OLDNAME replaced with NEWNAME in the internal representation of the
> dependency list.  It's tempting to say that the maintainer should just
> add NEWNAME to the "requires:" of OLDNAME, and let the requirement
> chain.  But I don't think that would fly because what we're saying with
> this keyword is that OLDNAME won't be installed any more.  This is in
> opposition to the current method where OLDNAME does actually stay
> installed (in the mind of setup) but contains an empty payload so has no
> footprint.
> 
> - the Replaced-by method would not allow a determined user to continue
> using an old version of a package without upgrading.  With the current
> scheme they can just mark the existing package as "Keep" (or select a
> "Prev" version) which has the effect of blocking the upgrade since it
> works on the principle of the bumped version triggering the action.  But
> with a Replaced-by keyword doing the action instead of the presence of a
> bumped version, there would be no choice.

IMHO, the replaced-by would add nothing in terms of less maintainence
burden.  The old package still has to be tweaked in one way or another.
The only extra work without replaced-by is to create empty tar archives
for the old packages, which is really simple.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Brian Dessent]

Brian Dessent wrote:

> - the Replaced-by method would not allow a determined user to continue
> using an old version of a package without upgrading.  With the current
> scheme they can just mark the existing package as "Keep" (or select a
> "Prev" version) which has the effect of blocking the upgrade since it
> works on the principle of the bumped version triggering the action.  But
> with a Replaced-by keyword doing the action instead of the presence of a
> bumped version, there would be no choice.

And thinking about this some more, we could work around that issue by
implementing the treatment of the Replaced-by keyword "as-if" the user
had manually deselected OLDNAME and selected NEWNAME.  Thus if they
select "Keep" then that action is undone and the list of operations is
cleared as expected, and all packages stay at what they're set to now. 
Of course we'd need to hook it into the right places so that these
"as-if" selections are not just done once but every time that the trust
level is set to "Curr".  Otherwise, you'd miss the upgrade if you
clicked on Keep then clicked again on Curr (or Exp for that matter.)

So to sum up an implementation skeleton:

- add the keyword to the lexer and parser
- add an additional packagedb data member that contains a vector of
OLDNAME->NEWNAME pairs found from the result of parsing the ini file
- add code that runs after reading the installed.db that changes any
occurance of OLDNAME to NEWNAME in any package's dependency spec
- add code in the "set global trustval" codepath that runs after all the
exising calculations to set the transitions "as-if" the user had
deselected OLDNAME and selected NEWNAME for each pair, recomputing full
recursive dependencies each time -- but *only* if setting the trustval
to curr or exp, not keep.

Brian

Re: [RFC] 1.7 Packaging: Obsolete packages

[Brian Dessent]

Corinna Vinschen wrote:

> IMHO, the replaced-by would add nothing in terms of less maintainence
> burden.  The old package still has to be tweaked in one way or another.
> The only extra work without replaced-by is to create empty tar archives
> for the old packages, which is really simple.

As I do not maintain a huge number of packages I really can't comment on
what maintainers need or want.  Certainly we shouldn't waste time
designing something that's not going to be helpful.  But I do seem to
recall that Yaakov stated early in this thread that some kind of help
with renaming/splitting packages would save a great deal of work for
him, so perhaps we should wait for him or someone else with tons of
packages to comment.

Brian

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Aug 22 11:24, Corinna Vinschen wrote:
> On Aug 22 01:52, Brian Dessent wrote:
> > As far as the actual freeing of memory, I think it would go something
> > like
> > 
> > for (vector <packagemeta *>::iterator i = packages.begin ();
> >      i != packages.end (); ++i)
> >   {
> >     delete *i;
> >   }
> > packages.clear();
> > 
> > (And likewise for sourcePackages.)
> 
> If that works, it should be fairly simple.  Since the installed.db is
> the first time read when pressing the "Next" button in the root dir
> dialog, it might be best to place the cleanup above in the same
> place, right before initializing the packagedb.

Too simple, apparently.  I created a new static method packagedb::clear()
which I call in RootPage::OnNext():

  void
  packagedb::clear ()
  {
    for (vector <packagemeta *>::iterator i = packages.begin ();
	 i != packages.end (); ++i)
      delete *i;
    packages.clear ();
    for (vector <packagemeta *>::iterator i = sourcePackages.begin ();
	 i != sourcePackages.end (); ++i)
      {

/* If you don't clear the categories, deleting the source packages
   SEGVs.  Are the categories lists shared with the binary packagemeta? */

	(*i)->categories.clear ();
	delete *i;
      }
    sourcePackages.clear ();
    installeddbread = 0;
  }

For testing I started setup and just clicked "Next" up to the package
selection dialog.  This picked up all information from a former 1.5
install and there's a list of packages to update. 

Then I clicked "Back" up to the root dir dialog, changed the root dir to
a non-existing "Cygwin-1.7" and clicked "Next" again up to the package
selection.  The result was that Setup now thinks it has nothing to
install/update.  Nothing at all.

Something's obviously missing...


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: [RFC] 1.7 Packaging: Obsolete packages

[Brian Dessent]

Corinna Vinschen wrote:

> Something's obviously missing...

Yes, I led you astray, sorry.  That is going to purge absolutely
everything that setup knows about packages.  In order to get that back
it would be necessary to re-parse setup.ini and re-scan the local
package directory for its contents to see what's available.

What's needed is rather than nuking the database, to undo the
information about what versions are currently installed.  This is rather
dicey because each packagemeta contains a std::set of packageversions,
each of which could have come from setup.ini, or from the parsing of the
filename in installed.db.  So we can't just remove everything from that
set.  But if you don't remove anything you'd leak (or end up with
duplicates, or end up with nonsense versions) because the code in the
installed.db reader (below) creates a new packageversion
unconditionally.  Really what we want to do is undo this:

      packageversion binary = 
        cygpackage::createInstance (pkgname, inst, instsz, f.ver,
                                    package_installed,
                                    package_binary);
 
      pkg->add_version (binary);
      pkg->set_installed (binary);
      pkg->desired = pkg->installed;

I think what might be required is to add another field to packageversion
to denote whether it came from reading setup.ini or from reading
installed.db.  (Remember that it's possible that the installed version
is not Curr nor Prev nor Exp, it could be anything.)  Then add a method
to packagemeta that removes from 'versions' any that came from
installed.db, leaving the ones that came from setup.ini.  (This will
need an operator delete to balance the operator new in
cygpackage::createInstance otherwise it leaks.)  Then we can call that
method on each packagemeta in the packagedb::packages vector.  We'll
also need to set the 'desired' and 'installed' members to point to
something sensible, because otherwise they are potentially dangling if
they pointed to something we removed.  I'm not sure what that something
is, but to find out you can put some debugging statements in and find
out what everything looks like after parsing setup.ini and scanning the
local package dir but prior to reading installed.db to find out how to
reverse the changes.

Brian

Re: [RFC] 1.7 Packaging: Obsolete packages

[Charles Wilson]

Brian Dessent wrote:
> Yes, it royally sucks that setup has
> this horrific mix of containers as well as explicit pointer management. 
> Ideally it would be great to eliminate all explicit new/delete
> housekeeping and instead make them smart pointers, but that's a big
> project.

Here's a start: a non-intrusive ref-counting smart pointer I wrote a few
years back:
http://code.google.com/p/optionparse/source/browse/trunk/lib/OptionParse/ExternalCountedPtr.h
(or you could grab the boost one)

'Course, actually converting all raw pointer use in setup over to smart
pointers is still a big project. <g>

--
Chuck

Re: [RFC] 1.7 Packaging: Obsolete packages

[Corinna Vinschen]

On Aug 22 03:53, Brian Dessent wrote:
> Corinna Vinschen wrote:
> 
> > Something's obviously missing...
> 
> Yes, I led you astray, sorry.  That is going to purge absolutely
> everything that setup knows about packages.  In order to get that back
> it would be necessary to re-parse setup.ini and re-scan the local
> package directory for its contents to see what's available.

That's what I had expected, actually.  If the user goes back to the
root dir dialog, I don't quite care if the user's selections are
not remembered.  I know, it's not overly good GUI design, but it
would have been at least better than what we have today.

> What's needed is rather than nuking the database, to undo the
> information about what versions are currently installed.  This is rather
> dicey because each packagemeta contains a std::set of packageversions,
> each of which could have come from setup.ini, or from the parsing of the
> filename in installed.db.  So we can't just remove everything from that
> set.  But if you don't remove anything you'd leak (or end up with
> duplicates, or end up with nonsense versions) because the code in the
> installed.db reader (below) creates a new packageversion
> unconditionally.  Really what we want to do is undo this:
> 
>       packageversion binary = 
>         cygpackage::createInstance (pkgname, inst, instsz, f.ver,
>                                     package_installed,
>                                     package_binary);
>  
>       pkg->add_version (binary);
>       pkg->set_installed (binary);
>       pkg->desired = pkg->installed;
> 
> I think what might be required is to add another field to packageversion
> to denote whether it came from reading setup.ini or from reading
> installed.db.  (Remember that it's possible that the installed version
> is not Curr nor Prev nor Exp, it could be anything.) 
> [and lots of stuff]

Baeh.  I'm not sure I can manage to implement that before my vacation.
Brian, do you have any spare cycles in the next couple of weeks?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat