[RFC] cygport pm for managing package releases

[RFC] cygport pm for managing package releases

[Andrew Schulman]

cygport has automated a lot of the work of building and maintaining
packages for Cygwin. But one area where it doesn't help yet is in managing
the available releases of a package. For me as the maintainer of a dozen or
so packages, there are routine tasks that I still find to be painful:

* Finding out which versions of a package are currently available in the
Cygwin repositories, and which if any are marked as "test".
* Marking or unmarking a version as "test".
* Removing versions from the repositories.
* Marking a package as obsolete.

All of these are still manual tasks. Most require digging through
documentation (though that's also much improved in the last few years),
manually editing .hint files or creating dummy package files, and manually
uploading files to the right places in sftp://cygwin.com. It's not fun, and
so I don't keep up with it as well as I should.

To alleviate that, I think cygport should add a set of "pm" commands to
automate package management. For example:

* cygport pm list - list versions available in the Cygwin repositories.
* cygport pm test - set/clear "test" status for a version.
* cygport pm del - remove a package version from the repositories.
* cygport pm obsolete - mark a package as obsolete.

And probably others. I think this would make maintainer's lives easier, and
make these management tasks more reliable.

I can spend some time planning and developing this, and if others want to
collaborate on it, so much the better. But before I start on that, I want
to get people's comments here about whether:

* It's worth doing;
* (More to the point) It'd be likely to be accepted upstream, assuming the
implementation is satisfactory; and 
* There may be problems in implementing it that I haven't thought of. 

I can think of a few problems or objections:

1. The "pm" commands will bake into cygport logic that's specific to how
the package repositories and upset currently work. So if those change,
cygport will have to change to match them. That's true, but not just for
cygport pm - other parts of cygport, such as cygport up, are basically
clients for upset. And at least it'll centralize the changes in one place,
so maintainers won't have to worry about them.

2. "pm list" will require finding and parsing an appropriate setup.ini
file, unlike the other "pm" commands which will manipulate
sftp://cygwin.com. 

I think these are surmountable, but I want to know if there's a general
agreement that it's worth doing.

BTW a successful example like this one is the "cygport up" command, which
we added a few years ago to automate uploading packages to cygwin.com. I
think it's working well.

Andrew

Re: [RFC] cygport pm for managing package releases

[Brian Inglis]

On 2020-09-11 10:07, Andrew Schulman via Cygwin-apps wrote:
> cygport has automated a lot of the work of building and maintaining
> packages for Cygwin. But one area where it doesn't help yet is in managing
> the available releases of a package. For me as the maintainer of a dozen or
> so packages, there are routine tasks that I still find to be painful:
> 
> * Finding out which versions of a package are currently available in the
> Cygwin repositories, and which if any are marked as "test".
> * Marking or unmarking a version as "test".
> * Removing versions from the repositories.
> * Marking a package as obsolete.
> 
> All of these are still manual tasks. Most require digging through
> documentation (though that's also much improved in the last few years),
> manually editing .hint files or creating dummy package files, and manually
> uploading files to the right places in sftp://cygwin.com. It's not fun, and
> so I don't keep up with it as well as I should.
> 
> To alleviate that, I think cygport should add a set of "pm" commands to
> automate package management. For example:
> 
> * cygport pm list - list versions available in the Cygwin repositories.
> * cygport pm test - set/clear "test" status for a version.
> * cygport pm del - remove a package version from the repositories.
> * cygport pm obsolete - mark a package as obsolete.
> 
> And probably others. I think this would make maintainer's lives easier, and
> make these management tasks more reliable.
> 
> I can spend some time planning and developing this, and if others want to
> collaborate on it, so much the better. But before I start on that, I want
> to get people's comments here about whether:
> 
> * It's worth doing;
> * (More to the point) It'd be likely to be accepted upstream, assuming the
> implementation is satisfactory; and 
> * There may be problems in implementing it that I haven't thought of. 
> 
> I can think of a few problems or objections:
> 
> 1. The "pm" commands will bake into cygport logic that's specific to how
> the package repositories and upset currently work. So if those change,
> cygport will have to change to match them. That's true, but not just for
> cygport pm - other parts of cygport, such as cygport up, are basically
> clients for upset. And at least it'll centralize the changes in one place,
> so maintainers won't have to worry about them.
> 
> 2. "pm list" will require finding and parsing an appropriate setup.ini
> file, unlike the other "pm" commands which will manipulate
> sftp://cygwin.com. 
> 
> I think these are surmountable, but I want to know if there's a general
> agreement that it's worth doing.
> 
> BTW a successful example like this one is the "cygport up" command, which
> we added a few years ago to automate uploading packages to cygwin.com. I
> think it's working well.

Also cygport commands package-test and all-test added more recently should make
test package management easier.

I and others have forked and hack updates to apt-cyg (static) on github, and
push changes to our forks when they are stable enough, and find that allows me
to easily do a lot with setup.ini and installed.db.

Perhaps we should first outline the maintainer packaging workflow, including
required functions such as creating a package directory and contents, checking
for other repos with a package, sending ITPs/ITAs and SSH public keys, checking
licensing, checking for new upstream releases, and less common functions such as
those you mention, sending upstream emails and submitting patches or PRs, and
others, with a summary like the cygport --help output, description as in the
cygport HTML help, specification of what is needed and why, and example commands
to execute, if known, to implement the function.

Perhaps the additional functions and commands could be guided by the functions
provided for Gentoo portage tools, Debian dh, Fedora fedpkg, and OpenSuSE osc
packaging tools.

Other functions could be like OpenSuSE osc submitrequest, requestmaintainership
to send standardized ITPs/ITAs, Debian equivs to create trivial package files,
lintian/fedpkg lint to check package metadata, etc.

Once we have a list of maintainer functions, we should consider what maintainers
consider pain points to add as cygport commands, plus quick and easy wins to
help maintainers contribute while understanding cygport development and docs.

I like the single word commands in cygport and other tools e.g. your pm list is
like apt show, pm test and pm obsolete remind me of apt mark, and possibly also
pm del, depending on whether you expect those commands to work on .hint files or
upload directories or both, and distinctions and expectations like those need to
be explained.

Should we work with patches, PRs, a dev repo (on sourceware? or github) against
https://cygwin.com/git/?p=cygwin-apps/cygport.git, or other storage space(s), to
maintain lists of workflows, suggested functions, maintainer pain points,
possible commands, command summaries, outlines, help descriptions, spec details,
and commands to execute in an implementation.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]

Re: [RFC] cygport pm for managing package releases

[Ken Brown]

On 9/11/2020 12:07 PM, Andrew Schulman via Cygwin-apps wrote:
> cygport has automated a lot of the work of building and maintaining
> packages for Cygwin. But one area where it doesn't help yet is in managing
> the available releases of a package. For me as the maintainer of a dozen or
> so packages, there are routine tasks that I still find to be painful:
> 
> * Finding out which versions of a package are currently available in the
> Cygwin repositories, and which if any are marked as "test".
> * Marking or unmarking a version as "test".
> * Removing versions from the repositories.
> * Marking a package as obsolete.
> 
> All of these are still manual tasks. Most require digging through
> documentation (though that's also much improved in the last few years),
> manually editing .hint files or creating dummy package files, and manually
> uploading files to the right places in sftp://cygwin.com. It's not fun, and
> so I don't keep up with it as well as I should.
> 
> To alleviate that, I think cygport should add a set of "pm" commands to
> automate package management. For example:
> 
> * cygport pm list - list versions available in the Cygwin repositories.
> * cygport pm test - set/clear "test" status for a version.
> * cygport pm del - remove a package version from the repositories.
> * cygport pm obsolete - mark a package as obsolete.
> 
> And probably others. I think this would make maintainer's lives easier, and
> make these management tasks more reliable.
> 
> I can spend some time planning and developing this, and if others want to
> collaborate on it, so much the better. But before I start on that, I want
> to get people's comments here about whether:
> 
> * It's worth doing;
> * (More to the point) It'd be likely to be accepted upstream, assuming the
> implementation is satisfactory; and
> * There may be problems in implementing it that I haven't thought of.
> 
> I can think of a few problems or objections:
> 
> 1. The "pm" commands will bake into cygport logic that's specific to how
> the package repositories and upset currently work. So if those change,
> cygport will have to change to match them. That's true, but not just for
> cygport pm - other parts of cygport, such as cygport up, are basically
> clients for upset. And at least it'll centralize the changes in one place,
> so maintainers won't have to worry about them.
> 
> 2. "pm list" will require finding and parsing an appropriate setup.ini
> file, unlike the other "pm" commands which will manipulate
> sftp://cygwin.com.
> 
> I think these are surmountable, but I want to know if there's a general
> agreement that it's worth doing.
> 
> BTW a successful example like this one is the "cygport up" command, which
> we added a few years ago to automate uploading packages to cygwin.com. I
> think it's working well.

Agreed.  Thanks for doing this.

Concerning your specific suggestions:

 > * cygport pm list - list versions available in the Cygwin repositories.

Good idea.  I often find myself looking at setup.ini to get this information, 
and it would be nice to have cygport automate the process.

 > * cygport pm test - set/clear "test" status for a version.

I like the idea of clearing test status, i.e., 'cygport pm untest'; this should 
be trivial to implement in view of Jon's recent work:

   https://cygwin.com/pipermail/cygwin-apps/2020-August/040440.html

But I'm not sure about going in the other direction.  Once users have already 
installed a non-test package, it could be very confusing to have that package 
retroactively declared to be a test release.

 > * cygport pm del - remove a package version from the repositories.

This would be very useful.  The current mechanism for removing a package version 
is very tedious.

 > * cygport pm obsolete - mark a package as obsolete.

I was about to question the need for this, but I'll bet you're thinking about 
unison2.48.  It will soon become obsolete, with two or more possible replacement 
packages.  So the usual mechanism of having a new package obsolete an old one 
doesn't quite work.

Ken

Re: [RFC] cygport pm for managing package releases

[Andrew Schulman]

> Perhaps we should first outline the maintainer packaging workflow, including
> required functions such as creating a package directory and contents, checking
> for other repos with a package, sending ITPs/ITAs and SSH public keys, checking
> licensing, checking for new upstream releases, and less common functions such as
> those you mention, sending upstream emails and submitting patches or PRs, and
> others, with a summary like the cygport --help output, description as in the
> cygport HTML help, specification of what is needed and why, and example commands
> to execute, if known, to implement the function.

<snip>

> Once we have a list of maintainer functions, we should consider what maintainers
> consider pain points to add as cygport commands, plus quick and easy wins to
> help maintainers contribute while understanding cygport development and docs.

That sounds like a good approach, although I wouldn't want to get too bogged
down in details at first. I like the idea of outlining the packaging workflow
first, then automating the most common+important+painful pieces, and adding
others later as the capability matures.

Another consideration is how cygport will be affected as we move to an automated
build system. We wouldn't want to build to a bunch of stuff that's just going to
go away. But actually I think that once the builds are automated and we're all
just uploading our cygport files, the package management functions will become
more important, since we'll be more hands-off of the .hint files.

> I like the single word commands in cygport and other tools e.g. your pm list is
> like apt show, pm test and pm obsolete remind me of apt mark, and possibly also
> pm del, depending on whether you expect those commands to work on .hint files or
> upload directories or both, and distinctions and expectations like those need to
> be explained.

Agree. And clarified first.

> Should we work with patches, PRs, a dev repo (on sourceware? or github) against
> https://cygwin.com/git/?p=cygwin-apps/cygport.git, or other storage space(s), to
> maintain lists of workflows, suggested functions, maintainer pain points,
> possible commands, command summaries, outlines, help descriptions, spec details,
> and commands to execute in an implementation.

A Github repo sounds like an easy way to start. I'm open to suggestions if there
are better ways.

Andrew

Re: [RFC] cygport pm for managing package releases

[Brian Inglis]

On 2020-09-12 09:03, Ken Brown via Cygwin-apps wrote:
> On 9/11/2020 12:07 PM, Andrew Schulman via Cygwin-apps wrote:
>> cygport has automated a lot of the work of building and maintaining
>> packages for Cygwin. But one area where it doesn't help yet is in managing
>> the available releases of a package. For me as the maintainer of a dozen or
>> so packages, there are routine tasks that I still find to be painful:
>>
>> * Finding out which versions of a package are currently available in the
>> Cygwin repositories, and which if any are marked as "test".
>> * Marking or unmarking a version as "test".
>> * Removing versions from the repositories.
>> * Marking a package as obsolete.
>>
>> All of these are still manual tasks. Most require digging through
>> documentation (though that's also much improved in the last few years),
>> manually editing .hint files or creating dummy package files, and manually
>> uploading files to the right places in sftp://cygwin.com. It's not fun, and
>> so I don't keep up with it as well as I should.
>>
>> To alleviate that, I think cygport should add a set of "pm" commands to
>> automate package management. For example:
>>
>> * cygport pm list - list versions available in the Cygwin repositories.
>> * cygport pm test - set/clear "test" status for a version.
>> * cygport pm del - remove a package version from the repositories.
>> * cygport pm obsolete - mark a package as obsolete.
>>
>> And probably others. I think this would make maintainer's lives easier, and
>> make these management tasks more reliable.
>>
>> I can spend some time planning and developing this, and if others want to
>> collaborate on it, so much the better. But before I start on that, I want
>> to get people's comments here about whether:
>>
>> * It's worth doing;
>> * (More to the point) It'd be likely to be accepted upstream, assuming the
>> implementation is satisfactory; and
>> * There may be problems in implementing it that I haven't thought of.
>>
>> I can think of a few problems or objections:
>>
>> 1. The "pm" commands will bake into cygport logic that's specific to how
>> the package repositories and upset currently work. So if those change,
>> cygport will have to change to match them. That's true, but not just for
>> cygport pm - other parts of cygport, such as cygport up, are basically
>> clients for upset. And at least it'll centralize the changes in one place,
>> so maintainers won't have to worry about them.
>>
>> 2. "pm list" will require finding and parsing an appropriate setup.ini
>> file, unlike the other "pm" commands which will manipulate
>> sftp://cygwin.com.
>>
>> I think these are surmountable, but I want to know if there's a general
>> agreement that it's worth doing.
>>
>> BTW a successful example like this one is the "cygport up" command, which
>> we added a few years ago to automate uploading packages to cygwin.com. I
>> think it's working well.
> 
> Agreed.  Thanks for doing this.
> 
> Concerning your specific suggestions:
> 
>> * cygport pm list - list versions available in the Cygwin repositories.
> 
> Good idea.  I often find myself looking at setup.ini to get this information,
> and it would be nice to have cygport automate the process.
> 
>> * cygport pm test - set/clear "test" status for a version.
> 
> I like the idea of clearing test status, i.e., 'cygport pm untest'; this should
> be trivial to implement in view of Jon's recent work:
> 
>   https://cygwin.com/pipermail/cygwin-apps/2020-August/040440.html
> 
> But I'm not sure about going in the other direction.  Once users have already
> installed a non-test package, it could be very confusing to have that package
> retroactively declared to be a test release.
> 
>> * cygport pm del - remove a package version from the repositories.
> 
> This would be very useful.  The current mechanism for removing a package version
> is very tedious.
> 
>> * cygport pm obsolete - mark a package as obsolete.
> 
> I was about to question the need for this, but I'll bet you're thinking about
> unison2.48.  It will soon become obsolete, with two or more possible replacement
> packages.  So the usual mechanism of having a new package obsolete an old one
> doesn't quite work.

Python 2/2.7 (308) packages also come to mind as being dropped at some point in
the next year as there is no longer any support.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]

Re: [RFC] cygport pm for managing package releases

[Hamish McIntyre-Bhatty]

[-- Attachment #1.1.1: Type: text/plain, Size: 4663 bytes --]

On 12/09/2020 22:20, Brian Inglis wrote:
> On 2020-09-12 09:03, Ken Brown via Cygwin-apps wrote:
>> On 9/11/2020 12:07 PM, Andrew Schulman via Cygwin-apps wrote:
>>> cygport has automated a lot of the work of building and maintaining
>>> packages for Cygwin. But one area where it doesn't help yet is in managing
>>> the available releases of a package. For me as the maintainer of a dozen or
>>> so packages, there are routine tasks that I still find to be painful:
>>>
>>> * Finding out which versions of a package are currently available in the
>>> Cygwin repositories, and which if any are marked as "test".
>>> * Marking or unmarking a version as "test".
>>> * Removing versions from the repositories.
>>> * Marking a package as obsolete.
>>>
>>> All of these are still manual tasks. Most require digging through
>>> documentation (though that's also much improved in the last few years),
>>> manually editing .hint files or creating dummy package files, and manually
>>> uploading files to the right places in sftp://cygwin.com. It's not fun, and
>>> so I don't keep up with it as well as I should.
>>>
>>> To alleviate that, I think cygport should add a set of "pm" commands to
>>> automate package management. For example:
>>>
>>> * cygport pm list - list versions available in the Cygwin repositories.
>>> * cygport pm test - set/clear "test" status for a version.
>>> * cygport pm del - remove a package version from the repositories.
>>> * cygport pm obsolete - mark a package as obsolete.
>>>
>>> And probably others. I think this would make maintainer's lives easier, and
>>> make these management tasks more reliable.
>>>
>>> I can spend some time planning and developing this, and if others want to
>>> collaborate on it, so much the better. But before I start on that, I want
>>> to get people's comments here about whether:
>>>
>>> * It's worth doing;
>>> * (More to the point) It'd be likely to be accepted upstream, assuming the
>>> implementation is satisfactory; and
>>> * There may be problems in implementing it that I haven't thought of.
>>>
>>> I can think of a few problems or objections:
>>>
>>> 1. The "pm" commands will bake into cygport logic that's specific to how
>>> the package repositories and upset currently work. So if those change,
>>> cygport will have to change to match them. That's true, but not just for
>>> cygport pm - other parts of cygport, such as cygport up, are basically
>>> clients for upset. And at least it'll centralize the changes in one place,
>>> so maintainers won't have to worry about them.
>>>
>>> 2. "pm list" will require finding and parsing an appropriate setup.ini
>>> file, unlike the other "pm" commands which will manipulate
>>> sftp://cygwin.com.
>>>
>>> I think these are surmountable, but I want to know if there's a general
>>> agreement that it's worth doing.
>>>
>>> BTW a successful example like this one is the "cygport up" command, which
>>> we added a few years ago to automate uploading packages to cygwin.com. I
>>> think it's working well.
>> Agreed.  Thanks for doing this.
>>
>> Concerning your specific suggestions:
>>
>>> * cygport pm list - list versions available in the Cygwin repositories.
>> Good idea.  I often find myself looking at setup.ini to get this information,
>> and it would be nice to have cygport automate the process.
>>
>>> * cygport pm test - set/clear "test" status for a version.
>> I like the idea of clearing test status, i.e., 'cygport pm untest'; this should
>> be trivial to implement in view of Jon's recent work:
>>
>>   https://cygwin.com/pipermail/cygwin-apps/2020-August/040440.html
>>
>> But I'm not sure about going in the other direction.  Once users have already
>> installed a non-test package, it could be very confusing to have that package
>> retroactively declared to be a test release.
>>
>>> * cygport pm del - remove a package version from the repositories.
>> This would be very useful.  The current mechanism for removing a package version
>> is very tedious.
>>
>>> * cygport pm obsolete - mark a package as obsolete.
>> I was about to question the need for this, but I'll bet you're thinking about
>> unison2.48.  It will soon become obsolete, with two or more possible replacement
>> packages.  So the usual mechanism of having a new package obsolete an old one
>> doesn't quite work.
> Python 2/2.7 (308) packages also come to mind as being dropped at some point in
> the next year as there is no longer any support.
>
I would definitely find these features helpful so you have my vote on
these additions for sure.

Hamish


[-- Attachment #1.1.2: 0x87B761FE07F548D6.asc --]
[-- Type: application/pgp-keys, Size: 3235 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] cygport pm for managing package releases

[Jon Turney]

On 11/09/2020 17:07, Andrew Schulman via Cygwin-apps wrote:
> cygport has automated a lot of the work of building and maintaining
> packages for Cygwin. But one area where it doesn't help yet is in managing
> the available releases of a package. For me as the maintainer of a dozen or
> so packages, there are routine tasks that I still find to be painful:
> 
> * Finding out which versions of a package are currently available in the
> Cygwin repositories, and which if any are marked as "test".

While this isn't the ideal way to find it, the package summary page now 
provides this information e.g. [1]

https://cygwin.com/packages/summary/socat-src.html

> * Marking or unmarking a version as "test".
> * Removing versions from the repositories.
> * Marking a package as obsolete.

I'd add to this list 'generating and sending the package announce email'.

> All of these are still manual tasks. Most require digging through
> documentation (though that's also much improved in the last few years),
> manually editing .hint files or creating dummy package files, and manually
> uploading files to the right places in sftp://cygwin.com. It's not fun, and
> so I don't keep up with it as well as I should.
> 
[...]
> 
> I think these are surmountable, but I want to know if there's a general
> agreement that it's worth doing.
> 
> BTW a successful example like this one is the "cygport up" command, which
> we added a few years ago to automate uploading packages to cygwin.com. I
> think it's working well.

I'm very keen on reducing the maintainer workload by increasing the 
automation available to them.

However, I'm not so sure about the approach proposed, which perpetuates 
the 'create strange files which have a special meaning when uploaded 
causing something non-obvious to happen' behaviour.

I think I'd perhaps rather just extend the work done with 'untest' [2] 
to allow maintainers to do these things directly.

[2] 
https://cygwin.com/git/?p=cygwin-apps/calm.git;a=commit;h=0a29ad572cbe1bcc64fd3624f5c38eee79c50445

Re: [RFC] cygport pm for managing package releases

[Ken Brown]

On 9/20/2020 3:19 PM, Jon Turney wrote:
> I'd add to this list 'generating and sending the package announce email'.

cygport already does this ('cygport *.cygport announce').  Or did you have 
something else in mind?

Ken

Re: [RFC] cygport pm for managing package releases

[Andrew Schulman]

> I'm very keen on reducing the maintainer workload by increasing the 
> automation available to them.

Good.

> However, I'm not so sure about the approach proposed, which perpetuates 
> the 'create strange files which have a special meaning when uploaded 
> causing something non-obvious to happen' behaviour.
> 
> I think I'd perhaps rather just extend the work done with 'untest' [2] 
> to allow maintainers to do these things directly.
> 
> [2] 
> https://cygwin.com/git/?p=cygwin-apps/calm.git;a=commit;h=0a29ad572cbe1bcc64fd3624f5c38eee79c50445

I'm all in favor of using simpler methods. I just want to collect them all
in one place in cygport, so maintainers -

* don't have to remember, care, or find documentation for what the details
are for different package management functions;
* don't have to keep track as those methods change or improve;
* can rely on a stable and documented set of cygport commands to manage
their packages.

So as better automation becomes available, like the untest feature, we can
build that into cygport. But the maintainers won't necessarily have to
know; they can still just run cygport pm untest, for example.

Andrew