GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

[Adam Dinwoodie]

On Sat, 16 Jan 2021 at 20:22, Adam Dinwoodie wrote:
> Version 0.65-1 of moreutils has been uploaded and should be coming
> soon to a distribution server near you.

In case anyone's interested or has thoughts:

As part of working on this release, I've been playing with GitHub's
automation tools. The entire build / test / package / release / upload
process was performed using free ephemeral GitHub-managed VMs. At
least in theory, this reduces the manual work for future releases to:

- Commit a version of the Cygport file with an updated version number.
- Create a tag and push that tag to GitHub
- Wait for the confirmation email to arrive
- Send the announcement email

This is obviously serving a similar purpose to the automated builds
that Scallywag provides; I'm not sure I'd have bothered with this
project had I not already been most of the way through it before I
spotted Scallywag existed. I suspect in theory Scallywag's access to
the Cygwin servers means it's potentially more powerful, but Scallywag
also comes with some general caveats ("at this stage, this is only
probably useful for verifying that BUILD_REQUIRES is correct"),
whereas I think with care this could replace local builds today.

The configuration to make this work is almost entirely in
<https://github.com/me-and/Cygwin-moreutils/blob/v0.65-1/.github/workflows/ci.yml>,
and I expect the modifications to make this work for most other
packages would be straightforward. I'm hoping to make the process and
required configuration simpler, by creating a pre-defined GitHub
action that hides most of the boilerplate.

If you're curious, you can see the full build output for the final
release build at
<https://github.com/me-and/Cygwin-moreutils/actions/runs/490534324>.

Re: GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

[Ken Brown]

On 1/16/2021 3:33 PM, Adam Dinwoodie wrote:
> On Sat, 16 Jan 2021 at 20:22, Adam Dinwoodie wrote:
>> Version 0.65-1 of moreutils has been uploaded and should be coming
>> soon to a distribution server near you.
> 
> In case anyone's interested or has thoughts:
> 
> As part of working on this release, I've been playing with GitHub's
> automation tools. The entire build / test / package / release / upload
> process was performed using free ephemeral GitHub-managed VMs. At
> least in theory, this reduces the manual work for future releases to:
> 
> - Commit a version of the Cygport file with an updated version number.
> - Create a tag and push that tag to GitHub
> - Wait for the confirmation email to arrive
> - Send the announcement email
> 
> This is obviously serving a similar purpose to the automated builds
> that Scallywag provides; I'm not sure I'd have bothered with this
> project had I not already been most of the way through it before I
> spotted Scallywag existed. I suspect in theory Scallywag's access to
> the Cygwin servers means it's potentially more powerful, but Scallywag
> also comes with some general caveats ("at this stage, this is only
> probably useful for verifying that BUILD_REQUIRES is correct"),

I assume you're quoting from https://cygwin.com/packaging/build.html.  Scallywag 
does have some limitations currently, but I think the statement you quoted is 
obsolete.  I often have Scallywag deploy my packages, as does Jon Turney.

The limitations I've bumped into are:

1. Scallywag will time out after an hour on each arch.

2. Several of my packages fail to build on x86 because of gcc crashes.

I think these limitations are outweighed by the fact that a Scallywag build is 
automatically triggered by a push to an official source repo 
(https://cygwin.com/packaging/repos.html).  All maintainers can use this without 
any special setup.

Ken

Re: GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

[Marco Atzeri]

On 16.01.2021 23:31, Ken Brown via Cygwin-apps wrote:
> On 1/16/2021 3:33 PM, Adam Dinwoodie wrote:
>> On Sat, 16 Jan 2021 at 20:22, Adam Dinwoodie wrote:
>>> Version 0.65-1 of moreutils has been uploaded and should be coming
>>> soon to a distribution server near you.

> 
> I assume you're quoting from https://cygwin.com/packaging/build.html.  
> Scallywag does have some limitations currently, but I think the 
> statement you quoted is obsolete.  I often have Scallywag deploy my 
> packages, as does Jon Turney.
> 
> The limitations I've bumped into are:
> 
> 1. Scallywag will time out after an hour on each arch

until now I was hit only during test. But I stil need to
deploy on our Git some of the heavy packages

> 
> 2. Several of my packages fail to build on x86 because of gcc crashes.

3. sometimes the failure is not in the build, but in missing or 
imperfect testing

> 
> I think these limitations are outweighed by the fact that a Scallywag 
> build is automatically triggered by a push to an official source repo 
> (https://cygwin.com/packaging/repos.html).  All maintainers can use this 
> without any special setup.

+1

> Ken

but of course Adam, more options are always better,

Thank for sharing
Marco

Re: GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

[Adam Dinwoodie]

On Sat, 16 Jan 2021 at 22:31, Ken Brown wrote:
> On 1/16/2021 3:33 PM, Adam Dinwoodie wrote:
> > On Sat, 16 Jan 2021 at 20:22, Adam Dinwoodie wrote:
> >> Version 0.65-1 of moreutils has been uploaded and should be coming
> >> soon to a distribution server near you.
> >
> > In case anyone's interested or has thoughts:
> >
> > As part of working on this release, I've been playing with GitHub's
> > automation tools. The entire build / test / package / release / upload
> > process was performed using free ephemeral GitHub-managed VMs. At
> > least in theory, this reduces the manual work for future releases to:
> >
> > - Commit a version of the Cygport file with an updated version number.
> > - Create a tag and push that tag to GitHub
> > - Wait for the confirmation email to arrive
> > - Send the announcement email
> >
> > This is obviously serving a similar purpose to the automated builds
> > that Scallywag provides; I'm not sure I'd have bothered with this
> > project had I not already been most of the way through it before I
> > spotted Scallywag existed. I suspect in theory Scallywag's access to
> > the Cygwin servers means it's potentially more powerful, but Scallywag
> > also comes with some general caveats ("at this stage, this is only
> > probably useful for verifying that BUILD_REQUIRES is correct"),
>
> I assume you're quoting from https://cygwin.com/packaging/build.html.  Scallywag
> does have some limitations currently, but I think the statement you quoted is
> obsolete.  I often have Scallywag deploy my packages, as does Jon Turney.

Yes, that was my source here. I experimented with Scallywag briefly,
but was put off by (a) that warning and (b) the fact that my first
builds failed because I apparently use the wrong quoting style in my
Cygport files. And, as I say, that I was already a significant way to
a working GitHub Action process.

> The limitations I've bumped into are:
>
> 1. Scallywag will time out after an hour on each arch.

This is a killer for me. Getting this working with moreutils was a
simple proof-of-concept; the key package where this will likely save
me time and energy is Git, and the Git test suite takes multiple hours
to run on Cygwin. GitHub actions have a per-job limit of 6 hours, and
a per-workflow limit of 72 hours.

> 2. Several of my packages fail to build on x86 because of gcc crashes.
>
> I think these limitations are outweighed by the fact that a Scallywag build is
> automatically triggered by a push to an official source repo
> (https://cygwin.com/packaging/repos.html).  All maintainers can use this without
> any special setup.

That's clearly incredibly valuable, yes. I'm hoping to reduce the
special setup using GitHub Actions requires, but it's clearly going to
require more than zero setup.

Adam

Re: GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

[Jon Turney]

On 17/01/2021 15:33, Adam Dinwoodie wrote:
> On Sat, 16 Jan 2021 at 22:31, Ken Brown wrote:
>> On 1/16/2021 3:33 PM, Adam Dinwoodie wrote:
>>> On Sat, 16 Jan 2021 at 20:22, Adam Dinwoodie wrote:
>>>> Version 0.65-1 of moreutils has been uploaded and should be coming
>>>> soon to a distribution server near you.
>>>
>>> In case anyone's interested or has thoughts:
>>>
>>> As part of working on this release, I've been playing with GitHub's
>>> automation tools. The entire build / test / package / release / upload
>>> process was performed using free ephemeral GitHub-managed VMs. At
>>> least in theory, this reduces the manual work for future releases to:
>>>
>>> - Commit a version of the Cygport file with an updated version number.
>>> - Create a tag and push that tag to GitHub
>>> - Wait for the confirmation email to arrive
>>> - Send the announcement email

Thanks for sharing this.

One architectural problem that requires solving in cygport that we both 
have is that we aren't building from the source package, but the package 
repository (so this doesn't verify that all the files used by the build 
are put into the source package).  Ideally we'd be able to instruct 
cygport to make just the source package, then unpack and build from that.

>>> This is obviously serving a similar purpose to the automated builds
>>> that Scallywag provides; I'm not sure I'd have bothered with this
>>> project had I not already been most of the way through it before I
>>> spotted Scallywag existed. I suspect in theory Scallywag's access to
>>> the Cygwin servers means it's potentially more powerful, but Scallywag
>>> also comes with some general caveats ("at this stage, this is only
>>> probably useful for verifying that BUILD_REQUIRES is correct"),
>>
>> I assume you're quoting from https://cygwin.com/packaging/build.html.  Scallywag
>> does have some limitations currently, but I think the statement you quoted is
>> obsolete.  I often have Scallywag deploy my packages, as does Jon Turney.

The caveat was mainly driven by the fact that 'it's only been tested by 
me' :)

> Yes, that was my source here. I experimented with Scallywag briefly,
> but was put off by (a) that warning and (b) the fact that my first
> builds failed because I apparently use the wrong quoting style in my
> Cygport files. And, as I say, that I was already a significant way to
> a working GitHub Action process.
> 
>> The limitations I've bumped into are:
>>
>> 1. Scallywag will time out after an hour on each arch.
> 
> This is a killer for me. Getting this working with moreutils was a
> simple proof-of-concept; the key package where this will likely save
> me time and energy is Git, and the Git test suite takes multiple hours
> to run on Cygwin. GitHub actions have a per-job limit of 6 hours, and
> a per-workflow limit of 72 hours.

Yeah.

>> 2. Several of my packages fail to build on x86 because of gcc crashes.
>>
>> I think these limitations are outweighed by the fact that a Scallywag build is
>> automatically triggered by a push to an official source repo
>> (https://cygwin.com/packaging/repos.html).  All maintainers can use this without
>> any special setup.
> 
> That's clearly incredibly valuable, yes. I'm hoping to reduce the
> special setup using GitHub Actions requires, but it's clearly going to
> require more than zero setup.

I'd rather not be replicating the tooling to do this into every package 
repository.

All the AppVeyor builds that scallywag does are of the scallywag 
repository itself, since the AppVeyor API lets me parameterize the build 
(by the package repository and commit-id)

In a brief investigation, I didn't find anything similar for github actions.

Re: GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

[Jon Turney]

On 09/05/2021 15:40, Jon Turney wrote:
> On 17/01/2021 15:33, Adam Dinwoodie wrote:
>> On Sat, 16 Jan 2021 at 22:31, Ken Brown wrote:
[...]
>>> I think these limitations are outweighed by the fact that a Scallywag 
>>> build is
>>> automatically triggered by a push to an official source repo
>>> (https://cygwin.com/packaging/repos.html).  All maintainers can use 
>>> this without
>>> any special setup.
>>
>> That's clearly incredibly valuable, yes. I'm hoping to reduce the
>> special setup using GitHub Actions requires, but it's clearly going to
>> require more than zero setup.
> 
> I'd rather not be replicating the tooling to do this into every package 
> repository.
> 
> All the AppVeyor builds that scallywag does are of the scallywag 
> repository itself, since the AppVeyor API lets me parameterize the build 
> (by the package repository and commit-id)
> 
> In a brief investigation, I didn't find anything similar for github 
> actions.

It seems that this has capability been added since the time I checked 
(the 'repository_dispatch' workflow trigger event).

I've deployed a change which switches the build back-end from AppVeyor 
to GitHub:  This should be faster (as it's not limited to a single job 
at a time) and more useful (since the job timeout is 6 hours, not 1 hour).


I'd like feedback on what other improvements are needed for this to 
become the preferable way to build packages.

Re: GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

[ASSI]

Jon Turney writes:
> I'd like feedback on what other improvements are needed for this to
> become the preferable way to build packages.

As long as you have to have a GitHub account to look at the results I
won't be able to use it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

[Brian Inglis]

On 2021-06-06 11:37, ASSI wrote:
> Jon Turney writes:
>> I'd like feedback on what other improvements are needed for this to
>> become the preferable way to build packages.
> 
> As long as you have to have a GitHub account to look at the results I
> won't be able to use it.

Check it out and see if you do:

	https://github.com/cygwin/scallywag/actions

I already have one and it works fine for me: only comment is that the 
log seems less readable than on Appveyor, so harder to spot the issues, 
but that's only an initial impression; will have to check and compare 
with Appveyor logs.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: GitHub automation for Cygwin builds [Was: Updated: moreutils v0.65-1]

[Jon Turney]

On 06/06/2021 19:32, Brian Inglis wrote:
> On 2021-06-06 11:37, ASSI wrote:
>> Jon Turney writes:
>>> I'd like feedback on what other improvements are needed for this to
>>> become the preferable way to build packages.
>>
>> As long as you have to have a GitHub account to look at the results I
>> won't be able to use it.
> 
> Check it out and see if you do:
> 
>      https://github.com/cygwin/scallywag/actions

Yeah, this complaint is slightly confusingly worded, but if you log out 
of github, you get a button which says "Sign in to view logs".

Which is annoying.

> I already have one and it works fine for me: only comment is that the 
> log seems less readable than on Appveyor, so harder to spot the issues, 
> but that's only an initial impression; will have to check and compare 
> with Appveyor logs.