How to avoid tying up scallywag

How to avoid tying up scallywag

[Ken Brown]

Jon,

I'll be ready to go with TeX Live 2023 in a couple days.  That involves 
about 60 packages.  If I push them all at once, I'm afraid that would 
tie up scallywag and make it unusable by others.  I was thinking of 
pushing them in batches of 5, with a couple hours in between batches. 
But I don't know how many jobs scallywag can do at once.  What do you think?

Ken

Re: How to avoid tying up scallywag

[marco atzeri]

On Mon, Mar 20, 2023 at 12:04 AM Ken Brown via Cygwin-apps
<cygwin-apps@cygwin.com> wrote:
>
> Jon,
>
> I'll be ready to go with TeX Live 2023 in a couple days.  That involves
> about 60 packages.  If I push them all at once, I'm afraid that would
> tie up scallywag and make it unusable by others.  I was thinking of
> pushing them in batches of 5, with a couple hours in between batches.
> But I don't know how many jobs scallywag can do at once.  What do you think?
>
> Ken

It is very easy in reality. Jon will surely implement the "nice" token
to lower the priority of your jobs
;-)

No preference for me , as I am out of service.

Regards
Marco

Re: How to avoid tying up scallywag

[Jon Turney]

On 19/03/2023 23:04, Ken Brown via Cygwin-apps wrote:
> Jon,
> 
> I'll be ready to go with TeX Live 2023 in a couple days.  That involves 
> about 60 packages.  If I push them all at once, I'm afraid that would 
> tie up scallywag and make it unusable by others.  I was thinking of 
> pushing them in batches of 5, with a couple hours in between batches. 
> But I don't know how many jobs scallywag can do at once.  What do you 
> think?

As far as I can tell, the documented limits for the GitHub free service 
currently used are currently:

* 20 concurrent jobs
* runs which are queued for more than 45 minutes without starting are 
discarded.


The implementation of how the build back-end is used in scallywag is 
moderately modularized, so if these restrictions become irksome, and we 
ever have access to a better compute service, that could be used instead.


Note that if you are just updating the repository, without using 
scallywag to deploy, then pushing with --push-option=nobuild is more 
slightly more efficient that SCALLYWAG="nobuild" in the cygport, as it 
can short-cut things, since it doesn't need to start a job to evaluate 
the tokens to determine if nobuild is set.

Re: How to avoid tying up scallywag

[Ken Brown]

On 3/20/2023 7:22 AM, Jon Turney wrote:
> On 19/03/2023 23:04, Ken Brown via Cygwin-apps wrote:
>> Jon,
>>
>> I'll be ready to go with TeX Live 2023 in a couple days.  That 
>> involves about 60 packages.  If I push them all at once, I'm afraid 
>> that would tie up scallywag and make it unusable by others.  I was 
>> thinking of pushing them in batches of 5, with a couple hours in 
>> between batches. But I don't know how many jobs scallywag can do at 
>> once.  What do you think?
> 
> As far as I can tell, the documented limits for the GitHub free service 
> currently used are currently:
> 
> * 20 concurrent jobs
> * runs which are queued for more than 45 minutes without starting are 
> discarded.

So I should even be able to do 10 or 15 at once without clogging the 
system.  Maybe I'll start with one batch of 15 and see what happens.

> The implementation of how the build back-end is used in scallywag is 
> moderately modularized, so if these restrictions become irksome, and we 
> ever have access to a better compute service, that could be used instead.
> 
> 
> Note that if you are just updating the repository, without using 
> scallywag to deploy, then pushing with --push-option=nobuild is more 
> slightly more efficient that SCALLYWAG="nobuild" in the cygport, as it 
> can short-cut things, since it doesn't need to start a job to evaluate 
> the tokens to determine if nobuild is set.

Good to know, but in the present case I'm planning to deploy.

Ken