Re: gcc git locked out for hours second day in a row

[Mikael Morin <morin-mikael@orange.fr>]

Hello,

Le 12/06/2024 à 16:57, Jakub Jelinek a écrit :
> On Wed, Jun 12, 2024 at 04:53:38PM +0200, Mikael Morin wrote:
>>> Perhaps you could create a mirror version of the repo and do some experiments locally on that to identify where the bottle-neck is coming from?
>>>
>> Not sure where to start for that.Are the hooks published somewhere?
> 
> Yes: https://github.com/AdaCore/git-hooks/tree/master
> 
> Note, we use some tweaks on top of that, but that is mostly for the
> release branches and trunk, so it would be interesting to just try
> to reproduce that with the stock AdaCore git hooks.
> 
I have finally taken some time to investigate this hook slowness, and 
here are my findings.

My tests were run with configs commit-extra-checker and 
commit-email-formatter disabled, and hooks.update-hook set to a minimal 
script (either "true" or "sleep 1").  With that config, I could not 
reproduce the slowness pushing to refs/users/mikael/*.  The push 
finishes in less than a minute.

However, trying to push to a normal tag, there is some email count check 
coming into play, and I can reproduce some slowness (details below). 
This email count check shouldn't happen on the gcc repository in my use 
case (as email checks don't apply to user references), but the slowness 
could well happen in other cases than email count check depending on the 
configuration, as the problem relates to the size of the list of new 
commits and is not restricted to email count.

Anyway, even with email count check triggering, each tag takes less than 
2 minutes to be rejected in my test.  With 330 tags to process, that 
would make an upper bound of 11 hours before rejecting the push in my 
test (I killed it after a few minutes).  On the other hand, with the 
information you gave upthread, the hook on the gcc repository seemed to 
be still processing the first tag after a few hours (assuming they are 
processed in alphabetical order, which seems to be the case).  So this 
still doesn't explain what was happening on the gcc repository.

Regarding the email count check slowness I mentioned above, I traced it 
back to the updates.AbstractUpdate class, whose (procedural) 
new_commits_for_ref attribute is a list of "new" commits, containing 
both really new commits and commits newly on the branch to be updated, 
but already known to the repository.  For a tag or branch creation, a 
list of "new on the branch" commits would be huge as everything is new, 
so parent commits of the oldest "repository-new" commit are not picked 
up.  But in my test the list still amounts to a little less than 80,000 
commits, basically what happened on trunk in the last 8 years.  Anything 
that walks such a big list is bound to be slow.

To sum up:
- The hooks support checking "new on the branch" commits additionally to 
"new on the repository" commits, and that is a feature, not a bug.
- In my use case, that means that the hooks process 80,000 commits, even 
if only 330 of them are new on the repository.
- As the hook is called on a per-reference basis, the same commits would 
be processed over and over again for every reference in my use case, so 
the best would be to push them one by one, in order.
- I still don't know why it took hours (without even finishing) to 
process just one tag the other day on the gcc repository.

Nikael