gcc git locked out for hours second day in a row

gcc git locked out for hours second day in a row

[Jakub Jelinek]

Hi!

Yesterday the gcc git repository was locked for 3 hours
locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167)
78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 0000000000000000000000000000000000000000 c2f9fe1d8111b9671bf0aa8362446516fd942f1d
process until overseers killed it but today we have the same
situation for 3 ours and counting again:
locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652)
78:06 python hooks/update.py refs/users/mikael/tags/toto 0000000000000000000000000000000000000000 cca005166dba2cefeb51afac3ea629b3972acea3

It is possible we have some bug in the git hook scripts, but it would
be helpful trying to understand what exactly you're trying to commit
and why nobody else (at least to my knowledge) has similarly stuck commits.

The effect is that nobody can push anything else to gcc git repo
for hours.

	Jakub

Re: gcc git locked out for hours second day in a row

[Mark Wielaard]

Hi,

On Wed, 2024-06-12 at 13:48 +0200, Jakub Jelinek via Gcc wrote:
> Yesterday the gcc git repository was locked for 3 hours
> locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167)
> 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 0000000000000000000000000000000000000000 c2f9fe1d8111b9671bf0aa8362446516fd942f1d
> process until overseers killed it but today we have the same
> situation for 3 ours and counting again:
> locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652)
> 78:06 python hooks/update.py refs/users/mikael/tags/toto 0000000000000000000000000000000000000000 cca005166dba2cefeb51afac3ea629b3972acea3

Just for the record, I kill the process and removed the git-
hooks::update.token.lock file. Sorry that killed mikael's push, but
hopefully that makes it possible for others to push again.

Cheers,

Mark

Re: gcc git locked out for hours second day in a row

[Mikael Morin]

Le 12/06/2024 à 13:48, Jakub Jelinek a écrit :
> Hi!
> 
> Yesterday the gcc git repository was locked for 3 hours
> locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167)
> 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 0000000000000000000000000000000000000000 c2f9fe1d8111b9671bf0aa8362446516fd942f1d
> process until overseers killed it but today we have the same
> situation for 3 ours and counting again:
> locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652)
> 78:06 python hooks/update.py refs/users/mikael/tags/toto 0000000000000000000000000000000000000000 cca005166dba2cefeb51afac3ea629b3972acea3
> 
> It is possible we have some bug in the git hook scripts, but it would
> be helpful trying to understand what exactly you're trying to commit
> and why nobody else (at least to my knowledge) has similarly stuck commits.
> 
> The effect is that nobody can push anything else to gcc git repo
> for hours.
> 
> 	Jakub
> 
Yes, sorry for the inconvenience.
I tried pushing a series of tags labeling merge points between the 
fortran-dev branch and recent years master.
The number of merge points is a bit high (329) but I expected it to be a 
manageable number.  I tried again today with just the most recent merge 
point, but it got stuck again.  I should try with the oldest one, but 
I'm afraid locking the repository again.

I waited for the push to finish for say one hour before killing it 
yesterday, and no more than 15 minutes today.  Unfortunately, killing 
the process doesn't seem to unlock things on the server side.

It may be a misconfiguration on my side, but I have never had this 
problem before.

Sorry again.

Re: gcc git locked out for hours second day in a row

[Mikael Morin]

Le 12/06/2024 à 14:14, Mark Wielaard a écrit :
> Hi,
> 
> On Wed, 2024-06-12 at 13:48 +0200, Jakub Jelinek via Gcc wrote:
>> Yesterday the gcc git repository was locked for 3 hours
>> locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167)
>> 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 0000000000000000000000000000000000000000 c2f9fe1d8111b9671bf0aa8362446516fd942f1d
>> process until overseers killed it but today we have the same
>> situation for 3 ours and counting again:
>> locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652)
>> 78:06 python hooks/update.py refs/users/mikael/tags/toto 0000000000000000000000000000000000000000 cca005166dba2cefeb51afac3ea629b3972acea3
> 
> Just for the record, I kill the process and removed the git-
> hooks::update.token.lock file. Sorry that killed mikael's push, but
> hopefully that makes it possible for others to push again.
> 
No problem, I had interrupted the push long before on my side.

Re: gcc git locked out for hours second day in a row

[Jonathan Wakely]

On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc <gcc@gcc.gnu.org> wrote:
>
> Le 12/06/2024 à 13:48, Jakub Jelinek a écrit :
> > Hi!
> >
> > Yesterday the gcc git repository was locked for 3 hours
> > locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167)
> > 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 0000000000000000000000000000000000000000 c2f9fe1d8111b9671bf0aa8362446516fd942f1d
> > process until overseers killed it but today we have the same
> > situation for 3 ours and counting again:
> > locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652)
> > 78:06 python hooks/update.py refs/users/mikael/tags/toto 0000000000000000000000000000000000000000 cca005166dba2cefeb51afac3ea629b3972acea3
> >
> > It is possible we have some bug in the git hook scripts, but it would
> > be helpful trying to understand what exactly you're trying to commit
> > and why nobody else (at least to my knowledge) has similarly stuck commits.
> >
> > The effect is that nobody can push anything else to gcc git repo
> > for hours.
> >
> >       Jakub
> >
> Yes, sorry for the inconvenience.
> I tried pushing a series of tags labeling merge points between the
> fortran-dev branch and recent years master.

Just pushing tags should not cause a problem, assuming all the commits
being tagged already exist. What exactly are you pushing?


> The number of merge points is a bit high (329) but I expected it to be a
> manageable number.  I tried again today with just the most recent merge
> point, but it got stuck again.  I should try with the oldest one, but
> I'm afraid locking the repository again.
>
> I waited for the push to finish for say one hour before killing it
> yesterday, and no more than 15 minutes today.  Unfortunately, killing
> the process doesn't seem to unlock things on the server side.
>
> It may be a misconfiguration on my side, but I have never had this
> problem before.
>
> Sorry again.
>
>

Re: gcc git locked out for hours second day in a row

[Mikael Morin]

Le 12/06/2024 à 14:58, Jonathan Wakely a écrit :
> On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc <gcc@gcc.gnu.org> wrote:
>>
>> Le 12/06/2024 à 13:48, Jakub Jelinek a écrit :
>>> Hi!
>>>
>>> Yesterday the gcc git repository was locked for 3 hours
>>> locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167)
>>> 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 0000000000000000000000000000000000000000 c2f9fe1d8111b9671bf0aa8362446516fd942f1d
>>> process until overseers killed it but today we have the same
>>> situation for 3 ours and counting again:
>>> locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652)
>>> 78:06 python hooks/update.py refs/users/mikael/tags/toto 0000000000000000000000000000000000000000 cca005166dba2cefeb51afac3ea629b3972acea3
>>>
>>> It is possible we have some bug in the git hook scripts, but it would
>>> be helpful trying to understand what exactly you're trying to commit
>>> and why nobody else (at least to my knowledge) has similarly stuck commits.
>>>
>>> The effect is that nobody can push anything else to gcc git repo
>>> for hours.
>>>
>>>        Jakub
>>>
>> Yes, sorry for the inconvenience.
>> I tried pushing a series of tags labeling merge points between the
>> fortran-dev branch and recent years master.
> 
> Just pushing tags should not cause a problem, assuming all the commits
> being tagged already exist. What exactly are you pushing?
> 
Well, the individual commits to be merged do exist, but the merge points 
don't and they are what I'm trying to push.

To be clear, the branch hasn't seen any update for years, and I'm trying 
to reapply what happened on trunk since, in a step-wise manner.  With 
300 merges I'm summing up 60000 commits of history.

> 
>> The number of merge points is a bit high (329) but I expected it to be a
>> manageable number.  I tried again today with just the most recent merge
>> point, but it got stuck again.  I should try with the oldest one, but
>> I'm afraid locking the repository again.
>>
>> I waited for the push to finish for say one hour before killing it
>> yesterday, and no more than 15 minutes today.  Unfortunately, killing
>> the process doesn't seem to unlock things on the server side.
>>
>> It may be a misconfiguration on my side, but I have never had this
>> problem before.
>>
>> Sorry again.
>>
>>

Re: gcc git locked out for hours second day in a row

[Richard Earnshaw (lists)]

On 12/06/2024 14:23, Mikael Morin via Gcc wrote:
> Le 12/06/2024 à 14:58, Jonathan Wakely a écrit :
>> On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc <gcc@gcc.gnu.org> wrote:
>>>
>>> Le 12/06/2024 à 13:48, Jakub Jelinek a écrit :
>>>> Hi!
>>>>
>>>> Yesterday the gcc git repository was locked for 3 hours
>>>> locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167)
>>>> 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 0000000000000000000000000000000000000000 c2f9fe1d8111b9671bf0aa8362446516fd942f1d
>>>> process until overseers killed it but today we have the same
>>>> situation for 3 ours and counting again:
>>>> locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652)
>>>> 78:06 python hooks/update.py refs/users/mikael/tags/toto 0000000000000000000000000000000000000000 cca005166dba2cefeb51afac3ea629b3972acea3
>>>>
>>>> It is possible we have some bug in the git hook scripts, but it would
>>>> be helpful trying to understand what exactly you're trying to commit
>>>> and why nobody else (at least to my knowledge) has similarly stuck commits.
>>>>
>>>> The effect is that nobody can push anything else to gcc git repo
>>>> for hours.
>>>>
>>>>        Jakub
>>>>
>>> Yes, sorry for the inconvenience.
>>> I tried pushing a series of tags labeling merge points between the
>>> fortran-dev branch and recent years master.
>>
>> Just pushing tags should not cause a problem, assuming all the commits
>> being tagged already exist. What exactly are you pushing?
>>
> Well, the individual commits to be merged do exist, but the merge points don't and they are what I'm trying to push.
> 
> To be clear, the branch hasn't seen any update for years, and I'm trying to reapply what happened on trunk since, in a step-wise manner.  With 300 merges I'm summing up 60000 commits of history.
> 
>>
>>> The number of merge points is a bit high (329) but I expected it to be a
>>> manageable number.  I tried again today with just the most recent merge
>>> point, but it got stuck again.  I should try with the oldest one, but
>>> I'm afraid locking the repository again.
>>>
>>> I waited for the push to finish for say one hour before killing it
>>> yesterday, and no more than 15 minutes today.  Unfortunately, killing
>>> the process doesn't seem to unlock things on the server side.
>>>
>>> It may be a misconfiguration on my side, but I have never had this
>>> problem before.
>>>
>>> Sorry again.
>>>
>>>
> 

Perhaps you could create a mirror version of the repo and do some experiments locally on that to identify where the bottle-neck is coming from?

R.

Re: gcc git locked out for hours second day in a row

[Mikael Morin]

Le 12/06/2024 à 16:34, Richard Earnshaw (lists) a écrit :
> On 12/06/2024 14:23, Mikael Morin via Gcc wrote:
>> Le 12/06/2024 à 14:58, Jonathan Wakely a écrit :
>>> On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc <gcc@gcc.gnu.org> wrote:
>>>>
>>>> Le 12/06/2024 à 13:48, Jakub Jelinek a écrit :
>>>>> Hi!
>>>>>
>>>>> Yesterday the gcc git repository was locked for 3 hours
>>>>> locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167)
>>>>> 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 0000000000000000000000000000000000000000 c2f9fe1d8111b9671bf0aa8362446516fd942f1d
>>>>> process until overseers killed it but today we have the same
>>>>> situation for 3 ours and counting again:
>>>>> locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652)
>>>>> 78:06 python hooks/update.py refs/users/mikael/tags/toto 0000000000000000000000000000000000000000 cca005166dba2cefeb51afac3ea629b3972acea3
>>>>>
>>>>> It is possible we have some bug in the git hook scripts, but it would
>>>>> be helpful trying to understand what exactly you're trying to commit
>>>>> and why nobody else (at least to my knowledge) has similarly stuck commits.
>>>>>
>>>>> The effect is that nobody can push anything else to gcc git repo
>>>>> for hours.
>>>>>
>>>>>         Jakub
>>>>>
>>>> Yes, sorry for the inconvenience.
>>>> I tried pushing a series of tags labeling merge points between the
>>>> fortran-dev branch and recent years master.
>>>
>>> Just pushing tags should not cause a problem, assuming all the commits
>>> being tagged already exist. What exactly are you pushing?
>>>
>> Well, the individual commits to be merged do exist, but the merge points don't and they are what I'm trying to push.
>>
>> To be clear, the branch hasn't seen any update for years, and I'm trying to reapply what happened on trunk since, in a step-wise manner.  With 300 merges I'm summing up 60000 commits of history.
>>
>>>
>>>> The number of merge points is a bit high (329) but I expected it to be a
>>>> manageable number.  I tried again today with just the most recent merge
>>>> point, but it got stuck again.  I should try with the oldest one, but
>>>> I'm afraid locking the repository again.
>>>>
>>>> I waited for the push to finish for say one hour before killing it
>>>> yesterday, and no more than 15 minutes today.  Unfortunately, killing
>>>> the process doesn't seem to unlock things on the server side.
>>>>
>>>> It may be a misconfiguration on my side, but I have never had this
>>>> problem before.
>>>>
>>>> Sorry again.
>>>>
>>>>
>>
> 
> Perhaps you could create a mirror version of the repo and do some experiments locally on that to identify where the bottle-neck is coming from?
> 
Not sure where to start for that.Are the hooks published somewhere?

Re: gcc git locked out for hours second day in a row

[Jakub Jelinek]

On Wed, Jun 12, 2024 at 04:53:38PM +0200, Mikael Morin wrote:
> > Perhaps you could create a mirror version of the repo and do some experiments locally on that to identify where the bottle-neck is coming from?
> > 
> Not sure where to start for that.Are the hooks published somewhere?

Yes: https://github.com/AdaCore/git-hooks/tree/master

Note, we use some tweaks on top of that, but that is mostly for the
release branches and trunk, so it would be interesting to just try
to reproduce that with the stock AdaCore git hooks.

	Jakub

Re: gcc git locked out for hours second day in a row

[Jonathan Wakely]

On Wed, 12 Jun 2024 at 14:23, Mikael Morin <morin-mikael@orange.fr> wrote:
>
> Le 12/06/2024 à 14:58, Jonathan Wakely a écrit :
> > On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc <gcc@gcc.gnu.org> wrote:
> >>
> >> Le 12/06/2024 à 13:48, Jakub Jelinek a écrit :
> >>> Hi!
> >>>
> >>> Yesterday the gcc git repository was locked for 3 hours
> >>> locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167)
> >>> 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 0000000000000000000000000000000000000000 c2f9fe1d8111b9671bf0aa8362446516fd942f1d
> >>> process until overseers killed it but today we have the same
> >>> situation for 3 ours and counting again:
> >>> locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652)
> >>> 78:06 python hooks/update.py refs/users/mikael/tags/toto 0000000000000000000000000000000000000000 cca005166dba2cefeb51afac3ea629b3972acea3
> >>>
> >>> It is possible we have some bug in the git hook scripts, but it would
> >>> be helpful trying to understand what exactly you're trying to commit
> >>> and why nobody else (at least to my knowledge) has similarly stuck commits.
> >>>
> >>> The effect is that nobody can push anything else to gcc git repo
> >>> for hours.
> >>>
> >>>        Jakub
> >>>
> >> Yes, sorry for the inconvenience.
> >> I tried pushing a series of tags labeling merge points between the
> >> fortran-dev branch and recent years master.
> >
> > Just pushing tags should not cause a problem, assuming all the commits
> > being tagged already exist. What exactly are you pushing?
> >
> Well, the individual commits to be merged do exist, but the merge points
> don't and they are what I'm trying to push.

I see. Merge commits are still commits, so they get processed by the hooks.

We don't even allow merge commits on trunk or the release branches, so
I don't know how the hooks handle them.

> To be clear, the branch hasn't seen any update for years, and I'm trying
> to reapply what happened on trunk since, in a step-wise manner.  With
> 300 merges I'm summing up 60000 commits of history.

Yes, that's going to give the hooks a ton of work to do. So it's
probably just  the number of commits being pushed to the branch, not
the fact they're merge commits (and certainly not the tags referring
to any of those commits, which should be very cheap).

If it was my branch, I'd be tempted to rebase the fortran_dev branch
on trunk, instead of merging years and years of trunk commits into the
ancient branch. Or create a new branch, say "fortran_dev_2", from
trunk, then merge the fortran_dev branch into that, and push the new
branch. That would presumably be a much smaller set of "new" commits
relative to trunk, so much less work for the hooks.

Re: gcc git locked out for hours second day in a row

[Mikael Morin]

Hello,

Le 12/06/2024 à 16:57, Jakub Jelinek a écrit :
> On Wed, Jun 12, 2024 at 04:53:38PM +0200, Mikael Morin wrote:
>>> Perhaps you could create a mirror version of the repo and do some experiments locally on that to identify where the bottle-neck is coming from?
>>>
>> Not sure where to start for that.Are the hooks published somewhere?
> 
> Yes: https://github.com/AdaCore/git-hooks/tree/master
> 
> Note, we use some tweaks on top of that, but that is mostly for the
> release branches and trunk, so it would be interesting to just try
> to reproduce that with the stock AdaCore git hooks.
> 
I have finally taken some time to investigate this hook slowness, and 
here are my findings.

My tests were run with configs commit-extra-checker and 
commit-email-formatter disabled, and hooks.update-hook set to a minimal 
script (either "true" or "sleep 1").  With that config, I could not 
reproduce the slowness pushing to refs/users/mikael/*.  The push 
finishes in less than a minute.

However, trying to push to a normal tag, there is some email count check 
coming into play, and I can reproduce some slowness (details below). 
This email count check shouldn't happen on the gcc repository in my use 
case (as email checks don't apply to user references), but the slowness 
could well happen in other cases than email count check depending on the 
configuration, as the problem relates to the size of the list of new 
commits and is not restricted to email count.

Anyway, even with email count check triggering, each tag takes less than 
2 minutes to be rejected in my test.  With 330 tags to process, that 
would make an upper bound of 11 hours before rejecting the push in my 
test (I killed it after a few minutes).  On the other hand, with the 
information you gave upthread, the hook on the gcc repository seemed to 
be still processing the first tag after a few hours (assuming they are 
processed in alphabetical order, which seems to be the case).  So this 
still doesn't explain what was happening on the gcc repository.

Regarding the email count check slowness I mentioned above, I traced it 
back to the updates.AbstractUpdate class, whose (procedural) 
new_commits_for_ref attribute is a list of "new" commits, containing 
both really new commits and commits newly on the branch to be updated, 
but already known to the repository.  For a tag or branch creation, a 
list of "new on the branch" commits would be huge as everything is new, 
so parent commits of the oldest "repository-new" commit are not picked 
up.  But in my test the list still amounts to a little less than 80,000 
commits, basically what happened on trunk in the last 8 years.  Anything 
that walks such a big list is bound to be slow.

To sum up:
- The hooks support checking "new on the branch" commits additionally to 
"new on the repository" commits, and that is a feature, not a bug.
- In my use case, that means that the hooks process 80,000 commits, even 
if only 330 of them are new on the repository.
- As the hook is called on a per-reference basis, the same commits would 
be processed over and over again for every reference in my use case, so 
the best would be to push them one by one, in order.
- I still don't know why it took hours (without even finishing) to 
process just one tag the other day on the gcc repository.

Nikael