[Patch] contrib/gcc-changelog/git_update_version.py: Add ignore commit, improve diagnostic

[Patch] contrib/gcc-changelog/git_update_version.py: Add ignore commit, improve diagnostic

[Tobias Burnus]

[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]

I noticed that the last bump happened on Thursday.

* * *

The error is according to
https://gcc.gnu.org/pipermail/gccadmin/2024q2/021298.html

2024-05-19 00:17:28,643:INFO:root:cannot find a ChangeLog location in message

That's the commit
---------------
     Revert "Revert: "Enable prange support.""

     This reverts commit d7bb8eaade3cd3aa70715c8567b4d7b08098e699 and enables prange
     support again.
---------------

* * * The attached patch adds this commit to the ignore list and helps 
with the diagnosis by showing the failing hash in the error message. OK 
for mainline? Post commit: Can someone install the new version + fix the 
ChangeLog for the ignored commit? * * * What I do not understand: Why does this commit get applied? I do see for both
contrib/gcc-changelog/git_check_commit.py -v -p 
da73261ce7731be7f2b164f1db796878cdc23365 and 
contrib/gcc-changelog/git_email.py 
0001-Revert-Revert-Enable-prange-support.patch the error above. - And I 
do not understand why it made it past the commit check but now fails?
Likewise for8057f9aa1f7e70490064de796d7a8d42d446caf8
Does the commit hook use an older version of the check scripts? Does it 
ignore the errors? Or what goes wrong here? Any idea? Tobias

[-- Attachment #2: 0001-contrib-gcc-changelog-git_update_version.py-Add-igno.patch --]
[-- Type: text/x-patch, Size: 1502 bytes --]

From f56b1764f2b5c2c83c6852607405e5be0a763a2c Mon Sep 17 00:00:00 2001
From: Tobias Burnus <tburnus@baylibre.com>
Date: Sun, 19 May 2024 08:17:42 +0200
Subject: [PATCH] contrib/gcc-changelog/git_update_version.py: Add ignore
 commit, improve diagnostic

contrib/ChangeLog:

        * gcc-changelog/git_update_version.py (IGNORED_COMMITS): Add
	cfceb070e2aea3cef9bd1f50d8d030c51449f45b.
	(prepend_to_changelog_files): Output git hash in case of error.

diff --git a/contrib/gcc-changelog/git_update_version.py b/contrib/gcc-changelog/git_update_version.py
index 24f6c43d0b2..ec0151b83fe 100755
--- a/contrib/gcc-changelog/git_update_version.py
+++ b/contrib/gcc-changelog/git_update_version.py
@@ -41,7 +41,8 @@ IGNORED_COMMITS = (
         '040e5b0edbca861196d9e2ea2af5e805769c8d5d',
         '8057f9aa1f7e70490064de796d7a8d42d446caf8',
         '109f1b28fc94c93096506e3df0c25e331cef19d0',
-        '39f81924d88e3cc197fc3df74204c9b5e01e12f7')
+        '39f81924d88e3cc197fc3df74204c9b5e01e12f7',
+        'da73261ce7731be7f2b164f1db796878cdc23365')
 
 FORMAT = '%(asctime)s:%(levelname)s:%(name)s:%(message)s'
 logging.basicConfig(level=logging.INFO, format=FORMAT,
@@ -58,6 +59,7 @@ def read_timestamp(path):
 
 def prepend_to_changelog_files(repo, folder, git_commit, add_to_git):
     if not git_commit.success:
+        logging.info(f"While processing {git_commit.info.hexsha}:")
         for error in git_commit.errors:
             logging.info(error)
         raise AssertionError()
-- 
2.45.0

[Patch] contrib/gcc-changelog/git_update_version.py: Improve diagnostic (was: [Patch] contrib/gcc-changelog/git_update_version.py: Add ignore commit, improve diagnostic)

[Tobias Burnus]

[-- Attachment #1.1: Type: text/plain, Size: 1630 bytes --]

Hmm, there were now two daily bumps:

Date:   Mon May 20 00:16:30 2024 +0000

Date:   Sun May 19 18:15:28 2024 +0000

I really wonder why.

I guess, the 'ignore commit' is hence not needed – but I think the 
improved diagnostic part still makes sense.

See updated patch.

On May 19, 24 Tobias Burnus wrote:
> I noticed that the last bump happened on Thursday.
>
> * * *
>
> The error is according to
> https://gcc.gnu.org/pipermail/gccadmin/2024q2/021298.html
>
> 2024-05-19 00:17:28,643:INFO:root:cannot find a ChangeLog location in 
> message
>
> That's the commit
> ---------------
>     Revert "Revert: "Enable prange support.""
>
>     This reverts commit d7bb8eaade3cd3aa70715c8567b4d7b08098e699 and 
> enables prange
>     support again.
> ---------------
>
> * * * The attached patch adds this commit to the ignore list and helps 
> with the diagnosis by showing the failing hash in the error message.
>
> OK for mainline?
>
> Post commit: Can someone install the new version + fix the ChangeLog 
> for the ignored commit?
>
> * * *
>
> What I do not understand: Why does this commit get applied? I do see 
> for both
>
> contrib/gcc-changelog/git_check_commit.py -v -p 
> da73261ce7731be7f2b164f1db796878cdc23365
>
> and
>
> contrib/gcc-changelog/git_email.py 
> 0001-Revert-Revert-Enable-prange-support.patch the error above. - And 
> I do not understand why it made it past the commit check but now fails?
>
> Likewise for8057f9aa1f7e70490064de796d7a8d42d446caf8
>
> Does the commit hook use an older version of the check scripts? Does 
> it ignore the errors? Or what goes wrong here? Any idea?

Tobias

[-- Attachment #2: 0001-contrib-gcc-changelog-git_update_version.py-Add-igno-v2.patch --]
[-- Type: text/x-patch, Size: 950 bytes --]

From f56b1764f2b5c2c83c6852607405e5be0a763a2c Mon Sep 17 00:00:00 2001
From: Tobias Burnus <tburnus@baylibre.com>
Date: Sun, 19 May 2024 08:17:42 +0200
Subject: [PATCH] contrib/gcc-changelog/git_update_version.py: Improve diagnostic

contrib/ChangeLog:

        * gcc-changelog/git_update_version.py (prepend_to_changelog_files): Output
	git hash in case errors occurred.

diff --git a/contrib/gcc-changelog/git_update_version.py b/contrib/gcc-changelog/git_update_version.py
index 24f6c43d0b2..ec0151b83fe 100755
--- a/contrib/gcc-changelog/git_update_version.py
+++ b/contrib/gcc-changelog/git_update_version.py
@@ -58,6 +58,7 @@ def read_timestamp(path):
 
 def prepend_to_changelog_files(repo, folder, git_commit, add_to_git):
     if not git_commit.success:
+        logging.info(f"While processing {git_commit.info.hexsha}:")
         for error in git_commit.errors:
             logging.info(error)
         raise AssertionError()
-- 
2.45.0

Re: [Patch] contrib/gcc-changelog/git_update_version.py: Improve diagnostic (was: [Patch] contrib/gcc-changelog/git_update_version.py: Add ignore commit, improve diagnostic)

[Jakub Jelinek]

On Mon, May 20, 2024 at 08:31:02AM +0200, Tobias Burnus wrote:
> Hmm, there were now two daily bumps:
> 
> Date:   Mon May 20 00:16:30 2024 +0000
> 
> Date:   Sun May 19 18:15:28 2024 +0000
> 
> I really wonder why.

Because I've done it by hand.
I have in ~gccadmin a gcc-changelog copy and adjusted update_version_git
script which doesn't use contrib/gcc-changelog subdirectory from the
checkout it makes but from the ~gccadmin directory, because I don't want to
constantly try to add some commit number to IGNORED_COMMITS, see that it
either works or doesn't (I think sometimes it needs the hash of the revert
commit, at other times the commit hash referenced in the revert commit)
or that further ones are needed.

> From f56b1764f2b5c2c83c6852607405e5be0a763a2c Mon Sep 17 00:00:00 2001
> From: Tobias Burnus <tburnus@baylibre.com>
> Date: Sun, 19 May 2024 08:17:42 +0200
> Subject: [PATCH] contrib/gcc-changelog/git_update_version.py: Improve diagnostic
> 
> contrib/ChangeLog:
> 
>         * gcc-changelog/git_update_version.py (prepend_to_changelog_files): Output

8 spaces rather than tab

> 	git hash in case errors occurred.
> 
> diff --git a/contrib/gcc-changelog/git_update_version.py b/contrib/gcc-changelog/git_update_version.py
> index 24f6c43d0b2..ec0151b83fe 100755
> --- a/contrib/gcc-changelog/git_update_version.py
> +++ b/contrib/gcc-changelog/git_update_version.py
> @@ -58,6 +58,7 @@ def read_timestamp(path):
>  
>  def prepend_to_changelog_files(repo, folder, git_commit, add_to_git):
>      if not git_commit.success:
> +        logging.info(f"While processing {git_commit.info.hexsha}:")
>          for error in git_commit.errors:
>              logging.info(error)
>          raise AssertionError()

So, your commit is useful part of it, I'm already using something similar in
my hack (just was doing it for even successful commits, but I think your
patch is better).
And, I think best would be if update_version_git script simply
accepted a list of ignored commits from the command line too,
passed it to the git_update_version.py script and that one
added those to IGNORED_COMMITS.
Because typically if the DATESTAMP/ChangeLog updates gets stuck,
one doesn't just adjust IGNORED_COMMITS and wait up to another
day to see if it worked, but runs the script by hand to make sure
it works.

--- gcc-checkout/contrib/gcc-changelog/git_update_version.py	2024-05-13 16:52:57.890151748 +0000
+++ gcc-changelog/git_update_version.py	2024-05-19 18:13:44.953648834 +0000
@@ -41,7 +41,21 @@ IGNORED_COMMITS = (
         '040e5b0edbca861196d9e2ea2af5e805769c8d5d',
         '8057f9aa1f7e70490064de796d7a8d42d446caf8',
         '109f1b28fc94c93096506e3df0c25e331cef19d0',
-        '39f81924d88e3cc197fc3df74204c9b5e01e12f7')
+        '39f81924d88e3cc197fc3df74204c9b5e01e12f7',
+        'd7bb8eaade3cd3aa70715c8567b4d7b08098e699',
+        '89feb3557a018893cfe50c2e07f91559bd3cde2b',
+        'ccf8d3e3d26c6ba3d5e11fffeed8d64018e9c060',
+        'e0c52905f666e3d23881f82dbf39466a24f009f4',
+        'b38472ffc1e631bd357573b44d956ce16d94e666',
+        'a0b13d0860848dd5f2876897ada1e22e4e681e91',
+        'b8c772cae97b54386f7853edf0f9897012bfa90b',
+        '810d35a7e054bcbb5b66d2e5924428e445f5fba9',
+        '0df1ee083434ac00ecb19582b1e5b25e105981b2',
+        '2c688f6afce4cbb414f5baab1199cd525f309fca',
+        '60dcb710b6b4aa22ea96abc8df6dfe9067f3d7fe',
+        '44968a0e00f656e9bb3e504bb2fa1a8282002015',
+        'd7bb8eaade3cd3aa70715c8567b4d7b08098e699',
+        'da73261ce7731be7f2b164f1db796878cdc23365')
 
 FORMAT = '%(asctime)s:%(levelname)s:%(name)s:%(message)s'
 logging.basicConfig(level=logging.INFO, format=FORMAT,
@@ -125,6 +139,7 @@ def update_current_branch(ref_name):
                                       % (commit.hexsha, head.hexsha), ref_name)
         commits = [c for c in commits if c.info.hexsha not in IGNORED_COMMITS]
         for git_commit in reversed(commits):
+            logging.info('trying %s', git_commit.info.hexsha)
             prepend_to_changelog_files(repo, args.git_path, git_commit,
                                        not args.dry_mode)
         if args.dry_mode:

	Jakub

Re: [Patch] contrib/gcc-changelog/git_update_version.py: Improve diagnostic

[Tobias Burnus]

[-- Attachment #1.1: Type: text/plain, Size: 1594 bytes --]

Hi Jakub,

Jakub Jelinek wrote:
> On Mon, May 20, 2024 at 08:31:02AM +0200, Tobias Burnus wrote:
>> Hmm, there were now two daily bumps: [...] I really wonder why.
> Because I've done it by hand.

Okay, that explains it.

I still do not understand why it slipped through at the first place; I 
tried old versions down to r12-709-g772e5e82e3114f and it still FAIL for 
the invalid commit ("ERR: cannot find a ChangeLog location in message").

Thus, I wonder whether the commit hook is active at all?!?

> I have in ~gccadmin a gcc-changelog copy and adjusted update_version_git
> script which doesn't use contrib/gcc-changelog subdirectory from the
> checkout it makes but from the ~gccadmin directory,
[...]
> I'm already using something similar in
> my hack (just was doing it for even successful commits, but I think your
> patch is better).
> And, I think best would be if update_version_git script simply
> accepted a list of ignored commits from the command line too,
> passed it to the git_update_version.py script and that one
> added those to IGNORED_COMMITS.

Updated version:

* Uses my diagnostic

* Adds an -i/--ignore argument for commits. Permits to use '-i hash1  -i 
hash2' but also '-i hash1,hash2' or '-i "hash1 hash2'

* I changed the global variable to lower case as Python's style guide 
states that all uppercase variables is for constants.

* The '=None' matches one of the current usages (no argument passed); 
hence, it is now explicit and 'pylint' is happy.

OK for mainline?

Tobias

PS: I have not updated the hashes. If needed/wanted, I leave that to 
you, Jakub.

[-- Attachment #2: git_update_version-v3.patch --]
[-- Type: text/x-patch, Size: 3559 bytes --]

contrib/gcc-changelog/git_update_version.py: Improve diagnostic

contrib/ChangeLog:

	* gcc-changelog/git_update_version.py: Add '-i'/'--ignore' argument
	to add to-be-ignored commits via the command line.
	(ignored_commits): Rename from IGNORED_COMMITS and change
	type from tuple to set.
	(prepend_to_changelog_files): Show git hash if errors occurred.
	(update_current_branch): Mark argument as optional by defaulting
	to None.

 contrib/gcc-changelog/git_update_version.py | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/contrib/gcc-changelog/git_update_version.py b/contrib/gcc-changelog/git_update_version.py
index 24f6c43d0b2..c69a3a6897a 100755
--- a/contrib/gcc-changelog/git_update_version.py
+++ b/contrib/gcc-changelog/git_update_version.py
@@ -22,6 +22,7 @@ import argparse
 import datetime
 import logging
 import os
+import re
 
 from git import Repo
 
@@ -30,7 +31,7 @@ from git_repository import parse_git_revisions
 current_timestamp = datetime.datetime.now().strftime('%Y%m%d\n')
 
 # Skip the following commits, they cannot be correctly processed
-IGNORED_COMMITS = (
+ignored_commits = {
         'c2be82058fb40f3ae891c68d185ff53e07f14f45',
         '04a040d907a83af54e0a98bdba5bfabc0ef4f700',
         '2e96b5f14e4025691b57d2301d71aa6092ed44bc',
@@ -41,7 +42,7 @@ IGNORED_COMMITS = (
         '040e5b0edbca861196d9e2ea2af5e805769c8d5d',
         '8057f9aa1f7e70490064de796d7a8d42d446caf8',
         '109f1b28fc94c93096506e3df0c25e331cef19d0',
-        '39f81924d88e3cc197fc3df74204c9b5e01e12f7')
+        '39f81924d88e3cc197fc3df74204c9b5e01e12f7'}
 
 FORMAT = '%(asctime)s:%(levelname)s:%(name)s:%(message)s'
 logging.basicConfig(level=logging.INFO, format=FORMAT,
@@ -58,6 +59,7 @@ def read_timestamp(path):
 
 def prepend_to_changelog_files(repo, folder, git_commit, add_to_git):
     if not git_commit.success:
+        logging.info(f"While processing {git_commit.info.hexsha}:")
         for error in git_commit.errors:
             logging.info(error)
         raise AssertionError()
@@ -93,13 +95,15 @@ parser.add_argument('-d', '--dry-mode',
                          ' is expected')
 parser.add_argument('-c', '--current', action='store_true',
                     help='Modify current branch (--push argument is ignored)')
+parser.add_argument('-i', '--ignore', action='append',
+                    help='list of commits to ignore')
 args = parser.parse_args()
 
 repo = Repo(args.git_path)
 origin = repo.remotes['origin']
 
 
-def update_current_branch(ref_name):
+def update_current_branch(ref_name=None):
     commit = repo.head.commit
     commit_count = 1
     while commit:
@@ -123,7 +127,7 @@ def update_current_branch(ref_name):
             head = head.parents[1]
         commits = parse_git_revisions(args.git_path, '%s..%s'
                                       % (commit.hexsha, head.hexsha), ref_name)
-        commits = [c for c in commits if c.info.hexsha not in IGNORED_COMMITS]
+        commits = [c for c in commits if c.info.hexsha not in ignored_commits]
         for git_commit in reversed(commits):
             prepend_to_changelog_files(repo, args.git_path, git_commit,
                                        not args.dry_mode)
@@ -153,6 +157,9 @@ def update_current_branch(ref_name):
     else:
         logging.info('DATESTAMP unchanged')
 
+if args.ignore is not None:
+    for item in args.ignore:
+        ignored_commits.update(set(i for i in re.split(r'\s*,\s*|\s+', item)))
 
 if args.current:
     logging.info('=== Working on the current branch ===')

Re: [Patch] contrib/gcc-changelog/git_update_version.py: Improve diagnostic

[Jakub Jelinek]

On Tue, May 21, 2024 at 09:36:05AM +0200, Tobias Burnus wrote:
> Jakub Jelinek wrote:
> > On Mon, May 20, 2024 at 08:31:02AM +0200, Tobias Burnus wrote:
> > > Hmm, there were now two daily bumps: [...] I really wonder why.
> > Because I've done it by hand.
> 
> Okay, that explains it.
> 
> I still do not understand why it slipped through at the first place; I tried
> old versions down to r12-709-g772e5e82e3114f and it still FAIL for the
> invalid commit ("ERR: cannot find a ChangeLog location in message").
> 
> Thus, I wonder whether the commit hook is active at all?!?

They are.  But
https://github.com/AdaCore/git-hooks/blob/master/hooks/updates/__init__.py#L836
with
https://github.com/AdaCore/git-hooks/blob/master/hooks/updates/commits.py#L230
bypasses all commits which contain just 3 magic words in a row.
And because that part is owned by AdaCore hooks, not the GCC customizations,
not sure what to do about that.

> > I have in ~gccadmin a gcc-changelog copy and adjusted update_version_git
> > script which doesn't use contrib/gcc-changelog subdirectory from the
> > checkout it makes but from the ~gccadmin directory,
> [...]
> > I'm already using something similar in
> > my hack (just was doing it for even successful commits, but I think your
> > patch is better).
> > And, I think best would be if update_version_git script simply
> > accepted a list of ignored commits from the command line too,
> > passed it to the git_update_version.py script and that one
> > added those to IGNORED_COMMITS.
> 
> Updated version:
> 
> * Uses my diagnostic
> 
> * Adds an -i/--ignore argument for commits. Permits to use '-i hash1  -i
> hash2' but also '-i hash1,hash2' or '-i "hash1 hash2'
> 
> * I changed the global variable to lower case as Python's style guide states
> that all uppercase variables is for constants.
> 
> * The '=None' matches one of the current usages (no argument passed); hence,
> it is now explicit and 'pylint' is happy.
> 
> OK for mainline?

Yes, thanks.

> PS: I have not updated the hashes. If needed/wanted, I leave that to you,
> Jakub.

Once some commit is ignored, we won't be processing it anymore, so I think
the -i option is all we need.

	Jakub