gcc/DATESTAMP not updated any longer

gcc/DATESTAMP not updated any longer

[Rainer Orth]

I noticed that gcc/DATESTAMP isn't updated any longer after this
Friday.  I doubt this is intentional...

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Mon, Nov 02, 2020 at 08:14:22PM +0100, Rainer Orth wrote:
> I noticed that gcc/DATESTAMP isn't updated any longer after this
> Friday.  I doubt this is intentional...

=== Working on: master ===
branch pulled and checked out
74 revisions since last Daily bump
writing to ./libstdc++-v3/ChangeLog
writing to ./gcc/ChangeLog
writing to ./gcc/testsuite/ChangeLog
writing to ./gcc/testsuite/ChangeLog
writing to ./gcc/cp/ChangeLog
writing to ./gcc/testsuite/ChangeLog
writing to ./gcc/ChangeLog
writing to ./gcc/testsuite/ChangeLog
writing to ./gcc/ChangeLog
writing to ./libgcc/ChangeLog
writing to ./gcc/ChangeLog
writing to ./gcc/ChangeLog
writing to ./gcc/testsuite/ChangeLog
empty group "()" found:"	* tree-vect-slp.c (): Update backedges in single-node cycles."
Traceback (most recent call last):
  File "../gcc-changelog/git_update_version.py", line 143, in <module>
    update_current_branch()
  File "../gcc-changelog/git_update_version.py", line 103, in update_current_branch
    not args.dry_mode)
  File "../gcc-changelog/git_update_version.py", line 39, in prepend_to_changelog_files
    raise AssertionError()
AssertionError

That's the Richard's commit that slipped through the verification, but
upsets ChangeLog generation.

	Jakub

Re: gcc/DATESTAMP not updated any longer

[Martin Liška]

On 11/2/20 8:43 PM, Jakub Jelinek wrote:
> On Mon, Nov 02, 2020 at 08:14:22PM +0100, Rainer Orth wrote:
>> I noticed that gcc/DATESTAMP isn't updated any longer after this
>> Friday.  I doubt this is intentional...
> 
> === Working on: master ===
> branch pulled and checked out
> 74 revisions since last Daily bump
> writing to ./libstdc++-v3/ChangeLog
> writing to ./gcc/ChangeLog
> writing to ./gcc/testsuite/ChangeLog
> writing to ./gcc/testsuite/ChangeLog
> writing to ./gcc/cp/ChangeLog
> writing to ./gcc/testsuite/ChangeLog
> writing to ./gcc/ChangeLog
> writing to ./gcc/testsuite/ChangeLog
> writing to ./gcc/ChangeLog
> writing to ./libgcc/ChangeLog
> writing to ./gcc/ChangeLog
> writing to ./gcc/ChangeLog
> writing to ./gcc/testsuite/ChangeLog
> empty group "()" found:"	* tree-vect-slp.c (): Update backedges in single-node cycles."
> Traceback (most recent call last):
>    File "../gcc-changelog/git_update_version.py", line 143, in <module>
>      update_current_branch()
>    File "../gcc-changelog/git_update_version.py", line 103, in update_current_branch
>      not args.dry_mode)
>    File "../gcc-changelog/git_update_version.py", line 39, in prepend_to_changelog_files
>      raise AssertionError()
> AssertionError
> 
> That's the Richard's commit that slipped through the verification, but
> upsets ChangeLog generation.
> 
> 	Jakub
> 

Whoops! Sorry for that.

Please run the script manually with the following patch:

diff --git a/contrib/gcc-changelog/git_commit.py b/contrib/gcc-changelog/git_commit.py
index 1d0860cddd8..70cd2c5c2b3 100755
--- a/contrib/gcc-changelog/git_commit.py
+++ b/contrib/gcc-changelog/git_commit.py
@@ -465,7 +465,7 @@ class GitCommit:
                                  for needle in ('()', '[]', '<>'):
                                      if ' ' + needle in parts[0]:
                                          msg = f'empty group "{needle}" found'
-                                        self.errors.append(Error(msg, line))
+                                        # self.errors.append(Error(msg, line))
                              last_entry.lines.append(line)
                      else:
                          if last_entry.is_empty:

Then all should be fine as new error should not appear.

Thanks,
Martin

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Mon, Nov 02, 2020 at 09:37:56PM +0100, Martin Liška wrote:
> > writing to ./gcc/testsuite/ChangeLog
> > empty group "()" found:"	* tree-vect-slp.c (): Update backedges in single-node cycles."
> > Traceback (most recent call last):
> >    File "../gcc-changelog/git_update_version.py", line 143, in <module>
> >      update_current_branch()
> >    File "../gcc-changelog/git_update_version.py", line 103, in update_current_branch
> >      not args.dry_mode)
> >    File "../gcc-changelog/git_update_version.py", line 39, in prepend_to_changelog_files
> >      raise AssertionError()
> > AssertionError
> > 
> > That's the Richard's commit that slipped through the verification, but
> > upsets ChangeLog generation.
> 
> Whoops! Sorry for that.
> 
> Please run the script manually with the following patch:
> 
> diff --git a/contrib/gcc-changelog/git_commit.py b/contrib/gcc-changelog/git_commit.py
> index 1d0860cddd8..70cd2c5c2b3 100755
> --- a/contrib/gcc-changelog/git_commit.py
> +++ b/contrib/gcc-changelog/git_commit.py
> @@ -465,7 +465,7 @@ class GitCommit:
>                                  for needle in ('()', '[]', '<>'):
>                                      if ' ' + needle in parts[0]:
>                                          msg = f'empty group "{needle}" found'
> -                                        self.errors.append(Error(msg, line))
> +                                        # self.errors.append(Error(msg, line))
>                              last_entry.lines.append(line)
>                      else:
>                          if last_entry.is_empty:
> 
> Then all should be fine as new error should not appear.

It isn't that easy (because update_version_git checks the gcc trunk and
so I had to insert a sh invocation in which I've tweaked it), but it worked,
thanks.  But something is really wrong with the hooks, as the gcc-cvs mail
for the trunk daily bump wasn't sent again (r10, r9 and r8 changes did).

	Jakub

Re: gcc/DATESTAMP not updated any longer

[Joseph Myers]

On Mon, 2 Nov 2020, Jakub Jelinek via Gcc wrote:

> It isn't that easy (because update_version_git checks the gcc trunk and
> so I had to insert a sh invocation in which I've tweaked it), but it worked,
> thanks.  But something is really wrong with the hooks, as the gcc-cvs mail
> for the trunk daily bump wasn't sent again (r10, r9 and r8 changes did).

I think any issue with an email not sent to gcc-cvs should be raised with 
overseers or postmaster; they'll need to check logs at the exact time to 
see if the email was ever submitted by the hooks to the MTA (and at least 
part of the mail sending from the hooks is I think asynchronous so any 
issues might not be reported back to the git commit command).  If it was 
submitted to the MTA, the problem is with the MTA or mailman.  If it was 
not submitted to the MTA, Joel might be able to advise on how to 
instrument the process of email sending from the hooks to see where it 
went wrong.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Tue, Nov 03, 2020 at 06:26:58PM +0000, Joseph Myers wrote:
> On Mon, 2 Nov 2020, Jakub Jelinek via Gcc wrote:
> 
> > It isn't that easy (because update_version_git checks the gcc trunk and
> > so I had to insert a sh invocation in which I've tweaked it), but it worked,
> > thanks.  But something is really wrong with the hooks, as the gcc-cvs mail
> > for the trunk daily bump wasn't sent again (r10, r9 and r8 changes did).
> 
> I think any issue with an email not sent to gcc-cvs should be raised with 
> overseers or postmaster; they'll need to check logs at the exact time to 
> see if the email was ever submitted by the hooks to the MTA (and at least 
> part of the mail sending from the hooks is I think asynchronous so any 
> issues might not be reported back to the git commit command).  If it was 
> submitted to the MTA, the problem is with the MTA or mailman.  If it was 
> not submitted to the MTA, Joel might be able to advise on how to 
> instrument the process of email sending from the hooks to see where it 
> went wrong.

As I didn't hear from overseers yet and we've seen again multiple missing
mails, after discussion with Martin Liska we've applied following hack
to log the mail addresses and mail subjects that are being passed to
sendmail.postfix, and a short while ago we got another case of missing mail.
And it didn't show up in the log file either:
gcc-cvs@gcc.gnu.org [gcc r11-4793] core: Rename DECL_IS_BUILTIN -> DECL_IS_UNDECLARED_BUILTIN
gcc-cvs@gcc.gnu.org [gcc r11-4794] Add PC as control register
gcc-cvs@gcc.gnu.org [gcc r11-4795] Combine new calculated ranges with existing range.
gcc-cvs@gcc.gnu.org [gcc r11-4796] rework PRE PHI translation cache
gcc-cvs@gcc.gnu.org [gcc r11-4798] c++: Propagate attributes to clones in duplicate_decls [PR67453]
As you can see, the r11-4797 commit is missing from that list and didn't
make gcc-cvs either.
But git log certainly shows that commit in between:
commit 556ab5125912fa2233986eb19d6cd995cf7de1d2
Author: Iain Sandoe <iain@sandoe.co.uk>
Date:   Fri Jul 31 21:05:28 2020 +0100

    Darwin: Darwin 20 is to be macOS 11 (Big Sur).
So, I'm afraid it must fail or bypass this code path somewhere earlier
in the hooks.

diff --git a/hooks/post_receive.py b/hooks/post_receive.py
index 3761c00..f368559 100644
--- a/hooks/post_receive.py
+++ b/hooks/post_receive.py
@@ -16,7 +16,7 @@ from init import init_all_globals
 from updates.emails import EmailQueue
 from updates.factory import new_update
 from utils import debug, warn
-
+import traceback
 
 def post_receive_one(ref_name, old_rev, new_rev, refs, submitter_email):
     """post-receive treatment for one reference.
@@ -44,8 +44,12 @@ def post_receive_one(ref_name, old_rev, new_rev, refs, submitter_email):
              "              old_rev = %s" % old_rev,
              "              new_rev = %s" % new_rev)
         return
-    update.send_email_notifications()
-
+    try:
+        update.send_email_notifications()
+    except Exception as e:
+        with open("/tmp/gcc-git-mail.log", "a") as f:
+            f.write(str(e) + '\n')
+            f.write(traceback.format_exc() + '\n')
 
 def post_receive(updated_refs, submitter_email):
     """Implement the post-receive hook for all given updated_refs.
diff --git a/hooks/updates/emails.py b/hooks/updates/emails.py
index 706124c..81f4beb 100644
--- a/hooks/updates/emails.py
+++ b/hooks/updates/emails.py
@@ -272,6 +272,9 @@ class Email(object):
         else:  # pragma: no cover (do not want real emails during testing)
             sendmail(self.email_info.email_from, email_recipients,
                      e_msg.as_string(), 'localhost')
+            with open("/tmp/gcc-git-mail.log", "a") as f:
+                f.write(e_msg['To'] + ' ' + e_msg['Subject'] + '\n')
+                f.close()
 
         if self.filer_cmd is not None:
             self.__call_filer_cmd()

	Jakub

Re: gcc/DATESTAMP not updated any longer

[Richard Biener]

On November 6, 2020 8:45:55 PM GMT+01:00, Jakub Jelinek via Gcc <gcc@gcc.gnu.org> wrote:
>On Tue, Nov 03, 2020 at 06:26:58PM +0000, Joseph Myers wrote:
>> On Mon, 2 Nov 2020, Jakub Jelinek via Gcc wrote:
>> 
>> > It isn't that easy (because update_version_git checks the gcc trunk
>and
>> > so I had to insert a sh invocation in which I've tweaked it), but
>it worked,
>> > thanks.  But something is really wrong with the hooks, as the
>gcc-cvs mail
>> > for the trunk daily bump wasn't sent again (r10, r9 and r8 changes
>did).
>> 
>> I think any issue with an email not sent to gcc-cvs should be raised
>with 
>> overseers or postmaster; they'll need to check logs at the exact time
>to 
>> see if the email was ever submitted by the hooks to the MTA (and at
>least 
>> part of the mail sending from the hooks is I think asynchronous so
>any 
>> issues might not be reported back to the git commit command).  If it
>was 
>> submitted to the MTA, the problem is with the MTA or mailman.  If it
>was 
>> not submitted to the MTA, Joel might be able to advise on how to 
>> instrument the process of email sending from the hooks to see where
>it 
>> went wrong.
>
>As I didn't hear from overseers yet and we've seen again multiple
>missing
>mails, after discussion with Martin Liska we've applied following hack
>to log the mail addresses and mail subjects that are being passed to
>sendmail.postfix, and a short while ago we got another case of missing
>mail.
>And it didn't show up in the log file either:
>gcc-cvs@gcc.gnu.org [gcc r11-4793] core: Rename DECL_IS_BUILTIN ->
>DECL_IS_UNDECLARED_BUILTIN
>gcc-cvs@gcc.gnu.org [gcc r11-4794] Add PC as control register
>gcc-cvs@gcc.gnu.org [gcc r11-4795] Combine new calculated ranges with
>existing range.
>gcc-cvs@gcc.gnu.org [gcc r11-4796] rework PRE PHI translation cache
>gcc-cvs@gcc.gnu.org [gcc r11-4798] c++: Propagate attributes to clones
>in duplicate_decls [PR67453]
>As you can see, the r11-4797 commit is missing from that list and
>didn't
>make gcc-cvs either.
>But git log certainly shows that commit in between:
>commit 556ab5125912fa2233986eb19d6cd995cf7de1d2
>Author: Iain Sandoe <iain@sandoe.co.uk>
>Date:   Fri Jul 31 21:05:28 2020 +0100
>
>    Darwin: Darwin 20 is to be macOS 11 (Big Sur).
>So, I'm afraid it must fail or bypass this code path somewhere earlier
>in the hooks.

Is that maybe already known to the repo when it is on some rebased user branch?

Richard. 

>diff --git a/hooks/post_receive.py b/hooks/post_receive.py
>index 3761c00..f368559 100644
>--- a/hooks/post_receive.py
>+++ b/hooks/post_receive.py
>@@ -16,7 +16,7 @@ from init import init_all_globals
> from updates.emails import EmailQueue
> from updates.factory import new_update
> from utils import debug, warn
>-
>+import traceback
> 
>def post_receive_one(ref_name, old_rev, new_rev, refs,
>submitter_email):
>     """post-receive treatment for one reference.
>@@ -44,8 +44,12 @@ def post_receive_one(ref_name, old_rev, new_rev,
>refs, submitter_email):
>              "              old_rev = %s" % old_rev,
>              "              new_rev = %s" % new_rev)
>         return
>-    update.send_email_notifications()
>-
>+    try:
>+        update.send_email_notifications()
>+    except Exception as e:
>+        with open("/tmp/gcc-git-mail.log", "a") as f:
>+            f.write(str(e) + '\n')
>+            f.write(traceback.format_exc() + '\n')
> 
> def post_receive(updated_refs, submitter_email):
>     """Implement the post-receive hook for all given updated_refs.
>diff --git a/hooks/updates/emails.py b/hooks/updates/emails.py
>index 706124c..81f4beb 100644
>--- a/hooks/updates/emails.py
>+++ b/hooks/updates/emails.py
>@@ -272,6 +272,9 @@ class Email(object):
>     else:  # pragma: no cover (do not want real emails during testing)
>             sendmail(self.email_info.email_from, email_recipients,
>                      e_msg.as_string(), 'localhost')
>+            with open("/tmp/gcc-git-mail.log", "a") as f:
>+                f.write(e_msg['To'] + ' ' + e_msg['Subject'] + '\n')
>+                f.close()
> 
>         if self.filer_cmd is not None:
>             self.__call_filer_cmd()
>
>	Jakub

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Fri, Nov 06, 2020 at 08:56:25PM +0100, Richard Biener wrote:
> >    Darwin: Darwin 20 is to be macOS 11 (Big Sur).
> >So, I'm afraid it must fail or bypass this code path somewhere earlier
> >in the hooks.
> 
> Is that maybe already known to the repo when it is on some rebased user branch?

I don't think so.
I think I'll work with Martin early next week to think about further spots
to add logging, so we narrow down where it is still called and where it
isn't.
But we really don't have a reliable reproducer, commit this and it won't
show up.

	Jakub

Re: gcc/DATESTAMP not updated any longer

[Iain Sandoe]

Jakub Jelinek via Gcc <gcc@gcc.gnu.org> wrote:

> On Fri, Nov 06, 2020 at 08:56:25PM +0100, Richard Biener wrote:
>>>   Darwin: Darwin 20 is to be macOS 11 (Big Sur).
>>> So, I'm afraid it must fail or bypass this code path somewhere earlier
>>> in the hooks.
>>
>> Is that maybe already known to the repo when it is on some rebased user  
>> branch?
>
> I don't think so.

That commit was only present on a local branch, so not “known” to the repo,  
for sure.
Iain

> I think I'll work with Martin early next week to think about further spots
> to add logging, so we narrow down where it is still called and where it
> isn't.
> But we really don't have a reliable reproducer, commit this and it won't
> show up.
>
> 	Jakub

Re: gcc/DATESTAMP not updated any longer

[Iain Sandoe]

Iain Sandoe via Gcc <gcc@gcc.gnu.org> wrote:

> Jakub Jelinek via Gcc <gcc@gcc.gnu.org> wrote:
>
>> On Fri, Nov 06, 2020 at 08:56:25PM +0100, Richard Biener wrote:
>>>>  Darwin: Darwin 20 is to be macOS 11 (Big Sur).
>>>> So, I'm afraid it must fail or bypass this code path somewhere earlier
>>>> in the hooks.
>>>
>>> Is that maybe already known to the repo when it is on some rebased user  
>>> branch?
>>
>> I don't think so.
>
> That commit was only present on a local branch, so not “known” to the  
> repo, for sure.

>> I think I'll work with Martin early next week to think about further spots
>> to add logging, so we narrow down where it is still called and where it
>> isn't.
>> But we really don't have a reliable reproducer, commit this and it won't
>> show up.

Here’s another one ;

r11-4824-g49393e266a2570cb6227777acd4674f922c8a26b

In this case:
1) the commit message seems simple - no quotes, paretheses or other such  
characters.
2) there were no other commits close in time to this.

(I don’t know if the server itself was busy or something of that nature, of  
course)

Iain

Re: gcc/DATESTAMP not updated any longer

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 269 bytes --]

On 11/6/20 8:59 PM, Jakub Jelinek wrote:
> I think I'll work with Martin early next week to think about further spots
> to add logging, so we narrow down where it is still called and where it
> isn't.

Hello.

I'm suggesting to use the following extra logging.

Martin

[-- Attachment #2: 0001-Another-debugging.patch --]
[-- Type: text/x-patch, Size: 1413 bytes --]

From a39631f6515b60ae2a3c8de0129b9920471df18b Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Mon, 9 Nov 2020 08:56:44 +0100
Subject: [PATCH] Another debugging.

---
 sourceware/projects/src-home/git-hooks/hooks/post_receive.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/sourceware/projects/src-home/git-hooks/hooks/post_receive.py b/sourceware/projects/src-home/git-hooks/hooks/post_receive.py
index 0e931f8..37166e6 100644
--- a/sourceware/projects/src-home/git-hooks/hooks/post_receive.py
+++ b/sourceware/projects/src-home/git-hooks/hooks/post_receive.py
@@ -18,6 +18,7 @@ from updates.factory import new_update
 from utils import debug, warn
 
 import traceback
+import datetime
 
 def post_receive_one(ref_name, old_rev, new_rev, refs, submitter_email):
     """post-receive treatment for one reference.
@@ -35,6 +36,9 @@ def post_receive_one(ref_name, old_rev, new_rev, refs, submitter_email):
           '                        new_rev=%s)'
           % (ref_name, old_rev, new_rev))
 
+    with open("/tmp/gcc-git-mail.log", "a") as f:
+        ts = str(datetime.datetime.now())
+        f.write('post_receive_one:%s: ref_name=%s, old_rev=%s, new_rev=%s\n' % (ts, ref_name, old_rev, new_rev))
     update = new_update(ref_name, old_rev, new_rev, refs, submitter_email)
     if update is None:
         # We emit a warning, rather than trigger an assertion, because
-- 
2.29.2

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Mon, Nov 09, 2020 at 08:57:48AM +0100, Martin Liška wrote:
> On 11/6/20 8:59 PM, Jakub Jelinek wrote:
> > I think I'll work with Martin early next week to think about further spots
> > to add logging, so we narrow down where it is still called and where it
> > isn't.
> 
> Hello.
> 
> I'm suggesting to use the following extra logging.

Didn't tell much, I think we have the first missing mail and the log says
just:
post_receive_one:2020-11-09 20:12:24.268657: ref_name=refs/heads/master, old_rev=3a5f8d745f8e26d973218b088788f22ad197ca67, new_rev=38b17c27ce5a8e0cc5baa14697d4b5542b91b9d1
post_receive_one:2020-11-09 20:18:57.087896: ref_name=refs/heads/master, old_rev=38b17c27ce5a8e0cc5baa14697d4b5542b91b9d1, new_rev=83f1c165d2a2ebfba1164c730c0a65210fc4b285
gcc-cvs@gcc.gnu.org [gcc r11-4852] c++: Improve error location for class using-decl.
gcc-cvs@gcc.gnu.org [gcc r11-4853] c++: Call tsubst_pack_expansion from tsubst.
post_receive_one:2020-11-09 20:32:41.238218: ref_name=refs/heads/master, old_rev=83f1c165d2a2ebfba1164c730c0a65210fc4b285, new_rev=6db082477ad839438c4b54fc61083276c68d47ec
post_receive_one:2020-11-09 21:10:36.565970: ref_name=refs/heads/master, old_rev=6db082477ad839438c4b54fc61083276c68d47ec, new_rev=32ff3309ae5a17b3a504aef3361a8c1c30e49f2c
gcc-cvs@gcc.gnu.org [gcc r11-4855] MAINTAINERS: Add myself for write after approval

The missing r11-4854 commit is the 6db082477ad8394 one.

So I think either new_update must be raising an exception, or returning
None for some reason.
Bet we want to add logging for that exception, as well as for it returning
None and inside of it perhaps even more detailed logging on where it
returned None.

> >From a39631f6515b60ae2a3c8de0129b9920471df18b Mon Sep 17 00:00:00 2001
> From: marxin <mliska@suse.cz>
> Date: Mon, 9 Nov 2020 08:56:44 +0100
> Subject: [PATCH] Another debugging.
> 
> ---
>  sourceware/projects/src-home/git-hooks/hooks/post_receive.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/sourceware/projects/src-home/git-hooks/hooks/post_receive.py b/sourceware/projects/src-home/git-hooks/hooks/post_receive.py
> index 0e931f8..37166e6 100644
> --- a/sourceware/projects/src-home/git-hooks/hooks/post_receive.py
> +++ b/sourceware/projects/src-home/git-hooks/hooks/post_receive.py
> @@ -18,6 +18,7 @@ from updates.factory import new_update
>  from utils import debug, warn
>  
>  import traceback
> +import datetime
>  
>  def post_receive_one(ref_name, old_rev, new_rev, refs, submitter_email):
>      """post-receive treatment for one reference.
> @@ -35,6 +36,9 @@ def post_receive_one(ref_name, old_rev, new_rev, refs, submitter_email):
>            '                        new_rev=%s)'
>            % (ref_name, old_rev, new_rev))
>  
> +    with open("/tmp/gcc-git-mail.log", "a") as f:
> +        ts = str(datetime.datetime.now())
> +        f.write('post_receive_one:%s: ref_name=%s, old_rev=%s, new_rev=%s\n' % (ts, ref_name, old_rev, new_rev))
>      update = new_update(ref_name, old_rev, new_rev, refs, submitter_email)
>      if update is None:
>          # We emit a warning, rather than trigger an assertion, because
> -- 
> 2.29.2
> 


	Jakub

Re: gcc/DATESTAMP not updated any longer

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 414 bytes --]

On 11/9/20 11:09 PM, Jakub Jelinek wrote:
> So I think either new_update must be raising an exception, or returning
> None for some reason.
> Bet we want to add logging for that exception, as well as for it returning
> None and inside of it perhaps even more detailed logging on where it
> returned None.

The following should expand both a returned None or an exception.
Jakub, please install it.

Thanks,
Martin

[-- Attachment #2: 0001-another-extra-logging.patch --]
[-- Type: text/x-patch, Size: 2981 bytes --]

From c3fb1dbededb037875428ced2b66ff40dc45c9d1 Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Tue, 10 Nov 2020 09:22:28 +0100
Subject: [PATCH] another extra logging

---
 .../src-home/git-hooks/hooks/post_receive.py  | 20 +++++++++----------
 .../git-hooks/hooks/updates/factory.py        |  2 ++
 2 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/sourceware/projects/src-home/git-hooks/hooks/post_receive.py b/sourceware/projects/src-home/git-hooks/hooks/post_receive.py
index 37166e6..9386e2f 100644
--- a/sourceware/projects/src-home/git-hooks/hooks/post_receive.py
+++ b/sourceware/projects/src-home/git-hooks/hooks/post_receive.py
@@ -39,17 +39,17 @@ def post_receive_one(ref_name, old_rev, new_rev, refs, submitter_email):
     with open("/tmp/gcc-git-mail.log", "a") as f:
         ts = str(datetime.datetime.now())
         f.write('post_receive_one:%s: ref_name=%s, old_rev=%s, new_rev=%s\n' % (ts, ref_name, old_rev, new_rev))
-    update = new_update(ref_name, old_rev, new_rev, refs, submitter_email)
-    if update is None:
-        # We emit a warning, rather than trigger an assertion, because
-        # it gives the script a chance to process any other reference
-        # that was updated, but not processed yet.
-        warn("post-receive: Unsupported reference update: %s (ignored)."
-             % ref_name,
-             "              old_rev = %s" % old_rev,
-             "              new_rev = %s" % new_rev)
-        return
     try:
+        update = new_update(ref_name, old_rev, new_rev, refs, submitter_email)
+        if update is None:
+            # We emit a warning, rather than trigger an assertion, because
+            # it gives the script a chance to process any other reference
+            # that was updated, but not processed yet.
+            warn("post-receive: Unsupported reference update: %s (ignored)."
+                 % ref_name,
+                 "              old_rev = %s" % old_rev,
+                 "              new_rev = %s" % new_rev)
+            return
         update.send_email_notifications()
     except Exception as e:
         with open("/tmp/gcc-git-mail.log", "a") as f:
diff --git a/sourceware/projects/src-home/git-hooks/hooks/updates/factory.py b/sourceware/projects/src-home/git-hooks/hooks/updates/factory.py
index 663a440..42cf9cc 100644
--- a/sourceware/projects/src-home/git-hooks/hooks/updates/factory.py
+++ b/sourceware/projects/src-home/git-hooks/hooks/updates/factory.py
@@ -207,6 +207,8 @@ def new_update(ref_name, old_rev, new_rev, all_refs, submitter_email):
         raise_unrecognized_ref_name(ref_name)
 
     new_cls = REF_CHANGE_MAP.get((ref_kind, change_type, object_type), None)
+    with open("/tmp/gcc-git-mail.log", "a") as f:
+        f.write('REF_CHANGE_MAP: ref_kind=%s, change_type=%s, object_type=%s, new_cls=%s' % (str(ref_kind), str(change_type), str(object_type), str(new_cls)))
     if new_cls is None:
         return None
 
-- 
2.29.2

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Tue, Nov 10, 2020 at 09:23:09AM +0100, Martin Liška wrote:
> On 11/9/20 11:09 PM, Jakub Jelinek wrote:
> > So I think either new_update must be raising an exception, or returning
> > None for some reason.
> > Bet we want to add logging for that exception, as well as for it returning
> > None and inside of it perhaps even more detailed logging on where it
> > returned None.
> 
> The following should expand both a returned None or an exception.
> Jakub, please install it.

Just FYI, we have finally the culprit today:
'ascii' codec can't decode byte 0xe2 in position 4882: ordinal not in range(128)
Traceback (most recent call last):
  File "hooks/post_receive.py", line 87, in post_receive
  File "/sourceware1/projects/src-home/git-hooks/hooks/daemon.py", line 82, in run_in_daemon
    fun()
  File "/sourceware1/projects/src-home/git-hooks/hooks/updates/emails.py", line 107, in flush
    email.send()
  File "/sourceware1/projects/src-home/git-hooks/hooks/updates/emails.py", line 233, in send
    e_msg_body = self.__email_body_with_diff
  File "/sourceware1/projects/src-home/git-hooks/hooks/updates/emails.py", line 310, in __email_body_with_diff
    email_body += diff
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4882: ordinal not in range(128)

and at least in a couple of past missed mails it is the git diff containing
non-ascii characters (usually valid utf-8, but I think we even shouldn't
throw away mails that contain something not valid utf-8).

	Jakub

Re: gcc/DATESTAMP not updated any longer

[Rainer Orth]

Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> writes:

> I noticed that gcc/DATESTAMP isn't updated any longer after this
> Friday.  I doubt this is intentional...

This has happened again tonight...

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: gcc/DATESTAMP not updated any longer

[Martin Liška]

On 12/11/20 2:17 PM, Rainer Orth wrote:
> Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> writes:
> 
>> I noticed that gcc/DATESTAMP isn't updated any longer after this
>> Friday.  I doubt this is intentional...
> 
> This has happened again tonight...
> 
> 	Rainer
> 

Thanks for heads up. I'm aware of it and I don't see reason why (running the
update script in dry mode works).

@Jakub: Can you please take a look?

Thanks,
Martin

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Fri, Dec 11, 2020 at 07:04:28PM +0100, Martin Liška wrote:
> On 12/11/20 2:17 PM, Rainer Orth wrote:
> > Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> writes:
> > 
> > > I noticed that gcc/DATESTAMP isn't updated any longer after this
> > > Friday.  I doubt this is intentional...
> > 
> > This has happened again tonight...
> > 
> > 	Rainer
> > 
> 
> Thanks for heads up. I'm aware of it and I don't see reason why (running the
> update script in dry mode works).
> 
> @Jakub: Can you please take a look?

When running it manually it just completed without problems.
So, no idea what happened.

	Jakub

Re: gcc/DATESTAMP not updated any longer

[Martin Liška]

On 12/11/20 7:26 PM, Jakub Jelinek wrote:
> When running it manually it just completed without problems.
> So, no idea what happened.

Morning.

Unfortunately, today we have similar problem. Master was bumped, but
not any of the release branches:

commit c5853240cdae1e727a65d1300e05307e5197bc69 (origin/releases/gcc-10)
Author: GCC Administrator <gccadmin@gcc.gnu.org>
Date:   Sun Dec 13 00:16:57 2020 +0000

     Daily bump.

Can you please Jakub redirect output of the script to a /tmp/xxx.log files?

Again, running that in dry mode is fine:

./contrib/gcc-changelog/git_update_version.py -d.
=== Working on: master ===
branch pulled and checked out
3 revisions since last Daily bump
writing to ./libstdc++-v3/ChangeLog
writing to ./gcc/ChangeLog
writing to ./gcc/testsuite/ChangeLog
branch diff written to ./master.patch
branch is done

=== Working on: releases/gcc-10 ===
branch pulled and checked out
1 revisions since last Daily bump
branch diff written to ./gcc-10.patch
branch is done

=== Working on: releases/gcc-8 ===
branch pulled and checked out
1 revisions since last Daily bump
branch diff written to ./gcc-8.patch
branch is done

=== Working on: releases/gcc-9 ===
branch pulled and checked out
1 revisions since last Daily bump
branch diff written to ./gcc-9.patch
branch is done

Thanks,
Martin

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Mon, Dec 14, 2020 at 09:59:22AM +0100, Martin Liška wrote:
> On 12/11/20 7:26 PM, Jakub Jelinek wrote:
> > When running it manually it just completed without problems.
> > So, no idea what happened.
> 
> Morning.
> 
> Unfortunately, today we have similar problem. Master was bumped, but
> not any of the release branches:
> 
> commit c5853240cdae1e727a65d1300e05307e5197bc69 (origin/releases/gcc-10)
> Author: GCC Administrator <gccadmin@gcc.gnu.org>
> Date:   Sun Dec 13 00:16:57 2020 +0000
> 
>     Daily bump.
> 
> Can you please Jakub redirect output of the script to a /tmp/xxx.log files?
> 
> Again, running that in dry mode is fine:

In non-dry mode it is fine too:
$ /home/gccadmin/scripts/update_version_git
=== Working on: master ===
branch pulled and checked out
3 revisions since last Daily bump
DATESTAMP unchanged
branch is done

=== Working on: releases/gcc-10 ===
branch pulled and checked out
1 revisions since last Daily bump
DATESTAMP will be changed:
branch is pushed
branch is done

=== Working on: releases/gcc-8 ===
branch pulled and checked out
1 revisions since last Daily bump
DATESTAMP will be changed:
branch is pushed
branch is done

=== Working on: releases/gcc-9 ===
branch pulled and checked out
1 revisions since last Daily bump
DATESTAMP will be changed:
branch is pushed
branch is done

Note, last night r11 DATESTAMP bump actually succeeded, it was the other
branches that failed.

	Jakub

Re: gcc/DATESTAMP not updated any longer

[Martin Liška]

On 12/14/20 10:05 AM, Jakub Jelinek wrote:
> Note, last night r11 DATESTAMP bump actually succeeded, it was the other
> branches that failed.

I know.

So please try to output the script output to a log. So that we can analyze
for the next time.

Thanks,
Martin

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Mon, Dec 14, 2020 at 10:21:15AM +0100, Martin Liška wrote:
> On 12/14/20 10:05 AM, Jakub Jelinek wrote:
> > Note, last night r11 DATESTAMP bump actually succeeded, it was the other
> > branches that failed.
> 
> I know.
> 
> So please try to output the script output to a log. So that we can analyze
> for the next time.

I've added that to the crontab, but it remains to be seen if it is the
process itself crashing, or something external killing it.

	Jakub

Re: gcc/DATESTAMP not updated any longer

[Martin Liška]

On 12/14/20 10:23 AM, Jakub Jelinek wrote:
> On Mon, Dec 14, 2020 at 10:21:15AM +0100, Martin Liška wrote:
>> On 12/14/20 10:05 AM, Jakub Jelinek wrote:
>>> Note, last night r11 DATESTAMP bump actually succeeded, it was the other
>>> branches that failed.
>>
>> I know.
>>
>> So please try to output the script output to a log. So that we can analyze
>> for the next time.
> 
> I've added that to the crontab, but it remains to be seen if it is the
> process itself crashing, or something external killing it.

Why so?

The following should log always

0 * * * * /path/to/script.. >> /tmp/gcc-daily-bump.log 2>&1

right?

Martin

> 
> 	Jakub
> 

Re: gcc/DATESTAMP not updated any longer

[Andreas Schwab]

On Dez 14 2020, Martin Liška wrote:

> On 12/11/20 7:26 PM, Jakub Jelinek wrote:
>> When running it manually it just completed without problems.
>> So, no idea what happened.
>
> Morning.
>
> Unfortunately, today we have similar problem. Master was bumped, but
> not any of the release branches:

Perhaps it was competing with an ongoing push?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: gcc/DATESTAMP not updated any longer

[Jakub Jelinek]

On Mon, Dec 14, 2020 at 11:02:44AM +0100, Andreas Schwab wrote:
> On Dez 14 2020, Martin Liška wrote:
> 
> > On 12/11/20 7:26 PM, Jakub Jelinek wrote:
> >> When running it manually it just completed without problems.
> >> So, no idea what happened.
> >
> > Morning.
> >
> > Unfortunately, today we have similar problem. Master was bumped, but
> > not any of the release branches:
> 
> Perhaps it was competing with an ongoing push?

Unlikely.  Because last successful commit to the repo before has been more
than 3 hours before the datestamp update time and first successful commit to
the repo after it has been more than 6 hours after it.

	Jakub

Re: gcc/DATESTAMP not updated any longer

[Joseph Myers]

On Mon, 14 Dec 2020, Martin Liška wrote:

> On 12/11/20 7:26 PM, Jakub Jelinek wrote:
> > When running it manually it just completed without problems.
> > So, no idea what happened.
> 
> Morning.
> 
> Unfortunately, today we have similar problem. Master was bumped, but
> not any of the release branches:

https://gcc.gnu.org/pipermail/gccadmin/2020q4/017046.html

FileNotFoundError: [Errno 2] No such file or directory: 
'/tmp/tmp.k6tFzmcsug/gcc/.git/objects/objb2sgga3i' -> 
'/tmp/tmp.k6tFzmcsug/gcc/.git/objects/06/788005095bcff66279c3254e910d934941ca9d'

No idea why the Python git implementation apparently being used by this 
code would produce such an error.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: gcc/DATESTAMP not updated any longer

[Joseph Myers]

On Fri, 11 Dec 2020, Martin Liška wrote:

> On 12/11/20 2:17 PM, Rainer Orth wrote:
> > Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> writes:
> > 
> > > I noticed that gcc/DATESTAMP isn't updated any longer after this
> > > Friday.  I doubt this is intentional...
> > 
> > This has happened again tonight...
> > 
> > 	Rainer
> > 
> 
> Thanks for heads up. I'm aware of it and I don't see reason why (running the
> update script in dry mode works).

https://gcc.gnu.org/pipermail/gccadmin/2020q4/017037.html

OSError: [Errno 28] No space left on device: 
'/tmp/tmp.Zq3p6D4MxS/gcc/.git/objects/objn31xpefh' -> 
'/tmp/tmp.Zq3p6D4MxS/gcc/.git/objects/db/ffb02a4bcdd4ec04af3db75d86b8cc2e52bdff'

Maybe change the script to use /sourceware/snapshot-tmp/gcc (which has 
rather more space) instead of /tmp?

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: gcc/DATESTAMP not updated any longer

[Martin Liška]

On 12/15/20 12:58 AM, Joseph Myers wrote:
> Maybe change the script to use /sourceware/snapshot-tmp/gcc (which has
> rather more space) instead of /tmp?

Jakub can you please do it?

Thanks,
Martin