[PATCH 1/4] contrib: add generate_snapshot_index.py

[PATCH 1/4] contrib: add generate_snapshot_index.py

[Sam James]

Script to create a map between weekly snapshots and the commit they're based on
with space-separated format BRANCH-DATE COMMIT.

For example:
8-20210107 5114ee0676e432493ada968e34071f02fb08114f
8-20210114 f9267925c648f2ccd9e4680b699e581003125bcf
...

This is helpful for bisects and quickly looking up the information from bug
reports.

contrib/:
    * generate_snapshot_index.py: New file.

Signed-off-by: Sam James <sam@gentoo.org>
---
 contrib/generate_snapshot_index.py | 79 ++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)
 create mode 100755 contrib/generate_snapshot_index.py

diff --git a/contrib/generate_snapshot_index.py b/contrib/generate_snapshot_index.py
new file mode 100755
index 000000000000..80fc14b2cf1e
--- /dev/null
+++ b/contrib/generate_snapshot_index.py
@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2023 Free Software Foundation, Inc.
+# Contributed by Sam James.
+#
+# This script is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# Script to create a map between weekly snapshots and the commit they're based on.
+# Creates known_snapshots.txt with space-separated format: BRANCH-DATE COMMIT
+# For example:
+# 8-20210107 5114ee0676e432493ada968e34071f02fb08114f
+# 8-20210114 f9267925c648f2ccd9e4680b699e581003125bcf
+
+import os
+import re
+import urllib.request
+
+MIRROR = "https://mirrorservice.org/sites/sourceware.org/pub/gcc/snapshots/"
+
+
+def get_remote_snapshot_list() -> list[str]:
+    # Parse the HTML index for links to snapshots
+    with urllib.request.urlopen(MIRROR) as index_response:
+        html = index_response.read().decode("utf-8")
+        snapshots = re.findall(r'href="([0-9]+-.*)"', html)
+
+    return snapshots
+
+
+def load_cached_entries() -> dict[str, str]:
+    local_snapshots = {}
+
+    with open("known_snapshots.txt", encoding="utf-8") as local_entries:
+        for entry in local_entries.readlines():
+            if not entry:
+                continue
+
+            date, commit = entry.strip().split(" ")
+            local_snapshots[date] = commit
+
+    return local_snapshots
+
+
+remote_snapshots = get_remote_snapshot_list()
+try:
+    known_snapshots = load_cached_entries()
+except FileNotFoundError:
+    # No cache available
+    known_snapshots = {}
+
+# This would give us chronological order (as in by creation)
+# snapshots.sort(reverse=False, key=lambda x: x.split('-')[1])
+# snapshots.sort(reverse=True, key=lambda x: x.split('-')[0])
+
+for snapshot in remote_snapshots:
+    # 8-20210107/ -> 8-20210107
+    snapshot = snapshot.strip("/")
+
+    # Don't fetch entries we already have stored.
+    if snapshot in known_snapshots:
+        continue
+
+    # The READMEs are plain text with several lines, one of which is:
+    # "with the following options: git://gcc.gnu.org/git/gcc.git branch releases/gcc-8 revision e4e5ad2304db534957c4af612aa288cb6ef51f25""
+    # We match after 'revision ' to grab the commit used.
+    with urllib.request.urlopen(f"{MIRROR}/{snapshot}/README") as readme_response:
+        data = readme_response.read().decode("utf-8")
+        parsed_commit = re.findall(r"revision (.*)", data)[0]
+        known_snapshots[snapshot] = parsed_commit
+
+# Dump it all back out to disk.
+with open("known_snapshots.txt.tmp", "w", encoding="utf-8") as known_entries:
+    for name, stored_commit in known_snapshots.items():
+        known_entries.write(f"{name} {stored_commit}\n")
+
+os.rename("known_snapshots.txt.tmp", "known_snapshots.txt")
-- 
2.42.0

[PATCH 2/4] maintainer-scripts/gcc_release: create index between snapshots <-> commits

[Sam James]

Create and maintain a known_snapshots.txt index with space-separated format
BRANCH-DATE COMMIT.

For example:
8-20210107 5114ee0676e432493ada968e34071f02fb08114f
8-20210114 f9267925c648f2ccd9e4680b699e581003125bcf
...

This is helpful for bisects and quickly looking up the information from bug
reports.

maintainer-scripts/
	* gcc_release: Create known_snapshots.txt as an index between snapshots
	and commits.

Signed-off-by: Sam James <sam@gentoo.org>
---
Note that there's a few different approaches we can take here. I've gone
for the simpler one of having it still fetch from the remote site and parse
because it's obviously hard for me to test a part which runs on the remote
machine.

We can skip this patch for now if desired. I have mixed feelings about complicating
the contrib/generate_snapshot_index.py script to take an URL / path as I'd ideally
like it to still be easily usable locally.

We could have it be generated locally and then uploaded as well, as another option.

 maintainer-scripts/gcc_release | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/maintainer-scripts/gcc_release b/maintainer-scripts/gcc_release
index 962b8efe99a7..4cd1fa799660 100755
--- a/maintainer-scripts/gcc_release
+++ b/maintainer-scripts/gcc_release
@@ -448,6 +448,9 @@ announce_snapshot() {
   SNAPSHOT_INDEX=${RELEASE}/index.html
 
   changedir "${SNAPSHOTS_DIR}"
+  # Create an index if it doesn't already exist and populate it before we add
+  # the new snapshot.
+  ${PYTHON} ${SOURCE_DIRECTORY}/contrib/generate_snapshot_index.py || error "Failed to generate snapshot index"
   echo \
 "Snapshot gcc-"${RELEASE}" is now available on
   https://gcc.gnu.org/pub/gcc/snapshots/"${RELEASE}"/
@@ -514,6 +517,9 @@ Last modified "${TEXT_DATE}"
   rm -f LATEST-${BRANCH}
   ln -s ${RELEASE} LATEST-${BRANCH}
 
+  # Add the snapshot we just made to the index
+  printf "${RELEASE} ${GITREV}\n" >> known_snapshots.txt
+
   inform "Sending mail"
 
   export QMAILHOST=gcc.gnu.org
@@ -617,6 +623,7 @@ GZIP="${GZIP:-gzip --best}"
 SCP="${SCP:-scp -p}"
 SSH="${SSH:-ssh}"
 TAR="${TAR:-tar}"
+PYTHON="${PYTHON:-python3}"
 
 ########################################################################
 # Command Line Processing
-- 
2.42.0

[PATCH 3/4] maintainer-scripts/gcc_release: use HTTPS for links

[Sam James]

maintainer-scripts/
	* gcc_release: Use HTTPS for links.

Signed-off-by: Sam James <sam@gentoo.org>
---
 maintainer-scripts/gcc_release | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/maintainer-scripts/gcc_release b/maintainer-scripts/gcc_release
index 4cd1fa799660..cf6a5731c609 100755
--- a/maintainer-scripts/gcc_release
+++ b/maintainer-scripts/gcc_release
@@ -25,7 +25,7 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with GCC; see the file COPYING3.  If not see
-# <http://www.gnu.org/licenses/>.
+# <https://www.gnu.org/licenses/>.
 #
 ########################################################################
 
@@ -454,7 +454,7 @@ announce_snapshot() {
   echo \
 "Snapshot gcc-"${RELEASE}" is now available on
   https://gcc.gnu.org/pub/gcc/snapshots/"${RELEASE}"/
-and on various mirrors, see http://gcc.gnu.org/mirrors.html for details.
+and on various mirrors, see https://gcc.gnu.org/mirrors.html for details.
 
 This snapshot has been generated from the GCC "${BRANCH}" git branch
 with the following options: "git://gcc.gnu.org/git/gcc.git branch ${GITBRANCH} revision ${GITREV}"
@@ -472,7 +472,7 @@ You'll find:
 <body>
 <h1>GCC "${RELEASE}" Snapshot</h1>
 
-<p>The <a href =\"http://gcc.gnu.org/\">GCC Project</a> makes
+<p>The <a href =\"https://gcc.gnu.org/\">GCC Project</a> makes
 periodic snapshots of the GCC source tree available to the public
 for testing purposes.</p>
 	
-- 
2.42.0

[PATCH 4/4] maintainer-scripts/gcc_release: cleanup whitespace

[Sam James]

maintainer-scripts/
	* gcc_release: Cleanup whitespace.

Signed-off-by: Sam James <sam@gentoo.org>
---
 maintainer-scripts/gcc_release | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/maintainer-scripts/gcc_release b/maintainer-scripts/gcc_release
index cf6a5731c609..965163b65b74 100755
--- a/maintainer-scripts/gcc_release
+++ b/maintainer-scripts/gcc_release
@@ -153,7 +153,7 @@ build_sources() {
       # Update this ChangeLog file only if it does not yet contain the
       # entry we are going to add.  (This is a safety net for repeated
       # runs of this script for the same release.)
-      if ! grep "GCC ${RELEASE} released." ${SOURCE_DIRECTORY}/${x} > /dev/null ; then       
+      if ! grep "GCC ${RELEASE} released." ${SOURCE_DIRECTORY}/${x} > /dev/null ; then
 	cat - ${SOURCE_DIRECTORY}/${x} > ${SOURCE_DIRECTORY}/${x}.new <<EOF
 ${LONG_DATE}  Release Manager
 
@@ -278,7 +278,7 @@ EOF
 
   # Create a "MD5SUMS" file to use for checking the validity of the release.
   echo \
-"# This file contains the MD5 checksums of the files in the 
+"# This file contains the MD5 checksums of the files in the
 # gcc-"${RELEASE}".tar.xz tarball.
 #
 # Besides verifying that all files in the tarball were correctly expanded,
@@ -323,7 +323,7 @@ build_tarfiles() {
   chmod -R a+r ${SOURCE_DIRECTORY}
   # And that all directories have mode 755.
   find ${SOURCE_DIRECTORY} -type d -exec chmod 755 {} \;
- 
+
   # Build one huge tarfile for the entire distribution.
   build_tarfile gcc-${RELEASE} `basename ${SOURCE_DIRECTORY}`
 }
@@ -442,7 +442,7 @@ snapshot_print() {
 # Announce a snapshot, both on the web and via mail.
 announce_snapshot() {
   inform "Updating links and READMEs on the FTP server"
-  
+
   TEXT_DATE=`date --date=$DATE +%B\ %d,\ %Y`
   SNAPSHOT_README=${RELEASE}/README
   SNAPSHOT_INDEX=${RELEASE}/index.html
@@ -475,7 +475,7 @@ You'll find:
 <p>The <a href =\"https://gcc.gnu.org/\">GCC Project</a> makes
 periodic snapshots of the GCC source tree available to the public
 for testing purposes.</p>
-	
+
 <p>If you are planning to download and use one of our snapshots, then
 we highly recommend you join the GCC developers list.  Details for
 how to sign up can be found on the GCC project home page.</p>
@@ -484,7 +484,7 @@ how to sign up can be found on the GCC project home page.</p>
 with the following options: <code>"git://gcc.gnu.org/git/gcc.git branch ${GITBRANCH} revision ${GITREV}"</code></p>
 
 <table>" > ${SNAPSHOT_INDEX}
-       
+
   snapshot_print gcc-${RELEASE}.tar.xz "Complete GCC"
 
   echo \
@@ -554,7 +554,7 @@ FTP_PATH=/var/ftp/pub/gcc
 # The directory in which snapshots will be placed.
 SNAPSHOTS_DIR=${FTP_PATH}/snapshots
 
-# The major number for the release.  For release `3.0.2' this would be 
+# The major number for the release.  For release `3.0.2' this would be
 # `3'
 RELEASE_MAJOR=""
 # The minor number for the release.  For release `3.0.2' this would be
@@ -566,7 +566,7 @@ RELEASE_REVISION=""
 # The complete name of the release.
 RELEASE=""
 
-# The name of the branch from which the release should be made, in a 
+# The name of the branch from which the release should be made, in a
 # user-friendly form.
 BRANCH=""
 
-- 
2.42.0

Re: [PATCH 2/4] maintainer-scripts/gcc_release: create index between snapshots <-> commits

[Jonathan Wakely]

On 02/11/23 08:39 +0000, Sam James wrote:
>Create and maintain a known_snapshots.txt index with space-separated format
>BRANCH-DATE COMMIT.
>
>For example:
>8-20210107 5114ee0676e432493ada968e34071f02fb08114f
>8-20210114 f9267925c648f2ccd9e4680b699e581003125bcf
>...
>
>This is helpful for bisects and quickly looking up the information from bug
>reports.


Is there any reason we don't just use git tags for this?

We could run a one-off job to create all the historical tags (setting
GIT_COMMITTER_DATE and GIT_AUTHOR_DATE so the tags are backdated), and
add a tagging step to the snapshot creation.

Git tags are cheap, but I can imagine a concern about hundreds of new
tags "littering" the output of 'git tag -l'. I don't _think_ you can
put tags under an alternative ref that isn't fetched by default (as we
do with refs/users and refs/vendor). I think tags have to go under
refs/tags. But grep -v could be used to filter out snapshot tags
easily.

We could use https://git-scm.com/docs/gitnamespaces for this though,
so that git --namespace=snapshots tag -l would show the snapshot tags.

Re: [PATCH 2/4] maintainer-scripts/gcc_release: create index between snapshots <-> commits

[Andreas Schwab]

On Nov 02 2023, Jonathan Wakely wrote:

> Git tags are cheap, but I can imagine a concern about hundreds of new
> tags "littering" the output of 'git tag -l'. I don't _think_ you can
> put tags under an alternative ref that isn't fetched by default (as we
> do with refs/users and refs/vendor). I think tags have to go under
> refs/tags. But grep -v could be used to filter out snapshot tags
> easily.

There is no inherent limitation on publishing tags outside of refs/tags,
to make them invisible by git tag.  There are already existing examples
of tags residing under various refs/users and refs/vendors namespaces.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [PATCH 2/4] maintainer-scripts/gcc_release: create index between snapshots <-> commits

[Jonathan Wakely]

On Thu, 2 Nov 2023 at 10:23, Andreas Schwab wrote:
>
> On Nov 02 2023, Jonathan Wakely wrote:
>
> > Git tags are cheap, but I can imagine a concern about hundreds of new
> > tags "littering" the output of 'git tag -l'. I don't _think_ you can
> > put tags under an alternative ref that isn't fetched by default (as we
> > do with refs/users and refs/vendor). I think tags have to go under
> > refs/tags. But grep -v could be used to filter out snapshot tags
> > easily.
>
> There is no inherent limitation on publishing tags outside of refs/tags,
> to make them invisible by git tag.  There are already existing examples
> of tags residing under various refs/users and refs/vendors namespaces.


Ah, good to know, thanks.

So then there's no reason that snapshots would have to clutter up the
list of default tags for anybody who isn't interested in them.

Re: [PATCH 3/4] maintainer-scripts/gcc_release: use HTTPS for links

[Joseph Myers]

On Thu, 2 Nov 2023, Sam James wrote:

> maintainer-scripts/
> 	* gcc_release: Use HTTPS for links.

OK.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH 4/4] maintainer-scripts/gcc_release: cleanup whitespace

[Joseph Myers]

On Thu, 2 Nov 2023, Sam James wrote:

> maintainer-scripts/
> 	* gcc_release: Cleanup whitespace.

OK.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH 2/4] maintainer-scripts/gcc_release: create index between snapshots <-> commits

[rep.dot.nop]

On 2 November 2023 11:25:47 CET, Jonathan Wakely <jwakely@redhat.com> wrote:
>On Thu, 2 Nov 2023 at 10:23, Andreas Schwab wrote:
>>
>> On Nov 02 2023, Jonathan Wakely wrote:
>>
>> > Git tags are cheap, but I can imagine a concern about hundreds of new
>> > tags "littering" the output of 'git tag -l'. I don't _think_ you can
>> > put tags under an alternative ref that isn't fetched by default (as we
>> > do with refs/users and refs/vendor). I think tags have to go under
>> > refs/tags. But grep -v could be used to filter out snapshot tags
>> > easily.
>>
>> There is no inherent limitation on publishing tags outside of refs/tags,
>> to make them invisible by git tag.  There are already existing examples
>> of tags residing under various refs/users and refs/vendors namespaces.
>
>
>Ah, good to know, thanks.
>
>So then there's no reason that snapshots would have to clutter up the
>list of default tags for anybody who isn't interested in them.
>

Thanks Andreas. Exactly. So, just to emphasise the obvious:
Let's please use refs/snapshot ?

Re: [PATCH 4/4] maintainer-scripts/gcc_release: cleanup whitespace

[Sam James]

Joseph Myers <joseph@codesourcery.com> writes:

> On Thu, 2 Nov 2023, Sam James wrote:
>
>> maintainer-scripts/
>> 	* gcc_release: Cleanup whitespace.
>
> OK.

Thanks. Would you mind pushing the two you approved?

Re: [PATCH 4/4] maintainer-scripts/gcc_release: cleanup whitespace

[Joseph Myers]

On Fri, 10 Nov 2023, Sam James wrote:

> Joseph Myers <joseph@codesourcery.com> writes:
> 
> > On Thu, 2 Nov 2023, Sam James wrote:
> >
> >> maintainer-scripts/
> >> 	* gcc_release: Cleanup whitespace.
> >
> > OK.
> 
> Thanks. Would you mind pushing the two you approved?

Done.

-- 
Joseph S. Myers
joseph@codesourcery.com