From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-x235.google.com (mail-lj1-x235.google.com [IPv6:2a00:1450:4864:20::235]) by sourceware.org (Postfix) with ESMTPS id D93523858D20 for ; Tue, 30 Apr 2024 12:50:56 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D93523858D20 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org D93523858D20 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::235 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1714481459; cv=none; b=MYb0lif+zxuZNhMX5afM/L5WBd84i2ETGfohP8oXA+c6dW+QiuXVnE3vFSQFPPWtJqQ2Roqq9BQ2FTDT14B0cJDaW7Ri6bEv6lxK3xt1ny8N6fN0NJeJxTcv4d4OK2RxU/OMPji5ENp9+QHXtIrv1Al+y+2Ah5/E+p11SIMDyMU= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1714481459; c=relaxed/simple; bh=+ywJelLPlgwJacDxoI6l3pULB76Y+9snSbIhmuonx/o=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=t1fa5luN9zQaPlYJqENdibc3a25jLlTVXjAWljIbGNwKvQgDAY2wspAz8AAKETIcbox08wlNo31FPE+2aa39oPyts3aM7IOs2x2DPwBA4uBrqQpBImk8QWb0LHZc9GidrbWluSyMd1b0598XNDW1tYZo7Ghb7xCxHRU5AoBBn98= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lj1-x235.google.com with SMTP id 38308e7fff4ca-2def8e58471so89053061fa.0 for ; Tue, 30 Apr 2024 05:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1714481455; x=1715086255; darn=sourceware.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=YAk0C/JRJ3GVQmPY5EMCo/FFaYvBnLBFhiHkrh1newc=; b=UnOiZz9gK497Yo4bqXzQIpZ4mA7rOVvyowX8Nf/1h1CDT4ZYc0YbAUGnfGsbTi0R/b XAzctigdx7/pDjSeVfCJ3bjouQdcwLt22IdA6h91/seS7nAeKe1LYccyJtGxbvHbiTk8 lEV0eQ3ZiR4aS4IZJ95O8Qn9nfMou57lRsxmIT16im2q/NHG2Ehuk7RSzC8Phy0TFkDK ikUmUvH/jVKPm81RtsjZJNTOWOg4YPgHr3PgCKJFGsOmYr1Tx8GCoiLKdszCrsHoXlkc xm2oJGuoXFvRecmzn6sWSY63DVO9sMFDl/YqLr0ZkyAhuUyX+F6+CsDCpk4n/5w2lTZ5 eNAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714481455; x=1715086255; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YAk0C/JRJ3GVQmPY5EMCo/FFaYvBnLBFhiHkrh1newc=; b=DPMt7mli7rdOQkaXOnq+PFowhDTlFFJoZN8T3o75mBixCcolIYUHqiAG2deEVsJLHq QmGmZdxy18XGjS3NgxCXbLAVrLjsRNujXkJQ1gfozJJ29j2MX4KirSLJJRsuQu6eHby+ avj51VvjueZoJ+ZQL5eEOfbIis60drJ6WH5Scbxy5DkprVgZigKYu6ZRxoVkarkJdHJl qZW1qbT3AR90BU1Tw16+v/o2InZfriH17Fe9M8Fu7KtMvKZHBlzzFXY+VUEmxNzUQhRe dvws3jl3Z5A3ZWn7rY0KjPfybHkbvid70SOZLzWbb633kKUzQPMVPehuJNCHAZSTGtuK DODA== X-Gm-Message-State: AOJu0Yz5GRqCEhu8pi5sTrj3xa1d3vQL8akUn9FfDynTw7pomAf5nfig ccbLhKRrK64/43XiEEtc6MSMWLsj3fy9e8eR8O3lHKf2mlHT+ymo+CNahIber4VA50AoAozBDxe ZN/c2tVWtpcOOb/zIU2msZFxP5WJJx2NGEeqfiw== X-Google-Smtp-Source: AGHT+IGpX6dJVzM6NVHi7iJU86WlfIzrgi+edt49H36kKOl/+ivA4gC1vTBs2OWxYrh4tWrc6kLZvS+DKh+EKmCP2NE= X-Received: by 2002:a2e:2203:0:b0:2e0:3ad2:b371 with SMTP id i3-20020a2e2203000000b002e03ad2b371mr5710214lji.25.1714481453023; Tue, 30 Apr 2024 05:50:53 -0700 (PDT) MIME-Version: 1.0 References: <20240419090903.2080043-1-christophe.lyon@linaro.org> <87sezh558o.fsf@gentoo.org> In-Reply-To: <87sezh558o.fsf@gentoo.org> From: Christophe Lyon Date: Tue, 30 Apr 2024 14:50:42 +0200 Message-ID: Subject: Re: [PATCH] contrib: Add autoregen.py To: Sam James Cc: gdb@sourceware.org Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-9.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi, On Sat, 20 Apr 2024 at 00:59, Sam James wrote: > > Christophe Lyon via Gdb writes: > > > This script is a copy of the current script used by Sourceware's > > autoregen buildbots. > > Please include a link to where it can be found and the commit > you're importing it from. I'd also consider CCing the builder ML. Good points, I agree. > It may be nice (not sure) to include `git log` or `git shortlog` > in the commit message to preserve some history too. > > > > > It is intended as a helper to regenerate files managed by autotools > > (autoconf, automake, aclocal, ....), as well as the toplevel > > Makefile.in which is created by autogen. > > > > Other files can be updated when using maintainer-mode, but this is not > > covered by this script. > > Who will own contrib/autoregen.py? Do we need to get all changes > reviewed across gdb, binutils, and gcc? The shared files tend to be a > huge pain for this. Yes, that's a concern I have. My motivation for adding it to contrib/ is to make it more visible (I don't think many contributors know where to find the sources of the autoregen buildbot, maybe most don't know it exists) It would also mean that the buildbot could do the equivalent of cd $project #project is one of gcc, binutils-gdb if [ -f .contrib/autoregen.py ]; then ./contrib/autoregen.py fi but that doesn't solve the maintenance pain indeed. Or maybe just drop this idea, and somehow make the existence of the buildbot more widely known? (how?) Thanks, Christophe > > > > > 2024-04-19 Christophe Lyon > > > > contrib/ > > * autoregen.py: New script. > > --- > > contrib/autoregen.py | 221 +++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 221 insertions(+) > > create mode 100755 contrib/autoregen.py > > > > diff --git a/contrib/autoregen.py b/contrib/autoregen.py > > new file mode 100755 > > index 00000000000..faffc88c5bd > > --- /dev/null > > +++ b/contrib/autoregen.py > > @@ -0,0 +1,221 @@ > > +#!/usr/bin/env python3 > > + > > +# This script helps to regenerate files managed by autotools and > > +# autogen in binutils-gdb and gcc repositories. > > + > > +# It can be used by buildbots to check that the current repository > > +# contents has been updated correctly, and by developers to update > > +# such files as expected. > > + > > +import os > > +import shutil > > +import subprocess > > +from pathlib import Path > > + > > + > > +# On Gentoo, vanilla unpatched autotools are packaged separately. > > +# We place the vanilla names first as we want to prefer those if both exist. > > +AUTOCONF_NAMES = ["autoconf-vanilla-2.69", "autoconf-2.69", "autoconf"] > > +AUTOMAKE_NAMES = ["automake-vanilla-1.15", "automake-1.15.1", "automake"] > > +ACLOCAL_NAMES = ["aclocal-vanilla-1.15", "aclocal-1.15.1", "aclocal"] > > +AUTOHEADER_NAMES = ["autoheader-vanilla-2.69", "autoheader-2.69", "autoheader"] > > +AUTORECONF_NAMES = ["autoreconf-vanilla-2.69", "autoreconf-2.69", "autoreconf"] > > + > > +# Pick the first for each list that exists on this system. > > +AUTOCONF_BIN = next(name for name in AUTOCONF_NAMES if shutil.which(name)) > > +AUTOMAKE_BIN = next(name for name in AUTOMAKE_NAMES if shutil.which(name)) > > +ACLOCAL_BIN = next(name for name in ACLOCAL_NAMES if shutil.which(name)) > > +AUTOHEADER_BIN = next(name for name in AUTOHEADER_NAMES if shutil.which(name)) > > +AUTORECONF_BIN = next(name for name in AUTORECONF_NAMES if shutil.which(name)) > > + > > +AUTOGEN_BIN = "autogen" > > + > > +# autoconf-wrapper and automake-wrapper from Gentoo look at this environment variable. > > +# It's harmless to set it on other systems though. > > +EXTRA_ENV = { > > + "WANT_AUTOCONF": AUTOCONF_BIN.split("-", 1)[1] if "-" in AUTOCONF_BIN else "", > > + "WANT_AUTOMAKE": AUTOMAKE_BIN.split("-", 1)[1] if "-" in AUTOMAKE_BIN else "", > > + "AUTOCONF": AUTOCONF_BIN, > > + "ACLOCAL": ACLOCAL_BIN, > > + "AUTOMAKE": AUTOMAKE_BIN, > > + "AUTOGEN": AUTOGEN_BIN, > > +} > > +ENV = os.environ.copy() > > +ENV.update(EXTRA_ENV) > > + > > + > > +# Directories we should skip entirely because they're vendored or have different > > +# autotools versions. > > +SKIP_DIRS = [ > > + # readline and minizip are maintained with different autotools versions > > + "readline", > > + "minizip", > > +] > > + > > +# these directories are known to be re-generatable with a simple autoreconf > > +# without special -I flags > > +# Entries commented out (and directories not listed) are handled by > > +# regenerate_manually(). > > +AUTORECONF_DIRS = [ > > + # subdirs common to binutils-gdb and gcc > > + "libbacktrace", > > + "libdecnumber", # No Makefile.am > > + "libiberty", # No Makefile.am > > + "zlib", > > + > > + # binutils-gdb subdirs > > + "bfd", > > + "binutils", > > + "etc", > > + "gas", > > + "gdb", > > + "gdbserver", > > + "gdbsupport", > > + "gnulib", > > + "gold", > > + "gprof", > > + "gprofng", > > + "gprofng/libcollector", > > + "ld", > > + "libctf", > > + "libsframe", > > + "opcodes", > > + "sim", > > + > > + # gcc subdirs > > + "c++tools", # No aclocal.m4 > > + "gcc", # No Makefile.am > > + #"fixincludes", # autoreconf complains about GCC_AC_FUNC_MMAP_BLACKLIST > > + "gnattools", # No aclocal.m4 > > + "gotools", > > + "libada", # No aclocal.m4 > > + "libatomic", > > + "libcc1", > > + "libcody", # No aclocal.m4 > > + "libcpp", # No Makefile.am > > + "libffi", > > + "libgcc", # No aclocal.m4 > > + "libgfortran", > > + # Hack: ACLOCAL_AMFLAGS = -I .. -I ../config in Makefile.in but we > > + # apply -I../config -I.. otherwise we do not match the current > > + # contents > > + #"libgm2", > > + "libgo", > > + "libgomp", > > + "libgrust", > > + "libitm", > > + "libobjc", # No Makefile.am > > + "libphobos", > > + "libquadmath", > > + "libsanitizer", > > + "libssp", > > + "libstdc++-v3", > > + # This does not cover libvtv/testsuite/other-tests/Makefile.in > > + "libvtv", > > + "lto-plugin", > > +] > > + > > + > > +# Run the shell command CMD. > > +# > > +# Print the command on stdout prior to running it. > > +def run_shell(cmd: str): > > + print(f"+ {cmd}", flush=True) > > + res = subprocess.run( > > + f"{cmd}", > > + shell=True, > > + encoding="utf8", > > + env=ENV, > > + ) > > + res.check_returncode() > > + > > + > > +def regenerate_with_autoreconf(): > > + run_shell(f"{AUTORECONF_BIN} -f") > > + > > +def regenerate_with_autogen(): > > + run_shell(f"{AUTOGEN_BIN} Makefile.def") > > + > > +def regenerate_manually(): > > + configure_lines = open("configure.ac").read().splitlines() > > + if folder.stem == "fixincludes" or folder.stem == "libgm2" or any( > > + True for line in configure_lines if line.startswith("AC_CONFIG_MACRO_DIR") > > + ): > > + include_arg = "" > > + include_arg2 = "" > > + if (folder / ".." / "config").is_dir(): > > + include_arg = "-I../config" > > + > > + # this is really a hack just for binutils-gdb/gprofng/libcollector > > + # make sure that the order of includes is done as --enable-maintainer-mode > > + if (folder / ".." / ".." / "config").is_dir(): > > + include_arg = "-I../.." > > + include_arg2 = "-I../../config" > > + > > + if folder.stem == "fixincludes": > > + include_arg = "-I.." > > + include_arg2 = "-I../config" > > + > > + if folder.stem == "libgm2": > > + include_arg = "-I../config" > > + include_arg2 = "-I.." > > + > > + # aclocal does not support the -f short option for force > > + run_shell(f"{ACLOCAL_BIN} --force {include_arg} {include_arg2}") > > + > > + if (folder / "config.in").is_file() or any( > > + True for line in configure_lines if line.startswith("AC_CONFIG_HEADERS") > > + ): > > + run_shell(f"{AUTOHEADER_BIN} -f") > > + > > + # The few lines below do not regenerate the exact same content as > > + # currently in the repo. Disable them for now. > > + # if (folder / "gm2-libs").is_dir(): > > + # run_shell(f"{AUTOCONF_BIN} -f gm2-libs/config-host.in > gm2-libs/config-host") > > + # run_shell(f"{AUTOHEADER_BIN} gm2-libs/config-host.in") > > + > > + > > + # apparently automake is somehow unstable -> skip it for gotools > > + if any( > > + True for line in configure_lines if line.startswith("AM_INIT_AUTOMAKE") > > + ) and not str(folder).endswith("gotools"): > > + run_shell(f"{AUTOMAKE_BIN} -f") > > + > > + run_shell(f"{AUTOCONF_BIN} -f") > > + > > + > > +run_shell(f"{AUTOCONF_BIN} --version") > > +run_shell(f"{AUTOMAKE_BIN} --version") > > +run_shell(f"{ACLOCAL_BIN} --version") > > +run_shell(f"{AUTOHEADER_BIN} --version") > > + > > +print(f"Extra environment: {EXTRA_ENV}", flush=True) > > + > > +config_folders: list[Path] = [] > > +autogen_folders: list[Path] = [] > > +repo_root = Path.cwd() > > + > > +for root, _, files in os.walk("."): > > + for file in files: > > + if file == "configure.ac": > > + config_folders.append(Path(root).resolve()) > > + if file == "Makefile.tpl": > > + autogen_folders.append(Path(root).resolve()) > > + > > +for folder in sorted(autogen_folders): > > + print(f"Entering directory {folder}", flush=True) > > + os.chdir(folder) > > + regenerate_with_autogen() > > + > > +for folder in sorted(config_folders): > > + if folder.stem in SKIP_DIRS: > > + print(f"Skipping directory {folder}", flush=True) > > + continue > > + > > + print(f"Entering directory {folder}", flush=True) > > + os.chdir(folder) > > + > > + if str(folder.relative_to(repo_root)) in AUTORECONF_DIRS: > > + regenerate_with_autoreconf() > > + else: > > + regenerate_manually()