* Add script to build many glibc configurations
@ 2016-11-09 16:27 Joseph Myers
2016-11-10 14:27 ` Joseph Myers
` (3 more replies)
0 siblings, 4 replies; 21+ messages in thread
From: Joseph Myers @ 2016-11-09 16:27 UTC (permalink / raw)
To: libc-alpha
This patch adds a Python (3.5 or later) script to build many different
configurations of glibc, including building the required cross
compilers first. It's not intended to change any patch testing
requirements, although some people may wish to use it for high-risk
patches such as adding warning options (and it can also be used to
test building, including compiling tests, for an individual
configuration, if e.g. you wish to do such a compilation test of a
patch for an architecture it touches).
The configurations include all the GNU/Linux ABI variants in
<https://sourceware.org/glibc/wiki/ABIList> (although some do not yet
build cleanly) and it would be desirable to cover enough other
variants e.g. for CPUs using different sysdeps directories to test
building each piece of code in glibc at least once. It would also be
desirable to extend it to cover Hurd and NaCl, which might best be
done by people familiar with those configurations.
You call the script as
build-many-glibcs.py /some/where thing-to-do <other-arguments>
where /some/where is a working directory for the script. It will
create and use subdirectories build, install, logs therein. You can
use it with thing-to-do being "checkout" to create a subdirectory src
therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc,
mpfr with the sources of those components, or create those directories
manually (all except glibc can be symlinks to sources elsewhere). In
the checkout case, by default it checks out GCC 6 branch, binutils
2.27 branch, glibc mainline and releases of other components. You can
specify <component>-<version> to choose a version to check out, where
<version> is "vcs-mainline" or "vcs-<branch>" to check out from
version control (only supported for gcc, binutils, glibc) and
otherwise a release version number to download and use a tarball;
components not specified on the command line have default versions
checked out.
Other than "checkout", thing-to-do is one of host-libraries,
compilers, glibcs. So you run, in that order:
build-many-glibcs.py /some/where host-libraries
build-many-glibcs.py /some/where compilers
build-many-glibcs.py /some/where glibcs
host-libraries is run once and then those libraries are used for all
the compilers. compilers can be run once and then used many times for
testing different glibc versions (so a bot only needs to update glibc
and rerun the glibcs task, if using stable GCC / binutils; if testing
the latest versions of the whole toolchain together including mainline
GCC, it would probably want to update everything and rerun both
compilers and glibcs). You can also name particular variants after
"compilers" or "glibcs" to build just those variants (the possible
variants are hardcoded in the script).
I may add support for updating existing checkouts as well as creating
the initial one. I may also add support for allowing the set of
configurations to depend on the GCC version (to get cleaner default
results), and optionally looping over architecture-independent glibc
variants of CFLAGS and configure options as well, for every glibc
configuration listed (e.g. -Os).
GCC versions before 4.9 are not expected to work (the code uses
--with-glibc-version to get the bootstrap GCC appropriately
configured). There are various problems for particular configurations
as well.
Command-line options to the script: -jN to run N jobs in parallel
(default the number of CPU cores reported by the system); --keep=all
or --keep=failed to control keeping around build directories (default
--keep=none).
The following eight submitted patches improve build or test results
for particular configurations:
1. https://sourceware.org/ml/libc-alpha/2016-11/msg00182.html (Make
Alpha <sys/user.h> self-contained)
2. https://sourceware.org/ml/libc-alpha/2016-11/msg00183.html (Make SH
<sys/user.h> self-contained)
3. https://sourceware.org/ml/libc-alpha/2016-11/msg00189.html (Ignore
-Wmaybe-uninitialized in stdlib/bug-getcontext.c)
4. https://sourceware.org/ml/libc-alpha/2016-11/msg00256.html (Make SH
ucontext always match current kernels)
5. https://sourceware.org/ml/libc-alpha/2016-11/msg00257.html (Fix SH4
register-dump.h for soft-float)
6. https://sourceware.org/ml/libc-alpha/2016-11/msg00312.html (Make
tilegx32 install libraries in lib32 directories)
7. https://sourceware.org/ml/libc-alpha/2016-11/msg00313.html (Fix
sysdeps/ia64/fpu/libm-symbols.h for inclusion in testcases)
8. https://sourceware.org/ml/libc-alpha/2016-11/msg00314.html (Work
around IA64 tst-setcontext2.c compile failure)
2016-11-09 Joseph Myers <joseph@codesourcery.com>
* scripts/build-many-glibcs.py: New file.
diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
new file mode 100755
index 0000000..d4632cb
--- /dev/null
+++ b/scripts/build-many-glibcs.py
@@ -0,0 +1,1132 @@
+#!/usr/bin/python3
+# Build many configurations of glibc.
+# Copyright (C) 2016 Free Software Foundation, Inc.
+# This file is part of the GNU C Library.
+#
+# The GNU C Library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# The GNU C Library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with the GNU C Library; if not, see
+# <http://www.gnu.org/licenses/>.
+
+"""Build many configurations of glibc.
+
+This script takes as arguments a directory name (containing a src
+subdirectory with sources of the relevant toolchain components) and a
+description of what to do: 'checkout', to check out sources into that
+directory, 'host-libraries', to build libraries required by the
+toolchain, 'compilers', to build cross-compilers for various
+configurations, or 'glibcs', to build glibc for various configurations
+and run the compilation parts of the testsuite. Subsequent arguments
+name the versions of components to check out (<component>-<version),
+for 'checkout', or, for actions other than 'checkout', name
+configurations for which compilers or glibc are to be built.
+"""
+
+import argparse
+import os
+import os.path
+import re
+import shutil
+import stat
+import subprocess
+import sys
+import urllib.request
+
+
+class Context:
+
+ """The global state associated with builds in a given directory."""
+
+ def __init__(self, topdir, parallelism, keep, action):
+ """Initialize the context."""
+ self.topdir = topdir
+ self.parallelism = parallelism
+ self.keep = keep
+ self.srcdir = os.path.join(topdir, 'src')
+ self.installdir = os.path.join(topdir, 'install')
+ self.host_libraries_installdir = os.path.join(self.installdir,
+ 'host-libraries')
+ self.builddir = os.path.join(topdir, 'build')
+ self.logsdir = os.path.join(topdir, 'logs')
+ self.makefile = os.path.join(self.builddir, 'Makefile')
+ self.wrapper = os.path.join(self.builddir, 'wrapper')
+ self.save_logs = os.path.join(self.builddir, 'save-logs')
+ if action != 'checkout':
+ self.build_triplet = self.get_build_triplet()
+ self.glibc_version = self.get_glibc_version()
+ self.configs = {}
+ self.glibc_configs = {}
+ self.makefile_pieces = ['.PHONY: all\n']
+ self.add_all_configs()
+
+ def get_build_triplet(self):
+ """Determine the build triplet with config.guess."""
+ config_guess = os.path.join(self.component_srcdir('gcc'),
+ 'config.guess')
+ cg_out = subprocess.run([config_guess], stdout=subprocess.PIPE,
+ check=True, universal_newlines=True).stdout
+ return cg_out.rstrip()
+
+ def get_glibc_version(self):
+ """Determine the glibc version number (major.minor)."""
+ version_h = os.path.join(self.component_srcdir('glibc'), 'version.h')
+ with open(version_h, 'r') as f:
+ lines = f.readlines()
+ starttext = '#define VERSION "'
+ for l in lines:
+ if l.startswith(starttext):
+ l = l[len(starttext):]
+ l = l.rstrip('"\n')
+ m = re.fullmatch('([0-9]+)\.([0-9]+)[.0-9]*', l)
+ return '%s.%s' % m.group(1, 2)
+ print('error: could not determine glibc version')
+ exit(1)
+
+ def add_all_configs(self):
+ """Add all known glibc build configurations."""
+ self.add_config(arch='aarch64',
+ os_name='linux-gnu')
+ self.add_config(arch='aarch64_be',
+ os_name='linux-gnu')
+ self.add_config(arch='alpha',
+ os_name='linux-gnu')
+ self.add_config(arch='arm',
+ os_name='linux-gnueabi')
+ self.add_config(arch='armeb',
+ os_name='linux-gnueabi')
+ self.add_config(arch='armeb',
+ os_name='linux-gnueabi',
+ variant='be8',
+ gcc_cfg=['--with-arch=armv7-a'])
+ self.add_config(arch='arm',
+ os_name='linux-gnueabihf')
+ self.add_config(arch='armeb',
+ os_name='linux-gnueabihf')
+ self.add_config(arch='armeb',
+ os_name='linux-gnueabihf',
+ variant='be8',
+ gcc_cfg=['--with-arch=armv7-a'])
+ self.add_config(arch='hppa',
+ os_name='linux-gnu')
+ self.add_config(arch='ia64',
+ os_name='linux-gnu',
+ first_gcc_cfg=['--with-system-libunwind'])
+ self.add_config(arch='m68k',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib'])
+ self.add_config(arch='m68k',
+ os_name='linux-gnu',
+ variant='coldfire',
+ gcc_cfg=['--with-arch=cf', '--disable-multilib'])
+ self.add_config(arch='microblaze',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib'])
+ self.add_config(arch='microblazeel',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib'])
+ self.add_config(arch='mips64',
+ os_name='linux-gnu',
+ gcc_cfg=['--with-mips-plt'],
+ glibcs=[{'variant': 'n32'},
+ {'arch': 'mips',
+ 'ccopts': '-mabi=32'},
+ {'variant': 'n64',
+ 'ccopts': '-mabi=64'}])
+ self.add_config(arch='mips64',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--with-mips-plt', '--with-float=soft'],
+ glibcs=[{'variant': 'n32-soft',
+ 'cfg': ['--without-fp']},
+ {'variant': 'soft',
+ 'arch': 'mips',
+ 'ccopts': '-mabi=32',
+ 'cfg': ['--without-fp']},
+ {'variant': 'n64-soft',
+ 'ccopts': '-mabi=64',
+ 'cfg': ['--without-fp']}])
+ self.add_config(arch='mips64',
+ os_name='linux-gnu',
+ variant='nan2008',
+ gcc_cfg=['--with-mips-plt', '--with-nan=2008',
+ '--with-arch-64=mips64r2',
+ '--with-arch-32=mips32r2'],
+ glibcs=[{'variant': 'n32-nan2008'},
+ {'variant': 'nan2008',
+ 'arch': 'mips',
+ 'ccopts': '-mabi=32'},
+ {'variant': 'n64-nan2008',
+ 'ccopts': '-mabi=64'}])
+ self.add_config(arch='mips64',
+ os_name='linux-gnu',
+ variant='nan2008-soft',
+ gcc_cfg=['--with-mips-plt', '--with-nan=2008',
+ '--with-arch-64=mips64r2',
+ '--with-arch-32=mips32r2',
+ '--with-float=soft'],
+ glibcs=[{'variant': 'n32-nan2008-soft',
+ 'cfg': ['--without-fp']},
+ {'variant': 'nan2008-soft',
+ 'arch': 'mips',
+ 'ccopts': '-mabi=32',
+ 'cfg': ['--without-fp']},
+ {'variant': 'n64-nan2008-soft',
+ 'ccopts': '-mabi=64',
+ 'cfg': ['--without-fp']}])
+ self.add_config(arch='mips64el',
+ os_name='linux-gnu',
+ gcc_cfg=['--with-mips-plt'],
+ glibcs=[{'variant': 'n32'},
+ {'arch': 'mipsel',
+ 'ccopts': '-mabi=32'},
+ {'variant': 'n64',
+ 'ccopts': '-mabi=64'}])
+ self.add_config(arch='mips64el',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--with-mips-plt', '--with-float=soft'],
+ glibcs=[{'variant': 'n32-soft',
+ 'cfg': ['--without-fp']},
+ {'variant': 'soft',
+ 'arch': 'mipsel',
+ 'ccopts': '-mabi=32',
+ 'cfg': ['--without-fp']},
+ {'variant': 'n64-soft',
+ 'ccopts': '-mabi=64',
+ 'cfg': ['--without-fp']}])
+ self.add_config(arch='mips64el',
+ os_name='linux-gnu',
+ variant='nan2008',
+ gcc_cfg=['--with-mips-plt', '--with-nan=2008',
+ '--with-arch-64=mips64r2',
+ '--with-arch-32=mips32r2'],
+ glibcs=[{'variant': 'n32-nan2008'},
+ {'variant': 'nan2008',
+ 'arch': 'mipsel',
+ 'ccopts': '-mabi=32'},
+ {'variant': 'n64-nan2008',
+ 'ccopts': '-mabi=64'}])
+ self.add_config(arch='mips64el',
+ os_name='linux-gnu',
+ variant='nan2008-soft',
+ gcc_cfg=['--with-mips-plt', '--with-nan=2008',
+ '--with-arch-64=mips64r2',
+ '--with-arch-32=mips32r2',
+ '--with-float=soft'],
+ glibcs=[{'variant': 'n32-nan2008-soft',
+ 'cfg': ['--without-fp']},
+ {'variant': 'nan2008-soft',
+ 'arch': 'mipsel',
+ 'ccopts': '-mabi=32',
+ 'cfg': ['--without-fp']},
+ {'variant': 'n64-nan2008-soft',
+ 'ccopts': '-mabi=64',
+ 'cfg': ['--without-fp']}])
+ self.add_config(arch='nios2',
+ os_name='linux-gnu')
+ self.add_config(arch='powerpc',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt'])
+ self.add_config(arch='powerpc',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--disable-multilib', '--with-float=soft',
+ '--enable-secureplt'],
+ glibcs=[{'variant': 'soft', 'cfg': ['--without-fp']}])
+ self.add_config(arch='powerpc64',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt'])
+ self.add_config(arch='powerpc64le',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt'])
+ self.add_config(arch='powerpc',
+ os_name='linux-gnuspe',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt',
+ '--enable-e500-double'],
+ glibcs=[{'cfg': ['--without-fp']}])
+ self.add_config(arch='powerpc',
+ os_name='linux-gnuspe',
+ variant='e500v1',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt'],
+ glibcs=[{'variant': 'e500v1', 'cfg': ['--without-fp']}])
+ self.add_config(arch='s390x',
+ os_name='linux-gnu',
+ glibcs=[{},
+ {'arch': 's390', 'ccopts': '-m31'}])
+ # SH is missing __builtin_trap support, so work around this;
+ # see <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70216>.
+ no_isolate = ('-fno-isolate-erroneous-paths-dereference'
+ ' -fno-isolate-erroneous-paths-attribute')
+ self.add_config(arch='sh3',
+ os_name='linux-gnu',
+ glibcs=[{'ccopts': no_isolate}])
+ self.add_config(arch='sh3eb',
+ os_name='linux-gnu',
+ glibcs=[{'ccopts': no_isolate}])
+ self.add_config(arch='sh4',
+ os_name='linux-gnu',
+ glibcs=[{'ccopts': no_isolate}])
+ self.add_config(arch='sh4eb',
+ os_name='linux-gnu',
+ glibcs=[{'ccopts': no_isolate}])
+ self.add_config(arch='sh4',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--without-fp'],
+ glibcs=[{'variant': 'soft',
+ 'cfg': ['--without-fp'],
+ 'ccopts': no_isolate}])
+ self.add_config(arch='sh4eb',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--without-fp'],
+ glibcs=[{'variant': 'soft',
+ 'cfg': ['--without-fp'],
+ 'ccopts': no_isolate}])
+ self.add_config(arch='sparc64',
+ os_name='linux-gnu',
+ glibcs=[{},
+ {'arch': 'sparcv9',
+ 'ccopts': '-m32 -mlong-double-128'}])
+ self.add_config(arch='tilegx',
+ os_name='linux-gnu',
+ glibcs=[{},
+ {'variant': '32', 'ccopts': '-m32'}])
+ self.add_config(arch='tilegxbe',
+ os_name='linux-gnu',
+ glibcs=[{},
+ {'variant': '32', 'ccopts': '-m32'}])
+ self.add_config(arch='tilepro',
+ os_name='linux-gnu')
+ self.add_config(arch='x86_64',
+ os_name='linux-gnu',
+ gcc_cfg=['--with-multilib-list=m64,m32,mx32'],
+ glibcs=[{},
+ {'variant': 'x32', 'ccopts': '-mx32'},
+ {'arch': 'i686', 'ccopts': '-m32 -march=i686'}],
+ extra_glibcs=[{'variant': 'disable-multi-arch',
+ 'cfg': ['--disable-multi-arch']},
+ {'variant': 'disable-multi-arch',
+ 'arch': 'i686',
+ 'ccopts': '-m32 -march=i686',
+ 'cfg': ['--disable-multi-arch']},
+ {'arch': 'i486',
+ 'ccopts': '-m32 -march=i486'},
+ {'arch': 'i586',
+ 'ccopts': '-m32 -march=i586'}])
+
+ def add_config(self, **args):
+ """Add an individual build configuration."""
+ cfg = Config(self, **args)
+ if cfg.name in self.configs:
+ print('error: duplicate config %s' % cfg.name)
+ exit(1)
+ self.configs[cfg.name] = cfg
+ for c in cfg.all_glibcs:
+ if c.name in self.glibc_configs:
+ print('error: duplicate glibc config %s' % c.name)
+ exit(1)
+ self.glibc_configs[c.name] = c
+
+ def component_srcdir(self, component):
+ """Return the source directory for a given component, e.g. gcc."""
+ return os.path.join(self.srcdir, component)
+
+ def component_builddir(self, action, config, component, subconfig=None):
+ """Return the directory to use for a build."""
+ if config is None:
+ # Host libraries.
+ assert subconfig is None
+ return os.path.join(self.builddir, action, component)
+ if subconfig is None:
+ return os.path.join(self.builddir, action, config, component)
+ else:
+ # glibc build as part of compiler build.
+ return os.path.join(self.builddir, action, config, component,
+ subconfig)
+
+ def compiler_installdir(self, config):
+ """Return the directory in which to install a compiler."""
+ return os.path.join(self.installdir, 'compilers', config)
+
+ def compiler_bindir(self, config):
+ """Return the directory in which to find compiler binaries."""
+ return os.path.join(self.compiler_installdir(config), 'bin')
+
+ def compiler_sysroot(self, config):
+ """Return the sysroot directory for a compiler."""
+ return os.path.join(self.compiler_installdir(config), 'sysroot')
+
+ def glibc_installdir(self, config):
+ """Return the directory in which to install glibc."""
+ return os.path.join(self.installdir, 'glibcs', config)
+
+ def run_builds(self, action, configs):
+ """Run the requested builds."""
+ if action == 'checkout':
+ self.checkout(configs)
+ return
+ elif action == 'host-libraries':
+ if configs:
+ print('error: configurations specified for host-libraries')
+ exit(1)
+ self.build_host_libraries()
+ elif action == 'compilers':
+ self.build_compilers(configs)
+ else:
+ self.build_glibcs(configs)
+ self.write_files()
+ self.do_build()
+
+ @staticmethod
+ def remove_dirs(*args):
+ """Remove directories and their contents if they exist."""
+ for dir in args:
+ shutil.rmtree(dir, ignore_errors=True)
+
+ @staticmethod
+ def remove_recreate_dirs(*args):
+ """Remove directories if they exist, and create them as empty."""
+ Context.remove_dirs(*args)
+ for dir in args:
+ os.makedirs(dir, exist_ok=True)
+
+ def add_makefile_cmdlist(self, target, cmdlist, logsdir):
+ """Add makefile text for a list of commands."""
+ commands = cmdlist.makefile_commands(self.wrapper, logsdir)
+ self.makefile_pieces.append('all: %s\n.PHONY: %s\n%s:\n%s\n' %
+ (target, target, target, commands))
+
+ def write_files(self):
+ """Write out the Makefile and wrapper script."""
+ mftext = ''.join(self.makefile_pieces)
+ with open(self.makefile, 'w') as f:
+ f.write(mftext)
+ wrapper_text = (
+ '#!/bin/sh\n'
+ 'prev_base=$1\n'
+ 'this_base=$2\n'
+ 'desc=$3\n'
+ 'dir=$4\n'
+ 'path=$5\n'
+ 'shift 5\n'
+ 'prev_status=$prev_base-status.txt\n'
+ 'this_status=$this_base-status.txt\n'
+ 'this_log=$this_base-log.txt\n'
+ 'date > "$this_log"\n'
+ 'echo >> "$this_log"\n'
+ 'echo "Description: $desc" >> "$this_log"\n'
+ 'echo "Command: $*" >> "$this_log"\n'
+ 'echo "Directory: $dir" >> "$this_log"\n'
+ 'echo "Path addition: $path" >> "$this_log"\n'
+ 'echo >> "$this_log"\n'
+ 'record_status ()\n'
+ '{\n'
+ ' echo >> "$this_log"\n'
+ ' echo "$1: $desc" > "$this_status"\n'
+ ' echo "$1: $desc" >> "$this_log"\n'
+ ' echo >> "$this_log"\n'
+ ' date >> "$this_log"\n'
+ ' echo "$1: $desc"\n'
+ ' exit 0\n'
+ '}\n'
+ 'check_error ()\n'
+ '{\n'
+ ' if [ "$1" != "0" ]; then\n'
+ ' record_status FAIL\n'
+ ' fi\n'
+ '}\n'
+ 'if [ "$prev_base" ] && ! grep -q "^PASS" "$prev_status"; then\n'
+ ' record_status UNRESOLVED\n'
+ 'fi\n'
+ 'if [ "$dir" ]; then\n'
+ ' cd "$dir"\n'
+ ' check_error "$?"\n'
+ 'fi\n'
+ 'if [ "$path" ]; then\n'
+ ' PATH=$path:$PATH\n'
+ 'fi\n'
+ '"$@" < /dev/null >> "$this_log" 2>&1\n'
+ 'check_error "$?"\n'
+ 'record_status PASS\n')
+ with open(self.wrapper, 'w') as f:
+ f.write(wrapper_text)
+ os.chmod(self.wrapper,
+ (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
+ stat.S_IROTH|stat.S_IXOTH))
+ save_logs_text = (
+ '#!/bin/sh\n'
+ 'if ! [ -f tests.sum ]; then\n'
+ ' echo "No test summary available."\n'
+ ' exit 0\n'
+ 'fi\n'
+ 'save_file ()\n'
+ '{\n'
+ ' echo "Contents of $1:"\n'
+ ' echo\n'
+ ' cat "$1"\n'
+ ' echo\n'
+ ' echo "End of contents of $1."\n'
+ ' echo\n'
+ '}\n'
+ 'save_file tests.sum\n'
+ 'non_pass_tests=$(grep -v "^PASS: " tests.sum | sed -e "s/^PASS: //")\n'
+ 'for t in $non_pass_tests; do\n'
+ ' if [ -f "$t.out" ]; then\n'
+ ' save_file "$t.out"\n'
+ ' fi\n'
+ 'done\n')
+ with open(self.save_logs, 'w') as f:
+ f.write(save_logs_text)
+ os.chmod(self.save_logs,
+ (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
+ stat.S_IROTH|stat.S_IXOTH))
+
+ def do_build(self):
+ """Do the actual build."""
+ cmd = ['make', '-j%d' % self.parallelism, '-s', '-C', self.builddir]
+ subprocess.run(cmd, check=True)
+
+ def build_host_libraries(self):
+ """Build the host libraries."""
+ installdir = self.host_libraries_installdir
+ builddir = os.path.join(self.builddir, 'host-libraries')
+ logsdir = os.path.join(self.logsdir, 'host-libraries')
+ self.remove_recreate_dirs(installdir, builddir, logsdir)
+ cmdlist = CommandList('host-libraries', self.keep)
+ self.build_host_library(cmdlist, 'gmp')
+ self.build_host_library(cmdlist, 'mpfr',
+ ['--with-gmp=%s' % installdir])
+ self.build_host_library(cmdlist, 'mpc',
+ ['--with-gmp=%s' % installdir,
+ '--with-mpfr=%s' % installdir])
+ cmdlist.add_command('done', ['touch', os.path.join(installdir, 'ok')])
+ self.add_makefile_cmdlist('host-libraries', cmdlist, logsdir)
+
+ def build_host_library(self, cmdlist, lib, extra_opts=None):
+ """Build one host library."""
+ srcdir = self.component_srcdir(lib)
+ builddir = self.component_builddir('host-libraries', None, lib)
+ installdir = self.host_libraries_installdir
+ cmdlist.push_subdesc(lib)
+ cmdlist.create_use_dir(builddir)
+ cfg_cmd = [os.path.join(srcdir, 'configure'),
+ '--prefix=%s' % installdir,
+ '--disable-shared']
+ if extra_opts:
+ cfg_cmd.extend (extra_opts)
+ cmdlist.add_command('configure', cfg_cmd)
+ cmdlist.add_command('build', ['make'])
+ cmdlist.add_command('check', ['make', 'check'])
+ cmdlist.add_command('install', ['make', 'install'])
+ cmdlist.cleanup_dir()
+ cmdlist.pop_subdesc()
+
+ def build_compilers(self, configs):
+ """Build the compilers."""
+ if not configs:
+ self.remove_dirs(os.path.join(self.builddir, 'compilers'))
+ self.remove_dirs(os.path.join(self.installdir, 'compilers'))
+ self.remove_dirs(os.path.join(self.logsdir, 'compilers'))
+ configs = sorted(self.configs.keys())
+ for c in configs:
+ self.configs[c].build()
+
+ def build_glibcs(self, configs):
+ """Build the glibcs."""
+ if not configs:
+ self.remove_dirs(os.path.join(self.builddir, 'glibcs'))
+ self.remove_dirs(os.path.join(self.installdir, 'glibcs'))
+ self.remove_dirs(os.path.join(self.logsdir, 'glibcs'))
+ configs = sorted(self.glibc_configs.keys())
+ for c in configs:
+ self.glibc_configs[c].build()
+
+ def checkout(self, versions):
+ """Check out the desired component versions."""
+ default_versions = {'binutils': 'vcs-2.27',
+ 'gcc': 'vcs-6',
+ 'glibc': 'vcs-mainline',
+ 'gmp': '6.1.1',
+ 'linux': '4.8.6',
+ 'mpc': '1.0.3',
+ 'mpfr': '3.1.5'}
+ use_versions = {}
+ for v in versions:
+ found_v = False
+ for k in default_versions.keys():
+ kx = k + '-'
+ if v.startswith(kx):
+ vx = v[len(kx):]
+ if k in use_versions:
+ print('error: multiple versions for %s' % k)
+ exit(1)
+ use_versions[k] = vx
+ found_v = True
+ break
+ if not found_v:
+ print('error: unknown component in %s' % v)
+ exit(1)
+ for k in default_versions.keys():
+ if k not in use_versions:
+ use_versions[k] = default_versions[k]
+ os.makedirs(self.srcdir, exist_ok=True)
+ for k in sorted(default_versions.keys()):
+ v = use_versions[k]
+ if v.startswith('vcs-'):
+ self.checkout_vcs(k, v[4:])
+ else:
+ self.checkout_tar(k, v)
+
+ def checkout_vcs(self, component, version):
+ """Check out the given version of the given component from version
+ control."""
+ if component == 'binutils':
+ git_url = 'git://sourceware.org/git/binutils-gdb.git'
+ if version == 'mainline':
+ git_branch = 'master'
+ else:
+ trans = str.maketrans({'.': '_'})
+ git_branch = 'binutils-%s-branch' % version.translate(trans)
+ self.git_checkout(component, git_url, git_branch)
+ elif component == 'gcc':
+ if version == 'mainline':
+ branch = 'trunk'
+ else:
+ trans = str.maketrans({'.': '_'})
+ branch = 'branches/gcc-%s-branch' % version.translate(trans)
+ svn_url = 'svn://gcc.gnu.org/svn/gcc/%s' % branch
+ self.gcc_checkout(svn_url)
+ elif component == 'glibc':
+ git_url = 'git://sourceware.org/git/glibc.git'
+ if version == 'mainline':
+ git_branch = 'master'
+ else:
+ git_branch = 'release/%s/master' % version
+ self.git_checkout(component, git_url, git_branch)
+ self.fix_glibc_timestamps()
+ else:
+ print('error: component %s coming from VCS' % component)
+ exit(1)
+
+ def git_checkout(self, component, git_url, git_branch):
+ """Check out a component from git."""
+ subprocess.run(['git', 'clone', '-q', '-b', git_branch, git_url,
+ self.component_srcdir(component)], check=True)
+
+ def fix_glibc_timestamps(self):
+ """Fix timestamps in a glibc checkout."""
+ # Ensure that builds do not try to regenerate generated files
+ # in the source tree.
+ srcdir = self.component_srcdir('glibc')
+ for dirpath, dirnames, filenames in os.walk(srcdir):
+ for f in filenames:
+ if (f == 'configure' or
+ f == 'preconfigure' or
+ f.endswith('-kw.h')):
+ to_touch = os.path.join(dirpath, f)
+ subprocess.run(['touch', to_touch], check=True)
+
+ def gcc_checkout(self, svn_url):
+ """Check out GCC from SVN."""
+ subprocess.run(['svn', 'co', '-q', svn_url,
+ self.component_srcdir('gcc')], check=True)
+ subprocess.run(['contrib/gcc_update', '--silent'],
+ cwd=self.component_srcdir('gcc'), check=True)
+
+ def checkout_tar(self, component, version):
+ """Check out the given version of the given component from a
+ tarball."""
+ url_map = {'binutils': 'https://ftp.gnu.org/gnu/binutils/binutils-%(version)s.tar.bz2',
+ 'gcc': 'https://ftp.gnu.org/gnu/gcc/gcc-%(version)s/gcc-%(version)s.tar.bz2',
+ 'gmp': 'https://ftp.gnu.org/gnu/gmp/gmp-%(version)s.tar.xz',
+ 'linux': 'https://www.kernel.org/pub/linux/kernel/v4.x/linux-%(version)s.tar.xz',
+ 'mpc': 'https://ftp.gnu.org/gnu/mpc/mpc-%(version)s.tar.gz',
+ 'mpfr': 'https://ftp.gnu.org/gnu/mpfr/mpfr-%(version)s.tar.xz'}
+ if component not in url_map:
+ print('error: component %s coming from tarball' % component)
+ exit(1)
+ url = url_map[component] % {'version': version}
+ filename = os.path.join(self.srcdir, url.split('/')[-1])
+ response = urllib.request.urlopen(url)
+ data = response.read()
+ with open(filename, 'wb') as f:
+ f.write(data)
+ subprocess.run(['tar', '-C', self.srcdir, '-x', '-f', filename],
+ check=True)
+ os.rename(os.path.join(self.srcdir, '%s-%s' % (component, version)),
+ self.component_srcdir(component))
+ os.remove(filename)
+
+
+class Config:
+
+ """A configuration for building a compiler and associated libraries."""
+
+ def __init__(self, ctx, arch, os_name, variant=None, gcc_cfg=None,
+ first_gcc_cfg=None, glibcs=None, extra_glibcs=None):
+ """Initialize a Config object."""
+ self.ctx = ctx
+ self.arch = arch
+ self.os = os_name
+ self.variant = variant
+ if variant is None:
+ self.name = '%s-%s' % (arch, os_name)
+ else:
+ self.name = '%s-%s-%s' % (arch, os_name, variant)
+ self.triplet = '%s-glibc-%s' % (arch, os_name)
+ if gcc_cfg is None:
+ self.gcc_cfg = []
+ else:
+ self.gcc_cfg = gcc_cfg
+ if first_gcc_cfg is None:
+ self.first_gcc_cfg = []
+ else:
+ self.first_gcc_cfg = first_gcc_cfg
+ if glibcs is None:
+ glibcs = [{'variant': variant}]
+ if extra_glibcs is None:
+ extra_glibcs = []
+ glibcs = [Glibc(self, **g) for g in glibcs]
+ extra_glibcs = [Glibc(self, **g) for g in extra_glibcs]
+ self.all_glibcs = glibcs + extra_glibcs
+ self.compiler_glibcs = glibcs
+ self.installdir = ctx.compiler_installdir(self.name)
+ self.bindir = ctx.compiler_bindir(self.name)
+ self.sysroot = ctx.compiler_sysroot(self.name)
+ self.builddir = os.path.join(ctx.builddir, 'compilers', self.name)
+ self.logsdir = os.path.join(ctx.logsdir, 'compilers', self.name)
+
+ def component_builddir(self, component):
+ """Return the directory to use for a (non-glibc) build."""
+ return self.ctx.component_builddir('compilers', self.name, component)
+
+ def build(self):
+ """Generate commands to build this compiler."""
+ self.ctx.remove_recreate_dirs(self.installdir, self.builddir,
+ self.logsdir)
+ cmdlist = CommandList('compilers-%s' % self.name, self.ctx.keep)
+ cmdlist.add_command('check-host-libraries',
+ ['test', '-f',
+ os.path.join(self.ctx.host_libraries_installdir,
+ 'ok')])
+ cmdlist.use_path(self.bindir)
+ self.build_cross_tool(cmdlist, 'binutils', 'binutils',
+ ['--disable-gdb',
+ '--disable-libdecnumber',
+ '--disable-readline',
+ '--disable-sim'])
+ if self.os.startswith('linux'):
+ self.install_linux_headers(cmdlist)
+ self.build_gcc(cmdlist, True)
+ for g in self.compiler_glibcs:
+ cmdlist.push_subdesc('glibc')
+ cmdlist.push_subdesc(g.name)
+ g.build_glibc(cmdlist, True)
+ cmdlist.pop_subdesc()
+ cmdlist.pop_subdesc()
+ self.build_gcc(cmdlist, False)
+ cmdlist.add_command('done', ['touch',
+ os.path.join(self.installdir, 'ok')])
+ self.ctx.add_makefile_cmdlist('compilers-%s' % self.name, cmdlist,
+ self.logsdir)
+
+ def build_cross_tool(self, cmdlist, tool_src, tool_build, extra_opts=None):
+ """Build one cross tool."""
+ srcdir = self.ctx.component_srcdir(tool_src)
+ builddir = self.component_builddir(tool_build)
+ cmdlist.push_subdesc(tool_build)
+ cmdlist.create_use_dir(builddir)
+ cfg_cmd = [os.path.join(srcdir, 'configure'),
+ '--prefix=%s' % self.installdir,
+ '--build=%s' % self.ctx.build_triplet,
+ '--host=%s' % self.ctx.build_triplet,
+ '--target=%s' % self.triplet,
+ '--with-sysroot=%s' % self.sysroot]
+ if extra_opts:
+ cfg_cmd.extend(extra_opts)
+ cmdlist.add_command('configure', cfg_cmd)
+ cmdlist.add_command('build', ['make'])
+ cmdlist.add_command('install', ['make', 'install'])
+ cmdlist.cleanup_dir()
+ cmdlist.pop_subdesc()
+
+ def install_linux_headers(self, cmdlist):
+ """Install Linux kernel headers."""
+ arch_map = {'aarch64': 'arm64',
+ 'alpha': 'alpha',
+ 'arm': 'arm',
+ 'hppa': 'parisc',
+ 'i486': 'x86',
+ 'i586': 'x86',
+ 'i686': 'x86',
+ 'i786': 'x86',
+ 'ia64': 'ia64',
+ 'm68k': 'm68k',
+ 'microblaze': 'microblaze',
+ 'mips': 'mips',
+ 'nios2': 'nios2',
+ 'powerpc': 'powerpc',
+ 's390': 's390',
+ 'sh': 'sh',
+ 'sparc': 'sparc',
+ 'tile': 'tile',
+ 'x86_64': 'x86'}
+ linux_arch = None
+ for k in arch_map:
+ if self.arch.startswith(k):
+ linux_arch = arch_map[k]
+ break
+ assert linux_arch is not None
+ srcdir = self.ctx.component_srcdir('linux')
+ builddir = self.component_builddir('linux')
+ headers_dir = os.path.join(self.sysroot, 'usr')
+ cmdlist.push_subdesc('linux')
+ cmdlist.create_use_dir(builddir)
+ cmdlist.add_command('install-headers',
+ ['make', '-C', srcdir, 'O=%s' % builddir,
+ 'ARCH=%s' % linux_arch,
+ 'INSTALL_HDR_PATH=%s' % headers_dir,
+ 'headers_install'])
+ cmdlist.cleanup_dir()
+ cmdlist.pop_subdesc()
+
+ def build_gcc(self, cmdlist, bootstrap):
+ """Build GCC."""
+ # libsanitizer commonly breaks because of glibc header
+ # changes, or on unusual targets. libssp is of little
+ # relevance with glibc's own stack checking support.
+ cfg_opts = list(self.gcc_cfg)
+ cfg_opts += ['--disable-libsanitizer', '--disable-libssp']
+ if bootstrap:
+ tool_build = 'gcc-first'
+ # Building a static-only, C-only compiler that is
+ # sufficient to build glibc. Various libraries and
+ # features that may require libc headers must be disabled.
+ # When configuring with a sysroot, --with-newlib is
+ # required to define inhibit_libc (to stop some parts of
+ # libgcc including libc headers); --without-headers is not
+ # sufficient.
+ cfg_opts += ['--enable-languages=c', '--disable-shared',
+ '--disable-threads',
+ '--disable-libatomic',
+ '--disable-decimal-float',
+ '--disable-libffi',
+ '--disable-libgomp',
+ '--disable-libitm',
+ '--disable-libmpx',
+ '--disable-libquadmath',
+ '--without-headers', '--with-newlib',
+ '--with-glibc-version=%s' % self.ctx.glibc_version
+ ]
+ cfg_opts += self.first_gcc_cfg
+ else:
+ tool_build = 'gcc'
+ cfg_opts += ['--enable-languages=c,c++', '--enable-shared',
+ '--enable-threads']
+ self.build_cross_tool(cmdlist, 'gcc', tool_build, cfg_opts)
+
+
+class Glibc:
+
+ """A configuration for building glibc."""
+
+ def __init__(self, compiler, arch=None, os_name=None, variant=None,
+ cfg=None, ccopts=None):
+ """Initialize a Glibc object."""
+ self.ctx = compiler.ctx
+ self.compiler = compiler
+ if arch is None:
+ self.arch = compiler.arch
+ else:
+ self.arch = arch
+ if os_name is None:
+ self.os = compiler.os
+ else:
+ self.os = os_name
+ self.variant = variant
+ if variant is None:
+ self.name = '%s-%s' % (self.arch, self.os)
+ else:
+ self.name = '%s-%s-%s' % (self.arch, self.os, variant)
+ self.triplet = '%s-glibc-%s' % (self.arch, self.os)
+ if cfg is None:
+ self.cfg = []
+ else:
+ self.cfg = cfg
+ self.ccopts = ccopts
+
+ def tool_name(self, tool):
+ """Return the name of a cross-compilation tool."""
+ ctool = '%s-%s' % (self.compiler.triplet, tool)
+ if self.ccopts and (tool == 'gcc' or tool == 'g++'):
+ ctool = '%s %s' % (ctool, self.ccopts)
+ return ctool
+
+ def build(self):
+ """Generate commands to build this glibc."""
+ builddir = self.ctx.component_builddir('glibcs', self.name, 'glibc')
+ installdir = self.ctx.glibc_installdir(self.name)
+ logsdir = os.path.join(self.ctx.logsdir, 'glibcs', self.name)
+ self.ctx.remove_recreate_dirs(installdir, builddir, logsdir)
+ cmdlist = CommandList('glibcs-%s' % self.name, self.ctx.keep)
+ cmdlist.add_command('check-compilers',
+ ['test', '-f',
+ os.path.join(self.compiler.installdir, 'ok')])
+ cmdlist.use_path(self.compiler.bindir)
+ self.build_glibc(cmdlist, False)
+ self.ctx.add_makefile_cmdlist('glibcs-%s' % self.name, cmdlist,
+ logsdir)
+
+ def build_glibc(self, cmdlist, for_compiler):
+ """Generate commands to build this glibc, either as part of a compiler
+ build or with the bootstrapped compiler (and in the latter case, run
+ tests as well)."""
+ srcdir = self.ctx.component_srcdir('glibc')
+ if for_compiler:
+ builddir = self.ctx.component_builddir('compilers',
+ self.compiler.name, 'glibc',
+ self.name)
+ installdir = self.compiler.sysroot
+ srcdir_copy = self.ctx.component_builddir('compilers',
+ self.compiler.name,
+ 'glibc-src',
+ self.name)
+ else:
+ builddir = self.ctx.component_builddir('glibcs', self.name,
+ 'glibc')
+ installdir = self.ctx.glibc_installdir(self.name)
+ srcdir_copy = self.ctx.component_builddir('glibcs', self.name,
+ 'glibc-src')
+ cmdlist.create_use_dir(builddir)
+ # glibc builds write into the source directory, and even if
+ # not intentionally there is a risk of bugs that involve
+ # writing into the working directory. To avoid possible
+ # concurrency issues, copy the source directory.
+ cmdlist.create_copy_dir(srcdir, srcdir_copy)
+ cfg_cmd = [os.path.join(srcdir_copy, 'configure'),
+ '--prefix=/usr',
+ '--enable-add-ons',
+ '--build=%s' % self.ctx.build_triplet,
+ '--host=%s' % self.triplet,
+ 'CC=%s' % self.tool_name('gcc'),
+ 'CXX=%s' % self.tool_name('g++'),
+ 'AR=%s' % self.tool_name('ar'),
+ 'AS=%s' % self.tool_name('as'),
+ 'LD=%s' % self.tool_name('ld'),
+ 'NM=%s' % self.tool_name('nm'),
+ 'OBJCOPY=%s' % self.tool_name('objcopy'),
+ 'OBJDUMP=%s' % self.tool_name('objdump'),
+ 'RANLIB=%s' % self.tool_name('ranlib'),
+ 'READELF=%s' % self.tool_name('readelf'),
+ 'STRIP=%s' % self.tool_name('strip')]
+ cfg_cmd += self.cfg
+ cmdlist.add_command('configure', cfg_cmd)
+ cmdlist.add_command('build', ['make'])
+ cmdlist.add_command('install', ['make', 'install',
+ 'install_root=%s' % installdir])
+ # GCC uses paths such as lib/../lib64, so make sure lib
+ # directories always exist.
+ cmdlist.add_command('mkdir-lib', ['mkdir', '-p',
+ os.path.join(installdir, 'lib'),
+ os.path.join(installdir,
+ 'usr', 'lib')])
+ if not for_compiler:
+ cmdlist.add_command('check', ['make', 'check'])
+ cmdlist.add_command('save-logs', [self.ctx.save_logs],
+ always_run=True)
+ cmdlist.cleanup_dir('cleanup-src', srcdir_copy)
+ cmdlist.cleanup_dir()
+
+
+class Command:
+
+ """A command run in the build process."""
+
+ def __init__(self, desc, num, dir, path, command, always_run=False):
+ """Initialize a Command object."""
+ self.dir = dir
+ self.path = path
+ self.desc = desc
+ trans = str.maketrans({' ': '-'})
+ self.logbase = '%03d-%s' % (num, desc.translate(trans))
+ self.command = command
+ self.always_run = always_run
+
+ @staticmethod
+ def shell_make_quote_string(s):
+ """Given a string not containing a newline, quote it for use by the
+ shell and make."""
+ assert '\n' not in s
+ if re.fullmatch('[]+,./0-9@A-Z_a-z-]+', s):
+ return s
+ strans = str.maketrans({"'": "'\\''"})
+ s = "'%s'" % s.translate(strans)
+ mtrans = str.maketrans({'$': '$$'})
+ return s.translate(mtrans)
+
+ @staticmethod
+ def shell_make_quote_list(l, translate_make):
+ """Given a list of strings not containing newlines, quote them for use
+ by the shell and make, returning a single string. If translate_make
+ is true and the first string is 'make', change it to $(MAKE)."""
+ l = [Command.shell_make_quote_string(s) for s in l]
+ if translate_make and l[0] == 'make':
+ l[0] = '$(MAKE)'
+ return ' '.join(l)
+
+ def shell_make_quote(self):
+ """Return this command quoted for the shell and make."""
+ return self.shell_make_quote_list(self.command, True)
+
+
+class CommandList:
+
+ """A list of commands run in the build process."""
+
+ def __init__(self, desc, keep):
+ """Initialize a CommandList object."""
+ self.cmdlist = []
+ self.dir = None
+ self.path = None
+ self.desc = [desc]
+ self.keep = keep
+
+ def desc_txt(self, desc):
+ """Return the description to use for a command."""
+ return '%s %s' % (' '.join(self.desc), desc)
+
+ def use_dir(self, dir):
+ """Set the default directory for subsequent commands."""
+ self.dir = dir
+
+ def use_path(self, path):
+ """Set a directory to be prepended to the PATH for subsequent
+ commands."""
+ self.path = path
+
+ def push_subdesc(self, subdesc):
+ """Set the default subdescription for subsequent commands (e.g., the
+ name of a component being built, within the series of commands
+ building it)."""
+ self.desc.append(subdesc)
+
+ def pop_subdesc(self):
+ """Pop a subdescription from the list of descriptions."""
+ self.desc.pop()
+
+ def create_use_dir(self, dir):
+ """Remove and recreate a directory and use it for subsequent
+ commands."""
+ self.add_command_dir('rm', None, ['rm', '-rf', dir])
+ self.add_command_dir('mkdir', None, ['mkdir', '-p', dir])
+ self.use_dir(dir)
+
+ def create_copy_dir(self, src, dest):
+ """Remove a directory and recreate it as a copy from the given
+ source."""
+ self.add_command_dir('copy-rm', None, ['rm', '-rf', dest])
+ parent = os.path.dirname(dest)
+ self.add_command_dir('copy-mkdir', None, ['mkdir', '-p', parent])
+ self.add_command_dir('copy', None, ['cp', '-a', src, dest])
+
+ def add_command_dir(self, desc, dir, command, always_run=False):
+ """Add a command to run in a given directory."""
+ cmd = Command(self.desc_txt(desc), len(self.cmdlist), dir, self.path,
+ command, always_run)
+ self.cmdlist.append(cmd)
+
+ def add_command(self, desc, command, always_run=False):
+ """Add a command to run in the default directory."""
+ cmd = Command(self.desc_txt(desc), len(self.cmdlist), self.dir,
+ self.path, command, always_run)
+ self.cmdlist.append(cmd)
+
+ def cleanup_dir(self, desc='cleanup', dir=None):
+ """Clean up a build directory. If no directory is specified, the
+ default directory is cleaned up and ceases to be the default
+ directory."""
+ if dir is None:
+ dir = self.dir
+ self.use_dir(None)
+ if self.keep != 'all':
+ self.add_command_dir(desc, None, ['rm', '-rf', dir],
+ always_run=(self.keep == 'none'))
+
+ def makefile_commands(self, wrapper, logsdir):
+ """Return the sequence of commands in the form of text for a Makefile.
+ The given wrapper script takes arguments: base of logs for
+ previous command, or empty; base of logs for this command;
+ description; directory; PATH addition; the command itself."""
+ # prev_base is the base of the name for logs of the previous
+ # command that is not always-run (that is, a build command,
+ # whose failure should stop subsequent build commands from
+ # being run, as opposed to a cleanup command, which is run
+ # even if previous commands failed).
+ prev_base = ''
+ cmds = []
+ for c in self.cmdlist:
+ ctxt = c.shell_make_quote()
+ if prev_base and not c.always_run:
+ prev_log = os.path.join(logsdir, prev_base)
+ else:
+ prev_log = ''
+ this_log = os.path.join(logsdir, c.logbase)
+ if not c.always_run:
+ prev_base = c.logbase
+ if c.dir is None:
+ dir = ''
+ else:
+ dir = c.dir
+ if c.path is None:
+ path = ''
+ else:
+ path = c.path
+ prelims = [wrapper, prev_log, this_log, c.desc, dir, path]
+ prelim_txt = Command.shell_make_quote_list(prelims, False)
+ cmds.append('\t@%s %s' % (prelim_txt, ctxt))
+ return '\n'.join(cmds)
+
+
+def get_parser():
+ """Return an argument parser for this module."""
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument('-j', dest='parallelism',
+ help='Run this number of jobs in parallel',
+ type=int, default=os.cpu_count())
+ parser.add_argument('--keep', dest='keep',
+ help='Whether to keep all build directories, '
+ 'none or only those from failed builds',
+ default='none', choices=('none', 'all', 'failed'))
+ parser.add_argument('topdir',
+ help='Toplevel working directory')
+ parser.add_argument('action',
+ help='What to do',
+ choices=('checkout', 'host-libraries', 'compilers',
+ 'glibcs'))
+ parser.add_argument('configs',
+ help='Versions to check out or configurations to build',
+ nargs='*')
+ return parser
+
+
+def main(argv):
+ """The main entry point."""
+ parser = get_parser()
+ opts = parser.parse_args(argv)
+ topdir = os.path.abspath(opts.topdir)
+ ctx = Context(topdir, opts.parallelism, opts.keep, opts.action)
+ ctx.run_builds(opts.action, opts.configs)
+
+
+if __name__ == '__main__':
+ main(sys.argv[1:])
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-09 16:27 Add script to build many glibc configurations Joseph Myers
@ 2016-11-10 14:27 ` Joseph Myers
2016-11-10 16:44 ` Joseph Myers
2016-11-23 17:36 ` Chris Metcalf
2016-11-10 17:09 ` Steve Ellcey
` (2 subsequent siblings)
3 siblings, 2 replies; 21+ messages in thread
From: Joseph Myers @ 2016-11-10 14:27 UTC (permalink / raw)
To: libc-alpha
Consider the use of '-s' removed from the make command in this script (it
was intended to make the script itself quieter by not reporting entering
and leaving a directory, just PASS and FAIL lines, but had the unintended
effect of quieting all the submakes as well).
Here are the failures I observe with this script, with the indicated eight
glibc patches applied (excluding check-compilers failures at the glibcs
stage from the compilers having failed to build in the compilers stage):
GCC 4.9:
FAIL: compilers-ia64-linux-gnu gcc-first build
inhibit_libc unsupported for ia64 before GCC 6.
FAIL: compilers-m68k-linux-gnu-coldfire gcc-first build
ICE building libgcc,
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68467>.
FAIL: compilers-nios2-linux-gnu glibc nios2-linux-gnu build
ICE building glibc.
FAIL: compilers-sh3-linux-gnu glibc sh3-linux-gnu build
FAIL: compilers-sh3eb-linux-gnu glibc sh3eb-linux-gnu build
FAIL: compilers-sh4-linux-gnu-soft glibc sh4-linux-gnu-soft build
FAIL: compilers-sh4-linux-gnu glibc sh4-linux-gnu build
FAIL: compilers-sh4eb-linux-gnu-soft glibc sh4eb-linux-gnu-soft build
FAIL: compilers-sh4eb-linux-gnu glibc sh4eb-linux-gnu build
Likely fixed by
<https://sourceware.org/ml/libc-alpha/2016-11/msg00326.html> (but the BE
case observed before to be broken with GCC before GCC 6 for unknown
reasons - like the problems observed with LE in the absence of the
-fno-isolate-erroneous-paths-* options but not fixed by those options).
FAIL: compilers-tilepro-linux-gnu glibc tilepro-linux-gnu build
__WORDSIZE32_PTRDIFF_LONG, __WORDSIZE32_SIZE_ULONG not defined -
omission from bits/wordsize.h patch?
FAIL: glibcs-hppa-linux-gnu check
elf/check-execstack, elf/check-textrel fail.
FAIL: glibcs-microblaze-linux-gnu build
FAIL: glibcs-microblazeel-linux-gnu build
EH issues.
FAIL: glibcs-tilegx-linux-gnu-32 check
FAIL: glibcs-tilegx-linux-gnu check
FAIL: glibcs-tilegxbe-linux-gnu-32 check
FAIL: glibcs-tilegxbe-linux-gnu check
Missing asm/dataplane.h.
GCC 5:
FAIL: compilers-ia64-linux-gnu gcc-first build
FAIL: compilers-m68k-linux-gnu-coldfire gcc-first build
FAIL: compilers-sh3-linux-gnu glibc sh3-linux-gnu build
FAIL: compilers-sh3eb-linux-gnu glibc sh3eb-linux-gnu build
FAIL: compilers-sh4-linux-gnu-soft glibc sh4-linux-gnu-soft build
FAIL: compilers-sh4-linux-gnu glibc sh4-linux-gnu build
FAIL: compilers-sh4eb-linux-gnu-soft glibc sh4eb-linux-gnu-soft build
FAIL: compilers-sh4eb-linux-gnu glibc sh4eb-linux-gnu build
FAIL: compilers-tilepro-linux-gnu glibc tilepro-linux-gnu build
FAIL: glibcs-hppa-linux-gnu check
FAIL: glibcs-microblaze-linux-gnu build
FAIL: glibcs-microblazeel-linux-gnu build
FAIL: glibcs-tilegx-linux-gnu-32 check
FAIL: glibcs-tilegx-linux-gnu check
FAIL: glibcs-tilegxbe-linux-gnu-32 check
FAIL: glibcs-tilegxbe-linux-gnu check
As above.
FAIL: glibcs-nios2-linux-gnu check
Missing asm/cachectl.h.
GCC 6:
FAIL: compilers-m68k-linux-gnu-coldfire gcc-first build
FAIL: compilers-sh3-linux-gnu glibc sh3-linux-gnu build
FAIL: compilers-sh3eb-linux-gnu glibc sh3eb-linux-gnu build
FAIL: compilers-sh4-linux-gnu-soft glibc sh4-linux-gnu-soft build
FAIL: compilers-sh4-linux-gnu glibc sh4-linux-gnu build
FAIL: compilers-sh4eb-linux-gnu-soft glibc sh4eb-linux-gnu-soft build
FAIL: compilers-sh4eb-linux-gnu glibc sh4eb-linux-gnu build
FAIL: compilers-tilepro-linux-gnu glibc tilepro-linux-gnu build
FAIL: glibcs-hppa-linux-gnu check
FAIL: glibcs-microblaze-linux-gnu build
FAIL: glibcs-microblazeel-linux-gnu build
FAIL: glibcs-nios2-linux-gnu check
FAIL: glibcs-tilegx-linux-gnu-32 check
FAIL: glibcs-tilegx-linux-gnu check
As above.
FAIL: compilers-tilegxbe-linux-gnu gcc-first build
ICE building libgcc.
FAIL: glibcs-ia64-linux-gnu check
elf/check-execstack fails.
FAIL: compilers-m68k-linux-gnu-coldfire gcc-first build
FAIL: compilers-sh3-linux-gnu glibc sh3-linux-gnu build
FAIL: compilers-sh3eb-linux-gnu glibc sh3eb-linux-gnu build
FAIL: compilers-sh4-linux-gnu-soft glibc sh4-linux-gnu-soft build
FAIL: compilers-sh4-linux-gnu glibc sh4-linux-gnu build
FAIL: compilers-sh4eb-linux-gnu-soft glibc sh4eb-linux-gnu-soft build
FAIL: compilers-sh4eb-linux-gnu glibc sh4eb-linux-gnu build
FAIL: compilers-tilepro-linux-gnu glibc tilepro-linux-gnu build
FAIL: glibcs-hppa-linux-gnu check
FAIL: glibcs-ia64-linux-gnu check
FAIL: glibcs-microblaze-linux-gnu build
FAIL: glibcs-microblazeel-linux-gnu build
FAIL: glibcs-nios2-linux-gnu check
As above.
FAIL: compilers-powerpc-linux-gnuspe-e500v1 glibc powerpc-linux-gnuspe-e500v1 build
ICE building glibc.
FAIL: compilers-powerpc-linux-gnuspe gcc-first build
ICE building libgcc.
FAIL: compilers-tilegxbe-linux-gnu gcc-first build
FAIL: compilers-tilegx-linux-gnu gcc-first build
Assembler errors building libgcc.
FAIL: glibcs-x86_64-linux-gnu-disable-multi-arch check
FAIL: glibcs-x86_64-linux-gnu-x32 check
FAIL: glibcs-x86_64-linux-gnu check
Spurious "used uninitialized" errors in libmvec tests, to investigate
further.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-10 14:27 ` Joseph Myers
@ 2016-11-10 16:44 ` Joseph Myers
2016-11-23 17:36 ` Chris Metcalf
1 sibling, 0 replies; 21+ messages in thread
From: Joseph Myers @ 2016-11-10 16:44 UTC (permalink / raw)
To: libc-alpha
On Thu, 10 Nov 2016, Joseph Myers wrote:
> FAIL: glibcs-x86_64-linux-gnu-disable-multi-arch check
> FAIL: glibcs-x86_64-linux-gnu-x32 check
> FAIL: glibcs-x86_64-linux-gnu check
>
> Spurious "used uninitialized" errors in libmvec tests, to investigate
> further.
Now filed as <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78295> since
these look like a clear GCC bug.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-09 16:27 Add script to build many glibc configurations Joseph Myers
2016-11-10 14:27 ` Joseph Myers
@ 2016-11-10 17:09 ` Steve Ellcey
2016-11-10 17:22 ` Joseph Myers
2016-11-11 15:20 ` Joseph Myers
2016-11-17 16:52 ` Zack Weinberg
3 siblings, 1 reply; 21+ messages in thread
From: Steve Ellcey @ 2016-11-10 17:09 UTC (permalink / raw)
To: Joseph Myers, libc-alpha
On Wed, 2016-11-09 at 16:27 +0000, Joseph Myers wrote:
> This patch adds a Python (3.5 or later) script to build many
> different
> configurations of glibc, including building the required cross
> compilers first.  It's not intended to change any patch testing
> requirements, although some people may wish to use it for high-risk
> patches such as adding warning options (and it can also be used to
> test building, including compiling tests, for an individual
> configuration, if e.g. you wish to do such a compilation test of a
> patch for an architecture it touches).
Joseph, it appears that when building the cross compilers, there is no
way to restrict that to just the compilers for certain targets. Â Is
that correct? Â On one hand I can understand that, since that gives you
the maximum flexibility in what glibc libraries to build later but on
the other hand, that is a lot of cross-compilers. Â I could see people
using this script to test the build of a particular platform that they
don't have access to as well as for testing multiple builds. Â What do
you think about allowing a list of targets on the compilers build?
Steve Ellcey
sellcey@caviumnetworks.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-10 17:09 ` Steve Ellcey
@ 2016-11-10 17:22 ` Joseph Myers
2016-11-10 17:39 ` Steve Ellcey
0 siblings, 1 reply; 21+ messages in thread
From: Joseph Myers @ 2016-11-10 17:22 UTC (permalink / raw)
To: Steve Ellcey; +Cc: libc-alpha
[-- Attachment #1: Type: text/plain, Size: 655 bytes --]
On Thu, 10 Nov 2016, Steve Ellcey wrote:
> Joseph, it appears that when building the cross compilers, there is no
> way to restrict that to just the compilers for certain targets. Â Is
> that correct? Â On one hand I can understand that, since that gives you
You can specify arguments <dir> compilers target1 target2 ...
(where the targets are ARCH-linux-gnu or ARCH-linux-gnu-variant for one of
the variants listed in the script). Then for the glibc step specify
similar arguments (note that ARCH and variant names may be different for
the glibcs in cases where a compiler has more than one glibc).
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-10 17:22 ` Joseph Myers
@ 2016-11-10 17:39 ` Steve Ellcey
0 siblings, 0 replies; 21+ messages in thread
From: Steve Ellcey @ 2016-11-10 17:39 UTC (permalink / raw)
To: Joseph Myers; +Cc: libc-alpha
On Thu, 2016-11-10 at 17:22 +0000, Joseph Myers wrote:
> On Thu, 10 Nov 2016, Steve Ellcey wrote:
>
> >
> > Joseph, it appears that when building the cross compilers, there is
> > no
> > way to restrict that to just the compilers for certain targets. Â Is
> > that correct? Â On one hand I can understand that, since that gives
> > you
> You can specify arguments <dir> compilers target1 target2 ...
>
> (where the targets are ARCH-linux-gnu or ARCH-linux-gnu-variant for
> one ofÂ
> the variants listed in the script).  Then for the glibc step specifyÂ
> similar arguments (note that ARCH and variant names may be different
> forÂ
> the glibcs in cases where a compiler has more than one glibc).
OK, I tried 'aarch64' instead of 'aarch64-linux-gnu' and when I got an
error I thought it meant that targets were not allowed and not that I
had just messed up on the target name. Â Using proper triplet target
names is working much better.
Steve Ellcey
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-09 16:27 Add script to build many glibc configurations Joseph Myers
2016-11-10 14:27 ` Joseph Myers
2016-11-10 17:09 ` Steve Ellcey
@ 2016-11-11 15:20 ` Joseph Myers
2016-11-11 19:37 ` Carlos O'Donell
2016-11-14 15:06 ` Mike Frysinger
2016-11-17 16:52 ` Zack Weinberg
3 siblings, 2 replies; 21+ messages in thread
From: Joseph Myers @ 2016-11-11 15:20 UTC (permalink / raw)
To: libc-alpha
Here is an updated version of this script with some support for updating
existing checkouts.
Add script to build many glibc configurations.
This patch adds a Python (3.5 or later) script to build many different
configurations of glibc, including building the required cross
compilers first. It's not intended to change any patch testing
requirements, although some people may wish to use it for high-risk
patches such as adding warning options (and it can also be used to
test building, including compiling tests, for an individual
configuration, if e.g. you wish to do such a compilation test of a
patch for an architecture it touches).
The configurations include all the GNU/Linux ABI variants in
<https://sourceware.org/glibc/wiki/ABIList> (although some do not yet
build cleanly) and it would be desirable to cover enough other
variants e.g. for CPUs using different sysdeps directories to test
building each piece of code in glibc at least once. It would also be
desirable to extend it to cover Hurd and NaCl, which might best be
done by people familiar with those configurations.
You call the script as
build-many-glibcs.py /some/where thing-to-do <other-arguments>
where /some/where is a working directory for the script. It will
create and use subdirectories build, install, logs therein. You can
use it with thing-to-do being "checkout" to create a subdirectory src
therein, with subdirectories binutils, gcc, glibc, gmp, linux, mpc,
mpfr with the sources of those components, or create those directories
manually (all except glibc can be symlinks to sources elsewhere). In
the checkout case, by default it checks out GCC 6 branch, binutils
2.27 branch, glibc mainline and releases of other components. You can
specify <component>-<version> to choose a version to check out, where
<version> is "vcs-mainline" or "vcs-<branch>" to check out from
version control (only supported for gcc, binutils, glibc) and
otherwise a release version number to download and use a tarball;
components not specified on the command line have default versions
checked out. If you rerun "checkout" (with the same version
specifications) it will update checkouts from version control, but
will not detect cases where the location something is expected to be
checked out from has changed.
Other than "checkout", thing-to-do is one of host-libraries,
compilers, glibcs. So you run, in that order:
build-many-glibcs.py /some/where host-libraries
build-many-glibcs.py /some/where compilers
build-many-glibcs.py /some/where glibcs
host-libraries is run once and then those libraries are used for all
the compilers. compilers can be run once and then used many times for
testing different glibc versions (so a bot only needs to update glibc
and rerun the glibcs task, if using stable GCC / binutils; if testing
the latest versions of the whole toolchain together including mainline
GCC, it would probably want to update everything and rerun both
compilers and glibcs). You can also name particular variants after
"compilers" or "glibcs" to build just those variants (the possible
variants are hardcoded in the script).
I may add support for allowing the set of configurations to depend on
the GCC version (to get cleaner default results), and optionally
looping over architecture-independent glibc variants of CFLAGS and
configure options as well, for every glibc configuration listed
(e.g. -Os).
GCC versions before 4.9 are not expected to work (the code uses
--with-glibc-version to get the bootstrap GCC appropriately
configured). There are various problems for particular configurations
as well.
Command-line options to the script: -jN to run N jobs in parallel
(default the number of CPU cores reported by the system); --keep=all
or --keep=failed to control keeping around build directories (default
--keep=none).
The following eight submitted patches improve build or test results
for particular configurations:
1. https://sourceware.org/ml/libc-alpha/2016-11/msg00182.html (Make
Alpha <sys/user.h> self-contained)
2. https://sourceware.org/ml/libc-alpha/2016-11/msg00183.html (Make SH
<sys/user.h> self-contained)
3. https://sourceware.org/ml/libc-alpha/2016-11/msg00189.html (Ignore
-Wmaybe-uninitialized in stdlib/bug-getcontext.c)
4. https://sourceware.org/ml/libc-alpha/2016-11/msg00256.html (Make SH
ucontext always match current kernels)
5. https://sourceware.org/ml/libc-alpha/2016-11/msg00257.html (Fix SH4
register-dump.h for soft-float)
6. https://sourceware.org/ml/libc-alpha/2016-11/msg00312.html (Make
tilegx32 install libraries in lib32 directories)
7. https://sourceware.org/ml/libc-alpha/2016-11/msg00313.html (Fix
sysdeps/ia64/fpu/libm-symbols.h for inclusion in testcases)
8. https://sourceware.org/ml/libc-alpha/2016-11/msg00314.html (Work
around IA64 tst-setcontext2.c compile failure)
2016-11-11 Joseph Myers <joseph@codesourcery.com>
* scripts/build-many-glibcs.py: New file.
diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
new file mode 100755
index 0000000..4b4e15c
--- /dev/null
+++ b/scripts/build-many-glibcs.py
@@ -0,0 +1,1142 @@
+#!/usr/bin/python3
+# Build many configurations of glibc.
+# Copyright (C) 2016 Free Software Foundation, Inc.
+# This file is part of the GNU C Library.
+#
+# The GNU C Library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# The GNU C Library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with the GNU C Library; if not, see
+# <http://www.gnu.org/licenses/>.
+
+"""Build many configurations of glibc.
+
+This script takes as arguments a directory name (containing a src
+subdirectory with sources of the relevant toolchain components) and a
+description of what to do: 'checkout', to check out sources into that
+directory, 'host-libraries', to build libraries required by the
+toolchain, 'compilers', to build cross-compilers for various
+configurations, or 'glibcs', to build glibc for various configurations
+and run the compilation parts of the testsuite. Subsequent arguments
+name the versions of components to check out (<component>-<version),
+for 'checkout', or, for actions other than 'checkout', name
+configurations for which compilers or glibc are to be built.
+"""
+
+import argparse
+import os
+import os.path
+import re
+import shutil
+import stat
+import subprocess
+import sys
+import urllib.request
+
+
+class Context:
+
+ """The global state associated with builds in a given directory."""
+
+ def __init__(self, topdir, parallelism, keep, action):
+ """Initialize the context."""
+ self.topdir = topdir
+ self.parallelism = parallelism
+ self.keep = keep
+ self.srcdir = os.path.join(topdir, 'src')
+ self.installdir = os.path.join(topdir, 'install')
+ self.host_libraries_installdir = os.path.join(self.installdir,
+ 'host-libraries')
+ self.builddir = os.path.join(topdir, 'build')
+ self.logsdir = os.path.join(topdir, 'logs')
+ self.makefile = os.path.join(self.builddir, 'Makefile')
+ self.wrapper = os.path.join(self.builddir, 'wrapper')
+ self.save_logs = os.path.join(self.builddir, 'save-logs')
+ if action != 'checkout':
+ self.build_triplet = self.get_build_triplet()
+ self.glibc_version = self.get_glibc_version()
+ self.configs = {}
+ self.glibc_configs = {}
+ self.makefile_pieces = ['.PHONY: all\n']
+ self.add_all_configs()
+
+ def get_build_triplet(self):
+ """Determine the build triplet with config.guess."""
+ config_guess = os.path.join(self.component_srcdir('gcc'),
+ 'config.guess')
+ cg_out = subprocess.run([config_guess], stdout=subprocess.PIPE,
+ check=True, universal_newlines=True).stdout
+ return cg_out.rstrip()
+
+ def get_glibc_version(self):
+ """Determine the glibc version number (major.minor)."""
+ version_h = os.path.join(self.component_srcdir('glibc'), 'version.h')
+ with open(version_h, 'r') as f:
+ lines = f.readlines()
+ starttext = '#define VERSION "'
+ for l in lines:
+ if l.startswith(starttext):
+ l = l[len(starttext):]
+ l = l.rstrip('"\n')
+ m = re.fullmatch('([0-9]+)\.([0-9]+)[.0-9]*', l)
+ return '%s.%s' % m.group(1, 2)
+ print('error: could not determine glibc version')
+ exit(1)
+
+ def add_all_configs(self):
+ """Add all known glibc build configurations."""
+ self.add_config(arch='aarch64',
+ os_name='linux-gnu')
+ self.add_config(arch='aarch64_be',
+ os_name='linux-gnu')
+ self.add_config(arch='alpha',
+ os_name='linux-gnu')
+ self.add_config(arch='arm',
+ os_name='linux-gnueabi')
+ self.add_config(arch='armeb',
+ os_name='linux-gnueabi')
+ self.add_config(arch='armeb',
+ os_name='linux-gnueabi',
+ variant='be8',
+ gcc_cfg=['--with-arch=armv7-a'])
+ self.add_config(arch='arm',
+ os_name='linux-gnueabihf')
+ self.add_config(arch='armeb',
+ os_name='linux-gnueabihf')
+ self.add_config(arch='armeb',
+ os_name='linux-gnueabihf',
+ variant='be8',
+ gcc_cfg=['--with-arch=armv7-a'])
+ self.add_config(arch='hppa',
+ os_name='linux-gnu')
+ self.add_config(arch='ia64',
+ os_name='linux-gnu',
+ first_gcc_cfg=['--with-system-libunwind'])
+ self.add_config(arch='m68k',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib'])
+ self.add_config(arch='m68k',
+ os_name='linux-gnu',
+ variant='coldfire',
+ gcc_cfg=['--with-arch=cf', '--disable-multilib'])
+ self.add_config(arch='microblaze',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib'])
+ self.add_config(arch='microblazeel',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib'])
+ self.add_config(arch='mips64',
+ os_name='linux-gnu',
+ gcc_cfg=['--with-mips-plt'],
+ glibcs=[{'variant': 'n32'},
+ {'arch': 'mips',
+ 'ccopts': '-mabi=32'},
+ {'variant': 'n64',
+ 'ccopts': '-mabi=64'}])
+ self.add_config(arch='mips64',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--with-mips-plt', '--with-float=soft'],
+ glibcs=[{'variant': 'n32-soft',
+ 'cfg': ['--without-fp']},
+ {'variant': 'soft',
+ 'arch': 'mips',
+ 'ccopts': '-mabi=32',
+ 'cfg': ['--without-fp']},
+ {'variant': 'n64-soft',
+ 'ccopts': '-mabi=64',
+ 'cfg': ['--without-fp']}])
+ self.add_config(arch='mips64',
+ os_name='linux-gnu',
+ variant='nan2008',
+ gcc_cfg=['--with-mips-plt', '--with-nan=2008',
+ '--with-arch-64=mips64r2',
+ '--with-arch-32=mips32r2'],
+ glibcs=[{'variant': 'n32-nan2008'},
+ {'variant': 'nan2008',
+ 'arch': 'mips',
+ 'ccopts': '-mabi=32'},
+ {'variant': 'n64-nan2008',
+ 'ccopts': '-mabi=64'}])
+ self.add_config(arch='mips64',
+ os_name='linux-gnu',
+ variant='nan2008-soft',
+ gcc_cfg=['--with-mips-plt', '--with-nan=2008',
+ '--with-arch-64=mips64r2',
+ '--with-arch-32=mips32r2',
+ '--with-float=soft'],
+ glibcs=[{'variant': 'n32-nan2008-soft',
+ 'cfg': ['--without-fp']},
+ {'variant': 'nan2008-soft',
+ 'arch': 'mips',
+ 'ccopts': '-mabi=32',
+ 'cfg': ['--without-fp']},
+ {'variant': 'n64-nan2008-soft',
+ 'ccopts': '-mabi=64',
+ 'cfg': ['--without-fp']}])
+ self.add_config(arch='mips64el',
+ os_name='linux-gnu',
+ gcc_cfg=['--with-mips-plt'],
+ glibcs=[{'variant': 'n32'},
+ {'arch': 'mipsel',
+ 'ccopts': '-mabi=32'},
+ {'variant': 'n64',
+ 'ccopts': '-mabi=64'}])
+ self.add_config(arch='mips64el',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--with-mips-plt', '--with-float=soft'],
+ glibcs=[{'variant': 'n32-soft',
+ 'cfg': ['--without-fp']},
+ {'variant': 'soft',
+ 'arch': 'mipsel',
+ 'ccopts': '-mabi=32',
+ 'cfg': ['--without-fp']},
+ {'variant': 'n64-soft',
+ 'ccopts': '-mabi=64',
+ 'cfg': ['--without-fp']}])
+ self.add_config(arch='mips64el',
+ os_name='linux-gnu',
+ variant='nan2008',
+ gcc_cfg=['--with-mips-plt', '--with-nan=2008',
+ '--with-arch-64=mips64r2',
+ '--with-arch-32=mips32r2'],
+ glibcs=[{'variant': 'n32-nan2008'},
+ {'variant': 'nan2008',
+ 'arch': 'mipsel',
+ 'ccopts': '-mabi=32'},
+ {'variant': 'n64-nan2008',
+ 'ccopts': '-mabi=64'}])
+ self.add_config(arch='mips64el',
+ os_name='linux-gnu',
+ variant='nan2008-soft',
+ gcc_cfg=['--with-mips-plt', '--with-nan=2008',
+ '--with-arch-64=mips64r2',
+ '--with-arch-32=mips32r2',
+ '--with-float=soft'],
+ glibcs=[{'variant': 'n32-nan2008-soft',
+ 'cfg': ['--without-fp']},
+ {'variant': 'nan2008-soft',
+ 'arch': 'mipsel',
+ 'ccopts': '-mabi=32',
+ 'cfg': ['--without-fp']},
+ {'variant': 'n64-nan2008-soft',
+ 'ccopts': '-mabi=64',
+ 'cfg': ['--without-fp']}])
+ self.add_config(arch='nios2',
+ os_name='linux-gnu')
+ self.add_config(arch='powerpc',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt'])
+ self.add_config(arch='powerpc',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--disable-multilib', '--with-float=soft',
+ '--enable-secureplt'],
+ glibcs=[{'variant': 'soft', 'cfg': ['--without-fp']}])
+ self.add_config(arch='powerpc64',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt'])
+ self.add_config(arch='powerpc64le',
+ os_name='linux-gnu',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt'])
+ self.add_config(arch='powerpc',
+ os_name='linux-gnuspe',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt',
+ '--enable-e500-double'],
+ glibcs=[{'cfg': ['--without-fp']}])
+ self.add_config(arch='powerpc',
+ os_name='linux-gnuspe',
+ variant='e500v1',
+ gcc_cfg=['--disable-multilib', '--enable-secureplt'],
+ glibcs=[{'variant': 'e500v1', 'cfg': ['--without-fp']}])
+ self.add_config(arch='s390x',
+ os_name='linux-gnu',
+ glibcs=[{},
+ {'arch': 's390', 'ccopts': '-m31'}])
+ # SH is missing __builtin_trap support, so work around this;
+ # see <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70216>.
+ no_isolate = ('-fno-isolate-erroneous-paths-dereference'
+ ' -fno-isolate-erroneous-paths-attribute')
+ self.add_config(arch='sh3',
+ os_name='linux-gnu',
+ glibcs=[{'ccopts': no_isolate}])
+ self.add_config(arch='sh3eb',
+ os_name='linux-gnu',
+ glibcs=[{'ccopts': no_isolate}])
+ self.add_config(arch='sh4',
+ os_name='linux-gnu',
+ glibcs=[{'ccopts': no_isolate}])
+ self.add_config(arch='sh4eb',
+ os_name='linux-gnu',
+ glibcs=[{'ccopts': no_isolate}])
+ self.add_config(arch='sh4',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--without-fp'],
+ glibcs=[{'variant': 'soft',
+ 'cfg': ['--without-fp'],
+ 'ccopts': no_isolate}])
+ self.add_config(arch='sh4eb',
+ os_name='linux-gnu',
+ variant='soft',
+ gcc_cfg=['--without-fp'],
+ glibcs=[{'variant': 'soft',
+ 'cfg': ['--without-fp'],
+ 'ccopts': no_isolate}])
+ self.add_config(arch='sparc64',
+ os_name='linux-gnu',
+ glibcs=[{},
+ {'arch': 'sparcv9',
+ 'ccopts': '-m32 -mlong-double-128'}])
+ self.add_config(arch='tilegx',
+ os_name='linux-gnu',
+ glibcs=[{},
+ {'variant': '32', 'ccopts': '-m32'}])
+ self.add_config(arch='tilegxbe',
+ os_name='linux-gnu',
+ glibcs=[{},
+ {'variant': '32', 'ccopts': '-m32'}])
+ self.add_config(arch='tilepro',
+ os_name='linux-gnu')
+ self.add_config(arch='x86_64',
+ os_name='linux-gnu',
+ gcc_cfg=['--with-multilib-list=m64,m32,mx32'],
+ glibcs=[{},
+ {'variant': 'x32', 'ccopts': '-mx32'},
+ {'arch': 'i686', 'ccopts': '-m32 -march=i686'}],
+ extra_glibcs=[{'variant': 'disable-multi-arch',
+ 'cfg': ['--disable-multi-arch']},
+ {'variant': 'disable-multi-arch',
+ 'arch': 'i686',
+ 'ccopts': '-m32 -march=i686',
+ 'cfg': ['--disable-multi-arch']},
+ {'arch': 'i486',
+ 'ccopts': '-m32 -march=i486'},
+ {'arch': 'i586',
+ 'ccopts': '-m32 -march=i586'}])
+
+ def add_config(self, **args):
+ """Add an individual build configuration."""
+ cfg = Config(self, **args)
+ if cfg.name in self.configs:
+ print('error: duplicate config %s' % cfg.name)
+ exit(1)
+ self.configs[cfg.name] = cfg
+ for c in cfg.all_glibcs:
+ if c.name in self.glibc_configs:
+ print('error: duplicate glibc config %s' % c.name)
+ exit(1)
+ self.glibc_configs[c.name] = c
+
+ def component_srcdir(self, component):
+ """Return the source directory for a given component, e.g. gcc."""
+ return os.path.join(self.srcdir, component)
+
+ def component_builddir(self, action, config, component, subconfig=None):
+ """Return the directory to use for a build."""
+ if config is None:
+ # Host libraries.
+ assert subconfig is None
+ return os.path.join(self.builddir, action, component)
+ if subconfig is None:
+ return os.path.join(self.builddir, action, config, component)
+ else:
+ # glibc build as part of compiler build.
+ return os.path.join(self.builddir, action, config, component,
+ subconfig)
+
+ def compiler_installdir(self, config):
+ """Return the directory in which to install a compiler."""
+ return os.path.join(self.installdir, 'compilers', config)
+
+ def compiler_bindir(self, config):
+ """Return the directory in which to find compiler binaries."""
+ return os.path.join(self.compiler_installdir(config), 'bin')
+
+ def compiler_sysroot(self, config):
+ """Return the sysroot directory for a compiler."""
+ return os.path.join(self.compiler_installdir(config), 'sysroot')
+
+ def glibc_installdir(self, config):
+ """Return the directory in which to install glibc."""
+ return os.path.join(self.installdir, 'glibcs', config)
+
+ def run_builds(self, action, configs):
+ """Run the requested builds."""
+ if action == 'checkout':
+ self.checkout(configs)
+ return
+ elif action == 'host-libraries':
+ if configs:
+ print('error: configurations specified for host-libraries')
+ exit(1)
+ self.build_host_libraries()
+ elif action == 'compilers':
+ self.build_compilers(configs)
+ else:
+ self.build_glibcs(configs)
+ self.write_files()
+ self.do_build()
+
+ @staticmethod
+ def remove_dirs(*args):
+ """Remove directories and their contents if they exist."""
+ for dir in args:
+ shutil.rmtree(dir, ignore_errors=True)
+
+ @staticmethod
+ def remove_recreate_dirs(*args):
+ """Remove directories if they exist, and create them as empty."""
+ Context.remove_dirs(*args)
+ for dir in args:
+ os.makedirs(dir, exist_ok=True)
+
+ def add_makefile_cmdlist(self, target, cmdlist, logsdir):
+ """Add makefile text for a list of commands."""
+ commands = cmdlist.makefile_commands(self.wrapper, logsdir)
+ self.makefile_pieces.append('all: %s\n.PHONY: %s\n%s:\n%s\n' %
+ (target, target, target, commands))
+
+ def write_files(self):
+ """Write out the Makefile and wrapper script."""
+ mftext = ''.join(self.makefile_pieces)
+ with open(self.makefile, 'w') as f:
+ f.write(mftext)
+ wrapper_text = (
+ '#!/bin/sh\n'
+ 'prev_base=$1\n'
+ 'this_base=$2\n'
+ 'desc=$3\n'
+ 'dir=$4\n'
+ 'path=$5\n'
+ 'shift 5\n'
+ 'prev_status=$prev_base-status.txt\n'
+ 'this_status=$this_base-status.txt\n'
+ 'this_log=$this_base-log.txt\n'
+ 'date > "$this_log"\n'
+ 'echo >> "$this_log"\n'
+ 'echo "Description: $desc" >> "$this_log"\n'
+ 'echo "Command: $*" >> "$this_log"\n'
+ 'echo "Directory: $dir" >> "$this_log"\n'
+ 'echo "Path addition: $path" >> "$this_log"\n'
+ 'echo >> "$this_log"\n'
+ 'record_status ()\n'
+ '{\n'
+ ' echo >> "$this_log"\n'
+ ' echo "$1: $desc" > "$this_status"\n'
+ ' echo "$1: $desc" >> "$this_log"\n'
+ ' echo >> "$this_log"\n'
+ ' date >> "$this_log"\n'
+ ' echo "$1: $desc"\n'
+ ' exit 0\n'
+ '}\n'
+ 'check_error ()\n'
+ '{\n'
+ ' if [ "$1" != "0" ]; then\n'
+ ' record_status FAIL\n'
+ ' fi\n'
+ '}\n'
+ 'if [ "$prev_base" ] && ! grep -q "^PASS" "$prev_status"; then\n'
+ ' record_status UNRESOLVED\n'
+ 'fi\n'
+ 'if [ "$dir" ]; then\n'
+ ' cd "$dir"\n'
+ ' check_error "$?"\n'
+ 'fi\n'
+ 'if [ "$path" ]; then\n'
+ ' PATH=$path:$PATH\n'
+ 'fi\n'
+ '"$@" < /dev/null >> "$this_log" 2>&1\n'
+ 'check_error "$?"\n'
+ 'record_status PASS\n')
+ with open(self.wrapper, 'w') as f:
+ f.write(wrapper_text)
+ os.chmod(self.wrapper,
+ (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
+ stat.S_IROTH|stat.S_IXOTH))
+ save_logs_text = (
+ '#!/bin/sh\n'
+ 'if ! [ -f tests.sum ]; then\n'
+ ' echo "No test summary available."\n'
+ ' exit 0\n'
+ 'fi\n'
+ 'save_file ()\n'
+ '{\n'
+ ' echo "Contents of $1:"\n'
+ ' echo\n'
+ ' cat "$1"\n'
+ ' echo\n'
+ ' echo "End of contents of $1."\n'
+ ' echo\n'
+ '}\n'
+ 'save_file tests.sum\n'
+ 'non_pass_tests=$(grep -v "^PASS: " tests.sum | sed -e "s/^PASS: //")\n'
+ 'for t in $non_pass_tests; do\n'
+ ' if [ -f "$t.out" ]; then\n'
+ ' save_file "$t.out"\n'
+ ' fi\n'
+ 'done\n')
+ with open(self.save_logs, 'w') as f:
+ f.write(save_logs_text)
+ os.chmod(self.save_logs,
+ (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
+ stat.S_IROTH|stat.S_IXOTH))
+
+ def do_build(self):
+ """Do the actual build."""
+ cmd = ['make', '-j%d' % self.parallelism]
+ subprocess.run(cmd, cwd=self.builddir, check=True)
+
+ def build_host_libraries(self):
+ """Build the host libraries."""
+ installdir = self.host_libraries_installdir
+ builddir = os.path.join(self.builddir, 'host-libraries')
+ logsdir = os.path.join(self.logsdir, 'host-libraries')
+ self.remove_recreate_dirs(installdir, builddir, logsdir)
+ cmdlist = CommandList('host-libraries', self.keep)
+ self.build_host_library(cmdlist, 'gmp')
+ self.build_host_library(cmdlist, 'mpfr',
+ ['--with-gmp=%s' % installdir])
+ self.build_host_library(cmdlist, 'mpc',
+ ['--with-gmp=%s' % installdir,
+ '--with-mpfr=%s' % installdir])
+ cmdlist.add_command('done', ['touch', os.path.join(installdir, 'ok')])
+ self.add_makefile_cmdlist('host-libraries', cmdlist, logsdir)
+
+ def build_host_library(self, cmdlist, lib, extra_opts=None):
+ """Build one host library."""
+ srcdir = self.component_srcdir(lib)
+ builddir = self.component_builddir('host-libraries', None, lib)
+ installdir = self.host_libraries_installdir
+ cmdlist.push_subdesc(lib)
+ cmdlist.create_use_dir(builddir)
+ cfg_cmd = [os.path.join(srcdir, 'configure'),
+ '--prefix=%s' % installdir,
+ '--disable-shared']
+ if extra_opts:
+ cfg_cmd.extend (extra_opts)
+ cmdlist.add_command('configure', cfg_cmd)
+ cmdlist.add_command('build', ['make'])
+ cmdlist.add_command('check', ['make', 'check'])
+ cmdlist.add_command('install', ['make', 'install'])
+ cmdlist.cleanup_dir()
+ cmdlist.pop_subdesc()
+
+ def build_compilers(self, configs):
+ """Build the compilers."""
+ if not configs:
+ self.remove_dirs(os.path.join(self.builddir, 'compilers'))
+ self.remove_dirs(os.path.join(self.installdir, 'compilers'))
+ self.remove_dirs(os.path.join(self.logsdir, 'compilers'))
+ configs = sorted(self.configs.keys())
+ for c in configs:
+ self.configs[c].build()
+
+ def build_glibcs(self, configs):
+ """Build the glibcs."""
+ if not configs:
+ self.remove_dirs(os.path.join(self.builddir, 'glibcs'))
+ self.remove_dirs(os.path.join(self.installdir, 'glibcs'))
+ self.remove_dirs(os.path.join(self.logsdir, 'glibcs'))
+ configs = sorted(self.glibc_configs.keys())
+ for c in configs:
+ self.glibc_configs[c].build()
+
+ def checkout(self, versions):
+ """Check out the desired component versions."""
+ default_versions = {'binutils': 'vcs-2.27',
+ 'gcc': 'vcs-6',
+ 'glibc': 'vcs-mainline',
+ 'gmp': '6.1.1',
+ 'linux': '4.8.6',
+ 'mpc': '1.0.3',
+ 'mpfr': '3.1.5'}
+ use_versions = {}
+ for v in versions:
+ found_v = False
+ for k in default_versions.keys():
+ kx = k + '-'
+ if v.startswith(kx):
+ vx = v[len(kx):]
+ if k in use_versions:
+ print('error: multiple versions for %s' % k)
+ exit(1)
+ use_versions[k] = vx
+ found_v = True
+ break
+ if not found_v:
+ print('error: unknown component in %s' % v)
+ exit(1)
+ for k in default_versions.keys():
+ if k not in use_versions:
+ use_versions[k] = default_versions[k]
+ os.makedirs(self.srcdir, exist_ok=True)
+ for k in sorted(default_versions.keys()):
+ update = os.access(self.component_srcdir(k), os.F_OK)
+ v = use_versions[k]
+ if v.startswith('vcs-'):
+ self.checkout_vcs(k, v[4:], update)
+ else:
+ self.checkout_tar(k, v, update)
+
+ def checkout_vcs(self, component, version, update):
+ """Check out the given version of the given component from version
+ control."""
+ if component == 'binutils':
+ git_url = 'git://sourceware.org/git/binutils-gdb.git'
+ if version == 'mainline':
+ git_branch = 'master'
+ else:
+ trans = str.maketrans({'.': '_'})
+ git_branch = 'binutils-%s-branch' % version.translate(trans)
+ self.git_checkout(component, git_url, git_branch, update)
+ elif component == 'gcc':
+ if version == 'mainline':
+ branch = 'trunk'
+ else:
+ trans = str.maketrans({'.': '_'})
+ branch = 'branches/gcc-%s-branch' % version.translate(trans)
+ svn_url = 'svn://gcc.gnu.org/svn/gcc/%s' % branch
+ self.gcc_checkout(svn_url, update)
+ elif component == 'glibc':
+ git_url = 'git://sourceware.org/git/glibc.git'
+ if version == 'mainline':
+ git_branch = 'master'
+ else:
+ git_branch = 'release/%s/master' % version
+ self.git_checkout(component, git_url, git_branch, update)
+ self.fix_glibc_timestamps()
+ else:
+ print('error: component %s coming from VCS' % component)
+ exit(1)
+
+ def git_checkout(self, component, git_url, git_branch, update):
+ """Check out a component from git."""
+ if update:
+ subprocess.run(['git', 'remote', 'prune', 'origin'],
+ cwd=self.component_srcdir(component), check=True)
+ subprocess.run(['git', 'pull', '-q'],
+ cwd=self.component_srcdir(component), check=True)
+ else:
+ subprocess.run(['git', 'clone', '-q', '-b', git_branch, git_url,
+ self.component_srcdir(component)], check=True)
+
+ def fix_glibc_timestamps(self):
+ """Fix timestamps in a glibc checkout."""
+ # Ensure that builds do not try to regenerate generated files
+ # in the source tree.
+ srcdir = self.component_srcdir('glibc')
+ for dirpath, dirnames, filenames in os.walk(srcdir):
+ for f in filenames:
+ if (f == 'configure' or
+ f == 'preconfigure' or
+ f.endswith('-kw.h')):
+ to_touch = os.path.join(dirpath, f)
+ subprocess.run(['touch', to_touch], check=True)
+
+ def gcc_checkout(self, svn_url, update):
+ """Check out GCC from SVN."""
+ if not update:
+ subprocess.run(['svn', 'co', '-q', svn_url,
+ self.component_srcdir('gcc')], check=True)
+ subprocess.run(['contrib/gcc_update', '--silent'],
+ cwd=self.component_srcdir('gcc'), check=True)
+
+ def checkout_tar(self, component, version, update):
+ """Check out the given version of the given component from a
+ tarball."""
+ if update:
+ return
+ url_map = {'binutils': 'https://ftp.gnu.org/gnu/binutils/binutils-%(version)s.tar.bz2',
+ 'gcc': 'https://ftp.gnu.org/gnu/gcc/gcc-%(version)s/gcc-%(version)s.tar.bz2',
+ 'gmp': 'https://ftp.gnu.org/gnu/gmp/gmp-%(version)s.tar.xz',
+ 'linux': 'https://www.kernel.org/pub/linux/kernel/v4.x/linux-%(version)s.tar.xz',
+ 'mpc': 'https://ftp.gnu.org/gnu/mpc/mpc-%(version)s.tar.gz',
+ 'mpfr': 'https://ftp.gnu.org/gnu/mpfr/mpfr-%(version)s.tar.xz'}
+ if component not in url_map:
+ print('error: component %s coming from tarball' % component)
+ exit(1)
+ url = url_map[component] % {'version': version}
+ filename = os.path.join(self.srcdir, url.split('/')[-1])
+ response = urllib.request.urlopen(url)
+ data = response.read()
+ with open(filename, 'wb') as f:
+ f.write(data)
+ subprocess.run(['tar', '-C', self.srcdir, '-x', '-f', filename],
+ check=True)
+ os.rename(os.path.join(self.srcdir, '%s-%s' % (component, version)),
+ self.component_srcdir(component))
+ os.remove(filename)
+
+
+class Config:
+
+ """A configuration for building a compiler and associated libraries."""
+
+ def __init__(self, ctx, arch, os_name, variant=None, gcc_cfg=None,
+ first_gcc_cfg=None, glibcs=None, extra_glibcs=None):
+ """Initialize a Config object."""
+ self.ctx = ctx
+ self.arch = arch
+ self.os = os_name
+ self.variant = variant
+ if variant is None:
+ self.name = '%s-%s' % (arch, os_name)
+ else:
+ self.name = '%s-%s-%s' % (arch, os_name, variant)
+ self.triplet = '%s-glibc-%s' % (arch, os_name)
+ if gcc_cfg is None:
+ self.gcc_cfg = []
+ else:
+ self.gcc_cfg = gcc_cfg
+ if first_gcc_cfg is None:
+ self.first_gcc_cfg = []
+ else:
+ self.first_gcc_cfg = first_gcc_cfg
+ if glibcs is None:
+ glibcs = [{'variant': variant}]
+ if extra_glibcs is None:
+ extra_glibcs = []
+ glibcs = [Glibc(self, **g) for g in glibcs]
+ extra_glibcs = [Glibc(self, **g) for g in extra_glibcs]
+ self.all_glibcs = glibcs + extra_glibcs
+ self.compiler_glibcs = glibcs
+ self.installdir = ctx.compiler_installdir(self.name)
+ self.bindir = ctx.compiler_bindir(self.name)
+ self.sysroot = ctx.compiler_sysroot(self.name)
+ self.builddir = os.path.join(ctx.builddir, 'compilers', self.name)
+ self.logsdir = os.path.join(ctx.logsdir, 'compilers', self.name)
+
+ def component_builddir(self, component):
+ """Return the directory to use for a (non-glibc) build."""
+ return self.ctx.component_builddir('compilers', self.name, component)
+
+ def build(self):
+ """Generate commands to build this compiler."""
+ self.ctx.remove_recreate_dirs(self.installdir, self.builddir,
+ self.logsdir)
+ cmdlist = CommandList('compilers-%s' % self.name, self.ctx.keep)
+ cmdlist.add_command('check-host-libraries',
+ ['test', '-f',
+ os.path.join(self.ctx.host_libraries_installdir,
+ 'ok')])
+ cmdlist.use_path(self.bindir)
+ self.build_cross_tool(cmdlist, 'binutils', 'binutils',
+ ['--disable-gdb',
+ '--disable-libdecnumber',
+ '--disable-readline',
+ '--disable-sim'])
+ if self.os.startswith('linux'):
+ self.install_linux_headers(cmdlist)
+ self.build_gcc(cmdlist, True)
+ for g in self.compiler_glibcs:
+ cmdlist.push_subdesc('glibc')
+ cmdlist.push_subdesc(g.name)
+ g.build_glibc(cmdlist, True)
+ cmdlist.pop_subdesc()
+ cmdlist.pop_subdesc()
+ self.build_gcc(cmdlist, False)
+ cmdlist.add_command('done', ['touch',
+ os.path.join(self.installdir, 'ok')])
+ self.ctx.add_makefile_cmdlist('compilers-%s' % self.name, cmdlist,
+ self.logsdir)
+
+ def build_cross_tool(self, cmdlist, tool_src, tool_build, extra_opts=None):
+ """Build one cross tool."""
+ srcdir = self.ctx.component_srcdir(tool_src)
+ builddir = self.component_builddir(tool_build)
+ cmdlist.push_subdesc(tool_build)
+ cmdlist.create_use_dir(builddir)
+ cfg_cmd = [os.path.join(srcdir, 'configure'),
+ '--prefix=%s' % self.installdir,
+ '--build=%s' % self.ctx.build_triplet,
+ '--host=%s' % self.ctx.build_triplet,
+ '--target=%s' % self.triplet,
+ '--with-sysroot=%s' % self.sysroot]
+ if extra_opts:
+ cfg_cmd.extend(extra_opts)
+ cmdlist.add_command('configure', cfg_cmd)
+ cmdlist.add_command('build', ['make'])
+ cmdlist.add_command('install', ['make', 'install'])
+ cmdlist.cleanup_dir()
+ cmdlist.pop_subdesc()
+
+ def install_linux_headers(self, cmdlist):
+ """Install Linux kernel headers."""
+ arch_map = {'aarch64': 'arm64',
+ 'alpha': 'alpha',
+ 'arm': 'arm',
+ 'hppa': 'parisc',
+ 'i486': 'x86',
+ 'i586': 'x86',
+ 'i686': 'x86',
+ 'i786': 'x86',
+ 'ia64': 'ia64',
+ 'm68k': 'm68k',
+ 'microblaze': 'microblaze',
+ 'mips': 'mips',
+ 'nios2': 'nios2',
+ 'powerpc': 'powerpc',
+ 's390': 's390',
+ 'sh': 'sh',
+ 'sparc': 'sparc',
+ 'tile': 'tile',
+ 'x86_64': 'x86'}
+ linux_arch = None
+ for k in arch_map:
+ if self.arch.startswith(k):
+ linux_arch = arch_map[k]
+ break
+ assert linux_arch is not None
+ srcdir = self.ctx.component_srcdir('linux')
+ builddir = self.component_builddir('linux')
+ headers_dir = os.path.join(self.sysroot, 'usr')
+ cmdlist.push_subdesc('linux')
+ cmdlist.create_use_dir(builddir)
+ cmdlist.add_command('install-headers',
+ ['make', '-C', srcdir, 'O=%s' % builddir,
+ 'ARCH=%s' % linux_arch,
+ 'INSTALL_HDR_PATH=%s' % headers_dir,
+ 'headers_install'])
+ cmdlist.cleanup_dir()
+ cmdlist.pop_subdesc()
+
+ def build_gcc(self, cmdlist, bootstrap):
+ """Build GCC."""
+ # libsanitizer commonly breaks because of glibc header
+ # changes, or on unusual targets. libssp is of little
+ # relevance with glibc's own stack checking support.
+ cfg_opts = list(self.gcc_cfg)
+ cfg_opts += ['--disable-libsanitizer', '--disable-libssp']
+ if bootstrap:
+ tool_build = 'gcc-first'
+ # Building a static-only, C-only compiler that is
+ # sufficient to build glibc. Various libraries and
+ # features that may require libc headers must be disabled.
+ # When configuring with a sysroot, --with-newlib is
+ # required to define inhibit_libc (to stop some parts of
+ # libgcc including libc headers); --without-headers is not
+ # sufficient.
+ cfg_opts += ['--enable-languages=c', '--disable-shared',
+ '--disable-threads',
+ '--disable-libatomic',
+ '--disable-decimal-float',
+ '--disable-libffi',
+ '--disable-libgomp',
+ '--disable-libitm',
+ '--disable-libmpx',
+ '--disable-libquadmath',
+ '--without-headers', '--with-newlib',
+ '--with-glibc-version=%s' % self.ctx.glibc_version
+ ]
+ cfg_opts += self.first_gcc_cfg
+ else:
+ tool_build = 'gcc'
+ cfg_opts += ['--enable-languages=c,c++', '--enable-shared',
+ '--enable-threads']
+ self.build_cross_tool(cmdlist, 'gcc', tool_build, cfg_opts)
+
+
+class Glibc:
+
+ """A configuration for building glibc."""
+
+ def __init__(self, compiler, arch=None, os_name=None, variant=None,
+ cfg=None, ccopts=None):
+ """Initialize a Glibc object."""
+ self.ctx = compiler.ctx
+ self.compiler = compiler
+ if arch is None:
+ self.arch = compiler.arch
+ else:
+ self.arch = arch
+ if os_name is None:
+ self.os = compiler.os
+ else:
+ self.os = os_name
+ self.variant = variant
+ if variant is None:
+ self.name = '%s-%s' % (self.arch, self.os)
+ else:
+ self.name = '%s-%s-%s' % (self.arch, self.os, variant)
+ self.triplet = '%s-glibc-%s' % (self.arch, self.os)
+ if cfg is None:
+ self.cfg = []
+ else:
+ self.cfg = cfg
+ self.ccopts = ccopts
+
+ def tool_name(self, tool):
+ """Return the name of a cross-compilation tool."""
+ ctool = '%s-%s' % (self.compiler.triplet, tool)
+ if self.ccopts and (tool == 'gcc' or tool == 'g++'):
+ ctool = '%s %s' % (ctool, self.ccopts)
+ return ctool
+
+ def build(self):
+ """Generate commands to build this glibc."""
+ builddir = self.ctx.component_builddir('glibcs', self.name, 'glibc')
+ installdir = self.ctx.glibc_installdir(self.name)
+ logsdir = os.path.join(self.ctx.logsdir, 'glibcs', self.name)
+ self.ctx.remove_recreate_dirs(installdir, builddir, logsdir)
+ cmdlist = CommandList('glibcs-%s' % self.name, self.ctx.keep)
+ cmdlist.add_command('check-compilers',
+ ['test', '-f',
+ os.path.join(self.compiler.installdir, 'ok')])
+ cmdlist.use_path(self.compiler.bindir)
+ self.build_glibc(cmdlist, False)
+ self.ctx.add_makefile_cmdlist('glibcs-%s' % self.name, cmdlist,
+ logsdir)
+
+ def build_glibc(self, cmdlist, for_compiler):
+ """Generate commands to build this glibc, either as part of a compiler
+ build or with the bootstrapped compiler (and in the latter case, run
+ tests as well)."""
+ srcdir = self.ctx.component_srcdir('glibc')
+ if for_compiler:
+ builddir = self.ctx.component_builddir('compilers',
+ self.compiler.name, 'glibc',
+ self.name)
+ installdir = self.compiler.sysroot
+ srcdir_copy = self.ctx.component_builddir('compilers',
+ self.compiler.name,
+ 'glibc-src',
+ self.name)
+ else:
+ builddir = self.ctx.component_builddir('glibcs', self.name,
+ 'glibc')
+ installdir = self.ctx.glibc_installdir(self.name)
+ srcdir_copy = self.ctx.component_builddir('glibcs', self.name,
+ 'glibc-src')
+ cmdlist.create_use_dir(builddir)
+ # glibc builds write into the source directory, and even if
+ # not intentionally there is a risk of bugs that involve
+ # writing into the working directory. To avoid possible
+ # concurrency issues, copy the source directory.
+ cmdlist.create_copy_dir(srcdir, srcdir_copy)
+ cfg_cmd = [os.path.join(srcdir_copy, 'configure'),
+ '--prefix=/usr',
+ '--enable-add-ons',
+ '--build=%s' % self.ctx.build_triplet,
+ '--host=%s' % self.triplet,
+ 'CC=%s' % self.tool_name('gcc'),
+ 'CXX=%s' % self.tool_name('g++'),
+ 'AR=%s' % self.tool_name('ar'),
+ 'AS=%s' % self.tool_name('as'),
+ 'LD=%s' % self.tool_name('ld'),
+ 'NM=%s' % self.tool_name('nm'),
+ 'OBJCOPY=%s' % self.tool_name('objcopy'),
+ 'OBJDUMP=%s' % self.tool_name('objdump'),
+ 'RANLIB=%s' % self.tool_name('ranlib'),
+ 'READELF=%s' % self.tool_name('readelf'),
+ 'STRIP=%s' % self.tool_name('strip')]
+ cfg_cmd += self.cfg
+ cmdlist.add_command('configure', cfg_cmd)
+ cmdlist.add_command('build', ['make'])
+ cmdlist.add_command('install', ['make', 'install',
+ 'install_root=%s' % installdir])
+ # GCC uses paths such as lib/../lib64, so make sure lib
+ # directories always exist.
+ cmdlist.add_command('mkdir-lib', ['mkdir', '-p',
+ os.path.join(installdir, 'lib'),
+ os.path.join(installdir,
+ 'usr', 'lib')])
+ if not for_compiler:
+ cmdlist.add_command('check', ['make', 'check'])
+ cmdlist.add_command('save-logs', [self.ctx.save_logs],
+ always_run=True)
+ cmdlist.cleanup_dir('cleanup-src', srcdir_copy)
+ cmdlist.cleanup_dir()
+
+
+class Command:
+
+ """A command run in the build process."""
+
+ def __init__(self, desc, num, dir, path, command, always_run=False):
+ """Initialize a Command object."""
+ self.dir = dir
+ self.path = path
+ self.desc = desc
+ trans = str.maketrans({' ': '-'})
+ self.logbase = '%03d-%s' % (num, desc.translate(trans))
+ self.command = command
+ self.always_run = always_run
+
+ @staticmethod
+ def shell_make_quote_string(s):
+ """Given a string not containing a newline, quote it for use by the
+ shell and make."""
+ assert '\n' not in s
+ if re.fullmatch('[]+,./0-9@A-Z_a-z-]+', s):
+ return s
+ strans = str.maketrans({"'": "'\\''"})
+ s = "'%s'" % s.translate(strans)
+ mtrans = str.maketrans({'$': '$$'})
+ return s.translate(mtrans)
+
+ @staticmethod
+ def shell_make_quote_list(l, translate_make):
+ """Given a list of strings not containing newlines, quote them for use
+ by the shell and make, returning a single string. If translate_make
+ is true and the first string is 'make', change it to $(MAKE)."""
+ l = [Command.shell_make_quote_string(s) for s in l]
+ if translate_make and l[0] == 'make':
+ l[0] = '$(MAKE)'
+ return ' '.join(l)
+
+ def shell_make_quote(self):
+ """Return this command quoted for the shell and make."""
+ return self.shell_make_quote_list(self.command, True)
+
+
+class CommandList:
+
+ """A list of commands run in the build process."""
+
+ def __init__(self, desc, keep):
+ """Initialize a CommandList object."""
+ self.cmdlist = []
+ self.dir = None
+ self.path = None
+ self.desc = [desc]
+ self.keep = keep
+
+ def desc_txt(self, desc):
+ """Return the description to use for a command."""
+ return '%s %s' % (' '.join(self.desc), desc)
+
+ def use_dir(self, dir):
+ """Set the default directory for subsequent commands."""
+ self.dir = dir
+
+ def use_path(self, path):
+ """Set a directory to be prepended to the PATH for subsequent
+ commands."""
+ self.path = path
+
+ def push_subdesc(self, subdesc):
+ """Set the default subdescription for subsequent commands (e.g., the
+ name of a component being built, within the series of commands
+ building it)."""
+ self.desc.append(subdesc)
+
+ def pop_subdesc(self):
+ """Pop a subdescription from the list of descriptions."""
+ self.desc.pop()
+
+ def create_use_dir(self, dir):
+ """Remove and recreate a directory and use it for subsequent
+ commands."""
+ self.add_command_dir('rm', None, ['rm', '-rf', dir])
+ self.add_command_dir('mkdir', None, ['mkdir', '-p', dir])
+ self.use_dir(dir)
+
+ def create_copy_dir(self, src, dest):
+ """Remove a directory and recreate it as a copy from the given
+ source."""
+ self.add_command_dir('copy-rm', None, ['rm', '-rf', dest])
+ parent = os.path.dirname(dest)
+ self.add_command_dir('copy-mkdir', None, ['mkdir', '-p', parent])
+ self.add_command_dir('copy', None, ['cp', '-a', src, dest])
+
+ def add_command_dir(self, desc, dir, command, always_run=False):
+ """Add a command to run in a given directory."""
+ cmd = Command(self.desc_txt(desc), len(self.cmdlist), dir, self.path,
+ command, always_run)
+ self.cmdlist.append(cmd)
+
+ def add_command(self, desc, command, always_run=False):
+ """Add a command to run in the default directory."""
+ cmd = Command(self.desc_txt(desc), len(self.cmdlist), self.dir,
+ self.path, command, always_run)
+ self.cmdlist.append(cmd)
+
+ def cleanup_dir(self, desc='cleanup', dir=None):
+ """Clean up a build directory. If no directory is specified, the
+ default directory is cleaned up and ceases to be the default
+ directory."""
+ if dir is None:
+ dir = self.dir
+ self.use_dir(None)
+ if self.keep != 'all':
+ self.add_command_dir(desc, None, ['rm', '-rf', dir],
+ always_run=(self.keep == 'none'))
+
+ def makefile_commands(self, wrapper, logsdir):
+ """Return the sequence of commands in the form of text for a Makefile.
+ The given wrapper script takes arguments: base of logs for
+ previous command, or empty; base of logs for this command;
+ description; directory; PATH addition; the command itself."""
+ # prev_base is the base of the name for logs of the previous
+ # command that is not always-run (that is, a build command,
+ # whose failure should stop subsequent build commands from
+ # being run, as opposed to a cleanup command, which is run
+ # even if previous commands failed).
+ prev_base = ''
+ cmds = []
+ for c in self.cmdlist:
+ ctxt = c.shell_make_quote()
+ if prev_base and not c.always_run:
+ prev_log = os.path.join(logsdir, prev_base)
+ else:
+ prev_log = ''
+ this_log = os.path.join(logsdir, c.logbase)
+ if not c.always_run:
+ prev_base = c.logbase
+ if c.dir is None:
+ dir = ''
+ else:
+ dir = c.dir
+ if c.path is None:
+ path = ''
+ else:
+ path = c.path
+ prelims = [wrapper, prev_log, this_log, c.desc, dir, path]
+ prelim_txt = Command.shell_make_quote_list(prelims, False)
+ cmds.append('\t@%s %s' % (prelim_txt, ctxt))
+ return '\n'.join(cmds)
+
+
+def get_parser():
+ """Return an argument parser for this module."""
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument('-j', dest='parallelism',
+ help='Run this number of jobs in parallel',
+ type=int, default=os.cpu_count())
+ parser.add_argument('--keep', dest='keep',
+ help='Whether to keep all build directories, '
+ 'none or only those from failed builds',
+ default='none', choices=('none', 'all', 'failed'))
+ parser.add_argument('topdir',
+ help='Toplevel working directory')
+ parser.add_argument('action',
+ help='What to do',
+ choices=('checkout', 'host-libraries', 'compilers',
+ 'glibcs'))
+ parser.add_argument('configs',
+ help='Versions to check out or configurations to build',
+ nargs='*')
+ return parser
+
+
+def main(argv):
+ """The main entry point."""
+ parser = get_parser()
+ opts = parser.parse_args(argv)
+ topdir = os.path.abspath(opts.topdir)
+ ctx = Context(topdir, opts.parallelism, opts.keep, opts.action)
+ ctx.run_builds(opts.action, opts.configs)
+
+
+if __name__ == '__main__':
+ main(sys.argv[1:])
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-11 15:20 ` Joseph Myers
@ 2016-11-11 19:37 ` Carlos O'Donell
2016-11-14 15:06 ` Mike Frysinger
1 sibling, 0 replies; 21+ messages in thread
From: Carlos O'Donell @ 2016-11-11 19:37 UTC (permalink / raw)
To: Joseph Myers, libc-alpha
On 11/11/2016 10:20 AM, Joseph Myers wrote:
> Here is an updated version of this script with some support for updating
> existing checkouts.
>
> Add script to build many glibc configurations.
> 2016-11-11 Joseph Myers <joseph@codesourcery.com>
>
> * scripts/build-many-glibcs.py: New file.
>
This looks great, and much better than what we have. If we do anything
else it should start with this script as the basis for discussion.
Please feel free to check this in.
--
Cheers,
Carlos.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-11 15:20 ` Joseph Myers
2016-11-11 19:37 ` Carlos O'Donell
@ 2016-11-14 15:06 ` Mike Frysinger
2016-11-14 15:23 ` Joseph Myers
2016-11-14 23:57 ` Joseph Myers
1 sibling, 2 replies; 21+ messages in thread
From: Mike Frysinger @ 2016-11-14 15:06 UTC (permalink / raw)
To: Joseph Myers; +Cc: libc-alpha
[-- Attachment #1: Type: text/plain, Size: 600 bytes --]
On 11 Nov 2016 15:20, Joseph Myers wrote:
> +import os.path
this import is unnecessary
> +class Context:
please no old style classes. all of them should inherit object.
class Context(object):
> +
> + """The global state associated with builds in a given directory."""
don't put a blank line between the class and its docstring.
> + os.chmod(self.wrapper,
> + (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
> + stat.S_IROTH|stat.S_IXOTH))
the stat constants are unreadable imo. better to never use them and
stick to sane octals like 0644 and such.
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-14 15:06 ` Mike Frysinger
@ 2016-11-14 15:23 ` Joseph Myers
2016-11-14 19:27 ` Mike Frysinger
2016-11-14 23:57 ` Joseph Myers
1 sibling, 1 reply; 21+ messages in thread
From: Joseph Myers @ 2016-11-14 15:23 UTC (permalink / raw)
To: Mike Frysinger; +Cc: libc-alpha
On Mon, 14 Nov 2016, Mike Frysinger wrote:
> > +class Context:
>
> please no old style classes. all of them should inherit object.
> class Context(object):
There's no such thing as an old-style class in Python 3 (and this script
requires 3.5 or later); inheritance from object happens by default.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-14 15:23 ` Joseph Myers
@ 2016-11-14 19:27 ` Mike Frysinger
2016-11-14 23:28 ` Joseph Myers
0 siblings, 1 reply; 21+ messages in thread
From: Mike Frysinger @ 2016-11-14 19:27 UTC (permalink / raw)
To: Joseph Myers; +Cc: libc-alpha
[-- Attachment #1: Type: text/plain, Size: 482 bytes --]
On 14 Nov 2016 15:22, Joseph Myers wrote:
> On Mon, 14 Nov 2016, Mike Frysinger wrote:
> > > +class Context:
> >
> > please no old style classes. all of them should inherit object.
> > class Context(object):
>
> There's no such thing as an old-style class in Python 3 (and this script
> requires 3.5 or later); inheritance from object happens by default.
the py3 behavior is irrelevant to the style choice: the object inherit
should be explicit regardless.
-mike
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-14 19:27 ` Mike Frysinger
@ 2016-11-14 23:28 ` Joseph Myers
0 siblings, 0 replies; 21+ messages in thread
From: Joseph Myers @ 2016-11-14 23:28 UTC (permalink / raw)
To: Mike Frysinger; +Cc: libc-alpha
On Mon, 14 Nov 2016, Mike Frysinger wrote:
> On 14 Nov 2016 15:22, Joseph Myers wrote:
> > On Mon, 14 Nov 2016, Mike Frysinger wrote:
> > > > +class Context:
> > >
> > > please no old style classes. all of them should inherit object.
> > > class Context(object):
> >
> > There's no such thing as an old-style class in Python 3 (and this script
> > requires 3.5 or later); inheritance from object happens by default.
>
> the py3 behavior is irrelevant to the style choice: the object inherit
> should be explicit regardless.
Is there some style guide we're following that recommends this for Python
3 code? I've only seen it mentioned as something to do when you're
writing code that's meant to work the same in Python 2 and Python 3, not
for code unconcerned with Python 2 compatibility.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-14 15:06 ` Mike Frysinger
2016-11-14 15:23 ` Joseph Myers
@ 2016-11-14 23:57 ` Joseph Myers
1 sibling, 0 replies; 21+ messages in thread
From: Joseph Myers @ 2016-11-14 23:57 UTC (permalink / raw)
To: Mike Frysinger; +Cc: libc-alpha
On Mon, 14 Nov 2016, Mike Frysinger wrote:
> On 11 Nov 2016 15:20, Joseph Myers wrote:
> > +import os.path
>
> this import is unnecessary
Removed in this patch.
> > +class Context:
>
> please no old style classes. all of them should inherit object.
> class Context(object):
Changed in this patch, though I've only ever seen this mentioned as a
Python 2 compatibility issue not as something relevant to Python 3 code
(otherwise I'd have expected Python 3 to remove the option not to specify
a class to inherit from).
> > + """The global state associated with builds in a given directory."""
>
> don't put a blank line between the class and its docstring.
Changed in this patch.
> > + os.chmod(self.wrapper,
> > + (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
> > + stat.S_IROTH|stat.S_IXOTH))
>
> the stat constants are unreadable imo. better to never use them and
> stick to sane octals like 0644 and such.
This patch puts the mode in a variable with a comment giving the 0o755
value (being Python 3, 0o is the notation for octal constants). While I
agree on the unreadability, hardcoding such values rather than using the
provided os interface seems inappropriate in Python code (in the absence
of Python defining the conventional values of those constants as being an
OS-independent interface).
Committed.
2016-11-14 Joseph Myers <joseph@codesourcery.com>
* scripts/build-many-glibcs.py (os.path): Do not import.
(Context): Inherit explicitly from object. Remove blank line
between class and docstring.
(Config): Likewise.
(Glibc): Likewise.
(Command): Likewise.
(CommandList): Likewise.
(Context.write_files): Store chmod mode in a variable.
diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
index 4b4e15c..be561c3 100755
--- a/scripts/build-many-glibcs.py
+++ b/scripts/build-many-glibcs.py
@@ -33,7 +33,6 @@ configurations for which compilers or glibc are to be built.
import argparse
import os
-import os.path
import re
import shutil
import stat
@@ -42,8 +41,7 @@ import sys
import urllib.request
-class Context:
-
+class Context(object):
"""The global state associated with builds in a given directory."""
def __init__(self, topdir, parallelism, keep, action):
@@ -460,9 +458,10 @@ class Context:
'record_status PASS\n')
with open(self.wrapper, 'w') as f:
f.write(wrapper_text)
- os.chmod(self.wrapper,
- (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
- stat.S_IROTH|stat.S_IXOTH))
+ # Mode 0o755.
+ mode_exec = (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
+ stat.S_IROTH|stat.S_IXOTH)
+ os.chmod(self.wrapper, mode_exec)
save_logs_text = (
'#!/bin/sh\n'
'if ! [ -f tests.sum ]; then\n'
@@ -487,9 +486,7 @@ class Context:
'done\n')
with open(self.save_logs, 'w') as f:
f.write(save_logs_text)
- os.chmod(self.save_logs,
- (stat.S_IRWXU|stat.S_IRGRP|stat.S_IXGRP|
- stat.S_IROTH|stat.S_IXOTH))
+ os.chmod(self.save_logs, mode_exec)
def do_build(self):
"""Do the actual build."""
@@ -678,8 +675,7 @@ class Context:
os.remove(filename)
-class Config:
-
+class Config(object):
"""A configuration for building a compiler and associated libraries."""
def __init__(self, ctx, arch, os_name, variant=None, gcc_cfg=None,
@@ -846,8 +842,7 @@ class Config:
self.build_cross_tool(cmdlist, 'gcc', tool_build, cfg_opts)
-class Glibc:
-
+class Glibc(object):
"""A configuration for building glibc."""
def __init__(self, compiler, arch=None, os_name=None, variant=None,
@@ -958,8 +953,7 @@ class Glibc:
cmdlist.cleanup_dir()
-class Command:
-
+class Command(object):
"""A command run in the build process."""
def __init__(self, desc, num, dir, path, command, always_run=False):
@@ -999,8 +993,7 @@ class Command:
return self.shell_make_quote_list(self.command, True)
-class CommandList:
-
+class CommandList(object):
"""A list of commands run in the build process."""
def __init__(self, desc, keep):
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-09 16:27 Add script to build many glibc configurations Joseph Myers
` (2 preceding siblings ...)
2016-11-11 15:20 ` Joseph Myers
@ 2016-11-17 16:52 ` Zack Weinberg
2016-11-17 17:26 ` Zack Weinberg
2016-11-17 17:51 ` Joseph Myers
3 siblings, 2 replies; 21+ messages in thread
From: Zack Weinberg @ 2016-11-17 16:52 UTC (permalink / raw)
To: libc-alpha
On 11/09/2016 11:27 AM, Joseph Myers wrote:
> This patch adds a Python (3.5 or later) script to build many different
> configurations of glibc, including building the required cross
> compilers first. It's not intended to change any patch testing
> requirements, although some people may wish to use it for high-risk
> patches such as adding warning options ...
Since this does its own glibc checkout, it's not clear to me how one
should use it to test a patch(set). I presume that whatever one does,
it only affects the "glibcs" step, but what actually do you do? Do you
manually update /some/where/src/glibc to contain the code you want
tested and then run "glibcs", or do you somehow tell
build-many-glibcs.py the name of a branch you want tested, or what?
Thanks,
zw
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-17 16:52 ` Zack Weinberg
@ 2016-11-17 17:26 ` Zack Weinberg
2016-11-17 17:47 ` Joseph Myers
2016-11-17 17:54 ` Andreas Schwab
2016-11-17 17:51 ` Joseph Myers
1 sibling, 2 replies; 21+ messages in thread
From: Zack Weinberg @ 2016-11-17 17:26 UTC (permalink / raw)
To: libc-alpha, Joseph Myers
On 11/17/2016 11:52 AM, Zack Weinberg wrote:
> On 11/09/2016 11:27 AM, Joseph Myers wrote:
>> This patch adds a Python (3.5 or later) script to build many different
>> configurations of glibc, including building the required cross
>> compilers first. It's not intended to change any patch testing
>> requirements, although some people may wish to use it for high-risk
>> patches such as adding warning options ...
>
> Since this does its own glibc checkout, it's not clear to me how one
> should use it to test a patch(set). I presume that whatever one does,
> it only affects the "glibcs" step, but what actually do you do? Do you
> manually update /some/where/src/glibc to contain the code you want
> tested and then run "glibcs", or do you somehow tell
> build-many-glibcs.py the name of a branch you want tested, or what?
I went ahead with the setup steps, and they failed during the compilers
phase. It seems that it's not picking up the host libraries I just
built. Now what?
$ ./glibc/scripts/build-many-glibcs.py $PWD/glibc-many host-libraries
PASS: host-libraries gmp rm
PASS: host-libraries gmp mkdir
PASS: host-libraries gmp configure
PASS: host-libraries gmp build
PASS: host-libraries gmp check
PASS: host-libraries gmp install
PASS: host-libraries gmp cleanup
PASS: host-libraries mpfr rm
PASS: host-libraries mpfr mkdir
PASS: host-libraries mpfr configure
PASS: host-libraries mpfr build
PASS: host-libraries mpfr check
PASS: host-libraries mpfr install
PASS: host-libraries mpfr cleanup
PASS: host-libraries mpc rm
PASS: host-libraries mpc mkdir
PASS: host-libraries mpc configure
PASS: host-libraries mpc build
PASS: host-libraries mpc check
PASS: host-libraries mpc install
PASS: host-libraries mpc cleanup
PASS: host-libraries done
$ ./glibc/scripts/build-many-glibcs.py $PWD/glibc-many compilers
PASS: compilers-aarch64_be-linux-gnu check-host-libraries
PASS: compilers-aarch64-linux-gnu check-host-libraries
PASS: compilers-alpha-linux-gnu check-host-libraries
PASS: compilers-arm-linux-gnueabi check-host-libraries
PASS: compilers-arm-linux-gnueabihf check-host-libraries
PASS: compilers-armeb-linux-gnueabi check-host-libraries
PASS: compilers-armeb-linux-gnueabi-be8 check-host-libraries
PASS: compilers-armeb-linux-gnueabihf check-host-libraries
...
PASS: compilers-arm-linux-gnueabi linux mkdir
FAIL: compilers-alpha-linux-gnu gcc-first configure
$ cat
logs/compilers/alpha-linux-gnu/013-compilers-alpha-linux-gnu-gcc-first-configure-log.txt
Thu Nov 17 11:50:29 EST 2016
Description: compilers-alpha-linux-gnu gcc-first configure
Command: /home/zack/projects/glibc-many/src/gcc/configure
--prefix=/home/zack/projects/glibc-many/install/compilers/alpha
-linux-gnu --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu
--target=alpha-glibc-linux-gnu --with-sysroot=/home/zac
k/projects/glibc-many/install/compilers/alpha-linux-gnu/sysroot
--disable-libsanitizer --disable-libssp --enable-languages=c
--disable-shared --disable-threads --disable-libatomic
--disable-decimal-float --disable-libffi --disable-libgomp
--disable-libitm --disable-libmpx --disable-libquadmath
--without-headers --with-newlib --with-glibc-version=2.24
Directory:
/home/zack/projects/glibc-many/build/compilers/alpha-linux-gnu/gcc-first
Path addition:
/home/zack/projects/glibc-many/install/compilers/alpha-linux-gnu/bin
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... alpha-glibc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether ln works... yes
checking whether ln -s works... yes
checking for a sed that does not truncate output... /bin/sed
checking for gawk... gawk
checking for libcilkrts support... no
checking for libvtv support... no
checking for x86_64-pc-linux-gnu-gcc... no
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for x86_64-pc-linux-gnu-g++... no
checking for x86_64-pc-linux-gnu-c++... no
checking for x86_64-pc-linux-gnu-gpp... no
checking for x86_64-pc-linux-gnu-aCC... no
checking for x86_64-pc-linux-gnu-CC... no
checking for x86_64-pc-linux-gnu-cxx... no
checking for x86_64-pc-linux-gnu-cc++... no
checking for x86_64-pc-linux-gnu-cl.exe... no
checking for x86_64-pc-linux-gnu-FCC... no
checking for x86_64-pc-linux-gnu-KCC... no
checking for x86_64-pc-linux-gnu-RCC... no
checking for x86_64-pc-linux-gnu-xlC_r... no
checking for x86_64-pc-linux-gnu-xlC... no
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking whether g++ accepts -static-libstdc++ -static-libgcc... yes
checking for x86_64-pc-linux-gnu-gnatbind... no
checking for gnatbind... no
checking for x86_64-pc-linux-gnu-gnatmake... no
checking for gnatmake... no
checking whether compiler driver understands Ada... no
checking how to compare bootstrapped objects... cmp --ignore-initial=16
$$f1 $$f2
checking for objdir... .libs
checking for the correct version of gmp.h... yes
checking for the correct version of mpfr.h... no
configure: error: Building GCC requires GMP 4.2+, MPFR 2.4.0+ and MPC
0.8.0+.
Try the --with-gmp, --with-mpfr and/or --with-mpc options to specify
their locations. Source code for these libraries can be found at
their respective hosting sites as well as at
ftp://gcc.gnu.org/pub/gcc/infrastructure/. See also
http://gcc.gnu.org/install/prerequisites.html for additional info. If
you obtained GMP, MPFR and/or MPC from a vendor distribution package,
make sure that you have installed both the libraries and the header
files. They may be located in separate packages.
FAIL: compilers-alpha-linux-gnu gcc-first configure
Thu Nov 17 11:50:30 EST 2016
(I see a libgmp.a (for instance) in install/host-libraries/lib/, but not
in install/compilers/alpha-linux-gnu/sysroot/, which is the only place
the above configure operation might have thought to look for it.)
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-17 17:26 ` Zack Weinberg
@ 2016-11-17 17:47 ` Joseph Myers
2016-11-17 17:54 ` Andreas Schwab
1 sibling, 0 replies; 21+ messages in thread
From: Joseph Myers @ 2016-11-17 17:47 UTC (permalink / raw)
To: Zack Weinberg; +Cc: libc-alpha
On Thu, 17 Nov 2016, Zack Weinberg wrote:
> On 11/17/2016 11:52 AM, Zack Weinberg wrote:
> > On 11/09/2016 11:27 AM, Joseph Myers wrote:
> >> This patch adds a Python (3.5 or later) script to build many different
> >> configurations of glibc, including building the required cross
> >> compilers first. It's not intended to change any patch testing
> >> requirements, although some people may wish to use it for high-risk
> >> patches such as adding warning options ...
> >
> > Since this does its own glibc checkout, it's not clear to me how one
> > should use it to test a patch(set). I presume that whatever one does,
> > it only affects the "glibcs" step, but what actually do you do? Do you
> > manually update /some/where/src/glibc to contain the code you want
> > tested and then run "glibcs", or do you somehow tell
> > build-many-glibcs.py the name of a branch you want tested, or what?
>
> I went ahead with the setup steps, and they failed during the compilers
> phase. It seems that it's not picking up the host libraries I just
> built. Now what?
I've committed this patch to fix this. (The systems I tested on had
suitable versions of these libraries installed where host GCC could find
them, so I didn't notice that the newly built libraries weren't being
used.)
Actually use newly built host libraries in build-many-glibcs.py.
This patch adds the missing GCC configure options required to make use
of the newly built host libraries in build-many-glibcs.py.
2016-11-17 Joseph Myers <joseph@codesourcery.com>
* scripts/build-many-glibcs.py (Config.build_gcc): Configure with
newly built gmp, mpfr and mpc.
diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
index be561c3..b0e0f5e 100755
--- a/scripts/build-many-glibcs.py
+++ b/scripts/build-many-glibcs.py
@@ -813,6 +813,10 @@ class Config(object):
# relevance with glibc's own stack checking support.
cfg_opts = list(self.gcc_cfg)
cfg_opts += ['--disable-libsanitizer', '--disable-libssp']
+ host_libs = self.ctx.host_libraries_installdir
+ cfg_opts += ['--with-gmp=%s' % host_libs,
+ '--with-mpfr=%s' % host_libs,
+ '--with-mpc=%s' % host_libs]
if bootstrap:
tool_build = 'gcc-first'
# Building a static-only, C-only compiler that is
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-17 16:52 ` Zack Weinberg
2016-11-17 17:26 ` Zack Weinberg
@ 2016-11-17 17:51 ` Joseph Myers
2016-11-18 16:28 ` Zack Weinberg
1 sibling, 1 reply; 21+ messages in thread
From: Joseph Myers @ 2016-11-17 17:51 UTC (permalink / raw)
To: Zack Weinberg; +Cc: libc-alpha
On Thu, 17 Nov 2016, Zack Weinberg wrote:
> On 11/09/2016 11:27 AM, Joseph Myers wrote:
> > This patch adds a Python (3.5 or later) script to build many different
> > configurations of glibc, including building the required cross
> > compilers first. It's not intended to change any patch testing
> > requirements, although some people may wish to use it for high-risk
> > patches such as adding warning options ...
>
> Since this does its own glibc checkout, it's not clear to me how one
> should use it to test a patch(set). I presume that whatever one does,
> it only affects the "glibcs" step, but what actually do you do? Do you
> manually update /some/where/src/glibc to contain the code you want
> tested and then run "glibcs", or do you somehow tell
> build-many-glibcs.py the name of a branch you want tested, or what?
You patch the script's glibc checkout (or switch it to a different branch,
or whatever) at some point after the checkout step and before running the
"glibcs" build. (The model is that if you're often running tests for many
configurations, you do the host-libraries and compilers steps once, with
some stable GCC version, and then can keep using the previously built
compilers when running the "glibcs" step with new and possibly patched
glibc - so saving time compared to rebuilding the compilers from scratch
every time.)
(The checkout step will not notice if you've manually patched or changed
the branch of a checkout, and will fail if git pull fails, e.g. if it
tries to update a file where you have local changes.)
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-17 17:26 ` Zack Weinberg
2016-11-17 17:47 ` Joseph Myers
@ 2016-11-17 17:54 ` Andreas Schwab
1 sibling, 0 replies; 21+ messages in thread
From: Andreas Schwab @ 2016-11-17 17:54 UTC (permalink / raw)
To: Zack Weinberg; +Cc: libc-alpha, Joseph Myers
On Nov 17 2016, Zack Weinberg <zackw@panix.com> wrote:
> (I see a libgmp.a (for instance) in install/host-libraries/lib/, but not
> in install/compilers/alpha-linux-gnu/sysroot/, which is the only place
> the above configure operation might have thought to look for it.)
Since libgmp is needed for the host the configure script will not search
the target sysroot.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-17 17:51 ` Joseph Myers
@ 2016-11-18 16:28 ` Zack Weinberg
2016-11-18 18:27 ` Joseph Myers
0 siblings, 1 reply; 21+ messages in thread
From: Zack Weinberg @ 2016-11-18 16:28 UTC (permalink / raw)
To: Joseph Myers; +Cc: GNU C Library
On Thu, Nov 17, 2016 at 12:51 PM, Joseph Myers <joseph@codesourcery.com> wrote:
> On Thu, 17 Nov 2016, Zack Weinberg wrote:
>> On 11/09/2016 11:27 AM, Joseph Myers wrote:
>> > This patch adds a Python (3.5 or later) script to build many different
>> > configurations of glibc ...
>>
>> Since this does its own glibc checkout, it's not clear to me how one
>> should use it to test a patch(set)...
>
> You patch the script's glibc checkout (or switch it to a different branch,
> or whatever) at some point after the checkout step and before running the
> "glibcs" build...
Thanks, I have it working now. I'd like to mention some more places
where the script could be more ergonomic for patch testing:
* Please don't delete the build tree unless configure, build, and
check all succeed, so that one can dig into a failure beyond what's
visible in the logs. Add an option to preserve the build tree even if
all three succeeded, to facilitate additional testing (for instance,
an "installed stripped libraries are unchanged" test - if that fails
one will want to go look at the object files).
* Please find some way of not needing to make a copy of the glibc
source tree for each build configuration, so that one can make edits
to .../src/glibc and immediately do incremental rebuilds in all
failing configurations.
* Please make sure that command lines written to logfiles are properly
quoted, so that they can be copied and pasted directly into a shell.
(The problem case I know about is CC="x-y-z-gcc -mthis -mthat" on a
configure command line, which currently gets written to the logfile
without any quotes.) (Python stdlib has a function `shlex.quote` that
will help.)
I'm kibitzing a lot, so let me say that this script is really, really
useful and I can see lots of cases where it will make complex changes
easier to validate. Thanks for writing it.
zw
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-18 16:28 ` Zack Weinberg
@ 2016-11-18 18:27 ` Joseph Myers
0 siblings, 0 replies; 21+ messages in thread
From: Joseph Myers @ 2016-11-18 18:27 UTC (permalink / raw)
To: Zack Weinberg; +Cc: GNU C Library
On Fri, 18 Nov 2016, Zack Weinberg wrote:
> * Please don't delete the build tree unless configure, build, and
> check all succeed, so that one can dig into a failure beyond what's
That's --keep=failed.
> visible in the logs. Add an option to preserve the build tree even if
> all three succeeded, to facilitate additional testing (for instance,
> an "installed stripped libraries are unchanged" test - if that fails
> one will want to go look at the object files).
That's --keep=all.
There's the question of what the best default is. The current --keep=none
default is oriented to the idea of a buildbot that may not want disk usage
to explode after a commit that causes all builds to fail, and will need to
delete the build directories anyway when starting the next build.
> * Please find some way of not needing to make a copy of the glibc
> source tree for each build configuration, so that one can make edits
> to .../src/glibc and immediately do incremental rebuilds in all
> failing configurations.
You'd need to fix bug 14121, then see if builds and testing with a
read-only source directory work (and try to find any cases where Makefile
dependencies mean something could get regenerated in the source directory
and the checkout step doesn't currently fix the timestamps to avoid that).
> * Please make sure that command lines written to logfiles are properly
> quoted, so that they can be copied and pasted directly into a shell.
> (The problem case I know about is CC="x-y-z-gcc -mthis -mthat" on a
> configure command line, which currently gets written to the logfile
> without any quotes.) (Python stdlib has a function `shlex.quote` that
> will help.)
Patch below committed.
Quote shell commands in logs from build-many-glibcs.py.
As requested in
<https://sourceware.org/ml/libc-alpha/2016-11/msg00664.html>, this
patch makes the commands recorded in build-many-glibcs.py quote words
so they can be cut-and-pasted back into a shell. (Note that these
logs are generated by the wrapper script generated to run commands
with logs, hence the needs for quoting logic to be implemented in that
shell script.)
2016-11-18 Joseph Myers <joseph@codesourcery.com>
* scripts/build-many-glibcs.py (Context.write_files): Make wrapper
script quote words in command output to log suitably for input to
the shell.
diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
index b0e0f5e..517dec4 100755
--- a/scripts/build-many-glibcs.py
+++ b/scripts/build-many-glibcs.py
@@ -423,7 +423,17 @@ class Context(object):
'date > "$this_log"\n'
'echo >> "$this_log"\n'
'echo "Description: $desc" >> "$this_log"\n'
- 'echo "Command: $*" >> "$this_log"\n'
+ 'printf "%s" "Command:" >> "$this_log"\n'
+ 'for word in "$@"; do\n'
+ ' if expr "$word" : "[]+,./0-9@A-Z_a-z-]\\\\{1,\\\\}\\$" > /dev/null; then\n'
+ ' printf " %s" "$word"\n'
+ ' else\n'
+ ' printf " \'"\n'
+ ' printf "%s" "$word" | sed -e "s/\'/\'\\\\\\\\\'\'/"\n'
+ ' printf "\'"\n'
+ ' fi\n'
+ 'done >> "$this_log"\n'
+ 'echo >> "$this_log"\n'
'echo "Directory: $dir" >> "$this_log"\n'
'echo "Path addition: $path" >> "$this_log"\n'
'echo >> "$this_log"\n'
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Add script to build many glibc configurations
2016-11-10 14:27 ` Joseph Myers
2016-11-10 16:44 ` Joseph Myers
@ 2016-11-23 17:36 ` Chris Metcalf
1 sibling, 0 replies; 21+ messages in thread
From: Chris Metcalf @ 2016-11-23 17:36 UTC (permalink / raw)
To: Joseph Myers, libc-alpha
On 11/10/2016 9:27 AM, Joseph Myers wrote:
> Here are the failures I observe with this script, with the indicated eight
> glibc patches applied (excluding check-compilers failures at the glibcs
> stage from the compilers having failed to build in the compilers stage):
I think glibc should build cleanly now with all the compilers you
cited in your email for both tilepro and tilegx now.
Thanks for setting this up - it's a great way to stay on top of
build issues.
--
Chris Metcalf, Mellanox Technologies
http://www.mellanox.com
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2016-11-23 17:36 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-09 16:27 Add script to build many glibc configurations Joseph Myers
2016-11-10 14:27 ` Joseph Myers
2016-11-10 16:44 ` Joseph Myers
2016-11-23 17:36 ` Chris Metcalf
2016-11-10 17:09 ` Steve Ellcey
2016-11-10 17:22 ` Joseph Myers
2016-11-10 17:39 ` Steve Ellcey
2016-11-11 15:20 ` Joseph Myers
2016-11-11 19:37 ` Carlos O'Donell
2016-11-14 15:06 ` Mike Frysinger
2016-11-14 15:23 ` Joseph Myers
2016-11-14 19:27 ` Mike Frysinger
2016-11-14 23:28 ` Joseph Myers
2016-11-14 23:57 ` Joseph Myers
2016-11-17 16:52 ` Zack Weinberg
2016-11-17 17:26 ` Zack Weinberg
2016-11-17 17:47 ` Joseph Myers
2016-11-17 17:54 ` Andreas Schwab
2016-11-17 17:51 ` Joseph Myers
2016-11-18 16:28 ` Zack Weinberg
2016-11-18 18:27 ` Joseph Myers
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).