Unfortunately, my inline PGP signature seems to have corrupted the patch, so I'll resend it as multipart/signed message. Sorry about that. [end top-post] From 3847ffad3f1c7757d34b7a24279f5bb3b64b4c87 Mon Sep 17 00:00:00 2001 Hi, On 26/09/12 18:56, Yann E. MORIN wrote: > On Wednesday 26 September 2012 17:30:12 Blair Burtan wrote: >> That was the problem but not quite the complete solution. I had to modify >> the grep path to point to the MacPorts verison when I built ct-ng. I also >> had to use ginstall instead of the OSX install. > > ./configure --help > [--SNIP--] > Optional Packages: > --with-PACKAGE[=ARG] use PACKAGE [ARG=yes] > --without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no) > --with-install=PATH Specify the full PATH to a BSD-compatible install > --with-sed=PATH Specify the full PATH to GNU sed > --with-objcopy=PATH Specify the full PATH to GNU objcopy > --with-objdump=PATH Specify the full PATH to GNU objdump > --with-ranlib=PATH Specify the full PATH to GNU ranlib > --with-readelf=PATH Specify the full PATH to GNU readelf > --with-bash=PATH Specify the full PATH to GNU bash >= 3.1 > --with-awk=PATH Specify the full PATH to GNU awk > --with-make=PATH Specify the full PATH to GNU make >= 3.80 > --with-libtool=PATH Specify the full PATH to GNU libtool >= 1.5.26 > --with-libtoolize=PATH Specify the full PATH to GNU libtoolize >= 1.5.26 > --with-automake=PATH Specify the full PATH to GNU automake >= 1.10 > [--SNIP--] > > That's how you tell crosstool-NG what tools to use if the default ones are > not the GNU ones. While it is possible to tell crosstool-NG to use custom GNU tools for several tools, it is not possible for all tools, especially for grep and sed. This causes ct-ng build to fail: [ERROR] >> [ERROR] >> Build failed in step '(top-level)' [ERROR] >> [ERROR] >> Error happened in: CT_DoForceRmdir[scripts/functions@458] [ERROR] >> called from: main[scripts/crosstool-NG.sh@238] [ERROR] >> One of the problems here is also that on BSD installs, including OS X, GNU tools are often installed with a prefixed 'g', in this case "ggrep" and "gsed". There is no way to tell crosstool-NG to use ggrep instead of grep, and adjusting $PATH does not help either. I have created a small (not counting the changes due to the scripts/functions rename) patch that adds --with-grep and substitues @@CT_sed@@ for sed in a few places where sed from PATH is used and GNU-specific options are used. I can confirm this works for me on Darwin using GNU tools installed via homebrew using the following ./configure options: ./configure --prefix=$(pwd) --exec-prefix=$(pwd) \ --with-objcopy=gobjcopy \ --with-objdump=gobjdump \ --with-readelf=greadelf \ --with-libtool=glibtool \ --with-libtoolize=glibtoolize \ --with-install=ginstall \ --with-sed=gsed \ --with-awk=gawk \ --with-grep=ggrep I have not yet tested that this patch works flawlessly on Linux using default ./configure options. I'd like to submit the following patch for testing. Regards, Fabian Freyer --- Makefile.in | 5 +- configure.ac | 6 + scripts/crosstool-NG.sh.in | 6 +- scripts/functions | 1395 -------------------------------------------- scripts/functions.in | 1395 ++++++++++++++++++++++++++++++++++++++++++++ scripts/xldd.in | 2 +- 6 files changed, 1409 insertions(+), 1400 deletions(-) delete mode 100644 scripts/functions create mode 100644 scripts/functions.in diff --git a/Makefile.in b/Makefile.in index 01759b6..cd68baa 100644 --- a/Makefile.in +++ b/Makefile.in @@ -59,7 +59,7 @@ export datarootdir := @datarootdir@ export install := @INSTALL@ export bash := @_BASH@ export awk := @_AWK@ -export grep := @GREP@ +export grep := @_GREP@ export make := @MAKE@ export sed := @SED@ export libtool := @LIBTOOL@ @@ -142,6 +142,7 @@ uninstall: real-uninstall # Build rules build-bin: $(PROG_NAME) \ + scripts/functions \ scripts/crosstool-NG.sh \ scripts/saveSample.sh \ scripts/showTuple.sh @@ -174,6 +175,8 @@ define sed_it -e 's,@@CT_make@@,$(make),g;' \ -e 's,@@CT_bash@@,$(bash),g;' \ -e 's,@@CT_awk@@,$(awk),g;' \ + -e 's,@@CT_grep@@,$(grep),g;' \ + -e 's,@@CT_sed@@,$(sed),g;' \ $< >$@ endef diff --git a/configure.ac b/configure.ac index f8c67be..27238ab 100644 --- a/configure.ac +++ b/configure.ac @@ -94,6 +94,12 @@ AC_ARG_WITH([install], [Specify the full PATH to a BSD-compatible install]), [INSTALL=$withval]) AC_PROG_INSTALL +AC_CACHE_VAL([ac_cv_path_GREP], + [AC_ARG_WITH([grep], + AS_HELP_STRING([--with-grep=PATH], + [Specify the full PATH to GNU grep]), + [ac_cv_path_GREP=$withval])]) +AC_SUBST([_GREP], [$ac_cv_path_GREP]) AC_PROG_GREP AC_PROG_EGREP AS_IF( diff --git a/scripts/crosstool-NG.sh.in b/scripts/crosstool-NG.sh.in index 3699500..4b9011d 100644 --- a/scripts/crosstool-NG.sh.in +++ b/scripts/crosstool-NG.sh.in @@ -125,7 +125,7 @@ CT_DoLog INFO "Build started ${CT_STAR_DATE_HUMAN}" # We really need to extract from ,config and not .config.2, as we # do want the kconfig's values, not our mangled config with arrays. CT_DoStep DEBUG "Dumping user-supplied crosstool-NG configuration" -CT_DoExecLog DEBUG grep -E '^(# |)CT_' .config +CT_DoExecLog DEBUG @@CT_grep@@ -E '^(# |)CT_' .config CT_EndStep CT_DoLog DEBUG "Unsetting and unexporting MAKEFLAGS" @@ -570,9 +570,9 @@ if [ -z "${CT_RESTART}" ]; then CT_DoLog EXTRA " build = ${CT_REAL_BUILD}" CT_DoLog EXTRA " host = ${CT_REAL_HOST}" CT_DoLog EXTRA " target = ${CT_TARGET}" - set |grep -E '^CT_.+=' |sort |CT_DoLog DEBUG + set |@@CT_grep@@ -E '^CT_.+=' |sort |CT_DoLog DEBUG CT_DoLog DEBUG "Other environment:" - printenv |grep -v -E '^CT_.+=' |CT_DoLog DEBUG + printenv |@@CT_grep@@ -v -E '^CT_.+=' |CT_DoLog DEBUG CT_EndStep fi diff --git a/scripts/functions b/scripts/functions deleted file mode 100644 index 2e4d4fa..0000000 --- a/scripts/functions +++ /dev/null @@ -1,1395 +0,0 @@ -# This file contains some usefull common functions -*- sh -*- -# Copyright 2007 Yann E. MORIN -# Licensed under the GPL v2. See COPYING in the root of this package - -# Prepare the fault handler -CT_OnError() { - local ret=$? - local result - local old_trap - local intro - local file line func - local step step_depth - - # To avoid printing the backtace for each sub-shell - # up to the top-level, just remember we've dumped it - if [ ! -f "${CT_WORK_DIR}/backtrace" ]; then - touch "${CT_WORK_DIR}/backtrace" - - # Print steps backtrace - step_depth=${CT_STEP_COUNT} - CT_STEP_COUNT=1 # To have a zero-indentation - CT_DoLog ERROR "" - CT_DoLog ERROR ">>" - intro="Build failed" - for((step=step_depth; step>0; step--)); do - CT_DoLog ERROR ">> ${intro} in step '${CT_STEP_MESSAGE[${step}]}'" - intro=" called" - done - - # Print functions backtrace - intro="Error happened in" - CT_DoLog ERROR ">>" - for((depth=1; ${BASH_LINENO[$((${depth}-1))]}>0; depth++)); do - file="${BASH_SOURCE[${depth}]#${CT_LIB_DIR}/}" - func="${FUNCNAME[${depth}]}" - line="@${BASH_LINENO[${depth}-1]:-?}" - CT_DoLog ERROR ">> ${intro}: ${func}[${file}${line}]" - intro=" called from" - done - - # If the user asked for interactive debugging, dump him/her to a shell - if [ "${CT_DEBUG_INTERACTIVE}" = "y" ]; then - # We do not want this sub-shell exit status to be caught, because - # it is absolutely legit that it exits with non-zero. - # Save the trap handler to restore it after our debug-shell - old_trap="$(trap -p ERR)" - trap -- ERR - ( - exec >&6 2>&7 <&8 - printf "\r \n\nCurrent command" - if [ -n "${cur_cmd}" ]; then - printf ":\n %s\n" "${cur_cmd}" - else - printf " (unknown), " - fi - printf "exited with error code: %d\n" ${ret} - printf "Please fix it up and finish by exiting the shell with one of these values:\n" - printf " 1 fixed, continue with next build command\n" - if [ -n "${cur_cmd}" ]; then - printf " 2 repeat this build command\n" - fi - printf " 3 abort build\n\n" - while true; do - ${bash} --rcfile <(printf "PS1='ct-ng:\w> '\nPROMPT_COMMAND=''\n") -i - result=$? - case $result in - 1) printf "\nContinuing past the failed command.\n\n" - break - ;; - 2) if [ -n "${cur_cmd}" ]; then - printf "\nRe-trying last command.\n\n" - break - fi - ;; - 3) break;; - esac - printf "\nPlease exit with one of these values:\n" - printf " 1 fixed, continue with next build command\n" - if [ -n "${cur_cmd}" ]; then - printf " 2 repeat this build command\n" - fi - printf " 3 abort build\n" - done - exit $result - ) - result=$? - # Restore the trap handler - eval "${old_trap}" - case "${result}" in - 1) rm -f "${CT_WORK_DIR}/backtrace"; touch "${CT_BUILD_DIR}/skip"; return;; - 2) rm -f "${CT_WORK_DIR}/backtrace"; touch "${CT_BUILD_DIR}/repeat"; return;; - # 3 is an abort, continue... - esac - fi - fi - - # And finally, in top-level shell, print some hints - if [ ${BASH_SUBSHELL} -eq 0 ]; then - # Help diagnose the error - CT_STEP_COUNT=1 # To have a zero-indentation - CT_DoLog ERROR ">>" - if [ "${CT_LOG_TO_FILE}" = "y" ]; then - CT_DoLog ERROR ">> For more info on this error, look at the file: '${tmp_log_file#${CT_TOP_DIR}/}'" - fi - CT_DoLog ERROR ">> There is a list of known issues, some with workarounds, in:" - CT_DoLog ERROR ">> '${CT_DOC_DIR#${CT_TOP_DIR}/}/B - Known issues.txt'" - - CT_DoLog ERROR "" - CT_DoEnd ERROR - rm -f "${CT_WORK_DIR}/backtrace" - fi - exit $ret -} - -# Install the fault handler -trap CT_OnError ERR - -# Inherit the fault handler in subshells and functions -set -E - -# Make pipes fail on the _first_ failed command -# Not supported on bash < 3.x, but we need it, so drop the obsoleting bash-2.x -set -o pipefail - -# Don't hash commands' locations, and search every time it is requested. -# This is slow, but needed because of the static/shared core gcc which shall -# always match to shared if it exists, and only fallback to static if the -# shared is not found -set +o hashall - -# Log policy: -# - first of all, save stdout so we can see the live logs: fd #6 -# (also save stdin and stderr for use by CT_DEBUG_INTERACTIVE) -exec 6>&1 7>&2 8<&0 -# - then point stdout to the log file -tmp_log_file="${CT_TOP_DIR}/build.log" -rm -f "${tmp_log_file}" -exec >>"${tmp_log_file}" - -# The different log levels: -CT_LOG_LEVEL_ERROR=0 -CT_LOG_LEVEL_WARN=1 -CT_LOG_LEVEL_INFO=2 -CT_LOG_LEVEL_EXTRA=3 -CT_LOG_LEVEL_CFG=4 -CT_LOG_LEVEL_FILE=5 -CT_LOG_LEVEL_STATE=6 -CT_LOG_LEVEL_ALL=7 -CT_LOG_LEVEL_DEBUG=8 - -# Make it easy to use \n and ! -CR=$(printf "\n") -BANG='!' - -# A function to log what is happening -# Different log level are available: -# - ERROR: A serious, fatal error occurred -# - WARN: A non fatal, non serious error occurred, take your responsbility with the generated build -# - INFO: Informational messages -# - EXTRA: Extra informational messages -# - CFG: Output of various "./configure"-type scripts -# - FILE: File / archive unpacking. -# - STATE: State save & restore -# - ALL: Component's build messages -# - DEBUG: Internal debug messages -# Usage: CT_DoLog [message] -# If message is empty, then stdin will be logged. -CT_DoLog() { - local max_level LEVEL level cur_l cur_L - local l - eval max_level="\${CT_LOG_LEVEL_${CT_LOG_LEVEL_MAX}}" - # Set the maximum log level to DEBUG if we have none - [ -z "${max_level}" ] && max_level=${CT_LOG_LEVEL_DEBUG} - - LEVEL="$1"; shift - eval level="\${CT_LOG_LEVEL_${LEVEL}}" - - if [ $# -eq 0 ]; then - cat - - else - printf "%s\n" "${*}" - fi |( IFS="${CR}" # We want the full lines, even leading spaces - _prog_bar_cpt=0 - _prog_bar[0]='/' - _prog_bar[1]='-' - _prog_bar[2]='\' - _prog_bar[3]='|' - indent=$((2*CT_STEP_COUNT)) - while read line; do - case "${CT_LOG_SEE_TOOLS_WARN},${line}" in - y,*"warning:"*) cur_L=WARN; cur_l=${CT_LOG_LEVEL_WARN};; - y,*"WARNING:"*) cur_L=WARN; cur_l=${CT_LOG_LEVEL_WARN};; - *"error:"*) cur_L=ERROR; cur_l=${CT_LOG_LEVEL_ERROR};; - *"make["*"]: *** ["*) cur_L=ERROR; cur_l=${CT_LOG_LEVEL_ERROR};; - *) cur_L="${LEVEL}"; cur_l="${level}";; - esac - # There will always be a log file (stdout, fd #1), be it /dev/null - printf "[%-5s]%*s%s%s\n" "${cur_L}" "${indent}" " " "${line}" - if [ ${cur_l} -le ${max_level} ]; then - # Only print to console (fd #6) if log level is high enough. - printf "${CT_LOG_PROGRESS_BAR:+\r}[%-5s]%*s%s%s\n" "${cur_L}" "${indent}" " " "${line}" >&6 - fi - if [ "${CT_LOG_PROGRESS_BAR}" = "y" ]; then - printf "\r[%02d:%02d] %s " $((SECONDS/60)) $((SECONDS%60)) "${_prog_bar[$((_prog_bar_cpt/10))]}" >&6 - _prog_bar_cpt=$(((_prog_bar_cpt+1)%40)) - fi - done - ) - - return 0 -} - -# Execute an action, and log its messages -# It is possible to even log local variable assignments (a-la: var=val ./cmd opts) -# Usage: CT_DoExecLog [VAR=val...] [parameters...] -CT_DoExecLog() { - local level="$1" - local cur_cmd - local ret - shift - ( - for i in "$@"; do - cur_cmd+="'${i}' " - done - while true; do - case "${1}" in - *=*) eval export "'${1}'"; shift;; - *) break;; - esac - done - # This while-loop goes hand-in-hand with the ERR trap handler: - # - if the command terminates successfully, then we hit the break - # statement, and we exit the loop - # - if the command terminates in error, then the ERR handler kicks - # in, then: - # - if the user did *not* ask for interactive debugging, the ERR - # handler exits, and we hit the end of the sub-shell - # - if the user did ask for interactive debugging, the ERR handler - # spawns a shell. Upon termination of this shell, the ERR handler - # examines the exit status of the shell: - # - if 1, the ERR handler returns; then we hit the else statement, - # then the break, and we exit the 'while' loop, to continue the - # build; - # - if 2, the ERR handler touches the repeat file, and returns; - # then we hit the if statement, and we loop for one more - # iteration; - # - if 3, the ERR handler exits with the command's exit status, - # and we're dead; - # - for any other exit status of the shell, the ERR handler - # prints an informational message, and respawns the shell - # - # This allows a user to get an interactive shell that has the same - # environment (PATH and so on) that the failed command was ran with. - while true; do - rm -f "${CT_BUILD_DIR}/repeat" - CT_DoLog DEBUG "==> Executing: ${cur_cmd}" - "${@}" 2>&1 |CT_DoLog "${level}" - ret="${?}" - if [ -f "${CT_BUILD_DIR}/repeat" ]; then - rm -f "${CT_BUILD_DIR}/repeat" - continue - elif [ -f "${CT_BUILD_DIR}/skip" ]; then - rm -f "${CT_BUILD_DIR}/skip" - ret=0 - break - else - break - fi - done - exit ${ret} - ) - # Catch failure of the sub-shell - [ $? -eq 0 ] -} - -# Tail message to be logged whatever happens -# Usage: CT_DoEnd -CT_DoEnd() -{ - local level="$1" - CT_STOP_DATE=$(CT_DoDate +%s%N) - CT_STOP_DATE_HUMAN=$(CT_DoDate +%Y%m%d.%H%M%S) - if [ "${level}" != "ERROR" ]; then - CT_DoLog "${level:-INFO}" "Build completed at ${CT_STOP_DATE_HUMAN}" - fi - elapsed=$((CT_STOP_DATE-CT_STAR_DATE)) - elapsed_min=$((elapsed/(60*1000*1000*1000))) - elapsed_sec=$(printf "%02d" $(((elapsed%(60*1000*1000*1000))/(1000*1000*1000)))) - elapsed_csec=$(printf "%02d" $(((elapsed%(1000*1000*1000))/(10*1000*1000)))) - CT_DoLog ${level:-INFO} "(elapsed: ${elapsed_min}:${elapsed_sec}.${elapsed_csec})" -} - -# Remove entries referring to . and other relative paths -# Usage: CT_SanitizePath -CT_SanitizePath() { - local new - local p - local IFS=: - for p in $PATH; do - # Only accept absolute paths; - # Note: as a special case the empty string in PATH is equivalent to . - if [ -n "${p}" -a -z "${p%%/*}" ]; then - new="${new}${new:+:}${p}" - fi - done - PATH="${new}" -} - -# Sanitise the directory name contained in the variable passed as argument: -# - remove duplicate / -# Usage: CT_SanitiseVarDir CT_PREFIX_DIR -CT_SanitiseVarDir() { - local var - local old_dir - local new_dir - - for var in "$@"; do - eval "old_dir=\"\${${var}}\"" - new_dir="$( printf "${old_dir}" \ - |sed -r -e 's:/+:/:g;' \ - )" - eval "${var}=\"${new_dir}\"" - CT_DoLog DEBUG "Sanitised '${var}': '${old_dir}' -> '${new_dir}'" - done -} - -# Abort the execution with an error message -# Usage: CT_Abort -CT_Abort() { - CT_DoLog ERROR "$1" - false -} - -# Test a condition, and print a message if satisfied -# Usage: CT_Test -CT_Test() { - local ret - local m="$1" - shift - CT_DoLog DEBUG "Testing '! ( $* )'" - test "$@" && CT_DoLog WARN "$m" - return 0 -} - -# Test a condition, and abort with an error message if satisfied -# Usage: CT_TestAndAbort -CT_TestAndAbort() { - local m="$1" - shift - CT_DoLog DEBUG "Testing '! ( $* )'" - test "$@" && CT_Abort "$m" - return 0 -} - -# Test a condition, and abort with an error message if not satisfied -# Usage: CT_TestAndAbort -CT_TestOrAbort() { - local m="$1" - shift - CT_DoLog DEBUG "Testing '$*'" - test "$@" || CT_Abort "$m" - return 0 -} - -# Test the presence of a tool, or abort if not found -# Usage: CT_HasOrAbort -CT_HasOrAbort() { - CT_TestAndAbort "'${1}' not found and needed for successful toolchain build." -z "$(CT_Which "${1}")" - return 0 -} - -# Search a program: wrap "which" for those system where "which" -# verbosely says there is no match (such as on Mandriva). -# Usage: CT_Which -CT_Which() { - which "$1" 2>/dev/null || true -} - -# Get current date with nanosecond precision -# On those system not supporting nanosecond precision, faked with rounding down -# to the highest entire second -# Usage: CT_DoDate -CT_DoDate() { - date "$1" |sed -r -e 's/%?N$/000000000/;' -} - -CT_STEP_COUNT=1 -CT_STEP_MESSAGE[${CT_STEP_COUNT}]="(top-level)" -# Memorise a step being done so that any error is caught -# Usage: CT_DoStep -CT_DoStep() { - local start=$(CT_DoDate +%s%N) - CT_DoLog "$1" "=================================================================" - CT_DoLog "$1" "$2" - CT_STEP_COUNT=$((CT_STEP_COUNT+1)) - CT_STEP_LEVEL[${CT_STEP_COUNT}]="$1"; shift - CT_STEP_START[${CT_STEP_COUNT}]="${start}" - CT_STEP_MESSAGE[${CT_STEP_COUNT}]="$1" - return 0 -} - -# End the step just being done -# Usage: CT_EndStep -CT_EndStep() { - local stop=$(CT_DoDate +%s%N) - local duration=$(printf "%032d" $((stop-${CT_STEP_START[${CT_STEP_COUNT}]})) |sed -r -e 's/([[:digit:]]{2})[[:digit:]]{7}$/\.\1/; s/^0+//; s/^\./0\./;') - local elapsed=$(printf "%02d:%02d" $((SECONDS/60)) $((SECONDS%60))) - local level="${CT_STEP_LEVEL[${CT_STEP_COUNT}]}" - local message="${CT_STEP_MESSAGE[${CT_STEP_COUNT}]}" - CT_STEP_COUNT=$((CT_STEP_COUNT-1)) - CT_DoLog "${level}" "${message}: done in ${duration}s (at ${elapsed})" - return 0 -} - -# Pushes into a directory, and pops back -CT_Pushd() { - CT_DoLog DEBUG "Entering '$1'" - pushd "$1" >/dev/null 2>&1 -} -CT_Popd() { - popd >/dev/null 2>&1 -} - -# Create a dir and cd or pushd into it -# Usage: CT_mkdir_cd -# CT_mkdir_pushd -CT_mkdir_cd() { - local dir="${1}" - - mkdir -p "${dir}" - cd "${dir}" -} -CT_mkdir_pushd() { - local dir="${1}" - - mkdir -p "${dir}" - CT_Pushd "${dir}" -} - -# Creates a temporary directory -# $1: variable to assign to -# Usage: CT_MktempDir foo -CT_MktempDir() { - # Some mktemp do not allow more than 6 Xs - eval "$1"=$(mktemp -q -d "${CT_BUILD_DIR}/tmp.XXXXXX") - CT_TestOrAbort "Could not make temporary directory" -n "${!1}" -a -d "${!1}" - CT_DoLog DEBUG "Made temporary directory '${!1}'" - return 0 -} - -# Removes one or more directories, even if it is read-only, or its parent is -# Usage: CT_DoForceRmdir dir [...] -CT_DoForceRmdir() { - local dir - local mode - for dir in "${@}"; do - [ -d "${dir}" ] || continue - case "$CT_SYS_OS" in - Linux|CYGWIN*) - mode="$(stat -c '%a' "$(dirname "${dir}")")" - ;; - Darwin|*BSD) - mode="$(stat -f '%Lp' "$(dirname "${dir}")")" - ;; - *) - CT_Abort "Unhandled host OS $CT_SYS_OS" - ;; - esac - CT_DoExecLog ALL chmod u+w "$(dirname "${dir}")" - CT_DoExecLog ALL chmod -R u+w "${dir}" - CT_DoExecLog ALL rm -rf "${dir}" - CT_DoExecLog ALL chmod ${mode} "$(dirname "${dir}")" - done -} - -# Echoes the specified string on stdout until the pipe breaks. -# Doesn't fail -# $1: string to echo -# Usage: CT_DoYes "" |make oldconfig -CT_DoYes() { - yes "$1" || true -} - -# Add the specified directory to LD_LIBRARY_PATH, and export it -# If the specified patch is already present, just export -# $1: path to add -# $2: add as 'first' or 'last' path, 'first' is assumed if $2 is empty -# Usage CT_SetLibPath /some/where/lib [first|last] -CT_SetLibPath() { - local path="$1" - local pos="$2" - - case ":${LD_LIBRARY_PATH}:" in - *:"${path}":*) ;; - *) case "${pos}" in - last) - CT_DoLog DEBUG "Adding '${path}' at end of LD_LIBRARY_PATH" - LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}${path}" - ;; - first|"") - CT_DoLog DEBUG "Adding '${path}' at start of LD_LIBRARY_PATH" - LD_LIBRARY_PATH="${path}${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" - ;; - *) - CT_Abort "Incorrect position '${pos}' to add '${path}' to LD_LIBRARY_PATH" - ;; - esac - ;; - esac - CT_DoLog DEBUG "==> LD_LIBRARY_PATH='${LD_LIBRARY_PATH}'" - export LD_LIBRARY_PATH -} - -# Build up the list of allowed tarball extensions -# Add them in the prefered order; most preferred comes first -CT_DoListTarballExt() { - if [ "${CT_CONFIGURE_has_xz}" = "y" ]; then - printf ".tar.xz\n" - fi - if [ "${CT_CONFIGURE_has_lzma}" = "y" \ - -o "${CT_CONFIGURE_has_xz}" = "y" ]; then - printf ".tar.lzma\n" - fi - printf ".tar.bz2\n" - printf ".tar.gz\n.tgz\n" - printf ".tar\n" - printf ".zip\n" -} - -# Get the file name extension of a component -# Usage: CT_GetFileExtension [extension] -# If found, echoes the extension to stdout, and return 0 -# If not found, echoes nothing on stdout, and return !0. -CT_GetFileExtension() { - local ext - local file="$1" - shift - local first_ext="$1" - - # we need to also check for an empty extension for those very - # peculiar components that don't have one (such as sstrip from - # buildroot). - for ext in ${first_ext} $(CT_DoListTarballExt) /.git ''; do - if [ -e "${CT_TARBALLS_DIR}/${file}${ext}" ]; then - echo "${ext}" - exit 0 - fi - done - - exit 1 -} - -# Try to retrieve the specified URL (HTTP or FTP) -# Usage: CT_DoGetFile -# This functions always returns true (0), as it can be legitimate not -# to find the requested URL (think about snapshots, different layouts -# for different gcc versions, etc...). -CT_DoGetFile() { - local url="${1}" - local dest="${CT_TARBALLS_DIR}/${url##*/}" - local tmp="${dest}.tmp-dl" - - # Remove potential left-over from a previous run - rm -f "${tmp}" - - # We also retry a few times, in case there is a transient error (eg. behind - # a dynamic IP that changes during the transfer...) - # With automated download as we are doing, it can be very dangerous to - # continue the downloads. It's far better to simply overwrite the - # destination file. - # Some company networks have firewalls to connect to the internet, but it's - # not easy to detect them, so force a global ${CT_CONNECT_TIMEOUT}-second - # timeout. - if [ ${CT_CONNECT_TIMEOUT} = -1 ]; then - T= - else - T="-T ${CT_CONNECT_TIMEOUT}" - fi - if CT_DoExecLog ALL wget --passive-ftp --tries=3 -nc \ - --progress=dot:binary \ - ${T} \ - -O "${tmp}" \ - "${url}" - then - # Success, we got it, good! - mv "${tmp}" "${dest}" - CT_DoLog DEBUG "Got it from: \"${url}\"" - else - # Woops... - rm -f "${tmp}" - CT_DoLog DEBUG "Not at this location: \"${url}\"" - fi -} - -# This function tries to retrieve a tarball form a local directory -# Usage: CT_GetLocal [.extension] -CT_GetLocal() { - local basename="$1" - local first_ext="$2" - local ext - - # Do we already have it in *our* tarballs dir? - if ext="$( CT_GetFileExtension "${basename}" ${first_ext} )"; then - CT_DoLog DEBUG "Already have '${basename}'" - return 0 - fi - - if [ -n "${CT_LOCAL_TARBALLS_DIR}" ]; then - CT_DoLog DEBUG "Trying to retrieve an already downloaded copy of '${basename}'" - # We'd rather have a bzip2'ed tarball, then gzipped tarball, plain tarball, - # or, as a failover, a file without extension. - for ext in ${first_ext} $(CT_DoListTarballExt) ''; do - CT_DoLog DEBUG "Trying '${CT_LOCAL_TARBALLS_DIR}/${basename}${ext}'" - if [ -r "${CT_LOCAL_TARBALLS_DIR}/${basename}${ext}" -a \ - "${CT_FORCE_DOWNLOAD}" != "y" ]; then - CT_DoLog DEBUG "Got '${basename}' from local storage" - CT_DoExecLog ALL ln -s "${CT_LOCAL_TARBALLS_DIR}/${basename}${ext}" "${CT_TARBALLS_DIR}/${basename}${ext}" - return 0 - fi - done - fi - return 1 -} - -# This function gets the custom source from either a tarball or directory -# Usage: CT_GetCustom -CT_GetCustom() { - local custom_component="$1" - local custom_version="$2" - local custom_location="$3" - local custom_name="${custom_component}-${custom_version}" - - CT_TestAndAbort "${custom_name}: CT_CUSTOM_LOCATION_ROOT_DIR or ${custom_component}'s CUSTOM_LOCATION must be set." \ - -z "${CT_CUSTOM_LOCATION_ROOT_DIR}" -a -z "${custom_location}" - - if [ -n "${CT_CUSTOM_LOCATION_ROOT_DIR}" \ - -a -z "${custom_location}" ]; then - custom_location="${CT_CUSTOM_LOCATION_ROOT_DIR}/${custom_component}" - fi - - CT_DoLog EXTRA "Using '${custom_name}' from custom location" - if [ ! -d "${custom_location}" ]; then - # We need to know the custom tarball extension, - # so we can create a properly-named symlink, which - # we use later on in 'extract' - case "${custom_location}" in - *.tar.xz) custom_name="${custom_name}.tar.xz";; - *.tar.bz2) custom_name="${custom_name}.tar.bz2";; - *.tar.gz|*.tgz) custom_name="${custom_name}.tar.gz";; - *.tar) custom_name="${custom_name}.tar";; - *) CT_Abort "Unknown extension for custom tarball '${custom_location}'";; - esac - CT_DoExecLog DEBUG ln -sf "${custom_location}" \ - "${CT_TARBALLS_DIR}/${custom_name}" - else - CT_DoExecLog DEBUG ln -snf "${custom_location}" \ - "${CT_SRC_DIR}/${custom_name}" - fi -} - -# This function saves the specified to local storage if possible, -# and if so, symlinks it for later usage -# Usage: CT_SaveLocal -CT_SaveLocal() { - local file="$1" - local basename="${file##*/}" - - if [ "${CT_SAVE_TARBALLS}" = "y" ]; then - CT_DoLog EXTRA "Saving '${basename}' to local storage" - # The file may already exist if downloads are forced: remove it first - CT_DoExecLog ALL rm -f "${CT_LOCAL_TARBALLS_DIR}/${basename}" - CT_DoExecLog ALL mv -f "${file}" "${CT_LOCAL_TARBALLS_DIR}" - CT_DoExecLog ALL ln -s "${CT_LOCAL_TARBALLS_DIR}/${basename}" "${file}" - fi -} - -# Download the file from one of the URLs passed as argument -# Usage: CT_GetFile [.extension] [url ...] -CT_GetFile() { - local ext - local -a URLS - local url - local file="$1" - local first_ext - shift - # If next argument starts with a dot, then this is not an URL, - # and we can consider that it is a preferred extension. - case "$1" in - .*) first_ext="$1" - shift - ;; - esac - - # Does it exist localy? - if CT_GetLocal "${file}" ${first_ext}; then - return 0 - fi - # No, it does not... - - # If not allowed to download from the Internet, don't - if [ "${CT_FORBID_DOWNLOAD}" = "y" ]; then - CT_DoLog DEBUG "Not allowed to download from the Internet, aborting ${file} download" - return 1 - fi - - # Try to retrieve the file - CT_DoLog EXTRA "Retrieving '${file}'" - - # Add URLs on the LAN mirror - if [ "${CT_USE_MIRROR}" = "y" ]; then - CT_TestOrAbort "Please set the mirror base URL" -n "${CT_MIRROR_BASE_URL}" - URLS+=( "${CT_MIRROR_BASE_URL}/${file%-*}" ) - URLS+=( "${CT_MIRROR_BASE_URL}" ) - fi - - if [ "${CT_FORCE_MIRROR}" != "y" ]; then - URLS+=( "${@}" ) - fi - - # Scan all URLs in turn, and try to grab a tarball from there - # Do *not* try git trees (ext=/.git), this is handled in a specific - # wrapper, below - for ext in ${first_ext} $(CT_DoListTarballExt) ''; do - # Try all urls in turn - for url in "${URLS[@]}"; do - [ -n "${url}" ] || continue - CT_DoLog DEBUG "Trying '${url}/${file}${ext}'" - CT_DoGetFile "${url}/${file}${ext}" - if [ -f "${CT_TARBALLS_DIR}/${file}${ext}" ]; then - CT_DoLog DEBUG "Got '${file}' from the Internet" - CT_SaveLocal "${CT_TARBALLS_DIR}/${file}${ext}" - return 0 - fi - done - done - - # Just return error, someone may want to catch and handle the error - # (eg. glibc/eglibc add-ons can be missing). - return 1 -} - -# Checkout from CVS, and build the associated tarball -# The tarball will be called ${basename}.tar.bz2 -# Prerequisite: either the server does not require password, -# or the user must already be logged in. -# 'tag' is the tag to retrieve. Must be specified, but can be empty. -# If dirname is specified, then module will be renamed to dirname -# prior to building the tarball. -# Usage: CT_GetCVS [dirname[=subdir]] -# Note: if '=subdir' is given, then it is used instead of 'module'. -CT_GetCVS() { - local basename="$1" - local uri="$2" - local module="$3" - local tag="${4:+-r ${4}}" - local dirname="$5" - local tmp_dir - - # First try locally, then the mirror - if CT_GetFile "${basename}"; then - # Got it! Return early! - return 0 - fi - - if [ "${CT_FORBID_DOWNLOAD}" = "y" ]; then - CT_DoLog WARN "Downloads forbidden, not trying cvs retrieval" - return 1 - fi - - CT_MktempDir tmp_dir - CT_Pushd "${tmp_dir}" - - CT_DoExecLog ALL cvs -z 9 -d "${uri}" co -P ${tag} "${module}" - if [ -n "${dirname}" ]; then - case "${dirname}" in - *=*) - CT_DoExecLog DEBUG mv "${dirname#*=}" "${dirname%%=*}" - CT_DoExecLog ALL tar cjf "${CT_TARBALLS_DIR}/${basename}.tar.bz2" "${dirname%%=*}" - ;; - *) - CT_DoExecLog ALL mv "${module}" "${dirname}" - CT_DoExecLog ALL tar cjf "${CT_TARBALLS_DIR}/${basename}.tar.bz2" "${dirname:-${module}}" - ;; - esac - fi - CT_SaveLocal "${CT_TARBALLS_DIR}/${basename}.tar.bz2" - - CT_Popd - CT_DoExecLog ALL rm -rf "${tmp_dir}" -} - -# Check out from SVN, and build the associated tarball -# The tarball will be called ${basename}.tar.bz2 -# Prerequisite: either the server does not require password, -# or the user must already be logged in. -# 'rev' is the revision to retrieve -# Usage: CT_GetSVN [rev] -CT_GetSVN() { - local basename="$1" - local uri="$2" - local rev="$3" - - # First try locally, then the mirror - if CT_GetFile "${basename}"; then - # Got it! Return early! - return 0 - fi - - if [ "${CT_FORBID_DOWNLOAD}" = "y" ]; then - CT_DoLog WARN "Downloads forbidden, not trying svn retrieval" - return 1 - fi - - CT_MktempDir tmp_dir - CT_Pushd "${tmp_dir}" - - if ! CT_DoExecLog ALL svn export ${rev:+-r ${rev}} "${uri}" "${basename}"; then - CT_DoLog WARN "Could not retrieve '${basename}'" - return 1 - fi - CT_DoExecLog ALL tar cjf "${CT_TARBALLS_DIR}/${basename}.tar.bz2" "${basename}" - CT_SaveLocal "${CT_TARBALLS_DIR}/${basename}.tar.bz2" - - CT_Popd - CT_DoExecLog ALL rm -rf "${tmp_dir}" -} - -# Clone a git tree -# Tries the given URLs in turn until one can get cloned. No tarball will be created. -# Prerequisites: either the server does not require password, -# or the user has already taken any action to authenticate to the server. -# The cloned tree will *not* be stored in the local tarballs dir! -# Usage: CT_GetGit -CT_GetGit() { - local basename="${1}" - local cset="${2}" - local url="${3}" - local file="${basename}-${cset}.tar.gz" - local dir="${CT_TARBALLS_DIR}/${basename}-${cset}.git" - local dest="${CT_TARBALLS_DIR}/${file}" - local tmp="${CT_TARBALLS_DIR}/${file}.tmp-dl" - - # Do we alreadyhave it? - if CT_GetLocal "${file}"; then - return 0 - fi - # Nope... - - if [ "${CT_FORBID_DOWNLOAD}" = "y" ]; then - CT_DoLog WARN "Downloads forbidden, not trying git retrieval" - return 1 - fi - - # Add URLs on the LAN mirror - # We subvert the normal download method, just to look for - # looking at the local mirror - if CT_GetFile "${basename}-${cset}" .tar.gz; then - return 0 - fi - - CT_DoLog EXTRA "Retrieving '${basename}-${cset}' (git)" - - # Remove potential left-over from a previous run - CT_DoExecLog ALL rm -rf "${tmp}.tar.gz" "${tmp}.tar" "${tmp}" "${dir}" - - if CT_DoExecLog ALL git clone "${url}" "${dir}"; then - # Yep, cloned OK - CT_Pushd "${dir}" - CT_DoExecLog ALL git archive --format=tar \ - --prefix="${basename}-${cset}/" \ - -o "${tmp}.tar" \ - "${cset}" - CT_DoExecLog ALL gzip -9 "${tmp}.tar" - CT_DoExecLog ALL mv -f "${tmp}.tar.gz" "${dest}" - CT_SaveLocal "${dest}" - CT_DoExecLog ALL rm -rf "${tmp}.tar.gz" "${tmp}.tar" "${tmp}" "${dir}" - CT_Popd - else - # Woops... - CT_DoExecLog ALL rm -rf "${dir}" - CT_DoLog Debug "Could not clone '${basename}'" - return 1 - fi -} - -# Extract a tarball -# Some tarballs need to be extracted in specific places. Eg.: glibc addons -# must be extracted in the glibc directory; uCLibc locales must be extracted -# in the extra/locale sub-directory of uClibc. This is taken into account -# by the caller, that did a 'cd' into the correct path before calling us -# and sets nochdir to 'nochdir'. -# Note also that this function handles the git trees! -# Usage: CT_Extract [nochdir] [options] -# where 'options' are dependent on the source (eg. git branch/tag...) -CT_Extract() { - local nochdir="$1" - local basename - local ext - local lzma_prog - local -a tar_opts - - if [ "${nochdir}" = "nochdir" ]; then - shift - nochdir="$(pwd)" - else - nochdir="${CT_SRC_DIR}" - fi - - basename="$1" - shift - - if ! ext="$(CT_GetFileExtension "${basename}")"; then - CT_DoLog WARN "'${basename}' not found in '${CT_TARBALLS_DIR}'" - return 1 - fi - local full_file="${CT_TARBALLS_DIR}/${basename}${ext}" - - # Check if already extracted - if [ -e "${CT_SRC_DIR}/.${basename}.extracted" ]; then - CT_DoLog DEBUG "Already extracted '${basename}'" - return 0 - fi - - # Check if previously partially extracted - if [ -e "${CT_SRC_DIR}/.${basename}.extracting" ]; then - CT_DoLog ERROR "The '${basename}' sources were partially extracted." - CT_DoLog ERROR "Please remove first:" - CT_DoLog ERROR " - the source dir for '${basename}', in '${CT_SRC_DIR}'" - CT_DoLog ERROR " - the file '${CT_SRC_DIR}/.${basename}.extracting'" - CT_Abort "I'll stop now to avoid any carnage..." - fi - CT_DoExecLog DEBUG touch "${CT_SRC_DIR}/.${basename}.extracting" - - CT_Pushd "${nochdir}" - - CT_DoLog EXTRA "Extracting '${basename}'" - CT_DoExecLog FILE mkdir -p "${basename}" - tar_opts=( "--strip-components=1" ) - tar_opts+=( "-C" "${basename}" ) - tar_opts+=( "-xv" ) - - # One note here: - # - lzma can be handled either with 'xz' or 'lzma' - # - we get lzma tarball only if either or both are available - # - so, if we get an lzma tarball, and either 'xz' or 'lzma' is - # missing, we can assume the other is available - if [ "${CT_CONFIGURE_has_lzma}" = "y" ]; then - lzma_prog="lzma -fdc" - else - lzma_prog="xz -fdc" - fi - case "${ext}" in - .tar.xz) xz -fdc "${full_file}" | CT_DoExecLog FILE tar "${tar_opts[@]}" -f -;; - .tar.lzma) ${lzma_prog} "${full_file}" | CT_DoExecLog FILE tar "${tar_opts[@]}" -f -;; - .tar.bz2) bzip2 -dc "${full_file}" | CT_DoExecLog FILE tar "${tar_opts[@]}" -f -;; - .tar.gz|.tgz) gzip -dc "${full_file}" | CT_DoExecLog FILE tar "${tar_opts[@]}" -f -;; - .tar) CT_DoExecLog FILE tar "${tar_opts[@]}" -f "${full_file}";; - .zip) CT_DoExecLog FILE unzip "${@}" "${full_file}";; - /.git) CT_ExtractGit "${basename}" "${@}";; - *) CT_DoLog WARN "Don't know how to handle '${basename}${ext}': unknown extension" - return 1 - ;; - esac - - # Don't mark as being extracted for git - case "${ext}" in - /.git) ;; - *) CT_DoExecLog DEBUG touch "${CT_SRC_DIR}/.${basename}.extracted";; - esac - CT_DoExecLog DEBUG rm -f "${CT_SRC_DIR}/.${basename}.extracting" - - CT_Popd -} - -# Create a working git clone of a local git repository -# Usage: CT_ExtractGit [ref] -# where 'ref' is the reference to use: -# the full name of a branch, like "remotes/origin/branch_name" -# a date as understandable by git, like "YYYY-MM-DD[ hh[:mm[:ss]]]" -# a tag name -# If 'ref' is not given, the current repository HEAD will be used -CT_ExtractGit() { - local basename="${1}" - local ref="${2}" - local repo - local ref_type - - # pushd now to be able to get git revlist in case ref is a date - repo="${CT_TARBALLS_DIR}/${basename}" - CT_Pushd "${repo}" - - # What kind of reference is ${ref} ? - if [ -z "${ref}" ]; then - ref_type=head - ref=$(git rev-list -n1 HEAD) - elif git tag |grep -E "^${ref}$" >/dev/null 2>&1; then - ref_type=tag - elif git branch -a --no-color |grep -E "^. ${ref}$" >/dev/null 2>&1; then - ref_type=branch - elif date -d "${ref}" >/dev/null 2>&1; then - ref_type=date - ref=$(git rev-list -n1 --before="${ref}") - else - CT_Abort "Reference '${ref}' is an incorrect git reference: neither tag, branch nor date" - fi - - CT_Popd - - CT_DoExecLog FILE rmdir "${basename}" - case "${ref_type}" in - branch) CT_DoExecLog FILE git clone -b "${ref}" "${repo}" "${basename}" ;; - *) CT_DoExecLog FILE git clone "${repo}" "${basename}" - CT_Pushd "${basename}" - CT_DoExecLog FILE git checkout "${ref}" - CT_Popd - ;; - esac -} - -# Patches the specified component -# See CT_Extract, above, for explanations on 'nochdir' -# Usage: CT_Patch [nochdir] -# If the package directory is *not* packagename-packageversion, then -# the caller must cd into the proper directory first, and call us -# with nochdir -CT_Patch() { - local nochdir="$1" - local pkgname - local version - local pkgdir - local base_file - local ver_file - local d - local -a patch_dirs - local bundled_patch_dir - local local_patch_dir - local bundled_exp_patch_dir - local local_exp_patch_dir - - if [ "${nochdir}" = "nochdir" ]; then - shift - pkgname="$1" - version="$2" - pkgdir="${pkgname}-${version}" - nochdir="$(pwd)" - else - pkgname="$1" - version="$2" - pkgdir="${pkgname}-${version}" - nochdir="${CT_SRC_DIR}/${pkgdir}" - fi - - # Check if already patched - if [ -e "${CT_SRC_DIR}/.${pkgdir}.patched" ]; then - CT_DoLog DEBUG "Already patched '${pkgdir}'" - return 0 - fi - - # Check if already partially patched - if [ -e "${CT_SRC_DIR}/.${pkgdir}.patching" ]; then - CT_DoLog ERROR "The '${pkgdir}' sources were partially patched." - CT_DoLog ERROR "Please remove first:" - CT_DoLog ERROR " - the source dir for '${pkgdir}', in '${CT_SRC_DIR}'" - CT_DoLog ERROR " - the file '${CT_SRC_DIR}/.${pkgdir}.extracted'" - CT_DoLog ERROR " - the file '${CT_SRC_DIR}/.${pkgdir}.patching'" - CT_Abort "I'll stop now to avoid any carnage..." - fi - touch "${CT_SRC_DIR}/.${pkgdir}.patching" - - CT_Pushd "${nochdir}" - - CT_DoLog EXTRA "Patching '${pkgdir}'" - - bundled_patch_dir="${CT_LIB_DIR}/patches/${pkgname}/${version}" - local_patch_dir="${CT_LOCAL_PATCH_DIR}/${pkgname}/${version}" - - # Check for experimental patches, if enabled. - if [ "${CT_EXPERIMENTAL_PATCHES}" = "y" ]; then - bundled_exp_patch_dir="${CT_LIB_DIR}/patches/experimental/${pkgname}/${version}" - local_exp_patch_dir="${CT_LOCAL_PATCH_DIR}/experimental/${pkgname}/${version}" - fi - - case "${CT_PATCH_ORDER}" in - bundled) patch_dirs=("${bundled_patch_dir}" "${bundled_exp_patch_dir}");; - local) patch_dirs=("${local_patch_dir}" "${local_exp_patch_dir}");; - bundled,local) patch_dirs=("${bundled_patch_dir}" "${bundled_exp_patch_dir}" "${local_patch_dir}" "${local_exp_patch_dir}");; - local,bundled) patch_dirs=("${local_patch_dir}" "${local_exp_patch_dir}" "${bundled_patch_dir}" "${bundled_exp_patch_dir}");; - none) patch_dirs=;; - esac - - for d in "${patch_dirs[@]}"; do - CT_DoLog DEBUG "Looking for patches in '${d}'..." - if [ -n "${d}" -a -d "${d}" ]; then - for p in "${d}"/*.patch; do - if [ -f "${p}" ]; then - CT_DoExecLog ALL patch --no-backup-if-mismatch -g0 -F1 -p1 -f -i "${p}" - fi - done - if [ "${CT_PATCH_SINGLE}" = "y" ]; then - break - fi - fi - done - - if [ "${CT_OVERIDE_CONFIG_GUESS_SUB}" = "y" ]; then - CT_DoLog ALL "Overiding config.guess and config.sub" - for cfg in config_guess config_sub; do - eval ${cfg}="${CT_LIB_DIR}/scripts/${cfg/_/.}" - [ -e "${CT_TOP_DIR}/scripts/${cfg/_/.}" ] && eval ${cfg}="${CT_TOP_DIR}/scripts/${cfg/_/.}" - # Can't use CT_DoExecLog because of the '{} \;' to be passed un-mangled to find - find . -type f -name "${cfg/_/.}" -exec cp -v "${!cfg}" {} \; |CT_DoLog ALL - done - fi - - CT_DoExecLog DEBUG touch "${CT_SRC_DIR}/.${pkgdir}.patched" - CT_DoExecLog DEBUG rm -f "${CT_SRC_DIR}/.${pkgdir}.patching" - - CT_Popd -} - -# Two wrappers to call config.(guess|sub) either from CT_TOP_DIR or CT_LIB_DIR. -# Those from CT_TOP_DIR, if they exist, will be be more recent than those from CT_LIB_DIR. -CT_DoConfigGuess() { - if [ -x "${CT_TOP_DIR}/scripts/config.guess" ]; then - "${CT_TOP_DIR}/scripts/config.guess" - else - "${CT_LIB_DIR}/scripts/config.guess" - fi -} - -CT_DoConfigSub() { - if [ -x "${CT_TOP_DIR}/scripts/config.sub" ]; then - "${CT_TOP_DIR}/scripts/config.sub" "$@" - else - "${CT_LIB_DIR}/scripts/config.sub" "$@" - fi -} - -# Compute the target tuple from what is provided by the user -# Usage: CT_DoBuildTargetTuple -# In fact this function takes the environment variables to build the target -# tuple. It is needed both by the normal build sequence, as well as the -# sample saving sequence. -CT_DoBuildTargetTuple() { - # Set the endianness suffix, and the default endianness gcc option - case "${CT_ARCH_ENDIAN}" in - big) - target_endian_eb=eb - target_endian_be=be - target_endian_el= - target_endian_le= - CT_ARCH_ENDIAN_CFLAG="-mbig-endian" - CT_ARCH_ENDIAN_LDFLAG="-Wl,-EB" - ;; - little) - target_endian_eb= - target_endian_be= - target_endian_el=el - target_endian_le=le - CT_ARCH_ENDIAN_CFLAG="-mlittle-endian" - CT_ARCH_ENDIAN_LDFLAG="-Wl,-EL" - ;; - esac - - # Set the bitness suffix - case "${CT_ARCH_BITNESS}" in - 32) - target_bits_32=32 - target_bits_64= - ;; - 64) - target_bits_32= - target_bits_64=64 - ;; - esac - - # Build the default architecture tuple part - CT_TARGET_ARCH="${CT_ARCH}${CT_ARCH_SUFFIX}" - - # Set defaults for the system part of the tuple. Can be overriden - # by architecture-specific values. - case "${CT_LIBC}" in - *glibc) CT_TARGET_SYS=gnu;; - uClibc) CT_TARGET_SYS=uclibc;; - *) CT_TARGET_SYS=elf;; - esac - - # Set the default values for ARCH, ABI, CPU, TUNE, FPU and FLOAT - unset CT_ARCH_ARCH_CFLAG CT_ARCH_ABI_CFLAG CT_ARCH_CPU_CFLAG CT_ARCH_TUNE_CFLAG CT_ARCH_FPU_CFLAG CT_ARCH_FLOAT_CFLAG - unset CT_ARCH_WITH_ARCH CT_ARCH_WITH_ABI CT_ARCH_WITH_CPU CT_ARCH_WITH_TUNE CT_ARCH_WITH_FPU CT_ARCH_WITH_FLOAT - [ "${CT_ARCH_ARCH}" ] && { CT_ARCH_ARCH_CFLAG="-march=${CT_ARCH_ARCH}"; CT_ARCH_WITH_ARCH="--with-arch=${CT_ARCH_ARCH}"; } - [ "${CT_ARCH_ABI}" ] && { CT_ARCH_ABI_CFLAG="-mabi=${CT_ARCH_ABI}"; CT_ARCH_WITH_ABI="--with-abi=${CT_ARCH_ABI}"; } - [ "${CT_ARCH_CPU}" ] && { CT_ARCH_CPU_CFLAG="-mcpu=${CT_ARCH_CPU}"; CT_ARCH_WITH_CPU="--with-cpu=${CT_ARCH_CPU}"; } - [ "${CT_ARCH_TUNE}" ] && { CT_ARCH_TUNE_CFLAG="-mtune=${CT_ARCH_TUNE}"; CT_ARCH_WITH_TUNE="--with-tune=${CT_ARCH_TUNE}"; } - [ "${CT_ARCH_FPU}" ] && { CT_ARCH_FPU_CFLAG="-mfpu=${CT_ARCH_FPU}"; CT_ARCH_WITH_FPU="--with-fpu=${CT_ARCH_FPU}"; } - - case "${CT_ARCH_FLOAT}" in - hard) - CT_ARCH_FLOAT_CFLAG="-mhard-float" - CT_ARCH_WITH_FLOAT="--with-float=hard" - ;; - soft) - CT_ARCH_FLOAT_CFLAG="-msoft-float" - CT_ARCH_WITH_FLOAT="--with-float=soft" - ;; - softfp) - CT_ARCH_FLOAT_CFLAG="-mfloat-abi=softfp" - CT_ARCH_WITH_FLOAT="--with-float=softfp" - ;; - esac - - # Build the default kernel tuple part - CT_TARGET_KERNEL="${CT_KERNEL}" - - # Overide the default values with the components specific settings - CT_DoArchTupleValues - CT_DoKernelTupleValues - - # Finish the target tuple construction - CT_TARGET="${CT_TARGET_ARCH}" - CT_TARGET="${CT_TARGET}${CT_TARGET_VENDOR:+-${CT_TARGET_VENDOR}}" - CT_TARGET="${CT_TARGET}${CT_TARGET_KERNEL:+-${CT_TARGET_KERNEL}}" - CT_TARGET="${CT_TARGET}${CT_TARGET_SYS:+-${CT_TARGET_SYS}}" - - # Sanity checks - __sed_alias="" - if [ -n "${CT_TARGET_ALIAS_SED_EXPR}" ]; then - __sed_alias=$(echo "${CT_TARGET}" |sed -r -e "${CT_TARGET_ALIAS_SED_EXPR}") - fi - case ":${CT_TARGET_VENDOR}:${CT_TARGET_ALIAS}:${__sed_alias}:" in - :*" "*:*:*:) CT_Abort "Don't use spaces in the vendor string, it breaks things.";; - :*"-"*:*:*:) CT_Abort "Don't use dashes in the vendor string, it breaks things.";; - :*:*" "*:*:) CT_Abort "Don't use spaces in the target alias, it breaks things.";; - :*:*:*" "*:) CT_Abort "Don't use spaces in the target sed transform, it breaks things.";; - esac - - # Canonicalise it - CT_TARGET=$(CT_DoConfigSub "${CT_TARGET}") - # Prepare the target CFLAGS - CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_ENDIAN_CFLAG}" - CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_ARCH_CFLAG}" - CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_ABI_CFLAG}" - CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_CPU_CFLAG}" - CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_TUNE_CFLAG}" - CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_FPU_CFLAG}" - CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_FLOAT_CFLAG}" - - # Now on for the target LDFLAGS - CT_ARCH_TARGET_LDFLAGS="${CT_ARCH_TARGET_LDFLAGS} ${CT_ARCH_ENDIAN_LDFLAG}" -} - -# This function does pause the build until the user strikes "Return" -# Usage: CT_DoPause [optional_message] -CT_DoPause() { - local foo - local message="${1:-Pausing for your pleasure}" - CT_DoLog INFO "${message}" - read -p "Press 'Enter' to continue, or Ctrl-C to stop..." foo >&6 - return 0 -} - -# This function creates a tarball of the specified directory, but -# only if it exists -# Usage: CT_DoTarballIfExists [extra_tar_options [...]] -CT_DoTarballIfExists() { - local dir="$1" - local tarball="$2" - shift 2 - local -a extra_tar_opts=( "$@" ) - local -a compress - - case "${CT_DEBUG_CT_SAVE_STEPS_GZIP}" in - y) compress=( gzip -c -3 - ); tar_ext=.gz;; - *) compress=( cat - ); tar_ext=;; - esac - - if [ -d "${dir}" ]; then - CT_DoLog DEBUG " Saving '${dir}'" - { tar c -C "${dir}" -v -f - "${extra_tar_opts[@]}" . \ - |"${compress[@]}" >"${tarball}.tar${tar_ext}" ; - } 2>&1 |sed -r -e 's/^/ /;' |CT_DoLog STATE - else - CT_DoLog STATE " Not saving '${dir}': does not exist" - fi -} - -# This function extracts a tarball to the specified directory, but -# only if the tarball exists -# Usage: CT_DoExtractTarballIfExists [extra_tar_options [...]] -CT_DoExtractTarballIfExists() { - local tarball="$1" - local dir="$2" - shift 2 - local -a extra_tar_opts=( "$@" ) - local -a uncompress - - case "${CT_DEBUG_CT_SAVE_STEPS_GZIP}" in - y) uncompress=( gzip -c -d ); tar_ext=.gz;; - *) uncompress=( cat ); tar_ext=;; - esac - - if [ -f "${tarball}.tar${tar_ext}" ]; then - CT_DoLog DEBUG " Restoring '${dir}'" - CT_DoForceRmdir "${dir}" - CT_DoExecLog DEBUG mkdir -p "${dir}" - { "${uncompress[@]}" "${tarball}.tar${tar_ext}" \ - |tar x -C "${dir}" -v -f - "${extra_tar_opts[@]}" ; - } 2>&1 |sed -r -e 's/^/ /;' |CT_DoLog STATE - else - CT_DoLog STATE " Not restoring '${dir}': does not exist" - fi -} - -# This function saves the state of the toolchain to be able to restart -# at any one point -# Usage: CT_DoSaveState -CT_DoSaveState() { - [ "${CT_DEBUG_CT_SAVE_STEPS}" = "y" ] || return 0 - local state_name="$1" - local state_dir="${CT_STATE_DIR}/${state_name}" - - # Log this to the log level required by the user - CT_DoLog ${CT_LOG_LEVEL_MAX} "Saving state to restart at step '${state_name}'..." - - rm -rf "${state_dir}" - mkdir -p "${state_dir}" - - CT_DoLog STATE " Saving environment and aliases" - # We must omit shell functions, and some specific bash variables - # that break when restoring the environment, later. We could do - # all the processing in the awk script, but a sed is easier... - set |awk ' - BEGIN { _p = 1; } - $0~/^[^ ]+ \(\)/ { _p = 0; } - _p == 1 - $0 == "}" { _p = 1; } - ' |sed -r -e '/^BASH_(ARGC|ARGV|LINENO|SOURCE|VERSINFO)=/d; - /^(UID|EUID)=/d; - /^(FUNCNAME|GROUPS|PPID|SHELLOPTS)=/d;' >"${state_dir}/env.sh" - - CT_DoTarballIfExists "${CT_BUILDTOOLS_PREFIX_DIR}" "${state_dir}/buildtools_dir" - CT_DoTarballIfExists "${CT_CONFIG_DIR}" "${state_dir}/config_dir" - CT_DoTarballIfExists "${CT_PREFIX_DIR}" "${state_dir}/prefix_dir" --exclude '*.log' - - CT_DoLog STATE " Saving log file" - exec >/dev/null - case "${CT_DEBUG_CT_SAVE_STEPS_GZIP}" in - y) gzip -3 -c "${tmp_log_file}" >"${state_dir}/log.gz";; - *) cat "${tmp_log_file}" >"${state_dir}/log";; - esac - exec >>"${tmp_log_file}" -} - -# This function restores a previously saved state -# Usage: CT_DoLoadState -CT_DoLoadState(){ - local state_name="$1" - local state_dir="${CT_STATE_DIR}/${state_name}" - local old_RESTART="${CT_RESTART}" - local old_STOP="${CT_STOP}" - - CT_TestOrAbort "The previous build did not reach the point where it could be restarted at '${CT_RESTART}'" -d "${state_dir}" - - # We need to do something special with the log file! - if [ "${CT_LOG_TO_FILE}" = "y" ]; then - exec >"${state_dir}/tail.log" - fi - - # Log this to the log level required by the user - CT_DoLog ${CT_LOG_LEVEL_MAX} "Restoring state at step '${state_name}', as requested." - - CT_DoExtractTarballIfExists "${state_dir}/prefix_dir" "${CT_PREFIX_DIR}" - CT_DoExtractTarballIfExists "${state_dir}/config_dir" "${CT_CONFIG_DIR}" - CT_DoExtractTarballIfExists "${state_dir}/buildtools_dir" "${CT_BUILDTOOLS_PREFIX_DIR}" - - # Restore the environment, discarding any error message - # (for example, read-only bash internals) - CT_DoLog STATE " Restoring environment" - . "${state_dir}/env.sh" >/dev/null 2>&1 || true - - # Restore the new RESTART and STOP steps - CT_RESTART="${old_RESTART}" - CT_STOP="${old_STOP}" - unset old_stop old_restart - - CT_DoLog STATE " Restoring log file" - exec >/dev/null - case "${CT_DEBUG_CT_SAVE_STEPS_GZIP}" in - y) gzip -dc "${state_dir}/log.gz" >"${tmp_log_file}";; - *) cat "${state_dir}/log" >"${tmp_log_file}";; - esac - cat "${state_dir}/tail.log" >>"${tmp_log_file}" - exec >>"${tmp_log_file}" - rm -f "${state_dir}/tail.log" -} diff --git a/scripts/functions.in b/scripts/functions.in new file mode 100644 index 0000000..51bf4a1 --- /dev/null +++ b/scripts/functions.in @@ -0,0 +1,1395 @@ +# This file contains some usefull common functions -*- sh -*- +# Copyright 2007 Yann E. MORIN +# Licensed under the GPL v2. See COPYING in the root of this package + +# Prepare the fault handler +CT_OnError() { + local ret=$? + local result + local old_trap + local intro + local file line func + local step step_depth + + # To avoid printing the backtace for each sub-shell + # up to the top-level, just remember we've dumped it + if [ ! -f "${CT_WORK_DIR}/backtrace" ]; then + touch "${CT_WORK_DIR}/backtrace" + + # Print steps backtrace + step_depth=${CT_STEP_COUNT} + CT_STEP_COUNT=1 # To have a zero-indentation + CT_DoLog ERROR "" + CT_DoLog ERROR ">>" + intro="Build failed" + for((step=step_depth; step>0; step--)); do + CT_DoLog ERROR ">> ${intro} in step '${CT_STEP_MESSAGE[${step}]}'" + intro=" called" + done + + # Print functions backtrace + intro="Error happened in" + CT_DoLog ERROR ">>" + for((depth=1; ${BASH_LINENO[$((${depth}-1))]}>0; depth++)); do + file="${BASH_SOURCE[${depth}]#${CT_LIB_DIR}/}" + func="${FUNCNAME[${depth}]}" + line="@${BASH_LINENO[${depth}-1]:-?}" + CT_DoLog ERROR ">> ${intro}: ${func}[${file}${line}]" + intro=" called from" + done + + # If the user asked for interactive debugging, dump him/her to a shell + if [ "${CT_DEBUG_INTERACTIVE}" = "y" ]; then + # We do not want this sub-shell exit status to be caught, because + # it is absolutely legit that it exits with non-zero. + # Save the trap handler to restore it after our debug-shell + old_trap="$(trap -p ERR)" + trap -- ERR + ( + exec >&6 2>&7 <&8 + printf "\r \n\nCurrent command" + if [ -n "${cur_cmd}" ]; then + printf ":\n %s\n" "${cur_cmd}" + else + printf " (unknown), " + fi + printf "exited with error code: %d\n" ${ret} + printf "Please fix it up and finish by exiting the shell with one of these values:\n" + printf " 1 fixed, continue with next build command\n" + if [ -n "${cur_cmd}" ]; then + printf " 2 repeat this build command\n" + fi + printf " 3 abort build\n\n" + while true; do + ${bash} --rcfile <(printf "PS1='ct-ng:\w> '\nPROMPT_COMMAND=''\n") -i + result=$? + case $result in + 1) printf "\nContinuing past the failed command.\n\n" + break + ;; + 2) if [ -n "${cur_cmd}" ]; then + printf "\nRe-trying last command.\n\n" + break + fi + ;; + 3) break;; + esac + printf "\nPlease exit with one of these values:\n" + printf " 1 fixed, continue with next build command\n" + if [ -n "${cur_cmd}" ]; then + printf " 2 repeat this build command\n" + fi + printf " 3 abort build\n" + done + exit $result + ) + result=$? + # Restore the trap handler + eval "${old_trap}" + case "${result}" in + 1) rm -f "${CT_WORK_DIR}/backtrace"; touch "${CT_BUILD_DIR}/skip"; return;; + 2) rm -f "${CT_WORK_DIR}/backtrace"; touch "${CT_BUILD_DIR}/repeat"; return;; + # 3 is an abort, continue... + esac + fi + fi + + # And finally, in top-level shell, print some hints + if [ ${BASH_SUBSHELL} -eq 0 ]; then + # Help diagnose the error + CT_STEP_COUNT=1 # To have a zero-indentation + CT_DoLog ERROR ">>" + if [ "${CT_LOG_TO_FILE}" = "y" ]; then + CT_DoLog ERROR ">> For more info on this error, look at the file: '${tmp_log_file#${CT_TOP_DIR}/}'" + fi + CT_DoLog ERROR ">> There is a list of known issues, some with workarounds, in:" + CT_DoLog ERROR ">> '${CT_DOC_DIR#${CT_TOP_DIR}/}/B - Known issues.txt'" + + CT_DoLog ERROR "" + CT_DoEnd ERROR + rm -f "${CT_WORK_DIR}/backtrace" + fi + exit $ret +} + +# Install the fault handler +trap CT_OnError ERR + +# Inherit the fault handler in subshells and functions +set -E + +# Make pipes fail on the _first_ failed command +# Not supported on bash < 3.x, but we need it, so drop the obsoleting bash-2.x +set -o pipefail + +# Don't hash commands' locations, and search every time it is requested. +# This is slow, but needed because of the static/shared core gcc which shall +# always match to shared if it exists, and only fallback to static if the +# shared is not found +set +o hashall + +# Log policy: +# - first of all, save stdout so we can see the live logs: fd #6 +# (also save stdin and stderr for use by CT_DEBUG_INTERACTIVE) +exec 6>&1 7>&2 8<&0 +# - then point stdout to the log file +tmp_log_file="${CT_TOP_DIR}/build.log" +rm -f "${tmp_log_file}" +exec >>"${tmp_log_file}" + +# The different log levels: +CT_LOG_LEVEL_ERROR=0 +CT_LOG_LEVEL_WARN=1 +CT_LOG_LEVEL_INFO=2 +CT_LOG_LEVEL_EXTRA=3 +CT_LOG_LEVEL_CFG=4 +CT_LOG_LEVEL_FILE=5 +CT_LOG_LEVEL_STATE=6 +CT_LOG_LEVEL_ALL=7 +CT_LOG_LEVEL_DEBUG=8 + +# Make it easy to use \n and ! +CR=$(printf "\n") +BANG='!' + +# A function to log what is happening +# Different log level are available: +# - ERROR: A serious, fatal error occurred +# - WARN: A non fatal, non serious error occurred, take your responsbility with the generated build +# - INFO: Informational messages +# - EXTRA: Extra informational messages +# - CFG: Output of various "./configure"-type scripts +# - FILE: File / archive unpacking. +# - STATE: State save & restore +# - ALL: Component's build messages +# - DEBUG: Internal debug messages +# Usage: CT_DoLog [message] +# If message is empty, then stdin will be logged. +CT_DoLog() { + local max_level LEVEL level cur_l cur_L + local l + eval max_level="\${CT_LOG_LEVEL_${CT_LOG_LEVEL_MAX}}" + # Set the maximum log level to DEBUG if we have none + [ -z "${max_level}" ] && max_level=${CT_LOG_LEVEL_DEBUG} + + LEVEL="$1"; shift + eval level="\${CT_LOG_LEVEL_${LEVEL}}" + + if [ $# -eq 0 ]; then + cat - + else + printf "%s\n" "${*}" + fi |( IFS="${CR}" # We want the full lines, even leading spaces + _prog_bar_cpt=0 + _prog_bar[0]='/' + _prog_bar[1]='-' + _prog_bar[2]='\' + _prog_bar[3]='|' + indent=$((2*CT_STEP_COUNT)) + while read line; do + case "${CT_LOG_SEE_TOOLS_WARN},${line}" in + y,*"warning:"*) cur_L=WARN; cur_l=${CT_LOG_LEVEL_WARN};; + y,*"WARNING:"*) cur_L=WARN; cur_l=${CT_LOG_LEVEL_WARN};; + *"error:"*) cur_L=ERROR; cur_l=${CT_LOG_LEVEL_ERROR};; + *"make["*"]: *** ["*) cur_L=ERROR; cur_l=${CT_LOG_LEVEL_ERROR};; + *) cur_L="${LEVEL}"; cur_l="${level}";; + esac + # There will always be a log file (stdout, fd #1), be it /dev/null + printf "[%-5s]%*s%s%s\n" "${cur_L}" "${indent}" " " "${line}" + if [ ${cur_l} -le ${max_level} ]; then + # Only print to console (fd #6) if log level is high enough. + printf "${CT_LOG_PROGRESS_BAR:+\r}[%-5s]%*s%s%s\n" "${cur_L}" "${indent}" " " "${line}" >&6 + fi + if [ "${CT_LOG_PROGRESS_BAR}" = "y" ]; then + printf "\r[%02d:%02d] %s " $((SECONDS/60)) $((SECONDS%60)) "${_prog_bar[$((_prog_bar_cpt/10))]}" >&6 + _prog_bar_cpt=$(((_prog_bar_cpt+1)%40)) + fi + done + ) + + return 0 +} + +# Execute an action, and log its messages +# It is possible to even log local variable assignments (a-la: var=val ./cmd opts) +# Usage: CT_DoExecLog [VAR=val...] [parameters...] +CT_DoExecLog() { + local level="$1" + local cur_cmd + local ret + shift + ( + for i in "$@"; do + cur_cmd+="'${i}' " + done + while true; do + case "${1}" in + *=*) eval export "'${1}'"; shift;; + *) break;; + esac + done + # This while-loop goes hand-in-hand with the ERR trap handler: + # - if the command terminates successfully, then we hit the break + # statement, and we exit the loop + # - if the command terminates in error, then the ERR handler kicks + # in, then: + # - if the user did *not* ask for interactive debugging, the ERR + # handler exits, and we hit the end of the sub-shell + # - if the user did ask for interactive debugging, the ERR handler + # spawns a shell. Upon termination of this shell, the ERR handler + # examines the exit status of the shell: + # - if 1, the ERR handler returns; then we hit the else statement, + # then the break, and we exit the 'while' loop, to continue the + # build; + # - if 2, the ERR handler touches the repeat file, and returns; + # then we hit the if statement, and we loop for one more + # iteration; + # - if 3, the ERR handler exits with the command's exit status, + # and we're dead; + # - for any other exit status of the shell, the ERR handler + # prints an informational message, and respawns the shell + # + # This allows a user to get an interactive shell that has the same + # environment (PATH and so on) that the failed command was ran with. + while true; do + rm -f "${CT_BUILD_DIR}/repeat" + CT_DoLog DEBUG "==> Executing: ${cur_cmd}" + "${@}" 2>&1 |CT_DoLog "${level}" + ret="${?}" + if [ -f "${CT_BUILD_DIR}/repeat" ]; then + rm -f "${CT_BUILD_DIR}/repeat" + continue + elif [ -f "${CT_BUILD_DIR}/skip" ]; then + rm -f "${CT_BUILD_DIR}/skip" + ret=0 + break + else + break + fi + done + exit ${ret} + ) + # Catch failure of the sub-shell + [ $? -eq 0 ] +} + +# Tail message to be logged whatever happens +# Usage: CT_DoEnd +CT_DoEnd() +{ + local level="$1" + CT_STOP_DATE=$(CT_DoDate +%s%N) + CT_STOP_DATE_HUMAN=$(CT_DoDate +%Y%m%d.%H%M%S) + if [ "${level}" != "ERROR" ]; then + CT_DoLog "${level:-INFO}" "Build completed at ${CT_STOP_DATE_HUMAN}" + fi + elapsed=$((CT_STOP_DATE-CT_STAR_DATE)) + elapsed_min=$((elapsed/(60*1000*1000*1000))) + elapsed_sec=$(printf "%02d" $(((elapsed%(60*1000*1000*1000))/(1000*1000*1000)))) + elapsed_csec=$(printf "%02d" $(((elapsed%(1000*1000*1000))/(10*1000*1000)))) + CT_DoLog ${level:-INFO} "(elapsed: ${elapsed_min}:${elapsed_sec}.${elapsed_csec})" +} + +# Remove entries referring to . and other relative paths +# Usage: CT_SanitizePath +CT_SanitizePath() { + local new + local p + local IFS=: + for p in $PATH; do + # Only accept absolute paths; + # Note: as a special case the empty string in PATH is equivalent to . + if [ -n "${p}" -a -z "${p%%/*}" ]; then + new="${new}${new:+:}${p}" + fi + done + PATH="${new}" +} + +# Sanitise the directory name contained in the variable passed as argument: +# - remove duplicate / +# Usage: CT_SanitiseVarDir CT_PREFIX_DIR +CT_SanitiseVarDir() { + local var + local old_dir + local new_dir + + for var in "$@"; do + eval "old_dir=\"\${${var}}\"" + new_dir="$( printf "${old_dir}" \ + |@@CT_sed@@ -r -e 's:/+:/:g;' \ + )" + eval "${var}=\"${new_dir}\"" + CT_DoLog DEBUG "Sanitised '${var}': '${old_dir}' -> '${new_dir}'" + done +} + +# Abort the execution with an error message +# Usage: CT_Abort +CT_Abort() { + CT_DoLog ERROR "$1" + false +} + +# Test a condition, and print a message if satisfied +# Usage: CT_Test +CT_Test() { + local ret + local m="$1" + shift + CT_DoLog DEBUG "Testing '! ( $* )'" + test "$@" && CT_DoLog WARN "$m" + return 0 +} + +# Test a condition, and abort with an error message if satisfied +# Usage: CT_TestAndAbort +CT_TestAndAbort() { + local m="$1" + shift + CT_DoLog DEBUG "Testing '! ( $* )'" + test "$@" && CT_Abort "$m" + return 0 +} + +# Test a condition, and abort with an error message if not satisfied +# Usage: CT_TestAndAbort +CT_TestOrAbort() { + local m="$1" + shift + CT_DoLog DEBUG "Testing '$*'" + test "$@" || CT_Abort "$m" + return 0 +} + +# Test the presence of a tool, or abort if not found +# Usage: CT_HasOrAbort +CT_HasOrAbort() { + CT_TestAndAbort "'${1}' not found and needed for successful toolchain build." -z "$(CT_Which "${1}")" + return 0 +} + +# Search a program: wrap "which" for those system where "which" +# verbosely says there is no match (such as on Mandriva). +# Usage: CT_Which +CT_Which() { + which "$1" 2>/dev/null || true +} + +# Get current date with nanosecond precision +# On those system not supporting nanosecond precision, faked with rounding down +# to the highest entire second +# Usage: CT_DoDate +CT_DoDate() { + date "$1" |@@CT_sed@@ -r -e 's/%?N$/000000000/;' +} + +CT_STEP_COUNT=1 +CT_STEP_MESSAGE[${CT_STEP_COUNT}]="(top-level)" +# Memorise a step being done so that any error is caught +# Usage: CT_DoStep +CT_DoStep() { + local start=$(CT_DoDate +%s%N) + CT_DoLog "$1" "=================================================================" + CT_DoLog "$1" "$2" + CT_STEP_COUNT=$((CT_STEP_COUNT+1)) + CT_STEP_LEVEL[${CT_STEP_COUNT}]="$1"; shift + CT_STEP_START[${CT_STEP_COUNT}]="${start}" + CT_STEP_MESSAGE[${CT_STEP_COUNT}]="$1" + return 0 +} + +# End the step just being done +# Usage: CT_EndStep +CT_EndStep() { + local stop=$(CT_DoDate +%s%N) + local duration=$(printf "%032d" $((stop-${CT_STEP_START[${CT_STEP_COUNT}]})) |@@CT_sed@@ -r -e 's/([[:digit:]]{2})[[:digit:]]{7}$/\.\1/; s/^0+//; s/^\./0\./;') + local elapsed=$(printf "%02d:%02d" $((SECONDS/60)) $((SECONDS%60))) + local level="${CT_STEP_LEVEL[${CT_STEP_COUNT}]}" + local message="${CT_STEP_MESSAGE[${CT_STEP_COUNT}]}" + CT_STEP_COUNT=$((CT_STEP_COUNT-1)) + CT_DoLog "${level}" "${message}: done in ${duration}s (at ${elapsed})" + return 0 +} + +# Pushes into a directory, and pops back +CT_Pushd() { + CT_DoLog DEBUG "Entering '$1'" + pushd "$1" >/dev/null 2>&1 +} +CT_Popd() { + popd >/dev/null 2>&1 +} + +# Create a dir and cd or pushd into it +# Usage: CT_mkdir_cd +# CT_mkdir_pushd +CT_mkdir_cd() { + local dir="${1}" + + mkdir -p "${dir}" + cd "${dir}" +} +CT_mkdir_pushd() { + local dir="${1}" + + mkdir -p "${dir}" + CT_Pushd "${dir}" +} + +# Creates a temporary directory +# $1: variable to assign to +# Usage: CT_MktempDir foo +CT_MktempDir() { + # Some mktemp do not allow more than 6 Xs + eval "$1"=$(mktemp -q -d "${CT_BUILD_DIR}/tmp.XXXXXX") + CT_TestOrAbort "Could not make temporary directory" -n "${!1}" -a -d "${!1}" + CT_DoLog DEBUG "Made temporary directory '${!1}'" + return 0 +} + +# Removes one or more directories, even if it is read-only, or its parent is +# Usage: CT_DoForceRmdir dir [...] +CT_DoForceRmdir() { + local dir + local mode + for dir in "${@}"; do + [ -d "${dir}" ] || continue + case "$CT_SYS_OS" in + Linux|CYGWIN*) + mode="$(stat -c '%a' "$(dirname "${dir}")")" + ;; + Darwin|*BSD) + mode="$(stat -f '%Lp' "$(dirname "${dir}")")" + ;; + *) + CT_Abort "Unhandled host OS $CT_SYS_OS" + ;; + esac + CT_DoExecLog ALL chmod u+w "$(dirname "${dir}")" + CT_DoExecLog ALL chmod -R u+w "${dir}" + CT_DoExecLog ALL rm -rf "${dir}" + CT_DoExecLog ALL chmod ${mode} "$(dirname "${dir}")" + done +} + +# Echoes the specified string on stdout until the pipe breaks. +# Doesn't fail +# $1: string to echo +# Usage: CT_DoYes "" |make oldconfig +CT_DoYes() { + yes "$1" || true +} + +# Add the specified directory to LD_LIBRARY_PATH, and export it +# If the specified patch is already present, just export +# $1: path to add +# $2: add as 'first' or 'last' path, 'first' is assumed if $2 is empty +# Usage CT_SetLibPath /some/where/lib [first|last] +CT_SetLibPath() { + local path="$1" + local pos="$2" + + case ":${LD_LIBRARY_PATH}:" in + *:"${path}":*) ;; + *) case "${pos}" in + last) + CT_DoLog DEBUG "Adding '${path}' at end of LD_LIBRARY_PATH" + LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}${path}" + ;; + first|"") + CT_DoLog DEBUG "Adding '${path}' at start of LD_LIBRARY_PATH" + LD_LIBRARY_PATH="${path}${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" + ;; + *) + CT_Abort "Incorrect position '${pos}' to add '${path}' to LD_LIBRARY_PATH" + ;; + esac + ;; + esac + CT_DoLog DEBUG "==> LD_LIBRARY_PATH='${LD_LIBRARY_PATH}'" + export LD_LIBRARY_PATH +} + +# Build up the list of allowed tarball extensions +# Add them in the prefered order; most preferred comes first +CT_DoListTarballExt() { + if [ "${CT_CONFIGURE_has_xz}" = "y" ]; then + printf ".tar.xz\n" + fi + if [ "${CT_CONFIGURE_has_lzma}" = "y" \ + -o "${CT_CONFIGURE_has_xz}" = "y" ]; then + printf ".tar.lzma\n" + fi + printf ".tar.bz2\n" + printf ".tar.gz\n.tgz\n" + printf ".tar\n" + printf ".zip\n" +} + +# Get the file name extension of a component +# Usage: CT_GetFileExtension [extension] +# If found, echoes the extension to stdout, and return 0 +# If not found, echoes nothing on stdout, and return !0. +CT_GetFileExtension() { + local ext + local file="$1" + shift + local first_ext="$1" + + # we need to also check for an empty extension for those very + # peculiar components that don't have one (such as sstrip from + # buildroot). + for ext in ${first_ext} $(CT_DoListTarballExt) /.git ''; do + if [ -e "${CT_TARBALLS_DIR}/${file}${ext}" ]; then + echo "${ext}" + exit 0 + fi + done + + exit 1 +} + +# Try to retrieve the specified URL (HTTP or FTP) +# Usage: CT_DoGetFile +# This functions always returns true (0), as it can be legitimate not +# to find the requested URL (think about snapshots, different layouts +# for different gcc versions, etc...). +CT_DoGetFile() { + local url="${1}" + local dest="${CT_TARBALLS_DIR}/${url##*/}" + local tmp="${dest}.tmp-dl" + + # Remove potential left-over from a previous run + rm -f "${tmp}" + + # We also retry a few times, in case there is a transient error (eg. behind + # a dynamic IP that changes during the transfer...) + # With automated download as we are doing, it can be very dangerous to + # continue the downloads. It's far better to simply overwrite the + # destination file. + # Some company networks have firewalls to connect to the internet, but it's + # not easy to detect them, so force a global ${CT_CONNECT_TIMEOUT}-second + # timeout. + if [ ${CT_CONNECT_TIMEOUT} = -1 ]; then + T= + else + T="-T ${CT_CONNECT_TIMEOUT}" + fi + if CT_DoExecLog ALL wget --passive-ftp --tries=3 -nc \ + --progress=dot:binary \ + ${T} \ + -O "${tmp}" \ + "${url}" + then + # Success, we got it, good! + mv "${tmp}" "${dest}" + CT_DoLog DEBUG "Got it from: \"${url}\"" + else + # Woops... + rm -f "${tmp}" + CT_DoLog DEBUG "Not at this location: \"${url}\"" + fi +} + +# This function tries to retrieve a tarball form a local directory +# Usage: CT_GetLocal [.extension] +CT_GetLocal() { + local basename="$1" + local first_ext="$2" + local ext + + # Do we already have it in *our* tarballs dir? + if ext="$( CT_GetFileExtension "${basename}" ${first_ext} )"; then + CT_DoLog DEBUG "Already have '${basename}'" + return 0 + fi + + if [ -n "${CT_LOCAL_TARBALLS_DIR}" ]; then + CT_DoLog DEBUG "Trying to retrieve an already downloaded copy of '${basename}'" + # We'd rather have a bzip2'ed tarball, then gzipped tarball, plain tarball, + # or, as a failover, a file without extension. + for ext in ${first_ext} $(CT_DoListTarballExt) ''; do + CT_DoLog DEBUG "Trying '${CT_LOCAL_TARBALLS_DIR}/${basename}${ext}'" + if [ -r "${CT_LOCAL_TARBALLS_DIR}/${basename}${ext}" -a \ + "${CT_FORCE_DOWNLOAD}" != "y" ]; then + CT_DoLog DEBUG "Got '${basename}' from local storage" + CT_DoExecLog ALL ln -s "${CT_LOCAL_TARBALLS_DIR}/${basename}${ext}" "${CT_TARBALLS_DIR}/${basename}${ext}" + return 0 + fi + done + fi + return 1 +} + +# This function gets the custom source from either a tarball or directory +# Usage: CT_GetCustom +CT_GetCustom() { + local custom_component="$1" + local custom_version="$2" + local custom_location="$3" + local custom_name="${custom_component}-${custom_version}" + + CT_TestAndAbort "${custom_name}: CT_CUSTOM_LOCATION_ROOT_DIR or ${custom_component}'s CUSTOM_LOCATION must be set." \ + -z "${CT_CUSTOM_LOCATION_ROOT_DIR}" -a -z "${custom_location}" + + if [ -n "${CT_CUSTOM_LOCATION_ROOT_DIR}" \ + -a -z "${custom_location}" ]; then + custom_location="${CT_CUSTOM_LOCATION_ROOT_DIR}/${custom_component}" + fi + + CT_DoLog EXTRA "Using '${custom_name}' from custom location" + if [ ! -d "${custom_location}" ]; then + # We need to know the custom tarball extension, + # so we can create a properly-named symlink, which + # we use later on in 'extract' + case "${custom_location}" in + *.tar.xz) custom_name="${custom_name}.tar.xz";; + *.tar.bz2) custom_name="${custom_name}.tar.bz2";; + *.tar.gz|*.tgz) custom_name="${custom_name}.tar.gz";; + *.tar) custom_name="${custom_name}.tar";; + *) CT_Abort "Unknown extension for custom tarball '${custom_location}'";; + esac + CT_DoExecLog DEBUG ln -sf "${custom_location}" \ + "${CT_TARBALLS_DIR}/${custom_name}" + else + CT_DoExecLog DEBUG ln -snf "${custom_location}" \ + "${CT_SRC_DIR}/${custom_name}" + fi +} + +# This function saves the specified to local storage if possible, +# and if so, symlinks it for later usage +# Usage: CT_SaveLocal +CT_SaveLocal() { + local file="$1" + local basename="${file##*/}" + + if [ "${CT_SAVE_TARBALLS}" = "y" ]; then + CT_DoLog EXTRA "Saving '${basename}' to local storage" + # The file may already exist if downloads are forced: remove it first + CT_DoExecLog ALL rm -f "${CT_LOCAL_TARBALLS_DIR}/${basename}" + CT_DoExecLog ALL mv -f "${file}" "${CT_LOCAL_TARBALLS_DIR}" + CT_DoExecLog ALL ln -s "${CT_LOCAL_TARBALLS_DIR}/${basename}" "${file}" + fi +} + +# Download the file from one of the URLs passed as argument +# Usage: CT_GetFile [.extension] [url ...] +CT_GetFile() { + local ext + local -a URLS + local url + local file="$1" + local first_ext + shift + # If next argument starts with a dot, then this is not an URL, + # and we can consider that it is a preferred extension. + case "$1" in + .*) first_ext="$1" + shift + ;; + esac + + # Does it exist localy? + if CT_GetLocal "${file}" ${first_ext}; then + return 0 + fi + # No, it does not... + + # If not allowed to download from the Internet, don't + if [ "${CT_FORBID_DOWNLOAD}" = "y" ]; then + CT_DoLog DEBUG "Not allowed to download from the Internet, aborting ${file} download" + return 1 + fi + + # Try to retrieve the file + CT_DoLog EXTRA "Retrieving '${file}'" + + # Add URLs on the LAN mirror + if [ "${CT_USE_MIRROR}" = "y" ]; then + CT_TestOrAbort "Please set the mirror base URL" -n "${CT_MIRROR_BASE_URL}" + URLS+=( "${CT_MIRROR_BASE_URL}/${file%-*}" ) + URLS+=( "${CT_MIRROR_BASE_URL}" ) + fi + + if [ "${CT_FORCE_MIRROR}" != "y" ]; then + URLS+=( "${@}" ) + fi + + # Scan all URLs in turn, and try to grab a tarball from there + # Do *not* try git trees (ext=/.git), this is handled in a specific + # wrapper, below + for ext in ${first_ext} $(CT_DoListTarballExt) ''; do + # Try all urls in turn + for url in "${URLS[@]}"; do + [ -n "${url}" ] || continue + CT_DoLog DEBUG "Trying '${url}/${file}${ext}'" + CT_DoGetFile "${url}/${file}${ext}" + if [ -f "${CT_TARBALLS_DIR}/${file}${ext}" ]; then + CT_DoLog DEBUG "Got '${file}' from the Internet" + CT_SaveLocal "${CT_TARBALLS_DIR}/${file}${ext}" + return 0 + fi + done + done + + # Just return error, someone may want to catch and handle the error + # (eg. glibc/eglibc add-ons can be missing). + return 1 +} + +# Checkout from CVS, and build the associated tarball +# The tarball will be called ${basename}.tar.bz2 +# Prerequisite: either the server does not require password, +# or the user must already be logged in. +# 'tag' is the tag to retrieve. Must be specified, but can be empty. +# If dirname is specified, then module will be renamed to dirname +# prior to building the tarball. +# Usage: CT_GetCVS [dirname[=subdir]] +# Note: if '=subdir' is given, then it is used instead of 'module'. +CT_GetCVS() { + local basename="$1" + local uri="$2" + local module="$3" + local tag="${4:+-r ${4}}" + local dirname="$5" + local tmp_dir + + # First try locally, then the mirror + if CT_GetFile "${basename}"; then + # Got it! Return early! + return 0 + fi + + if [ "${CT_FORBID_DOWNLOAD}" = "y" ]; then + CT_DoLog WARN "Downloads forbidden, not trying cvs retrieval" + return 1 + fi + + CT_MktempDir tmp_dir + CT_Pushd "${tmp_dir}" + + CT_DoExecLog ALL cvs -z 9 -d "${uri}" co -P ${tag} "${module}" + if [ -n "${dirname}" ]; then + case "${dirname}" in + *=*) + CT_DoExecLog DEBUG mv "${dirname#*=}" "${dirname%%=*}" + CT_DoExecLog ALL tar cjf "${CT_TARBALLS_DIR}/${basename}.tar.bz2" "${dirname%%=*}" + ;; + *) + CT_DoExecLog ALL mv "${module}" "${dirname}" + CT_DoExecLog ALL tar cjf "${CT_TARBALLS_DIR}/${basename}.tar.bz2" "${dirname:-${module}}" + ;; + esac + fi + CT_SaveLocal "${CT_TARBALLS_DIR}/${basename}.tar.bz2" + + CT_Popd + CT_DoExecLog ALL rm -rf "${tmp_dir}" +} + +# Check out from SVN, and build the associated tarball +# The tarball will be called ${basename}.tar.bz2 +# Prerequisite: either the server does not require password, +# or the user must already be logged in. +# 'rev' is the revision to retrieve +# Usage: CT_GetSVN [rev] +CT_GetSVN() { + local basename="$1" + local uri="$2" + local rev="$3" + + # First try locally, then the mirror + if CT_GetFile "${basename}"; then + # Got it! Return early! + return 0 + fi + + if [ "${CT_FORBID_DOWNLOAD}" = "y" ]; then + CT_DoLog WARN "Downloads forbidden, not trying svn retrieval" + return 1 + fi + + CT_MktempDir tmp_dir + CT_Pushd "${tmp_dir}" + + if ! CT_DoExecLog ALL svn export ${rev:+-r ${rev}} "${uri}" "${basename}"; then + CT_DoLog WARN "Could not retrieve '${basename}'" + return 1 + fi + CT_DoExecLog ALL tar cjf "${CT_TARBALLS_DIR}/${basename}.tar.bz2" "${basename}" + CT_SaveLocal "${CT_TARBALLS_DIR}/${basename}.tar.bz2" + + CT_Popd + CT_DoExecLog ALL rm -rf "${tmp_dir}" +} + +# Clone a git tree +# Tries the given URLs in turn until one can get cloned. No tarball will be created. +# Prerequisites: either the server does not require password, +# or the user has already taken any action to authenticate to the server. +# The cloned tree will *not* be stored in the local tarballs dir! +# Usage: CT_GetGit +CT_GetGit() { + local basename="${1}" + local cset="${2}" + local url="${3}" + local file="${basename}-${cset}.tar.gz" + local dir="${CT_TARBALLS_DIR}/${basename}-${cset}.git" + local dest="${CT_TARBALLS_DIR}/${file}" + local tmp="${CT_TARBALLS_DIR}/${file}.tmp-dl" + + # Do we alreadyhave it? + if CT_GetLocal "${file}"; then + return 0 + fi + # Nope... + + if [ "${CT_FORBID_DOWNLOAD}" = "y" ]; then + CT_DoLog WARN "Downloads forbidden, not trying git retrieval" + return 1 + fi + + # Add URLs on the LAN mirror + # We subvert the normal download method, just to look for + # looking at the local mirror + if CT_GetFile "${basename}-${cset}" .tar.gz; then + return 0 + fi + + CT_DoLog EXTRA "Retrieving '${basename}-${cset}' (git)" + + # Remove potential left-over from a previous run + CT_DoExecLog ALL rm -rf "${tmp}.tar.gz" "${tmp}.tar" "${tmp}" "${dir}" + + if CT_DoExecLog ALL git clone "${url}" "${dir}"; then + # Yep, cloned OK + CT_Pushd "${dir}" + CT_DoExecLog ALL git archive --format=tar \ + --prefix="${basename}-${cset}/" \ + -o "${tmp}.tar" \ + "${cset}" + CT_DoExecLog ALL gzip -9 "${tmp}.tar" + CT_DoExecLog ALL mv -f "${tmp}.tar.gz" "${dest}" + CT_SaveLocal "${dest}" + CT_DoExecLog ALL rm -rf "${tmp}.tar.gz" "${tmp}.tar" "${tmp}" "${dir}" + CT_Popd + else + # Woops... + CT_DoExecLog ALL rm -rf "${dir}" + CT_DoLog Debug "Could not clone '${basename}'" + return 1 + fi +} + +# Extract a tarball +# Some tarballs need to be extracted in specific places. Eg.: glibc addons +# must be extracted in the glibc directory; uCLibc locales must be extracted +# in the extra/locale sub-directory of uClibc. This is taken into account +# by the caller, that did a 'cd' into the correct path before calling us +# and sets nochdir to 'nochdir'. +# Note also that this function handles the git trees! +# Usage: CT_Extract [nochdir] [options] +# where 'options' are dependent on the source (eg. git branch/tag...) +CT_Extract() { + local nochdir="$1" + local basename + local ext + local lzma_prog + local -a tar_opts + + if [ "${nochdir}" = "nochdir" ]; then + shift + nochdir="$(pwd)" + else + nochdir="${CT_SRC_DIR}" + fi + + basename="$1" + shift + + if ! ext="$(CT_GetFileExtension "${basename}")"; then + CT_DoLog WARN "'${basename}' not found in '${CT_TARBALLS_DIR}'" + return 1 + fi + local full_file="${CT_TARBALLS_DIR}/${basename}${ext}" + + # Check if already extracted + if [ -e "${CT_SRC_DIR}/.${basename}.extracted" ]; then + CT_DoLog DEBUG "Already extracted '${basename}'" + return 0 + fi + + # Check if previously partially extracted + if [ -e "${CT_SRC_DIR}/.${basename}.extracting" ]; then + CT_DoLog ERROR "The '${basename}' sources were partially extracted." + CT_DoLog ERROR "Please remove first:" + CT_DoLog ERROR " - the source dir for '${basename}', in '${CT_SRC_DIR}'" + CT_DoLog ERROR " - the file '${CT_SRC_DIR}/.${basename}.extracting'" + CT_Abort "I'll stop now to avoid any carnage..." + fi + CT_DoExecLog DEBUG touch "${CT_SRC_DIR}/.${basename}.extracting" + + CT_Pushd "${nochdir}" + + CT_DoLog EXTRA "Extracting '${basename}'" + CT_DoExecLog FILE mkdir -p "${basename}" + tar_opts=( "--strip-components=1" ) + tar_opts+=( "-C" "${basename}" ) + tar_opts+=( "-xv" ) + + # One note here: + # - lzma can be handled either with 'xz' or 'lzma' + # - we get lzma tarball only if either or both are available + # - so, if we get an lzma tarball, and either 'xz' or 'lzma' is + # missing, we can assume the other is available + if [ "${CT_CONFIGURE_has_lzma}" = "y" ]; then + lzma_prog="lzma -fdc" + else + lzma_prog="xz -fdc" + fi + case "${ext}" in + .tar.xz) xz -fdc "${full_file}" | CT_DoExecLog FILE tar "${tar_opts[@]}" -f -;; + .tar.lzma) ${lzma_prog} "${full_file}" | CT_DoExecLog FILE tar "${tar_opts[@]}" -f -;; + .tar.bz2) bzip2 -dc "${full_file}" | CT_DoExecLog FILE tar "${tar_opts[@]}" -f -;; + .tar.gz|.tgz) gzip -dc "${full_file}" | CT_DoExecLog FILE tar "${tar_opts[@]}" -f -;; + .tar) CT_DoExecLog FILE tar "${tar_opts[@]}" -f "${full_file}";; + .zip) CT_DoExecLog FILE unzip "${@}" "${full_file}";; + /.git) CT_ExtractGit "${basename}" "${@}";; + *) CT_DoLog WARN "Don't know how to handle '${basename}${ext}': unknown extension" + return 1 + ;; + esac + + # Don't mark as being extracted for git + case "${ext}" in + /.git) ;; + *) CT_DoExecLog DEBUG touch "${CT_SRC_DIR}/.${basename}.extracted";; + esac + CT_DoExecLog DEBUG rm -f "${CT_SRC_DIR}/.${basename}.extracting" + + CT_Popd +} + +# Create a working git clone of a local git repository +# Usage: CT_ExtractGit [ref] +# where 'ref' is the reference to use: +# the full name of a branch, like "remotes/origin/branch_name" +# a date as understandable by git, like "YYYY-MM-DD[ hh[:mm[:ss]]]" +# a tag name +# If 'ref' is not given, the current repository HEAD will be used +CT_ExtractGit() { + local basename="${1}" + local ref="${2}" + local repo + local ref_type + + # pushd now to be able to get git revlist in case ref is a date + repo="${CT_TARBALLS_DIR}/${basename}" + CT_Pushd "${repo}" + + # What kind of reference is ${ref} ? + if [ -z "${ref}" ]; then + ref_type=head + ref=$(git rev-list -n1 HEAD) + elif git tag |@@CT_grep@@ -E "^${ref}$" >/dev/null 2>&1; then + ref_type=tag + elif git branch -a --no-color | @@CT_grep@@ -E "^. ${ref}$" >/dev/null 2>&1; then + ref_type=branch + elif date -d "${ref}" >/dev/null 2>&1; then + ref_type=date + ref=$(git rev-list -n1 --before="${ref}") + else + CT_Abort "Reference '${ref}' is an incorrect git reference: neither tag, branch nor date" + fi + + CT_Popd + + CT_DoExecLog FILE rmdir "${basename}" + case "${ref_type}" in + branch) CT_DoExecLog FILE git clone -b "${ref}" "${repo}" "${basename}" ;; + *) CT_DoExecLog FILE git clone "${repo}" "${basename}" + CT_Pushd "${basename}" + CT_DoExecLog FILE git checkout "${ref}" + CT_Popd + ;; + esac +} + +# Patches the specified component +# See CT_Extract, above, for explanations on 'nochdir' +# Usage: CT_Patch [nochdir] +# If the package directory is *not* packagename-packageversion, then +# the caller must cd into the proper directory first, and call us +# with nochdir +CT_Patch() { + local nochdir="$1" + local pkgname + local version + local pkgdir + local base_file + local ver_file + local d + local -a patch_dirs + local bundled_patch_dir + local local_patch_dir + local bundled_exp_patch_dir + local local_exp_patch_dir + + if [ "${nochdir}" = "nochdir" ]; then + shift + pkgname="$1" + version="$2" + pkgdir="${pkgname}-${version}" + nochdir="$(pwd)" + else + pkgname="$1" + version="$2" + pkgdir="${pkgname}-${version}" + nochdir="${CT_SRC_DIR}/${pkgdir}" + fi + + # Check if already patched + if [ -e "${CT_SRC_DIR}/.${pkgdir}.patched" ]; then + CT_DoLog DEBUG "Already patched '${pkgdir}'" + return 0 + fi + + # Check if already partially patched + if [ -e "${CT_SRC_DIR}/.${pkgdir}.patching" ]; then + CT_DoLog ERROR "The '${pkgdir}' sources were partially patched." + CT_DoLog ERROR "Please remove first:" + CT_DoLog ERROR " - the source dir for '${pkgdir}', in '${CT_SRC_DIR}'" + CT_DoLog ERROR " - the file '${CT_SRC_DIR}/.${pkgdir}.extracted'" + CT_DoLog ERROR " - the file '${CT_SRC_DIR}/.${pkgdir}.patching'" + CT_Abort "I'll stop now to avoid any carnage..." + fi + touch "${CT_SRC_DIR}/.${pkgdir}.patching" + + CT_Pushd "${nochdir}" + + CT_DoLog EXTRA "Patching '${pkgdir}'" + + bundled_patch_dir="${CT_LIB_DIR}/patches/${pkgname}/${version}" + local_patch_dir="${CT_LOCAL_PATCH_DIR}/${pkgname}/${version}" + + # Check for experimental patches, if enabled. + if [ "${CT_EXPERIMENTAL_PATCHES}" = "y" ]; then + bundled_exp_patch_dir="${CT_LIB_DIR}/patches/experimental/${pkgname}/${version}" + local_exp_patch_dir="${CT_LOCAL_PATCH_DIR}/experimental/${pkgname}/${version}" + fi + + case "${CT_PATCH_ORDER}" in + bundled) patch_dirs=("${bundled_patch_dir}" "${bundled_exp_patch_dir}");; + local) patch_dirs=("${local_patch_dir}" "${local_exp_patch_dir}");; + bundled,local) patch_dirs=("${bundled_patch_dir}" "${bundled_exp_patch_dir}" "${local_patch_dir}" "${local_exp_patch_dir}");; + local,bundled) patch_dirs=("${local_patch_dir}" "${local_exp_patch_dir}" "${bundled_patch_dir}" "${bundled_exp_patch_dir}");; + none) patch_dirs=;; + esac + + for d in "${patch_dirs[@]}"; do + CT_DoLog DEBUG "Looking for patches in '${d}'..." + if [ -n "${d}" -a -d "${d}" ]; then + for p in "${d}"/*.patch; do + if [ -f "${p}" ]; then + CT_DoExecLog ALL patch --no-backup-if-mismatch -g0 -F1 -p1 -f -i "${p}" + fi + done + if [ "${CT_PATCH_SINGLE}" = "y" ]; then + break + fi + fi + done + + if [ "${CT_OVERIDE_CONFIG_GUESS_SUB}" = "y" ]; then + CT_DoLog ALL "Overiding config.guess and config.sub" + for cfg in config_guess config_sub; do + eval ${cfg}="${CT_LIB_DIR}/scripts/${cfg/_/.}" + [ -e "${CT_TOP_DIR}/scripts/${cfg/_/.}" ] && eval ${cfg}="${CT_TOP_DIR}/scripts/${cfg/_/.}" + # Can't use CT_DoExecLog because of the '{} \;' to be passed un-mangled to find + find . -type f -name "${cfg/_/.}" -exec cp -v "${!cfg}" {} \; |CT_DoLog ALL + done + fi + + CT_DoExecLog DEBUG touch "${CT_SRC_DIR}/.${pkgdir}.patched" + CT_DoExecLog DEBUG rm -f "${CT_SRC_DIR}/.${pkgdir}.patching" + + CT_Popd +} + +# Two wrappers to call config.(guess|sub) either from CT_TOP_DIR or CT_LIB_DIR. +# Those from CT_TOP_DIR, if they exist, will be be more recent than those from CT_LIB_DIR. +CT_DoConfigGuess() { + if [ -x "${CT_TOP_DIR}/scripts/config.guess" ]; then + "${CT_TOP_DIR}/scripts/config.guess" + else + "${CT_LIB_DIR}/scripts/config.guess" + fi +} + +CT_DoConfigSub() { + if [ -x "${CT_TOP_DIR}/scripts/config.sub" ]; then + "${CT_TOP_DIR}/scripts/config.sub" "$@" + else + "${CT_LIB_DIR}/scripts/config.sub" "$@" + fi +} + +# Compute the target tuple from what is provided by the user +# Usage: CT_DoBuildTargetTuple +# In fact this function takes the environment variables to build the target +# tuple. It is needed both by the normal build sequence, as well as the +# sample saving sequence. +CT_DoBuildTargetTuple() { + # Set the endianness suffix, and the default endianness gcc option + case "${CT_ARCH_ENDIAN}" in + big) + target_endian_eb=eb + target_endian_be=be + target_endian_el= + target_endian_le= + CT_ARCH_ENDIAN_CFLAG="-mbig-endian" + CT_ARCH_ENDIAN_LDFLAG="-Wl,-EB" + ;; + little) + target_endian_eb= + target_endian_be= + target_endian_el=el + target_endian_le=le + CT_ARCH_ENDIAN_CFLAG="-mlittle-endian" + CT_ARCH_ENDIAN_LDFLAG="-Wl,-EL" + ;; + esac + + # Set the bitness suffix + case "${CT_ARCH_BITNESS}" in + 32) + target_bits_32=32 + target_bits_64= + ;; + 64) + target_bits_32= + target_bits_64=64 + ;; + esac + + # Build the default architecture tuple part + CT_TARGET_ARCH="${CT_ARCH}${CT_ARCH_SUFFIX}" + + # Set defaults for the system part of the tuple. Can be overriden + # by architecture-specific values. + case "${CT_LIBC}" in + *glibc) CT_TARGET_SYS=gnu;; + uClibc) CT_TARGET_SYS=uclibc;; + *) CT_TARGET_SYS=elf;; + esac + + # Set the default values for ARCH, ABI, CPU, TUNE, FPU and FLOAT + unset CT_ARCH_ARCH_CFLAG CT_ARCH_ABI_CFLAG CT_ARCH_CPU_CFLAG CT_ARCH_TUNE_CFLAG CT_ARCH_FPU_CFLAG CT_ARCH_FLOAT_CFLAG + unset CT_ARCH_WITH_ARCH CT_ARCH_WITH_ABI CT_ARCH_WITH_CPU CT_ARCH_WITH_TUNE CT_ARCH_WITH_FPU CT_ARCH_WITH_FLOAT + [ "${CT_ARCH_ARCH}" ] && { CT_ARCH_ARCH_CFLAG="-march=${CT_ARCH_ARCH}"; CT_ARCH_WITH_ARCH="--with-arch=${CT_ARCH_ARCH}"; } + [ "${CT_ARCH_ABI}" ] && { CT_ARCH_ABI_CFLAG="-mabi=${CT_ARCH_ABI}"; CT_ARCH_WITH_ABI="--with-abi=${CT_ARCH_ABI}"; } + [ "${CT_ARCH_CPU}" ] && { CT_ARCH_CPU_CFLAG="-mcpu=${CT_ARCH_CPU}"; CT_ARCH_WITH_CPU="--with-cpu=${CT_ARCH_CPU}"; } + [ "${CT_ARCH_TUNE}" ] && { CT_ARCH_TUNE_CFLAG="-mtune=${CT_ARCH_TUNE}"; CT_ARCH_WITH_TUNE="--with-tune=${CT_ARCH_TUNE}"; } + [ "${CT_ARCH_FPU}" ] && { CT_ARCH_FPU_CFLAG="-mfpu=${CT_ARCH_FPU}"; CT_ARCH_WITH_FPU="--with-fpu=${CT_ARCH_FPU}"; } + + case "${CT_ARCH_FLOAT}" in + hard) + CT_ARCH_FLOAT_CFLAG="-mhard-float" + CT_ARCH_WITH_FLOAT="--with-float=hard" + ;; + soft) + CT_ARCH_FLOAT_CFLAG="-msoft-float" + CT_ARCH_WITH_FLOAT="--with-float=soft" + ;; + softfp) + CT_ARCH_FLOAT_CFLAG="-mfloat-abi=softfp" + CT_ARCH_WITH_FLOAT="--with-float=softfp" + ;; + esac + + # Build the default kernel tuple part + CT_TARGET_KERNEL="${CT_KERNEL}" + + # Overide the default values with the components specific settings + CT_DoArchTupleValues + CT_DoKernelTupleValues + + # Finish the target tuple construction + CT_TARGET="${CT_TARGET_ARCH}" + CT_TARGET="${CT_TARGET}${CT_TARGET_VENDOR:+-${CT_TARGET_VENDOR}}" + CT_TARGET="${CT_TARGET}${CT_TARGET_KERNEL:+-${CT_TARGET_KERNEL}}" + CT_TARGET="${CT_TARGET}${CT_TARGET_SYS:+-${CT_TARGET_SYS}}" + + # Sanity checks + __sed_alias="" + if [ -n "${CT_TARGET_ALIAS_SED_EXPR}" ]; then + __sed_alias=$(echo "${CT_TARGET}" |@@CT_sed@@ -r -e "${CT_TARGET_ALIAS_SED_EXPR}") + fi + case ":${CT_TARGET_VENDOR}:${CT_TARGET_ALIAS}:${__sed_alias}:" in + :*" "*:*:*:) CT_Abort "Don't use spaces in the vendor string, it breaks things.";; + :*"-"*:*:*:) CT_Abort "Don't use dashes in the vendor string, it breaks things.";; + :*:*" "*:*:) CT_Abort "Don't use spaces in the target alias, it breaks things.";; + :*:*:*" "*:) CT_Abort "Don't use spaces in the target sed transform, it breaks things.";; + esac + + # Canonicalise it + CT_TARGET=$(CT_DoConfigSub "${CT_TARGET}") + # Prepare the target CFLAGS + CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_ENDIAN_CFLAG}" + CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_ARCH_CFLAG}" + CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_ABI_CFLAG}" + CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_CPU_CFLAG}" + CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_TUNE_CFLAG}" + CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_FPU_CFLAG}" + CT_ARCH_TARGET_CFLAGS="${CT_ARCH_TARGET_CFLAGS} ${CT_ARCH_FLOAT_CFLAG}" + + # Now on for the target LDFLAGS + CT_ARCH_TARGET_LDFLAGS="${CT_ARCH_TARGET_LDFLAGS} ${CT_ARCH_ENDIAN_LDFLAG}" +} + +# This function does pause the build until the user strikes "Return" +# Usage: CT_DoPause [optional_message] +CT_DoPause() { + local foo + local message="${1:-Pausing for your pleasure}" + CT_DoLog INFO "${message}" + read -p "Press 'Enter' to continue, or Ctrl-C to stop..." foo >&6 + return 0 +} + +# This function creates a tarball of the specified directory, but +# only if it exists +# Usage: CT_DoTarballIfExists [extra_tar_options [...]] +CT_DoTarballIfExists() { + local dir="$1" + local tarball="$2" + shift 2 + local -a extra_tar_opts=( "$@" ) + local -a compress + + case "${CT_DEBUG_CT_SAVE_STEPS_GZIP}" in + y) compress=( gzip -c -3 - ); tar_ext=.gz;; + *) compress=( cat - ); tar_ext=;; + esac + + if [ -d "${dir}" ]; then + CT_DoLog DEBUG " Saving '${dir}'" + { tar c -C "${dir}" -v -f - "${extra_tar_opts[@]}" . \ + |"${compress[@]}" >"${tarball}.tar${tar_ext}" ; + } 2>&1 |@@CT_sed@@ -r -e 's/^/ /;' |CT_DoLog STATE + else + CT_DoLog STATE " Not saving '${dir}': does not exist" + fi +} + +# This function extracts a tarball to the specified directory, but +# only if the tarball exists +# Usage: CT_DoExtractTarballIfExists [extra_tar_options [...]] +CT_DoExtractTarballIfExists() { + local tarball="$1" + local dir="$2" + shift 2 + local -a extra_tar_opts=( "$@" ) + local -a uncompress + + case "${CT_DEBUG_CT_SAVE_STEPS_GZIP}" in + y) uncompress=( gzip -c -d ); tar_ext=.gz;; + *) uncompress=( cat ); tar_ext=;; + esac + + if [ -f "${tarball}.tar${tar_ext}" ]; then + CT_DoLog DEBUG " Restoring '${dir}'" + CT_DoForceRmdir "${dir}" + CT_DoExecLog DEBUG mkdir -p "${dir}" + { "${uncompress[@]}" "${tarball}.tar${tar_ext}" \ + |tar x -C "${dir}" -v -f - "${extra_tar_opts[@]}" ; + } 2>&1 |@@CT_sed@@ -r -e 's/^/ /;' |CT_DoLog STATE + else + CT_DoLog STATE " Not restoring '${dir}': does not exist" + fi +} + +# This function saves the state of the toolchain to be able to restart +# at any one point +# Usage: CT_DoSaveState +CT_DoSaveState() { + [ "${CT_DEBUG_CT_SAVE_STEPS}" = "y" ] || return 0 + local state_name="$1" + local state_dir="${CT_STATE_DIR}/${state_name}" + + # Log this to the log level required by the user + CT_DoLog ${CT_LOG_LEVEL_MAX} "Saving state to restart at step '${state_name}'..." + + rm -rf "${state_dir}" + mkdir -p "${state_dir}" + + CT_DoLog STATE " Saving environment and aliases" + # We must omit shell functions, and some specific bash variables + # that break when restoring the environment, later. We could do + # all the processing in the awk script, but a sed is easier... + set |awk ' + BEGIN { _p = 1; } + $0~/^[^ ]+ \(\)/ { _p = 0; } + _p == 1 + $0 == "}" { _p = 1; } + ' |@@CT_sed@@ -r -e '/^BASH_(ARGC|ARGV|LINENO|SOURCE|VERSINFO)=/d; + /^(UID|EUID)=/d; + /^(FUNCNAME|GROUPS|PPID|SHELLOPTS)=/d;' >"${state_dir}/env.sh" + + CT_DoTarballIfExists "${CT_BUILDTOOLS_PREFIX_DIR}" "${state_dir}/buildtools_dir" + CT_DoTarballIfExists "${CT_CONFIG_DIR}" "${state_dir}/config_dir" + CT_DoTarballIfExists "${CT_PREFIX_DIR}" "${state_dir}/prefix_dir" --exclude '*.log' + + CT_DoLog STATE " Saving log file" + exec >/dev/null + case "${CT_DEBUG_CT_SAVE_STEPS_GZIP}" in + y) gzip -3 -c "${tmp_log_file}" >"${state_dir}/log.gz";; + *) cat "${tmp_log_file}" >"${state_dir}/log";; + esac + exec >>"${tmp_log_file}" +} + +# This function restores a previously saved state +# Usage: CT_DoLoadState +CT_DoLoadState(){ + local state_name="$1" + local state_dir="${CT_STATE_DIR}/${state_name}" + local old_RESTART="${CT_RESTART}" + local old_STOP="${CT_STOP}" + + CT_TestOrAbort "The previous build did not reach the point where it could be restarted at '${CT_RESTART}'" -d "${state_dir}" + + # We need to do something special with the log file! + if [ "${CT_LOG_TO_FILE}" = "y" ]; then + exec >"${state_dir}/tail.log" + fi + + # Log this to the log level required by the user + CT_DoLog ${CT_LOG_LEVEL_MAX} "Restoring state at step '${state_name}', as requested." + + CT_DoExtractTarballIfExists "${state_dir}/prefix_dir" "${CT_PREFIX_DIR}" + CT_DoExtractTarballIfExists "${state_dir}/config_dir" "${CT_CONFIG_DIR}" + CT_DoExtractTarballIfExists "${state_dir}/buildtools_dir" "${CT_BUILDTOOLS_PREFIX_DIR}" + + # Restore the environment, discarding any error message + # (for example, read-only bash internals) + CT_DoLog STATE " Restoring environment" + . "${state_dir}/env.sh" >/dev/null 2>&1 || true + + # Restore the new RESTART and STOP steps + CT_RESTART="${old_RESTART}" + CT_STOP="${old_STOP}" + unset old_stop old_restart + + CT_DoLog STATE " Restoring log file" + exec >/dev/null + case "${CT_DEBUG_CT_SAVE_STEPS_GZIP}" in + y) gzip -dc "${state_dir}/log.gz" >"${tmp_log_file}";; + *) cat "${state_dir}/log" >"${tmp_log_file}";; + esac + cat "${state_dir}/tail.log" >>"${tmp_log_file}" + exec >>"${tmp_log_file}" + rm -f "${state_dir}/tail.log" +} diff --git a/scripts/xldd.in b/scripts/xldd.in index c906240..4cbe333 100755 --- a/scripts/xldd.in +++ b/scripts/xldd.in @@ -159,7 +159,7 @@ fi sysroot="$( "${gcc}" -print-sysroot 2>/dev/null )" if [ -z "${sysroot}" ]; then sysroot="$( "${gcc}" -print-file-name=libc.so 2>/dev/null \ - |sed -r -e 's:/usr/lib/libc.so$::;' \ + |"${sed}" -r -e 's:/usr/lib/libc.so$::;' \ )" fi if [ -z "${sysroot}" ]; then -- 1.8.5.2 (Apple Git-48)