* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
@ 2016-09-14 17:38 Moritz Klammler
2016-09-14 18:12 ` Joseph Myers
0 siblings, 1 reply; 17+ messages in thread
From: Moritz Klammler @ 2016-09-14 17:38 UTC (permalink / raw)
To: GCC Patches; +Cc: Joseph Myers
Joseph Myers <joseph@codesourcery.com> writes:
> On Wed, 14 Sep 2016, Moritz Klammler wrote:
>
>> be cleaner to only include those checksums that are actually needed. On
>> the other hand, it means an increased maintenance burden each time the
>> version of the dependency is changed. In order to mitigate this and
>
> I really don't see it as an increased burden. The maintainer shouldn't be
> using the checksum files on the server at all. What they should do is:
>
> * Download the tar file from ftp.gnu.org (at least for GMP / MPFR / MPC),
> *verify the GPG signature* and test with it. (I'm not sure if the GNU
> keyring is currently published.) The GPG signatures on ftp.gnu.org are
> from the maintainer who uploaded the package, whereas the checksum files
> on gcc.gnu.org are automatically generated from cron. (I don't know if a
> secure way to download ISL from its origin has been added since
> <https://gcc.gnu.org/ml/gcc/2016-07/msg00003.html> raised the issue.)
>
> * Update the script and the to-be-checked-in checksums, using the file
> they just downloaded and verified the signature of.
>
> * Add the new file to the server before the script changes get checked in.
Ok, I didn't know about the workflow. Do you think I should dike the
`--strip-sums` option out again then?
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-09-14 17:38 Verify package integrity of downloaded prerequisites (partially fixes 61439) Moritz Klammler
@ 2016-09-14 18:12 ` Joseph Myers
0 siblings, 0 replies; 17+ messages in thread
From: Joseph Myers @ 2016-09-14 18:12 UTC (permalink / raw)
To: Moritz Klammler; +Cc: GCC Patches
On Wed, 14 Sep 2016, Moritz Klammler wrote:
> Ok, I didn't know about the workflow. Do you think I should dike the
> `--strip-sums` option out again then?
I don't see any use for such an option. Anyone changing the versions
should always have a copy of the new tarball (obtained securely if
possible) and should determine the sums from that.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-10-25 0:16 Moritz Klammler
@ 2016-10-25 15:18 ` Jeff Law
0 siblings, 0 replies; 17+ messages in thread
From: Jeff Law @ 2016-10-25 15:18 UTC (permalink / raw)
To: Moritz Klammler, Richard Biener; +Cc: GCC Patches
On 10/24/2016 06:16 PM, Moritz Klammler wrote:
> Jeff Law <law@redhat.com> writes:
>
>> On 10/24/2016 02:44 AM, Richard Biener wrote:
>>> On Fri, Oct 7, 2016 at 3:10 PM, Moritz Klammler <moritz@klammler.eu> wrote:
>>>> I would like to bump my patch that makes the
>>>> `contrib/download_prerequisites` script verify the checksums of the
>>>> downloaded packages and augments it with a few additional options.
>>>> All feedback I have received has been incorporated. Is it okay like
>>>> this?
>>>>
>>>> Below is again the latest iteration of the patch.
>>>
>>> Looks generally ok. Can you remove -eu from the /bin/sh command and
>>> leave the copyright in place?
>>>
>>> The patch is ok with that change. Do you have commit privileges?
>> I'm pretty sure Moritz doesn't have commit privs. So I reverted the
>> gratutious changes to the copyright notice and cobbled together a
>> changlog and committed the result.
>>
>> jeff
>>
>
> Thank you very much both of you.
>
> Indeed, I do not have commit privileges. In fact, this was my very
> first patch to be accepted. I did went through the copyright assignment
> process already, though.
>
> Now that the patch is applied, can this bug be closed?
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61439
Done. And ChangeLog updated to include appropriate tags.
jeff
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
@ 2016-10-25 0:16 Moritz Klammler
2016-10-25 15:18 ` Jeff Law
0 siblings, 1 reply; 17+ messages in thread
From: Moritz Klammler @ 2016-10-25 0:16 UTC (permalink / raw)
To: Jeff Law, Richard Biener; +Cc: GCC Patches
Jeff Law <law@redhat.com> writes:
> On 10/24/2016 02:44 AM, Richard Biener wrote:
>> On Fri, Oct 7, 2016 at 3:10 PM, Moritz Klammler <moritz@klammler.eu> wrote:
>>> I would like to bump my patch that makes the
>>> `contrib/download_prerequisites` script verify the checksums of the
>>> downloaded packages and augments it with a few additional options.
>>> All feedback I have received has been incorporated. Is it okay like
>>> this?
>>>
>>> Below is again the latest iteration of the patch.
>>
>> Looks generally ok. Can you remove -eu from the /bin/sh command and
>> leave the copyright in place?
>>
>> The patch is ok with that change. Do you have commit privileges?
> I'm pretty sure Moritz doesn't have commit privs. So I reverted the
> gratutious changes to the copyright notice and cobbled together a
> changlog and committed the result.
>
> jeff
>
Thank you very much both of you.
Indeed, I do not have commit privileges. In fact, this was my very
first patch to be accepted. I did went through the copyright assignment
process already, though.
Now that the patch is applied, can this bug be closed?
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61439
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-10-24 8:44 ` Richard Biener
@ 2016-10-24 16:06 ` Jeff Law
0 siblings, 0 replies; 17+ messages in thread
From: Jeff Law @ 2016-10-24 16:06 UTC (permalink / raw)
To: Richard Biener, Moritz Klammler; +Cc: GCC Patches
On 10/24/2016 02:44 AM, Richard Biener wrote:
> On Fri, Oct 7, 2016 at 3:10 PM, Moritz Klammler <moritz@klammler.eu> wrote:
>> I would like to bump my patch that makes the
>> `contrib/download_prerequisites` script verify the checksums of
>> the downloaded packages and augments it with a few additional options.
>> All feedback I have received has been incorporated. Is it okay like
>> this?
>>
>> Below is again the latest iteration of the patch.
>
> Looks generally ok. Can you remove -eu from the /bin/sh command and leave
> the copyright in place?
>
> The patch is ok with that change. Do you have commit privileges?
I'm pretty sure Moritz doesn't have commit privs. So I reverted the
gratutious changes to the copyright notice and cobbled together a
changlog and committed the result.
jeff
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-10-07 13:10 Moritz Klammler
@ 2016-10-24 8:44 ` Richard Biener
2016-10-24 16:06 ` Jeff Law
0 siblings, 1 reply; 17+ messages in thread
From: Richard Biener @ 2016-10-24 8:44 UTC (permalink / raw)
To: Moritz Klammler; +Cc: GCC Patches
On Fri, Oct 7, 2016 at 3:10 PM, Moritz Klammler <moritz@klammler.eu> wrote:
> I would like to bump my patch that makes the
> `contrib/download_prerequisites` script verify the checksums of
> the downloaded packages and augments it with a few additional options.
> All feedback I have received has been incorporated. Is it okay like
> this?
>
> Below is again the latest iteration of the patch.
Looks generally ok. Can you remove -eu from the /bin/sh command and leave
the copyright in place?
The patch is ok with that change. Do you have commit privileges?
Thanks,
Richard.
>
>
> Index: contrib/download_prerequisites
> ===================================================================
> --- contrib/download_prerequisites (revision 240137)
> +++ contrib/download_prerequisites (working copy)
> @@ -1,60 +1,215 @@
> -#! /bin/sh
> +#! /bin/sh -eu
> +#! -*- coding:utf-8; mode:shell-script; -*-
>
> -# Download some prerequisites needed by gcc.
> -# Run this from the top level of the gcc source tree and the gcc
> -# build will do the right thing.
> +# Download some prerequisites needed by GCC.
> +# Run this from the top level of the GCC source tree and the GCC build will do
> +# the right thing. Run it with the `--help` option for more information.
> #
> -# (C) 2010-2016 Free Software Foundation
> +# (C) 2016 Free Software Foundation
> #
> -# This program is free software: you can redistribute it and/or modify
> -# it under the terms of the GNU General Public License as published by
> -# the Free Software Foundation, either version 3 of the License, or
> -# (at your option) any later version.
> -#
> -# This program is distributed in the hope that it will be useful, but
> -# WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> -# General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program. If not, see http://www.gnu.org/licenses/.
> +# This program is free software: you can redistribute it and/or modify it under
> +# the terms of the GNU General Public License as published by the Free Software
> +# Foundation, either version 3 of the License, or (at your option) any later
> +# version.
> +#
> +# This program is distributed in the hope that it will be useful, but WITHOUT
> +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
> +# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
> +# details.
> +#
> +# You should have received a copy of the GNU General Public License along with
> +# this program. If not, see http://www.gnu.org/licenses/.
>
> -# If you want to disable Graphite loop optimizations while building GCC,
> -# DO NOT set GRAPHITE_LOOP_OPT as yes so that the isl package will not
> -# be downloaded.
> -GRAPHITE_LOOP_OPT=yes
>
> -if [ ! -e gcc/BASE-VER ] ; then
> - echo "You must run this script in the top level GCC source directory."
> - exit 1
> -fi
> +program='download_prerequisites'
> +version='(unversioned)'
>
> -# Necessary to build GCC.
> -MPFR=mpfr-3.1.4
> -GMP=gmp-6.1.0
> -MPC=mpc-1.0.3
> +# MAINTAINERS: If you update the package versions below, please
> +# remember to also update the files `contrib/prerequisites.sha512` and
> +# `contrib/prerequisites.md5` with the new checksums.
>
> -wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPFR.tar.bz2 || exit 1
> -tar xjf $MPFR.tar.bz2 || exit 1
> -if test -L mpfr; then rm -f mpfr; fi
> -ln -sf $MPFR mpfr || exit 1
> +gmp='gmp-6.1.0.tar.bz2'
> +mpfr='mpfr-3.1.4.tar.bz2'
> +mpc='mpc-1.0.3.tar.gz'
> +isl='isl-0.16.1.tar.bz2'
>
> -wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$GMP.tar.bz2 || exit 1
> -tar xjf $GMP.tar.bz2 || exit 1
> -if test -L gmp; then rm -f gmp; fi
> -ln -sf $GMP gmp || exit 1
> +base_url='ftp://gcc.gnu.org/pub/gcc/infrastructure/'
>
> -wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPC.tar.gz || exit 1
> -tar xzf $MPC.tar.gz || exit 1
> -if test -L mpc; then rm -f mpc; fi
> -ln -sf $MPC mpc || exit 1
> +echo_archives() {
> + echo "${gmp}"
> + echo "${mpfr}"
> + echo "${mpc}"
> + if [ ${graphite} -gt 0 ]; then echo "${isl}"; fi
> +}
>
> -# Necessary to build GCC with the Graphite loop optimizations.
> -if [ "$GRAPHITE_LOOP_OPT" = "yes" ] ; then
> - ISL=isl-0.16.1
> +graphite=1
> +verify=1
> +force=0
> +chksum='sha512'
> +directory='.'
>
> - wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$ISL.tar.bz2 || exit 1
> - tar xjf $ISL.tar.bz2 || exit 1
> - if test -L isl; then rm -f isl; fi
> - ln -sf $ISL isl || exit 1
> +helptext="usage: ${program} [OPTION...]
> +
> +Downloads some prerequisites needed by GCC. Run this from the top level of the
> +GCC source tree and the GCC build will do the right thing.
> +
> +The following options are available:
> +
> + --directory=DIR download and unpack packages into DIR instead of '.'
> + --force download again overwriting existing packages
> + --no-force do not download existing packages again (default)
> + --isl download ISL, needed for Graphite loop optimizations (default)
> + --graphite same as --isl
> + --no-isl don't download ISL
> + --no-graphite same as --no-isl
> + --verify verify package integrity after download (default)
> + --no-verify don't verify package integrity
> + --sha512 use SHA512 checksum to verify package integrity (default)
> + --md5 use MD5 checksum to verify package integrity
> + --help show this text and exit
> + --version show version information and exit
> +"
> +
> +versiontext="${program} ${version}
> +Copyright (C) 2016 Free Software Foundation, Inc.
> +This is free software; see the source for copying conditions. There is NO
> +warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
> +
> +die() {
> + echo "error: $@" >&2
> + exit 1
> +}
> +
> +for arg in "$@"
> +do
> + case "${arg}" in
> + --help)
> + echo "${helptext}"
> + exit
> + ;;
> + --version)
> + echo "${versiontext}"
> + exit
> + ;;
> + esac
> +done
> +unset arg
> +
> +argnext=
> +for arg in "$@"
> +do
> + if [ "x${argnext}" = x ]
> + then
> + case "${arg}" in
> + --directory)
> + argnext='directory'
> + ;;
> + --directory=*)
> + directory="${arg#--directory=}"
> + ;;
> + --force)
> + force=1
> + ;;
> + --no-force)
> + force=0
> + ;;
> + --isl|--graphite)
> + graphite=1
> + ;;
> + --no-isl|--no-graphite)
> + graphite=0
> + ;;
> + --verify)
> + verify=1
> + ;;
> + --no-verify)
> + verify=0
> + ;;
> + --sha512)
> + chksum='sha512'
> + verify=1
> + ;;
> + --md5)
> + chksum='md5'
> + verify=1
> + ;;
> + -*)
> + die "unknown option: ${arg}"
> + ;;
> + *)
> + die "too many arguments"
> + ;;
> + esac
> + else
> + case "${arg}" in
> + -*)
> + die "Missing argument for option --${argnext}"
> + ;;
> + esac
> + case "${argnext}" in
> + directory)
> + directory="${arg}"
> + ;;
> + *)
> + die "The impossible has happened"
> + ;;
> + esac
> + argnext=
> + fi
> +done
> +[ "x${argnext}" = x ] || die "Missing argument for option --${argnext}"
> +unset arg argnext
> +
> +[ -e ./gcc/BASE-VER ] \
> + || die "You must run this script in the top-level GCC source directory"
> +
> +[ -d "${directory}" ] \
> + || die "No such directory: ${directory}"
> +
> +for ar in $(echo_archives)
> +do
> + if [ ${force} -gt 0 ]; then rm -f "${directory}/${ar}"; fi
> + [ -e "${directory}/${ar}" ] \
> + || wget --no-verbose -O "${directory}/${ar}" "${base_url}${ar}" \
> + || die "Cannot download ${ar} from ${base_url}"
> +done
> +unset ar
> +
> +if [ ${verify} -gt 0 ]
> +then
> + chksumfile="contrib/prerequisites.${chksum}"
> + [ -r "${chksumfile}" ] || die "No checksums available"
> + for ar in $(echo_archives)
> + do
> + grep "${ar}" "${chksumfile}" \
> + | ( cd "${directory}" && "${chksum}sum" --check ) \
> + || die "Cannot verify integrity of possibly corrupted file ${ar}"
> + done
> + unset chksumfile
> fi
> +unset ar
> +
> +for ar in $(echo_archives)
> +do
> + package="${ar%.tar*}"
> + if [ ${force} -gt 0 ]; then rm -rf "${directory}/${package}"; fi
> + [ -e "${directory}/${package}" ] \
> + || ( cd "${directory}" && tar -xf "${ar}" ) \
> + || die "Cannot extract package from ${ar}"
> + unset package
> +done
> +unset ar
> +
> +for ar in $(echo_archives)
> +do
> + target="${directory}/${ar%.tar*}/"
> + linkname="${ar%-*}"
> + if [ ${force} -gt 0 ]; then rm -f "${linkname}"; fi
> + [ -e "${linkname}" ] \
> + || ln -s "${target}" "${linkname}" \
> + || die "Cannot create symbolic link ${linkname} --> ${target}"
> + unset target linkname
> +done
> +unset ar
> +
> +echo "All prerequisites downloaded successfully."
> Index: contrib/prerequisites.md5
> ===================================================================
> --- contrib/prerequisites.md5 (nonexistent)
> +++ contrib/prerequisites.md5 (working copy)
> @@ -0,0 +1,4 @@
> +86ee6e54ebfc4a90b643a65e402c4048 gmp-6.1.0.tar.bz2
> +b8a2f6b0e68bef46e53da2ac439e1cf4 mpfr-3.1.4.tar.bz2
> +d6a1d5f8ddea3abd2cc3e98f58352d26 mpc-1.0.3.tar.gz
> +ac1f25a0677912952718a51f5bc20f32 isl-0.16.1.tar.bz2
> Index: contrib/prerequisites.sha512
> ===================================================================
> --- contrib/prerequisites.sha512 (nonexistent)
> +++ contrib/prerequisites.sha512 (working copy)
> @@ -0,0 +1,4 @@
> +3c82aeab9c1596d4da8afac2eec38e429e84f3211e1a572cf8fd2b546493c44c039b922a1133eaaa48bd7f3e11dbe795a384e21ed95cbe3ecc58d7ac02246117 gmp-6.1.0.tar.bz2
> +51066066ff2c12ed2198605ecf68846b0c96b548adafa5b80e0c786d0df488411a5e8973358fce7192dc977ad4e68414cf14500e3c39746de62465eb145bb819 mpfr-3.1.4.tar.bz2
> +0028b76df130720c1fad7de937a0d041224806ce5ef76589f19c7b49d956071a683e2f20d154c192a231e69756b19e48208f2889b0c13950ceb7b3cfaf059a43 mpc-1.0.3.tar.gz
> +c188667a84dc5bdddb4ab7c35f89c91bf15a8171f4fcaf41301cf285fb7328846d9a367c096012fec4cc69d244f0bc9e95d84c09ec097394cd4093076f2a041b isl-0.16.1.tar.bz2
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
@ 2016-10-07 13:10 Moritz Klammler
2016-10-24 8:44 ` Richard Biener
0 siblings, 1 reply; 17+ messages in thread
From: Moritz Klammler @ 2016-10-07 13:10 UTC (permalink / raw)
To: GCC Patches
[-- Attachment #1: Type: text/plain, Size: 308 bytes --]
I would like to bump my patch that makes the
`contrib/download_prerequisites` script verify the checksums of
the downloaded packages and augments it with a few additional options.
All feedback I have received has been incorporated. Is it okay like
this?
Below is again the latest iteration of the patch.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 4th iteration of the patch against SVN revision 240137 --]
[-- Type: text/x-diff, Size: 10109 bytes --]
Index: contrib/download_prerequisites
===================================================================
--- contrib/download_prerequisites (revision 240137)
+++ contrib/download_prerequisites (working copy)
@@ -1,60 +1,215 @@
-#! /bin/sh
+#! /bin/sh -eu
+#! -*- coding:utf-8; mode:shell-script; -*-
-# Download some prerequisites needed by gcc.
-# Run this from the top level of the gcc source tree and the gcc
-# build will do the right thing.
+# Download some prerequisites needed by GCC.
+# Run this from the top level of the GCC source tree and the GCC build will do
+# the right thing. Run it with the `--help` option for more information.
#
-# (C) 2010-2016 Free Software Foundation
+# (C) 2016 Free Software Foundation
#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see http://www.gnu.org/licenses/.
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation, either version 3 of the License, or (at your option) any later
+# version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program. If not, see http://www.gnu.org/licenses/.
-# If you want to disable Graphite loop optimizations while building GCC,
-# DO NOT set GRAPHITE_LOOP_OPT as yes so that the isl package will not
-# be downloaded.
-GRAPHITE_LOOP_OPT=yes
-if [ ! -e gcc/BASE-VER ] ; then
- echo "You must run this script in the top level GCC source directory."
- exit 1
-fi
+program='download_prerequisites'
+version='(unversioned)'
-# Necessary to build GCC.
-MPFR=mpfr-3.1.4
-GMP=gmp-6.1.0
-MPC=mpc-1.0.3
+# MAINTAINERS: If you update the package versions below, please
+# remember to also update the files `contrib/prerequisites.sha512` and
+# `contrib/prerequisites.md5` with the new checksums.
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPFR.tar.bz2 || exit 1
-tar xjf $MPFR.tar.bz2 || exit 1
-if test -L mpfr; then rm -f mpfr; fi
-ln -sf $MPFR mpfr || exit 1
+gmp='gmp-6.1.0.tar.bz2'
+mpfr='mpfr-3.1.4.tar.bz2'
+mpc='mpc-1.0.3.tar.gz'
+isl='isl-0.16.1.tar.bz2'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$GMP.tar.bz2 || exit 1
-tar xjf $GMP.tar.bz2 || exit 1
-if test -L gmp; then rm -f gmp; fi
-ln -sf $GMP gmp || exit 1
+base_url='ftp://gcc.gnu.org/pub/gcc/infrastructure/'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPC.tar.gz || exit 1
-tar xzf $MPC.tar.gz || exit 1
-if test -L mpc; then rm -f mpc; fi
-ln -sf $MPC mpc || exit 1
+echo_archives() {
+ echo "${gmp}"
+ echo "${mpfr}"
+ echo "${mpc}"
+ if [ ${graphite} -gt 0 ]; then echo "${isl}"; fi
+}
-# Necessary to build GCC with the Graphite loop optimizations.
-if [ "$GRAPHITE_LOOP_OPT" = "yes" ] ; then
- ISL=isl-0.16.1
+graphite=1
+verify=1
+force=0
+chksum='sha512'
+directory='.'
- wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$ISL.tar.bz2 || exit 1
- tar xjf $ISL.tar.bz2 || exit 1
- if test -L isl; then rm -f isl; fi
- ln -sf $ISL isl || exit 1
+helptext="usage: ${program} [OPTION...]
+
+Downloads some prerequisites needed by GCC. Run this from the top level of the
+GCC source tree and the GCC build will do the right thing.
+
+The following options are available:
+
+ --directory=DIR download and unpack packages into DIR instead of '.'
+ --force download again overwriting existing packages
+ --no-force do not download existing packages again (default)
+ --isl download ISL, needed for Graphite loop optimizations (default)
+ --graphite same as --isl
+ --no-isl don't download ISL
+ --no-graphite same as --no-isl
+ --verify verify package integrity after download (default)
+ --no-verify don't verify package integrity
+ --sha512 use SHA512 checksum to verify package integrity (default)
+ --md5 use MD5 checksum to verify package integrity
+ --help show this text and exit
+ --version show version information and exit
+"
+
+versiontext="${program} ${version}
+Copyright (C) 2016 Free Software Foundation, Inc.
+This is free software; see the source for copying conditions. There is NO
+warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
+
+die() {
+ echo "error: $@" >&2
+ exit 1
+}
+
+for arg in "$@"
+do
+ case "${arg}" in
+ --help)
+ echo "${helptext}"
+ exit
+ ;;
+ --version)
+ echo "${versiontext}"
+ exit
+ ;;
+ esac
+done
+unset arg
+
+argnext=
+for arg in "$@"
+do
+ if [ "x${argnext}" = x ]
+ then
+ case "${arg}" in
+ --directory)
+ argnext='directory'
+ ;;
+ --directory=*)
+ directory="${arg#--directory=}"
+ ;;
+ --force)
+ force=1
+ ;;
+ --no-force)
+ force=0
+ ;;
+ --isl|--graphite)
+ graphite=1
+ ;;
+ --no-isl|--no-graphite)
+ graphite=0
+ ;;
+ --verify)
+ verify=1
+ ;;
+ --no-verify)
+ verify=0
+ ;;
+ --sha512)
+ chksum='sha512'
+ verify=1
+ ;;
+ --md5)
+ chksum='md5'
+ verify=1
+ ;;
+ -*)
+ die "unknown option: ${arg}"
+ ;;
+ *)
+ die "too many arguments"
+ ;;
+ esac
+ else
+ case "${arg}" in
+ -*)
+ die "Missing argument for option --${argnext}"
+ ;;
+ esac
+ case "${argnext}" in
+ directory)
+ directory="${arg}"
+ ;;
+ *)
+ die "The impossible has happened"
+ ;;
+ esac
+ argnext=
+ fi
+done
+[ "x${argnext}" = x ] || die "Missing argument for option --${argnext}"
+unset arg argnext
+
+[ -e ./gcc/BASE-VER ] \
+ || die "You must run this script in the top-level GCC source directory"
+
+[ -d "${directory}" ] \
+ || die "No such directory: ${directory}"
+
+for ar in $(echo_archives)
+do
+ if [ ${force} -gt 0 ]; then rm -f "${directory}/${ar}"; fi
+ [ -e "${directory}/${ar}" ] \
+ || wget --no-verbose -O "${directory}/${ar}" "${base_url}${ar}" \
+ || die "Cannot download ${ar} from ${base_url}"
+done
+unset ar
+
+if [ ${verify} -gt 0 ]
+then
+ chksumfile="contrib/prerequisites.${chksum}"
+ [ -r "${chksumfile}" ] || die "No checksums available"
+ for ar in $(echo_archives)
+ do
+ grep "${ar}" "${chksumfile}" \
+ | ( cd "${directory}" && "${chksum}sum" --check ) \
+ || die "Cannot verify integrity of possibly corrupted file ${ar}"
+ done
+ unset chksumfile
fi
+unset ar
+
+for ar in $(echo_archives)
+do
+ package="${ar%.tar*}"
+ if [ ${force} -gt 0 ]; then rm -rf "${directory}/${package}"; fi
+ [ -e "${directory}/${package}" ] \
+ || ( cd "${directory}" && tar -xf "${ar}" ) \
+ || die "Cannot extract package from ${ar}"
+ unset package
+done
+unset ar
+
+for ar in $(echo_archives)
+do
+ target="${directory}/${ar%.tar*}/"
+ linkname="${ar%-*}"
+ if [ ${force} -gt 0 ]; then rm -f "${linkname}"; fi
+ [ -e "${linkname}" ] \
+ || ln -s "${target}" "${linkname}" \
+ || die "Cannot create symbolic link ${linkname} --> ${target}"
+ unset target linkname
+done
+unset ar
+
+echo "All prerequisites downloaded successfully."
Index: contrib/prerequisites.md5
===================================================================
--- contrib/prerequisites.md5 (nonexistent)
+++ contrib/prerequisites.md5 (working copy)
@@ -0,0 +1,4 @@
+86ee6e54ebfc4a90b643a65e402c4048 gmp-6.1.0.tar.bz2
+b8a2f6b0e68bef46e53da2ac439e1cf4 mpfr-3.1.4.tar.bz2
+d6a1d5f8ddea3abd2cc3e98f58352d26 mpc-1.0.3.tar.gz
+ac1f25a0677912952718a51f5bc20f32 isl-0.16.1.tar.bz2
Index: contrib/prerequisites.sha512
===================================================================
--- contrib/prerequisites.sha512 (nonexistent)
+++ contrib/prerequisites.sha512 (working copy)
@@ -0,0 +1,4 @@
+3c82aeab9c1596d4da8afac2eec38e429e84f3211e1a572cf8fd2b546493c44c039b922a1133eaaa48bd7f3e11dbe795a384e21ed95cbe3ecc58d7ac02246117 gmp-6.1.0.tar.bz2
+51066066ff2c12ed2198605ecf68846b0c96b548adafa5b80e0c786d0df488411a5e8973358fce7192dc977ad4e68414cf14500e3c39746de62465eb145bb819 mpfr-3.1.4.tar.bz2
+0028b76df130720c1fad7de937a0d041224806ce5ef76589f19c7b49d956071a683e2f20d154c192a231e69756b19e48208f2889b0c13950ceb7b3cfaf059a43 mpc-1.0.3.tar.gz
+c188667a84dc5bdddb4ab7c35f89c91bf15a8171f4fcaf41301cf285fb7328846d9a367c096012fec4cc69d244f0bc9e95d84c09ec097394cd4093076f2a041b isl-0.16.1.tar.bz2
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-09-14 21:36 Moritz Klammler
@ 2016-09-14 21:49 ` Mike Stump
0 siblings, 0 replies; 17+ messages in thread
From: Mike Stump @ 2016-09-14 21:49 UTC (permalink / raw)
To: Moritz Klammler; +Cc: GCC Patches, Joseph Myers
On Sep 14, 2016, at 1:19 PM, Moritz Klammler <moritz@klammler.eu> wrote:
>
> Joseph Myers <joseph@codesourcery.com> writes:
>
>> On Wed, 14 Sep 2016, Moritz Klammler wrote:
>>
>>> Ok, I didn't know about the workflow. Do you think I should dike the
>>> `--strip-sums` option out again then?
>>
>> I don't see any use for such an option. Anyone changing the versions
>> should always have a copy of the new tarball (obtained securely if
>> possible) and should determine the sums from that.
>
> Alright, below then without the option again.
>
> Btw, how frequently am I supposed to post revisions of my patch to this
> list?
Feel free to address all reasonable concerns raised, and then post.
> Is it considered okay to do it immediately
Yes, as long as you address all the concerns you plan to address. Before that point and it is usually premature.
> And should I also attach a "diff of the diffs" or will those interested be happy to
> produce it themselves?
No diffs of diffs please. You should just regenerate the entire patch and include it in a email as you discuss each point raised and the outcome of those points.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
@ 2016-09-14 21:36 Moritz Klammler
2016-09-14 21:49 ` Mike Stump
0 siblings, 1 reply; 17+ messages in thread
From: Moritz Klammler @ 2016-09-14 21:36 UTC (permalink / raw)
To: GCC Patches; +Cc: Joseph Myers
[-- Attachment #1: Type: text/plain, Size: 761 bytes --]
Joseph Myers <joseph@codesourcery.com> writes:
> On Wed, 14 Sep 2016, Moritz Klammler wrote:
>
>> Ok, I didn't know about the workflow. Do you think I should dike the
>> `--strip-sums` option out again then?
>
> I don't see any use for such an option. Anyone changing the versions
> should always have a copy of the new tarball (obtained securely if
> possible) and should determine the sums from that.
Alright, below then without the option again.
Btw, how frequently am I supposed to post revisions of my patch to this
list? Is it considered okay to do it immediately or should I wait (how
long?) and gather more conclusive feedback first? And should I also
attach a "diff of the diffs" or will those interested be happy to
produce it themselves?
[-- Attachment #2: Suggested ChangeLog entry --]
[-- Type: text/plain, Size: 188 bytes --]
* contrib/download_prerequisites: Verify integrity of downloaded
packages and added more command-line options.
* contrib/prerequisites.sha512: New.
* contrib/prerequisites.md5: New.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 4th revision of the patch --]
[-- Type: text/x-diff, Size: 10109 bytes --]
Index: contrib/download_prerequisites
===================================================================
--- contrib/download_prerequisites (revision 240137)
+++ contrib/download_prerequisites (working copy)
@@ -1,60 +1,215 @@
-#! /bin/sh
+#! /bin/sh -eu
+#! -*- coding:utf-8; mode:shell-script; -*-
-# Download some prerequisites needed by gcc.
-# Run this from the top level of the gcc source tree and the gcc
-# build will do the right thing.
+# Download some prerequisites needed by GCC.
+# Run this from the top level of the GCC source tree and the GCC build will do
+# the right thing. Run it with the `--help` option for more information.
#
-# (C) 2010-2016 Free Software Foundation
+# (C) 2016 Free Software Foundation
#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see http://www.gnu.org/licenses/.
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation, either version 3 of the License, or (at your option) any later
+# version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program. If not, see http://www.gnu.org/licenses/.
-# If you want to disable Graphite loop optimizations while building GCC,
-# DO NOT set GRAPHITE_LOOP_OPT as yes so that the isl package will not
-# be downloaded.
-GRAPHITE_LOOP_OPT=yes
-if [ ! -e gcc/BASE-VER ] ; then
- echo "You must run this script in the top level GCC source directory."
- exit 1
-fi
+program='download_prerequisites'
+version='(unversioned)'
-# Necessary to build GCC.
-MPFR=mpfr-3.1.4
-GMP=gmp-6.1.0
-MPC=mpc-1.0.3
+# MAINTAINERS: If you update the package versions below, please
+# remember to also update the files `contrib/prerequisites.sha512` and
+# `contrib/prerequisites.md5` with the new checksums.
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPFR.tar.bz2 || exit 1
-tar xjf $MPFR.tar.bz2 || exit 1
-if test -L mpfr; then rm -f mpfr; fi
-ln -sf $MPFR mpfr || exit 1
+gmp='gmp-6.1.0.tar.bz2'
+mpfr='mpfr-3.1.4.tar.bz2'
+mpc='mpc-1.0.3.tar.gz'
+isl='isl-0.16.1.tar.bz2'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$GMP.tar.bz2 || exit 1
-tar xjf $GMP.tar.bz2 || exit 1
-if test -L gmp; then rm -f gmp; fi
-ln -sf $GMP gmp || exit 1
+base_url='ftp://gcc.gnu.org/pub/gcc/infrastructure/'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPC.tar.gz || exit 1
-tar xzf $MPC.tar.gz || exit 1
-if test -L mpc; then rm -f mpc; fi
-ln -sf $MPC mpc || exit 1
+echo_archives() {
+ echo "${gmp}"
+ echo "${mpfr}"
+ echo "${mpc}"
+ if [ ${graphite} -gt 0 ]; then echo "${isl}"; fi
+}
-# Necessary to build GCC with the Graphite loop optimizations.
-if [ "$GRAPHITE_LOOP_OPT" = "yes" ] ; then
- ISL=isl-0.16.1
+graphite=1
+verify=1
+force=0
+chksum='sha512'
+directory='.'
- wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$ISL.tar.bz2 || exit 1
- tar xjf $ISL.tar.bz2 || exit 1
- if test -L isl; then rm -f isl; fi
- ln -sf $ISL isl || exit 1
+helptext="usage: ${program} [OPTION...]
+
+Downloads some prerequisites needed by GCC. Run this from the top level of the
+GCC source tree and the GCC build will do the right thing.
+
+The following options are available:
+
+ --directory=DIR download and unpack packages into DIR instead of '.'
+ --force download again overwriting existing packages
+ --no-force do not download existing packages again (default)
+ --isl download ISL, needed for Graphite loop optimizations (default)
+ --graphite same as --isl
+ --no-isl don't download ISL
+ --no-graphite same as --no-isl
+ --verify verify package integrity after download (default)
+ --no-verify don't verify package integrity
+ --sha512 use SHA512 checksum to verify package integrity (default)
+ --md5 use MD5 checksum to verify package integrity
+ --help show this text and exit
+ --version show version information and exit
+"
+
+versiontext="${program} ${version}
+Copyright (C) 2016 Free Software Foundation, Inc.
+This is free software; see the source for copying conditions. There is NO
+warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
+
+die() {
+ echo "error: $@" >&2
+ exit 1
+}
+
+for arg in "$@"
+do
+ case "${arg}" in
+ --help)
+ echo "${helptext}"
+ exit
+ ;;
+ --version)
+ echo "${versiontext}"
+ exit
+ ;;
+ esac
+done
+unset arg
+
+argnext=
+for arg in "$@"
+do
+ if [ "x${argnext}" = x ]
+ then
+ case "${arg}" in
+ --directory)
+ argnext='directory'
+ ;;
+ --directory=*)
+ directory="${arg#--directory=}"
+ ;;
+ --force)
+ force=1
+ ;;
+ --no-force)
+ force=0
+ ;;
+ --isl|--graphite)
+ graphite=1
+ ;;
+ --no-isl|--no-graphite)
+ graphite=0
+ ;;
+ --verify)
+ verify=1
+ ;;
+ --no-verify)
+ verify=0
+ ;;
+ --sha512)
+ chksum='sha512'
+ verify=1
+ ;;
+ --md5)
+ chksum='md5'
+ verify=1
+ ;;
+ -*)
+ die "unknown option: ${arg}"
+ ;;
+ *)
+ die "too many arguments"
+ ;;
+ esac
+ else
+ case "${arg}" in
+ -*)
+ die "Missing argument for option --${argnext}"
+ ;;
+ esac
+ case "${argnext}" in
+ directory)
+ directory="${arg}"
+ ;;
+ *)
+ die "The impossible has happened"
+ ;;
+ esac
+ argnext=
+ fi
+done
+[ "x${argnext}" = x ] || die "Missing argument for option --${argnext}"
+unset arg argnext
+
+[ -e ./gcc/BASE-VER ] \
+ || die "You must run this script in the top-level GCC source directory"
+
+[ -d "${directory}" ] \
+ || die "No such directory: ${directory}"
+
+for ar in $(echo_archives)
+do
+ if [ ${force} -gt 0 ]; then rm -f "${directory}/${ar}"; fi
+ [ -e "${directory}/${ar}" ] \
+ || wget --no-verbose -O "${directory}/${ar}" "${base_url}${ar}" \
+ || die "Cannot download ${ar} from ${base_url}"
+done
+unset ar
+
+if [ ${verify} -gt 0 ]
+then
+ chksumfile="contrib/prerequisites.${chksum}"
+ [ -r "${chksumfile}" ] || die "No checksums available"
+ for ar in $(echo_archives)
+ do
+ grep "${ar}" "${chksumfile}" \
+ | ( cd "${directory}" && "${chksum}sum" --check ) \
+ || die "Cannot verify integrity of possibly corrupted file ${ar}"
+ done
+ unset chksumfile
fi
+unset ar
+
+for ar in $(echo_archives)
+do
+ package="${ar%.tar*}"
+ if [ ${force} -gt 0 ]; then rm -rf "${directory}/${package}"; fi
+ [ -e "${directory}/${package}" ] \
+ || ( cd "${directory}" && tar -xf "${ar}" ) \
+ || die "Cannot extract package from ${ar}"
+ unset package
+done
+unset ar
+
+for ar in $(echo_archives)
+do
+ target="${directory}/${ar%.tar*}/"
+ linkname="${ar%-*}"
+ if [ ${force} -gt 0 ]; then rm -f "${linkname}"; fi
+ [ -e "${linkname}" ] \
+ || ln -s "${target}" "${linkname}" \
+ || die "Cannot create symbolic link ${linkname} --> ${target}"
+ unset target linkname
+done
+unset ar
+
+echo "All prerequisites downloaded successfully."
Index: contrib/prerequisites.md5
===================================================================
--- contrib/prerequisites.md5 (nonexistent)
+++ contrib/prerequisites.md5 (working copy)
@@ -0,0 +1,4 @@
+86ee6e54ebfc4a90b643a65e402c4048 gmp-6.1.0.tar.bz2
+b8a2f6b0e68bef46e53da2ac439e1cf4 mpfr-3.1.4.tar.bz2
+d6a1d5f8ddea3abd2cc3e98f58352d26 mpc-1.0.3.tar.gz
+ac1f25a0677912952718a51f5bc20f32 isl-0.16.1.tar.bz2
Index: contrib/prerequisites.sha512
===================================================================
--- contrib/prerequisites.sha512 (nonexistent)
+++ contrib/prerequisites.sha512 (working copy)
@@ -0,0 +1,4 @@
+3c82aeab9c1596d4da8afac2eec38e429e84f3211e1a572cf8fd2b546493c44c039b922a1133eaaa48bd7f3e11dbe795a384e21ed95cbe3ecc58d7ac02246117 gmp-6.1.0.tar.bz2
+51066066ff2c12ed2198605ecf68846b0c96b548adafa5b80e0c786d0df488411a5e8973358fce7192dc977ad4e68414cf14500e3c39746de62465eb145bb819 mpfr-3.1.4.tar.bz2
+0028b76df130720c1fad7de937a0d041224806ce5ef76589f19c7b49d956071a683e2f20d154c192a231e69756b19e48208f2889b0c13950ceb7b3cfaf059a43 mpc-1.0.3.tar.gz
+c188667a84dc5bdddb4ab7c35f89c91bf15a8171f4fcaf41301cf285fb7328846d9a367c096012fec4cc69d244f0bc9e95d84c09ec097394cd4093076f2a041b isl-0.16.1.tar.bz2
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-09-14 17:04 Moritz Klammler
@ 2016-09-14 17:35 ` Joseph Myers
0 siblings, 0 replies; 17+ messages in thread
From: Joseph Myers @ 2016-09-14 17:35 UTC (permalink / raw)
To: Moritz Klammler; +Cc: GCC Patches, Richard Biener
On Wed, 14 Sep 2016, Moritz Klammler wrote:
> be cleaner to only include those checksums that are actually needed. On
> the other hand, it means an increased maintenance burden each time the
> version of the dependency is changed. In order to mitigate this and
I really don't see it as an increased burden. The maintainer shouldn't be
using the checksum files on the server at all. What they should do is:
* Download the tar file from ftp.gnu.org (at least for GMP / MPFR / MPC),
*verify the GPG signature* and test with it. (I'm not sure if the GNU
keyring is currently published.) The GPG signatures on ftp.gnu.org are
from the maintainer who uploaded the package, whereas the checksum files
on gcc.gnu.org are automatically generated from cron. (I don't know if a
secure way to download ISL from its origin has been added since
<https://gcc.gnu.org/ml/gcc/2016-07/msg00003.html> raised the issue.)
* Update the script and the to-be-checked-in checksums, using the file
they just downloaded and verified the signature of.
* Add the new file to the server before the script changes get checked in.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
@ 2016-09-14 17:04 Moritz Klammler
2016-09-14 17:35 ` Joseph Myers
0 siblings, 1 reply; 17+ messages in thread
From: Moritz Klammler @ 2016-09-14 17:04 UTC (permalink / raw)
To: GCC Patches; +Cc: Richard Biener, Joseph Myers
[-- Attachment #1: Type: text/plain, Size: 1986 bytes --]
Richard Biener <richard.guenther@gmail.com> writes:
> On Tue, Sep 13, 2016 at 6:01 PM, Joseph Myers <joseph@codesourcery.com> wrote:
>> On Tue, 13 Sep 2016, Moritz Klammler wrote:
>>
>>> I have made an actual diff now, containing also the checksum files.
>>> I
>>
>> I don't think checksums of lots of miscellaneous files should be
>> included, just the checksums for those files the current script will
>> actually use.
>
> I generally like the script but agree with Joseph here. We should be
> able to verify (upon committing changes to the script) to verify the
> sums by performing ./contrib/download_prerequesites [--md5] which
> means only including those that can be verified that way.
>
> Richard.
Thanks both of you for your feedback. I generally agree that it would
be cleaner to only include those checksums that are actually needed. On
the other hand, it means an increased maintenance burden each time the
version of the dependency is changed. In order to mitigate this and
eventually get the best of both worlds, I have added another
(intentionally undocumented) option `--strip-sums` to the script that
will strip the checksum file to include only the packages it will
download. A maintainer can run the script with this option to both
strip the checksum file and immediately test the new configuration. A
comment in the script next to where the version numbers are set reminds
you to do that.
If there were also GPG signatures of the checksum files on the server, I
could further modify the script to even download (and verify) the new
checksum files automatically, assuming that the maintainer who applies
the update will have the GPG key of the person who maintains the FTP
directory. (Might well be the same person anyway, I don't know.)
By the way: as mentioned in the bug tracker, there also is the
`./contrib/download_ecj` script but I believe that it will become
obsolete with the upcoming deletion of GCJ so we can ignore it for now.
Is this right?
[-- Attachment #2: Suggested ChangeLog entry (unchanged) --]
[-- Type: text/plain, Size: 188 bytes --]
* contrib/download_prerequisites: Verify integrity of downloaded
packages and added more command-line options.
* contrib/prerequisites.sha512: New.
* contrib/prerequisites.md5: New.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 3rd iteration of the patch --]
[-- Type: text/x-diff, Size: 11098 bytes --]
Index: contrib/download_prerequisites
===================================================================
--- contrib/download_prerequisites (revision 240137)
+++ contrib/download_prerequisites (working copy)
@@ -1,60 +1,240 @@
-#! /bin/sh
+#! /bin/sh -eu
+#! -*- coding:utf-8; mode:shell-script; -*-
-# Download some prerequisites needed by gcc.
-# Run this from the top level of the gcc source tree and the gcc
-# build will do the right thing.
+# Download some prerequisites needed by GCC.
+# Run this from the top level of the GCC source tree and the GCC build will do
+# the right thing. Run it with the `--help` option for more information.
#
-# (C) 2010-2016 Free Software Foundation
+# (C) 2016 Free Software Foundation
#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see http://www.gnu.org/licenses/.
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation, either version 3 of the License, or (at your option) any later
+# version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program. If not, see http://www.gnu.org/licenses/.
-# If you want to disable Graphite loop optimizations while building GCC,
-# DO NOT set GRAPHITE_LOOP_OPT as yes so that the isl package will not
-# be downloaded.
-GRAPHITE_LOOP_OPT=yes
-if [ ! -e gcc/BASE-VER ] ; then
- echo "You must run this script in the top level GCC source directory."
- exit 1
-fi
+program='download_prerequisites'
+version='(unversioned)'
-# Necessary to build GCC.
-MPFR=mpfr-3.1.4
-GMP=gmp-6.1.0
-MPC=mpc-1.0.3
+# MAINTAINERS: If you update the package versions below, please
+# replace the files `contrib/prerequisites.sha512` and
+# `contrib/prerequisites.md5` with a new authoritative copy of the
+# checksum files and then run this script with the (intentionally
+# undocumented) `--strip-sums` option. This will strip any unneeded
+# entries from the checksum file and immediately self-test the new
+# configuration. You have to do this twice, once with the `--sha512`
+# (or none) and once with the `--md5` option in order to strip the
+# checksum files for both hash algorithms.
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPFR.tar.bz2 || exit 1
-tar xjf $MPFR.tar.bz2 || exit 1
-if test -L mpfr; then rm -f mpfr; fi
-ln -sf $MPFR mpfr || exit 1
+gmp='gmp-6.1.0.tar.bz2'
+mpfr='mpfr-3.1.4.tar.bz2'
+mpc='mpc-1.0.3.tar.gz'
+isl='isl-0.16.1.tar.bz2'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$GMP.tar.bz2 || exit 1
-tar xjf $GMP.tar.bz2 || exit 1
-if test -L gmp; then rm -f gmp; fi
-ln -sf $GMP gmp || exit 1
+base_url='ftp://gcc.gnu.org/pub/gcc/infrastructure/'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPC.tar.gz || exit 1
-tar xzf $MPC.tar.gz || exit 1
-if test -L mpc; then rm -f mpc; fi
-ln -sf $MPC mpc || exit 1
+echo_archives() {
+ echo "${gmp}"
+ echo "${mpfr}"
+ echo "${mpc}"
+ if [ ${graphite} -gt 0 ]; then echo "${isl}"; fi
+}
-# Necessary to build GCC with the Graphite loop optimizations.
-if [ "$GRAPHITE_LOOP_OPT" = "yes" ] ; then
- ISL=isl-0.16.1
+graphite=1
+verify=1
+force=0
+chksum='sha512'
+directory='.'
+stripsums=0
- wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$ISL.tar.bz2 || exit 1
- tar xjf $ISL.tar.bz2 || exit 1
- if test -L isl; then rm -f isl; fi
- ln -sf $ISL isl || exit 1
+helptext="usage: ${program} [OPTION...]
+
+Downloads some prerequisites needed by GCC. Run this from the top level of the
+GCC source tree and the GCC build will do the right thing.
+
+The following options are available:
+
+ --directory=DIR download and unpack packages into DIR instead of '.'
+ --force download again overwriting existing packages
+ --no-force do not download existing packages again (default)
+ --isl download ISL, needed for Graphite loop optimizations (default)
+ --graphite same as --isl
+ --no-isl don't download ISL
+ --no-graphite same as --no-isl
+ --verify verify package integrity after download (default)
+ --no-verify don't verify package integrity
+ --sha512 use SHA512 checksum to verify package integrity (default)
+ --md5 use MD5 checksum to verify package integrity
+ --help show this text and exit
+ --version show version information and exit
+"
+
+versiontext="${program} ${version}
+Copyright (C) 2016 Free Software Foundation, Inc.
+This is free software; see the source for copying conditions. There is NO
+warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
+
+die() {
+ echo "error: $@" >&2
+ exit 1
+}
+
+for arg in "$@"
+do
+ case "${arg}" in
+ --help)
+ echo "${helptext}"
+ exit
+ ;;
+ --version)
+ echo "${versiontext}"
+ exit
+ ;;
+ esac
+done
+unset arg
+
+argnext=
+for arg in "$@"
+do
+ if [ "x${argnext}" = x ]
+ then
+ case "${arg}" in
+ --directory)
+ argnext='directory'
+ ;;
+ --directory=*)
+ directory="${arg#--directory=}"
+ ;;
+ --force)
+ force=1
+ ;;
+ --no-force)
+ force=0
+ ;;
+ --isl|--graphite)
+ graphite=1
+ ;;
+ --no-isl|--no-graphite)
+ graphite=0
+ ;;
+ --verify)
+ verify=1
+ ;;
+ --no-verify)
+ verify=0
+ ;;
+ --sha512)
+ chksum='sha512'
+ verify=1
+ ;;
+ --md5)
+ chksum='md5'
+ verify=1
+ ;;
+ --strip-sums)
+ stripsums=1
+ ;;
+ -*)
+ die "unknown option: ${arg}"
+ ;;
+ *)
+ die "too many arguments"
+ ;;
+ esac
+ else
+ case "${arg}" in
+ -*)
+ die "Missing argument for option --${argnext}"
+ ;;
+ esac
+ case "${argnext}" in
+ directory)
+ directory="${arg}"
+ ;;
+ *)
+ die "The impossible has happened"
+ ;;
+ esac
+ argnext=
+ fi
+done
+[ "x${argnext}" = x ] || die "Missing argument for option --${argnext}"
+unset arg argnext
+
+[ -e ./gcc/BASE-VER ] \
+ || die "You must run this script in the top-level GCC source directory"
+
+[ -d "${directory}" ] \
+ || die "No such directory: ${directory}"
+
+if [ ${stripsums} -gt 0 ]
+then
+ chksumfile="contrib/prerequisites.${chksum}"
+ stripsumfile="${chksumfile}.stripped"
+ rm -f "${stripsumfile}"
+ for ar in $(echo_archives)
+ do
+ grep "${ar}" "${chksumfile}" >> "${stripsumfile}" \
+ || die "Cannot find checksum for '${ar}' in file '${chksumfile}'"
+ done
+ mv "${stripsumfile}" "${chksumfile}" \
+ || die "Cannot overwrite '${chksumfile}' with '${stripsumfile}'"
+ unset ar chksumfile stripsumfile
fi
+
+for ar in $(echo_archives)
+do
+ if [ ${force} -gt 0 ]; then rm -f "${directory}/${ar}"; fi
+ [ -e "${directory}/${ar}" ] \
+ || wget --no-verbose -O "${directory}/${ar}" "${base_url}${ar}" \
+ || die "Cannot download ${ar} from ${base_url}"
+done
+unset ar
+
+if [ ${verify} -gt 0 ]
+then
+ chksumfile="contrib/prerequisites.${chksum}"
+ [ -r "${chksumfile}" ] || die "No checksums available"
+ for ar in $(echo_archives)
+ do
+ grep "${ar}" "${chksumfile}" \
+ | ( cd "${directory}" && "${chksum}sum" --check ) \
+ || die "Cannot verify integrity of possibly corrupted file ${ar}"
+ done
+ unset chksumfile
+fi
+unset ar
+
+for ar in $(echo_archives)
+do
+ package="${ar%.tar*}"
+ if [ ${force} -gt 0 ]; then rm -rf "${directory}/${package}"; fi
+ [ -e "${directory}/${package}" ] \
+ || ( cd "${directory}" && tar -xf "${ar}" ) \
+ || die "Cannot extract package from ${ar}"
+ unset package
+done
+unset ar
+
+for ar in $(echo_archives)
+do
+ target="${directory}/${ar%.tar*}/"
+ linkname="${ar%-*}"
+ if [ ${force} -gt 0 ]; then rm -f "${linkname}"; fi
+ [ -e "${linkname}" ] \
+ || ln -s "${target}" "${linkname}" \
+ || die "Cannot create symbolic link ${linkname} --> ${target}"
+ unset target linkname
+done
+unset ar
+
+echo "All prerequisites downloaded successfully."
Index: contrib/prerequisites.md5
===================================================================
--- contrib/prerequisites.md5 (nonexistent)
+++ contrib/prerequisites.md5 (working copy)
@@ -0,0 +1,4 @@
+86ee6e54ebfc4a90b643a65e402c4048 gmp-6.1.0.tar.bz2
+b8a2f6b0e68bef46e53da2ac439e1cf4 mpfr-3.1.4.tar.bz2
+d6a1d5f8ddea3abd2cc3e98f58352d26 mpc-1.0.3.tar.gz
+ac1f25a0677912952718a51f5bc20f32 isl-0.16.1.tar.bz2
Index: contrib/prerequisites.sha512
===================================================================
--- contrib/prerequisites.sha512 (nonexistent)
+++ contrib/prerequisites.sha512 (working copy)
@@ -0,0 +1,4 @@
+3c82aeab9c1596d4da8afac2eec38e429e84f3211e1a572cf8fd2b546493c44c039b922a1133eaaa48bd7f3e11dbe795a384e21ed95cbe3ecc58d7ac02246117 gmp-6.1.0.tar.bz2
+51066066ff2c12ed2198605ecf68846b0c96b548adafa5b80e0c786d0df488411a5e8973358fce7192dc977ad4e68414cf14500e3c39746de62465eb145bb819 mpfr-3.1.4.tar.bz2
+0028b76df130720c1fad7de937a0d041224806ce5ef76589f19c7b49d956071a683e2f20d154c192a231e69756b19e48208f2889b0c13950ceb7b3cfaf059a43 mpc-1.0.3.tar.gz
+c188667a84dc5bdddb4ab7c35f89c91bf15a8171f4fcaf41301cf285fb7328846d9a367c096012fec4cc69d244f0bc9e95d84c09ec097394cd4093076f2a041b isl-0.16.1.tar.bz2
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-09-13 16:07 ` Joseph Myers
@ 2016-09-14 8:24 ` Richard Biener
0 siblings, 0 replies; 17+ messages in thread
From: Richard Biener @ 2016-09-14 8:24 UTC (permalink / raw)
To: Joseph Myers; +Cc: Moritz Klammler, GCC Patches
On Tue, Sep 13, 2016 at 6:01 PM, Joseph Myers <joseph@codesourcery.com> wrote:
> On Tue, 13 Sep 2016, Moritz Klammler wrote:
>
>> I have made an actual diff now, containing also the checksum files. I
>
> I don't think checksums of lots of miscellaneous files should be included,
> just the checksums for those files the current script will actually use.
I generally like the script but agree with Joseph here. We should be able
to verify (upon committing changes to the script) to verify the sums by
performing ./contrib/download_prerequesites [--md5] which means only
including those that can be verified that way.
Richard.
> --
> Joseph S. Myers
> joseph@codesourcery.com
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-09-13 8:37 Moritz Klammler
@ 2016-09-13 16:07 ` Joseph Myers
2016-09-14 8:24 ` Richard Biener
0 siblings, 1 reply; 17+ messages in thread
From: Joseph Myers @ 2016-09-13 16:07 UTC (permalink / raw)
To: Moritz Klammler; +Cc: gcc-patches
On Tue, 13 Sep 2016, Moritz Klammler wrote:
> I have made an actual diff now, containing also the checksum files. I
I don't think checksums of lots of miscellaneous files should be included,
just the checksums for those files the current script will actually use.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
@ 2016-09-13 8:37 Moritz Klammler
2016-09-13 16:07 ` Joseph Myers
0 siblings, 1 reply; 17+ messages in thread
From: Moritz Klammler @ 2016-09-13 8:37 UTC (permalink / raw)
To: gcc-patches; +Cc: Joseph Myers
[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]
Joseph Myers <joseph@codesourcery.com> writes:
> On Sun, 11 Sep 2016, Moritz Klammler wrote:
>
>> gmp='gmp-4.3.2.tar.bz2'
>> mpfr='mpfr-2.4.2.tar.bz2'
>> mpc='mpc-0.8.1.tar.gz'
>> isl='isl-0.15.tar.bz2'
>
> These are not the versions used in the current script (given which,
> presumably you need to check for any other changes to the script since
> you started work on your version that need to be reflected in it).
Thanks for checking, you're right; I should have checked that. I
couldn't spot any other relevant changes, however.
Attached is the patch for the updated script. FWIW, I have also tested
bootstrapping the default configuration plus `--disable-multilib` on
x86_64 GNU/Linux.
I have made an actual diff now, containing also the checksum files. I
can make no guarantees that these are accurate, though. I've simply
downloaded them over the same insecure FTP connection that is used to
download the archives the signatures are supposed to verify. Please
somebody with a means of doing so double-check the signature files.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Diff against current SVN trunk --]
[-- Type: text/x-diff, Size: 17082 bytes --]
Index: contrib/download_prerequisites
===================================================================
--- contrib/download_prerequisites (revision 240106)
+++ contrib/download_prerequisites (working copy)
@@ -1,60 +1,211 @@
-#! /bin/sh
+#! /bin/sh -eu
+#! -*- coding:utf-8; mode:shell-script; -*-
-# Download some prerequisites needed by gcc.
-# Run this from the top level of the gcc source tree and the gcc
-# build will do the right thing.
+# Download some prerequisites needed by GCC.
+# Run this from the top level of the GCC source tree and the GCC build will do
+# the right thing. Run it with the `--help` option for more information.
#
-# (C) 2010-2016 Free Software Foundation
+# (C) 2016 Free Software Foundation
#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see http://www.gnu.org/licenses/.
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation, either version 3 of the License, or (at your option) any later
+# version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program. If not, see http://www.gnu.org/licenses/.
-# If you want to disable Graphite loop optimizations while building GCC,
-# DO NOT set GRAPHITE_LOOP_OPT as yes so that the isl package will not
-# be downloaded.
-GRAPHITE_LOOP_OPT=yes
-if [ ! -e gcc/BASE-VER ] ; then
- echo "You must run this script in the top level GCC source directory."
- exit 1
-fi
+program='download_prerequisites'
+version='(unversioned)'
-# Necessary to build GCC.
-MPFR=mpfr-3.1.4
-GMP=gmp-6.1.0
-MPC=mpc-1.0.3
+graphite=1
+verify=1
+force=0
+chksum='sha512'
+directory='.'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPFR.tar.bz2 || exit 1
-tar xjf $MPFR.tar.bz2 || exit 1
-if test -L mpfr; then rm -f mpfr; fi
-ln -sf $MPFR mpfr || exit 1
+gmp='gmp-6.1.0.tar.bz2'
+mpfr='mpfr-3.1.4.tar.bz2'
+mpc='mpc-1.0.3.tar.gz'
+isl='isl-0.16.1.tar.bz2'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$GMP.tar.bz2 || exit 1
-tar xjf $GMP.tar.bz2 || exit 1
-if test -L gmp; then rm -f gmp; fi
-ln -sf $GMP gmp || exit 1
+base_url='ftp://gcc.gnu.org/pub/gcc/infrastructure/'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPC.tar.gz || exit 1
-tar xzf $MPC.tar.gz || exit 1
-if test -L mpc; then rm -f mpc; fi
-ln -sf $MPC mpc || exit 1
+echo_archives() {
+ echo "${gmp}"
+ echo "${mpfr}"
+ echo "${mpc}"
+ if [ ${graphite} -gt 0 ]; then echo "${isl}"; fi
+}
-# Necessary to build GCC with the Graphite loop optimizations.
-if [ "$GRAPHITE_LOOP_OPT" = "yes" ] ; then
- ISL=isl-0.16.1
+helptext="usage: ${program} [OPTION...]
- wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$ISL.tar.bz2 || exit 1
- tar xjf $ISL.tar.bz2 || exit 1
- if test -L isl; then rm -f isl; fi
- ln -sf $ISL isl || exit 1
+Downloads some prerequisites needed by GCC. Run this from the top level of the
+GCC source tree and the GCC build will do the right thing.
+
+The following options are available:
+
+ --directory=DIR download and unpack packages into DIR instead of '.'
+ --force download again overwriting existing packages
+ --no-force do not download existing packages again (default)
+ --isl download ISL, needed for Graphite loop optimizations (default)
+ --graphite same as --isl
+ --no-isl don't download ISL
+ --no-graphite same as --no-isl
+ --verify verify package integrity after download (default)
+ --no-verify don't verify package integrity
+ --sha512 use SHA512 checksum to verify package integrity (default)
+ --md5 use MD5 checksum to verify package integrity
+ --help show this text and exit
+ --version show version information and exit
+"
+
+versiontext="${program} ${version}
+Copyright (C) 2016 Free Software Foundation, Inc.
+This is free software; see the source for copying conditions. There is NO
+warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
+
+die() {
+ echo "error: $@" >&2
+ exit 1
+}
+
+for arg in "$@"
+do
+ case "${arg}" in
+ --help)
+ echo "${helptext}"
+ exit
+ ;;
+ --version)
+ echo "${versiontext}"
+ exit
+ ;;
+ esac
+done
+unset arg
+
+argnext=
+for arg in "$@"
+do
+ if [ "x${argnext}" = x ]
+ then
+ case "${arg}" in
+ --directory)
+ argnext='directory'
+ ;;
+ --directory=*)
+ directory="${arg#--directory=}"
+ ;;
+ --force)
+ force=1
+ ;;
+ --no-force)
+ force=0
+ ;;
+ --isl|--graphite)
+ graphite=1
+ ;;
+ --no-isl|--no-graphite)
+ graphite=0
+ ;;
+ --verify)
+ verify=1
+ ;;
+ --no-verify)
+ verify=0
+ ;;
+ --sha512)
+ chksum='sha512'
+ verify=1
+ ;;
+ --md5)
+ chksum='md5'
+ verify=1
+ ;;
+ -*)
+ die "unknown option: ${arg}"
+ ;;
+ *)
+ die "too many arguments"
+ ;;
+ esac
+ else
+ case "${arg}" in
+ -*)
+ die "Missing argument for option --${argnext}"
+ ;;
+ esac
+ case "${argnext}" in
+ directory)
+ directory="${arg}"
+ ;;
+ *)
+ die "The impossible has happened"
+ ;;
+ esac
+ argnext=
+ fi
+done
+[ "x${argnext}" = x ] || die "Missing argument for option --${argnext}"
+unset arg argnext
+
+[ -e ./gcc/BASE-VER ] \
+ || die "You must run this script in the top-level GCC source directory"
+
+[ -d "${directory}" ] \
+ || die "No such directory: ${directory}"
+
+for ar in $(echo_archives)
+do
+ if [ ${force} -gt 0 ]; then rm -f "${directory}/${ar}"; fi
+ [ -e "${directory}/${ar}" ] \
+ || wget --no-verbose -O "${directory}/${ar}" "${base_url}${ar}" \
+ || die "Cannot download ${ar} from ${base_url}"
+done
+unset ar
+
+if [ ${verify} -gt 0 ]
+then
+ chksumfile="contrib/prerequisites.${chksum}"
+ [ -r "${chksumfile}" ] || die "No checksums available"
+ for ar in $(echo_archives)
+ do
+ grep "${ar}" "${chksumfile}" \
+ | ( cd "${directory}" && "${chksum}sum" --check ) \
+ || die "Cannot verify integrity of possibly corrupted file ${ar}"
+ done
+ unset chksumfile
fi
+unset ar
+
+for ar in $(echo_archives)
+do
+ package="${ar%.tar*}"
+ if [ ${force} -gt 0 ]; then rm -rf "${directory}/${package}"; fi
+ [ -e "${directory}/${package}" ] \
+ || ( cd "${directory}" && tar -xf "${ar}" ) \
+ || die "Cannot extract package from ${ar}"
+ unset package
+done
+unset ar
+
+for ar in $(echo_archives)
+do
+ target="${directory}/${ar%.tar*}/"
+ linkname="${ar%-*}"
+ if [ ${force} -gt 0 ]; then rm -f "${linkname}"; fi
+ [ -e "${linkname}" ] \
+ || ln -s "${target}" "${linkname}" \
+ || die "Cannot create symbolic link ${linkname} --> ${target}"
+ unset target linkname
+done
+unset ar
+
+echo "All prerequisites downloaded successfully."
Index: contrib/prerequisites.md5
===================================================================
--- contrib/prerequisites.md5 (nonexistent)
+++ contrib/prerequisites.md5 (working copy)
@@ -0,0 +1,39 @@
+dd6d26b592ed806a2f1b5b3deb28aeec brik2.tar.gz
+0ebb1a56b1af0e21d9de30b644c1c059 cctools-576.dmg
+3b9a5dd3db6b4a7e9c8de02198faea25 cctools-576.tar.bz2
+0043796eff1b3187f5e7b4db6e3fc5e5 cctools-590.12.dmg
+410dd3c1471d31e24a193c674432a7f5 cctools-590.12.tar.bz2
+36232fb9525ef5101ed9089db7083d4d cctools-590.36.dmg
+e90cf2e459c431b65a8989f79e4d625c cctools-590.36.tar.bz2
+947123350d1ff6dcb4b0774947ac015a cloog-0.16.1.tar.gz
+83877caaa879c7160063138bb18348e7 cloog-0.16.2.tar.gz
+be78a47bd82523250eb3e91646db5b3d cloog-0.18.0.tar.gz
+e34fca0540d840e5d0f6427e98c92252 cloog-0.18.1.tar.gz
+f483539b30a60a3478eea70c77b26bef cloog-parma-0.16.1.tar.gz
+060ae4df6fb8176e021b4d033a6c0b9e cloog-ppl-0.15.11.tar.gz
+294f381eaa5ce6f96bf5c50094cb5f19 gccrepo.tar.rz
+dd60683d7057917e34630b4a787932e8 gmp-4.3.2.tar.bz2
+86ee6e54ebfc4a90b643a65e402c4048 gmp-6.1.0.tar.bz2
+3206f67d214bef0e1de1068a87c22455 gperf-2.7-19981006.pat
+bce1586384d8635a76d2f017fb067cd2 isl-0.11.1.tar.bz2
+e039bfcfb6c2ab039b8ee69bf883e824 isl-0.12.2.tar.bz2
+acd347243fca5609e3df37dba47fd0bb isl-0.14.tar.bz2
+8428efbbc6f6e2810ce5c1ba73ecf98c isl-0.15.tar.bz2
+ac1f25a0677912952718a51f5bc20f32 isl-0.16.1.tar.bz2
+906124171f15ee1585d840ed7d174009 libg++-2.8.1.3-20000312.diff.gz
+47b93312badd9550ccb7d113bbf0242a libg++-2.8.1.3-20000419.diff.gz
+9e00e62b8fb7af3e41364b7c6d9f4cf8 libg++-2.8.1.3-20000816.diff.gz
+625dd5a953661b901c876f92c2c5e7a6 libg++-2.8.1.3-20000914.diff.gz
+2826dbbd081646c459b1774145ffd7bf libg++-2.8.1.3.tar.gz
+5b34aa804d514cc295414a963aedb6bf mpc-0.8.1.tar.gz
+d6a1d5f8ddea3abd2cc3e98f58352d26 mpc-1.0.3.tar.gz
+89e59fe665e2b3ad44a6789f40b059a0 mpfr-2.4.2.tar.bz2
+b8a2f6b0e68bef46e53da2ac439e1cf4 mpfr-3.1.4.tar.bz2
+918d22e272274f96f3c26ea5b51723e9 pooma-2.3.0-gcc.tar.gz
+918d22e272274f96f3c26ea5b51723e9 pooma-gcc.tar.gz
+e7dd265afdeaea81f7e87a72b182d875 ppl-0.10.2.tar.gz
+ba527ec0ffc830ce16fad8a4195a337e ppl-0.11.tar.gz
+1e30484839440c95c98b8372cf2d2c49 sha512.sum
+58572309d5916942643b20ed385317ce svk-all-entire-history.tar.rz
+ea45e1dc06856e709ada659888a78411 svk-trunk-3.4-onward.tar.rz
+642d834f4f8655d080e0be610af68e2e svk-trunk-entire-history.tar.rz
Index: contrib/prerequisites.sha512
===================================================================
--- contrib/prerequisites.sha512 (nonexistent)
+++ contrib/prerequisites.sha512 (working copy)
@@ -0,0 +1,38 @@
+9b01839bbdf185e9a236ca87efc60f23b7fff7e59264c9ef05c93d19aed72295829b9c3458b19f3c01ca0c038947ad63ab2fc64e0179846db9a5a5e137c94524 brik2.tar.gz
+378e5718efb0edaf9d33dbf6f07f1c1931bb60b210539fe3cc9053b06ba66e88ca1dc1880958ba1d51a1fe365afeb9ac3eec903f4cf14b38bfc274be6cca8d35 cctools-576.dmg
+820b4628563cdac9cb87d4014330dfa5c2f94fb60c57f075602dc1153ed3fc910e2eea54d5ea4a1c63c33149e8e6ff8d6320ac9a3e8bc1e011caa09fb45e6896 cctools-576.tar.bz2
+82844046c0babc67364849214fc90284935953dbf7040d7c5301df3070a8d66aa04338f03865a4dc0936bb420732f125fba0aff1d5bca0da9e8a7d1b2c1174b7 cctools-590.12.dmg
+4cd431d11b234a9856cd3e357b4b701ca1f726b097597452004aa4845c83026de5a6d1a2ccca4c6482d5df9cd76d395b9f1f9d8597f31998a424cbec43d028dd cctools-590.12.tar.bz2
+44e50658350e59b8d9a8ae786d357e4aaa7d35460d5aed76048a616d8ba020011705436f6468908737779a0f00496681a0d13e3f52a8dd0a42c24e9897ddfbef cctools-590.36.dmg
+0603ea4f6db75af6a59e02fa3b9f325c3f21eaeef44acc618a2ac51bb51f4fa7c1d238aa6ef8bfa1a7886378124aa8d52141f209d4388b17f36c85df261ad581 cctools-590.36.tar.bz2
+3741f6085df362482ef7062d8cf3bfff54ca72890e837c3f13d10a8e159efe2b1d6e224d65911a0ba8bac57cc9b23d2cf192f416cbc9937f699d711febb8a111 cloog-0.16.1.tar.gz
+acf6123cc194d4556bb92471a73ecb52dc6ed54d09cf1e22fee28f6658a07d8a85eb27106a4f15835865e108bfc4e1a9f6340c9bb578719055a295e2d7a36075 cloog-0.16.2.tar.gz
+34404984c4269dfc19673712271b08f220b6b49e451e7479cefbc5d2f27ab8e780f5d51fbbc7cecb974dab655673d8348f2a1ece3a60feaeeb9413fccf16eb1d cloog-0.18.0.tar.gz
+0b12d9f3c39a2425e28e1d7c0a2b3787287fe3e6e3052f094d2ab6cffeb205ce19044100cbfd805659b3e6b3d21ac2f5a3c92848f476de54edfe6b1cbd2172e9 cloog-0.18.1.tar.gz
+bcf0ed5cd4349ab183feb51f657915cbffb0e0da050a174a0d8ef8336aa87b0f1923fecfebcc43bb4163a48a54e5997e4e6c80d7e1a5983674c91f3d36462f04 cloog-parma-0.16.1.tar.gz
+71a7c42c1b0c410c52c0d7d2a7d458bdae655b816bc89d010ef471c0abcf0ccfee8cabce4a93ff066a17f756c20d5440383302c95558c9456d5db6c421840b36 cloog-ppl-0.15.11.tar.gz
+e0ca3c60481d0f2f1db68baa8b5dfbda17adabb723fea4a21279fa833a5ab9fa215e72170a14ceead9ac837945013ba20c7bbe0c13b8b49f090c1fb1ec4a9220 gccrepo.tar.rz
+2e0b0fd23e6f10742a5517981e5171c6e88b0a93c83da701b296f5c0861d72c19782daab589a7eac3f9032152a0fc7eff7f5362db8fccc4859564a9aa82329cf gmp-4.3.2.tar.bz2
+3c82aeab9c1596d4da8afac2eec38e429e84f3211e1a572cf8fd2b546493c44c039b922a1133eaaa48bd7f3e11dbe795a384e21ed95cbe3ecc58d7ac02246117 gmp-6.1.0.tar.bz2
+af3c51991692d0467b1520f6f39745cffc45699b5133709dbb8700b044517b29c71453b7e728f50a591c486f7103095450bbfc0ba247a40efbd2fd5dae680d18 gperf-2.7-19981006.pat
+f021ef6982b30a4834bc459af3520bd74b9c6af3e479b344c2d0e1bb4550b85cd30612e342ac761d85f1d2e2160ab3183e1252696726f438e61fb070a7a7bd22 isl-0.11.1.tar.bz2
+b5dea4d6b504fac22f2d8cd9c47d6896994d3966635a514d45010243db44cfbc009574b808f3587f35d91de629110bc11b52941d4075e21f8646c59656e62aea isl-0.12.2.tar.bz2
+117c0c6f31d91a9284a8f54a748df6494ca2ac21507a45611e911403e9610579b9323ce624aea1de0b8089b3194e59d4364f01bdb71ddcf8f6c24c749d11c4b7 isl-0.14.tar.bz2
+1e27b7798f7428abcb5e9b2e3fbe3842fede54c03bbd7bd3cf83703e1e4cca7d95c51326ab90253fe55b38c002183e8e78dfbb4d2cf20b0aabe02443c8e7d50f isl-0.15.tar.bz2
+c188667a84dc5bdddb4ab7c35f89c91bf15a8171f4fcaf41301cf285fb7328846d9a367c096012fec4cc69d244f0bc9e95d84c09ec097394cd4093076f2a041b isl-0.16.1.tar.bz2
+abb57356eee18eca395aa4b2ceee20360983f666458e1064f142bda4710ae02d41a47ad2c3fca6b2f3a20d40296277d31e76d2845817df4652c70da4fa3960f8 libg++-2.8.1.3-20000312.diff.gz
+7d8c160d93ad977528bbdbf8a313b4c2299b4ae24c662c491c421852604b2270cade04d34acc199b609a670ebd7a25ab1fb6636afe06f02cdd62c25484d25762 libg++-2.8.1.3-20000419.diff.gz
+6f2f4c2c94dc9b682b8cbe6601698b88c5d69fcb75bba9d0a056ec0a5939851f96e5a73aa8af6e4e32828191ea057b669475e0cf301f5dd15bdcc5acca5541dc libg++-2.8.1.3-20000816.diff.gz
+d828131e3391ef44854073edc4ebdcf59bd2e7d28229759f61d52b77c24494bc463a0646094d2fdf1d70fd2af2d4c536965ddadf625dfc916d14daa0fea4a282 libg++-2.8.1.3-20000914.diff.gz
+dc8289b700630407d932fc28d182b14716c52df7dcdc52cd5d44bd05a9b7406f0ab639821d191f8a8544fb3ee57ae58457909a875df495dd8794268fde0afc87 libg++-2.8.1.3.tar.gz
+8cbf00a3f046a7f0ee5bb7cb3f0582b6a0221b5f34ae282752d48689bf509858a9abd613ce2b2f349606742321784ad547ae67ac7af6bf0e421c3f4d491932bf md5.sum
+14cb9ae3d33caed24d5ae648eed28b2e00ad047a8baeff25981129af88245b4def2948573d7a00d65c5bd34e53524aa6a7351b76703c9f888b41830c1a1daae2 mpc-0.8.1.tar.gz
+0028b76df130720c1fad7de937a0d041224806ce5ef76589f19c7b49d956071a683e2f20d154c192a231e69756b19e48208f2889b0c13950ceb7b3cfaf059a43 mpc-1.0.3.tar.gz
+c004b3dbf86c04960e4a1f8db37a409a7cc4cb76135e76e98dcc5ad93aaa8deb62334ee13ff84447a7c12a5e8cb57f25c62ac908c24920f1fb1a38d79d4a4c5e mpfr-2.4.2.tar.bz2
+51066066ff2c12ed2198605ecf68846b0c96b548adafa5b80e0c786d0df488411a5e8973358fce7192dc977ad4e68414cf14500e3c39746de62465eb145bb819 mpfr-3.1.4.tar.bz2
+a3d2a187a6ed3767cf5998fb4942bde237a0ebadfa75ab9ce8de02f786564b45613fc6954c53f1f3b76b45933cda3c763b53819b9f8d5046662ef4d9475f68ae pooma-2.3.0-gcc.tar.gz
+df0eed010aa79ed984db158a4aeda39936b1efcba84cc497197f37b48cfb416d77690e4ecfaf74de91eb39745c2b5b9a74cd4509ef0972104065b5f2c5fdd013 ppl-0.10.2.tar.gz
+dca1d6a26708f08a99ac7a42f73059c171ceb63d3617338dd6608d93aab614838bfdd1b7e39c2df4559d7ed7bb86eb8f020ad59197681757a23a9a1bd8df9ebe ppl-0.11.tar.gz
+85999ba5bbede5fda08500b0323910d3f759c4a92218af9ca54e70b217b980a2eab6ca96bc49b198187bc8606d4c013ed07990d2638cd62119dd6da6822e6bb6 svk-all-entire-history.tar.rz
+028f9403032c0b4d38c030668302ec5bc14f43ae7bd0ec86ef65cf7a75e9011d21730cd030a69a4ccd5187ce5d0fdd93bd8d0399355e2aefaa69d2f5b9b91c38 svk-trunk-3.4-onward.tar.rz
+03f597cc36ea4823028a0377071e280a1137d8ad198416cc32982a1cd04e255a9e28ca5481202b749438dbb4d22d770282a839f1e8098c32e23a89cde4f9ef01 svk-trunk-entire-history.tar.rz
[-- Attachment #3: ChangeLog entry --]
[-- Type: text/plain, Size: 188 bytes --]
* contrib/download_prerequisites: Verify integrity of downloaded
packages and added more command-line options.
* contrib/prerequisites.sha512: New.
* contrib/prerequisites.md5: New.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-09-11 17:19 Moritz Klammler
2016-09-11 19:05 ` Mike Stump
@ 2016-09-12 22:08 ` Joseph Myers
1 sibling, 0 replies; 17+ messages in thread
From: Joseph Myers @ 2016-09-12 22:08 UTC (permalink / raw)
To: Moritz Klammler; +Cc: gcc-patches
On Sun, 11 Sep 2016, Moritz Klammler wrote:
> gmp='gmp-4.3.2.tar.bz2'
> mpfr='mpfr-2.4.2.tar.bz2'
> mpc='mpc-0.8.1.tar.gz'
> isl='isl-0.15.tar.bz2'
These are not the versions used in the current script (given which,
presumably you need to check for any other changes to the script since you
started work on your version that need to be reflected in it).
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Verify package integrity of downloaded prerequisites (partially fixes 61439)
2016-09-11 17:19 Moritz Klammler
@ 2016-09-11 19:05 ` Mike Stump
2016-09-12 22:08 ` Joseph Myers
1 sibling, 0 replies; 17+ messages in thread
From: Mike Stump @ 2016-09-11 19:05 UTC (permalink / raw)
To: Moritz Klammler; +Cc: gcc-patches
On Sep 11, 2016, at 8:35 AM, Moritz Klammler <moritz@klammler.eu> wrote:
>
> There is a long-standing
> [bug report](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61439)
> pointing out that the `download_prerequisites` script doesn't verify the
> integrity of the packages it downloads.
I like the script.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Verify package integrity of downloaded prerequisites (partially fixes 61439)
@ 2016-09-11 17:19 Moritz Klammler
2016-09-11 19:05 ` Mike Stump
2016-09-12 22:08 ` Joseph Myers
0 siblings, 2 replies; 17+ messages in thread
From: Moritz Klammler @ 2016-09-11 17:19 UTC (permalink / raw)
To: gcc-patches
[-- Attachment #1.1: Type: text/plain, Size: 3245 bytes --]
There is a long-standing
[bug report](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61439)
pointing out that the `download_prerequisites` script doesn't verify the
integrity of the packages it downloads. The original bug report is only
concerned about stability but for me, this is first and foremost a
security issue. Even more so since the downloads happen over unsecured
FTP connections.
I have already posted an improved version of the script that does verify
checksums on the Bugzilla thread in February this year but apparently,
it went unnoticed there.
Coincidentally, another person posted an alternative solution at the
same place just about the same time that didn't seem to have received
any attention either. Their fix is minimal invasive while I essentially
re-wrote the script and added a bunch of command-line options. I
believe those are useful but you could also arguably call it
over-engineering. I'm okay with either fix but I'm not going to speak
for the other person so I'll only explain my own patch here.
The script relies on the files `contrib/prerequisites.sha512` and
`contrib/prerequisites.md5` to be present in the GCC tarball and will
use them to verify the package integrity. Said files are tiny and
readily available here
ftp://gcc.gnu.org/pub/gcc/infrastructure/sha512.sum
ftp://gcc.gnu.org/pub/gcc/infrastructure/md5.sum
but I didn't include them into the patch because I have no way of
verifying their integrity. Somebody who has secure access to the GNU
servers should verify that we add the correct checksums into the GCC
archive. Every time a new version of the prerequisites is used, the
checksum files have to be updated as well.
The integrity of the downloaded prerequisites then follows transitively
assuming that GCC's own tarball hasn't been tampered with which we have
to assume anyway. (Ensuring this is beyond the scope of anything we can
put *inside* the tarball itself.)
Since my patch doesn't contain said checksum files and other than that,
merely replaces a single file completely, I'm attaching just the new
version of the `download_prerequisites` scripts and not a diff. It
would be nice of somebody with trusted access to the GNU servers could
be so kind to prepare the actual SVN patch for me.
After adding the checksum files, the ChangeLog entry might be like this:
* contrib/download_prerequisites: Verify integrity of downloaded
packages and added more command-line options.
* contrib/prerequisites.sha512: New.
* contrib/prerequisites.md5: New.
I already went through the copyright assignment process for GCC so I
think you should be able to include my patch if it is accepted. (I also
took the liberty to mention the FSF as copyright holder in the script
without waiting for the patch being accepted.)
I believe that attacking a system by compromising the compiler is a very
serious threat so I hope that you'll consider patching the script. If
there are any portability or other issues with the way I wrote the
script, I'll be ready to fix them.
Moritz
--
OpenPGP:
Public Key: http://openpgp.klammler.eu
Fingerprint: 2732 DA32 C8D0 EEEC A081 BE9D CF6C 5166 F393 A9C0
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: New version of the download_prerequisites script that verifies checksums --]
[-- Type: text/x-sh, Size: 6219 bytes --]
#! /bin/sh -eu
#! -*- coding:utf-8; mode:shell-script; -*-
# Download some prerequisites needed by GCC.
# Run this from the top level of the Gcc source tree and the Gcc build will do
# the right thing. Run it with the `--help` option for more information.
#
# (C) 2016 Free Software Foundation
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free Software
# Foundation, either version 3 of the License, or (at your option) any later
# version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
# details.
#
# You should have received a copy of the GNU General Public License along with
# this program. If not, see http://www.gnu.org/licenses/.
program='download_prerequisites'
version='(unversioned)'
graphite=1
verify=1
force=0
chksum='sha512'
directory='.'
gmp='gmp-4.3.2.tar.bz2'
mpfr='mpfr-2.4.2.tar.bz2'
mpc='mpc-0.8.1.tar.gz'
isl='isl-0.15.tar.bz2'
base_url='ftp://gcc.gnu.org/pub/gcc/infrastructure/'
echo_archives() {
echo "${gmp}"
echo "${mpfr}"
echo "${mpc}"
if [ ${graphite} -gt 0 ]; then echo "${isl}"; fi
}
helptext="usage: ${program} [OPTION...]
Downloads some prerequisites needed by GCC. Run this from the top level of the
GCC source tree and the GCC build will do the right thing.
The following options are available:
--directory=DIR download and unpack packages into DIR instead of '.'
--force download again overwriting existing packages
--no-force do not download existing packages again (default)
--isl download ISL, needed for Graphite loop optimizations (default)
--graphite same as --isl
--no-isl don't download ISL
--no-graphite same as --no-isl
--verify verify package integrity after download (default)
--no-verify don't verify package integrity
--sha512 use SHA512 checksum to verify package integrity (default)
--md5 use MD5 checksum to verify package integrity
--help show this text and exit
--version show version information and exit
"
versiontext="${program} ${version}
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
die() {
echo "error: $@" >&2
exit 1
}
for arg in "$@"
do
case "${arg}" in
--help)
echo "${helptext}"
exit
;;
--version)
echo "${versiontext}"
exit
;;
esac
done
unset arg
argnext=
for arg in "$@"
do
if [ "x${argnext}" = x ]
then
case "${arg}" in
--directory)
argnext='directory'
;;
--directory=*)
directory="${arg#--directory=}"
;;
--force)
force=1
;;
--no-force)
force=0
;;
--isl|--graphite)
graphite=1
;;
--no-isl|--no-graphite)
graphite=0
;;
--verify)
verify=1
;;
--no-verify)
verify=0
;;
--sha512)
chksum='sha512'
verify=1
;;
--md5)
chksum='md5'
verify=1
;;
-*)
die "unknown option: ${arg}"
;;
*)
die "too many arguments"
;;
esac
else
case "${arg}" in
-*)
die "Missing argument for option --${argnext}"
;;
esac
case "${argnext}" in
directory)
directory="${arg}"
;;
*)
die "The impossible has happened"
;;
esac
argnext=
fi
done
[ "x${argnext}" = x ] || die "Missing argument for option --${argnext}"
unset arg argnext
[ -e ./gcc/BASE-VER ] \
|| die "You must run this script in the top-level GCC source directory"
[ -d "${directory}" ] \
|| die "No such directory: ${directory}"
for ar in $(echo_archives)
do
if [ ${force} -gt 0 ]; then rm -f "${directory}/${ar}"; fi
[ -e "${directory}/${ar}" ] \
|| wget --no-verbose -O "${directory}/${ar}" "${base_url}${ar}" \
|| die "Cannot download ${ar} from ${base_url}"
done
unset ar
if [ ${verify} -gt 0 ]
then
chksumfile="contrib/prerequisites.${chksum}"
[ -r "${chksumfile}" ] || die "No checksums available"
for ar in $(echo_archives)
do
grep "${ar}" "${chksumfile}" \
| ( cd "${directory}" && "${chksum}sum" --check ) \
|| die "Cannot verify integrity of possibly corrupted file ${ar}"
done
unset chksumfile
fi
unset ar
for ar in $(echo_archives)
do
package="${ar%.tar*}"
if [ ${force} -gt 0 ]; then rm -rf "${directory}/${package}"; fi
[ -e "${directory}/${package}" ] \
|| ( cd "${directory}" && tar -xf "${ar}" ) \
|| die "Cannot extract package from ${ar}"
unset package
done
unset ar
for ar in $(echo_archives)
do
target="${directory}/${ar%.tar*}/"
linkname="${ar%-*}"
if [ ${force} -gt 0 ]; then rm -f "${linkname}"; fi
[ -e "${linkname}" ] \
|| ln -s "${target}" "${linkname}" \
|| die "Cannot create symbolic link ${linkname} --> ${target}"
unset target linkname
done
unset ar
echo "All prerequisites downloaded successfully."
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2016-10-25 15:18 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-09-14 17:38 Verify package integrity of downloaded prerequisites (partially fixes 61439) Moritz Klammler
2016-09-14 18:12 ` Joseph Myers
-- strict thread matches above, loose matches on Subject: below --
2016-10-25 0:16 Moritz Klammler
2016-10-25 15:18 ` Jeff Law
2016-10-07 13:10 Moritz Klammler
2016-10-24 8:44 ` Richard Biener
2016-10-24 16:06 ` Jeff Law
2016-09-14 21:36 Moritz Klammler
2016-09-14 21:49 ` Mike Stump
2016-09-14 17:04 Moritz Klammler
2016-09-14 17:35 ` Joseph Myers
2016-09-13 8:37 Moritz Klammler
2016-09-13 16:07 ` Joseph Myers
2016-09-14 8:24 ` Richard Biener
2016-09-11 17:19 Moritz Klammler
2016-09-11 19:05 ` Mike Stump
2016-09-12 22:08 ` Joseph Myers
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).