port contrib/download_prerequisites script to macOS

port contrib/download_prerequisites script to macOS

[Damian Rouson]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

All,

The attached patch modifies the contrib/download_prerequisites script to work on macOS. 
The revised script detects the operating system and adjusts the shasum and md5 commands 
to their expected name and arguments on macOS.  The revised script also uses curl if 
wget is not present.  macOS ships with curl but not wget.

Tested on macOS and Lubuntu and Fedora Linux distributions. 

Ok for trunk?

Damian


2017-04-05  Damian Rouson  <damian@sourceryinstitute.org>

        * download_prerequisites (md5_check): New function emulates Linux
        'md5 --check' on macOS.  Modified script for macOS compatibility.


[-- Attachment #2: macOS-download_prerequisites.diff --]
[-- Type: application/octet-stream, Size: 3495 bytes --]

diff --git a/contrib/download_prerequisites b/contrib/download_prerequisites
index a9eac67..a3918cb 100755
--- a/contrib/download_prerequisites
+++ b/contrib/download_prerequisites
@@ -20,7 +20,6 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see http://www.gnu.org/licenses/.
 
-
 program='download_prerequisites'
 version='(unversioned)'
 
@@ -45,7 +44,23 @@ echo_archives() {
 graphite=1
 verify=1
 force=0
-chksum='sha512'
+OS=$(uname)
+
+case $OS in
+  "Darwin")
+    chksum='shasum -a 512 --check'
+  ;;
+  *)
+    chksum='sha512sum --check'
+  ;;
+esac
+
+if type wget > /dev/null ; then
+  fetch='wget'
+else
+  fetch='curl -LO -u anonymous:'
+fi
+chksum_extension='sha512'
 directory='.'
 
 helptext="usage: ${program} [OPTION...]
@@ -95,6 +110,19 @@ do
 done
 unset arg
 
+# Emulate Linux's 'md5 --check' on macOS
+md5_check() {
+  md5_checksum_line=$(cat -)                        # Store the standard input: a line from contrib/prerequisites.md5:
+  md5_checksum_expected="${md5_checksum_line%% *}"  # Grab the text before the first space
+  file_to_check="${md5_checksum_line##* }"          # Grab the text after the first space
+  
+  md5_checksum_output=$(md5 -r "${file_to_check}")    # Calculate the md5 checksum for the downloaded file
+  md5_checksum_detected="${md5_checksum_output%% *}"  # Grab the text before the first space
+  [ "${md5_checksum_expected}" == "${md5_checksum_detected}" ] || die "Cannot verify integrity of possibly corrupted file ${file_to_check}"
+  echo "${file_to_check}: OK"
+}
+
+
 argnext=
 for arg in "$@"
 do
@@ -126,11 +154,27 @@ do
                 verify=0
                 ;;
             --sha512)
-                chksum='sha512'
+                case $OS in
+                  "Darwin")
+                    chksum='shasum -a 512 --check'
+                  ;;
+                  *)
+                    chksum='sha512sum --check'
+                  ;;
+                esac
+                chksum_extension='sha512'
                 verify=1
                 ;;
             --md5)
-                chksum='md5'
+                case $OS in
+                  "Darwin")
+                    chksum='md5_check'
+                  ;;
+                  *)
+                    chksum='md5 --check'
+                  ;;
+                esac
+                chksum_extension='md5'
                 verify=1
                 ;;
             -*)
@@ -170,19 +214,19 @@ for ar in $(echo_archives)
 do
     if [ ${force} -gt 0 ]; then rm -f "${directory}/${ar}"; fi
     [ -e "${directory}/${ar}" ]                                               \
-        || wget --no-verbose -O "${directory}/${ar}" "${base_url}${ar}"       \
+        || ${fetch} --no-verbose -O "${directory}/${ar}" "${base_url}${ar}"       \
         || die "Cannot download ${ar} from ${base_url}"
 done
 unset ar
 
 if [ ${verify} -gt 0 ]
 then
-    chksumfile="contrib/prerequisites.${chksum}"
+    chksumfile="contrib/prerequisites.${chksum_extension}"
     [ -r "${chksumfile}" ] || die "No checksums available"
     for ar in $(echo_archives)
     do
         grep "${ar}" "${chksumfile}"                                          \
-            | ( cd "${directory}" && "${chksum}sum" --check )                 \
+            | ( cd "${directory}" && ${chksum} )                              \
             || die "Cannot verify integrity of possibly corrupted file ${ar}"
     done
     unset chksumfile

[PATCH, contrib] Re: port contrib/download_prerequisites script to macOS

[Jerry DeLisle]

ping

On 04/04/2017 06:10 PM, Damian Rouson wrote:
> All,
> 
> The attached patch modifies the contrib/download_prerequisites script to work on macOS. 
> The revised script detects the operating system and adjusts the shasum and md5 commands 
> to their expected name and arguments on macOS.  The revised script also uses curl if 
> wget is not present.  macOS ships with curl but not wget.
> 
> Tested on macOS and Lubuntu and Fedora Linux distributions. 
> 
> Ok for trunk?
> 
> Damian
> 
> 
> 2017-04-05  Damian Rouson  <damian@sourceryinstitute.org>
> 
>         * download_prerequisites (md5_check): New function emulates Linux
>         'md5 --check' on macOS.  Modified script for macOS compatibility.
> 

Re: port contrib/download_prerequisites script to macOS

[Mike Stump]

On Apr 4, 2017, at 6:10 PM, Damian Rouson <damian@sourceryinstitute.org> wrote:
> 
> The attached patch modifies the contrib/download_prerequisites script to work on macOS. 

> Ok for trunk?

Don't know if others would prefer to review...  assuming no strong feelings against...

Ok.

Re: port contrib/download_prerequisites script to macOS

[Jerry DeLisle]

On 04/10/2017 11:59 AM, Damian Rouson wrote:
> Thanks!  I'm asking Jerry to commit it.
>
> Damian
>
> On April 10, 2017 at 9:48:01 AM, Mike Stump (mikestump@comcast.net
> <mailto:mikestump@comcast.net>) wrote:
>
>> On Apr 4, 2017, at 6:10 PM, Damian Rouson <damian@sourceryinstitute.org> wrote:
>> >
>> > The attached patch modifies the contrib/download_prerequisites script to work on macOS.
>>
>> > Ok for trunk?
>>
>> Don't know if others would prefer to review... assuming no strong feelings
>> against...
>>
>> Ok.
>>

No objections noted.

	M	contrib/ChangeLog
	M	contrib/download_prerequisites
Committed r246845

Thanks all,

Jerry

Re: port contrib/download_prerequisites script to macOS

[Jeff Law]

On 04/04/2017 07:10 PM, Damian Rouson wrote:
> All,
>
> The attached patch modifies the contrib/download_prerequisites script to work on macOS.
> The revised script detects the operating system and adjusts the shasum and md5 commands
> to their expected name and arguments on macOS.  The revised script also uses curl if
> wget is not present.  macOS ships with curl but not wget.
>
> Tested on macOS and Lubuntu and Fedora Linux distributions.
>
> Ok for trunk?
>
> Damian
>
>
> 2017-04-05  Damian Rouson  <damian@sourceryinstitute.org>
>
>         * download_prerequisites (md5_check): New function emulates Linux
>         'md5 --check' on macOS.  Modified script for macOS compatibility.
I wonder if we should just switch to curl from wget in general rather 
than conditionalizing the code at all.

For the sums, rather than doing a check of the OS, just see if 
sha512/md5sum exists.  If not, then fallback to the Darwin defaults.

Jeff

Re: port contrib/download_prerequisites script to macOS

[Martin Sebor]

On 04/12/2017 04:03 PM, Jeff Law wrote:
> On 04/04/2017 07:10 PM, Damian Rouson wrote:
>> All,
>>
>> The attached patch modifies the contrib/download_prerequisites script
>> to work on macOS.
>> The revised script detects the operating system and adjusts the shasum
>> and md5 commands
>> to their expected name and arguments on macOS.  The revised script
>> also uses curl if
>> wget is not present.  macOS ships with curl but not wget.
>>
>> Tested on macOS and Lubuntu and Fedora Linux distributions.
>>
>> Ok for trunk?
>>
>> Damian
>>
>>
>> 2017-04-05  Damian Rouson  <damian@sourceryinstitute.org>
>>
>>         * download_prerequisites (md5_check): New function emulates Linux
>>         'md5 --check' on macOS.  Modified script for macOS compatibility.
> I wonder if we should just switch to curl from wget in general rather
> than conditionalizing the code at all.

That was going to be my suggestion as well.  It will make updating
the script easier.

Martin

>
> For the sums, rather than doing a check of the OS, just see if
> sha512/md5sum exists.  If not, then fallback to the Darwin defaults.
>
> Jeff

Re: port contrib/download_prerequisites script to macOS

[Jerry DeLisle]

On 04/12/2017 05:40 PM, Martin Sebor wrote:
> On 04/12/2017 04:03 PM, Jeff Law wrote:
>> On 04/04/2017 07:10 PM, Damian Rouson wrote:
>>> All,
>>>
>>> The attached patch modifies the contrib/download_prerequisites script
>>> to work on macOS.
>>> The revised script detects the operating system and adjusts the shasum
>>> and md5 commands
>>> to their expected name and arguments on macOS.  The revised script
>>> also uses curl if
>>> wget is not present.  macOS ships with curl but not wget.
>>>
>>> Tested on macOS and Lubuntu and Fedora Linux distributions.
>>>
>>> Ok for trunk?
>>>
>>> Damian
>>>
>>>
>>> 2017-04-05  Damian Rouson  <damian@sourceryinstitute.org>
>>>
>>>         * download_prerequisites (md5_check): New function emulates Linux
>>>         'md5 --check' on macOS.  Modified script for macOS compatibility.
>> I wonder if we should just switch to curl from wget in general rather
>> than conditionalizing the code at all.
> 
> That was going to be my suggestion as well.  It will make updating
> the script easier.
> 
> Martin
> 
>>
>> For the sums, rather than doing a check of the OS, just see if
>> sha512/md5sum exists.  If not, then fallback to the Darwin defaults.
>>
>> Jeff
> 

I did not wait long enough for your comments and comitted it already. We can
certainly adjust it.

Jerry

Re: port contrib/download_prerequisites script to macOS

[Damian Rouson]

Resending as plain text:

 On April 12, 2017 at 3:03:19 PM, Jeff Law (law@redhat.com(mailto:law@redhat.com)) wrote:
 
 > > 
 > > 2017-04-05 Damian Rouson 
 > > 
 > > * download_prerequisites (md5_check): New function emulates Linux 
 > > 'md5 --check' on macOS. Modified script for macOS compatibility. 
 > I wonder if we should just switch to curl from wget in general rather 
 > than conditionalizing the code at all. 
 
 Hi Jeff,
 
 
Thanks for your comments. The conditionals are more portable than hardwiring one choice. On macOS, curl is always present but not wget. On the Linux distribution that I use (Lubuntu), wget is always present but curl isn’t installed by default. My goal was to support as many users as possible without requiring them to install prerequisites just to download the prerequisites. :) As my first contribution of a patch to GCC, I took the baby step of allowing for the use of curl. If acceptable, my next step would be to allow for the use of ftp when neither curl nor wget is present. That’s what I do in the scripts from which I culled the code in the patch. In those scripts, I invoke ftp via the following function: 
 
https://github.com/sourceryinstitute/OpenCoarrays/blob/master/prerequisites/build-functions/ftp_url.sh
 
 > 
 > 
 > For the sums, rather than doing a check of the OS, just see if  
 > sha512/md5sum exists. If not, then fallback to the Darwin defaults. 
 
I think I tried that first and ran into some difficulties. I’ll make another attempt. My recent shell programming experience is with the bash shell and I have some experience with the C-shell from years ago, but I had no experience with writing Bourne shell scripts before writing this patch and I think the problem I encountered was related to differences between Bourne and bash.
 
 
 Damian

Re: port contrib/download_prerequisites script to macOS

[Martin Sebor]

On 04/13/2017 05:29 PM, Damian Rouson wrote:
> Resending as plain text:
>
>  On April 12, 2017 at 3:03:19 PM, Jeff Law (law@redhat.com(mailto:law@redhat.com)) wrote:
>
>  > >
>  > > 2017-04-05 Damian Rouson
>  > >
>  > > * download_prerequisites (md5_check): New function emulates Linux
>  > > 'md5 --check' on macOS. Modified script for macOS compatibility.
>  > I wonder if we should just switch to curl from wget in general rather
>  > than conditionalizing the code at all.
>
>  Hi Jeff,
>
>
> Thanks for your comments. The conditionals are more portable than hardwiring one choice. On macOS, curl is always present but not wget. On the Linux distribution that I use (Lubuntu), wget is always present but curl isn’t installed by default.

It sounds like using the first of the two utilities that's in PATH,
similarly to the sha512/md5sum approach discussed below, would be
a more robust solution.  That way it will work on any distribution
that provides either utility, and it will also be possible to test
the script both ways on either kind of system without having to make
changes to it.

Martin

> My goal was to support as many users as possible without requiring them to install prerequisites just to download the prerequisites. :) As my first contribution of a patch to GCC, I took the baby step of allowing for the use of curl. If acceptable, my next step would be to allow for the use of ftp when neither curl nor wget is present. That’s what I do in the scripts from which I culled the code in the patch. In those scripts, I invoke ftp via the following function:
>
> https://github.com/sourceryinstitute/OpenCoarrays/blob/master/prerequisites/build-functions/ftp_url.sh
>
>  >
>  >
>  > For the sums, rather than doing a check of the OS, just see if
>  > sha512/md5sum exists. If not, then fallback to the Darwin defaults.
>
> I think I tried that first and ran into some difficulties. I’ll make another attempt. My recent shell programming experience is with the bash shell and I have some experience with the C-shell from years ago, but I had no experience with writing Bourne shell scripts before writing this patch and I think the problem I encountered was related to differences between Bourne and bash.
>
>
>  Damian
>