[patch] build xz (instead of bz2) compressed tarballs and diffs

[patch] build xz (instead of bz2) compressed tarballs and diffs

[Matthias Klose]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

As discussed on IRC with Jakub and Richard here are is a small patch which
builds xz compressed tarballs and diff files.

Tested with

  maintainer-scripts/gcc_release \
	-s snap:trunk -p <old bz2 tarball> diffs sources tarfiles
  maintainer-scripts/gcc_release \
	-s snap:trunk -p <old xz tarball> diffs sources tarfiles

and checked that the new tarball and diff files are compressed using xz.

Ok for the trunk and the gcc-7-branch?

Matthias


[-- Attachment #2: xz-releases.diff --]
[-- Type: text/plain, Size: 5027 bytes --]


maintainer-scripts/

2017-05-14  Matthias Klose  <doko@ubuntu.com>

	* gcc_release (build_gzip): Build xz tarball instead of bz2 tarball.
	(build_diffs): Handle building diffs from either bz2 or xz tarballs,
	compress diffs using xz instead of bz2.
	(build_diff): Likewise.
	(upload_files): Check for *.xz files instead of *.bz2 files.
	(announce_snapshot): Announce xz tarball instead of bz2 tarball.
	(XZ): New definition.
	(<toplevel>): Look for both bz2 and xz compressed old tarballs.

 
Index: maintainer-scripts/gcc_release
===================================================================
--- maintainer-scripts/gcc_release	(revision 248041)
+++ maintainer-scripts/gcc_release	(working copy)
@@ -221,7 +221,7 @@
   # Create a "MD5SUMS" file to use for checking the validity of the release.
   echo \
 "# This file contains the MD5 checksums of the files in the 
-# gcc-"${RELEASE}".tar.bz2 tarball.
+# gcc-"${RELEASE}".tar.xz tarball.
 #
 # Besides verifying that all files in the tarball were correctly expanded,
 # it also can be used to determine if any files have changed since the
@@ -244,11 +244,11 @@
 
 build_tarfile() {
   # Get the name of the destination tar file.
-  TARFILE="$1.tar.bz2"
+  TARFILE="$1.tar.xz"
   shift
 
   # Build the tar file itself.
-  (${TAR} cf - "$@" | ${BZIP2} > ${TARFILE}) || \
+  (${TAR} cf - "$@" | ${XZ} > ${TARFILE}) || \
     error "Could not build tarfile"
   FILE_LIST="${FILE_LIST} ${TARFILE}"
 }
@@ -273,8 +273,8 @@
 # Build .gz files.
 build_gzip() {
   for f in ${FILE_LIST}; do
-    target=${f%.bz2}.gz
-    (${BZIP2} -d -c $f | ${GZIP} > ${target}) || error "Could not create ${target}"
+    target=${f%.xz}.gz
+    (${XZ} -d -c $f | ${GZIP} > ${target}) || error "Could not create ${target}"
   done
 }
 
@@ -282,12 +282,19 @@
 build_diffs() {
   old_dir=${1%/*}
   old_file=${1##*/}
-  old_vers=${old_file%.tar.bz2}
+  case "$old_file" in
+    *.tar.xz) old_vers=${old_file%.tar.xz};;
+    *) old_vers=${old_file%.tar.bz2};;
+  esac
   old_vers=${old_vers#gcc-}
   inform "Building diffs against version $old_vers"
   for f in gcc; do
-    old_tar=${old_dir}/${f}-${old_vers}.tar.bz2
-    new_tar=${WORKING_DIRECTORY}/${f}-${RELEASE}.tar.bz2
+    if [ -e ${old_dir}/${f}-${old_vers}.tar.xz ]; then
+      old_tar=${old_dir}/${f}-${old_vers}.tar.xz
+    else
+      old_tar=${old_dir}/${f}-${old_vers}.tar.bz2
+    fi
+    new_tar=${WORKING_DIRECTORY}/${f}-${RELEASE}.tar.xz
     if [ ! -e $old_tar ]; then
       inform "$old_tar not found; not generating diff file"
     elif [ ! -e $new_tar ]; then
@@ -294,7 +301,7 @@
       inform "$new_tar not found; not generating diff file"
     else
       build_diff $old_tar gcc-${old_vers} $new_tar gcc-${RELEASE} \
-        ${f}-${old_vers}-${RELEASE}.diff.bz2
+        ${f}-${old_vers}-${RELEASE}.diff.xz
     fi
   done
 }
@@ -305,13 +312,20 @@
   tmpdir=gccdiff.$$
   mkdir $tmpdir || error "Could not create directory $tmpdir"
   changedir $tmpdir
-  (${BZIP2} -d -c $1 | ${TAR} xf - ) || error "Could not unpack $1 for diffs"
-  (${BZIP2} -d -c $3 | ${TAR} xf - ) || error "Could not unpack $3 for diffs"
-  ${DIFF} $2 $4 > ../${5%.bz2}
+  case "$1" in
+    *.tar.bz2)
+      (${BZIP2} -d -c $1 | ${TAR} xf - ) || error "Could not unpack $1 for diffs"
+      ;;
+    *.tar.xz)
+      (${XZ} -d -c $1 | ${TAR} xf - ) || error "Could not unpack $1 for diffs"
+      ;;
+  esac
+  (${XZ} -d -c $3 | ${TAR} xf - ) || error "Could not unpack $3 for diffs"
+  ${DIFF} $2 $4 > ../${5%.xz}
   if [ $? -eq 2 ]; then
     error "Trouble making diffs from $1 to $3"
   fi
-  ${BZIP2} ../${5%.bz2} || error "Could not generate ../$5"
+  ${XZ} ../${5%.xz} || error "Could not generate ../$5"
   changedir ..
   rm -rf $tmpdir
   FILE_LIST="${FILE_LIST} $5"
@@ -335,7 +349,7 @@
   fi
 
   # Then copy files to their respective (sub)directories.
-  for x in gcc*.gz gcc*.bz2; do
+  for x in gcc*.gz gcc*.xz; do
     if [ -e ${x} ]; then
       # Make sure the file will be readable on the server.
       chmod a+r ${x}
@@ -410,7 +424,7 @@
 
 <table>" > ${SNAPSHOT_INDEX}
        
-  snapshot_print gcc-${RELEASE}.tar.bz2 "Complete GCC"
+  snapshot_print gcc-${RELEASE}.tar.xz "Complete GCC"
 
   echo \
 "Diffs from "${BRANCH}"-"${LAST_DATE}" are available in the diffs/ subdirectory.
@@ -528,12 +542,13 @@
 MODE_TARFILES=0
 MODE_UPLOAD=0
 
-# List of archive files generated; used to create .gz files from .bz2.
+# List of archive files generated; used to create .gz files from .xz.
 FILE_LIST=""
 
 # Programs we use.
 
 BZIP2="${BZIP2:-bzip2}"
+XZ="${XZ:-xz --best}"
 CVS="${CVS:-cvs -f -Q -z9}"
 DIFF="${DIFF:-diff -Nrcpad}"
 ENV="${ENV:-env}"
@@ -644,6 +659,9 @@
   if [ $MODE_DIFFS -ne 0 ] && [ $LOCAL -ne 0 ] && [ -z "${OLD_TARS}" ]; then
     LAST_DATE=`cat ~/.snapshot_date-${BRANCH}`
     OLD_TARS=${SNAPSHOTS_DIR}/${BRANCH}-${LAST_DATE}/gcc-${BRANCH}-${LAST_DATE}.tar.bz2
+    if [ ! -e $OLD_TARS ]; then
+      OLD_TARS=${SNAPSHOTS_DIR}/${BRANCH}-${LAST_DATE}/gcc-${BRANCH}-${LAST_DATE}.tar.xz
+    fi
   fi
 fi
 

Re: [patch] build xz (instead of bz2) compressed tarballs and diffs

[Joseph Myers]

The xz manpage warns against blindly using -9 (for which --best is a 
deprecated alias) because of the implications for memory requirements for 
decompressing.  If there's a reason it's considered appropriate here, I 
think it needs an explanatory comment.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [patch] build xz (instead of bz2) compressed tarballs and diffs

[Markus Trippelsdorf]

On 2017.05.15 at 14:02 +0000, Joseph Myers wrote:
> The xz manpage warns against blindly using -9 (for which --best is a 
> deprecated alias) because of the implications for memory requirements for 
> decompressing.  If there's a reason it's considered appropriate here, I 
> think it needs an explanatory comment.

I think it is unacceptable, because it would increase memory usage when
decompressing over 20x compared to bz2 (and over 100x while compressing).

The default -6 should be good enough (3x more memory when decompressing).

-- 
Markus

Re: [patch] build xz (instead of bz2) compressed tarballs and diffs

[Jakub Jelinek]

On Mon, May 15, 2017 at 04:13:44PM +0200, Markus Trippelsdorf wrote:
> On 2017.05.15 at 14:02 +0000, Joseph Myers wrote:
> > The xz manpage warns against blindly using -9 (for which --best is a 
> > deprecated alias) because of the implications for memory requirements for 
> > decompressing.  If there's a reason it's considered appropriate here, I 
> > think it needs an explanatory comment.
> 
> I think it is unacceptable, because it would increase memory usage when
> decompressing over 20x compared to bz2 (and over 100x while compressing).

The memory using during compressing isn't that interesting as long as it
isn't prohibitive for sourceware or the machines RMs use.
For the decompression, I guess it matters what is actually the memory needed
for decompression the -9 gcc tarball, and compare that to minimal memory
requirements to compile (not bootstrap) the compiler using typical system
compilers.  If compilation of gcc takes more memory than the decompression,
then it should be fine, why would anyone try to decompress gcc not to build
it afterwards?

	Jakub

Re: [patch] build xz (instead of bz2) compressed tarballs and diffs

[Markus Trippelsdorf]

On 2017.05.15 at 16:24 +0200, Jakub Jelinek wrote:
> On Mon, May 15, 2017 at 04:13:44PM +0200, Markus Trippelsdorf wrote:
> > On 2017.05.15 at 14:02 +0000, Joseph Myers wrote:
> > > The xz manpage warns against blindly using -9 (for which --best is a 
> > > deprecated alias) because of the implications for memory requirements for 
> > > decompressing.  If there's a reason it's considered appropriate here, I 
> > > think it needs an explanatory comment.
> > 
> > I think it is unacceptable, because it would increase memory usage when
> > decompressing over 20x compared to bz2 (and over 100x while compressing).
> 
> The memory using during compressing isn't that interesting as long as it
> isn't prohibitive for sourceware or the machines RMs use.
> For the decompression, I guess it matters what is actually the memory needed
> for decompression the -9 gcc tarball, and compare that to minimal memory
> requirements to compile (not bootstrap) the compiler using typical system
> compilers.  If compilation of gcc takes more memory than the decompression,
> then it should be fine, why would anyone try to decompress gcc not to build
> it afterwards?

Ok, it doesn't really matter. With gcc-7.1 tarball:

size: 533084160 (uncompressed)

-9:
 xz -d gcc.tar.xz
4.71user 0.26system 0:04.97elapsed 100%CPU (0avgtext+0avgdata 67804maxresident)k
size: 60806928

-6 (default):
 xz -d gcc.tar.xz
4.88user 0.28system 0:05.17elapsed 99%CPU (0avgtext+0avgdata 10324maxresident)k
size: 65059664

So -9 is actually just fine.

-- 
Markus

Re: [patch] build xz (instead of bz2) compressed tarballs and diffs

[Richard Biener]

On Mon, May 15, 2017 at 3:11 AM, Matthias Klose <doko@ubuntu.com> wrote:
> As discussed on IRC with Jakub and Richard here are is a small patch which
> builds xz compressed tarballs and diff files.
>
> Tested with
>
>   maintainer-scripts/gcc_release \
>         -s snap:trunk -p <old bz2 tarball> diffs sources tarfiles
>   maintainer-scripts/gcc_release \
>         -s snap:trunk -p <old xz tarball> diffs sources tarfiles
>
> and checked that the new tarball and diff files are compressed using xz.
>
> Ok for the trunk and the gcc-7-branch?

Ok.  The version on trunk can get the bz2 old-tar support removed after the next
snapshot generation I  think.  Likewise the branch version after 7.2
was released.

Richard.

> Matthias
>

Re: [patch] build xz (instead of bz2) compressed tarballs and diffs

[Matthias Klose]

On 15.05.2017 12:04, Markus Trippelsdorf wrote:
> On 2017.05.15 at 16:24 +0200, Jakub Jelinek wrote:
>> On Mon, May 15, 2017 at 04:13:44PM +0200, Markus Trippelsdorf wrote:
>>> On 2017.05.15 at 14:02 +0000, Joseph Myers wrote:
>>>> The xz manpage warns against blindly using -9 (for which --best is a 
>>>> deprecated alias) because of the implications for memory requirements for 
>>>> decompressing.  If there's a reason it's considered appropriate here, I 
>>>> think it needs an explanatory comment.
>>>
>>> I think it is unacceptable, because it would increase memory usage when
>>> decompressing over 20x compared to bz2 (and over 100x while compressing).
>>
>> The memory using during compressing isn't that interesting as long as it
>> isn't prohibitive for sourceware or the machines RMs use.
>> For the decompression, I guess it matters what is actually the memory needed
>> for decompression the -9 gcc tarball, and compare that to minimal memory
>> requirements to compile (not bootstrap) the compiler using typical system
>> compilers.  If compilation of gcc takes more memory than the decompression,
>> then it should be fine, why would anyone try to decompress gcc not to build
>> it afterwards?
> 
> Ok, it doesn't really matter. With gcc-7.1 tarball:
> 
> size: 533084160 (uncompressed)
> 
> -9:
>  xz -d gcc.tar.xz
> 4.71user 0.26system 0:04.97elapsed 100%CPU (0avgtext+0avgdata 67804maxresident)k
> size: 60806928
> 
> -6 (default):
>  xz -d gcc.tar.xz
> 4.88user 0.28system 0:05.17elapsed 99%CPU (0avgtext+0avgdata 10324maxresident)k
> size: 65059664
> 
> So -9 is actually just fine.

ok, updated the script to use xz --best by default. trunk and the gcc-7-branch.

Matthias

Re: [patch] build xz (instead of bz2) compressed tarballs and diffs

[Matthias Klose]

On 18.05.2017 03:34, Richard Biener wrote:
> On Mon, May 15, 2017 at 3:11 AM, Matthias Klose <doko@ubuntu.com> wrote:
>> As discussed on IRC with Jakub and Richard here are is a small patch which
>> builds xz compressed tarballs and diff files.
>>
>> Tested with
>>
>>   maintainer-scripts/gcc_release \
>>         -s snap:trunk -p <old bz2 tarball> diffs sources tarfiles
>>   maintainer-scripts/gcc_release \
>>         -s snap:trunk -p <old xz tarball> diffs sources tarfiles
>>
>> and checked that the new tarball and diff files are compressed using xz.
>>
>> Ok for the trunk and the gcc-7-branch?
> 
> Ok.  The version on trunk can get the bz2 old-tar support removed after the next
> snapshot generation I  think.  Likewise the branch version after 7.2
> was released.

Looks like the copy of the script on gcc.gnu.org affects all active branches.
the May 23 GCC 5 snapshot was created successfully.  Is this acceptable? If yes,
then the patch should probably go to the 5 and 6 branches as well.

Please copy the script again to enable the xz --best compression.

Matthias

Re: [patch] build xz (instead of bz2) compressed tarballs and diffs

[Richard Biener]

On May 24, 2017 1:22:42 AM GMT+02:00, Matthias Klose <doko@ubuntu.com> wrote:
>On 18.05.2017 03:34, Richard Biener wrote:
>> On Mon, May 15, 2017 at 3:11 AM, Matthias Klose <doko@ubuntu.com>
>wrote:
>>> As discussed on IRC with Jakub and Richard here are is a small patch
>which
>>> builds xz compressed tarballs and diff files.
>>>
>>> Tested with
>>>
>>>   maintainer-scripts/gcc_release \
>>>         -s snap:trunk -p <old bz2 tarball> diffs sources tarfiles
>>>   maintainer-scripts/gcc_release \
>>>         -s snap:trunk -p <old xz tarball> diffs sources tarfiles
>>>
>>> and checked that the new tarball and diff files are compressed using
>xz.
>>>
>>> Ok for the trunk and the gcc-7-branch?
>> 
>> Ok.  The version on trunk can get the bz2 old-tar support removed
>after the next
>> snapshot generation I  think.  Likewise the branch version after 7.2
>> was released.
>
>Looks like the copy of the script on gcc.gnu.org affects all active
>branches.

Yes.  Only the trunk script is actually used, so ...

>the May 23 GCC 5 snapshot was created successfully.  Is this
>acceptable? If yes,
>then the patch should probably go to the 5 and 6 branches as well.

... This isn't really necessary.

>Please copy the script again to enable the xz --best compression.
>
>Matthias

Re: [patch] build xz (instead of bz2) compressed tarballs and diffs

[Gerald Pfeifer]

On Tue, 23 May 2017, Matthias Klose wrote:
> Looks like the copy of the script on gcc.gnu.org affects all active branches.

Yeah, I just run into this myself (and scratched my had until I
debugged this, admittedly quickly).

> the May 23 GCC 5 snapshot was created successfully.  Is this acceptable? 

Well, I guess it's a done deed.  Probably a good idea to send a note
to gcc@gcc.gnu.org at least; I added the one below to our main page.

Gerald

Index: index.html
===================================================================
RCS file: /cvs/gcc/wwwdocs/htdocs/index.html,v
retrieving revision 1.1053
diff -u -r1.1053 index.html
--- index.html	9 May 2017 07:20:38 -0000	1.1053
+++ index.html	24 May 2017 19:08:02 -0000
@@ -46,6 +46,10 @@
 <h2 id="news">News</h2>
 <dl>
 
+<dt><span>Weekly snapshots now use xz compression</span>
+    <span class="date">[2017-05-24]</span></dt>
+    <dd>...instead of bzip2.</dd>
+
 <dt><span><a href="gcc-7/">GCC 7.1</a> released</span>
     <span class="date">[2017-05-02]</span></dt>
     <dd></dd>