xz-compressed release tarballs?

xz-compressed release tarballs?

[Jim Meyering]

I have just downloaded 20MB of gdb-7.4.tar.bz2
and lamented that it wasn't xz-compressed.

With xz -9e, that would have been 25% smaller,
at 15282412 bytes (contrast w/20614020 for .bz2).
The .xz tarball would have uncompressed more quickly, too.

For reference, gnome went xz-only in September:

  http://thread.gmane.org/gmane.comp.gnome.devel.announce/210

and I have done the same with coreutils-8.14 and 8.15 and grep-2.10.
No one has complained, since xz is available nearly everywhere,
these days, and on the few/aging systems for which it is not
already packaged, it's easy to build the latest from source.

Obviously, gdb needn't drop .gz and .bz2 tarballs now or ever,
but please do consider distributing .xz-compressed tarballs.

Re: xz-compressed release tarballs?

[Jan Kratochvil]

On Thu, 26 Jan 2012 21:21:47 +0100, Jim Meyering wrote:
> With xz -9e, that would have been 25% smaller,
> at 15282412 bytes (contrast w/20614020 for .bz2).
> The .xz tarball would have uncompressed more quickly, too.

+1

That slow bzip2 decompression is a continuous pain during packaging, staring
at bzip2 -dc on each gdb.spec debugging cycle.

bzip2 -dc = 0m3.641s -> xz -dc = 0m1.264s

Previous proposal by me:
	xz for the new release tip
	http://sourceware.org/ml/gdb/2009-09/msg00341.html


Thanks,
Jan

Re: xz-compressed release tarballs?

[Eli Zaretskii]

> Date: Thu, 26 Jan 2012 21:31:48 +0100
> From: Jan Kratochvil <jan.kratochvil@redhat.com>
> Cc: gdb@sourceware.org
> 
> On Thu, 26 Jan 2012 21:21:47 +0100, Jim Meyering wrote:
> > With xz -9e, that would have been 25% smaller,
> > at 15282412 bytes (contrast w/20614020 for .bz2).
> > The .xz tarball would have uncompressed more quickly, too.
> 
> +1
> 
> That slow bzip2 decompression is a continuous pain during packaging, staring
> at bzip2 -dc on each gdb.spec debugging cycle.

As long as we don't drop bz2 and gz, that pain will never go away.

> Previous proposal by me:
> 	xz for the new release tip
> 	http://sourceware.org/ml/gdb/2009-09/msg00341.html

Please think a little about those who don't necessarily have a tar
that knows about xz.  The world doesn't end with GNU/Linux.

Re: xz-compressed release tarballs?

[Jan Kratochvil]

On Thu, 26 Jan 2012 22:30:39 +0100, Eli Zaretskii wrote:
> > That slow bzip2 decompression is a continuous pain during packaging, staring
> > at bzip2 -dc on each gdb.spec debugging cycle.
> 
> As long as we don't drop bz2 and gz, that pain will never go away.

gdb.src.rpm will just use gdb-7.4.tar.xz so the pain will be gone.


> Please think a little about those who don't necessarily have a tar
> that knows about xz.  The world doesn't end with GNU/Linux.

Current state: Shipped .tar.gz and .tar.bz2.
Possibility (a): Ship .tar.gz, .tar.bz2 and .tar.xz.
Possibility (b) - preferred by me: Ship only .tar.gz and .tar.xz.

IMO there are only few systems which already handle .tar.bz2 and which still
do not handle .tar.xz.  And those few system can use .tar.gz instead of
.tar.bz2.


Thanks,
Jan

Re: xz-compressed release tarballs?

[Samuel Bronson]

On Thu, Jan 26, 2012 at 4:36 PM, Jan Kratochvil
<jan.kratochvil@redhat.com> wrote:
> On Thu, 26 Jan 2012 22:30:39 +0100, Eli Zaretskii wrote:
>> > That slow bzip2 decompression is a continuous pain during packaging, staring
>> > at bzip2 -dc on each gdb.spec debugging cycle.
>>
>> As long as we don't drop bz2 and gz, that pain will never go away.
>
> gdb.src.rpm will just use gdb-7.4.tar.xz so the pain will be gone.
>
>> Please think a little about those who don't necessarily have a tar
>> that knows about xz.  The world doesn't end with GNU/Linux.
>
> Current state: Shipped .tar.gz and .tar.bz2.
> Possibility (a): Ship .tar.gz, .tar.bz2 and .tar.xz.
> Possibility (b) - preferred by me: Ship only .tar.gz and .tar.xz.

Hear, hear! I was just wondering why GCC didn't switch to this yet the
other day.

> IMO there are only few systems which already handle .tar.bz2 and which still
> do not handle .tar.xz.  And those few system can use .tar.gz instead of
> .tar.bz2.

And often already do because of the speed problem.

Re: xz-compressed release tarballs?

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 433 bytes --]

On Thursday 26 January 2012 16:30:39 Eli Zaretskii wrote:
> > Previous proposal by me:
> > 	xz for the new release tip
> > 	http://sourceware.org/ml/gdb/2009-09/msg00341.html
> 
> Please think a little about those who don't necessarily have a tar
> that knows about xz.  The world doesn't end with GNU/Linux.

why does tar have to know about xz ?  stdin hasn't suddenly stopped working.
	xzcat foo.tar.xz | tar xf -
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: xz-compressed release tarballs?

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 180 bytes --]

On Thursday 26 January 2012 16:36:12 Jan Kratochvil wrote:
> Possibility (b) - preferred by me: Ship only .tar.gz and .tar.xz.

this seems to address everyone's concerns ...
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: xz-compressed release tarballs?

[Ralf Corsepius]

On 01/26/2012 10:30 PM, Eli Zaretskii wrote:
>> Date: Thu, 26 Jan 2012 21:31:48 +0100
>> From: Jan Kratochvil<jan.kratochvil@redhat.com>
>> Cc: gdb@sourceware.org
>>
>> On Thu, 26 Jan 2012 21:21:47 +0100, Jim Meyering wrote:
>>> With xz -9e, that would have been 25% smaller,
>>> at 15282412 bytes (contrast w/20614020 for .bz2).

+1

>> Previous proposal by me:
>> 	xz for the new release tip
>> 	http://sourceware.org/ml/gdb/2009-09/msg00341.html
>
> Please think a little about those who don't necessarily have a tar
> that knows about xz.  The world doesn't end with GNU/Linux.

These days, even FreeBSD is shipping their OS tar.xz (rsp. txz) packaged:
c.f. ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/9.0-RELEASE/

Ralf

Re: xz-compressed release tarballs?

[Joel Brobecker]

> Current state: Shipped .tar.gz and .tar.bz2.
> Possibility (a): Ship .tar.gz, .tar.bz2 and .tar.xz.
> Possibility (b) - preferred by me: Ship only .tar.gz and .tar.xz.

To me, it is not a question of whether tar handles .xz or not.
It's a question whether the system has an xz decompresser or not
(tar just calls gzip/bzip/xz as far as I can tell).  The xzutils seem
to be supported on quite a wide range of platforms, but I do not know
whether they are now readily available. I know that on my (oldish)
system, it was not installed by default.

I think we can make everyone happy by simply adding the .xz format,
giving us 3 release tarballs. That would cost about 25-30MB extra
times 2-3 releases a year. I am not opposed to getting rid of the
.bz2 format, however.

-- 
Joel

Re: xz-compressed release tarballs?

[Eli Zaretskii]

> From: Mike Frysinger <vapier@gentoo.org>
> Date: Thu, 26 Jan 2012 20:49:32 -0500
> Cc: Jan Kratochvil <jan.kratochvil@redhat.com>,
>  jim@meyering.net
> 
> On Thursday 26 January 2012 16:30:39 Eli Zaretskii wrote:
> > > Previous proposal by me:
> > > 	xz for the new release tip
> > > 	http://sourceware.org/ml/gdb/2009-09/msg00341.html
> > 
> > Please think a little about those who don't necessarily have a tar
> > that knows about xz.  The world doesn't end with GNU/Linux.
> 
> why does tar have to know about xz ?  stdin hasn't suddenly stopped working.
> 	xzcat foo.tar.xz | tar xf -

Because (a) this is less convenient to type, and (b) you assume that
ports of tar to non-Posix platforms know how to read binary streams
from stdin, but that assumption is false in quite a few existing and
widely used ports of tar.

Re: xz-compressed release tarballs?

[Eli Zaretskii]

> From: Mike Frysinger <vapier@gentoo.org>
> Date: Thu, 26 Jan 2012 20:50:13 -0500
> Cc: Jan Kratochvil <jan.kratochvil@redhat.com>, Eli Zaretskii <eliz@gnu.org>, jim@meyering.net
> 
> On Thursday 26 January 2012 16:36:12 Jan Kratochvil wrote:
> > Possibility (b) - preferred by me: Ship only .tar.gz and .tar.xz.
> 
> this seems to address everyone's concerns ...

As long as .tar.gz is not dropped (which several GNU projects did
already), yes.

Re: xz-compressed release tarballs?

[Jan Kratochvil]

On Thu, 26 Jan 2012 21:31:48 +0100, Jan Kratochvil wrote:
> That slow bzip2 decompression is a continuous pain during packaging, staring
> at bzip2 -dc on each gdb.spec debugging cycle.

I have realized it may not help in this case as the packaging commonly
involves snapshots
	ftp://sourceware.org/pub/gdb/snapshots/current/

Which are already shipped only as .tar.bz2, with no .tar.gz there.

I am fine with providing the snapshots only as .tar.xz but that may bring more
disagreements.  The snapshots may not be so world visible like the releases.
Are there concerns about snapshots only in the .tar.xz format?


Regards,
Jan

Re: xz-compressed release tarballs?

[Eli Zaretskii]

> Date: Fri, 27 Jan 2012 09:01:51 +0400
> From: Joel Brobecker <brobecker@adacore.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, jim@meyering.net, gdb@sourceware.org
> 
> I think we can make everyone happy by simply adding the .xz format,
> giving us 3 release tarballs. That would cost about 25-30MB extra
> times 2-3 releases a year. I am not opposed to getting rid of the
> .bz2 format, however.

Fine with me, although I don't really understand what's problem, as
disk space is no longer at a premium and neither is network bandwidth
(certainly not for those who lobby for .xz).  Sometimes it looks to me
like people are simply pushing to use newer tools just because they
shine brighter.

Re: xz-compressed release tarballs?

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1070 bytes --]

On Friday 27 January 2012 04:10:32 Eli Zaretskii wrote:
> From: Mike Frysinger <vapier@gentoo.org>
> > On Thursday 26 January 2012 16:30:39 Eli Zaretskii wrote:
> > > > Previous proposal by me:
> > > > 	xz for the new release tip
> > > > 	http://sourceware.org/ml/gdb/2009-09/msg00341.html
> > > 
> > > Please think a little about those who don't necessarily have a tar
> > > that knows about xz.  The world doesn't end with GNU/Linux.
> > 
> > why does tar have to know about xz ?  stdin hasn't suddenly stopped
> > working.
> > 
> > 	xzcat foo.tar.xz | tar xf -
> 
> Because (a) this is less convenient to type

you gotta be pulling my leg with this

> and (b) you assume that ports of tar to non-Posix platforms know how to read
> binary streams from stdin, but that assumption is false in quite a few
> existing and widely used ports of tar.

still a non-issue:
	unxz foo.tar.xz
	tar xf foo.tar

forcing people to live in the past because you can't be bothered to type a few 
extra characters on your legacy systems is ridiculous
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: xz-compressed release tarballs?

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 303 bytes --]

On Friday 27 January 2012 04:26:24 Jan Kratochvil wrote:
> I am fine with providing the snapshots only as .tar.xz but that may bring
> more disagreements.  The snapshots may not be so world visible like the
> releases. Are there concerns about snapshots only in the .tar.xz format?

fine by me :)
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: xz-compressed release tarballs?

[Marek Polacek]

On 01/27/2012 09:28 PM, Mike Frysinger wrote:
> On Friday 27 January 2012 04:26:24 Jan Kratochvil wrote:
>> I am fine with providing the snapshots only as .tar.xz but that may bring
>> more disagreements.  The snapshots may not be so world visible like the
>> releases. Are there concerns about snapshots only in the .tar.xz format?
> 
> fine by me :)

FWIW, I'm all for this idea.

	Marek

Re: xz-compressed release tarballs?

[Eli Zaretskii]

> From: Mike Frysinger <vapier@gentoo.org>
> Date: Fri, 27 Jan 2012 13:41:40 -0500
> Cc: gdb@sourceware.org,
>  jan.kratochvil@redhat.com,
>  jim@meyering.net
> 
> On Friday 27 January 2012 04:10:32 Eli Zaretskii wrote:
> > From: Mike Frysinger <vapier@gentoo.org>
> > > On Thursday 26 January 2012 16:30:39 Eli Zaretskii wrote:
> > > > > Previous proposal by me:
> > > > > 	xz for the new release tip
> > > > > 	http://sourceware.org/ml/gdb/2009-09/msg00341.html
> > > > 
> > > > Please think a little about those who don't necessarily have a tar
> > > > that knows about xz.  The world doesn't end with GNU/Linux.
> > > 
> > > why does tar have to know about xz ?  stdin hasn't suddenly stopped
> > > working.
> > > 
> > > 	xzcat foo.tar.xz | tar xf -
> > 
> > Because (a) this is less convenient to type
> 
> you gotta be pulling my leg with this

No more than you are pulling mine with a few extra bytes.

> > and (b) you assume that ports of tar to non-Posix platforms know how to read
> > binary streams from stdin, but that assumption is false in quite a few
> > existing and widely used ports of tar.
> 
> still a non-issue:
> 	unxz foo.tar.xz
> 	tar xf foo.tar

So is pulling a few extra bytes through the wire.

> forcing people to live in the past because you can't be bothered to type a few 
> extra characters on your legacy systems is ridiculous

I see nothing ridiculous in a request to cater to a situation of your
fellow developer of Free Software, and don't understand how can you
ridicule such a request.

Re: xz-compressed release tarballs?

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 650 bytes --]

On Saturday 28 January 2012 03:18:47 Eli Zaretskii wrote:
> I see nothing ridiculous in a request to cater to a situation of your
> fellow developer of Free Software, and don't understand how can you
> ridicule such a request.

sorry, but i really can't find anything worth replying to here.  being a free 
software developer doesn't mean being able to force people to adhere to what 
works on their personal legacy systems where the user refuses to upgrade the 
software.  that is unreasonable.

i've got plenty of old systems, but i don't try and force everyone else to try 
and support them at the detriment of future progress.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: xz-compressed release tarballs?

[Eli Zaretskii]

> From: Mike Frysinger <vapier@gentoo.org>
> Date: Sat, 28 Jan 2012 18:09:15 -0500
> Cc: gdb@sourceware.org,
>  jan.kratochvil@redhat.com,
>  jim@meyering.net
> 
> On Saturday 28 January 2012 03:18:47 Eli Zaretskii wrote:
> > I see nothing ridiculous in a request to cater to a situation of your
> > fellow developer of Free Software, and don't understand how can you
> > ridicule such a request.
> 
> sorry, but i really can't find anything worth replying to here.

But you replied anyway.

> being a free software developer doesn't mean being able to force
> people to adhere to what works on their personal legacy systems
> where the user refuses to upgrade the software.  that is
> unreasonable.

I cannot and didn't force anyone to do anything.  I made a request.
Disagreeing with that request and acting against it is entirely
appropriate and understood; calling it ridiculous or using other
derogatory language is not.

> i've got plenty of old systems

There's nothing in MS-Windows systems that can be called "legacy" or
"old".

> i don't try and force everyone else to try and support them at the
> detriment of future progress.

Neither did I.

Re: xz-compressed release tarballs?

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1102 bytes --]

On Saturday 28 January 2012 22:53:13 Eli Zaretskii wrote:
> From: Mike Frysinger <vapier@gentoo.org>
> > being a free software developer doesn't mean being able to force
> > people to adhere to what works on their personal legacy systems
> > where the user refuses to upgrade the software.  that is
> > unreasonable.
> 
> I cannot and didn't force anyone to do anything.  I made a request.
> Disagreeing with that request and acting against it is entirely
> appropriate and understood; calling it ridiculous or using other
> derogatory language is not.

sorry, but i guess that's my interpretation.  asking people to not use xz even 
though you've shown no reason other than tar itself on deficient platforms 
cannot directly read the compression is ridiculous.  it is absolutely trivial 
to decompress it first if you are on such a system as i showed.

> There's nothing in MS-Windows systems that can be called "legacy" or
> "old".

seriously ?  there is *plenty* in MS-Windows that can be called both those 
things.  even MS documents these in great detail in the MSDN.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: xz-compressed release tarballs?

[Alfred M. Szmidt]

Please move this nonsense off-list.

Re: xz-compressed release tarballs?

[Tom Tromey]

>>>>> "Joel" == Joel Brobecker <brobecker@adacore.com> writes:

Joel> I think we can make everyone happy by simply adding the .xz format,
Joel> giving us 3 release tarballs. That would cost about 25-30MB extra
Joel> times 2-3 releases a year. I am not opposed to getting rid of the
Joel> .bz2 format, however.

I'm in favor of adding .xz and removing .bz2.
It doesn't hurt anybody and it will help some people with limited
bandwidth.  Other GNU programs seem to have done this without issue.

Tom

Re: xz-compressed release tarballs?

[Stan Shebs]

On 1/28/12 3:09 PM, Mike Frysinger wrote:
> On Saturday 28 January 2012 03:18:47 Eli Zaretskii wrote:
>> I see nothing ridiculous in a request to cater to a situation of your
>> fellow developer of Free Software, and don't understand how can you
>> ridicule such a request.
> sorry, but i really can't find anything worth replying to here.  being a free
> software developer doesn't mean being able to force people to adhere to what
> works on their personal legacy systems where the user refuses to upgrade the
> software.  that is unreasonable.
>
> i've got plenty of old systems, but i don't try and force everyone else to try
> and support them at the detriment of future progress.
> -mike

I have to take exception to this; there are several good reasons to make 
allowance for older and legacy systems.

One is that our volunteers are just that - working on the software on 
their own time, and without spare cash (or time) to upgrade to the 
latest and greatest hardware or software.

Another is that free software has frequently been able to displace 
proprietary software on older systems, as a way to extend useful life.  
This has been a major factor in the spread and acceptance of free software.

A third is that the requirement to run on a variety of systems is good 
for self-discipline.  When our code has "all the world's a Vax", "all 
the world's an x86", or "all the world is Linux" assumptions wired into 
it, that reduces our ability to adapt when all the world changes to 
something else.

So while we can certainly debate whether this feature or that helps or 
hinders future progress, I don't think it's in our interests to 
disparage all the legacy support.

Stan

Re: xz-compressed release tarballs?

[Pedro Alves]

On 01/30/2012 07:54 PM, Stan Shebs wrote:

> So while we can certainly debate whether this feature or that helps or hinders future progress, I don't think it's in our interests to disparage all the legacy support.

Indeed.

Keeping things in perspective, it doesn't seem like we're at the verge of a shift of
mentality where people are rushing out to wack out all legacy support from GDB.  Although, it has
been pointed out that several GNU packages are already shipping xz only, including coreutils and
grep, without complaint -- this is a very good indication that dropping .gz isn't that much of
an obstacle as one might think at first sight.  But, for GDB, let's leave that for some other
day in the somewhat distant future, and move on.

The options on the table were:

 (a) tar.gz, tar.bz2, tar.xz
 (b) tar.gz, tar.xz
 (c) tar.gz, tar.bz2  (status quo, do nothing, for completeness)

tar.gz was still there in all proposals, which caters to older, legacy systems.

I'm also in favor of replacing .bz2 with .xz, keeping .gz, option (b), and it
seemed to me it got the popular vote.  IMO, Just Do It (TM).

-- 
Pedro Alves

Re: xz-compressed release tarballs?

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 824 bytes --]

On Monday 30 January 2012 14:54:04 Stan Shebs wrote:
> So while we can certainly debate whether this feature or that helps or
> hinders future progress, I don't think it's in our interests to
> disparage all the legacy support.

i didn't say "legacy isn't supported at all".  xz will work on legacy systems 
just fine provided you d/l and build the decompressor (which is freely 
available).  what i did say is that we can't restrict ourselves based on what 
random old versions of tar people have on their random systems which happens 
to not detect xz transparently.

your `tar` doesn't support -J, or -I, or -a doesn't detect xz ?  then run 
`unxz` yourself first.  there is no actual problem here.  the only thing i've 
seen from Eli is that he doesn't want to have to run `unxz ... && tar xf ...`.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]