RE: Compressing hippos really fast

RE: Compressing hippos really fast

[Phil Betts]

Corinna Vinschen wrote on Tuesday, March 04, 2008 3:43 PM::

> Hi,
> 
> 
> does anybody know about a compression tool which is above all capable
> of compressing really fast?  The compression ratio is only a mild
> concern, it's rather more important that the tool is not acting as
> bottleneck when compressing files which are badly compressable. 
> Unfortunately 
> the usual compression tools are rather interested in a good
> compression than in a good speed when streaming lots of data.
> 
> Here are a couple of disks which are supposed to be backed up.  Right
> now this is done using a script which creats tar.gz archives of all
> disks.  Some of this disks are quite big and contains many files which
> are already compressed.  It turns out that gzipping these disks is
> *the* bottleneck when backing up.
> 
> When not compressing, tar creates archives with 37MB/s.  When creating
> tar.gz archives, the compression takes so much time that the speed
> goes down to 6MB/s.  Using gzip --fast doesn't help much.  bzip is a
> lot slower than gzip.
> 
> So the question is, does anybody know a compression tool which can be
> used with tar, which doesn't slow down the backup by a factor of 6? 
> It would be cool to have a tool which is as quick as the hardware
> compression used in modern tape drives, but that's just dreaming...
> 
> 
> May the hippos be with you,
> Corinna

I had this problem ages ago.  My solution was to run two backups.  
One uncompressed including only files globbing *.gz, *.t[bg]z, *.[zZ], 
*.bz2, *.zip etc, and one for the remainder which was piped 
through gzip.

Even a fast compression algorithm is just wasting time trying to 
compress previously compressed files, and as most compressors work 
on some variant of Lempel Ziv, if they're fed a mixture of 
compressible and incompressible data, the incompressible data 
flushes the dictionary making the compression of the compressible 
part worse.

Phil

Re: Compressing hippos really fast

[Lee D. Rothstein]

Sounds like he needs data-dedupe. Google "data de-duplication" for an 
array of vendors.

Phil Betts wrote:
> Corinna Vinschen wrote on Tuesday, March 04, 2008 3:43 PM::
>
>   
>> Hi,
>>
>>
>> does anybody know about a compression tool which is above all capable
>> of compressing really fast?  The compression ratio is only a mild
>> concern, it's rather more important that the tool is not acting as
>> bottleneck when compressing files which are badly compressable. 
>> Unfortunately 
>> the usual compression tools are rather interested in a good
>> compression than in a good speed when streaming lots of data.
>>
>> Here are a couple of disks which are supposed to be backed up.  Right
>> now this is done using a script which creats tar.gz archives of all
>> disks.  Some of this disks are quite big and contains many files which
>> are already compressed.  It turns out that gzipping these disks is
>> *the* bottleneck when backing up.
>>
>> When not compressing, tar creates archives with 37MB/s.  When creating
>> tar.gz archives, the compression takes so much time that the speed
>> goes down to 6MB/s.  Using gzip --fast doesn't help much.  bzip is a
>> lot slower than gzip.
>>
>> So the question is, does anybody know a compression tool which can be
>> used with tar, which doesn't slow down the backup by a factor of 6? 
>> It would be cool to have a tool which is as quick as the hardware
>> compression used in modern tape drives, but that's just dreaming...
>>
>>
>> May the hippos be with you,
>> Corinna
>>     
>
> I had this problem ages ago.  My solution was to run two backups.  
> One uncompressed including only files globbing *.gz, *.t[bg]z, *.[zZ], 
> *.bz2, *.zip etc, and one for the remainder which was piped 
> through gzip.
>
> Even a fast compression algorithm is just wasting time trying to 
> compress previously compressed files, and as most compressors work 
> on some variant of Lempel Ziv, if they're fed a mixture of 
> compressible and incompressible data, the incompressible data 
> flushes the dictionary making the compression of the compressible 
> part worse.
>
> Phil
>
>   

Re: Compressing hippos really fast

[Corinna Vinschen]

On Mar  4 18:35, Phil Betts wrote:
> I had this problem ages ago.  My solution was to run two backups.  
> One uncompressed including only files globbing *.gz, *.t[bg]z, *.[zZ], 
> *.bz2, *.zip etc, and one for the remainder which was piped 
> through gzip.

I guess that's the way to go in the long run.


Corinna

Re: Compressing hippos really fast

[Robert Pendell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Corinna Vinschen wrote:
| On Mar  4 16:23, Buchbinder, Barry (NIH/NIAID) [E] wrote:
|> Corinna Vinschen wrote:
|>> does anybody know about a compression tool which is above all
|>> capable of compressing really fast?
|> I know that this is very "un-Linuxy", but if you don't really
|> need tar, you might consider zip.
|
| Urgh.  I'm not sure that's really an option.  I never tried to use
| zip on a terabyte of data...
|
|
| Corinna
|
I don't think that would work.  I think the linux version is based on
Info-Zip and that one was limited to 2GB archive size.

- --
Robert Pendell
shinji@elite-systems.org

Thawte Web of Trust Notary
CAcert Assurer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH0w1bs1pR2j1qW+sRAuNcAJ41P5hGXU+ruPuK0lQbue36We5gHgCfaIqK
PA277TzK78wmnB7llQMwQoA=
=rlR8
-----END PGP SIGNATURE-----

Re: Compressing hippos really fast

[Corinna Vinschen]

On Mar  4 16:23, Buchbinder, Barry (NIH/NIAID) [E] wrote:
> Corinna Vinschen wrote:
> > does anybody know about a compression tool which is above all
> > capable of compressing really fast?
> 
> I know that this is very "un-Linuxy", but if you don't really
> need tar, you might consider zip.

Urgh.  I'm not sure that's really an option.  I never tried to use
zip on a terabyte of data...


Corinna

RE: Compressing hippos really fast

[Buchbinder, Barry (NIH/NIAID) [E]]

Corinna Vinschen wrote:
> does anybody know about a compression tool which is above all
> capable of compressing really fast?

I know that this is very "un-Linuxy", but if you don't really
need tar, you might consider zip.

The -u (update) option will only compress changed files.  The
rest of the time, is devoted to coping the already compressed
part of the zip and doing the date comparisons.  (One might
let find do the latter.)

And to adopt a previously suggestion, the -n option will just
store (no compression) files with certain suffixes.

In my experience, zip -u doesn't seem particularly fast
(though that is with cygwin over a busy LAN on busy servers),
but it might be worth an experiment.  (And be prepared for
the initial creation of the zip archives being slow.)

Good luck.

- Barry

Re: Compressing hippos really fast

[Igor Peshansky]

On Tue, 4 Mar 2008, Corinna Vinschen wrote:

> On Mar  4 16:34, Owen Rees wrote:
>
> > ...but a passing hippo suggested that lzop <http://www.lzop.org/>
> > would be worth investigating.
>
> Dave Korn is a hippo?
>
> http://cygwin.com/ml/cygwin-talk/2008-q1/msg00096.html

On the internet, nobody knows if you're a hippo.
	Igor
-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_	    pechtcha@cs.nyu.edu | igor@watson.ibm.com
ZZZzz /,`.-'`'    -.  ;-;;,_		Igor Peshansky, Ph.D. (name changed!)
     |,4-  ) )-,_. ,\ (  `'-'		old name: Igor Pechtchanski
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

"That which is hateful to you, do not do to your neighbor.  That is the whole
Torah; the rest is commentary.  Go and study it." -- Rabbi Hillel

RE: Compressing hippos really fast

[Dave Korn]

On 04 March 2008 16:45, maybe.i'll.try@rolling.it.back.one.version wrote:

> On Mar  4 16:34, Owen Rees wrote:
>> --On 04 March 2008 16:43 +0100 Corinna Vinschen wrote:
>> 
>>> does anybody know about a compression tool which is above all capable of
>>> compressing really fast?
>> 
>> I have not used it myself, but a passing hippo suggested that lzop
>> <http://www.lzop.org/> would be worth investigating.
> 
> Dave Korn is a hippo?


   Ssshhhhh!  You're giving my secret away!

<wallows>

    cheers,
      DaveK

-- 
Can't think of a witty .sigline today....

Re: Compressing hippos really fast

[Corinna Vinschen]

On Mar  4 16:33, a hippo wrote:
>   It should be possible to use lzop as an inline filter compressor, more or
> less as a direct drop-in replacement for gzip
> 
> http://www.lzop.org/

Thanks, I'll give it a try.


Corinna

Re: Compressing hippos really fast

[Corinna Vinschen]

On Mar  4 16:34, Owen Rees wrote:
> --On 04 March 2008 16:43 +0100 Corinna Vinschen wrote:
>
>> does anybody know about a compression tool which is above all capable of
>> compressing really fast?
>
> I have not used it myself, but a passing hippo suggested that lzop 
> <http://www.lzop.org/> would be worth investigating.

Dave Korn is a hippo?

http://cygwin.com/ml/cygwin-talk/2008-q1/msg00096.html

;)


Thanks,
Corinna

Re: Compressing hippos really fast

[Owen Rees]

--On 04 March 2008 16:43 +0100 Corinna Vinschen wrote:

> does anybody know about a compression tool which is above all capable of
> compressing really fast?

I have not used it myself, but a passing hippo suggested that lzop 
<http://www.lzop.org/> would be worth investigating.

-- 
Owen Rees
========================================================
Hewlett-Packard Limited.   Registered No: 690597 England
Registered Office:  Cain Road, Bracknell, Berks RG12 1HN

RE: Compressing hippos really fast

[Dave Korn]

On 04 March 2008 15:43, oh.no.my@quotefix.is.borken wrote:

> does anybody know about a compression tool which is above all capable of
> compressing really fast?  The compression ratio is only a mild concern,
> it's rather more important that the tool is not acting as bottleneck
> when compressing files which are badly compressable.  Unfortunately
> the usual compression tools are rather interested in a good compression
> than in a good speed when streaming lots of data.

  Hmm, I came across something much like this lately while researching
compression: take a look at LZO.  It focusses more on uncompression speed
but is also supposed to be fairly fast for compression. 

http://www.oberhumer.com/opensource/lzo/

  It should be possible to use lzop as an inline filter compressor, more or
less as a direct drop-in replacement for gzip

http://www.lzop.org/



    cheers,
      DaveK

-- 
Can't think of a witty .sigline today....

Compressing hippos really fast

[Corinna Vinschen]

Hi,


does anybody know about a compression tool which is above all capable of
compressing really fast?  The compression ratio is only a mild concern,
it's rather more important that the tool is not acting as bottleneck
when compressing files which are badly compressable.  Unfortunately
the usual compression tools are rather interested in a good compression
than in a good speed when streaming lots of data.

Here are a couple of disks which are supposed to be backed up.  Right
now this is done using a script which creats tar.gz archives of all
disks.  Some of this disks are quite big and contains many files which
are already compressed.  It turns out that gzipping these disks is *the*
bottleneck when backing up.

When not compressing, tar creates archives with 37MB/s.  When creating
tar.gz archives, the compression takes so much time that the speed goes
down to 6MB/s.  Using gzip --fast doesn't help much.  bzip is a lot
slower than gzip.

So the question is, does anybody know a compression tool which can be
used with tar, which doesn't slow down the backup by a factor of 6?  It
would be cool to have a tool which is as quick as the hardware
compression used in modern tape drives, but that's just dreaming...


May the hippos be with you,
Corinna