Re: [ECOS] ZLIB: using gzip decompression in combination with libpng

Re: [ECOS] ZLIB: using gzip decompression in combination with libpng

[stephane.deltour]

Libpng uses inflateInit/deflateInit. So it will rely on the default windowBits parameter. Libpng doesn't seem to work when MAX_WBITS is defined to 15+16. 
To decompress gzip data, I use inflateInit2 with the windowBits set to 15+16. I don't use gzopen because I don't have/use a file system. The data I need to decompress is located in memory.

It seems indeed that the CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP is not ideal. I am sure there must be a combination that works for all, but I haven't figured it out yet.
Thanks for the reply in any case.

Regards,
Stephane 



-----Original Message-----
From: ecos-discuss-owner@ecos.sourceware.org [mailto:ecos-discuss-owner@ecos.sourceware.org] On Behalf Of Andrew Lunn
Sent: zaterdag 26 november 2005 17:32
To: stephane.deltour@i4surf.net
Cc: ecos-discuss@ecos.sourceware.org
Subject: Re: [ECOS] ZLIB: using gzip decompression in combination with libpng

On Fri, Nov 18, 2005 at 02:44:28PM +0100, stephane.deltour@i4surf.net wrote:
> 
> Hi,
> 

> I experienced some problems when trying to read PNG files and 
> decompress gzip compressed data in the same application
> 
> In order to be able to read PNG files I have to set the cdl option 
> CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP to 0. When it is set to 1 an 
> "incorrect header" error is returned when trying to read the PNG file.
> But if the cdl option is set to 0 then gzip compression or 
> decompression is disabled.  Surely it must be possible to be able to 
> do gzip decompression and using PNG files.
> 
> When looking into the zconf.h zlib header file I found the following lines:
> 
> #ifdef CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP
> #undef  MAX_WBITS
> #define MAX_WBITS   15+16 /* 32K LZ77 window */
> #else
> #define NO_GZIP
> #define NO_GZCOMPRESS
> #endif
> 
> When the CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP is set it causes 
> MAX_WBITS to be equal to (15+16) instead of the default 15.
> And this causes DEF_WBITS (in zutil.h) to be also equal to (15+16).
> As a result the default windowBits for decompression is equal to
> (15+16) and this seems to be somehow the cause why libpng can't read 
> png files anymore.
> 
> I was able to fix the issue by either setting MAX_WBITS to 15 (instead 
> of 15+16) or leaving MAX_WBITS to (15+16) but explicitily setting 
> DEF_WBITS to 15 in zconf.h.
>
> This made libpng work with CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP to 1.
> 
> Now my question is:
> Does anybody know why MAX_WBITS was set to 15+16 instead of the default 15 ?

It looks like gzip data is simply zlib data with a header on the front. I guess PNG is using plain zlib, where as gzip is using the gzip format.

There is some documentation about the window bits in zlib.h

----------------------------------------------------------------------

ZEXTERN int ZEXPORT deflateInit2 OF((z_streamp strm,
                                     int  level,
                                     int  method,
                                     int  windowBits,
                                     int  memLevel,
                                     int  strategy));

  This is another version of deflateInit with more compression options. The fields next_in, zalloc, zfree and opaque must be initialized before by the caller.

  The method parameter is the compression method. It must be Z_DEFLATED in this version of the library.

  The windowBits parameter is the base two logarithm of the window size (the size of the history buffer). It should be in the range 8..15 for this version of the library. Larger values of this parameter result in better compression at the expense of memory usage. The default value is 15 if deflateInit is used instead.

  windowBits can also be -8..-15 for raw deflate. In this case, -windowBits determines the window size. deflate() will then generate raw deflate data with no zlib header or trailer, and will not compute an adler32 check value.

  windowBits can also be greater than 15 for optional gzip encoding. Add 16 to windowBits to write a simple gzip header and trailer around the compressed data instead of a zlib wrapper. The gzip header will have no file name, no extra data, no comment, no modification time (set to zero), no header crc, and the operating system will be set to 255 (unknown).  If a gzip stream is being written, strm->adler is a crc32 instead of an adler32.

----------------------------------------------------------------------

So the core of the library should be able to do both formats. The problem is probably in the way you are calling the core library. How does libpng use the library? How do you call it to decompress your gzip data?

It looks like CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP is a horrible hack. It makes uncompress() work on gzip format data where as it is supposed to take zlib format data. I guess libpng is expecting zlib to do what it normally does and this eCos hack is breaking it.

It looks like if you call gzopen it will correctly handle the header when CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP is zero. However i guess you are not using gzopen(), but uncompress()?

        Andrew



--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss





-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] ZLIB: using gzip decompression in combination with libpng

[Andrew Lunn]

On Mon, Nov 28, 2005 at 04:43:17PM +0100, stephane.deltour@i4surf.net wrote:
> 

> Libpng uses inflateInit/deflateInit. So it will rely on the default
> windowBits parameter. Libpng doesn't seem to work when MAX_WBITS is
> defined to 15+16.  To decompress gzip data, I use inflateInit2 with
> the windowBits set to 15+16. I don't use gzopen because I don't
> have/use a file system. The data I need to decompress is located in
> memory.

You probably want to take a look at gzopen and see how it works. You
can then see how to stip the gzip header off your data and call
inflateInit2() etc.

> It seems indeed that the CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP is
> not ideal.

No. I think we need to add an eCos function gzuncompress() which
correctly handles the gzip header and it would be nice to check the
default of CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP to zero, but i
suspect that will break too many systems.

        Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] ZLIB: using gzip decompression in combination with libpng

[Andrew Lunn]

On Fri, Nov 18, 2005 at 02:44:28PM +0100, stephane.deltour@i4surf.net wrote:
> 
> Hi,
> 

> I experienced some problems when trying to read PNG files and
> decompress gzip compressed data in the same application
> 
> In order to be able to read PNG files I have to set the cdl option
> CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP to 0. When it is set to 1 an
> "incorrect header" error is returned when trying to read the PNG
> file.  But if the cdl option is set to 0 then gzip compression or
> decompression is disabled.  Surely it must be possible to be able to
> do gzip decompression and using PNG files.
> 
> When looking into the zconf.h zlib header file I found the following lines:
> 
> #ifdef CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP
> #undef  MAX_WBITS
> #define MAX_WBITS   15+16 /* 32K LZ77 window */
> #else
> #define NO_GZIP
> #define NO_GZCOMPRESS
> #endif
> 
> When the CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP is set it causes
> MAX_WBITS to be equal to (15+16) instead of the default 15.
> And this causes DEF_WBITS (in zutil.h) to be also equal to (15+16).
> As a result the default windowBits for decompression is equal to
> (15+16) and this seems to be somehow the cause why libpng can't read
> png files anymore.
> 
> I was able to fix the issue by either setting MAX_WBITS to 15
> (instead of 15+16) or leaving MAX_WBITS to (15+16) but explicitily
> setting DEF_WBITS to 15 in zconf.h.
>
> This made libpng work with CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP to 1.
> 
> Now my question is:
> Does anybody know why MAX_WBITS was set to 15+16 instead of the default 15 ?

It looks like gzip data is simply zlib data with a header on the
front. I guess PNG is using plain zlib, where as gzip is using the
gzip format.

There is some documentation about the window bits in zlib.h

----------------------------------------------------------------------

ZEXTERN int ZEXPORT deflateInit2 OF((z_streamp strm,
                                     int  level,
                                     int  method,
                                     int  windowBits,
                                     int  memLevel,
                                     int  strategy));

  This is another version of deflateInit with more compression
options. The fields next_in, zalloc, zfree and opaque must be
initialized before by the caller.

  The method parameter is the compression method. It must be Z_DEFLATED
in this version of the library.

  The windowBits parameter is the base two logarithm of the window
size (the size of the history buffer). It should be in the range 8..15
for this version of the library. Larger values of this parameter
result in better compression at the expense of memory usage. The
default value is 15 if deflateInit is used instead.

  windowBits can also be -8..-15 for raw deflate. In this case,
-windowBits determines the window size. deflate() will then generate
raw deflate data with no zlib header or trailer, and will not compute
an adler32 check value.

  windowBits can also be greater than 15 for optional gzip
encoding. Add 16 to windowBits to write a simple gzip header and
trailer around the compressed data instead of a zlib wrapper. The gzip
header will have no file name, no extra data, no comment, no
modification time (set to zero), no header crc, and the operating
system will be set to 255 (unknown).  If a gzip stream is being
written, strm->adler is a crc32 instead of an adler32.

----------------------------------------------------------------------

So the core of the library should be able to do both formats. The
problem is probably in the way you are calling the core library. How
does libpng use the library? How do you call it to decompress your
gzip data?

It looks like CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP is a horrible
hack. It makes uncompress() work on gzip format data where as it is
supposed to take zlib format data. I guess libpng is expecting zlib to
do what it normally does and this eCos hack is breaking it.

It looks like if you call gzopen it will correctly handle the header
when CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP is zero. However i guess
you are not using gzopen(), but uncompress()?

        Andrew



-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] ZLIB: using gzip decompression in combination with libpng

[stephane.deltour]

Hi,

I experienced some problems when trying to read PNG files and decompress gzip compressed data in the same application

In order to be able to read PNG files I have to set the cdl option CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP  to 0. When it is set to 1 an "incorrect header" error is returned when trying to read the PNG file.

But if the cdl option is set to 0 then gzip compression or decompression is disabled.
Surely it must be possible to be able to do gzip decompression and using PNG files.

When looking into the zconf.h zlib header file I found the following lines:

#ifdef CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP
#undef  MAX_WBITS
#define MAX_WBITS   15+16 /* 32K LZ77 window */
#else
#define NO_GZIP
#define NO_GZCOMPRESS
#endif

When the CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP is set it causes MAX_WBITS to be equal to (15+16) instead of the default 15.
And this causes DEF_WBITS (in zutil.h) to be also equal to (15+16).
As a result the default windowBits for decompression is equal to (15+16) and this seems to be somehow the cause why libpng can't read png files anymore.

I was able to fix the issue by either setting MAX_WBITS to 15 (instead of 15+16) or leaving MAX_WBITS to (15+16) but explicitily setting DEF_WBITS to 15 in zconf.h.
This made libpng work with CYGSEM_COMPRESS_ZLIB_DEFLATE_MAKES_GZIP to 1.

Now my question is:
Does anybody know why MAX_WBITS was set to 15+16 instead of the default 15 ?

Thanks,
regards Stephane Deltour




-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss