gzip 1.4-1 creates corrupt archive

gzip 1.4-1 creates corrupt archive

[Peter]

To trim a log file on my windows box I tried the lines below in a bash
script. (my bash is 4.1.10(4))
 
cat "$LOG" | gzip > "$LOG.old.gz"
> "$LOG"
 
The resulting archive was corrupt.
Directly compressing like  gzip "$LOG"  works OK but is not suitable in my
case as the logging service constantly writes to the file
I also tried the -c and -a options of gzip but did not succeed.
From my investigation I suspect there is something with textmode and binmode
but I do not fully understand how to deal with it.
 
What I finally found is that bzip2 works in the above line.
So my solution is to use bzip2 instead of gzip.
 
So my question is:  why can't  gzip  do it automatically if bzip2 can?
 
 


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gzip 1.4-1 creates corrupt archive

[K Stahl]

> cat "$LOG" | gzip > "$LOG.old.gz" > "$LOG"

Don't have access to my machine at the moment, but try adding the '-c'
options to gzip (e.g. gzip -c ...)

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gzip 1.4-1 creates corrupt archive

[Ryan Johnson]

On 25/08/2012 11:45 AM, K Stahl wrote:
>> cat "$LOG" | gzip > "$LOG.old.gz" > "$LOG"
> Don't have access to my machine at the moment, but try adding the '-c'
> options to gzip (e.g. gzip -c ...)
He said he tried that and it didn't work...

Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gzip 1.4-1 creates corrupt archive

[K Stahl]

Missed that section, was reading from my phone.  Anyway, how about
instead of chastising someone for trying to help, you come up with an
alternative solution?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gzip 1.4-1 creates corrupt archive

[Earnie Boyd]

On Sun, Aug 26, 2012 at 12:38 AM, K Stahl wrote:
> Missed that section, was reading from my phone.  Anyway, how about
> instead of chastising someone for trying to help, you come up with an
> alternative solution?

Stating that you missed something in a post isn't really chastising as
much as it is criticism that you need to pay attention more.  Reading
from a phone isn't an excuse for miscommunication.

-- 
Earnie
-- https://sites.google.com/site/earnieboyd

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gzip 1.4-1 creates corrupt archive

[Ryan Johnson]

On 27/08/2012 7:29 AM, Earnie Boyd wrote:
> On Sun, Aug 26, 2012 at 12:38 AM, K Stahl wrote:
>> Missed that section, was reading from my phone.  Anyway, how about
>> instead of chastising someone for trying to help, you come up with an
>> alternative solution?
> Stating that you missed something in a post isn't really chastising as
> much as it is criticism that you need to pay attention more.  Reading
> from a phone isn't an excuse for miscommunication.
>
*sigh*

It was actually neither. Somebody who seemed to have a clue how to 
diagnose this weirdness happened to miss a relevant detail. Happens to 
all of us, but I figured if I pointed it out early on it would save the 
OP a round trip latency in getting to the next phase of diagnosis, since 
the advice given had already been tried.

Unfortunately, it backfired and introduced 2+ rounds of latency instead.

Back on topic, I can't repro using text or binary (executable) files, 
which makes me wonder whether the issue is the constant appending the OP 
mentioned. Perhaps gzip reads twice from the growing log file in a way 
that confuses it, and bzip2 doesn't? Perhaps there's some weirdness with 
atimes that makes gzip think the file is unchanged when it actually has 
grown?

Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gzip 1.4-1 creates corrupt archive

[KHMan]

On 8/27/2012 8:01 PM, Ryan Johnson wrote:
> On 27/08/2012 7:29 AM, Earnie Boyd wrote:
>> On Sun, Aug 26, 2012 at 12:38 AM, K Stahl wrote:
>>> [snip snip snip]
>> [snip snip snip]
> [snip snip snip]
>
> Back on topic, I can't repro using text or binary (executable)
> files, which makes me wonder whether the issue is the constant
> appending the OP mentioned. Perhaps gzip reads twice from the
> growing log file in a way that confuses it, and bzip2 doesn't?
> Perhaps there's some weirdness with atimes that makes gzip think
> the file is unchanged when it actually has grown?

Same for me on text and binary files. I think OP should test 
non-updated text and binary files, that would certainly help to 
prove or disprove some theories.

Also failed to create a corrupted gzip file by manually appending 
from another terminal during compression. After all, gzip is 
reading from stdin only what it gets from cat, and gzip has been 
handling stdin/stdout since forever. Sounds simple enough...

OP gave (reproduced as follows):
 > To trim a log file on my windows box I tried the lines
 > below in a bash script. (my bash is 4.1.10(4))

cat "$LOG" | gzip > "$LOG.old.gz"
 > "$LOG"

What was the error message reported by gzip for the corrupted 
file? gzip has a very simple file structure, if OP can share a 
corrupt file, it should be pretty easy to learn exactly what 
aspect of the gzip file is broken.

-- 
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gzip 1.4-1 creates corrupt archive

[Warren Young]

On 8/25/2012 12:11 AM, Peter wrote:
>
> cat "$LOG" | gzip > "$LOG.old.gz"

Generally speaking, cat with a single file is a sign you're not using 
redirection to its fullest.  In this case, you get the same effect with 
much less overhead with:

     gzip < "$LOG" > "$LOG.old.gz"

> The resulting archive was corrupt.

Evidenced by...?

I don't disbelieve you, but can't you at least produce a "hello world" 
example file that shows the problem and send it to the list, or put it 
on a public web server somewhere?

>From my investigation I suspect there is something with textmode and binmode
> but I do not fully understand how to deal with it.

Why do you need the textmode hack?

That is to say, what non-Cygwin tools are you using that truly cannot 
cope with Unix LF line endings, thereby requiring Cygwin to do strange 
things in order to appease the native tools?

(Oh, and in case anyone reading this things I'm being a hypocrite after 
my arguments in the SQLite/svn thread, I'll reiterate that I don't like 
interop hacks and would like to avoid them wherever possible.  I'm 
hoping to find out that it is actually possible here.)

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple