snapshots (archive files) are too big ... Why?

snapshots (archive files) are too big ... Why?

[Houder]

Hi Corinna,

Just curious if there is a reason ...

I was able to extract cygpath.exe from cygwin-inst-20160121.tar.xz 
(snapshots)
in order to test your last modification to cygpath.cc. No problem here.

However, when examining the contents of the archive, I was surprised to 
find
the SAME version of cygpath.exe at least three times ...

The size of cygwin-inst-<date>.tar.xz (and cygwin-src-<date>.tar.xz) has 
grown
(suddenly) by a factor of 3 or 4 since 2015-07-20 ...

The same applies to winsup-src-<date>.tar.xz (since 2016-01-15) ...

In all cases it is because the archive contains the SAME version of a 
file at
least three times (as far as I can tell).

To summarize: No, I am not reporting a problem here; I am just _curious_ 
as to
why these archive are so much bigger than they (apparently) need to be 
...

Regards,
Henri

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: snapshots (archive files) are too big ... Why?

[Andrey Repin]

Greetings, Houder!

Judging from

tar tf cygwin-inst-20160121.tar.xz

the archive is composed in more than one step.
The packaging script is really ought for a thorough investigation.


-- 
With best regards,
Andrey Repin
Saturday, January 23, 2016 17:23:33

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: snapshots (archive files) are too big ... Why?

[Andrey Repin]

Greetings, Houder!

> Just curious if there is a reason ...

> I was able to extract cygpath.exe from cygwin-inst-20160121.tar.xz 
> (snapshots)
> in order to test your last modification to cygpath.cc. No problem here.

> However, when examining the contents of the archive, I was surprised to 
> find
> the SAME version of cygpath.exe at least three times ...

> The size of cygwin-inst-<date>.tar.xz (and cygwin-src-<date>.tar.xz) has 
> grown
> (suddenly) by a factor of 3 or 4 since 2015-07-20 ...

> The same applies to winsup-src-<date>.tar.xz (since 2016-01-15) ...

> In all cases it is because the archive contains the SAME version of a 
> file at
> least three times (as far as I can tell).

> To summarize: No, I am not reporting a problem here; I am just _curious_ 
> as to
> why these archive are so much bigger than they (apparently) need to be 
> ...

But this is actually a problem.

http://cygwin.com/snapshots/x86/cygwin-inst-20160121.tar

All tools reporting from 3 to 5 entries of each file in the archive.


-- 
With best regards,
Andrey Repin
Saturday, January 23, 2016 17:17:59

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: snapshots (archive files) are too big ... Why?

[Houder]

Hi Andrey,

On 2016-01-23 15:21, Andrey Repin wrote:

> But this is actually a problem.

Not sure ...extraction appears NOT to be a problem ... (yes, the same
file is extracted multiple times).

On the other hand, the "big" archive file might indicate there is room
for improvement ...

However, it might be so that the person who wrote the script believes
that she/he can only rely on Lamport's clock:

The "version" of the file, that survives the extraction process, will
be the one that was last appended to the archive file ...

     http://www.gnu.org/software/tar/manual/tar.html#SEC59

See 4.2.2.2 Multiple Members with the Same Name.

Of course, I am just guessing ... that is why I wrote: I am curious.

Regards,
Henri

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: snapshots (archive files) are too big ... Why?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1629 bytes --]

On Jan 23 13:59, Houder wrote:
> Hi Corinna,
> 
> Just curious if there is a reason ...
> 
> I was able to extract cygpath.exe from cygwin-inst-20160121.tar.xz
> (snapshots)
> in order to test your last modification to cygpath.cc. No problem here.
> 
> However, when examining the contents of the archive, I was surprised to find
> the SAME version of cygpath.exe at least three times ...
> 
> The size of cygwin-inst-<date>.tar.xz (and cygwin-src-<date>.tar.xz) has
> grown
> (suddenly) by a factor of 3 or 4 since 2015-07-20 ...
> 
> The same applies to winsup-src-<date>.tar.xz (since 2016-01-15) ...
> 
> In all cases it is because the archive contains the SAME version of a file
> at
> least three times (as far as I can tell).
> 
> To summarize: No, I am not reporting a problem here; I am just _curious_ as
> to
> why these archive are so much bigger than they (apparently) need to be ...

I found out why this happens, I just don't know why it only occurs since
2015-07-20.

The reason is the script is using an expression along the lines of

  find ... | tar -T - --no-recursion -cjf ...

It turns out that the --no-recursion option only works for me, if it
comes *prior* to the expression specifying the filenames to archive.
That is, I had to change the script to use

  find ... | tar --no-recursion -T - -cjf ...

instead.  Funny enough, `info tar' still contains an example using
the original order...


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: snapshots (archive files) are too big ... Why?

[Houder]

On 2016-01-23 19:10, Corinna Vinschen wrote:
> On Jan 23 13:59, Houder wrote:
>> Hi Corinna,
>> 
>> Just curious if there is a reason ...
>> 
>> I was able to extract cygpath.exe from cygwin-inst-20160121.tar.xz
>> (snapshots)
>> in order to test your last modification to cygpath.cc. No problem 
>> here.
>> 
>> However, when examining the contents of the archive, I was surprised 
>> to find
>> the SAME version of cygpath.exe at least three times ...
>> 
>> The size of cygwin-inst-<date>.tar.xz (and cygwin-src-<date>.tar.xz) 
>> has
>> grown
>> (suddenly) by a factor of 3 or 4 since 2015-07-20 ...
>> 
>> The same applies to winsup-src-<date>.tar.xz (since 2016-01-15) ...
>> 
>> In all cases it is because the archive contains the SAME version of a 
>> file
>> at
>> least three times (as far as I can tell).
>> 
>> To summarize: No, I am not reporting a problem here; I am just 
>> _curious_ as
>> to
>> why these archive are so much bigger than they (apparently) need to be 
>> ...
> 
> I found out why this happens, I just don't know why it only occurs 
> since
> 2015-07-20.
> 
> The reason is the script is using an expression along the lines of
> 
>   find ... | tar -T - --no-recursion -cjf ...
> 
> It turns out that the --no-recursion option only works for me, if it
> comes *prior* to the expression specifying the filenames to archive.
> That is, I had to change the script to use
> 
>   find ... | tar --no-recursion -T - -cjf ...
> 
> instead.  Funny enough, `info tar' still contains an example using
> the original order...

Ah, thank you for the effort you took and for your explanation. (yes,
the reason for the "big" files was simpler than I was guessing at).

However, I cannot confirm your finding at my end (using Cygwin). Still,
I am sure you will take another look at the size of a snapshot when you
create one the next time :-)

Thanks!

Regards,
Henri

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: snapshots (archive files) are too big ... Why?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2368 bytes --]

On Jan 23 20:09, Houder wrote:
> On 2016-01-23 19:10, Corinna Vinschen wrote:
> >On Jan 23 13:59, Houder wrote:
> >>Hi Corinna,
> >>
> >>Just curious if there is a reason ...
> >>
> >>I was able to extract cygpath.exe from cygwin-inst-20160121.tar.xz
> >>(snapshots)
> >>in order to test your last modification to cygpath.cc. No problem here.
> >>
> >>However, when examining the contents of the archive, I was surprised to
> >>find
> >>the SAME version of cygpath.exe at least three times ...
> >>
> >>The size of cygwin-inst-<date>.tar.xz (and cygwin-src-<date>.tar.xz) has
> >>grown
> >>(suddenly) by a factor of 3 or 4 since 2015-07-20 ...
> >>
> >>The same applies to winsup-src-<date>.tar.xz (since 2016-01-15) ...
> >>
> >>In all cases it is because the archive contains the SAME version of a
> >>file
> >>at
> >>least three times (as far as I can tell).
> >>
> >>To summarize: No, I am not reporting a problem here; I am just _curious_
> >>as
> >>to
> >>why these archive are so much bigger than they (apparently) need to be
> >>...
> >
> >I found out why this happens, I just don't know why it only occurs since
> >2015-07-20.
> >
> >The reason is the script is using an expression along the lines of
> >
> >  find ... | tar -T - --no-recursion -cjf ...
> >
> >It turns out that the --no-recursion option only works for me, if it
> >comes *prior* to the expression specifying the filenames to archive.
> >That is, I had to change the script to use
> >
> >  find ... | tar --no-recursion -T - -cjf ...
> >
> >instead.  Funny enough, `info tar' still contains an example using
> >the original order...
> 
> Ah, thank you for the effort you took and for your explanation. (yes,
> the reason for the "big" files was simpler than I was guessing at).
> 
> However, I cannot confirm your finding at my end (using Cygwin). Still,
> I am sure you will take another look at the size of a snapshot when you
> create one the next time :-)

Of course I created local test snapshots using the above change.  They
only have one version of each file and are considerably smaller than the
previous versions.  I'm building on Fedora Linux if that matters.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: snapshots (archive files) are too big ... Why?

[Houder]

On 2016-01-23 20:25, Corinna Vinschen wrote:
> On Jan 23 20:09, Houder wrote:
>> On 2016-01-23 19:10, Corinna Vinschen wrote:

>> >I found out why this happens, I just don't know why it only occurs since
>> >2015-07-20.
>> >
>> >The reason is the script is using an expression along the lines of
>> >
>> >  find ... | tar -T - --no-recursion -cjf ...
>> >
>> >It turns out that the --no-recursion option only works for me, if it
>> >comes *prior* to the expression specifying the filenames to archive.
>> >That is, I had to change the script to use
>> >
>> >  find ... | tar --no-recursion -T - -cjf ...
>> >
>> >instead.  Funny enough, `info tar' still contains an example using
>> >the original order...
>> 
>> Ah, thank you for the effort you took and for your explanation. (yes,
>> the reason for the "big" files was simpler than I was guessing at).
>> 
>> However, I cannot confirm your finding at my end (using Cygwin). 
>> Still,
>> I am sure you will take another look at the size of a snapshot when 
>> you
>> create one the next time :-)
> 
> Of course I created local test snapshots using the above change.  They
> only have one version of each file and are considerably smaller than 
> the
> previous versions.  I'm building on Fedora Linux if that matters.

@@ mkdir x
@@ cd x
@@ tar xJf ../cygwin-inst-20160121.tar.xz
@@ find etc usr | wc -l
381

# preferred order (options)? tar manual: 6.9 Descending into 
Directories)
@@ find etc usr | tar --no-recursion -T- -cJf ../foo.tar.xz

# NOT the preferred order (options)?
@@ find etc usr | tar -T- --no-recursion -cJf ../foo2.tar.xz
@@ tar tvJf ../foo.tar.xz > foo1
@@ tar tvJf ../foo2.tar.xz > foo2
@@ diff foo1 foo2
@@ wc -l foo*
   381 foo1
   381 foo2
   762 total
@@

Did the same test on FC19. Same result.

But, let it rest for the moment, I would advice. Next time when you will
build a snapshot, you (we) will know.

Regards,
Henri

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: snapshots (archive files) are too big ... Why?

[Achim Gratz]

Houder writes:
> Did the same test on FC19. Same result.

The patch most likely responsible is only applied to FC21/22/23 best I
can tell.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: snapshots (archive files) are too big ... Why?

[Houder]

On 2016-01-23 21:18, Achim Gratz wrote:
> Houder writes:
>> Did the same test on FC19. Same result.
> 
> The patch most likely responsible is only applied to FC21/22/23 best I
> can tell.

As I realized, that Corinna is using "the latest and the greatest" of 
the
Fedora distribution, I go myself

      tar-1.27.1-8.fc21.i686.rpm

and extracted the tar program from it.

Again using FC19, but now the tar program that I extracted:

@@ pwd
/home/henri/x
@@ ls -l
total 7172
-rwxr-xr-x 1 henri henri 6952352 Jan 23 20:28 
cygwin-inst-20160121.tar.xz
drwxr-xr-x 3 henri henri    4096 Jan 21 18:42 etc
-rwxr-xr-x 1 henri henri  377916 Jan 23 22:38 tar
drwxr-xr-x 7 henri henri    4096 Jan 21 18:42 usr
@@ ./tar --version
tar (GNU tar) 1.27.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
@@ find etc usr | ./tar -T - --no-recursion -cJf foo.tar.xz
@@ find etc usr | ./tar --no-recursion -T - -cJf foo2.tar.xz
@@ ./tar tvJf foo.tar.xz > foo1
@@ ./tar tvJf foo2.tar.xz > foo2
@@ wc -l foo1 foo2
   1429 foo1 <==== Oh, ah!
    381 foo2
   1810 total
@@

As noted by Achim, this modification of tar took place in July 2015.

That tells it all, does it not, Corinna?

Thanks, Achim.

Regards,
Henri

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: snapshots (archive files) are too big ... Why?

[Achim Gratz]

Corinna Vinschen writes:
> I found out why this happens, I just don't know why it only occurs since
> 2015-07-20.

Looks like this patch

http://pkgs.fedoraproject.org/cgit/rpms/tar.git/commit/?id=132f2c79628ff5f11bb7d41c5c34f94228d63c54

and the following two fixups would be a likely source of that change in
behaviour and it was probably the following package update that threw
the script off course.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: snapshots (archive files) are too big ... Why?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

On Jan 23 21:13, Achim Gratz wrote:
> Corinna Vinschen writes:
> > I found out why this happens, I just don't know why it only occurs since
> > 2015-07-20.
> 
> Looks like this patch
> 
> http://pkgs.fedoraproject.org/cgit/rpms/tar.git/commit/?id=132f2c79628ff5f11bb7d41c5c34f94228d63c54
> 
> and the following two fixups would be a likely source of that change in
> behaviour and it was probably the following package update that threw
> the script off course.

Looks like you nailed it.  It seems these patches have gone upstream in
the first place, there just hasn't been a new tar version released yet.

Eric, any chance you could include these patches in a new tar release as
well to catch up with Fedora (or nudge the tar maintainers to release
1.29)?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: snapshots (archive files) are too big ... Why?

[Achim Gratz]

Corinna Vinschen writes:
> Eric, any chance you could include these patches in a new tar release as
> well to catch up with Fedora (or nudge the tar maintainers to release
> 1.29)?

That release should certainly have better documentation of the change
since while I can see where it's coming from I think it has potential
for a lot more scripts failing to do what they did before.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple