Dedup x86/x86_64 --> noarch

Dedup x86/x86_64 --> noarch

[Achim Gratz]

After a discussion on IRC about de-duping the noarch content out of
package files (where I was told this would be too difficult), I've just
tried what would happen for two of my packages, maxima and perl.  Maxima
is practically a noarch package, save for the clisp memory image.  Perl
has gobs and gobs of non-arch-specific files mixed in with quite a bit
of arch-specific stuff.  I've used hashdeep for finding the dupes since
it is really fast, so the files are only de-duped if the are bit-for-bit
identical.

set p=perl
# reference
( cd $p.x86/inst    ; hashdeep -c sha256 -lr * ) > $p.x86.hash
# matching files
( cd $p.x86_64/inst ; hashdeep -c sha256 -k ../../$p.x86.hash -mlr * )
# non-matching files
( cd $p.x86_64/inst ; hashdeep -c sha256 -k ../../$p.x86.hash -xlr * )

For Maxima, there are a few files that should be identical, but aren't:
these are leakages from the build environment that I'll have to patch
out later (one of these leakages is actually a bug, affecting parts of
the documentation).

For Perl, the GZip compressed man-pages are flagged as different,
because gzip leaks the time-stamp (but that could be avoided using the
-n option to gzip in cygport).  Fixing that, the documentation packages
for Perl are completely shared between the two arches (well, duh), but
even the binary packages perl, perl-debginfo and perl_base would share
about a quarter of their content (so they'd need to be split into
something like perl_base / perl_base-common).

Looking at the current repo content we'd save about 30GB from the dedup
of the src abd doc packages alone and probably about 20GB from dedup in
the remaining packages.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Dedup x86/x86_64 --> noarch

[Achim Gratz]

Achim Gratz writes:
> Looking at the current repo content we'd save about 30GB from the dedup
> of the src abd doc packages alone and probably about 20GB from dedup in
> the remaining packages.

I've implemented some POC code and deduped my Cygwin mirror (it is
missing most of KDE and the cross-Cygwin compilation toolchains).  This
took a solid 12 hours of flat out 400% CPU load on my SandyBridge laptop
and ballooned the page file to 21GiB.  But it also removed almost
exactly a third from the repo's size (going from 81.2GiB to 51.4GiB), so
projected to the full repo it's slightly more than my original estimate.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Dedup x86/x86_64 --> noarch

[Jon Turney]

On 16/04/2016 11:03, Achim Gratz wrote:
> After a discussion on IRC about de-duping the noarch content out of
> package files (where I was told this would be too difficult), I've just

I think it was more along the lines of 'not yet' :)

In any case, we need noarch support in calm, before it's useful to have 
dedup of arch packages to noarch.

I think I have implemented the changes to calm to support all-or-nothing 
noarch (i.e. where all packages produced from a source package must be 
noarch), so if you can nominate a suitable, unimportant perl package, we 
can test it with that, initially.

(This wasn't quite as straightforward as just looking in another 
directory for packages, as the upload validation becomes more complex: 
we must check that consistent package sets result for both x86 and 
x86_64 before we can move noarch packages)

To make full use of this, cygport upload will need a feature to upload 
noarch packages from dist/ to noarch/ rather than <arch>/.

On 18/04/2016 20:44, Achim Gratz wrote:
>> Looking at the current repo content we'd save about 30GB from the dedup
>> of the src abd doc packages alone and probably about 20GB from dedup in
>> the remaining packages.
>
> I've implemented some POC code and deduped my Cygwin mirror (it is
> missing most of KDE and the cross-Cygwin compilation toolchains).  This
> took a solid 12 hours of flat out 400% CPU load on my SandyBridge laptop
> and ballooned the page file to 21GiB.  But it also removed almost
> exactly a third from the repo's size (going from 81.2GiB to 51.4GiB), so
> projected to the full repo it's slightly more than my original estimate.

Thanks.  It's very useful to have some numbers.

I don't think this distinguishes between packages which are (or should 
be) marked ARCH="noarch" in the cygport, and those where the build 
products happen to be identical and can be deduped?

I would guess that this saving is dominated by some very large, 
data-only noarch packages, but who knows?

(Also, looking forward, perhaps cygport needs a separate command to 
build the source package, rather than building it for each arch and then 
deduping it?)

Re: Dedup x86/x86_64 --> noarch

[Achim Gratz]

Jon Turney writes:
> I think I have implemented the changes to calm to support
> all-or-nothing noarch (i.e. where all packages produced from a source
> package must be noarch), so if you can nominate a suitable,
> unimportant perl package, we can test it with that, initially.

I'll have a look, there are quite a few packages that should have no
arch-dependent components by virtue of them being pure Perl.  In fact,
some of these were noarch while they were still in my private repo since
I kept the noarch alongside like Cygwin Ports has been doing.  You just
want to move them in the repo?

> (This wasn't quite as straightforward as just looking in another
> directory for packages, as the upload validation becomes more complex:
> we must check that consistent package sets result for both x86 and
> x86_64 before we can move noarch packages)

Sure.  For pure noarch this isn't much of a problem (or it never has
been with genini anyway).  Mixed packages might present a more
interesting challenge.  I think however, that if you simply treat
everything in noarch/<srcname>/ the same as <arch>/<srcname> and
coalesce these two directories logically, this problem should go away.

> To make full use of this, cygport upload will need a feature to upload
> noarch packages from dist/ to noarch/ rather than <arch>/.

I haven't looked into that, but I'm not using cygport for the upload
anyway.

> I don't think this distinguishes between packages which are (or should
> be) marked ARCH="noarch" in the cygport, and those where the build
> products happen to be identical and can be deduped?

No, I just took the archives and ran with it.  In fact most of the big
packages that I had memory problems with would clearly be "noarch" from
the outset (fonts and some of texlive).

> I would guess that this saving is dominated by some very large,
> data-only noarch packages, but who knows?

Thosa packages exists, but the commonalities in source and debuginfo
packages are summarily larger.  The scripting languages provide another
set of common files that yould be split off into a noarch package that
is then a dependency for the arch-dependent stuff.  But that's not
possible to set up right now I take it.

> (Also, looking forward, perhaps cygport needs a separate command to
> build the source package, rather than building it for each arch and
> then deduping it?)

I'm not sure what Yaakov thinks of that, but I really wanted to move the
whole build under a single directory and indeed dedup on the packaging
step (where I can just hardlink in the fs rather than needing to copy in
memory).  I haven't looked into that yet, because the changes necessary
might be quite pervasive.  But for the moment deduping the archives
doesn't look too bad either, so for folks building on different machines
and not sharing the build resources that would be another option
(vs. unpacking the build results from one arch into the build directory
of another).

Another thing that cygport needs is a way to mark sub-packages as noarch
so that the dedup can be skipped for files that are known to be noarch
already (I think their correct designation should still be checked
w.r.t. the actual build results).  Again, I've not yet looked at what
that will need in terms of implementation.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Dedup x86/x86_64 --> noarch

[Achim Gratz]

Jon Turney writes:
> I think I have implemented the changes to calm to support
> all-or-nothing noarch (i.e. where all packages produced from a source
> package must be noarch), so if you can nominate a suitable,
> unimportant perl package, we can test it with that, initially.

Not sure about "unimportant", but I'd suggest moving the developer
dependencies (perl-ExtUtils-*, perl-Test-*) first to see if there's any
problems.  The full list of Perl packages that dedup fully moved to
noarch (all source packages are also noarch):

perl-Algorithm-Diff
perl-Archive-Zip
perl-Authen-SASL
perl-Business-ISBN
perl-Business-ISBN-Data
perl-Business-ISMN
perl-Business-ISSN
perl-Canary-Stability
perl-Capture-Tiny
perl-Carp
perl-CGI
perl-Class-Accessor
perl-Class-Singleton
perl-Config-AutoConf
perl-Config-Perl-V
perl-Config-Tiny
perl-CPAN-Changes
perl-CPAN-DistnameInfo
perl-CPAN-Meta
perl-CPAN-Meta-Check
perl-CPAN-Meta-Requirements
perl-CPAN-Meta-YAML
perl-CPAN-Reporter
perl-CPAN-Testers-Report
perl-Data-Compare
perl-Data-Diver
perl-Data-Dump
perl-Data-GUID
perl-Data-OptList
perl-Data-Uniqid
perl-Date-Manip
perl-DateTime-Locale
perl-DateTime-TimeZone
perl-Devel-Autoflush
perl-Devel-CheckLib
perl-Devel-Symdump
perl-Digest-BubbleBabble
perl-Digest-HMAC
perl-Encode-Locale
perl-Error
perl-Exporter-Tiny
perl-ExtUtils-CBuilder
perl-ExtUtils-Depends
perl-ExtUtils-F77
perl-ExtUtils-LibBuilder
perl-ExtUtils-MakeMaker
perl-ExtUtils-PkgConfig
perl-File-Copy-Recursive
perl-File-Find-Rule
perl-File-HomeDir
perl-File-Listing
perl-File-pushd
perl-File-Slurp
perl-File-Slurp-Tiny
perl-File-Which
perl-Finance-Quote
perl-Font-TTF
perl-Getopt-ArgvFile
perl-GStreamer1
perl-Gtk3
perl-HTML-Element-Extended
perl-HTML-Format
perl-HTML-TableExtract
perl-HTML-Tagset
perl-HTML-Tree
perl-HTTP-Cookies
perl-HTTP-Daemon
perl-HTTP-Date
perl-HTTP-Message
perl-HTTP-Negotiate
perl-HTTP-Tiny
perl-inc-latest
perl-Inline
perl-Inline-Files
perl-IO-CaptureOutput
perl-IO-HTML
perl-IO-Prompt-Tiny
perl-IO-Socket-SSL
perl-IO-String
perl-IPC-Cmd
perl-IPC-Run
perl-IPC-Run3
perl-JSON
perl-JSON-MaybeXS
perl-JSON-PP
perl-libwww-perl
perl-libxml-perl
perl-List-AllUtils
perl-Log-Log4perl
perl-LWP-MediaTypes
perl-LWP-Protocol-https
perl-MailTools
perl-Metabase-Client-Simple
perl-Metabase-Fact
perl-MIME-Base32
perl-MIME-Charset
perl-MIME-tools
perl-Module-Implementation
perl-Module-Metadata
perl-Module-Pluggable
perl-Module-Runtime
perl-Module-ScanDeps
perl-Module-Signature
perl-Mojolicious
perl-Mozilla-CA
perl-Net-DNS
perl-Net-DNS-SEC
perl-Net-HTTP
perl-Net-IP
perl-Net-SMTP-SSL
perl-Number-Compare
perl-PAR
perl-Params-Util
perl-PAR-Dist
perl-Parse-CPAN-Meta
perl-Parse-RecDescent
perl-POD2-Base
perl-Pod-Coverage
perl-Pod-Escapes
perl-Pod-Simple
perl-Probe-Perl
perl-Readonly
perl-Regexp-Common
perl-SGMLSpm
perl-Spiffy
perl-Sub-Exporter
perl-Sub-Install
perl-Sub-Uplevel
perl-SUPER
perl-Tee
perl-Term-ReadLine-Perl
perl-Test-Base
perl-Test-CPAN-Meta
perl-Test-Deep
perl-Test-Exception
perl-Test-Fatal
perl-Test-Harness
perl-Test-MockModule
perl-Test-NoWarnings
perl-Test-Pod
perl-Test-Pod-Coverage
perl-Test-Reporter
perl-Test-Reporter-Transport-Metabase
perl-Test-Requires
perl-Test-Script
perl-Test-Simple
perl-Test-Warn
perl-Test-Warnings
perl-Test-Without-Module
perl-Test-YAML
perl-Text-CSV
perl-Text-Diff
perl-Text-Roman
perl-Text-Template
perl-Text-WrapI18N
perl-Tie-Cycle
perl-TimeDate
perl-Tk-Canvas-GradientColor
perl-Tk-ColoredButton
perl-Tk-EntryCheck
perl-Tk-Getopt
perl-Tk-Pod
perl-Try-Tiny
perl-Types-Serialiser
perl-URI
perl-WWW-RobotRules
perl-XML-DOM
perl-XML-LibXML-Simple
perl-XML-NamespaceSupport
perl-XML-RegExp
perl-XML-SAX
perl-XML-SAX-Base
perl-XML-SAX-Expat
perl-XML-Simple
perl-XML-Writer
perl-XML-XPath
perl-YAML


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Dedup x86/x86_64 --> noarch

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]

On Apr 23 16:19, Achim Gratz wrote:
> Jon Turney writes:
> > I think I have implemented the changes to calm to support
> > all-or-nothing noarch (i.e. where all packages produced from a source
> > package must be noarch), so if you can nominate a suitable,
> > unimportant perl package, we can test it with that, initially.
> 
> Not sure about "unimportant", but I'd suggest moving the developer
> dependencies (perl-ExtUtils-*, perl-Test-*) first to see if there's any
> problems.  The full list of Perl packages that dedup fully moved to
> noarch (all source packages are also noarch):

The src packages would ideally be in a src subdir, parallel to the
noarch and $arch dirs.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dedup x86/x86_64 --> noarch

[Achim Gratz]

Corinna Vinschen writes:
> The src packages would ideally be in a src subdir, parallel to the
> noarch and $arch dirs.

Hmm, I'm not soure I'd like that.

The src packages are not generally arch-less.  There are several
examples where either the cygport files, the patches or even the source
archives are arch-specific.  Of these, the .cygwin patches are almost
certainly spurious since they get time-stamped on creation.  The other
differences most likely aren't, although it should be possible to
reconcile them in most cases.

On another note, it still irks me that the debuginfo packages aren't
following the naming convention of the src packages; they should be
named ${PKG}-${PVR}-debuginfo.tar.xz and treated with their respective
box in setup.exe just like the sources.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Dedup x86/x86_64 --> noarch

[Jon Turney]

On 23/04/2016 16:43, Achim Gratz wrote:
> Corinna Vinschen writes:
>> The src packages would ideally be in a src subdir, parallel to the
>> noarch and $arch dirs.
>
> Hmm, I'm not sure I'd like that.
>
> The src packages are not generally arch-less.  There are several
> examples where either the cygport files, the patches or even the source
> archives are arch-specific.

I think 'generally' is over-stating the case, the vast majority of 
source packages should be arch-less.

If the source package really is arch-specific, then it should be marked 
so with ARCH [1]

If it contains arch-specific patches, they should be made conditional on 
ARCH.

[1] https://cygwinports.github.io/cygport/compilers_cygpart.html#robo872

 > Of these, the .cygwin patches are almost
> certainly spurious since they get time-stamped on creation.  The other
> differences most likely aren't, although it should be possible to
> reconcile them in most cases.

But yes, this is not straightforward because the way we generate source 
packages at the moment means there is no guarantee that the same source 
package is used to build the different arch variants.

> On another note, it still irks me that the debuginfo packages aren't
> following the naming convention of the src packages; they should be
> named ${PKG}-${PVR}-debuginfo.tar.xz and treated with their respective
> box in setup.exe just like the sources.
We have the source code :)

Re: Dedup x86/x86_64 --> noarch

[Achim Gratz]

Jon Turney writes:
> I think 'generally' is over-stating the case, the vast majority of
> source packages should be arch-less.

I said "not generally", which I think makes a slightly less sweeping
statement.  In any case, I just wanted to point out that some of the
existing source packages have differences between the two arches that
are not all trivial to remove (texlive, for whatever reason seems to
have arch specific source archives, for instance).

> If the source package really is arch-specific, then it should be
> marked so with ARCH [1]

This is currently making the whole build including all sub-packages
noarch.  There is currently no way to specify arch on a sub-package
granularity and the source package is implied by cygport anyway.

> If it contains arch-specific patches, they should be made conditional
> on ARCH.

Yes, that's the way to go in this case.  That wouldn't work in cases
where the sources themselves are already different, unless you suggest
that the source archive itself should be arch-dependent.  In that case
we'd have the interesting problem that cygport needs to recognize that
it should package the source for the other arch in the same source
package just to make the result non-arch-specific.

> But yes, this is not straightforward because the way we generate
> source packages at the moment means there is no guarantee that the
> same source package is used to build the different arch variants.

That is fixable as long as it can be done by cygport.  I'm more worried
about those packages that still use a different packaging system (I know
of Jari's, and maybe one or the other odd package is still built by hand)


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Dedup x86/x86_64 --> noarch

[Jon Turney]

On 23/04/2016 15:19, Achim Gratz wrote:
> Jon Turney writes:
>> I think I have implemented the changes to calm to support
>> all-or-nothing noarch (i.e. where all packages produced from a source
>> package must be noarch), so if you can nominate a suitable,
>> unimportant perl package, we can test it with that, initially.
>
> Not sure about "unimportant", but I'd suggest moving the developer
> dependencies (perl-ExtUtils-*, perl-Test-*) first to see if there's any
> problems.  The full list of Perl packages that dedup fully moved to
> noarch (all source packages are also noarch):

I've deployed an updated calm, and moved perl-Test-Base to noarch.

 From my brief testing, setup handles this layout with no problems. I 
also checked that noarch files in a download package directory shared 
between x86 and x86_64 setup works as expected.

Uploads to noarch should also now be accepted, although cygport needs 
teaching how to do that.

Thanks for the list.  I'll look at moving more stuff if no problems appear.

Re: Dedup x86/x86_64 --> noarch

[Achim Gratz]

Jon Turney writes:
> I've deployed an updated calm, and moved perl-Test-Base to noarch.

Thanks.

> From my brief testing, setup handles this layout with no problems. I
> also checked that noarch files in a download package directory shared
> between x86 and x86_64 setup works as expected.
>
> Uploads to noarch should also now be accepted, although cygport needs
> teaching how to do that.
>
> Thanks for the list.  I'll look at moving more stuff if no problems appear.

If you want to test something with a bit more exposure, move
_autorebase.  It is actually a "real" cygport noarch package that just
gets duplicated on upload currently.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Dedup x86/x86_64 --> noarch

[Andrew Schulman]

> After a discussion on IRC about de-duping the noarch content out of
> package files (where I was told this would be too difficult), I've just
> tried what would happen for two of my packages, maxima and perl.

I feel as though I've once again missed some important discussion about package
maintenance.  Your question implies that we can now upload packages with arch
"noarch".  Is that true?

I remember a discussion of this question a year or two ago, where IIRC the
result was that we wouldn't have any "noarch" packages.  Did that change?  If
so, I seem to have missed the discussion here.  It should be worth a "HEADSUP".

If this is what's happened, we need to update https://cygwin.com/setup.html with
the new information (the page is overdue for an overhaul anyway). In particular
I wonder if cygport is ready to handle the change.

Or, maybe I'm just misunderstanding.

Thanks,
Andrew

Re: Dedup x86/x86_64 --> noarch

[Achim Gratz]

Andrew Schulman writes:
> I feel as though I've once again missed some important discussion about package
> maintenance.  Your question implies that we can now upload packages with arch
> "noarch".  Is that true?

Meanwhile it has become true, although Jon and Yaakov are still in the
early stages of adding support for that.  But my posting really was
meant as a demonstration that there are substantial gains to be had from
going to this layout and that therefore it'd be worthwile adding
official support for it.

> I remember a discussion of this question a year or two ago, where IIRC the
> result was that we wouldn't have any "noarch" packages.  Did that change?  If
> so, I seem to have missed the discussion here.  It should be worth a "HEADSUP".

Yes, this needs to be announced soonish (Yaakov?) as otherwise some
folks will be a bit surprised to see their mirror scripts not working
anymore.

> If this is what's happened, we need to update https://cygwin.com/setup.html with
> the new information (the page is overdue for an overhaul anyway). In particular
> I wonder if cygport is ready to handle the change.

AFAIK, not quite yet.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Dedup x86/x86_64 --> noarch

[Andrew Schulman]

> Andrew Schulman writes:
> > I feel as though I've once again missed some important discussion about package
> > maintenance.  Your question implies that we can now upload packages with arch
> > "noarch".  Is that true?
> 
> Meanwhile it has become true, although Jon and Yaakov are still in the
> early stages of adding support for that.  But my posting really was
> meant as a demonstration that there are substantial gains to be had from
> going to this layout and that therefore it'd be worthwile adding
> official support for it.
> 
> > I remember a discussion of this question a year or two ago, where IIRC the
> > result was that we wouldn't have any "noarch" packages.  Did that change?  If
> > so, I seem to have missed the discussion here.  It should be worth a "HEADSUP".
> 
> Yes, this needs to be announced soonish (Yaakov?) as otherwise some
> folks will be a bit surprised to see their mirror scripts not working
> anymore.
> 
> > If this is what's happened, we need to update https://cygwin.com/setup.html with
> > the new information (the page is overdue for an overhaul anyway). In particular
> > I wonder if cygport is ready to handle the change.
> 
> AFAIK, not quite yet.

OK, thanks!
Andrew

Re: Dedup x86/x86_64 --> noarch

[Jon Turney]

On 10/05/2016 07:20, Andrew Schulman wrote:
>>> If this is what's happened, we need to update https://cygwin.com/setup.html with
>>> the new information (the page is overdue for an overhaul anyway).

I agree that page is very bad at communicating the information it needs 
to communicate.  Perhaps it would be much more helpful if it had a few 
examples of .cygport files rather than diving into the minutiae of 
setup.hint files.

However, I took a look over it, but I don't see what needs updating for 
this particular change.  Can you be more specific about what you think 
needs updating?

I have updated https://sourceware.org/cygwin-apps/package-upload.html

That didn't actually explain before what to do if you are manually 
uploading a noarch package ("if you are doing a manual upload, and your 
package is noarch, you'll need to upload to both x86 and x86_64"), so 
I'm not sure how to update it for the hopefully more straightforward 
present situation ("if you are doing a manual upload, and your package 
is noarch, upload to noarch").