[RFE] calm needs to know about ZStandard compressed archives

[RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

While waiting for the Perl bootstrap I've been experimenting with having
cygport produce .tar.zst packages.  Calm (mksetupini) does not recognize
these at the moment, but I've been able to monkey-patch it by adding
'|zst' in the obvious places.  Can this be done officially, please?  I
can't help noting that these really shouldn't be literal strings so that
you only need to change them in a single place if necessary; so perhaps
common_constants.py would be a good place for that definition.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [RFE] calm needs to know about ZStandard compressed archives

[Jon Turney]

On 25/12/2020 19:59, Achim Gratz wrote:
> 
> While waiting for the Perl bootstrap I've been experimenting with having
> cygport produce .tar.zst packages.  Calm (mksetupini) does not recognize
> these at the moment, but I've been able to monkey-patch it by adding
> '|zst' in the obvious places.  Can this be done officially, please?  I
> can't help noting that these really shouldn't be literal strings so that
> you only need to change them in a single place if necessary; so perhaps
> common_constants.py would be a good place for that definition.

Yeah, this list should be done in a less ad-hoc way.

This isn't totally straightforward as calm uses the python tar module to 
validate the upload archives, but that doesn't (yet) support zstd 
compression.

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Jon Turney writes:
> This isn't totally straightforward as calm uses the python tar module
> to validate the upload archives, but that doesn't (yet) support zstd 
> compression.

Maybe this would help until upstream gets their act together…

https://pypi.org/project/xtarfile/


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFE] calm needs to know about ZStandard compressed archives

[Marco Atzeri]

On 31.12.2020 16:47, Achim Gratz wrote:
> Jon Turney writes:
>> This isn't totally straightforward as calm uses the python tar module
>> to validate the upload archives, but that doesn't (yet) support zstd
>> compression.
> 
> Maybe this would help until upstream gets their act together…
> 
> https://pypi.org/project/xtarfile/

do you need it as python package ?

> 
> Regards,
> Achim.

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Marco Atzeri via Cygwin-apps writes:
> On 31.12.2020 16:47, Achim Gratz wrote:
>> Jon Turney writes:
>>> This isn't totally straightforward as calm uses the python tar module
>>> to validate the upload archives, but that doesn't (yet) support zstd
>>> compression.
>> Maybe this would help until upstream gets their act together…
>> https://pypi.org/project/xtarfile/
>
> do you need it as python package ?

The tricky part is if it's available on sourceware, I'd guess.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [RFE] calm needs to know about ZStandard compressed archives

[Marco Atzeri]

On 31.12.2020 16:52, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>> On 31.12.2020 16:47, Achim Gratz wrote:
>>> Jon Turney writes:
>>>> This isn't totally straightforward as calm uses the python tar module
>>>> to validate the upload archives, but that doesn't (yet) support zstd
>>>> compression.
>>> Maybe this would help until upstream gets their act together…
>>> https://pypi.org/project/xtarfile/
>>
>> do you need it as python package ?
> 
> The tricky part is if it's available on sourceware, I'd guess.

locate says no, for the time being

Re: [RFE] calm needs to know about ZStandard compressed archives

[Jon Turney]

On 31/12/2020 16:01, Marco Atzeri via Cygwin-apps wrote:
> On 31.12.2020 16:52, Achim Gratz wrote:
>> Marco Atzeri via Cygwin-apps writes:
>>> On 31.12.2020 16:47, Achim Gratz wrote:
>>>> Jon Turney writes:
>>>>> This isn't totally straightforward as calm uses the python tar module
>>>>> to validate the upload archives, but that doesn't (yet) support zstd
>>>>> compression.
>>>> Maybe this would help until upstream gets their act together…
>>>> https://pypi.org/project/xtarfile/
>>>
>>> do you need it as python package ?
>>
>> The tricky part is if it's available on sourceware, I'd guess.
> 
> locate says no, for the time being

Thanks, but since this is only done in one spot, I think I can just 
arrange to pipe ztd archives through 'unzstd' before validating.

Achim,

Can you publish your cygport changes, when you are happy with them, so I 
can make some packages to test with?

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Jon Turney writes:
> Can you publish your cygport changes, when you are happy with them, so
> I can make some packages to test with?

https://repo.or.cz/cygport/rpm-style.git/shortlog/refs/heads/to-upstream

Then you need to put this into .cygport.conf to actually produce
.tar.zst archives:

CYGPORT_TAR_EXT=".tar.xz"

The default compression level can be changed as follows:

CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner nobody:65534 -J"



Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: [RFE] calm needs to know about ZStandard compressed archives

[Brian Inglis]

On 2020-12-31 11:33, Achim Gratz wrote:
> Jon Turney writes:
>> Can you publish your cygport changes, when you are happy with them, so
>> I can make some packages to test with?
> 
> https://repo.or.cz/cygport/rpm-style.git/shortlog/refs/heads/to-upstream
> 
> Then you need to put this into .cygport.conf to actually produce
> .tar.zst archives:
> 
> CYGPORT_TAR_EXT=".tar.xz"
   CYGPORT_TAR_EXT=".tar.zst" ???

> The default compression level can be changed as follows:
> 
> CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner nobody:65534 -J"
   CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner 
nobody:65534 --zstd" ???

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Brian Inglis writes:
>   CYGPORT_TAR_EXT=".tar.zst" ???

Yes, typo.

>   CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner
>   nobody:65534 --zstd" ???

No.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [RFE] calm needs to know about ZStandard compressed archives

[Brian Inglis]

On 2020-12-31 12:18, Achim Gratz wrote:
> Brian Inglis writes:
>> CYGPORT_TAR_EXT=".tar.zst" ???
> 
> Yes, typo.

>> CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner
>> nobody:65534 --zstd" ???
> 
> No.

>>> CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner 
>>> nobody:65534 -J"
To which part does "No" apply: you specified -J which is --xz; you could instead 
specify -a, --auto-compress to use the archive suffix.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Brian Inglis writes:
> To which part does "No" apply: you specified -J which is --xz; you
> could instead specify -a, --auto-compress to use the archive suffix.

Here's the actual content of my test installation that I've used for the
Perl repository:

--8<---------------cut here---------------start------------->8---
CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner nobody:65534 -a"
CYGPORT_TAR_EXT=".tar.zst"
--8<---------------cut here---------------end--------------->8---


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [RFE] calm needs to know about ZStandard compressed archives

[Jon Turney]

On 31/12/2020 17:08, Jon Turney wrote:
> On 31/12/2020 16:01, Marco Atzeri via Cygwin-apps wrote:
>> On 31.12.2020 16:52, Achim Gratz wrote:
>>> Marco Atzeri via Cygwin-apps writes:
>>>> On 31.12.2020 16:47, Achim Gratz wrote:
>>>>> Jon Turney writes:
>>>>>> This isn't totally straightforward as calm uses the python tar module
>>>>>> to validate the upload archives, but that doesn't (yet) support zstd
>>>>>> compression.
>>>>> Maybe this would help until upstream gets their act together…
>>>>> https://pypi.org/project/xtarfile/
>>>>
>>>> do you need it as python package ?
>>>
>>> The tricky part is if it's available on sourceware, I'd guess.
>>
>> locate says no, for the time being
> 
> Thanks, but since this is only done in one spot, I think I can just 
> arrange to pipe ztd archives through 'unzstd' before validating.

Actually, on second look, we need to look inside the archive in a couple 
of places (to check if it's empty, and to generate the package file 
listings), so using xtarfile seems like the right approach.

I can arrange to install it on sourceware, but it would be nice if 
python-xtarfile was packaged so it can be made a prerequisite of the 
calm package.

Re: [RFE] calm needs to know about ZStandard compressed archives

[Marco Atzeri]

On 03.01.2021 16:19, Jon Turney wrote:
> On 31/12/2020 17:08, Jon Turney wrote:
>> On 31/12/2020 16:01, Marco Atzeri via Cygwin-apps wrote:
>>> On 31.12.2020 16:52, Achim Gratz wrote:
>>>> Marco Atzeri via Cygwin-apps writes:
>>>>> On 31.12.2020 16:47, Achim Gratz wrote:
>>>>>> Jon Turney writes:
>>>>>>> This isn't totally straightforward as calm uses the python tar 
>>>>>>> module
>>>>>>> to validate the upload archives, but that doesn't (yet) support zstd
>>>>>>> compression.

> 
> I can arrange to install it on sourceware, but it would be nice if 
> python-xtarfile was packaged so it can be made a prerequisite of the 
> calm package.

I will do it, but I will appreciate if package maintainers
can validate the python/pip test release I recently uploaded.
As I am rebuiling/adding python packages using latest test
and going backward is not so nice.

We will also need to modify cygport to stop automatically add
OBSOLETES as most of this packages never existed

eg python3-pytest

 >>> python36-pytest requires: python36 python36-attrs 
python36-iniconfig python36-pluggy python36-py python36-packaging 
python36-toml
 >>> python36-pytest OBSOLETES: python3-pytest
 >>> python37-pytest requires: python37 python37-attrs 
python37-iniconfig python37-pluggy python37-py python37-packaging 
python37-toml
 >>> python38-pytest requires: python38 python38-attrs 
python38-iniconfig python38-pluggy python38-py python38-packaging 
python38-toml

an overall cleaning of obsoletes package may be useful in the
near future. I need to work on it.

regards
Marco

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Marco Atzeri via Cygwin-apps writes:
> We will also need to modify cygport to stop automatically add
> OBSOLETES as most of this packages never existed

I think the right thing to do is to stop having actual packages for
OBSOLETES and just emit the obsoltes: hint instead.

> an overall cleaning of obsoletes package may be useful in the
> near future. I need to work on it.

The mailing list is full of folks that seem to update their
installations only after several years, so while I'm all for removing
the empty packages, the obsolete hints need to stay.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [RFE] calm needs to know about ZStandard compressed archives

[Marco Atzeri]

On 03.01.2021 16:57, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>> We will also need to modify cygport to stop automatically add
>> OBSOLETES as most of this packages never existed
> 
> I think the right thing to do is to stop having actual packages for
> OBSOLETES and just emit the obsoltes: hint instead.

>> an overall cleaning of obsoletes package may be useful in the
>> near future. I need to work on it.
> 
> The mailing list is full of folks that seem to update their
> installations only after several years, so while I'm all for removing
> the empty packages, the obsolete hints need to stay.

I was referring only to the "empty packages" artificially created
of course.

> Regards,
> Achim.
> 

Re: [RFE] calm needs to know about ZStandard compressed archives

[Marco Atzeri]

On 03.01.2021 16:19, Jon Turney wrote:

> 
> I can arrange to install it on sourceware, but it would be nice if 
> python-xtarfile was packaged so it can be made a prerequisite of the 
> calm package.

I just uploaded test versions of
python3X-xtarfile and python3X-zstandard
that seems are needed for that compression format..

let me know if find any issue.

Regards
Marco

Re: [RFE] calm needs to know about ZStandard compressed archives

[Jon Turney]

On 03/01/2021 15:57, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>> We will also need to modify cygport to stop automatically add
>> OBSOLETES as most of this packages never existed
> 
> I think the right thing to do is to stop having actual packages for
> OBSOLETES and just emit the obsoltes: hint instead.

Yeah, this change to cygport should be made and we can turn off support 
for setup versions which don't understand the obsoletes: hint.

>> an overall cleaning of obsoletes package may be useful in the
>> near future. I need to work on it.

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Jon Turney writes:
>> I think the right thing to do is to stop having actual packages for
>> OBSOLETES and just emit the obsoltes: hint instead.
>
> Yeah, this change to cygport should be made and we can turn off
> support for setup versions which don't understand the obsoletes: hint.

The obsolete hints are already generated for some time, so at least the
newer packages should have them.  The older packages would need to
retroactively get them (or keep their equally old obsoleteion packages
around still).

So I think this is what's left to do in cygport (untested, on top of my
to-upstream branch):

--8<---------------cut here---------------start------------->8---
From 0491ea2934d83c656ba5dacf7f02c31f02b46a38 Mon Sep 17 00:00:00 2001
Subject: [PATCH] lib/pkg_pkg.cygpart: stop generating packages for obsoletions

---
 lib/pkg_pkg.cygpart | 46 ---------------------------------------------
 1 file changed, 46 deletions(-)

diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
index 2b2f8bc..80003e9 100644
--- a/lib/pkg_pkg.cygpart
+++ b/lib/pkg_pkg.cygpart
@@ -828,32 +828,6 @@ _EOF
 			warning "${pkg_hint[${n}]%.hint}.hint is missing";
 		fi
 
-		for obspkg in ${!pkg_obsoletes_var}
-		do
-			if [ ${obspkg} = ${PN} ]
-			then
-				obssubdir= ;
-			else
-				obssubdir=${obspkg};
-			fi
-
-			mkdir -p ${distdir}/${PN}/${obssubdir};
-			${CYGPORT_TAR_CMD-tar -J} -cf ${distdir}/${PN}/${obssubdir}/${obspkg}-${PVR}${CYGPORT_TAR_EXT-.tar.xz} --files-from /dev/null
-
-			__step "${pkg_name[${n}]} OBSOLETES: ${obspkg}"
-
-			cat > ${distdir}/${PN}/${obssubdir}/${obspkg}-${PVR}.hint <<-_EOF
-category: _obsolete
-requires: ${pkg_name[${n}]}
-sdesc: "Obsoleted by ${pkg_name[${n}]}"
-ldesc: "The ${obspkg} package is obsolete.  Selecting this package for
-installation will cause the ${pkg_name[${n}]} package, which replaces this
-one, to be installed instead."
-${obssubdir:+external-source: ${PN}}
-${pkg_tag}
-_EOF
-		done
-
 		n+=1;
 	done
 
@@ -887,26 +861,6 @@ _EOF
 			fi
 		fi
 
-		for obspkg in ${!dbg_obsoletes_var}
-		do
-			mkdir -p ${distdir}/${PN}/${obspkg};
-			${CYGPORT_TAR_CMD-tar -J} -cf ${distdir}/${PN}/${obspkg}/${obspkg}-${PVR}${CYGPORT_TAR_EXT-.tar.xz} --files-from /dev/null
-
-			__step "${PN}-debuginfo OBSOLETES: ${obspkg}"
-
-			cat > ${distdir}/${PN}/${obspkg}/${obspkg}-${PVR}.hint <<-_EOF
-category: _obsolete
-requires: ${PN}-debuginfo
-sdesc: "Obsoleted by ${PN}-debuginfo"
-ldesc: "The ${obspkg} package is obsolete.  Selecting this package for
-installation will cause the ${PN}-debuginfo package, which replaces this
-one, to be installed instead."
-external-source: ${PN}
-${pkg_tag}
-_EOF
-		done
-	fi
-
 	# source package hint
 	if [ ! -f ${distdir}/${PN}/${PN}-${PVR}-src.hint ]
 	then
-- 
2.29.2
--8<---------------cut here---------------end--------------->8---



Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFE] calm needs to know about ZStandard compressed archives

[Jon Turney]

On 03/01/2021 21:55, Marco Atzeri via Cygwin-apps wrote:
> On 03.01.2021 16:19, Jon Turney wrote:
> 
>>
>> I can arrange to install it on sourceware, but it would be nice if 
>> python-xtarfile was packaged so it can be made a prerequisite of the 
>> calm package.
> 
> I just uploaded test versions of
> python3X-xtarfile and python3X-zstandard
> that seems are needed for that compression format..

Thanks.

I uploaded a calm-20210110-1 package, which adds support for .zst 
package archives.

This is also deployed on sourceware.

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Jon Turney writes:
> I uploaded a calm-20210110-1 package, which adds support for .zst
> package archives.
>
> This is also deployed on sourceware.

Awesome, thank you!


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Jon Turney writes:
> I uploaded a calm-20210110-1 package, which adds support for .zst
> package archives.

The package seems to be missing a dependency on python38-xtarfile,
though.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: [RFE] calm needs to know about ZStandard compressed archives

[Achim Gratz]

Jon Turney writes:
> This is also deployed on sourceware.

At least one of xtarfile or zstandard Pythono modules seems to be
missing, though.  I tried to upload binutils and got this:

ERROR: exception NotImplementedError while reading /sourceware/cygwin-staging/home/Achim Gratz/x86/release/binutils/binutils-2.35.2-1-src.tar.zst


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFE] calm needs to know about ZStandard compressed archives

[Jon Turney]

On 24/01/2021 09:35, Achim Gratz wrote:
> Jon Turney writes:
>> I uploaded a calm-20210110-1 package, which adds support for .zst
>> package archives.
> 
> The package seems to be missing a dependency on python38-xtarfile,
> though.

Thanks for reporting.  Should be fixed.

Re: [RFE] calm needs to know about ZStandard compressed archives

[Jon Turney]

On 31/01/2021 17:13, Achim Gratz wrote:
> Jon Turney writes:
>> This is also deployed on sourceware.
> 
> At least one of xtarfile or zstandard Pythono modules seems to be
> missing, though.  I tried to upload binutils and got this:
> 
> ERROR: exception NotImplementedError while reading /sourceware/cygwin-staging/home/Achim Gratz/x86/release/binutils/binutils-2.35.2-1-src.tar.zst
> 

Yeah, seems like I screwed up installing the python modules.  Should be 
fixed now.


Thanks for reporting.