public inbox for bzip2-devel@sourceware.org
 help / color / mirror / Atom feed
* Some cherry-picks (Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.)
  2019-01-01  0:00   ` Mark Wielaard
@ 2019-01-01  0:00     ` Mark Wielaard
  2019-01-01  0:00       ` Federico Mena Quintero
  2019-01-01  0:00     ` Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q Mark Wielaard
  1 sibling, 1 reply; 14+ messages in thread
From: Mark Wielaard @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok,
	Federico Mena Quintero

On Mon, 2019-06-24 at 11:34 +0200, Mark Wielaard wrote:
> On Mon, 2019-06-24 at 10:31 +0200, Santiago Ruano Rincón wrote:
> > For information, Federico Mena (in CC) is taking over the development
> > of
> > bzip2: https://people.gnome.org/~federico/blog/maintaining-bzip2.html
> > 
> > You should consider coordinating your efforts!
> 
> Thanks. I assumed Federico was already on the bzip2-devel mailinglist.
> We did email earlier to discuss bzip2 maintenance and that all the
> infrastructure was already setup on the new sourceware.org bzip2
> project. But it seems we did some duplicate work. Sorry for the
> miscommunication.
> 
> It looks like we picked at least similar patches for the C sources, so
> those look mostly identical. I'll go over the remaining differences and
> try to cherry-pick or merge them into the bzip2 git repo.

I cherry-picked the following:

commit ff986850159a1ea0c75617ffa792d1bb2069856e
Author: Federico Mena Quintero <federico@gnome.org>
Date:   Wed May 29 17:14:27 2019 -0500

    Change a magic number (6) for a constant (BZ_N_GROUPS).
    
    decompress.c (BZ2_decompress): Check nGroups against BZ_N_GROUPS.

commit 7ed62bfb46e87a9e878712603469440e6882b184
Author: Albert Astals Cid <aacid@kde.org>
Date:   Tue May 28 19:35:18 2019 +0200

    Make sure nSelectors is not out of range
    
    nSelectors is used in a loop from 0 to nSelectors to access selectorMtf
    which is
        UChar    selectorMtf[BZ_MAX_SELECTORS];
    so if nSelectors is bigger than BZ_MAX_SELECTORS it'll do an invalid memory
    access
    
    Fixes out of bounds access discovered while fuzzying karchive
    
    This was reported as CVE-2019-12900
    BZ2_decompress in decompress.c in bzip2 through 1.0.6 has an
    out-of-bounds write when there are many selectors.

commit 16f2c753f9959e8d7c7e1fa771b8ccc5821427aa
Author: Paul Kehrer <paul.l.kehrer@gmail.com>
Date:   Sat Jun 8 10:06:40 2019 -0400

    Fix undefined behavior in the macros SET_BH, CLEAR_BH, & ISSET_BH
    
    These macros contain this pattern:
    1 << ((Int32_value) & 31
    
    This causes the undefined behavior sanitizers in clang and gcc to
    complain because the shift, while ultimately stored to an unsigned
    variable, is done as a signed value. Adding a cast to unsigned for
    the int32 value resolves this issue.

That makes the sources almost identical, modulo some whitespace issues
(inconsistent use of tab/space as indent). And some Windows specific
tweaks that I am not able to test (but they are probably correct though
).

The only remaining difference between the trees (for the C sources) is
the fix for O_CLOEXEC. I would like to better understand the
(different) Debian solution for that:

https://sources.debian.org/patches/bzip2/1.0.6-9/bzip2recover-race-open-output.diff/

Cheers,

Mark

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00               ` Mark Wielaard
@ 2019-01-01  0:00                 ` Mark Wielaard
  2019-01-01  0:00                   ` Federico Mena Quintero
  0 siblings, 1 reply; 14+ messages in thread
From: Mark Wielaard @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Federico Mena Quintero, Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok

On Wed, 2019-06-26 at 12:55 +0200, Mark Wielaard wrote:
> On Tue, 2019-06-25 at 21:22 -0500, Federico Mena Quintero wrote:
> > What do you think of me doing this:
> > 
> > - Create a "bzip2-1.0" branch with the commits and contents of the
> > sourceware repo.
> > 
> > - Sync the sources?  Are there any changes in the gitlab master
> > branch
> > that you *don't* want?
> > 
> > - Release 1.0.7 out of the bzip2-1.0 branch; deprecate that branch
> > immediately.  Advertise it as targeting people who embed bzip2-1.0.6
> > in
> > their sources and probably already have customizations to the build
> > system themselves.
> > 
> > - Continue with the current development in the master branch; I'll
> > probably release 1.1.0 as an experimental release with the new build
> > stuff and all the fixes.  Advertise this as targeted to distros and
> > everything else.
> > 
> > - Rust goes in a separate branch as usual; it's going to need some
> > big
> > changes to the new build system anyway.
> 
> I would like to do the release from the sourceware.org git repo first
> as is. And only then create a 1.0.x branch from it. That way we have
> something that simply replaces the bzip.org releases and establishes 
> https://sourceware.org/bzip2/ as new (or technically old) home for
> bzip2, have an official download location, updated regenerated manuals,
> etc. Then we can tweak the homepage through bzip2-htdocs.git to point
> to a future development tree if you don't like to use the sourceware
> bzip2.git repo as main development platform.

OK, lets go with this plan. There is this outstanding CVE and people
have been asking for a release to resolve it. 

I'll run the prepare-release.sh and release-update.sh scripts creating
the 1.0.7 release and will then update bzip-htdocs.git to add a pointer
to the homepage for the current development future 1.1.x development
branch https://gitlab.com/federicomenaquintero/bzip2 for people who
want a new build system and more fixes.

I'll sent mail to the list when done.

Cheers,

Mark

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00 [PATCH] bzip2: Fix return value when combining --test,-t and -q Mark Wielaard
@ 2019-01-01  0:00 ` Santiago Ruano Rincón
  2019-01-01  0:00   ` Mark Wielaard
  0 siblings, 1 reply; 14+ messages in thread
From: Santiago Ruano Rincón @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Mark Wielaard
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok,
	Federico Mena Quintero

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

Hi,

El 24/06/19 a las 09:40, Mark Wielaard escribió:
> Hi,
> 
> bzip2 lost its domain and got a new home at https://sourceware.org/bzip2/
> It also didn't see a release for a very long time. Causing various patches
> used by distros to not have been integrated upstream. We are trying to
> collect them all and do a new release.

Thanks for this.

>
> The following patch comes from Debian.
> Please let us know if we missed some others.

For information, Federico Mena (in CC) is taking over the development of
bzip2: https://people.gnome.org/~federico/blog/maintaining-bzip2.html

You should consider coordinating your efforts!

Cheers,

 -- Santiago

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00                 ` Mark Wielaard
@ 2019-01-01  0:00                   ` Federico Mena Quintero
  2019-01-01  0:00                     ` Mark Wielaard
  0 siblings, 1 reply; 14+ messages in thread
From: Federico Mena Quintero @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Mark Wielaard, Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok

On Thu, 2019-06-27 at 20:00 +0200, Mark Wielaard wrote:
> 
> I'll run the prepare-release.sh and release-update.sh scripts
> creating
> the 1.0.7 release and will then update bzip-htdocs.git to add a
> pointer
> to the homepage for the current development future 1.1.x development
> branch https://gitlab.com/federicomenaquintero/bzip2 for people who
> want a new build system and more fixes.
> 

Thank you!  Once you are done, I'll push a branch to the gitlab repo
with those changes.  In the meantime I'll blog about your plans to keep
the 1.0.7 code there, and about 1.1.x being the version with the new
build systems.

  Federico

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00             ` Federico Mena Quintero
@ 2019-01-01  0:00               ` Mark Wielaard
  2019-01-01  0:00                 ` Mark Wielaard
  0 siblings, 1 reply; 14+ messages in thread
From: Mark Wielaard @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Federico Mena Quintero, Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok

On Tue, 2019-06-25 at 21:22 -0500, Federico Mena Quintero wrote:
> What do you think of me doing this:
> 
> - Create a "bzip2-1.0" branch with the commits and contents of the
> sourceware repo.
> 
> - Sync the sources?  Are there any changes in the gitlab master
> branch
> that you *don't* want?
> 
> - Release 1.0.7 out of the bzip2-1.0 branch; deprecate that branch
> immediately.  Advertise it as targeting people who embed bzip2-1.0.6
> in
> their sources and probably already have customizations to the build
> system themselves.
> 
> - Continue with the current development in the master branch; I'll
> probably release 1.1.0 as an experimental release with the new build
> stuff and all the fixes.  Advertise this as targeted to distros and
> everything else.
> 
> - Rust goes in a separate branch as usual; it's going to need some
> big
> changes to the new build system anyway.

I would like to do the release from the sourceware.org git repo first
as is. And only then create a 1.0.x branch from it. That way we have
something that simply replaces the bzip.org releases and establishes 
https://sourceware.org/bzip2/ as new (or technically old) home for
bzip2, have an official download location, updated regenerated manuals,
etc. Then we can tweak the homepage through bzip2-htdocs.git to point
to a future development tree if you don't like to use the sourceware
bzip2.git repo as main development platform.

But I like to keep a (mirror) of the 1.0.x branch on sourceware so that
it is easy to keep making (emergency) releases from it if necessary. I
don't expect there to be many of those though. Given that the last 10
years provided 5 meaningful bug/security issues, there might be a bzip2
1.0.x release maybe once every 2 years.

I don't think we want any of the other changes from the gitlab tree
because they are all potential build issues, not purely bug fixes. e.g.
even the simple #if BZ_UNIX scares me a little.

I am sorry we didn't properly discuss and communicated what we expected
from resurrecting/maintaining bzip2. Especially since we seem to be
somewhat opposite developer/personalities. I am probably a conservative
minimalist, while you seem more of a maximalist modernist. But I hope
we can make cooperation work, because we probably need both types of
people/developers to maintain bzip2 for the next 20 years :)

For me the most important thing is that we provide a more stable home
for bzip2 than bzip.org was. And sourceware is that to me because it
existed even before bzip2, bzip2 had its home originally on
sourceware.org/bzip2, it is community managed and the site, git sources
and downloads/releases are widely mirrored. And that we don't break
anything, not even accidentally, which is why my 1.0.7 release really
only contains the minimal amount of changes just to update the
homepage, release number and fix any known bug/security issues, but
absolutely nothing more.

For you it is probably more important to have a modern build system, a
place to hack on it that makes it easy to exchange patches,
automagically mixes them with "pull requests" and issue tracking,
making the barrier to entry for people as low as possible, make it easy
to clean up and refactor the code, etc. all as long as it doesn't break
ABI of course. So that the code/community lives again to provide some
innovation.

Those things bite a little. Ideally they wouldn't of course, so we get
both stability and progress. But I would have never guessed someone
would pick a personal repo on a commercial (partially proprietary) run
platform. While you probably cannot imagine why someone would think a
place that offers bugzilla as issue tracker and primarily uses
mailinglists to exchange patches would be a good home for the project.
The conservative in me already freaks out about a simple change like
you just pushed to change the format of the version string returned by
BZ2_bzlibVersion (). What if someone parses it and expects that comma!
screams a little voice inside my head! Even though of course any code
that does that is obviously broken, and if we cannot even change that,
then what innovation can we do?

Anyway, I hope that explains why I care so deeply about what is there
now in the sourceware bzip2.git repo, and why I think we should keep
releasing that branch as is through sourceware.org. Even if we then
turn it into a branch that will never really change again and we point
the homepage repo sources links, to somewhere else for all future
development (in which case I would happily setup a mirror to make sure
the sources, releases and updated documentation are also available on
sourceware).

I do think you take the SONAME change too lightly though. But I don't
have a good answer for it. So my conservative reaction is DO NOT CHANGE
IT! Even though I think OpenSuse, Fedora and some other distros were
right to "fix" it to just contain one version digit. But sad fact is
that they were also wrong, because upstream is always right (even if it
is wrong) when it comes to compatibility. Changing the SONAME is
breaking ABI. And a lot of other distros didn't change it, and so
cannot without breaking ABI for the foreseeable future (I noticed even
the freedesktop flatpak runtime contains bzip2.so.1.0).

Cheers,

Mark

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Some cherry-picks (Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.)
  2019-01-01  0:00     ` Some cherry-picks (Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.) Mark Wielaard
@ 2019-01-01  0:00       ` Federico Mena Quintero
  0 siblings, 0 replies; 14+ messages in thread
From: Federico Mena Quintero @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Mark Wielaard, Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok

On Mon, 2019-06-24 at 15:55 +0200, Mark Wielaard wrote:
> The only remaining difference between the trees (for the C sources)
> is the fix for O_CLOEXEC. I would like to better understand the
> (different) Debian solution for that:
> 
> https://sources.debian.org/patches/bzip2/1.0.6-9/bzip2recover-race-open-output.diff/

This fix is partially correct, and I've pushed it to the repository in
gitlab with one change:

- Use practically the same fopen_output_safely() that is used in
bzip2.c.  This has a change to *not* use IntNative, to avoid
cutting&pasting even more #ifdefs from bzip2.c.

To make the fix fully correct, it would actually print different errors
when the output file exists, versus when it cannot be opened due to an
I/O error.  But that can wait.

I think the Debian patch may be confusing because it maintains the
description "fix unsafe race condition in opening output files" from
the bug report for bzip2-0.9.5... back then it *was* for a minor race
condition in bzip2, but these days it's only to avoid overwriting files
in bzip2recover.

  Federico

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00                   ` Federico Mena Quintero
@ 2019-01-01  0:00                     ` Mark Wielaard
  0 siblings, 0 replies; 14+ messages in thread
From: Mark Wielaard @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Federico Mena Quintero, Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok

On Thu, 2019-06-27 at 13:29 -0500, Federico Mena Quintero wrote:
> On Thu, 2019-06-27 at 20:00 +0200, Mark Wielaard wrote:
> > 
> > I'll run the prepare-release.sh and release-update.sh scripts
> > creating
> > the 1.0.7 release and will then update bzip-htdocs.git to add a
> > pointer
> > to the homepage for the current development future 1.1.x development
> > branch https://gitlab.com/federicomenaquintero/bzip2 for people who
> > want a new build system and more fixes.
> > 
> 
> Thank you!  Once you are done, I'll push a branch to the gitlab repo
> with those changes.  In the meantime I'll blog about your plans to keep
> the 1.0.7 code there, and about 1.1.x being the version with the new
> build systems.

Thanks. I also posted a blog with just the release announcement
and a link to your blog:
https://gnu.wildebeest.org/blog/mjw/2019/06/27/bzip2-1-0-7/

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00   ` Mark Wielaard
  2019-01-01  0:00     ` Some cherry-picks (Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.) Mark Wielaard
@ 2019-01-01  0:00     ` Mark Wielaard
  2019-01-01  0:00       ` Mark Wielaard
  1 sibling, 1 reply; 14+ messages in thread
From: Mark Wielaard @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok,
	Federico Mena Quintero

Hi,

On Mon, 2019-06-24 at 11:34 +0200, Mark Wielaard wrote:
> So I was thinking, first do a new 1.0.7 release with just the distro
> fixes and the updated project home, and then look at updating the
> rest of the build infrastructure.

I have been pondering what to do about the duplicate efforts to get a
bzip2 1.0.7 release out.

I am really conservative and really think it would be good to do a
release as is from the sourceware.org bzip2.git with just the
security/bug fixes. And I think it would not be good to force the
SO_NAME issue by updating the build system simultaneously. I am also
reluctant to include the "e" and/or O_CLOEXEC fix as is, because a) not
all distros have it currently (or different variants of it) and b) it
might not exist on all platforms, so would need some kind of check for
that first.

On the other hand, I do actually think it would be good to have a more
modern build system, and to force the SO_NAME issue (although I think
we should provide upstream support for changing it back to
libbz2.so.1.0 for those people who have been using it and don't want to
break backward compatibility). And using newer, more modern, possibly
GNU/Linux only features in 2019 would be a good thing.

So what about we do a conservative release of 1.0.7 from the current
sourceware.org bzip2.git and simultaneously do a 1.1.0 release from the
gitlab hosted git repo? Referencing each other as the "classic" and
"modern" bzip2 versions.

Then we can keep the sourceware.org bzip2.git around for ultra
conservative releases based on the current build "system" (that
basically only takes new security issues), using 1.0.8, 1.0.9, etc. But
encourage people towards the 1.1.1, 1.1.2, etc. releases for a more
modern experience, if at all possible.

Cheers,

Mark

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00         ` Federico Mena Quintero
@ 2019-01-01  0:00           ` Mark Wielaard
  2019-01-01  0:00             ` Federico Mena Quintero
  0 siblings, 1 reply; 14+ messages in thread
From: Mark Wielaard @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Federico Mena Quintero, Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok

Hi,

On Tue, 2019-06-25 at 14:08 -0500, Federico Mena Quintero wrote:
> On Tue, 2019-06-25 at 20:09 +0200, Mark Wielaard wrote:
> 
> > With that I think we are ready to do a bzip2 1.0.7 release simply
> > by
> > running
> > ./prepare-release.sh 1.0.7 and ./update-release.sh 1.0.7.
> 
> I like this script!  Thanks for writing it!

It is a bit of a hack to be honest. I mostly did it this way because
there is no modern build system yet :) I don't know how Julian did
releases. But there are way too many embedded version strings and dates
in the sources (and they don't even all use the same format!)

> > So if the above sounds like a good plan, then we can execute it
> > now.
> > Push out bzip2 1.0.7 and put the 1.0.x branch in hibernation, just
> > for
> > security or major bug fixes. Then we can concentrate on making sure
> > that a bzip2 1.1.X (or maybe even call it 2.0?) has a modern build
> > system and adds things like O_CLOEXEC support, etc.
> 
> I'm sorry for not replying to your mails earlier.  At work they just
> moved us to another email provider, and I've had some trouble fixing
> my
> email setup (SMTP servers, etc.) to make everything work as before.
> 
> I am... also ready to release 1.0.7 :-P  The work you've done in
> updating the release scripts is valuable, and I'd like to propose
> this
> instead:
> 
> * Put prepare-release.sh in the gitlab repository, and adjust it for
> the changes there.

If we are moving things around and have a modern build system it might
be better to actually remove most of the embedded versions/dates from
the sources and where they are really needed use configure to set them.
There actually already seems to be a merge request for that:
https://gitlab.com/federicomenaquintero/bzip2/merge_requests
(Although for autotools, which already has been removed again?)

> * Release 1.0.7 from the gitlab repo (it already has the NEWS file
> with
> updates, while I think your repo needs a CHANGES to be
> written?).  I've
> already tested that this works; "ninja dist" takes care of it.

Yes, the CHANGES file is the only thing the prepare-release.sh script
currently makes you do by hand (because there are no older tags yet, it
could in theory be replaced with a simple git shortlog), but there are
only 5 meaningful changes:

* Fix undefined behavior in the macros SET_BH, CLEAR_BH, & ISSET_BH

* bzip2: Fix return value when combining --test,-t and -q.

* bzip2recover: Fix buffer overflow for large argv[0] 

* bzip2recover: Fix use after free issue with outFile (CVE-2016-3189)

* Make sure nSelectors is not out of range (CVE-2019-12900)

> * Keep the soname changes from the gitlab repo.  I think the original
> Makefile-libbz2_so is incorrect in how it sets the soname (or at
> least
> libtool thinks so), and both openSUSE and Fedora managed to change it
> to a single digit instead of "1.0".  I *think* Debian, and distros
> which picked up their patches from there, simply never changed this
> and
> that's why they kept the "1.0" in place.

libtool doesn't support it, but libtool has to work on lots of unix-
like systems. The SONAME isn't technically wrong. It is just not what
we consider good practice these days. Unfortunately lots of distros
rely on this particular SONAME (at least everything based on Debian,
including Ubuntu and PureOS) use libbz2.so.1.0 and they simply cannot
change it, or they break backward compatibility. So I think any new
build system should at least provide the option of keeping the old
SONAME.

> Whatever we change the soname to, it's going to be incompatible with
> some distros - and they are already used to patching things, anyway. 
> Maybe purely for my own convenience I'd like to remove the patches
> from
> openSUSE - the current soname in the gitlab repo allows for that.

Yes, and I think lots of other distros (Fedora and CentOS based ones at
least) would also like that.

> I would be fine with releasing 1.0.7 from the sourceware repo, and
> then
> a subsequent 1.1 with all the changes from gitlab - but I think this
> will cause extra work for the distros.  First they'll have to see
> which
> of their patches need to be removed, and they'll still need to patch
> the build system.  And with 1.1 released, they'll need to do it all
> over again.
>
> I think we can minimize the work for distros if we update the patches
> and the build system in a single release.  They already know how to
> handle Meson more or less automatically, while leaving the original
> bzip2 Makefiles in place means that each distro has to patch to their
> own hand-rolled build system all over again.

I am not too concerned about the (larger) distros. Although I think it
would be good to at least provide a simple option to change the SONAME.
The "classic" build does that by simply editing/patching it, what the
distros probably already do. But the distros will probably just yell
and scream at us, and figure it out. I am more concerned about users we
don't really know about, maybe on non-GNU/Linux systems that have just
been using 1.0.6 for years as is. Putting out something with a new
build system, that by default generates an incompatible SONAME and some
changes that maybe need newer features not easily detected on those
systems might cause them to not upgrade at all.

I realize I am probably overly conservative, but I really like to keep
a 1.0.x branch going just for ancient users (I expect the distro to
switch to a more modern release, once they figure out the SONAME
thingy). And if we have that ancient branch, where we can just put
security fixes on, and nothing else, then we can be even more creative
on the new modern 1.1.x (2.0.x?) branch.

> [I've sent a few subscribe requests to this mailing list as
> federico@gnome.org, but it's not sending me the confirmation mail...
> do you know who may be able to kick the list software?]

That is odd. You could try contacting overseers@sourceware.org or go on
irc.freenode.net #overseers to see if someone knows what is going on.

Cheers,

Mark

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00           ` Mark Wielaard
@ 2019-01-01  0:00             ` Federico Mena Quintero
  2019-01-01  0:00               ` Mark Wielaard
  0 siblings, 1 reply; 14+ messages in thread
From: Federico Mena Quintero @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Mark Wielaard, Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok

On Wed, 2019-06-26 at 01:04 +0200, Mark Wielaard wrote:

> If we are moving things around and have a modern build system it
> might
> be better to actually remove most of the embedded versions/dates from
> the sources and where they are really needed use configure to set
> them.
> There actually already seems to be a merge request for that:

Yes, I want to update that MR tomorrow for the new build stuff.  Just
today I did it for BZ_VERSION in the header file; I think just the docs
are missing the changes from that MR now.

> I realize I am probably overly conservative, but I really like to
> keep a 1.0.x branch going just for ancient users (I expect the distro
> to
> switch to a more modern release, once they figure out the SONAME
> thingy). And if we have that ancient branch, where we can just put
> security fixes on, and nothing else, then we can be even more
> creative on the new modern 1.1.x (2.0.x?) branch.

I like this idea more and more.  And now I feel sheepish, because it's
basically the same approach I took for librsvg 2.40.20, the last C
version without Rust code in it.  This is from that version's release
notes:

Version 2.40.20
- Except for emergencies, this will be the LAST RELEASE of the
  librsvg-2.40.x series.  We are moving to 2.41, which is vastly
  improved over the 2.40 series.  The API/ABI there remain unchaged,
  so we strongly encourage you to upgrade your sources and binaries to
  librsvg-2.41.x.

I fetched from the sourceware repo onto my gitlab repo to see the
changes, and indeed the sources are practically identical (the build
stuff and docs are of course changed).

What do you think of me doing this:

- Create a "bzip2-1.0" branch with the commits and contents of the
sourceware repo.

- Sync the sources?  Are there any changes in the gitlab master branch
that you *don't* want?

- Release 1.0.7 out of the bzip2-1.0 branch; deprecate that branch
immediately.  Advertise it as targeting people who embed bzip2-1.0.6 in
their sources and probably already have customizations to the build
system themselves.

- Continue with the current development in the master branch; I'll
probably release 1.1.0 as an experimental release with the new build
stuff and all the fixes.  Advertise this as targeted to distros and
everything else.

- Rust goes in a separate branch as usual; it's going to need some big
changes to the new build system anyway.

  Federico

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00     ` Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q Mark Wielaard
@ 2019-01-01  0:00       ` Mark Wielaard
  2019-01-01  0:00         ` Federico Mena Quintero
  0 siblings, 1 reply; 14+ messages in thread
From: Mark Wielaard @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok,
	Federico Mena Quintero

Hi,

On Tue, 2019-06-25 at 02:05 +0200, Mark Wielaard wrote:
> So what about we do a conservative release of 1.0.7 from the current
> sourceware.org bzip2.git and simultaneously do a 1.1.0 release from the
> gitlab hosted git repo? Referencing each other as the "classic" and
> "modern" bzip2 versions.
> 
> Then we can keep the sourceware.org bzip2.git around for ultra
> conservative releases based on the current build "system" (that
> basically only takes new security issues), using 1.0.8, 1.0.9, etc. But
> encourage people towards the 1.1.1, 1.1.2, etc. releases for a more
> modern experience, if at all possible.

I just pushed the following commit:

commit f1e937776c5f331e24cc63a9d8e7ae9445a76761
Author: Mark Wielaard <mark@klomp.org>
Date:   Tue Jun 25 19:22:37 2019 +0200

    Add prepare-release.sh script.
    
    Script to run to prepare a new release.
    It will update the release number and tell you to update the
    CHANGES file and to double check everything looks before doing
    the release commit and tagging.
    
    Afterwards you probably want to run release-update.sh to upload
    the release and update the website at https://sourceware.org/bzip2/
    
    There are embedded version strings and dates in a couple of places.
    To keep the script simple remove some that aren't absolutely necessary.
    
    README now just points to CHANGES.
    README.COMPILATION.PROBLEMS only mentions the version once at the top.
    bzip2.c only mentions the version once when doing --version.
    manual.xml now doesn't have any embedded versions, just uses &bz-version;
    everywhere.

With that I think we are ready to do a bzip2 1.0.7 release simply by running
./prepare-release.sh 1.0.7 and ./update-release.sh 1.0.7.

So if the above sounds like a good plan, then we can execute it now.
Push out bzip2 1.0.7 and put the 1.0.x branch in hibernation, just for
security or major bug fixes. Then we can concentrate on making sure
that a bzip2 1.1.X (or maybe even call it 2.0?) has a modern build
system and adds things like O_CLOEXEC support, etc.

Cheers,

Mark

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH] bzip2: Fix return value when combining --test,-t and -q.
@ 2019-01-01  0:00 Mark Wielaard
  2019-01-01  0:00 ` Santiago Ruano Rincón
  0 siblings, 1 reply; 14+ messages in thread
From: Mark Wielaard @ 2019-01-01  0:00 UTC (permalink / raw)
  To: bzip2-devel
  Cc: Anibal Monsalve Salazar, Santiago Ruano Rincón, Anthony Fok,
	Mark Wielaard

Hi,

bzip2 lost its domain and got a new home at https://sourceware.org/bzip2/
It also didn't see a release for a very long time. Causing various patches
used by distros to not have been integrated upstream. We are trying to
collect them all and do a new release.

The following patch comes from Debian.
Please let us know if we missed some others.

Thanks,

Mark

When passing -q to get quiet output --test would not display an error
message, but would also suppress the exit 2 code to indicate the file
was corrupt. Only suppress the error message with -q, not the exit value.

This patch comes from Debian.
"bunzip2 -qt returns 0 for corrupt archives"
https://bugs.debian.org/279025
---
 bzip2.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/bzip2.c b/bzip2.c
index 854a2bb..63649f6 100644
--- a/bzip2.c
+++ b/bzip2.c
@@ -2003,12 +2003,14 @@ IntNative main ( IntNative argc, Char *argv[] )
             testf ( aa->name );
 	 }
       }
-      if (testFailsExist && noisy) {
-         fprintf ( stderr,
-           "\n"
-           "You can use the `bzip2recover' program to attempt to recover\n"
-           "data from undamaged sections of corrupted files.\n\n"
-         );
+      if (testFailsExist) {
+	 if (noisy) {
+            fprintf ( stderr,
+               "\n"
+               "You can use the `bzip2recover' program to attempt to recover\n"
+               "data from undamaged sections of corrupted files.\n\n"
+            );
+	 }
          setExit(2);
          exit(exitValue);
       }
-- 
1.8.3.1

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00 ` Santiago Ruano Rincón
@ 2019-01-01  0:00   ` Mark Wielaard
  2019-01-01  0:00     ` Some cherry-picks (Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.) Mark Wielaard
  2019-01-01  0:00     ` Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q Mark Wielaard
  0 siblings, 2 replies; 14+ messages in thread
From: Mark Wielaard @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok,
	Federico Mena Quintero

On Mon, 2019-06-24 at 10:31 +0200, Santiago Ruano Rincón wrote:
> For information, Federico Mena (in CC) is taking over the development
> of
> bzip2: https://people.gnome.org/~federico/blog/maintaining-bzip2.html
> 
> You should consider coordinating your efforts!

Thanks. I assumed Federico was already on the bzip2-devel mailinglist.
We did email earlier to discuss bzip2 maintenance and that all the
infrastructure was already setup on the new sourceware.org bzip2
project. But it seems we did some duplicate work. Sorry for the
miscommunication.

It looks like we picked at least similar patches for the C sources, so
those look mostly identical. I'll go over the remaining differences and
try to cherry-pick or merge them into the bzip2 git repo.

Some other files have been moved around, so it is harder to see whether
we did the same for the manual and other docs. And I explicitly didn't
touch the build system. I was looking at the autotools setup opensuse
uses, but that also uses libtool which would mean changing the bzip.so
verion/name. Which would be OK for Fedora and OpenSuse, since they
already changed it long ago, but it would not be good for Debian, which
uses the upstream libbz2.so.1.0 name. Also the buildbot, release
scripts and the automatic updating of the homepage/manual is somewhat
tied to the current setup.

So I was thinking, first do a new 1.0.7 release with just the distro
fixes and the updated project home, and then look at updating the rest
of the build infrastructure.

Cheers,

Mark

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.
  2019-01-01  0:00       ` Mark Wielaard
@ 2019-01-01  0:00         ` Federico Mena Quintero
  2019-01-01  0:00           ` Mark Wielaard
  0 siblings, 1 reply; 14+ messages in thread
From: Federico Mena Quintero @ 2019-01-01  0:00 UTC (permalink / raw)
  To: Mark Wielaard, Santiago Ruano Rincón
  Cc: bzip2-devel, Anibal Monsalve Salazar, Anthony Fok

On Tue, 2019-06-25 at 20:09 +0200, Mark Wielaard wrote:

> With that I think we are ready to do a bzip2 1.0.7 release simply by
> running
> ./prepare-release.sh 1.0.7 and ./update-release.sh 1.0.7.

I like this script!  Thanks for writing it!

> So if the above sounds like a good plan, then we can execute it now.
> Push out bzip2 1.0.7 and put the 1.0.x branch in hibernation, just
> for
> security or major bug fixes. Then we can concentrate on making sure
> that a bzip2 1.1.X (or maybe even call it 2.0?) has a modern build
> system and adds things like O_CLOEXEC support, etc.

I'm sorry for not replying to your mails earlier.  At work they just
moved us to another email provider, and I've had some trouble fixing my
email setup (SMTP servers, etc.) to make everything work as before.

I am... also ready to release 1.0.7 :-P  The work you've done in
updating the release scripts is valuable, and I'd like to propose this
instead:

* Put prepare-release.sh in the gitlab repository, and adjust it for
the changes there.

* Release 1.0.7 from the gitlab repo (it already has the NEWS file with
updates, while I think your repo needs a CHANGES to be written?).  I've
already tested that this works; "ninja dist" takes care of it.

* Keep the soname changes from the gitlab repo.  I think the original
Makefile-libbz2_so is incorrect in how it sets the soname (or at least
libtool thinks so), and both openSUSE and Fedora managed to change it
to a single digit instead of "1.0".  I *think* Debian, and distros
which picked up their patches from there, simply never changed this and
that's why they kept the "1.0" in place.

Whatever we change the soname to, it's going to be incompatible with
some distros - and they are already used to patching things, anyway. 
Maybe purely for my own convenience I'd like to remove the patches from
openSUSE - the current soname in the gitlab repo allows for that.

I would be fine with releasing 1.0.7 from the sourceware repo, and then
a subsequent 1.1 with all the changes from gitlab - but I think this
will cause extra work for the distros.  First they'll have to see which
of their patches need to be removed, and they'll still need to patch
the build system.  And with 1.1 released, they'll need to do it all
over again.

I think we can minimize the work for distros if we update the patches
and the build system in a single release.  They already know how to
handle Meson more or less automatically, while leaving the original
bzip2 Makefiles in place means that each distro has to patch to their
own hand-rolled build system all over again.

[I've sent a few subscribe requests to this mailing list as
federico@gnome.org, but it's not sending me the confirmation mail... do you know who may be able to kick the list software?]

  Federico

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2019-06-27 19:08 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-01  0:00 [PATCH] bzip2: Fix return value when combining --test,-t and -q Mark Wielaard
2019-01-01  0:00 ` Santiago Ruano Rincón
2019-01-01  0:00   ` Mark Wielaard
2019-01-01  0:00     ` Some cherry-picks (Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q.) Mark Wielaard
2019-01-01  0:00       ` Federico Mena Quintero
2019-01-01  0:00     ` Releasing bzip2 1.0.7 and 1.1.0 ? Was: [PATCH] bzip2: Fix return value when combining --test,-t and -q Mark Wielaard
2019-01-01  0:00       ` Mark Wielaard
2019-01-01  0:00         ` Federico Mena Quintero
2019-01-01  0:00           ` Mark Wielaard
2019-01-01  0:00             ` Federico Mena Quintero
2019-01-01  0:00               ` Mark Wielaard
2019-01-01  0:00                 ` Mark Wielaard
2019-01-01  0:00                   ` Federico Mena Quintero
2019-01-01  0:00                     ` Mark Wielaard

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).