* bzip2 next steps - goals for 1.0.9
@ 2019-01-01 0:00 Mark Wielaard
0 siblings, 0 replies; only message in thread
From: Mark Wielaard @ 2019-01-01 0:00 UTC (permalink / raw)
To: bzip2-devel; +Cc: Julian Seward
After the fire drills that were necessary to get the bzip2 1.0.7 "Help
a security issues!" and bzip2 1.0.8 "O, wait, maybe that was a little
too secure!" releases out I hope we will get a lot more time to do a
bzip2 1.0.9 release.
But having been forced to do two releases was also good. We got the old
website, including all old releases on sourceware.org. There is now a
git repository, also including the code for older releases. The whole
release process is now automated. Updates to the code, website and
manual are now all synchronized. We have the start of a more
comprehensive testsuite now. With buildbots on various architectures
running it on every commit. bzip2 is now part of oss-fuzz pulling from
the new git repository. And we did manage to integrate several changes
from distros and other forks/downstream back into the upstream bzip2
If we had more time (which I think we have, no rush to push out 1.0.9
quickly) then I think we want to do the following things:
- Extend the testsuite with more bz2 files that show interesting corner
cases (also as new seeds for fuzzers).
- Add more tests than just plain compress and decompress targets.
In particular I believe -f has some subtle behavior.
But it would also be good to have at least tests for all the libbz2
interfaces (some of which are not official/documented, see below).
- Add a Windows (cross) build and test (under wine) to the buildbot
as the major non-unix build that is supported.
- Provide more fuzzer targets and make them part of the upstream code
with a small wrapper so they can double as regression tests.
At least create targets for the low level BZ2_bzCompress,
BZ2_bzDecompress, high-level BZ2_bzRead (BZ2_bzReadGetUnused),
BZ2_bzWrite (BZ2_bzWriteFinish) and utility BZ2_bzBuffToBuffCompress,
BZ2_bzBuffToBuffDecompress functions in various configurations
and in compress/decompress mode to double check we can decompress
anything we compress ourselves.
- Update the manual to at least include documentation for the zlib
compatibility functions (see below). And double check it for any
other changes we made since the project moved to sourceware.
- Figure out how to produce the pdf version of the manual on more
setups. Currently it works perfectly on my RHEL7 setup, but some of
the buildbots cannot do a make dist because they don't produce a
correct pdf version (the html variant seems fine though).
- Some distros have some fixes for the man pages. Mainly symlinks for
some binaries. Look whether that should be upstreamed.
- Related, it would be good to generate the man page from the manual
again. Or find some other arrangement so that one or the other
is the main copy from which the other is generated/imported.
- We didn't pick up the Debian patch to add O_EXCL/O_CLOEXEC because
it seemed not portable (if we had a cross windows builder it would
probably have shown that the BZ_UNIX guards were incorrect). This
patch should probably be split in two.
- The O_EXCL part for bzip2recover should be easy to work exactly
like with bzip2. It would be good to not override the output
file if it already exists.
- The O_CLOEXEC one, through fopen "e" mode, part is trickier though.
It is only for the zlib compatible functions BZ2_bzopen and
BZ2_bzdopen. But those are not officially supported. Or as the
manual says: " These functions are not (yet) officially part of the
library, and are minimally documented here. If they break, you get
to keep all the pieces." And, worse, they seem to not be really
api compatible with the zlib versions. In particular the zlib
variants actually pass-through the "e" mode (!) so you don't need
to do that inside those functions themselves...
- We really should come up with a good path forward for the SONAME
mess. Currently Debian based or freedesktop SDK based distros/builds
use the current upstream libbz2.so.1.0 name. While Fedora and
openSUSE based distros will use the "sane" libbz2.so.1 name.
This means programs build against libbz2.so on those variants
don't run on the other. And so updating the upstream default
will break backward compatibility for one or the other.
I hope we can come up with an upgrade path that makes it possible
to run both binaries against a future libbz2.
My idea is simply to switch to the "sane" libbz2.so.1 name,
but also provide a wrapper library with the old libbz2.so.1.0 name.
Which might be as simply as:
gcc -shared -Wl,-soname -Wl,libbz2.so.1.0 -o libbz2.so.1.0 -lbz2
(where that -lbz2 linked against is the "sane" libbz2.so.1)
But there might be subtle issues with that I haven't thought about.
- Related, we probably also should tweak the symbol visibility so
libbz2 only exports functions/symbols that are actually supported.
- Nobody seems to really like our current build system and the Makefile
(fragments) you might have to hand edit. It is really basic and
simple and should work almost everywhere. But some of the above might
be helped with a slightly better build system. The best candidate
seems to be the autotools system that is already in use by various
distros and by Nix Packages collection (including for building cross
MinGW packages) since it was explicitly created to be integrated
(We will have to update the manual though, which currently says that
autotools aren't necessary, but I think just making cross compiling
to Windows easy proofs the original author was wrong, sorry Julian!)
Please let me know if I missed any other obvious goals for bzip2 1.0.9.
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2019-07-16 23:25 UTC | newest]
Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-01 0:00 bzip2 next steps - goals for 1.0.9 Mark Wielaard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).