From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 49549 invoked by alias); 27 Jun 2019 22:44:34 -0000 Mailing-List: contact bzip2-devel-help@sourceware.org; run by ezmlm Precedence: bulk List-Post: List-Help: List-Subscribe: List-Id: Sender: bzip2-devel-owner@sourceware.org Received: (qmail 49538 invoked by uid 89); 27 Jun 2019 22:44:34 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Checked: by ClamAV 0.100.3 on sourceware.org X-Virus-Found: No X-Spam-SWARE-Status: No, score=-0.7 required=5.0 tests=AWL,BAYES_00,SPF_NEUTRAL autolearn=no version=3.3.1 spammy=argued, hear X-Spam-Status: No, score=-0.7 required=5.0 tests=AWL,BAYES_00,SPF_NEUTRAL autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on sourceware.org X-Spam-Level: X-HELO: mx1.suse.de X-Virus-Scanned: by amavisd-new at test-mx.suse.de Message-ID: <1b88cafb93f6b37ced1c8bbfb04a115cf6a15200.camel@gnome.org> Subject: Re: bzip2 1.0.7 released From: Federico Mena Quintero To: Mark Wielaard , bzip2-devel@sourceware.org Cc: Julian Seward Date: Tue, 01 Jan 2019 00:00:00 -0000 In-Reply-To: <20190627205837.GD9273@wildebeest.org> References: <20190627205837.GD9273@wildebeest.org> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.30.4 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-SW-Source: 2019-q2/txt/msg00026.txt.bz2 On Thu, 2019-06-27 at 22:58 +0200, Mark Wielaard wrote: > The problem with the file in the report is that it does contain some > nSelectors that are slightly larger than BZ_MAX_SELECTORS. There's a discussion here: https://gitlab.com/federicomenaquintero/bzip2/commit/74de1e2e6ffc9d51ef9824db71a8ffee5962cdbc It has links to the lbzip2 bug, and the commit that fixed it. Summary: a bug in lbzip2 was causing it to write more than bzip2's allowed number of selectors; it got fixed in December 2018. If lbzip2's goals is to produce files which are compatible with bzip2, it can be argued that indeed it was producing broken files for a while. > -#define BZ_MAX_SELECTORS (2 + (900000 / BZ_G_SIZE)) > +#define BZ_MAX_SELECTORS (7 + (900000 / BZ_G_SIZE)) > > > > But of course I cannot tell why increasing the max with 5 is correct. > It might well be that the file is invalid. Before the fix bunzip2 > would overwrite some memory after the selectorMtf array. So it might > be the file decompressed by accident in the past. The comments in the lbzip2 commit that fixes this refer to the padding required to make the block size a multiple of 8 bits. I don't know enough about bzip2's format yet, or how the move-to-front algorithm works, to be able to tell if *that* simple fix for bzip2 of increasing BZ_MAX_SELECTORS is enough to make bzip2 able to read all of lbzip2's files. I think some distros ship lbzip2 instead of bzip2. If it was generating non-fully-compatible files... due to a bug in bzip2... it's probably fair to allow these files in bzip2 and be done with it. I'd still like to hear some confirmation (from Julian or the lbzip2 authors?) on whether the analysis above is correct. Federico