Re: New GNU C Library (glibc) security flaw reported on 30 Jan 2024

[Alexander Monakov <amonakov@ispras.ru>]

[-- Attachment #1: Type: text/plain, Size: 1468 bytes --]


On Wed, 7 Feb 2024, Zack Weinberg wrote:

> >> and C2011 7.22.5p2 actually makes this a hard requirement: “The
> >> implementation shall ensure that both arguments [of the comparison
> >> function called by qsort] are pointers to elements of the array.”
> >> It looks to me like there are situations where our implementation
> >> doesn’t do this:
> >
> > I don't see that in the glibc source. Are you sure about that?
> 
> I was wrong about this; I misread the code in qsort.c.  Specifically,
> I missed that msort_with_tmp always merges *from* two subsets of the
> main array, *into* the temporary array, and then copies the result back
> to the main array.  Still, I think it is likely that some C libraries in
> the past have failed to implement this requirement (intentionally or not).
> In the attached patch I revised that suggestion again and I hope it satisfies
> everyone's expectations now.

I don't understand how that is likely: was it common to use merge sort for
implementation of qsort? For in-place sorting algorithms that mistake is
pretty much impossible to make.

In any case it seems unnecessary (and even impolite maybe, towards the authors
of those other C libraries) to make such implication in the Glibc manual.
It's much nicer to stick to the facts about Glibc itself.

(Glibc VCS briefly carried an implementation with such a mistake around 2002
when Roger Sayle's "Towers of Hanoi merge sort" was applied and then reverted)

Alexander