[RFC] jpeg library

[RFC] jpeg library

[Charles Wilson]

For many years -- since the first "net release" after B20.1 in fact --
cygwin's jpeg library has included the so-called "lossless jpeg" patch.
 This has been somewhat controversial.

The patch modifies certain data structures that are marked "private" in
the jpeg headers, but they *are* in the headers.  Therefore, many
clients have manipulated those structures directly.  To interoperate
with cygwin's lossless-patched version, they had to make certain
accomodations:

http://cygwin.com/ml/cygwin/2005-07/msg01005.html

So the question becomes, why did we (I) add it in the first place, and
why do we still needed it?  I added it because I needed it to deal with
certain medical imagery, and frankly was unaware at the time that
changing "private" data structures would really affect anybody else.

Isn't that what "private" means?

Well, I no longer need to deal with that sort of imagery, and "private"
never really was very private, and it DID cause lots of people grief.
And, our build environment for packages is much nicer now than it was in
the early days, so any Joe who needs lossless jpeg can easily build it
themselves.  So, it'd be nice to get rid of the lossless jpeg patch.

Why shouldn't we get rid of it?  Well, because over the years those
other clients have added lots of workarounds to accommodate cygwin's
jpeg library.  If we removed the lossless jpeg stuff, then they wouldn't
need them -- but until they too removed their workarounds, their builds
would break on cygwin again!

Well, I see an opportunity to do exactly that: we are soon to release
cygwin-1.7, AND upgrade to gcc4/dw2/shared-libgcc as the standard
compiler.  Acccording to:
  http://cygwin.com/ml/cygwin-apps/2009-04/msg00034.html
this gives us a free pass to break ABI on DLLs -- like jpeg.

(Sortof. It's really only a free pass IF the ABI breakage is
specifically caused by the switch to cygwin1.7/gcc4/dw2/shared-libgcc.
NOT if the ABI breakage is caused by some other internal change...like
(reverting) modifications to data structures which are publicly exposed
but marked private in header file comments.  So, the proposal below is
stretching the rules a bit.  Which is why I want comments.)

So...

I propose to release a new libjpeg for cygwin-1.7 using gcc4 with the
shared libgcc runtime, once both cygwin-1.7 and gcc4 are officially
non-beta (e.g. moved to curr, not test).  This libjpeg will be named:

jpeg                            6b-N
libjpeg-devel                   6b-N
libjpeg62     "cygjpeg-62.dll"  6b-N

just like the current packages, where N > 20.  The DLL number, name, and
package name will NOT be incremented.  Existing packages that use
libjpeg, AND that "illegally" accessed the so-called "private" data
members, will break and will need to be recompiled, without whatever
internal workarounds they had previously implemented to deal with our
lossless stuff.

I could be persuaded to release it earlier.  That is, before cygwin-1.7
goes live although I'd rather not cause instability in the distro this
close to cygwin-1.7's release.  Or, sometime after cygwin-1.7.1, but
before gcc-4.x goes gold.

Alternative:

Another alternative is to release the lossless-jpeg-less libjpeg now,
for cygwin-1.7(beta), using gcc-3.4.5, under a new DLL name.  This way,
there is an ABI breakage but it's in a new DLL with a new number
(cygjpeg-63? -100? something), so nobody has a problem; doing this won't
destabilize the cygwin distribution at all.  It's just a normal DLL
update.  Then, when cygwin1.7 and gcc4 go live, I rebuild the
cygjpeg-100.dll using the new gcc4 and we have (maybe) a second ABI
breakage, only this one isn't accompanied by a DLL version bump, because
it would be due solely to issues related to the compiler switch, per:
  http://cygwin.com/ml/cygwin-apps/2009-04/msg00034.html

The downside of this approach is cygwin's jpeg DLL would have a
different name than is normally implied by the stock build machinery, and
  a) we would continue to have to patch our jpeg builds to use the new
numbering sequence in perpetuity
  b) any applications that dlopen libjpeg would need patching, in
perpetuity. I'm not sure this is much of an issue.

Open for discussion...

--
Chuck


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [RFC] jpeg library

[Yaakov (Cygwin/X)]

On 27/06/2009 13:30, Charles Wilson wrote:
> Well, I no longer need to deal with that sort of imagery, and "private"
> never really was very private, and it DID cause lots of people grief.

Moreso, the patch removed the height_in_blocks/width_in_blocks members 
from struct jpeg_component_info, which are used in a transupp.c file 
which is part of several desktop image viewers (I know of at least six), 
as well as the progressive_mode member from the jpeg_compress_struct 
struct, which is used in gtk+-2.12 and up.

> And, our build environment for packages is much nicer now than it was in
> the early days, so any Joe who needs lossless jpeg can easily build it
> themselves.  So, it'd be nice to get rid of the lossless jpeg patch.

+1

> Why shouldn't we get rid of it?  Well, because over the years those
> other clients have added lots of workarounds to accommodate cygwin's
> jpeg library.  If we removed the lossless jpeg stuff, then they wouldn't
> need them -- but until they too removed their workarounds, their builds
> would break on cygwin again!

What is the chance of breakage for regular clients?

> I propose to release a new libjpeg for cygwin-1.7 using gcc4 with the
> shared libgcc runtime, once both cygwin-1.7 and gcc4 are officially
> non-beta (e.g. moved to curr, not test).  This libjpeg will be named:

I presume that an upstream (ABI) version bump -- the easiest time to 
make changes like this -- is not in sight.

> just like the current packages, where N>  20.  The DLL number, name, and
> package name will NOT be incremented.  Existing packages that use
> libjpeg, AND that "illegally" accessed the so-called "private" data
> members, will break and will need to be recompiled, without whatever
> internal workarounds they had previously implemented to deal with our
> lossless stuff.

May I suggest making these available for a limited-time manual download 
for maintainers to test and, if necessary, rebuild their own packages to 
be ready when these are released.

> I could be persuaded to release it earlier.  That is, before cygwin-1.7
> goes live although I'd rather not cause instability in the distro this
> close to cygwin-1.7's release.  Or, sometime after cygwin-1.7.1, but
> before gcc-4.x goes gold.

I think earlier is better, but it should be coordinated.

> Another alternative is to release the lossless-jpeg-less libjpeg now,
> for cygwin-1.7(beta), using gcc-3.4.5, under a new DLL name.  This way,
> there is an ABI breakage but it's in a new DLL with a new number
> (cygjpeg-63? -100? something), so nobody has a problem; doing this won't
> destabilize the cygwin distribution at all.  It's just a normal DLL
> update.  Then, when cygwin1.7 and gcc4 go live, I rebuild the
> cygjpeg-100.dll using the new gcc4 and we have (maybe) a second ABI
> breakage, only this one isn't accompanied by a DLL version bump, because
> it would be due solely to issues related to the compiler switch, per:
>    http://cygwin.com/ml/cygwin-apps/2009-04/msg00034.html
>
> The downside of this approach is cygwin's jpeg DLL would have a
> different name than is normally implied by the stock build machinery, and
>    a) we would continue to have to patch our jpeg builds to use the new
> numbering sequence in perpetuity

Which is one reason I was against gcc4-ABI-bumps in the first place.

>    b) any applications that dlopen libjpeg would need patching, in
> perpetuity. I'm not sure this is much of an issue.

Can't think of any off the top of my head, but it's possible.

> Open for discussion...

My $0.02 CAD.


Yaakov

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [RFC] jpeg library

[Charles Wilson]

Yaakov (Cygwin/X) wrote:
> On 27/06/2009 13:30, Charles Wilson wrote:
>> Well, I no longer need to deal with that sort of imagery, and "private"
>> never really was very private, and it DID cause lots of people grief.
> 
> Moreso, the patch removed the height_in_blocks/width_in_blocks members
> from struct jpeg_component_info, which are used in a transupp.c file
> which is part of several desktop image viewers (I know of at least six),
> as well as the progressive_mode member from the jpeg_compress_struct
> struct, which is used in gtk+-2.12 and up.

This is precisely what I'm talking about.  Those fields in struct
jpeg_component_info are explicitly marked private in jpeglib.h:

  /* Remaining fields should be treated as private by applications. */

But they ARE PRESENT in that file, so there is an argument as to exactly
how private they really are. (There's a well established technique for
using opaque pointers to truly private data structures; why doesn't
libjpeg use those techniques?)

So, arguably, those six desktop image viewers as well as gtk+2.12 are
being slightly naughty.

But that's all by the by.  I want to revert the lossless patch going
forward; the question here is how to manage the transition.

>> And, our build environment for packages is much nicer now than it was in
>> the early days, so any Joe who needs lossless jpeg can easily build it
>> themselves.  So, it'd be nice to get rid of the lossless jpeg patch.
> 
> +1
> 
>> Why shouldn't we get rid of it?  Well, because over the years those
>> other clients have added lots of workarounds to accommodate cygwin's
>> jpeg library.  If we removed the lossless jpeg stuff, then they wouldn't
>> need them -- but until they too removed their workarounds, their builds
>> would break on cygwin again!
> 
> What is the chance of breakage for regular clients?

You mean for clients that aren't naughty, and do not/never did access
these "private" fields?  None, as far as I can tell.  The *size* of the
struct doesn't change. Only the names of some of the fields (not their
type), and their meaning.  Some purely internal structs change
significantly, but that shouldn't affect external clients, even naughty
ones.

Although the actual strings associated with various error code numbers
changes, so that could lead to some surprising incorrect error messages
until clients are recompiled.

In any event, if I release a new cygjpeg-62.dll (same as current name)
without the lossless jpeg patch, and it DOES break everything in sight,
well then, I'll withdraw it and we'll go with the plan B transition.

>> I propose to release a new libjpeg for cygwin-1.7 using gcc4 with the
>> shared libgcc runtime, once both cygwin-1.7 and gcc4 are officially
>> non-beta (e.g. moved to curr, not test).  This libjpeg will be named:
> 
> I presume that an upstream (ABI) version bump -- the easiest time to
> make changes like this -- is not in sight.

Nope. Upstream development is DEAD.  There was some flurry of activity
about two years ago, but it never went anywhere.   If IJG's libjpeg
wasn't so widespread and widely used, I'd want to look at some other
library that supports the format...

>> just like the current packages, where N>  20  The DLL number, name, and
>> package name will NOT be incremented.  Existing packages that use
>> libjpeg, AND that "illegally" accessed the so-called "private" data
>> members, will break and will need to be recompiled, without whatever
>> internal workarounds they had previously implemented to deal with our
>> lossless stuff.
> 
> May I suggest making these available for a limited-time manual download
> for maintainers to test and, if necessary, rebuild their own packages to
> be ready when these are released.

Well, "test" release, sure.  But given the interlocking dependencies of
libraries (esp. graphics libraries), you'd need an ever-increasing
number of these libraries in pending/test state. I don't have the
bandwidth to manage that outside of the sourceware facilities.  Since we
have a perfectly good mirror system and "test status" already available
for such things...

>> I could be persuaded to release it earlier.  That is, before cygwin-1.7
>> goes live although I'd rather not cause instability in the distro this
>> close to cygwin-1.7's release.  Or, sometime after cygwin-1.7.1, but
>> before gcc-4.x goes gold.
> 
> I think earlier is better, but it should be coordinated.

Well, with 1.7 due out sometime in the next week or so, I really don't
want to destabilize that.

>> The downside of this approach is cygwin's jpeg DLL would have a
>> different name than is normally implied by the stock build machinery, and
>>    a) we would continue to have to patch our jpeg builds to use the new
>> numbering sequence in perpetuity
> 
> Which is one reason I was against gcc4-ABI-bumps in the first place.

Where I begin acting as the devil's advocate:

Yeah, but on the other hand, every distro out there has a stack of
patches for libjpeg, because upstream is dead but the code has bugs.
Thus, debian has a list of libjpeg patches that, for all intents and
purposes, they must now "maintain in perpetuity".

That's the danger of such heavy reliance on a dead project; no place to
send the fixes.  So either live with the bugs, or maintain external patches.

This would not be all that different -- and we, too, have a stack of
bugfix patches for libjpeg (extracted from debian).

But this isn't a "pseudo-incompatibility" caused by a simple switch to
gcc4/shared-libgcc. This is a real ABI violation we're contemplating.
It's exactly what DLL version number bumps are FOR.

>>    b) any applications that dlopen libjpeg would need patching, in
>> perpetuity. I'm not sure this is much of an issue.
> 
> Can't think of any off the top of my head, but it's possible.

Yeah, I couldn't either. libjpeg isn't really the sort of library you
dlopen, is it?  Besides, does this hypothetical app know already about
our 'cyg*-n.dll' naming convention? If not then it's already broken and
needs patching for cygiwn. If so, then it already has cygwin-specific
workaround code that could be tweaked.

--
Chuck


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [RFC] jpeg library

[Yaakov (Cygwin/X)]

On 28/06/2009 01:17, Charles Wilson wrote:
> You mean for clients that aren't naughty, and do not/never did access
> these "private" fields?  None, as far as I can tell.  The *size* of the
> struct doesn't change. Only the names of some of the fields (not their
> type), and their meaning.  Some purely internal structs change
> significantly, but that shouldn't affect external clients, even naughty
> ones.
>
> Although the actual strings associated with various error code numbers
> changes, so that could lead to some surprising incorrect error messages
> until clients are recompiled.
>
> In any event, if I release a new cygjpeg-62.dll (same as current name)
> without the lossless jpeg patch, and it DOES break everything in sight,
> well then, I'll withdraw it and we'll go with the plan B transition.

That should be easily verified with a bit of testing on everyone's part, 
*before* the release.

> Well, "test" release, sure.  But given the interlocking dependencies of
> libraries (esp. graphics libraries), you'd need an ever-increasing
> number of these libraries in pending/test state. I don't have the
> bandwidth to manage that outside of the sourceware facilities.  Since we
> have a perfectly good mirror system and "test status" already available
> for such things...

If there is indeed no ABI breakage in the normal case, there shouldn't 
be any need for a bunch of packages in testing; everyone just tests the 
new libjpeg62, and if their package(s) still work, nothing more to do, 
right?

> Well, with 1.7 due out sometime in the next week or so, I really don't
> want to destabilize that.

That shouldn't stop you from a test: version ASAP.

> Yeah, but on the other hand, every distro out there has a stack of
> patches for libjpeg, because upstream is dead but the code has bugs.
> Thus, debian has a list of libjpeg patches that, for all intents and
> purposes, they must now "maintain in perpetuity".

True; thankfully the Linux distros are doing the hard work for us. :-)

> But this isn't a "pseudo-incompatibility" caused by a simple switch to
> gcc4/shared-libgcc. This is a real ABI violation we're contemplating.
> It's exactly what DLL version number bumps are FOR.

Freetype went through a similar transition some time ago[1]: they used 
to expose many internal symbols and headers, and before 2.2.0 they 
stopped.  Interestingly, they didn't change the ABI number, since the 
only changes were to things supposed to be internal.  Everyone fixed 
their packages (and the FT devs did help provide packages), and life 
moved on.

[1] http://freetype.sourceforge.net/freetype2/freetype-2.2.0.html


Yaakov

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [RFC] jpeg library

[Yaakov (Cygwin/X)]

On 28/06/2009 01:17, Charles Wilson wrote:
> Nope. Upstream development is DEAD.  There was some flurry of activity
> about two years ago, but it never went anywhere.   If IJG's libjpeg
> wasn't so widespread and widely used, I'd want to look at some other
> library that supports the format...

Actually, are you sure about that?  http://www.ijg.org/ lists a version 
7, released yesterday, as the current version.


Yaakov

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [RFC] jpeg library

[Dave Korn]

Yaakov (Cygwin/X) wrote:
> On 28/06/2009 01:17, Charles Wilson wrote:
>> Nope. Upstream development is DEAD.  There was some flurry of activity
>> about two years ago, but it never went anywhere.   If IJG's libjpeg
>> wasn't so widespread and widely used, I'd want to look at some other
>> library that supports the format...
> 
> Actually, are you sure about that?  http://www.ijg.org/ lists a version
> 7, released yesterday, as the current version.

  I was just about to say "We could always try and find a security
vulnerability, that's the only thing that would cause upstream to update
libjpeg these days!  :)  But then I took a look at the new change.log and it's
actually crammed with new and improved functionality, so this is an ideal
opportunity.

  I think the makes freezing (obsoleting) libjpeg-6b forever with the lossless
patch incorporated, and just moving to a straight unpatched release of
libjpeg-7 a no-brainer doesn't it?  What a fortuitous bit of timing!

    cheers,
      DaveK


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [RFC] jpeg library

[Charles Wilson]

Yaakov (Cygwin/X) wrote:
> On 28/06/2009 01:17, Charles Wilson wrote:
>> Nope. Upstream development is DEAD.  There was some flurry of activity
>> about two years ago, but it never went anywhere.   If IJG's libjpeg
>> wasn't so widespread and widely used, I'd want to look at some other
>> library that supports the format...
> 
> Actually, are you sure about that?  http://www.ijg.org/ lists a version
> 7, released yesterday, as the current version.

WTH?  You're kidding me...I'm floored.  There's a private mailing list
(with no online archives, natch) for the IJG project, but my
subscription is with an different email account, so I haven't checked it
in a while...nope, no activity on THAT list recently.  Maybe there's a
new list?

Anyway, wow.  OK, this changes things...

From the (competing?) libjpeg on sourceforge, there was a conversation
earlier this week, involving Guido Vollbeding (current IJG jpeg
maintainer), Tom Lane (previous IJG jpeg maintainer), and Bob
Friesenhahn (sourceforge libjpeg maintainer, GraphicsMagick).  Guido said:

> I have taken great care that v6b can be replaced seamlessly with v7,
> and even v8 will remain API compatible while introducing essential
> novelties.

BUT...they went ahead and changed the library version number anyway. In
the sourceforge code (on which cygwin's current jpeg is based), it was
-version-info=62.  In the new jpeg-v7 code, it is -version-info=7:0

In effect, this represents a change from "cygjpeg-62.dll" to
"cygjpeg-7.dll".  Given the speed of jpeg development, I doubt any of us
will be alive when the new DLL number series gets up to 61 (at over 10
years per major number...it'll be 2549 A.D. before we get there).

In that same conversation, there is a lot of mention of the use of
symbol versioning as the panacea for all possible version conflicts.
Nobody has seemed to point out that it works only on ELF systems.

Looks like jpeg-7 incorporates a number of the Debian patches, and many
of the patches from jpegclub.org, so that's good (Hmmm. Actually, the
jpegclub.org website has been updated and now claims
"JPEGclub.org...maintains the Independent JPEG Group's (IJG) software.".
I guess this isn't surprising, as jpegclub.org has always been Guido
Vollbeding's site; now that he's taken over from Tom Lane officially,
this only makes sense).

So, to sum up: ORDINARILY, the new jpeg library should be binary
compatible (and definitely API compatible) with the old one.  It has
lots of new functionality, but in a backwards-compatible way.  However,
they changed the SONAME anyway, which gives us the perfect opportunity
we're looking for.

OK...new plan:  jpeg-v7 will be released for cygwin-1.7 only, using
gcc4/dw2/shared-libgcc only, and will have the name "cygjpeg-7.dll".  It
will NOT have lossless jpeg support.

I'll do this soon.

--
Chuck


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [RFC] jpeg library

[Charles Wilson]

Dave Korn wrote:
>   I was just about to say "We could always try and find a security
> vulnerability, that's the only thing that would cause upstream to update
> libjpeg these days!  :)  But then I took a look at the new change.log and it's
> actually crammed with new and improved functionality, so this is an ideal
> opportunity.
> 
>   I think the makes freezing (obsoleting) libjpeg-6b forever with the lossless
> patch incorporated, and just moving to a straight unpatched release of
> libjpeg-7 a no-brainer doesn't it?  What a fortuitous bit of timing!

Yep.  Sometimes you get lucky.  Not often -- that's why it's called luck.

--
Chuck

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [RFC] jpeg library

[Dave Korn]

Charles Wilson wrote:

> In that same conversation, there is a lot of mention of the use of
> symbol versioning as the panacea for all possible version conflicts.
> Nobody has seemed to point out that it works only on ELF systems.

  This will not be a problem for us forever, I hope :)

    cheers,
      DaveK


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [RFC] jpeg library

[Charles Wilson]

Charles Wilson wrote:

> OK...new plan:  jpeg-v7 will be released for cygwin-1.7 only, using
> gcc4/dw2/shared-libgcc only, and will have the name "cygjpeg-7.dll".  It
> will NOT have lossless jpeg support.
> 
> I'll do this soon.

I've just posted, for 1.7 only, an update for libjpeg to jpeg v7. This
version no longer supports lossless jpeg, and therefore no longer
requires ill-behaved client packages that manipulate "private" jpeg data
structures to take special care when compiling on cygwin.

However, this means that if you maintain a client package that DOES
currently take special care on cygwin, to accommodate our old lossless
jpeg support, then your build may break NOW, until you remove those
modifications from your package.

However, contrary to the statement above, libjpeg7 was compiled using
gcc3, not gcc4/dw2/shared-libgcc.  A later update will switch to gcc4
after gcc4 goes "gold".

I've also uploaded (for cygwin-1.7 only) a "backwards compatibility" DLL
package, libjpeg62, containing the old-style DLL -- so existing
applications will continue to work properly.

Finally, I've uploaded (for both cygwin-1.5 and cygwin-1.7) a new
version of the tiff packages:

1) cygwin-1.7: tiff-3.8.2-11 compiled against the new libjpeg7, as a
sort of 'smoke test'.  Also, imported Debian fixes for CVE-2009-2285 and
CVE-2009-2347.

2) cygwin-1.5: tiff-3.8.2-5 recompiled to bring in Debian fixes for
CVE-2009-2285 and CVE-2009-2347.

--
Chuck

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple