public inbox for mauve-discuss@sourceware.org
 help / color / mirror / Atom feed
* Testing JDK bugs?
@ 2006-07-27  9:49 Roman Kennke
  2006-07-27 13:08 ` David Gilbert
  0 siblings, 1 reply; 8+ messages in thread
From: Roman Kennke @ 2006-07-27  9:49 UTC (permalink / raw)
  To: David Gilbert, classpath, mauve-discuss

Hi lists, hi David (you wrote most of the tests I'm gonna talk about..)

While trying to clean up some Mauve failures I came upon a couple of 
tests that fail on JDK because they test strictly against the spec where 
the JDK isn't as strict. This is mostly bounds checking, where the spec 
says throws BadLocationException if invalid or similar. I think the JDK 
simply doesn't perform explicit checks in the Content implementations. 
See for example the tests for GapContent and StringContent.

Now I am not sure how to handle this. I've commented these tests out 
locally, simply to avoid clutter in the Mauve output. The question is 
how to interpret the spec. Adding the throws BadLocationException does 
mean (to me) that the impl may or may not throw a BadLocationException, 
but the application should be prepared to deal with it anyways. 
Moreover, the throws BadLocationException is specified in the interface. 
The implementations are not required to throw the BadLocationException 
if they decide to deal with wrong input themselves. For instance, the 
GapContent implementation (ours as well as JDK) can very well handle 
Position outside the range, because it only calculates offsets.

The situation gets worse. There are a number of tests both in Mauve and 
in the Intel testsuite that actually test the JDK behaviour of _not_ 
throwing the BLE, sometimes indirectly (via a Document impl or so). So 
we can't get to fully PASS with Mauve.
We should decide if we want to test strict spec compliance or reference 
impl compatibility. So far the decisions in GNU Classpath have been made 
in favour of (bug-) compatibility over strict spec compliance, so I 
think we should do the same for Mauve.

Anyway, I think we should either disable the spec-compliance checks or 
the RI-compatibility checks or both in Mauve so that we have at least a 
chance to reach 100%.

Any opinions on that?

/Roman

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Testing JDK bugs?
  2006-07-27  9:49 Testing JDK bugs? Roman Kennke
@ 2006-07-27 13:08 ` David Gilbert
  2006-07-27 14:56   ` Jeroen Frijters
  0 siblings, 1 reply; 8+ messages in thread
From: David Gilbert @ 2006-07-27 13:08 UTC (permalink / raw)
  To: Roman Kennke; +Cc: classpath, mauve-discuss

Roman Kennke wrote:

> Hi lists, hi David (you wrote most of the tests I'm gonna talk about..)
>
> While trying to clean up some Mauve failures I came upon a couple of 
> tests that fail on JDK because they test strictly against the spec 
> where the JDK isn't as strict. This is mostly bounds checking, where 
> the spec says throws BadLocationException if invalid or similar. I 
> think the JDK simply doesn't perform explicit checks in the Content 
> implementations. See for example the tests for GapContent and 
> StringContent.
>
> Now I am not sure how to handle this. I've commented these tests out 
> locally, simply to avoid clutter in the Mauve output. The question is 
> how to interpret the spec. Adding the throws BadLocationException does 
> mean (to me) that the impl may or may not throw a 
> BadLocationException, but the application should be prepared to deal 
> with it anyways. Moreover, the throws BadLocationException is 
> specified in the interface. The implementations are not required to 
> throw the BadLocationException if they decide to deal with wrong input 
> themselves. For instance, the GapContent implementation (ours as well 
> as JDK) can very well handle Position outside the range, because it 
> only calculates offsets.
>
> The situation gets worse. There are a number of tests both in Mauve 
> and in the Intel testsuite that actually test the JDK behaviour of 
> _not_ throwing the BLE, sometimes indirectly (via a Document impl or 
> so). So we can't get to fully PASS with Mauve.
> We should decide if we want to test strict spec compliance or 
> reference impl compatibility. So far the decisions in GNU Classpath 
> have been made in favour of (bug-) compatibility over strict spec 
> compliance, so I think we should do the same for Mauve.
>
> Anyway, I think we should either disable the spec-compliance checks or 
> the RI-compatibility checks or both in Mauve so that we have at least 
> a chance to reach 100%.
>
> Any opinions on that?
>
> /Roman

Hi Roman,

The theory is easy:  Mauve should test AN implementation against THE 
spec.  The reality is...well, you know already what the reality is.   
Send me the names of the tests that are causing problems, and I'll try 
to recode them to pass on Sun's JRE 1.5.0, otherwise I'll file a bug 
report with Sun.  I may recode some tests with two paths, so I can set a 
system property (say) and check for strict spec compliance if I want or 
need to.

I never heard back from Sun about the GapContent bug I filed in 
January.  I filed a few other bug reports since then, with little to 
show for it:

http://www.object-refinery.com/jre/bugs/

...but with 100,000 bug reports in the first half of 2006, I guess 
they're busy!

Regards,

Dave

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Testing JDK bugs?
  2006-07-27 13:08 ` David Gilbert
@ 2006-07-27 14:56   ` Jeroen Frijters
  2006-07-27 16:32     ` David Gilbert
  2006-07-27 17:01     ` Andrew Haley
  0 siblings, 2 replies; 8+ messages in thread
From: Jeroen Frijters @ 2006-07-27 14:56 UTC (permalink / raw)
  To: David Gilbert, Roman Kennke; +Cc: classpath, mauve-discuss

David Gilbert wrote:
> The theory is easy:  Mauve should test AN implementation against THE 
> spec.

Pardon me for beating my favorite horse again, but this assumes that the
spec is somehow more valuable than code and/or that the spec doesn't
contain bugs. In the real world both are buggy and users rarely care
about the spec, especially when their app works on the RI, but not on
our implementation.

Allow me to rebut another issue that often comes up: "We'll make it spec
compliant and when someone finds an application that depends on the RI
behavior then we'll copy that behavior."

IMNSHO, this is actually a very dumb approach. It makes our
implementation worse than the RI in two ways:

1) Apps coded against the RI (possibly) don't work out of the box.
2) Apps coded against our implementation (and spec) run the risk of
breaking in the future when we randomly decide to start emulating the RI
instead of the spec.

Of course, things aren't black and white and issues should be decided on
a case by case basis, but considering the spec holy is not doing anybody
any service.

Regards,
Jeroen

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Testing JDK bugs?
  2006-07-27 14:56   ` Jeroen Frijters
@ 2006-07-27 16:32     ` David Gilbert
  2006-07-27 17:01     ` Andrew Haley
  1 sibling, 0 replies; 8+ messages in thread
From: David Gilbert @ 2006-07-27 16:32 UTC (permalink / raw)
  To: Jeroen Frijters; +Cc: Roman Kennke, classpath, mauve-discuss

Jeroen Frijters wrote:

>David Gilbert wrote:
>  
>
>>The theory is easy:  Mauve should test AN implementation against THE 
>>spec.
>>    
>>
>
>Pardon me for beating my favorite horse again, but this assumes that the
>spec is somehow more valuable than code and/or that the spec doesn't
>contain bugs. In the real world both are buggy and users rarely care
>about the spec, especially when their app works on the RI, but not on
>our implementation.
>
>Allow me to rebut another issue that often comes up: "We'll make it spec
>compliant and when someone finds an application that depends on the RI
>behavior then we'll copy that behavior."
>
>IMNSHO, this is actually a very dumb approach. It makes our
>implementation worse than the RI in two ways:
>
>1) Apps coded against the RI (possibly) don't work out of the box.
>2) Apps coded against our implementation (and spec) run the risk of
>breaking in the future when we randomly decide to start emulating the RI
>instead of the spec.
>
>Of course, things aren't black and white and issues should be decided on
>a case by case basis, but considering the spec holy is not doing anybody
>any service.
>  
>

I'm sorry to have set you off.  Bear in mind that there is a difference 
between the theory and the practice here...I'm not too hung up on the 
theory, and I think the remainder of my original post made that clear.

Regards,

Dave

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Testing JDK bugs?
  2006-07-27 14:56   ` Jeroen Frijters
  2006-07-27 16:32     ` David Gilbert
@ 2006-07-27 17:01     ` Andrew Haley
  2006-07-28  7:56       ` Jeroen Frijters
  1 sibling, 1 reply; 8+ messages in thread
From: Andrew Haley @ 2006-07-27 17:01 UTC (permalink / raw)
  To: Jeroen Frijters; +Cc: David Gilbert, Roman Kennke, classpath, mauve-discuss

Jeroen Frijters writes:
 > David Gilbert wrote:
 > > The theory is easy:  Mauve should test AN implementation against THE 
 > > spec.
 > 
 > Pardon me for beating my favorite horse again, but this assumes that the
 > spec is somehow more valuable than code and/or that the spec doesn't
 > contain bugs. In the real world both are buggy and users rarely care
 > about the spec, especially when their app works on the RI, but not on
 > our implementation.
 > 
 > Allow me to rebut another issue that often comes up: "We'll make it spec
 > compliant and when someone finds an application that depends on the RI
 > behavior then we'll copy that behavior."

That depends on whether or not we think the spec or some
implementation is buggy.  It is often possible to tell -- some
behaviours are just Obviously Wrong.  When it's not possible to tell,
we have to make a judgment call.

 > IMNSHO, this is actually a very dumb approach. It makes our
 > implementation worse than the RI in two ways:
 > 
 > 1) Apps coded against the RI (possibly) don't work out of the box.
 > 2) Apps coded against our implementation (and spec) run the risk of
 > breaking in the future when we randomly decide to start emulating
 > the RI instead of the spec.

Or, just as likely, the "RI" eventually gets fixed.  Should we have to
wait for that other implementation to be fixed before will allow
ourselves to fix ours?  That really would be dumb!

 > Of course, things aren't black and white and issues should be
 > decided on a case by case basis, but considering the spec holy

Beat that straw man!

 > is not doing anybody any service.

Andrew.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Testing JDK bugs?
  2006-07-27 17:01     ` Andrew Haley
@ 2006-07-28  7:56       ` Jeroen Frijters
  2006-07-28 12:33         ` Sven de Marothy
  2006-07-28 14:51         ` Tom Tromey
  0 siblings, 2 replies; 8+ messages in thread
From: Jeroen Frijters @ 2006-07-28  7:56 UTC (permalink / raw)
  To: Andrew Haley; +Cc: David Gilbert, Roman Kennke, classpath, mauve-discuss

Andrew Haley wrote:
> That depends on whether or not we think the spec or some
> implementation is buggy.  It is often possible to tell -- some
> behaviours are just Obviously Wrong. 

Even if something is "Obviously Wrong", it may not be a good idea to fix
it because it would be a breaking change. For example:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6227069

> When it's not possible to tell, we have to make a judgment call.

We obviously agree on this, but I'm trying to point out that things
aren't always as "obvious" as one would think at first glance.

Regards,
Jeroen

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Testing JDK bugs?
  2006-07-28  7:56       ` Jeroen Frijters
@ 2006-07-28 12:33         ` Sven de Marothy
  2006-07-28 14:51         ` Tom Tromey
  1 sibling, 0 replies; 8+ messages in thread
From: Sven de Marothy @ 2006-07-28 12:33 UTC (permalink / raw)
  To: Jeroen Frijters; +Cc: mauve-discuss, classpath

On Fri, 2006-07-28 at 09:56 +0200, Jeroen Frijters wrote:

> Even if something is "Obviously Wrong", it may not be a good idea to fix
> it because it would be a breaking change. For example:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6227069
> 

Obvious? Anyway, IMHO "fixing" something in a compatibility-breaking
way simply isn't a fix at all. It's the opposite. 

However, this goes in the opposite direction as well. And that's when
you really need to make a judgement call. I just did so in the
BigDecimal patch I just commited (which is why I was rude enough to
cross-post it to the main list).

Which is that setScale(scale, rounding) should throw an
ArithmeticException if scale < 0. The spec says so, and the JRE
does so prior to v1.5. In 1.5 it does not (with no change to the doc).

So without thinking too hard I figured it was a 1.5 regression and 
put the exception back in (fixing a mauve regression on our part).
After all, this could break compatibility, in theory.

Although now I'm not quite so sure, because the probable cause
of this is that the divide(BigDecimal, scale, rounding) method
which we (and Sun, presumably) implement this on, shows the same
behaviour, only the docs have changed. 

/Sven

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: Testing JDK bugs?
  2006-07-28  7:56       ` Jeroen Frijters
  2006-07-28 12:33         ` Sven de Marothy
@ 2006-07-28 14:51         ` Tom Tromey
  1 sibling, 0 replies; 8+ messages in thread
From: Tom Tromey @ 2006-07-28 14:51 UTC (permalink / raw)
  To: Jeroen Frijters
  Cc: Andrew Haley, David Gilbert, Roman Kennke, classpath, mauve-discuss

Jeroen> We obviously agree on this, but I'm trying to point out that things
Jeroen> aren't always as "obvious" as one would think at first glance.

How about distilling this wisdom into a section in the Classpath
hacking guide?

Tom

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2006-07-28 14:51 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-07-27  9:49 Testing JDK bugs? Roman Kennke
2006-07-27 13:08 ` David Gilbert
2006-07-27 14:56   ` Jeroen Frijters
2006-07-27 16:32     ` David Gilbert
2006-07-27 17:01     ` Andrew Haley
2006-07-28  7:56       ` Jeroen Frijters
2006-07-28 12:33         ` Sven de Marothy
2006-07-28 14:51         ` Tom Tromey

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).