Hi lists, hi David (you wrote most of the tests I'm gonna talk about..) While trying to clean up some Mauve failures I came upon a couple of tests that fail on JDK because they test strictly against the spec where the JDK isn't as strict. This is mostly bounds checking, where the spec says throws BadLocationException if invalid or similar. I think the JDK simply doesn't perform explicit checks in the Content implementations. See for example the tests for GapContent and StringContent. Now I am not sure how to handle this. I've commented these tests out locally, simply to avoid clutter in the Mauve output. The question is how to interpret the spec. Adding the throws BadLocationException does mean (to me) that the impl may or may not throw a BadLocationException, but the application should be prepared to deal with it anyways. Moreover, the throws BadLocationException is specified in the interface. The implementations are not required to throw the BadLocationException if they decide to deal with wrong input themselves. For instance, the GapContent implementation (ours as well as JDK) can very well handle Position outside the range, because it only calculates offsets. The situation gets worse. There are a number of tests both in Mauve and in the Intel testsuite that actually test the JDK behaviour of _not_ throwing the BLE, sometimes indirectly (via a Document impl or so). So we can't get to fully PASS with Mauve. We should decide if we want to test strict spec compliance or reference impl compatibility. So far the decisions in GNU Classpath have been made in favour of (bug-) compatibility over strict spec compliance, so I think we should do the same for Mauve. Anyway, I think we should either disable the spec-compliance checks or the RI-compatibility checks or both in Mauve so that we have at least a chance to reach 100%. Any opinions on that? /Roman
Roman Kennke wrote: > Hi lists, hi David (you wrote most of the tests I'm gonna talk about..) > > While trying to clean up some Mauve failures I came upon a couple of > tests that fail on JDK because they test strictly against the spec > where the JDK isn't as strict. This is mostly bounds checking, where > the spec says throws BadLocationException if invalid or similar. I > think the JDK simply doesn't perform explicit checks in the Content > implementations. See for example the tests for GapContent and > StringContent. > > Now I am not sure how to handle this. I've commented these tests out > locally, simply to avoid clutter in the Mauve output. The question is > how to interpret the spec. Adding the throws BadLocationException does > mean (to me) that the impl may or may not throw a > BadLocationException, but the application should be prepared to deal > with it anyways. Moreover, the throws BadLocationException is > specified in the interface. The implementations are not required to > throw the BadLocationException if they decide to deal with wrong input > themselves. For instance, the GapContent implementation (ours as well > as JDK) can very well handle Position outside the range, because it > only calculates offsets. > > The situation gets worse. There are a number of tests both in Mauve > and in the Intel testsuite that actually test the JDK behaviour of > _not_ throwing the BLE, sometimes indirectly (via a Document impl or > so). So we can't get to fully PASS with Mauve. > We should decide if we want to test strict spec compliance or > reference impl compatibility. So far the decisions in GNU Classpath > have been made in favour of (bug-) compatibility over strict spec > compliance, so I think we should do the same for Mauve. > > Anyway, I think we should either disable the spec-compliance checks or > the RI-compatibility checks or both in Mauve so that we have at least > a chance to reach 100%. > > Any opinions on that? > > /Roman Hi Roman, The theory is easy: Mauve should test AN implementation against THE spec. The reality is...well, you know already what the reality is. Send me the names of the tests that are causing problems, and I'll try to recode them to pass on Sun's JRE 1.5.0, otherwise I'll file a bug report with Sun. I may recode some tests with two paths, so I can set a system property (say) and check for strict spec compliance if I want or need to. I never heard back from Sun about the GapContent bug I filed in January. I filed a few other bug reports since then, with little to show for it: http://www.object-refinery.com/jre/bugs/ ...but with 100,000 bug reports in the first half of 2006, I guess they're busy! Regards, Dave
David Gilbert wrote:
> The theory is easy: Mauve should test AN implementation against THE
> spec.
Pardon me for beating my favorite horse again, but this assumes that the
spec is somehow more valuable than code and/or that the spec doesn't
contain bugs. In the real world both are buggy and users rarely care
about the spec, especially when their app works on the RI, but not on
our implementation.
Allow me to rebut another issue that often comes up: "We'll make it spec
compliant and when someone finds an application that depends on the RI
behavior then we'll copy that behavior."
IMNSHO, this is actually a very dumb approach. It makes our
implementation worse than the RI in two ways:
1) Apps coded against the RI (possibly) don't work out of the box.
2) Apps coded against our implementation (and spec) run the risk of
breaking in the future when we randomly decide to start emulating the RI
instead of the spec.
Of course, things aren't black and white and issues should be decided on
a case by case basis, but considering the spec holy is not doing anybody
any service.
Regards,
Jeroen
Jeroen Frijters wrote:
>David Gilbert wrote:
>
>
>>The theory is easy: Mauve should test AN implementation against THE
>>spec.
>>
>>
>
>Pardon me for beating my favorite horse again, but this assumes that the
>spec is somehow more valuable than code and/or that the spec doesn't
>contain bugs. In the real world both are buggy and users rarely care
>about the spec, especially when their app works on the RI, but not on
>our implementation.
>
>Allow me to rebut another issue that often comes up: "We'll make it spec
>compliant and when someone finds an application that depends on the RI
>behavior then we'll copy that behavior."
>
>IMNSHO, this is actually a very dumb approach. It makes our
>implementation worse than the RI in two ways:
>
>1) Apps coded against the RI (possibly) don't work out of the box.
>2) Apps coded against our implementation (and spec) run the risk of
>breaking in the future when we randomly decide to start emulating the RI
>instead of the spec.
>
>Of course, things aren't black and white and issues should be decided on
>a case by case basis, but considering the spec holy is not doing anybody
>any service.
>
>
I'm sorry to have set you off. Bear in mind that there is a difference
between the theory and the practice here...I'm not too hung up on the
theory, and I think the remainder of my original post made that clear.
Regards,
Dave
Jeroen Frijters writes: > David Gilbert wrote: > > The theory is easy: Mauve should test AN implementation against THE > > spec. > > Pardon me for beating my favorite horse again, but this assumes that the > spec is somehow more valuable than code and/or that the spec doesn't > contain bugs. In the real world both are buggy and users rarely care > about the spec, especially when their app works on the RI, but not on > our implementation. > > Allow me to rebut another issue that often comes up: "We'll make it spec > compliant and when someone finds an application that depends on the RI > behavior then we'll copy that behavior." That depends on whether or not we think the spec or some implementation is buggy. It is often possible to tell -- some behaviours are just Obviously Wrong. When it's not possible to tell, we have to make a judgment call. > IMNSHO, this is actually a very dumb approach. It makes our > implementation worse than the RI in two ways: > > 1) Apps coded against the RI (possibly) don't work out of the box. > 2) Apps coded against our implementation (and spec) run the risk of > breaking in the future when we randomly decide to start emulating > the RI instead of the spec. Or, just as likely, the "RI" eventually gets fixed. Should we have to wait for that other implementation to be fixed before will allow ourselves to fix ours? That really would be dumb! > Of course, things aren't black and white and issues should be > decided on a case by case basis, but considering the spec holy Beat that straw man! > is not doing anybody any service. Andrew.
Andrew Haley wrote: > That depends on whether or not we think the spec or some > implementation is buggy. It is often possible to tell -- some > behaviours are just Obviously Wrong. Even if something is "Obviously Wrong", it may not be a good idea to fix it because it would be a breaking change. For example: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6227069 > When it's not possible to tell, we have to make a judgment call. We obviously agree on this, but I'm trying to point out that things aren't always as "obvious" as one would think at first glance. Regards, Jeroen
On Fri, 2006-07-28 at 09:56 +0200, Jeroen Frijters wrote:
> Even if something is "Obviously Wrong", it may not be a good idea to fix
> it because it would be a breaking change. For example:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6227069
>
Obvious? Anyway, IMHO "fixing" something in a compatibility-breaking
way simply isn't a fix at all. It's the opposite.
However, this goes in the opposite direction as well. And that's when
you really need to make a judgement call. I just did so in the
BigDecimal patch I just commited (which is why I was rude enough to
cross-post it to the main list).
Which is that setScale(scale, rounding) should throw an
ArithmeticException if scale < 0. The spec says so, and the JRE
does so prior to v1.5. In 1.5 it does not (with no change to the doc).
So without thinking too hard I figured it was a 1.5 regression and
put the exception back in (fixing a mauve regression on our part).
After all, this could break compatibility, in theory.
Although now I'm not quite so sure, because the probable cause
of this is that the divide(BigDecimal, scale, rounding) method
which we (and Sun, presumably) implement this on, shows the same
behaviour, only the docs have changed.
/Sven
Jeroen> We obviously agree on this, but I'm trying to point out that things Jeroen> aren't always as "obvious" as one would think at first glance. How about distilling this wisdom into a section in the Classpath hacking guide? Tom