problems with "InputStreamReader.read" tests in "java.io.Utf8Encoding"

problems with "InputStreamReader.read" tests in "java.io.Utf8Encoding"

[Steve Murry]

Can someone check the 'negative' testcases within
java.io.Utf8Encoding.mojo to see if these are valid tests or not?
Specifically, I'm referring to the 9 testcases with data values that are declared 'test5_bytes' through 'test13_bytes'.  The testcases expect a
CharEncodingException when decoding illegal UTF-8 byte strings.  In some
cases the UTF-8 data is incorrect and in others it represents codepoints
that have not yet been assigned (at least for Unicode 3.2).  As I read the
Sun API description for InputStreamReader.read, I would expect either
MalformedInputException or UTFDataFormatException to be thrown instead (the API description doesn't seem very precise in this area).  In fact, most of the platforms that we have run these testcases against do not throw any type of exception at all!  Only the IBM JREs throw the expected CharConversionException.   We also found the following paragraph in one of the Sun bug descriptions:


>This is a bug in the tests. The specification of java.io.InputStreamReader
>does not require that an implementation throw IOExceptions on malformed
>input when decoding bytes in the UTF-8 charset. That our implementation has
>done this historically is a bug that was fixed as part of 4503732.

I'm not sure I agree with this statement myself (BTW - I believe they are referring to one of their own internal tests, not Mauve), but I'm just trying to get Mauve's take on all of this.  I would appreciate anyone's thoughts.

Thanks,

Steve Murry
SAS Institute

Re: problems with "InputStreamReader.read" tests in "java.io.Utf8Encoding"

[Brian Jones]

Steve Murry <stmurr@unx.sas.com> writes:

> (I'm reposting my earlier email due to format problems. Sorry 'bout that) 
> 
> 
> Can someone check the 'negative' testcases within
> java.io.Utf8Encoding.mojo to see if these are valid tests or not?
> Specifically, I'm referring to the 9 testcases with data values that
> are declared 'test5_bytes' through 'test13_bytes'.  The testcases
> expect a CharEncodingException when decoding illegal UTF-8 byte
> strings.  In some cases the UTF-8 data is incorrect and in others it
> represents codepoints that have not yet been assigned (at least for
> Unicode 3.2).  As I read the Sun API description for
> InputStreamReader.read, I would expect either
> MalformedInputException or UTFDataFormatException to be thrown
> instead (the API description doesn't seem very precise in this
> area).  In fact, most of the platforms that we have run these
> testcases against do not throw any type of exception at all!  Only
> the IBM JREs throw the expected CharConversionException.  We also
> found the following paragraph in one of the Sun bug descriptions:
> 
> >This is a bug in the tests. The specification of
> >java.io.InputStreamReader does not require that an implementation
> >throw IOExceptions on malformed input when decoding bytes in the
> >UTF-8 charset. That our implementation has done this historically
> >is a bug that was fixed as part of 4503732.
> 
> I'm not sure I agree with this statement myself (BTW - I believe
> they are referring to one of their own internal tests, not Mauve),
> but I'm just trying to get Mauve's take on all of this.  I would
> appreciate anyone's thoughts.

I'm not a Character/UTF-8/Unicode expert so I don't think I can answer
your question.

Brian
-- 
Brian Jones <cbj@gnu.org>

problems with "InputStreamReader.read" tests in "java.io.Utf8Encoding"

[Steve Murry]

(I'm reposting my earlier email due to format problems. Sorry 'bout that) 


Can someone check the 'negative' testcases within java.io.Utf8Encoding.mojo 
to see if these are valid tests or not? Specifically, I'm referring to the 
9 testcases with data values that are declared 'test5_bytes' through 
'test13_bytes'.  The testcases expect a CharEncodingException when decoding 
illegal UTF-8 byte strings.  In some cases the UTF-8 data is incorrect and 
in others it represents codepoints that have not yet been assigned (at least 
for Unicode 3.2).  As I read the Sun API description for InputStreamReader.read, 
I would expect either MalformedInputException or UTFDataFormatException to 
be thrown instead (the API description doesn't seem very precise in this area). 
In fact, most of the platforms that we have run these testcases against do not 
throw any type of exception at all!  Only the IBM JREs throw the expected 
CharConversionException.   We also found the following paragraph in one 
of the Sun bug descriptions: 

>This is a bug in the tests. The specification of 
>java.io.InputStreamReader does not require that an implementation throw 
>IOExceptions on malformed input when decoding bytes in the UTF-8 
>charset. That our implementation has done this historically is a bug 
>that was fixed as part of 4503732. 

I'm not sure I agree with this statement myself (BTW - I believe they are 
referring to one of their own internal tests, not Mauve), but I'm just 
trying to get Mauve's take on all of this.  I would appreciate anyone's thoughts. 

Thanks, 

Steve Murry 
SAS Institute