libgcj/1198: StreamTokenizer doesn't scan mixed token types correctly

libgcj/1198: StreamTokenizer doesn't scan mixed token types correctly

[jsturm]

>Number:         1198
>Category:       libgcj
>Synopsis:       StreamTokenizer doesn't scan mixed token types correctly
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    aph
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Dec 20 12:17:26 PST 2000
>Closed-Date:    Mon Sep 04 13:35:49 PDT 2000
>Last-Modified:  Mon Sep  4 13:40:00 PDT 2000
>Originator:     Jeff Sturm
>Release:        libgcj (current)
>Organization:
>Environment:
alphapca56-unknown-linux-gnu
>Description:
While trying to scan the input below:

(a).(b)

output of StreamTokenizer (from libgcj) doesn't match that
of the JDK.
>How-To-Repeat:
Compile/run the attached program:

gcj StreamTokenizerTest.java --main=StreamTokenizerTest -o st
./st '(a).(b)'

Expected output is:

(
[TT_WORD]: a
)
[TT_NUMBER]: 0.0
(
[TT_WORD]: b
)
[TT_EOF]
>Fix:

>Release-Note:

>Audit-Trail:

Formerly PR java.io/141


From: green@sourceware.cygnus.com
To: java-gnats@sourceware.cygnus.com
Cc:  
Subject: java.io/141
Date: 6 Mar 2000 03:48:34 -0000

 CVSROOT:	/cvs/java
 Module name:	libgcj
 Changes by:	green@sourceware.cygnus.com	00/03/05 19:48:34
 
 Modified files:
 	libjava/testsuite: ChangeLog 
 Added files:
 	libjava/testsuite/libjava.lang: PR141.java PR141.out 
 
 Log message:
 	2000-03-05  Anthony Green  <green@redhat.com>
 	
 	* libjava.lang/PR141.java: New file.
 	* libjava.lang/PR141.out: New file.
 	For PR java.io/141.
 
 Patches:
 http://gcc.gnu.org/cgi-bin/gcc/libjava/testsuite/ChangeLog.diff?cvsroot=java&r1=1.58&r2=1.59
 http://gcc.gnu.org/cgi-bin/gcc/libjava/testsuite/libjava.lang/PR141.java.diff?cvsroot=java&r1=NONE&r2=1.1
 http://gcc.gnu.org/cgi-bin/gcc/libjava/testsuite/libjava.lang/PR141.out.diff?cvsroot=java&r1=NONE&r2=1.1
 

From: Tom Tromey <tromey@cygnus.com>
To: Java Gnats Server <java-gnats@sourceware.cygnus.com>
Cc:  
Subject: java.io/141
Date: Wed, 9 Aug 2000 17:26:20 -0700 (PDT)

 I checked in a Mauve test case for this bug.
 Tom

From: Tom Tromey <tromey@cygnus.com>
To: Jeff Sturm <jsturm@sigma6.com>
Cc: Java Gnats Server <java-gnats@sourceware.cygnus.com>,
        Warren Levy <warrenl@cygnus.com>
Subject: java.io/141
Date: 09 Aug 2000 18:56:11 -0600

 I looked at this PR a bit more.
 
 I believe the JDK is wrong.
 If you follow the JLS definition of a floating point constant, `.' is
 not one.
 
 Alternatively, one could argue that compatibility is important here,
 and that the StreamTokenizer definition of a number is different from
 the JLS definition.
 
 I personally think that the JDK is wrong, though.  Warren, what do you
 think?  I read the JCL book on this subject but it isn't very helpful
 either.
 
 Tom

From: Jeff Sturm <jsturm@detroit.appnet.com>
To: tromey@cygnus.com
Cc: Java Gnats Server <java-gnats@sourceware.cygnus.com>,
        Warren Levy <warrenl@cygnus.com>
Subject: Re: java.io/141
Date: Fri, 11 Aug 2000 12:19:21 -0400

 Tom Tromey wrote:
 > I believe the JDK is wrong.
 > If you follow the JLS definition of a floating point constant, `.' is
 > not one.
 
 That sounds right.
 
 > Alternatively, one could argue that compatibility is important here,
 > and that the StreamTokenizer definition of a number is different from
 > the JLS definition.
 
 Since the JCL doesn't really say how `.' should be scanned, you can make an
 argument either way.  (I don't really want `.' to be scanned as a number, but I
 can live with it.)
 
 IIRC an exception was getting thrown when I tried to scan some source code with
 StreamTokenizer.  That made StreamTokenizer useless for me.  I don't think
 nextToken() should ever throw an exception other than IOException.  Perhaps I
 should have stated that in the bug report.
 
 
 --
 Jeff Sturm
 jeff.sturm@appnet.com
Responsible-Changed-From-To: tromey->warrenl
Responsible-Changed-By: tromey
Responsible-Changed-When: Sat Aug 19 18:25:20 2000
Responsible-Changed-Why:
    Warren, this is yours.
    Currently StreamTokenizer throws NumberFormatException
    for `.', but I think it shouldn't.

From: tromey@cygnus.com
To: java-gnats@sourceware.cygnus.com, jsturm@sigma6.com, tromey@cygnus.com,
  warrenl@cygnus.com
Cc:  
Subject: Re: java.io/141
Date: 20 Aug 2000 01:25:20 -0000

 Synopsis: StreamTokenizer doesn't scan mixed token types correctly
 
 Responsible-Changed-From-To: tromey->warrenl
 Responsible-Changed-By: tromey
 Responsible-Changed-When: Sat Aug 19 18:25:20 2000
 Responsible-Changed-Why:
     Warren, this is yours.
     Currently StreamTokenizer throws NumberFormatException
     for `.', but I think it shouldn't.
 
 http://sources.redhat.com/cgi-bin/gnatsweb.pl?cmd=view&pr=141&database=java
Responsible-Changed-From-To: warrenl->aph
Responsible-Changed-By: tromey
Responsible-Changed-When: Mon Sep  4 13:35:49 2000
Responsible-Changed-Why:
    Andrew fixed it.
State-Changed-From-To: open->closed
State-Changed-By: tromey
State-Changed-When: Mon Sep  4 13:35:49 2000
State-Changed-Why:
    Andrew checked in a patch for this bug.

From: tromey@cygnus.com
To: aph@cygnus.com, java-gnats@sourceware.cygnus.com, jsturm@sigma6.com,
  warrenl@cygnus.com
Cc:  
Subject: Re: java.io/141
Date: 4 Sep 2000 20:35:49 -0000

 Synopsis: StreamTokenizer doesn't scan mixed token types correctly
 
 Responsible-Changed-From-To: warrenl->aph
 Responsible-Changed-By: tromey
 Responsible-Changed-When: Mon Sep  4 13:35:49 2000
 Responsible-Changed-Why:
     Andrew fixed it.
 State-Changed-From-To: open->closed
 State-Changed-By: tromey
 State-Changed-When: Mon Sep  4 13:35:49 2000
 State-Changed-Why:
     Andrew checked in a patch for this bug.
 
 http://sources.redhat.com/cgi-bin/gnatsweb.pl?cmd=view&pr=141&database=java
>Unformatted:
----gnatsweb-attachment----
Content-Type: application/octet-stream; name="StreamTokenizerTest.java"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="StreamTokenizerTest.java"

aW1wb3J0IGphdmEuaW8uSU9FeGNlcHRpb247Ci8vaW1wb3J0IGphdmEuaW8uU3RyZWFtVG9rZW5p
emVyOwppbXBvcnQgamF2YS5pby5TdHJpbmdSZWFkZXI7CgpwdWJsaWMgY2xhc3MgU3RyZWFtVG9r
ZW5pemVyVGVzdCB7CglwdWJsaWMgc3RhdGljIHZvaWQgdGVzdChTdHJpbmcgc3RyKSB0aHJvd3Mg
SU9FeGNlcHRpb24gewoJCVN0cmluZ1JlYWRlciByZWFkZXIgPSBuZXcgU3RyaW5nUmVhZGVyKHN0
cik7CgkJU3RyZWFtVG9rZW5pemVyIHN0ID0gbmV3IFN0cmVhbVRva2VuaXplcihyZWFkZXIpOwoJ
CWludCB0ID0gMDsKCgkJZG8gewoJCQl0ID0gc3QubmV4dFRva2VuKCk7CgoJCQlzd2l0Y2ggKHQp
IHsKCQkJCWNhc2UgU3RyZWFtVG9rZW5pemVyLlRUX1dPUkQ6CgkJCQkJU3lzdGVtLm91dC5wcmlu
dGxuKCJbVFRfV09SRF06ICIgKyBzdC5zdmFsKTsKCQkJCQlicmVhazsKCQkJCWNhc2UgU3RyZWFt
VG9rZW5pemVyLlRUX05VTUJFUjoKCQkJCQlTeXN0ZW0ub3V0LnByaW50bG4oIltUVF9OVU1CRVJd
OiAiICsgc3QubnZhbCk7CgkJCQkJYnJlYWs7CgkJCQljYXNlIFN0cmVhbVRva2VuaXplci5UVF9F
T0w6CgkJCQkJU3lzdGVtLm91dC5wcmludGxuKCJbVFRfRU9MXSIpOwoJCQkJCWJyZWFrOwoJCQkJ
Y2FzZSBTdHJlYW1Ub2tlbml6ZXIuVFRfRU9GOgoJCQkJCVN5c3RlbS5vdXQucHJpbnRsbigiW1RU
X0VPRl0iKTsKCQkJCQlicmVhazsKCQkJCWRlZmF1bHQ6CgkJCQkJU3lzdGVtLm91dC5wcmludGxu
KChjaGFyKXQpOwoJCQkJCWJyZWFrOwoJCQl9CgkJfSB3aGlsZSAodCAhPSBTdHJlYW1Ub2tlbml6
ZXIuVFRfRU9GKTsKCX0KCglwdWJsaWMgc3RhdGljIHZvaWQgbWFpbihTdHJpbmdbXSBhcmdzKSB7
CgkJZm9yIChpbnQgbiA9IDA7IG4gPCBhcmdzLmxlbmd0aDsgbisrKSB7CgkJCXRyeSB7CgkJCQl0
ZXN0KGFyZ3Nbbl0pOwoJCQl9IGNhdGNoIChUaHJvd2FibGUgdCkgewoJCQkJdC5wcmludFN0YWNr
VHJhY2UoKTsKCQkJfQoJCX0KCX0KfQo=