libgcj/1338: StreamTokenizer does not handle /* comments correctly

libgcj/1338: StreamTokenizer does not handle /* comments correctly

[osk]

>Number:         1338
>Category:       libgcj
>Synopsis:       StreamTokenizer does not handle /* comments correctly
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    warrenl
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Dec 20 12:19:10 PST 2000
>Closed-Date:    
>Last-Modified:  Wed Aug  9 12:50:01 PDT 2000
>Originator:     Oskar Liljeblad
>Release:        recent libgcj (not the latest though)
>Organization:
>Environment:

>Description:
The class java.io.StreamTokenizer doesn't handle '/*'-comments
correctly. This is because the libgcj/Classpath StreamTokenizer
checks for commentChar's first, then slashStar-comments.
The '/' character is by default set as a comment character - 
a comment that ends at EOL.

Oskar
>How-To-Repeat:
import java.io.IOException;
import java.io.StringReader;
import java.io.StreamTokenizer;

public class PRxxx {
	public static void printTokenization(String s) {
		StringReader r = new StringReader(s);
		StreamTokenizer st = new StreamTokenizer(r);
		st.slashStarComments(true);
		try {
			while (st.nextToken() != StreamTokenizer.TT_EOF) {
				System.out.print(st.ttype+"["+st.sval+","+st.nval+"] ");
			}
		} catch (IOException e) {
			System.out.println("[IOException]");
		}
		System.out.println();
	}

	public static void main(String[] args)
	throws IOException {
		printTokenization("alpha /* bleh */ gamma");
		printTokenization("alpha / bleh");
		printTokenization("alpha /* bleh");
	}
}

// Good output (note the extra space at end of lines):
// -3[alpha,0.0] -3[gamma,0.0] 
// -3[alpha,0.0] 
// -3[alpha,0.0] 
>Fix:
See the patch0 file. This patch may introduce other (subtle)
bugs so it would be nice if the original author (Warren Levy)
or someone else could verify it.
>Release-Note:

>Audit-Trail:

Formerly PR java.io/313

Responsible-Changed-From-To: tromey->warrenl
Responsible-Changed-By: tromey
Responsible-Changed-When: Wed Aug  9 12:43:59 2000
Responsible-Changed-Why:
    Warren wrote this class.
    Warren, this PR includes a patch.
    Please look at it when you have time.

From: Tom Tromey <tromey@cygnus.com>
To: osk@hem.passagen.se
Cc: java-gnats@sourceware.cygnus.com
Subject: Re: java.io/313: StreamTokenizer does not handle /* comments correctly
Date: Wed, 9 Aug 2000 12:41:30 -0700 (PDT)

 I'm putting this test case into Mauve.
 
 Thanks,
 Tom

From: tromey@cygnus.com
To: java-gnats@sourceware.cygnus.com, osk@hem.passagen.se, tromey@cygnus.com,
  warrenl@cygnus.com
Cc:  
Subject: Re: java.io/313
Date: 9 Aug 2000 19:43:59 -0000

 Synopsis: StreamTokenizer does not handle /* comments correctly
 
 Responsible-Changed-From-To: tromey->warrenl
 Responsible-Changed-By: tromey
 Responsible-Changed-When: Wed Aug  9 12:43:59 2000
 Responsible-Changed-Why:
     Warren wrote this class.
     Warren, this PR includes a patch.
     Please look at it when you have time.
 
 http://sources.redhat.com/cgi-bin/gnatsweb.pl?cmd=view&pr=313&database=java
>Unformatted:
----gnatsweb-attachment----
Content-Type: application/octet-stream; name="patch0"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="patch0"

LS0tIFN0cmVhbVRva2VuaXplci5qYXZhLm9sZAlTYXQgQXVnICA1IDA5OjMzOjU0IDIwMDAKKysr
IFN0cmVhbVRva2VuaXplci5qYXZhCVdlZCBBdWcgIDkgMjA6NTI6MjIgMjAwMApAQCAtMjg5LDYg
KzI4OSw1MCBAQAogCSAgICByZXR1cm4gKHR0eXBlID0gVFRfRU9MKTsKIAl9CiAKKyAgICBpZiAo
Y2ggPT0gJy8nKQorICAgICAgaWYgKChjaCA9IGluLnJlYWQoKSkgPT0gJy8nICYmIHNsYXNoU2xh
c2gpCisgICAgCXsKKyAgICAJICB3aGlsZSAoKGNoID0gaW4ucmVhZCgpKSAhPSAnXG4nICYmIGNo
ICE9ICdccicgJiYgY2ggIT0gVFRfRU9GKQorICAgIAkgICAgOworICAgIAkgIGlmIChjaCAhPSBU
VF9FT0YpCisgICAgCSAgICBpbi51bnJlYWQoY2gpOworICAgIAkgIHJldHVybiBuZXh0VG9rZW4o
KTsgLy8gUmVjdXJzaXZlLCBidXQgbm90IHRvbyBkZWVwIGluIG5vcm1hbCBjYXNlcworICAgIAl9
CisgICAgICBlbHNlIGlmIChjaCA9PSAnKicgJiYgc2xhc2hTdGFyKQorICAgIAl7CisgICAgCSAg
d2hpbGUgKHRydWUpCisgICAgCSAgICB7CisgICAgCSAgICAgIGNoID0gaW4ucmVhZCgpOworICAg
IAkgICAgICBpZiAoY2ggPT0gJyonKQorICAgIAkJeworICAgIAkJICBpZiAoKGNoID0gaW4ucmVh
ZCgpKSA9PSAnLycpCisgICAgCQkgICAgYnJlYWs7CisgICAgCQkgIGVsc2UgaWYgKGNoICE9IFRU
X0VPRikKKyAgICAJCSAgICBpbi51bnJlYWQoY2gpOworICAgIAkJfQorICAgIAkgICAgICBlbHNl
IGlmIChjaCA9PSAnXG4nIHx8IGNoID09ICdccicpCisgICAgCQl7CisgICAgCQkgIGxpbmVOdW1i
ZXIrKzsKKyAgICAJCSAgaWYgKGNoID09ICdccicgJiYgKGNoID0gaW4ucmVhZCgpKSAhPSAnXG4n
KQorICAgIAkJICAgIHsKKyAgICAJCSAgICAgIGlmIChjaCAhPSBUVF9FT0YpCisgICAgCQkJaW4u
dW5yZWFkKGNoKTsKKyAgICAJCSAgICB9CisgICAgCQl9CisgICAgCSAgICAgIGVsc2UgaWYgKGNo
ID09IFRUX0VPRikKKyAgICAJCXsKKyAgICAJCSAgYnJlYWs7CisgICAgCQl9CisgICAgCSAgICB9
CisgICAgCSAgcmV0dXJuIG5leHRUb2tlbigpOyAvLyBSZWN1cnNpdmUsIGJ1dCBub3QgdG9vIGRl
ZXAgaW4gbm9ybWFsIGNhc2VzCisgICAgCX0KKyAgICAgIGVsc2UKKyAgICAJeworICAgIAkgIGlm
IChjaCAhPSBUVF9FT0YpCisgICAgCSAgICBpbi51bnJlYWQoY2gpOworICAgIAkgIGNoID0gJy8n
OworICAgIAl9CisKICAgICBpZiAoY2ggPT0gVFRfRU9GKQogICAgICAgdHR5cGUgPSBUVF9FT0Y7
CiAgICAgZWxzZSBpZiAoaXNOdW1lcmljKGNoKSkKQEAgLTQwNSw1MCArNDQ5LDYgQEAKICAgICAg
IH0KICAgICBlbHNlCiAgICAgICB7Ci0gICAgICAgIGlmIChjaCA9PSAnLycpCi0JICBpZiAoKGNo
ID0gaW4ucmVhZCgpKSA9PSAnLycgJiYgc2xhc2hTbGFzaCkKLQkgICAgewotCSAgICAgIHdoaWxl
ICgoY2ggPSBpbi5yZWFkKCkpICE9ICdcbicgJiYgY2ggIT0gJ1xyJyAmJiBjaCAhPSBUVF9FT0Yp
Ci0JCTsKLQkgICAgICBpZiAoY2ggIT0gVFRfRU9GKQotCQlpbi51bnJlYWQoY2gpOwotCSAgICAg
IHJldHVybiBuZXh0VG9rZW4oKTsgLy8gUmVjdXJzaXZlLCBidXQgbm90IHRvbyBkZWVwIGluIG5v
cm1hbCBjYXNlcwotCSAgICB9Ci0JICBlbHNlIGlmIChjaCA9PSAnKicgJiYgc2xhc2hTdGFyKSAK
LQkgICAgewotCSAgICAgIHdoaWxlICh0cnVlKQotCQl7Ci0JICAgICAgICAgIGNoID0gaW4ucmVh
ZCgpOwotCQkgIGlmIChjaCA9PSAnKicpCi0JCSAgICB7Ci0JCSAgICAgIGlmICgoY2ggPSBpbi5y
ZWFkKCkpID09ICcvJykKLQkJCWJyZWFrOwotCQkgICAgICBlbHNlIGlmIChjaCAhPSBUVF9FT0Yp
Ci0JCQlpbi51bnJlYWQoY2gpOwotCQkgICAgfQotCQkgIGVsc2UgaWYgKGNoID09ICdcbicgfHwg
Y2ggPT0gJ1xyJykKLQkJICAgIHsKLQkJICAgICAgbGluZU51bWJlcisrOwotCQkgICAgICBpZiAo
Y2ggPT0gJ1xyJyAmJiAoY2ggPSBpbi5yZWFkKCkpICE9ICdcbicpCi0JCQl7Ci0JCSAgICAgICAg
ICBpZiAoY2ggIT0gVFRfRU9GKQotCQkJICAgIGluLnVucmVhZChjaCk7Ci0JCQl9Ci0JCSAgICB9
Ci0JCSAgZWxzZSBpZiAoY2ggPT0gVFRfRU9GKQotCQkgICAgewotCQkgICAgICBicmVhazsKLQkJ
ICAgIH0KLQkJfQotCSAgICAgIHJldHVybiBuZXh0VG9rZW4oKTsgLy8gUmVjdXJzaXZlLCBidXQg
bm90IHRvbyBkZWVwIGluIG5vcm1hbCBjYXNlcwotCSAgICB9Ci0JICBlbHNlCi0JICAgIHsKLQkg
ICAgICBpZiAoY2ggIT0gVFRfRU9GKQotCQlpbi51bnJlYWQoY2gpOwotCSAgICAgIGNoID0gJy8n
OwotCSAgICB9Ci0KIAl0dHlwZSA9IGNoOwogICAgICAgfQogCg==