Re: [PATCH] Allow non-ASCII characters in Rust identifiers

[Tom Tromey <tom@tromey.com>]

Andrew> So I put this into a text file 'unicode.tcl':
Andrew>   puts "print 𝕯"
Andrew> (just in case that gets mangled in transit, that's the same unicode
Andrew> character as is used in the gdb.rust/unicode.exp test)

Andrew> I'm currently running tcl 8.6.  I wonder if you could compare this to
Andrew> the behaviour of your tclsh.

This works for me.  I'm using the system tclsh on Fedora 34.
tclsh doesn't seem to support --version, but:

prentzel. rpm -q tcl
tcl-8.6.10-5.fc34.x86_64

Andrew> Compared to the original, the first '0xf0' changes to '0xc3 0xb0', while
Andrew> all the subequent bytes get a 0xc2 byte before them.

Unicode U+00f0 is represented as 0xc3 0xb0 in UTF-8.  So one idea is if
tclsh thinks the input is Latin-1, where the code points generally map
identically to Unicode code points, then this conversion would be done
if converting from the file encoding to UTF-8.

That is, tclsh reads 0xf0, but thinking it is reading a Latin-1
character, converts that to the corresponding Uncode character U+00f0,
and from there to the bytes that are seen.

I have LANG=en_US.UTF-8, which may explain why my default encoding is
UTF-8.  Perhaps this setting is the problem - if you are running in a
Latin-1 locale then the .exp files will be recoded incorrectly as they
are read by the interpreter.

I don't know if there's a way to set the file encoding in a way that Tcl
will recognize.  We could maybe try a UTF-8 BOM.

Tom