internationalized text processing

internationalized text processing

[Nellis, Kenneth]

Never posted here, but somewhat off-topic for the general Cygwin list...

I've been frustrated at the lack of guidance I've found online regarding 
how to build an internationalized text-based Unix application, in particular 
with reference to handling different character sets. For example, "man"
recognizes the character set specified in locale settings and uses Unicode 
punctuation appropriately. OTOH, the cygutils "ascii -e" utility does not 
recognize that my LANG specifies UTF-8 and outputs garbage for the 
extended half. Should this be considered a bug?

Closer to a real application, how should a program that reads, processes, 
and outputs text detect the character set so that it can process the text 
correctly? In the absence of a BOM in the input file, should the program 
blindly presume the locale-specified character set or analyze the file to 
guess it? And what advice is there for guessing? And, once detected, what 
takes precedence for output: the input file's character set or the locale 
setting?

I presume these issues are either settled or being debated somewhere.

--Ken Nellis

Re: internationalized text processing

[Charles Wilson]

On 5/23/2011 9:44 AM, Nellis, Kenneth wrote:
> OTOH, the cygutils "ascii -e" utility does not 
> recognize that my LANG specifies UTF-8 and outputs garbage for the 
> extended half. Should this be considered a bug?

No, ascii is deliberately intended to be stupid.  In fact, the -e option
itself is simply a workaround.  Originally, ascii always printed the
first 256 codepoints unconditionally.  With the change in 1.7 to
defaulting to UTF-8, we recognized that this was bad, and changed
ascii's default behavior to print only the first 128 codepoints, and
added -e to restore the original behavior.

A bit of history: the reason ascii was originally written was as a
diagnostic tool so that folks could check whether their font settings
and (old, obsolete, do not do this anymore: CYGWIN var contained
charset:oem) var settings were correct, so that the DOS line-drawing
characters could be used in a bash shell (running in a cmd box).

E.g. poor man's hack to get CP437 working

This predated "real" codepage and $LANG handling, in cygwin-1.7.

Now, with "real" $LANG handling, line draw stuff Just Works(tm) when
LANG=*.UTF-8, at least for ncurses programs: try
	/usr/lib/ncurses/test/lrtest.exe
from the ncurses-demo package, in a bash shell running in a cmd box (or
pstree -G).  So, the need for ascii (-e) as a diagnostic tool is
kinda...not needed anymore.

Thus, I'm not too fussed about this "bug" in an obsolete and no longer
needed diag tool -- but I also don't see a need to remove it from
cygutils. So...mark this "bug" as either NOTABUG or WONTFIX. :-)

With regards to your other questions...I dunno.  Maybe somebody else does.

--
Chuck

RE: internationalized text processing

[Buchbinder, Barry (NIH/NIAID) [E]]

Charles Wilson sent the following at Monday, May 23, 2011 10:51 AM
>Thus, I'm not too fussed about this "bug" in an obsolete and no longer
>needed diag tool -- but I also don't see a need to remove it from
>cygutils. So...mark this "bug" as either NOTABUG or WONTFIX. :-)

Please keep ascii.  I use it to find how various characters might be
coded (most commonly when dealing with a URL).  And there are enough
times that I'm interested in the upper characters that I've now
aliased ascii='ascii -e'.

Thanks,

- Barry
  Disclaimer: Statements made herein are not made on behalf of NIAID.