internationalized text processing

["Nellis, Kenneth" <Kenneth.Nellis@acs-inc.com>]

Never posted here, but somewhat off-topic for the general Cygwin list...

I've been frustrated at the lack of guidance I've found online regarding 
how to build an internationalized text-based Unix application, in particular 
with reference to handling different character sets. For example, "man"
recognizes the character set specified in locale settings and uses Unicode 
punctuation appropriately. OTOH, the cygutils "ascii -e" utility does not 
recognize that my LANG specifies UTF-8 and outputs garbage for the 
extended half. Should this be considered a bug?

Closer to a real application, how should a program that reads, processes, 
and outputs text detect the character set so that it can process the text 
correctly? In the absence of a BOM in the input file, should the program 
blindly presume the locale-specified character set or analyze the file to 
guess it? And what advice is there for guessing? And, once detected, what 
takes precedence for output: the input file's character set or the locale 
setting?

I presume these issues are either settled or being debated somewhere.

--Ken Nellis