Re: Perl Unidecode modules - which to use (if not Text::Unidecode)?

[L A Walsh <cygwin@tlinx.org>]

On 2021/04/04 14:26, Joel Rees via Cygwin wrote:
>
>> 1. What perl Unicode modules should I consider, if not Text::Unidecode?
>> The present need
>> is to be able to convert those few "foreign" characters (like
>> ÇĆĈĊçĉċĜĞĠĢĝģğġËÌÍÎÏÒÓÔÕ)
>> that are basically ASCII with accent marks to their closest ASCII
>> equivalents, but I'd
>> like to do more with Unicode in the future, without going down any
>> dead-ends as far as
>> being able to run under cygwin is concerned.
>>
>>     
>
> "Stripping those few foreign accent characters" is probably not really what
> you want to do.
>   
----
    Why not?  You don't know his use case and you are misinterpreting his
example as random garbage.

Those aren't a random foreign encoding -- those are C's G's then E, I O
with accent variations that he may want to collapse for purposes of storing
in a text storage and retrieval (search) application.  They are all well
formed/well-coded UTF-8 characters -- they are not some 8-bit encoding
that was remangled during a no-recoding display of them in a UTF-8
context.

I didn't know about Text::Unidecode -- but it specifically to create
Latinized alternatives to foreign characters.  That was another hint
that it wasn't a random mistake.  The manpage for it says:

       It often happens that you have non-Roman text data in Unicode, 
but you
       can't display it -- usually because you're trying to show it to a 
user
       via an application that doesn't support Unicode, or because the fonts
       you need aren't accessible.  You could represent the Unicode 
characters
       as "???????" or "\15BA\15A0\1610...", but that's nearly useless 
to the
       user who actually wants to read what the text says.

An example was like:

tperl
use utf8;
use Text::Unidecode;
my $name="\x{5317}\x{4EB0}";

printf "name, %s == %s\n", $name, unidecode($name);
'
name, 北亰 == Bei Jing

It's not just about removing accents but getting an English
like translation based on the foreign text.






All of the characters he used as example were well coded utf-8
characters --



> Those "accent characters" are misinterpreted foreign encoding (likely not
> to be Unicode) characters. Simply "stripping" the "accent characters" will
> basically convert them to truly meaningless junk. I suppose the meaningless
> junk can then be interpreted by the reader as "used to be a be a foreign
> word here", but why bother contributing further to information entropy?
>
> 2. I see some talk of Internationalization in Chapter 2 of "Setting up
>   
>> Cygwin", but
>> cannot see anything relating to perl modules, and I don't see any easy way
>> to search many
>> months of the mailing list for a keyword... is there any information I
>> should know about?
>>     
>
>
> Have you read the perldoc on internationalization?
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
>
>