[Bug libc/12830] New: ISO-2022-JP-2 maps C1 control characters incorrectly

[Bug libc/12830] New: ISO-2022-JP-2 maps C1 control characters incorrectly

[glibcbugz at ghalkes dot nl]

http://sourceware.org/bugzilla/show_bug.cgi?id=12830

           Summary: ISO-2022-JP-2 maps C1 control characters incorrectly
           Product: glibc
           Version: 2.13
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: drepper.fsp@gmail.com
        ReportedBy: glibcbugz@ghalkes.nl


In the ISO-2022-JP-2 converter, the C1 control codes (U0080-U009F) are encoded
as 1B 2E 41 1B 4E [00 - 1F] (i.e., load ISO-8859-1 in the G2 graphics set, use
single shift to set G2 and encode the byte [00 - 1F]). However, if I understand
the standard correctly, switching to the G2 set _only_ changes the mapping of
the 96 characters in the range 20-7F (or the 94 charaacters in the range 21-7E
if a smaller set is used). The control characters are unaffected. To access the
C1 control set, one should use 1B [40 - 5F]. This is actually done for the
encoding of the "single shift 2" control (U+008E) in the sequence above, which
is encoded as 1B 4E.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12830] ISO-2022-JP-2 maps C1 control characters incorrectly

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12830

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING

--- Comment #1 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-09-09 19:49:24 UTC ---
Provide an self-contained test case and a reference to the specification which
you base your opinion on.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12830] ISO-2022-JP-2 maps C1 control characters incorrectly

[glibcbugz at ghalkes dot nl]

http://sourceware.org/bugzilla/show_bug.cgi?id=12830

--- Comment #2 from G. Halkes <glibcbugz at ghalkes dot nl> 2011-09-20 13:20:34 UTC ---
Testcase: in bash, using GNU libc iconv (converts U+0081 from C1):

echo -e -n '\x00\x81' | iconv -f UTF-16BE -t ISO-2022-JP-2 | od -t x1

result:

0000000 1b 2e 41 1b 4e 01
0000006

expected result:

0000000 1b 41
0000002

The standard I base my opinion on is ECMA-35, which can be found at
http://www.ecma-international.org/publications/standards/Ecma-035.htm and
which, according to ECMA itself, is "fully identical with International
Standard ISO/IEC 2022:1994". However, the ECMA-35 specification is freely
available, contrary to the ISO-2022 spec.

Specifically, section 9 discusses the structure of 7-bit codes, such as
ISO-2022-JP-2. It references section 7.2, which discusses the definitions of G0
through G3 and C0 and C1. In the specification of graphics sets G0 - G3, it
notes that it uses "column numbers" 02 through 07, i.e. has values between 0x20
and 0x7f. For C1 codes, it defines that they use column numbers 08 and 09, or
ESC Fe. The meaning of the Fe is explained in section 6.4.3 and 13.2, and
basically means a byte in the range 0x40 - 0x5f.

In my reading of the standard, changing GL to one of G0 through G3, using any
of the shift mechanisms, has no impact on the control codes in CL (the range
0x00 through 0x1f). Therefore, the generated sequence is incorrect, and is
essentially equal to the sequence "01".

Because columns 08 and 09 are not used in a 7 bit code such as ISO-2022-JP-2,
it has to use the ESC Fe construct for representing C1 control codes. Thus the
correct sequence would be "1b 41". This actually corresponds to how the control
character SS2 (U+008E) from C1 is already encoded in the example (i.e. "1b
4e"). See also Figure 8 on page 22, for a graphical representation of the
structure of a 7 bit code.

I hope this sufficiently clarifies my previous report.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12830] ISO-2022-JP-2 maps C1 control characters incorrectly

[aj at suse dot de]

http://sourceware.org/bugzilla/show_bug.cgi?id=12830

Andreas Jaeger <aj at suse dot de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW
                 CC|                            |aj at suse dot de

--- Comment #3 from Andreas Jaeger <aj at suse dot de> 2012-01-04 20:02:18 UTC ---
Information was provided, resetting state.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12830] ISO-2022-JP-2 maps C1 control characters incorrectly

[schwab@linux-m68k.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12830

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|drepper.fsp at gmail dot    |unassigned at sourceware
                   |com                         |dot org

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug localedata/12830] ISO-2022-JP-2 maps C1 control characters incorrectly

[jsm28 at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=12830

Joseph Myers <jsm28 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |libc-locales at sourceware dot org
          Component|libc                        |localedata

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/12830] ISO-2022-JP-2 maps C1 control characters incorrectly

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12830

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.