[Bug locale/31030] New: ERA segments are nul-separated rather than semicolon-separated

[Bug locale/31030] New: ERA segments are nul-separated rather than semicolon-separated

[bugzilla at tecnocode dot co.uk]

https://sourceware.org/bugzilla/show_bug.cgi?id=31030

            Bug ID: 31030
           Summary: ERA segments are nul-separated rather than
                    semicolon-separated
           Product: glibc
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: locale
          Assignee: unassigned at sourceware dot org
          Reporter: bugzilla at tecnocode dot co.uk
  Target Milestone: ---

nl_langinfo(3) says “Era description segments are separated by semicolons”, but
it appears that `nl_langinfo (ERA)` returns them separated by nul bytes
instead, and the only way to parse them is to additionally call the
(undocumented) `nl_langinfo (_NL_TIME_ERA_NUM_ENTRIES)` to find out how many
segments there are meant to be.

This can be demonstrated with the following example program (compile with `gcc
-o era era.c -Wall`):
```
#include <langinfo.h>
#include <locale.h>
#include <stdint.h>
#include <stdio.h>

int
main (void)
{
  setlocale (LC_ALL, "");
  const char *era = nl_langinfo (ERA);
  int n_entries = (int) (intptr_t) nl_langinfo (_NL_TIME_ERA_NUM_ENTRIES);

  printf ("n_entries: %d, era: %s\n", n_entries, era);

  return 0;
}
```

If run with `LANG=th_TH.UTF-8 ./era` it prints
```
n_entries: 1, era: +:1:-543/01/01:+*:พ.ศ.:%EC %Ey
```
which is all good.

However, if run with `LANG=ja_JP.UTF-8 ./era`, it prints:
```
n_entries: 11, era: +:2:2020/01/01:+*:令和:%EC%Ey年
```

There clearly aren’t 11 segments in the ’era’ description there — only one.
Looking at the ja_JP locale definition, there are correctly 11 segments defined
in it:
https://github.com/bminor/glibc/blob/master/localedata/locales/ja_JP#L14949-L14977

If I read past 10 nul terminators in the `era` string, I can retrieve all 11
segments. So the locale definition does work. It just doesn’t match up to what
nl_langinfo(3) says, and requires using the undocumented
`_NL_TIME_ERA_NUM_ENTRIES` to read all segments.

Is there a reason `era` is using nul separators? Could it be switched to using
semicolons please? :)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/31030] ERA segments are nul-separated rather than semicolon-separated

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31030

--- Comment #1 from Andreas Schwab <schwab@linux-m68k.org> ---
The ERA entry is actually terminated by a double null, and
_NL_TIME_ERA_NUM_ENTRIES is only related to _NL_TIME_ERA_ENTRIES (which
contains a structured representation of ERA and what is used internally
instead).

It is not clear whether changing the format of ERA would constitute an ABI
break.  The ALT_DIGITS entry has the same issue.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/31030] ERA segments are nul-separated rather than semicolon-separated

[smcv at collabora dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31030

Simon McVittie <smcv at collabora dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |smcv at collabora dot com

--- Comment #2 from Simon McVittie <smcv at collabora dot com> ---
GLib currently uses (intptr_t) nl_langinfo(_NL_TIME_ERA_NUM_ENTRIES) in its
workaround for the ERA not being as documented, but it seems that there is a
separate problem with that on 64-bit big-endian architectures:
https://gitlab.gnome.org/GNOME/glib/-/issues/3225

On x86_64, in the Japanese (ja_JP.utf8) locale, (intptr_t)
nl_langinfo(_NL_TIME_ERA_NUM_ENTRIES) is 0x0000'0000'0000'000b (decimal 11)
which seems to be what is expected.

But on s390x, the result I see is 0x0000'000b'0000'0000 (a very large number):
it looks as though the two 32-bit words have been swapped? There is a test
program attached to the GLib issue which might be useful.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/31030] ERA segments are nul-separated rather than semicolon-separated

[smcv at collabora dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31030

--- Comment #3 from Simon McVittie <smcv at collabora dot com> ---
(In reply to Andreas Schwab from comment #1)
> The ERA entry is actually terminated by a double null

Is that an API guarantee that glibc users can rely on, or an implementation
detail?

My concern is that if we rely on a double NUL, and then a future glibc version
changes to the documented behaviour with semicolon delimiters and single NUL
termination, we will be reading beyond the valid bounds of the returned string
and probably crash.

If glibc is willing to document/guarantee that the returned pointer is to
memory delimited by semicolons and/or single NULs, terminated by a double NUL,
then we could do something like this pseudocode:

    era = nl_langinfo (ERA);

    #ifdef __GLIBC__
    find length by searching for double NUL;
    era_copy = malloc (length);
    memcpy (era_copy, era, length);
    replace internal NULs with semicolons in era_copy;
    #else
    era_copy = strdup (era);
    #endif

The cost of doing that is that if glibc ever changes to returning the
documented format "segment;...;segment\0", it would have to use a slightly
longer buffer with a double NUL at the end, "segment;...;segment\0\0", to keep
that guarantee true.

(Obviously in real life the segments are more complicated than the word
"segment" but I'm sure you see what I mean.)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/31030] ERA segments are nul-separated rather than semicolon-separated

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31030

--- Comment #4 from Andreas Schwab <schwab@linux-m68k.org> ---
You cannot use _NL_TIME_ERA_NUM_ENTRIES like this since it is not a
pointer-sized entry.  Like all _NL_* definitions it is internal-only, not for
use with nl_langinfo.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/31030] ERA segments are nul-separated rather than semicolon-separated

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31030

--- Comment #5 from Andreas Schwab <schwab@linux-m68k.org> ---
Like I said, it is an ABI problem.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/31030] ERA segments are nul-separated rather than semicolon-separated

[bugzilla at tecnocode dot co.uk]

https://sourceware.org/bugzilla/show_bug.cgi?id=31030

--- Comment #6 from Philip Withnall <bugzilla at tecnocode dot co.uk> ---
Do you have any recommendations for how `nl_langinfo (ERA)` is meant to be used
with glibc in a POSIX-compliant way?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/31030] ERA segments are nul-separated rather than semicolon-separated

[aurelien at aurel32 dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=31030

Aurelien Jarno <aurelien at aurel32 dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |aurelien at aurel32 dot net

-- 
You are receiving this mail because:
You are on the CC list for the bug.