Variable length date strings in glibc locales?

Variable length date strings in glibc locales?

[Marko Myllynen]

Hi,

in some languages dates are written without leading zeroes so that May 3
would be "3.5.". The same for time, 08:07:00 would be "8.07.00".

In glibc locales it would be possible to write dates and times in such
fashion but do we know how that would affect existing applications? Are
they expecting dates and times to be fixed length and would variable
length date strings cause formatting or layout issues? Looking at
existing locales, almost all of them use fixed length strings for
d_fmt/t_fmt/date_fmt/d_t_fmt.

Ideally of course it would be nice to change certain locales to use date
and time formats according to their cultural conventions and national
recommendations but if that would lead to wonky layout in applications
then it's probably better to be pragmatic and use fixed length dates.

I could add few words about this to our Locales wiki page if someone
happens to know what's the best approach here.

Thanks,

-- 
Marko Myllynen

Re: Variable length date strings in glibc locales?

[Carlos O'Donell]

On 05/27/2014 02:58 AM, Marko Myllynen wrote:
> Hi,
> 
> in some languages dates are written without leading zeroes so that May 3
> would be "3.5.". The same for time, 08:07:00 would be "8.07.00".
> 
> In glibc locales it would be possible to write dates and times in such
> fashion but do we know how that would affect existing applications? Are
> they expecting dates and times to be fixed length and would variable
> length date strings cause formatting or layout issues? Looking at
> existing locales, almost all of them use fixed length strings for
> d_fmt/t_fmt/date_fmt/d_t_fmt.
> 
> Ideally of course it would be nice to change certain locales to use date
> and time formats according to their cultural conventions and national
> recommendations but if that would lead to wonky layout in applications
> then it's probably better to be pragmatic and use fixed length dates.
> 
> I could add few words about this to our Locales wiki page if someone
> happens to know what's the best approach here.

I know of no guarantees given about constant length date string.

Therefore I believe that applications will have to put up with
variable length dates if that is what the locale specifies.

The guiding principle is that we want to represent dates as
expected by the native speaker. If the application wants a constant
length they will need to arrange that by breaking up the string
and spacing it out themselves?

Cheers,
Carlos.

Re: Variable length date strings in glibc locales?

[Marko Myllynen]

Hi,

On 2014-05-27 16:37, Carlos O'Donell wrote:
> On 05/27/2014 02:58 AM, Marko Myllynen wrote:
>>
>> in some languages dates are written without leading zeroes so that May 3
>> would be "3.5.". The same for time, 08:07:00 would be "8.07.00".
>>
>> In glibc locales it would be possible to write dates and times in such
>> fashion but do we know how that would affect existing applications? Are
>> they expecting dates and times to be fixed length and would variable
>> length date strings cause formatting or layout issues? Looking at
>> existing locales, almost all of them use fixed length strings for
>> d_fmt/t_fmt/date_fmt/d_t_fmt.
>>
>> Ideally of course it would be nice to change certain locales to use date
>> and time formats according to their cultural conventions and national
>> recommendations but if that would lead to wonky layout in applications
>> then it's probably better to be pragmatic and use fixed length dates.
>>
>> I could add few words about this to our Locales wiki page if someone
>> happens to know what's the best approach here.
> 
> I know of no guarantees given about constant length date string.
> 
> Therefore I believe that applications will have to put up with
> variable length dates if that is what the locale specifies.
> 
> The guiding principle is that we want to represent dates as
> expected by the native speaker. If the application wants a constant
> length they will need to arrange that by breaking up the string
> and spacing it out themselves?

indeed that seems to be a very reasonable expectation, a quick check
with the current locales shows rather great variation, few leading
zeroes omitted in a locale wouldn't make a difference at all:

localhost:~> cat t.sh
#!/bin/bash

for f in date_fmt d_t_fmt d_fmt t_fmt ; do
  echo $f:
  for l in $(ls -1 /usr/share/i18n/locales/* | grep -Ev
'(@|i18n$|iso14651|translit|POSIX)') ; do
    echo -n "$(LC_ALL=$(basename $l.UTF-8) date --date="2007-05-03
08:07:00" +"$(LC_ALL=$(basename $l.UTF-8) locale $f)" | wc -L)" ; echo
-e "\t$(basename $l).UTF-8"
  done | sort -un | sed -n '1p;$p'
done
localhost:~> unset LC_ALL
localhost:~> bash ./t.sh 2>/dev/null
date_fmt:
21      ku_TR.UTF-8
55      so_ET.UTF-8
d_t_fmt:
19      tk_TM.UTF-8
56      km_KH.UTF-8
d_fmt:
5       or_IN.UTF-8
25      mt_MT.UTF-8
t_fmt:
8       aa_DJ.UTF-8
20      bo_CN.UTF-8
localhost:~> l=ku_TR.UTF-8
localhost:~> LC_ALL=$(basename $l) date --date="2007-05-03 08:07:00"
+"$(LC_ALL=$(basename $l) locale date_fmt)"
pêncsêm 03 Gulan 2007
localhost:~> l=en_US.UTF-8
localhost:~> LC_ALL=$(basename $l) date --date="2007-05-03 08:07:00"
+"$(LC_ALL=$(basename $l) locale date_fmt)"
Thu May  3 08:07:00 EEST 2007
localhost:~> l=so_ET.UTF-8
localhost:~> LC_ALL=$(basename $l) date --date="2007-05-03 08:07:00"
+"$(LC_ALL=$(basename $l) locale date_fmt)"
Khamiis, Bisha Shanaad  3,  8:07:00 subaxnimo EEST 2007
localhost:~>

So I think I'll add a note to the wiki page that although in few places
the resulting string is guaranteed to be of the same size in all locales
(like int_curr_symbol), in many cases the resulting string can wary in
length quite considerably.

Thanks,

-- 
Marko Myllynen

Re: Variable length date strings in glibc locales?

[Carlos O'Donell]

On 05/27/2014 02:24 PM, Marko Myllynen wrote:
> So I think I'll add a note to the wiki page that although in few places
> the resulting string is guaranteed to be of the same size in all locales
> (like int_curr_symbol), in many cases the resulting string can wary in
> length quite considerably.

Sounds good to me.

Cheers,
Carlos.

Re: Variable length date strings in glibc locales?

[Keld Simonsen]

On Tue, May 27, 2014 at 09:58:56AM +0300, Marko Myllynen wrote:
> Hi,
> 
> in some languages dates are written without leading zeroes so that May 3
> would be "3.5.". The same for time, 08:07:00 would be "8.07.00".
> 
> In glibc locales it would be possible to write dates and times in such
> fashion but do we know how that would affect existing applications? Are
> they expecting dates and times to be fixed length and would variable
> length date strings cause formatting or layout issues? Looking at
> existing locales, almost all of them use fixed length strings for
> d_fmt/t_fmt/date_fmt/d_t_fmt.
> 
> Ideally of course it would be nice to change certain locales to use date
> and time formats according to their cultural conventions and national
> recommendations but if that would lead to wonky layout in applications
> then it's probably better to be pragmatic and use fixed length dates.
> 
> I could add few words about this to our Locales wiki page if someone
> happens to know what's the best approach here.

I think there are two consideraation schools.

1. What is expected by users using POSIX like systems.

2. What is expected by the linguistic oriented users.

This varies from program to program.

For examle "ls" would expect constant width sizzes.
Also logging would expect constant width. 
"date" could be used for either POSIX-like or linguistic like purposes..
We have data for both constant width and variable with.
applications can then use the variant that is wanted.
For the figures, I am not fully sure how to do it in the way
the user would expect it. Maybe we need more formats here.

Also what is expected in specific applications may vary.
Eg. in accounting programs like spreadsheetd you would
normally want a thousands separator different from SPACE,
while in letters and documents you would want the SPACE as the thousands delimiter.

Best regards
keld

Re: Variable length date strings in glibc locales?

[Carlos O'Donell]

On 05/27/2014 05:02 PM, Keld Simonsen wrote:
> On Tue, May 27, 2014 at 09:58:56AM +0300, Marko Myllynen wrote:
>> Hi,
>>
>> in some languages dates are written without leading zeroes so that May 3
>> would be "3.5.". The same for time, 08:07:00 would be "8.07.00".
>>
>> In glibc locales it would be possible to write dates and times in such
>> fashion but do we know how that would affect existing applications? Are
>> they expecting dates and times to be fixed length and would variable
>> length date strings cause formatting or layout issues? Looking at
>> existing locales, almost all of them use fixed length strings for
>> d_fmt/t_fmt/date_fmt/d_t_fmt.
>>
>> Ideally of course it would be nice to change certain locales to use date
>> and time formats according to their cultural conventions and national
>> recommendations but if that would lead to wonky layout in applications
>> then it's probably better to be pragmatic and use fixed length dates.
>>
>> I could add few words about this to our Locales wiki page if someone
>> happens to know what's the best approach here.
> 
> I think there are two consideraation schools.
> 
> 1. What is expected by users using POSIX like systems.
> 
> 2. What is expected by the linguistic oriented users.
> 
> This varies from program to program.

Certainly.

So I'm going to take a policy position for glibc. The GNU C 
Library should accept locales with dates and times of variable
widths.

If the author of a program wishes a constant width they need
to use an ISO standard date format, or pad the individual
components themselves.

Cheers,
Carlos.

Re: Variable length date strings in glibc locales?

[Keld Simonsen]

On Tue, May 27, 2014 at 05:20:44PM -0400, Carlos O'Donell wrote:
> On 05/27/2014 05:02 PM, Keld Simonsen wrote:
> > On Tue, May 27, 2014 at 09:58:56AM +0300, Marko Myllynen wrote:
> >> Hi,
> >>
> >> in some languages dates are written without leading zeroes so that May 3
> >> would be "3.5.". The same for time, 08:07:00 would be "8.07.00".
> >>
> >> In glibc locales it would be possible to write dates and times in such
> >> fashion but do we know how that would affect existing applications? Are
> >> they expecting dates and times to be fixed length and would variable
> >> length date strings cause formatting or layout issues? Looking at
> >> existing locales, almost all of them use fixed length strings for
> >> d_fmt/t_fmt/date_fmt/d_t_fmt.
> >>
> >> Ideally of course it would be nice to change certain locales to use date
> >> and time formats according to their cultural conventions and national
> >> recommendations but if that would lead to wonky layout in applications
> >> then it's probably better to be pragmatic and use fixed length dates.
> >>
> >> I could add few words about this to our Locales wiki page if someone
> >> happens to know what's the best approach here.
> > 
> > I think there are two consideraation schools.
> > 
> > 1. What is expected by users using POSIX like systems.
> > 
> > 2. What is expected by the linguistic oriented users.
> > 
> > This varies from program to program.
> 
> Certainly.
> 
> So I'm going to take a policy position for glibc. The GNU C 
> Library should accept locales with dates and times of variable
> widths.
> 
> If the author of a program wishes a constant width they need
> to use an ISO standard date format, or pad the individual
> components themselves.

Hmm, I think that would not be according to users' expectations. 
For example "ls" and logging programs would be hard to change to that new 
convention. I think it would be better to go along with existing practice
and then provide guidance for how to use the info for new ways of using the
data.

I think the main problem area is the abbreviated day names and month names. 
I would advise that these be kept fixed, as this is what programs and users
expect in utilities. Then we should introduce a new set of abbreviated
day names and month names, that can have varying length. 
In this way we will be backwards compatible, and in line with programmers
and users expectations.

Best regards
Keld

Re: Variable length date strings in glibc locales?

[Marko Myllynen]

Hi,

On 2014-05-28 10:17, Keld Simonsen wrote:
> On Tue, May 27, 2014 at 05:20:44PM -0400, Carlos O'Donell wrote:
>> On 05/27/2014 05:02 PM, Keld Simonsen wrote:
>>> On Tue, May 27, 2014 at 09:58:56AM +0300, Marko Myllynen wrote:
>>>> Hi,
>>>>
>>>> in some languages dates are written without leading zeroes so that May 3
>>>> would be "3.5.". The same for time, 08:07:00 would be "8.07.00".
>>>>
>>>> In glibc locales it would be possible to write dates and times in such
>>>> fashion but do we know how that would affect existing applications? Are
>>>> they expecting dates and times to be fixed length and would variable
>>>> length date strings cause formatting or layout issues? Looking at
>>>> existing locales, almost all of them use fixed length strings for
>>>> d_fmt/t_fmt/date_fmt/d_t_fmt.
>>>>
>>>> Ideally of course it would be nice to change certain locales to use date
>>>> and time formats according to their cultural conventions and national
>>>> recommendations but if that would lead to wonky layout in applications
>>>> then it's probably better to be pragmatic and use fixed length dates.
>>>>
>>>> I could add few words about this to our Locales wiki page if someone
>>>> happens to know what's the best approach here.
>>>
>>> I think there are two consideraation schools.
>>>
>>> 1. What is expected by users using POSIX like systems.
>>>
>>> 2. What is expected by the linguistic oriented users.
>>>
>>> This varies from program to program.
>>
>> Certainly.
>>
>> So I'm going to take a policy position for glibc. The GNU C 
>> Library should accept locales with dates and times of variable
>> widths.
>>
>> If the author of a program wishes a constant width they need
>> to use an ISO standard date format, or pad the individual
>> components themselves.
> 
> Hmm, I think that would not be according to users' expectations. 
> For example "ls" and logging programs would be hard to change to that new 
> convention. I think it would be better to go along with existing practice
> and then provide guidance for how to use the info for new ways of using the
> data.
> 
> I think the main problem area is the abbreviated day names and month names. 
> I would advise that these be kept fixed, as this is what programs and users
> expect in utilities. Then we should introduce a new set of abbreviated
> day names and month names, that can have varying length. 
> In this way we will be backwards compatible, and in line with programmers
> and users expectations.

currently abbreviated day names and month names vary in length between
locales and within locales, too (e.g. fr_FR). Using the simple test
program again, we see:

localhost:~> cat t2.sh
#!/bin/bash

for f in a b ; do
  echo $f:
  for l in $(ls -1 /usr/share/i18n/locales/* | grep -Ev
'(@|i18n$|iso14651|translit|POSIX)') ; do
    echo -n "$(LC_ALL=$(basename $l.UTF-8) date --date="2007-05-03
08:07:00" +"%$f" | wc -L)" ; echo -e "\t$(basename $l).UTF-8"
  done | sort -un | sed -n '1p;$p'
done
localhost:~> unset LC_ALL
localhost:~> bash ./t2.sh
a:
1	ar_AE.UTF-8
7	fa_IR.UTF-8
b:
1	gu_IN.UTF-8
7	gv_GB.UTF-8
localhost:~>

In the past there was a request from an application developer to change
a locale which had fixed length month abbreviations in use:

https://bugzilla.redhat.com/show_bug.cgi?id=657572

So not sure what would be the ideal solution here but perhaps we could
at least try to mitigate the confusion with some additional instructions
in the wiki page, suggestions on that would warmly be welcome.

Thanks,

-- 
Marko Myllynen

Re: Variable length date strings in glibc locales?

[Marko Myllynen]

Hi,

On 2014-05-28 10:50, Marko Myllynen wrote:
> On 2014-05-28 10:17, Keld Simonsen wrote:
>> On Tue, May 27, 2014 at 05:20:44PM -0400, Carlos O'Donell wrote:
>>> On 05/27/2014 05:02 PM, Keld Simonsen wrote:
>>>> On Tue, May 27, 2014 at 09:58:56AM +0300, Marko Myllynen wrote:
>>>>>
>>>>> in some languages dates are written without leading zeroes so that May 3
>>>>> would be "3.5.". The same for time, 08:07:00 would be "8.07.00".
>>>>>
>>>>> In glibc locales it would be possible to write dates and times in such
>>>>> fashion but do we know how that would affect existing applications? Are
>>>>> they expecting dates and times to be fixed length and would variable
>>>>> length date strings cause formatting or layout issues? Looking at
>>>>> existing locales, almost all of them use fixed length strings for
>>>>> d_fmt/t_fmt/date_fmt/d_t_fmt.
>>>>>
>>>>> Ideally of course it would be nice to change certain locales to use date
>>>>> and time formats according to their cultural conventions and national
>>>>> recommendations but if that would lead to wonky layout in applications
>>>>> then it's probably better to be pragmatic and use fixed length dates.
>>>>>
>>>>> I could add few words about this to our Locales wiki page if someone
>>>>> happens to know what's the best approach here.
>>>>
>>>> I think there are two consideraation schools.
>>>>
>>>> 1. What is expected by users using POSIX like systems.
>>>>
>>>> 2. What is expected by the linguistic oriented users.
>>>>
>>>> This varies from program to program.
>>>
>>> Certainly.
>>>
>>> So I'm going to take a policy position for glibc. The GNU C 
>>> Library should accept locales with dates and times of variable
>>> widths.
>>>
>>> If the author of a program wishes a constant width they need
>>> to use an ISO standard date format, or pad the individual
>>> components themselves.
>>
>> Hmm, I think that would not be according to users' expectations. 
>> For example "ls" and logging programs would be hard to change to that new 
>> convention. I think it would be better to go along with existing practice
>> and then provide guidance for how to use the info for new ways of using the
>> data.
>>
>> I think the main problem area is the abbreviated day names and month names. 
>> I would advise that these be kept fixed, as this is what programs and users
>> expect in utilities. Then we should introduce a new set of abbreviated
>> day names and month names, that can have varying length. 
>> In this way we will be backwards compatible, and in line with programmers
>> and users expectations.
> 
> currently abbreviated day names and month names vary in length between
> locales and within locales, too (e.g. fr_FR).
> 
> So not sure what would be the ideal solution here but perhaps we could
> at least try to mitigate the confusion with some additional instructions
> in the wiki page, suggestions on that would warmly be welcome.

since there were no conclusions on this I'll add something like this to
the wiki page in a few days if no objections which basically sets the
expectations based on what is implemented today:

Application developers should note that while some of the resulting
strings (like int_curr_symbol of LC_MONETARY) are required to be of
certain length other resulting strings may vary even greatly, especially
between locales. Thus, testing applications with several locales is
recommended to make sure different length strings do not cause
inconsistencies in user experience.

Thanks,

-- 
Marko Myllynen

Re: Variable length date strings in glibc locales?

[Carlos O'Donell]

On 06/23/2014 11:17 AM, Marko Myllynen wrote:
> since there were no conclusions on this I'll add something like this to
> the wiki page in a few days if no objections which basically sets the
> expectations based on what is implemented today:
> 
> Application developers should note that while some of the resulting
> strings (like int_curr_symbol of LC_MONETARY) are required to be of
> certain length other resulting strings may vary even greatly, especially
> between locales. Thus, testing applications with several locales is
> recommended to make sure different length strings do not cause
> inconsistencies in user experience.

Agreed.

Please note that we need not reach a conclusion, just consensus.

Cheers,
Carlos.

Re: Variable length date strings in glibc locales?

[Keld Simonsen]

On Mon, Jun 23, 2014 at 01:40:07PM -0400, Carlos O'Donell wrote:
> On 06/23/2014 11:17 AM, Marko Myllynen wrote:
> > since there were no conclusions on this I'll add something like this to
> > the wiki page in a few days if no objections which basically sets the
> > expectations based on what is implemented today:
> > 
> > Application developers should note that while some of the resulting
> > strings (like int_curr_symbol of LC_MONETARY) are required to be of
> > certain length other resulting strings may vary even greatly, especially
> > between locales. Thus, testing applications with several locales is
> > recommended to make sure different length strings do not cause
> > inconsistencies in user experience.
> 
> Agreed.
> 
> Please note that we need not reach a conclusion, just consensus.

My opinion is that we need new keywords for linguistic abbreviations of
daynames and monthnames, which can vary in length, and a format string for that.
And then clarify that the existing abbreviations should be equal length and for
system things like logs.

I will write up a proposal for that.

Best regards
Keld