LC_* number reduction

LC_* number reduction

[Jakub Jelinek]

Hi!

> A call to setlocale opens currently 11 locale files. The needed file operations are expensive. The files
> for one locale should be consolidated.
>
> One drawback is, though, that this will probably eliminate the size optimization currently implemented
> which reduces entire locale files to hard links of existing files. The space savings can be large.
> Therefore this scheme should be optional.

I completely missed this todo item. Looking at how big and hardlinkable the
files are, I'd say glibc should use 3 locale files, not 11 nor 1:
LC_COLLATE
LC_CTYPE
LC_LOCALE (or however would be the rest of other LC_* files would be called together)

This scheme IMHO doesn't have to be optional.
The largest non-COLLATE non-CTYPE LC_* files on my box (with lots of .utf8
locales) are LC_TIME, where largest is a tad bit over 3KB, smallest tad bit
over 2KB. The rest of LC_* files are < 470B.
LC_TIME has at most 21 nlinks (arabic UTF8 locales) but on average 1-2, and
at least ATM LC_LOCALE could fit under 4KB for all locales (and 4KB block
filesystems are most common in these days IMHO).

The WWW page lists this as started, though I cannot find any changes related
to this.

	Jakub

Re: LC_* number reduction

[Ulrich Drepper]

Jakub Jelinek <jakub@redhat.com> writes:

> I completely missed this todo item. Looking at how big and hardlinkable the
> files are, I'd say glibc should use 3 locale files, not 11 nor 1:

I do have a different mechanism in mind.  Even simpler than that.  All
can be put in a simple flat "database".  I.e., one file which
immitates a filesystem with hardlinks etc.  It'll also include the
locale.lias information.  The advantage is that only one file has to
be opened, one name lookup in the hash table, and then potentially
only one single mmap() call if all the locale data is in consecutive
addresses.

> The WWW page lists this as started, though I cannot find any changes related
> to this.

I've stopped working on this before having something committed.  I
have to finish the TLS stuff first since it requires work on other
architectures to follow.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: LC_* number reduction

[Jakub Jelinek]

On Tue, Feb 05, 2002 at 10:15:29AM -0800, Ulrich Drepper wrote:
> > I completely missed this todo item. Looking at how big and hardlinkable the
> > files are, I'd say glibc should use 3 locale files, not 11 nor 1:
> 
> I do have a different mechanism in mind.  Even simpler than that.  All
> can be put in a simple flat "database".  I.e., one file which
> immitates a filesystem with hardlinks etc.  It'll also include the
> locale.lias information.  The advantage is that only one file has to
> be opened, one name lookup in the hash table, and then potentially
> only one single mmap() call if all the locale data is in consecutive
> addresses.

Well, this has obvious advantages, but some disadvantages too:
a) if you have this 25M locale.db file installed and want to upgrade glibc
   which e.g. contains some locales fixes, you need to decide either that
   you replace this file with the new one (which means blowing away possible
   user's localedef additions/tweaks/whatever), or leave the old one there
   (thus the user won't upgrade the locales for which fixes exist).
   Of course, there could be some "merge" operation for them.
   I don't consider here rebuilding all locales from sources at glibc
   package install time, since that takes too long.
b) it is no longer possible to install a subset of locales which users will
   only use (say at install time one chooses he will only be interested in
   english, french and german locales) - you end up with the 25M file all
   the time.

	Jakub

Re: LC_* number reduction

[Ulrich Drepper]

Jakub Jelinek <jakub@redhat.com> writes:

> a) if you have this 25M locale.db file installed and want to upgrade glibc
>    which e.g. contains some locales fixes, you need to decide either that
>    you replace this file with the new one (which means blowing away possible
>    user's localedef additions/tweaks/whatever), or leave the old one there
>    (thus the user won't upgrade the locales for which fixes exist).
>    Of course, there could be some "merge" operation for them.
>    I don't consider here rebuilding all locales from sources at glibc
>    package install time, since that takes too long.

This isn't much of a problem.  What I had in mind is providing some
kind of 'ar' functionality.  If you have to do major upgrades etc you
extract all files from the database and create normal files.  Then you
update the separate files and finally create a brand new database.

> b) it is no longer possible to install a subset of locales which users will
>    only use (say at install time one chooses he will only be interested in
>    english, french and german locales) - you end up with the 25M file all
>    the time.

Again, no problem.  I would not expect the database to be shipped
(maybe as well).  You install the single files as it is in the moment.
Than run localedef to insert the file and remove the separate file
then.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------