[CT-NG 1.11.1] eglibc internationalization support

[CT-NG 1.11.1] eglibc internationalization support

[Benoît THÉBAUDEAU]

Hi Yann, all,

To my surprise, the eglibc locales are not built by CT-NG although the localedef
tarball is downloaded.

I propose the patch below for that purpose (tested successfully). Please tell me
what you think.

Best regards,
Benoît
--------------------------------------------------------------------------------
This patch adds support for eglibc localedef.

localedef has to be extracted and symlinked like external eglibc add-ons.

However, it requires a dedicated configure/build/install sequence, which implies
setting only the required add-ons in --enable-add-ons in order to avoid breaking
the libc configure because of localedef being detected as an add-on.

Signed-off-by: Benoît Thébaudeau <benoit.thebaudeau@advansee.com>
---
diff -Nrdup crosstool-ng-1.11.1.orig/config/libc/glibc-eglibc.in-common crosstool-ng-1.11.1/config/libc/glibc-eglibc.in-common
--- crosstool-ng-1.11.1.orig/config/libc/glibc-eglibc.in-common	2011-05-02 18:11:24.000000000 +0200
+++ crosstool-ng-1.11.1/config/libc/glibc-eglibc.in-common	2011-05-17 18:02:52.466466861 +0200
@@ -141,7 +141,7 @@ config LIBC_ADDONS_LIST

       You need to specify neither linuxthreads nor nptl, as they are added
       automagically for you depending on the threading model you choosed
-      earlier.
+      earlier. Same for localedef.

       Eg.: crypt        (for very old libces)

diff -Nrdup crosstool-ng-1.11.1.orig/scripts/build/libc/eglibc.sh crosstool-ng-1.11.1/scripts/build/libc/eglibc.sh
--- crosstool-ng-1.11.1.orig/scripts/build/libc/eglibc.sh	2011-05-02 18:11:24.000000000 +0200
+++ crosstool-ng-1.11.1/scripts/build/libc/eglibc.sh	2011-05-17 19:38:50.447246778 +0200
@@ -104,7 +104,7 @@ do_libc_extract() {
     CT_Patch "eglibc" "${CT_LIBC_VERSION}"

     # C library addons
-    for addon in $(do_libc_add_ons_list " "); do
+    for addon in $(do_libc_add_ons_list " ") localedef; do
         # NPTL addon is not to be extracted, in any case
         [ "${addon}" = "nptl" ] && continue || true
         CT_Pushd "${CT_SRC_DIR}/eglibc-${CT_LIBC_VERSION}"
@@ -174,3 +174,56 @@ do_libc_check_config() {

     CT_EndStep
 }
+
+# This function builds and installs the C library locales
+do_eglibc_localedef() {
+    local libc_src_dir="${CT_SRC_DIR}/${CT_LIBC}-${CT_LIBC_VERSION}"
+    local src_dir="${libc_src_dir}/localedef"
+    local -a localedef_opts
+
+    CT_DoStep INFO "Installing C library locales"
+
+    mkdir -p "${CT_BUILD_DIR}/build-libc/localedef"
+    cd "${CT_BUILD_DIR}/build-libc/localedef"
+
+    CT_DoLog EXTRA "Configuring C library locales"
+
+    # ./configure is misled by our tools override wrapper for bash
+    # so just tell it where the real bash is _on_the_target_!
+    # Notes:
+    # - ${ac_cv_path_BASH_SHELL} is only used to set BASH_SHELL
+    # - ${BASH_SHELL}            is only used to set BASH
+    # - ${BASH}                  is only used to set the shebang
+    #                            in two scripts to run on the target
+    # So we can safely bypass bash detection at compile time.
+    # Should this change in a future eglibc release, we'd better
+    # directly mangle the generated scripts _after_ they get built,
+    # or even after they get installed... eglibc is such a sucker...
+    echo "ac_cv_path_BASH_SHELL=/bin/bash" >>config.cache
+
+    # Configure with --prefix the way we want it on the target...
+
+    CT_DoExecLog CFG                                                \
+    "${src_dir}/configure"                                          \
+        --prefix=/usr                                               \
+        --cache-file="$(pwd)/config.cache"                          \
+        --with-glibc="${libc_src_dir}"
+
+    CT_DoLog EXTRA "Building C library locales"
+    CT_DoExecLog ALL make ${JOBSFLAGS}
+
+    # Set the localedef endianness option
+    case "${CT_ARCH_BE},${CT_ARCH_LE}" in
+        y,) localedef_opts+=(--big-endian);;
+        ,y) localedef_opts+=(--little-endian);;
+    esac
+
+    # Set the localedef option for the target's uint32_t alignment in bytes
+    localedef_opts+=(--uint32-align=4)
+
+    CT_DoLog EXTRA "Installing C library locales"
+    CT_DoExecLog ALL make ${JOBSFLAGS}                              \
+                          "LOCALEDEF_OPTS=${localedef_opts[*]}"     \
+                          install_root="${CT_SYSROOT_DIR}"          \
+                          install-locales
+}
diff -Nrdup crosstool-ng-1.11.1.orig/scripts/build/libc/glibc-eglibc.sh-common crosstool-ng-1.11.1/scripts/build/libc/glibc-eglibc.sh-common
--- crosstool-ng-1.11.1.orig/scripts/build/libc/glibc-eglibc.sh-common	2011-05-02 18:11:24.000000000 +0200
+++ crosstool-ng-1.11.1/scripts/build/libc/glibc-eglibc.sh-common	2011-05-17 18:23:49.700466215 +0200
@@ -3,6 +3,7 @@
 # Build and install headers and start files
 do_libc_start_files() {
     local src_dir="${CT_SRC_DIR}/${CT_LIBC}-${CT_LIBC_VERSION}"
+    local -a extra_config

     CT_DoStep INFO "Installing C library headers & start files"

@@ -19,6 +20,11 @@ do_libc_start_files() {
             ;;
     esac

+    case "$(do_libc_add_ons_list ,)" in
+        "") extra_config+=("--enable-add-ons=no");;
+        *)  extra_config+=("--enable-add-ons=$(do_libc_add_ons_list ,)");;
+    esac
+
     cross_cc=$(CT_Which "${CT_TARGET}-gcc")
     cross_cxx=$(CT_Which "${CT_TARGET}-g++")
     cross_ar=$(CT_Which "${CT_TARGET}-ar")
@@ -53,7 +59,7 @@ do_libc_start_files() {
         --disable-profile                               \
         --without-gd                                    \
         --without-cvs                                   \
-        --enable-add-ons
+        "${extra_config[@]}"

     CT_DoLog EXTRA "Installing C library headers"

@@ -185,7 +191,7 @@ do_libc() {
     fi

     case "$(do_libc_add_ons_list ,)" in
-        "") ;;
+        "") extra_config+=("--enable-add-ons=no");;
         *)  extra_config+=("--enable-add-ons=$(do_libc_add_ons_list ,)");;
     esac

@@ -272,6 +278,10 @@ do_libc() {
                           install_root="${CT_SYSROOT_DIR}"  \
                           install

+    if [ "${CT_LIBC}" = "eglibc" ]; then
+        do_eglibc_localedef
+    fi
+
     CT_EndStep
 }


--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [CT-NG 1.11.1] eglibc internationalization support

[Yann E. MORIN]

Benoît, All,

On Wednesday 18 May 2011 17:38:52 Benoît THÉBAUDEAU wrote:
[--SNIP--]
> This patch adds support for eglibc localedef.
> 
> localedef has to be extracted and symlinked like external eglibc add-ons.
> 
> However, it requires a dedicated configure/build/install sequence, which implies
> setting only the required add-ons in --enable-add-ons in order to avoid breaking
> the libc configure because of localedef being detected as an add-on.

So, localedef is an add-on, but is not built like an add-on? Sigh...

> Signed-off-by: Benoît Thébaudeau <benoit.thebaudeau@advansee.com>
[--SNIP--]

I have nothing important to say about it for now. Looks good.
With the add-ons selection chmanges on-going, I thinks that it should be
added as an available add-on.

More review later, off to rest for now...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [CT-NG 1.11.1] eglibc internationalization support

[Benoît THÉBAUDEAU]

Yann, all,

> So, localedef is an add-on, but is not built like an add-on? Sigh...

Indeed.

> I have nothing important to say about it for now. Looks good.
> With the add-ons selection chmanges on-going, I thinks that it should
> be
> added as an available add-on.

Sure, but as an add-on handled automatically by CT-NG like nptl (this is
what my patch does), or as an optional add-on from CT_LIBC_ADDONS_LIST?

Also note that it is possible to limit the set of locales to install with
SUPPORTED-LOCALES, but I'm note sure adding such a config would be very
useful for CT-NG.

> More review later, off to rest for now...

Alright.

Regards,
Benoît

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [CT-NG 1.11.1] eglibc internationalization support

[Benoît THÉBAUDEAU]

Hi Yann, all!

> > I have nothing important to say about it for now. Looks good.
> > With the add-ons selection chmanges on-going, I thinks that it
> > should
> > be
> > added as an available add-on.
>
> Sure, but as an add-on handled automatically by CT-NG like nptl (this
> is
> what my patch does), or as an optional add-on from
> CT_LIBC_ADDONS_LIST?
>
> Also note that it is possible to limit the set of locales to install
> with
> SUPPORTED-LOCALES, but I'm note sure adding such a config would be
> very
> useful for CT-NG.

I have briefly looked at glibc's and eglibc's Makefiles. Both have a
localedata/install-locales target, but apparently only for native builds
since they use the built localedef. A host localedef is required in order
to install the locales, hence eglibc's localedef add-on. Unfortunately,
there doesn't seem to be such an add-on for glibc.

The options for eglibc are:
1) Use the localedef add-on.
2) Use a pre-installed host version of localedef.

The options for glibc are:
1) Use eglibc's localedef add-on if it works, but it's probably better to
   avoid mixing packages.
2) Use a pre-installed host version of localedef.
3) Do not support installing target locales, which would be a shame.

I think 2) would be the best for both since it would avoid eglibc/glibc
branches. The host localedef would be detected by CT-NG's configure.

Do you think the installation of locales should be mandatory, or a config
option?

Do you think that the list of locales to install should be a configuration,
or that all locales should be installed?

Tools like BuildRoot have an option to purge the target locales installed
from a toolchain, so too many config options concerning locales is perhaps
useless for CT-NG.

Tell me what you think.

Regards,
Benoît

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [CT-NG 1.11.1] eglibc internationalization support

[Benoît THÉBAUDEAU]

Hi Yann, all,

Does somebody have suggestions/comments/preferences or need more
time to answer, or should I propose a patch first, then discuss?

Regards,
Benoît

----- Original Message -----
> From: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> To: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
> Cc: crossgcc@sourceware.org
> Sent: Monday, June 6, 2011 7:32:06 PM
> Subject: Re: [CT-NG 1.11.1] eglibc internationalization support
>
> Hi Yann, all!
>
> > > I have nothing important to say about it for now. Looks good.
> > > With the add-ons selection chmanges on-going, I thinks that it
> > > should
> > > be
> > > added as an available add-on.
> >
> > Sure, but as an add-on handled automatically by CT-NG like nptl
> > (this
> > is
> > what my patch does), or as an optional add-on from
> > CT_LIBC_ADDONS_LIST?
> >
> > Also note that it is possible to limit the set of locales to
> > install
> > with
> > SUPPORTED-LOCALES, but I'm note sure adding such a config would be
> > very
> > useful for CT-NG.
>
> I have briefly looked at glibc's and eglibc's Makefiles. Both have a
> localedata/install-locales target, but apparently only for native
> builds
> since they use the built localedef. A host localedef is required in
> order
> to install the locales, hence eglibc's localedef add-on.
> Unfortunately,
> there doesn't seem to be such an add-on for glibc.
>
> The options for eglibc are:
> 1) Use the localedef add-on.
> 2) Use a pre-installed host version of localedef.
>
> The options for glibc are:
> 1) Use eglibc's localedef add-on if it works, but it's probably
> better to
>    avoid mixing packages.
> 2) Use a pre-installed host version of localedef.
> 3) Do not support installing target locales, which would be a shame.
>
> I think 2) would be the best for both since it would avoid
> eglibc/glibc
> branches. The host localedef would be detected by CT-NG's configure.
>
> Do you think the installation of locales should be mandatory, or a
> config
> option?
>
> Do you think that the list of locales to install should be a
> configuration,
> or that all locales should be installed?
>
> Tools like BuildRoot have an option to purge the target locales
> installed
> from a toolchain, so too many config options concerning locales is
> perhaps
> useless for CT-NG.
>
> Tell me what you think.
>
> Regards,
> Benoît

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [CT-NG 1.11.1] eglibc internationalization support

[Yann E. MORIN]

Benoit, All,

On Thursday 09 June 2011 15:42:07 Benoît THÉBAUDEAU wrote:
> Does somebody have suggestions/comments/preferences or need more
> time to answer, or should I propose a patch first, then discuss?

The real reason we need localedef is to build locales, right? So
maybe we'd better add an option:
  [ ] Build and install locales

And then build localedef and locales. I mean, rather that list it in the
add-ons list.

Below are a few quotes of your original patch:

> +# This function builds and installs the C library locales
> +do_eglibc_localedef() {

Rename to 'do_libc_locales' and add the same function in the common
glibc/eglibc script.

Then, instead of calling do_eglibc_localedef conditionnaly on CT_LIBC ==
eglibc, you can call it always. Then, once we know how to build locales
for glibc, infrastructure is there.

> +    CT_DoStep INFO "Installing C library locales"
> +
> +    mkdir -p "${CT_BUILD_DIR}/build-libc/localedef"
> +    cd "${CT_BUILD_DIR}/build-libc/localedef"

Although localedef is part of the C library, its build procedure is really
completely decorelated. Can't we build localedef in its own build dir:
    mkdir -p "${CT_BUILD_DIR}/build-localedef"
    cd "${CT_BUILD_DIR}/build-localedef"

Or, is it required that it be built as a sub-dir of eglibc? Where are the
locales definitions stored? Is that what --with-glibc= is for?

> +    CT_DoExecLog CFG                                                \
> +    "${src_dir}/configure"                                          \
> +        --prefix=/usr                                               \
> +        --cache-file="$(pwd)/config.cache"                          \
> +        --with-glibc="${libc_src_dir}"
> +

Indeed, we do want to build localedef natively. It runs on the host.
Correct?

Also, configure accepts --with-pkgversion and --with-bugurl, so we could
add them as well if they are usefull, or add a comment stating they are
not used for locales.

Would it be worth having the localedef binary on the target, in case there
is a need to build the locales on-the-fly there?

> +    # Set the localedef option for the target's uint32_t alignment in
> bytes
> +    localedef_opts+=(--uint32-align=4)

Even on 64-bit archs?

> @@ -19,6 +20,11 @@ do_libc_start_files() {
> +    case "$(do_libc_add_ons_list ,)" in
> +        "") extra_config+=("--enable-add-ons=no");;
> +        *)  extra_config+=("--enable-add-ons=$(do_libc_add_ons_list ,)");;
> +    esac
> +

...and...

> @@ -185,7 +191,7 @@ do_libc() {
>      case "$(do_libc_add_ons_list ,)" in
> -        "") ;;
> +        "") extra_config+=("--enable-add-ons=no");;
>          *)  extra_config+=("--enable-add-ons=$(do_libc_add_ons_list ,)");;
>      esac

... -> Could you make those two hunks (and related) into a separate
patch, please?

> > I have briefly looked at glibc's and eglibc's Makefiles. Both have a
> > localedata/install-locales target, but apparently only for native
> > builds
> > since they use the built localedef. A host localedef is required in
> > order
> > to install the locales, hence eglibc's localedef add-on.
> > Unfortunately,
> > there doesn't seem to be such an add-on for glibc.

Sad...

> > The options for eglibc are:
> > 1) Use the localedef add-on.
> > 2) Use a pre-installed host version of localedef.

1) as you did. We'll be needing it on hosts that do not have it ( yes,
there are guys using uClibc-based desktops out there... ;-) )

> > The options for glibc are:
> > 1) Use eglibc's localedef add-on if it works, but it's probably
> > better to
> >    avoid mixing packages.

That's not an option, indeed.

> > 2) Use a pre-installed host version of localedef.
> > 3) Do not support installing target locales, which would be a shame.

3) is better in the short term. We can see later if we can devise a plan
to build them.

> > I think 2) would be the best for both since it would avoid
> > eglibc/glibc
> > branches. The host localedef would be detected by CT-NG's configure.

The issue is about people who are building on hosts that may not have
have a localedef, for example uClibc-based desktops, on maybe MacOS.

> > Do you think the installation of locales should be mandatory, or a
> > config
> > option?

Config option, default n.

> > Do you think that the list of locales to install should be a
> > configuration,
> > or that all locales should be installed?

All locales. It would be up to the rootfs build system to choose what
locales to install. So maybe we could add a script that accepts a list
of locales, a destination rootfs, and copies required locales, and
handling patterns, a bit like:
  tuple-locales --dest-dir /some/place/rootfs 'en_US*' fr_FR.UTF8

Optionally, it should be possible to pass a file that lists such patterns:
  tuple-locales -d /some/place/rootfs -f my-locales-patterns

It may even makes sense to add this option to populate.
I can take care of that at a later stage...

> > Tools like BuildRoot have an option to purge the target locales
> > installed
> > from a toolchain, so too many config options concerning locales is
> > perhaps
> > useless for CT-NG.

Yes, indeed, it should be the role of such a build system to install
needed locales.

Sorry for the time it took for me to answer... Plus, I'm off for the WE
starting tomorrow morning early, although I may have a bit of time to
read emails at some point...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [CT-NG 1.11.1] eglibc internationalization support

[Benoît THÉBAUDEAU]

Yann, all,

> The real reason we need localedef is to build locales, right? So
> maybe we'd better add an option:
>   [ ] Build and install locales
>
> And then build localedef and locales. I mean, rather that list it in
> the
> add-ons list.

Yes, that's better, all the more eglibc-localedef does not really
behave like other add-ons.

> Below are a few quotes of your original patch:
>
> > +# This function builds and installs the C library locales
> > +do_eglibc_localedef() {
>
> Rename to 'do_libc_locales' and add the same function in the common
> glibc/eglibc script.
>
> Then, instead of calling do_eglibc_localedef conditionnaly on CT_LIBC
> ==
> eglibc, you can call it always. Then, once we know how to build
> locales
> for glibc, infrastructure is there.

OK. Is there the same kind of issue with uClibc, or is it specific to
(e)glibc?

> > +    CT_DoStep INFO "Installing C library locales"
> > +
> > +    mkdir -p "${CT_BUILD_DIR}/build-libc/localedef"
> > +    cd "${CT_BUILD_DIR}/build-libc/localedef"
>
> Although localedef is part of the C library, its build procedure is
> really
> completely decorelated. Can't we build localedef in its own build
> dir:
>     mkdir -p "${CT_BUILD_DIR}/build-localedef"
>     cd "${CT_BUILD_DIR}/build-localedef"
>
> Or, is it required that it be built as a sub-dir of eglibc? Where are
> the
> locales definitions stored? Is that what --with-glibc= is for?

eglibc-localedef can probably be extracted and built outside of eglibc.
I have to check that. What is certain is that it uses at least
eglibc/localedata. I will do that if it is supported since that would
avoid disturbing the configuration of normal add-ons.

> > +    CT_DoExecLog CFG
> >                                                \
> > +    "${src_dir}/configure"
> >                                          \
> > +        --prefix=/usr
> >                                               \
> > +        --cache-file="$(pwd)/config.cache"
> >                          \
> > +        --with-glibc="${libc_src_dir}"
> > +
>
> Indeed, we do want to build localedef natively. It runs on the host.
> Correct?

Yes.

> Also, configure accepts --with-pkgversion and --with-bugurl, so we
> could
> add them as well if they are usefull, or add a comment stating they
> are
> not used for locales.

OK. I'll see if these options are put somewhere when building locales.

> Would it be worth having the localedef binary on the target, in case
> there
> is a need to build the locales on-the-fly there?

The native target localedef is already built and installed by (e)glibc,
so nothing to do here.

> > +    # Set the localedef option for the target's uint32_t alignment
> > in
> > bytes
> > +    localedef_opts+=(--uint32-align=4)
>
> Even on 64-bit archs?

I'll check.

> > @@ -19,6 +20,11 @@ do_libc_start_files() {
> > +    case "$(do_libc_add_ons_list ,)" in
> > +        "") extra_config+=("--enable-add-ons=no");;
> > +        *)
> >  extra_config+=("--enable-add-ons=$(do_libc_add_ons_list ,)");;
> > +    esac
> > +
>
> ...and...
>
> > @@ -185,7 +191,7 @@ do_libc() {
> >      case "$(do_libc_add_ons_list ,)" in
> > -        "") ;;
> > +        "") extra_config+=("--enable-add-ons=no");;
> >          *)
> >           extra_config+=("--enable-add-ons=$(do_libc_add_ons_list
> >          ,)");;
> >      esac
>
> ... -> Could you make those two hunks (and related) into a separate
> patch, please?

OK. Do you know if add-ons can be skipped in the do_libc_start_files()
step? Perhaps "--enable-add-ons=no" would be fine in all cases at this
step. That would save some configure time. On the other hand, it may be
too risky for the future maintainability.

Add-ons can also be auto-detected (it's the default). Do you think it
would be worth adding a way to choose "all" add-ons in the config? Note
that this can already be achieved by adding "--enable-add-ons" to the
_EXTRA_CONFIG_ARRAY.

> > > I have briefly looked at glibc's and eglibc's Makefiles. Both
> > > have a
> > > localedata/install-locales target, but apparently only for native
> > > builds
> > > since they use the built localedef. A host localedef is required
> > > in
> > > order
> > > to install the locales, hence eglibc's localedef add-on.
> > > Unfortunately,
> > > there doesn't seem to be such an add-on for glibc.
>
> Sad...
>
> > > The options for eglibc are:
> > > 1) Use the localedef add-on.
> > > 2) Use a pre-installed host version of localedef.
>
> 1) as you did. We'll be needing it on hosts that do not have it (
> yes,
> there are guys using uClibc-based desktops out there... ;-) )

OK.

> > > The options for glibc are:
> > > 1) Use eglibc's localedef add-on if it works, but it's probably
> > > better to
> > >    avoid mixing packages.
>
> That's not an option, indeed.
>
> > > 2) Use a pre-installed host version of localedef.
> > > 3) Do not support installing target locales, which would be a
> > > shame.
>
> 3) is better in the short term. We can see later if we can devise a
> plan
> to build them.

OK. It would perhaps be feasible to extract/configure/build glibc
somewhere else for native host, which would build the native host
localedef. A restricted build like 'make [...] locale/localedef'
might also work after the configure. Overkill?

> > > I think 2) would be the best for both since it would avoid
> > > eglibc/glibc
> > > branches. The host localedef would be detected by CT-NG's
> > > configure.
>
> The issue is about people who are building on hosts that may not have
> have a localedef, for example uClibc-based desktops, on maybe MacOS.

OK.

> > > Do you think the installation of locales should be mandatory, or
> > > a
> > > config
> > > option?
>
> Config option, default n.
>
> > > Do you think that the list of locales to install should be a
> > > configuration,
> > > or that all locales should be installed?
>
> All locales. It would be up to the rootfs build system to choose what
> locales to install. So maybe we could add a script that accepts a
> list
> of locales, a destination rootfs, and copies required locales, and
> handling patterns, a bit like:
>   tuple-locales --dest-dir /some/place/rootfs 'en_US*' fr_FR.UTF8
>
> Optionally, it should be possible to pass a file that lists such
> patterns:
>   tuple-locales -d /some/place/rootfs -f my-locales-patterns
>
> It may even makes sense to add this option to populate.
> I can take care of that at a later stage...

OK.

> > > Tools like BuildRoot have an option to purge the target locales
> > > installed
> > > from a toolchain, so too many config options concerning locales
> > > is
> > > perhaps
> > > useless for CT-NG.
>
> Yes, indeed, it should be the role of such a build system to install
> needed locales.
>
> Sorry for the time it took for me to answer... Plus, I'm off for the
> WE
> starting tomorrow morning early, although I may have a bit of time to
> read emails at some point...

No problem. I just feared my previous message got forgotten in traffic.

I'll cook a new patch series when I have some time.

Best regards,
Benoît

--
For unsubscribe information see http://sourceware.org/lists.html#faq

[PATCH 3 of 5] scripts/eglibc: add support for locales

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311851059 -7200
# Node ID 3b3715d0f1d412df91e472ed563d0cde0afedc6b
# Parent  4fed02058cc3276180d6014899062f83e620eb37
scripts/eglibc: add support for locales

This patch adds support for eglibc locales.

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/scripts/build/libc/eglibc.sh b/scripts/build/libc/eglibc.sh
--- a/scripts/build/libc/eglibc.sh
+++ b/scripts/build/libc/eglibc.sh
@@ -14,6 +14,7 @@
 # snapshots available.
 do_libc_get() {
     local addon
+    local -a extra_addons
     local svn_base
 
     if [ "${CT_EGLIBC_HTTP}" = "y" ]; then
@@ -31,7 +32,11 @@
               "${svn_base}/libc"            \
               "${CT_EGLIBC_REVISION:-HEAD}"
 
-    for addon in $(do_libc_add_ons_list " "); do
+    if [ "${CT_LIBC_LOCALES}" = "y" ]; then
+        extra_addons+=("localedef")
+    fi
+
+    for addon in $(do_libc_add_ons_list " ") ${extra_addons[@]}; do
         # Never ever try to download these add-ons,
         # they've always been internal
         case "${addon}" in
@@ -97,13 +102,69 @@
 }
 
 # Extract the files required for the libc locales
-# Nothing to do
 do_libc_locales_extract() {
-    :
+    CT_Extract "eglibc-localedef-${CT_LIBC_VERSION}"
+    CT_Patch "eglibc" "localedef-${CT_LIBC_VERSION}"
 }
 
 # Build and install the libc locales
-# Not yet supported
 do_libc_locales() {
-    :
+    local libc_src_dir="${CT_SRC_DIR}/eglibc-${CT_LIBC_VERSION}"
+    local src_dir="${CT_SRC_DIR}/eglibc-localedef-${CT_LIBC_VERSION}"
+    local -a extra_config
+    local -a localedef_opts
+
+    mkdir -p "${CT_BUILD_DIR}/build-localedef"
+    cd "${CT_BUILD_DIR}/build-localedef"
+
+    CT_DoLog EXTRA "Configuring C library localedef"
+
+    if [ "${CT_LIBC_EGLIBC_HAS_PKGVERSION_BUGURL}" = "y" ]; then
+        extra_config+=("--with-pkgversion=${CT_PKGVERSION}")
+        [ -n "${CT_TOOLCHAIN_BUGURL}" ] && extra_config+=("--with-bugurl=${CT_TOOLCHAIN_BUGURL}")
+    fi
+
+    CT_DoLog DEBUG "Extra config args passed: '${extra_config[*]}'"
+
+    # ./configure is misled by our tools override wrapper for bash
+    # so just tell it where the real bash is _on_the_target_!
+    # Notes:
+    # - ${ac_cv_path_BASH_SHELL} is only used to set BASH_SHELL
+    # - ${BASH_SHELL}            is only used to set BASH
+    # - ${BASH}                  is only used to set the shebang
+    #                            in two scripts to run on the target
+    # So we can safely bypass bash detection at compile time.
+    # Should this change in a future eglibc release, we'd better
+    # directly mangle the generated scripts _after_ they get built,
+    # or even after they get installed... eglibc is such a sucker...
+    echo "ac_cv_path_BASH_SHELL=/bin/bash" >>config.cache
+
+    # Configure with --prefix the way we want it on the target...
+
+    CT_DoExecLog CFG                                                \
+    "${src_dir}/configure"                                          \
+        --prefix=/usr                                               \
+        --cache-file="$(pwd)/config.cache"                          \
+        --with-glibc="${libc_src_dir}"                              \
+        "${extra_config[@]}"
+
+    CT_DoLog EXTRA "Building C library localedef"
+    CT_DoExecLog ALL make ${JOBSFLAGS}
+
+    # Set the localedef endianness option
+    case "${CT_ARCH_BE},${CT_ARCH_LE}" in
+        y,) localedef_opts+=(--big-endian);;
+        ,y) localedef_opts+=(--little-endian);;
+    esac
+
+    # Set the localedef option for the target's uint32_t alignment in bytes.
+    # This is target-specific, but for now, 32-bit alignment should work for all
+    # supported targets, even 64-bit ones.
+    localedef_opts+=(--uint32-align=4)
+
+    CT_DoLog EXTRA "Installing C library locales"
+    CT_DoExecLog ALL make ${JOBSFLAGS}                              \
+                          "LOCALEDEF_OPTS=${localedef_opts[*]}"     \
+                          install_root="${CT_SYSROOT_DIR}"          \
+                          install-locales
 }

[PATCH 5 of 5] scripts/functions: force extract folder to archive basename

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311867232 -7200
# Node ID 50c28edc0a0ebe6bf285b6656233365e2ed88df0
# Parent  ac0f1f93198fca8e2f28b9499e197fa387c67234
scripts/functions: force extract folder to archive basename

Some archives like those of the 2011.07 revisions of Linaro GCC contain a folder
name different from the archive basename, which leads to errors afterwards, e.g.
when patching. E.g.:
gcc-linaro-4.5-2011.07.tar.bz2 extracts to gcc-linaro-4.5-2011.07-0/

This patch changes CT_Extract() to force the extraction of all archives to a
folder named like the archive basename. E.g.:
gcc-linaro-4.5-2011.07.tar.bz2 now extracts to gcc-linaro-4.5-2011.07/

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/scripts/functions b/scripts/functions
--- a/scripts/functions
+++ b/scripts/functions
@@ -728,10 +728,15 @@
     CT_Pushd "${nochdir}"
 
     CT_DoLog EXTRA "Extracting '${basename}'"
+    CT_DoExecLog FILE mkdir -p "${basename}"
     case "${ext}" in
-        .tar.bz2)     CT_DoExecLog FILE tar xvjf "${full_file}";;
-        .tar.gz|.tgz) CT_DoExecLog FILE tar xvzf "${full_file}";;
-        .tar)         CT_DoExecLog FILE tar xvf  "${full_file}";;
+        .tar.bz2)     { bzcat "${full_file}" \
+                        |tar --strip-components=1 -C "${basename}" -xvf - ;
+                      } 2>&1 |CT_DoLog FILE;;
+        .tar.gz|.tgz) { gzip -d -c "${full_file}" \
+                        |tar --strip-components=1 -C "${basename}" -xvf - ;
+                      } 2>&1 |CT_DoLog FILE;;
+        .tar)         CT_DoExecLog FILE tar --strip-components=1 -C "${basename}" -xvf "${full_file}";;
         /.git)        CT_ExtractGit "${basename}" "${@}";;
         *)            CT_DoLog WARN "Don't know how to handle '${basename}${ext}': unknown extension"
                       return 1

[PATCH 2 of 5] libc: create an infrastructure to build and install the libc locales

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311780145 -7200
# Node ID 4fed02058cc3276180d6014899062f83e620eb37
# Parent  0ee44cad2124dcf96979819f0ee4851dd6ef2334
libc: create an infrastructure to build and install the libc locales

This patch adds a common glibc/eglibc infrastructure to build and install the
libc locales.

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/config/libc/glibc-eglibc.in-common b/config/libc/glibc-eglibc.in-common
--- a/config/libc/glibc-eglibc.in-common
+++ b/config/libc/glibc-eglibc.in-common
@@ -144,6 +144,13 @@
 comment "|  or try again later... :-(                            "
 endif
 
+config LIBC_LOCALES
+    bool
+    prompt "Build and install locales"
+    help
+      Whether to build and install the libc locale files for the target,
+      which is required in order to support internationalization.
+
 if KERNEL_linux
 
 choice LIBC_GLIBC_SUPPORTED_KERNEL
diff --git a/scripts/build/libc/eglibc.sh b/scripts/build/libc/eglibc.sh
--- a/scripts/build/libc/eglibc.sh
+++ b/scripts/build/libc/eglibc.sh
@@ -95,3 +95,15 @@
 
     CT_EndStep
 }
+
+# Extract the files required for the libc locales
+# Nothing to do
+do_libc_locales_extract() {
+    :
+}
+
+# Build and install the libc locales
+# Not yet supported
+do_libc_locales() {
+    :
+}
diff --git a/scripts/build/libc/glibc-eglibc.sh-common b/scripts/build/libc/glibc-eglibc.sh-common
--- a/scripts/build/libc/glibc-eglibc.sh-common
+++ b/scripts/build/libc/glibc-eglibc.sh-common
@@ -47,6 +47,10 @@
     find . -type f -name configure -exec touch {} \; 2>&1 |CT_DoLog ALL
 
     CT_Popd
+
+    if [ "${CT_LIBC_LOCALES}" = "y" ]; then
+        do_libc_locales_extract
+    fi
 }
 
 # Build and install headers and start files
@@ -294,6 +298,10 @@
                               "${extra_make_args[@]}"           \
                               install_root="${CT_SYSROOT_DIR}"  \
                               install
+
+        if [ "${CT_LIBC_LOCALES}" = "y" ]; then
+            do_libc_locales
+        fi
     fi
 
     CT_EndStep
diff --git a/scripts/build/libc/glibc.sh b/scripts/build/libc/glibc.sh
--- a/scripts/build/libc/glibc.sh
+++ b/scripts/build/libc/glibc.sh
@@ -55,3 +55,15 @@
 do_libc_check_config() {
     :
 }
+
+# Extract the files required for the libc locales
+# Nothing to do
+do_libc_locales_extract() {
+    :
+}
+
+# Build and install the libc locales
+# Not yet supported
+do_libc_locales() {
+    :
+}

[PATCH 1 of 5] scripts/libc: do not build add-ons by default

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311766042 -7200
# Node ID 0ee44cad2124dcf96979819f0ee4851dd6ef2334
# Parent  2651341cc084a0b6e5858f666c8a851fa6cbfa92
scripts/libc: do not build add-ons by default

Currently, no --enable-add-ons option is passed to libc configure when
"$(do_libc_add_ons_list ,)" is empty, which makes configure automatically search
for present add-ons. In that case, all present add-ons are built, although
no add-on was selected by the user in the config. Moreover, this can make the
configure fail if some non-standard add-ons like eglibc-localedef are present.

This behavior also leads to an inconsistency from a user point of view between
the following cases:
 - LIBC_ADDONS_LIST="", LIBC_GLIBC_USE_PORTS=n and THREADS="none" in the config,
   which makes "$(do_libc_add_ons_list ,)" return "", so all present add-ons
   are built.
 - LIBC_ADDONS_LIST="", LIBC_GLIBC_USE_PORTS=n and THREADS!="none" in the
   config, which makes "$(do_libc_add_ons_list ,)" return the add-on supporting
   the chosen threading implementation, e.g. "nptl", so only this add-on is
   built.

This patch disables the building of all add-ons in that case.

diff --git a/scripts/build/libc/glibc-eglibc.sh-common b/scripts/build/libc/glibc-eglibc.sh-common
--- a/scripts/build/libc/glibc-eglibc.sh-common
+++ b/scripts/build/libc/glibc-eglibc.sh-common
@@ -142,7 +142,7 @@
     fi
 
     case "$(do_libc_add_ons_list ,)" in
-        "") ;;
+        "") extra_config+=("--enable-add-ons=no");;
         *)  extra_config+=("--enable-add-ons=$(do_libc_add_ons_list ,)");;
     esac
 

[PATCH 0 of 5] libc internationalization support

[Benoît THÉBAUDEAU]

Hi Yann, all,

This is a new revision of my patch series adding support for locales. I have
tested it successfully with eglibc and glibc 2.13.

The changes are:
 - It is updated against the latest CT-NG changes.
 - It takes your comments into account.
 - It adds partial support for glibc locales.
 - It fixes the extraction/patch step of Linaro GCC 2011.07 (related to the
   previous patch I sent).

As to the alignment of uint32_t on 64-bit targets, it is target-specific, but so
far, a 32-bit alignment should be fine for all supported targets, which seems to
be confirmed by:
http://cgit.openembedded.org/cgit.cgi/openembedded/tree/classes/libc-package.bbclass
However, I do not have access to all these targets to check that this is correct.

Concerning the support of glibc locales, it works by building the whole native
glibc for the host, then it uses the built localedef to build the locales. I
have tried a partial build limited to localedef, but it does not work because of
make dependencies with other parts of the libc.

As explained in the patch, there is a huge limitation for glibc: The host and
target endianness and uint32_t alignment must be the same for the locales to be
supported by the target. Hence, I'm not certain this patch is worth keeping.
It's better than nothing, but it is not helpful for half the targets.

Fixing this would require a patch for each version of glibc to make its
localedef more like eglibc's (i.e. supporting endianness and alignment options,
and with a Makefile allowing to pass extra options to localedef). Another
solution would be to build eglibc's localedef natively for the host, then to use
it to build glibc's locales, but this may lead to compatibility issues. Issues
like this one show that glibc is really not suited for cross-compiling, which is
eglibc's job.

Also note that by default, eglibc's install-locales generates separate locale
files while glibc's puts all locales into usr/lib/locale/locale-archive.
eglibc's way of doing this is easier to handle for tools like BuildRoot to
remove unwanted locales, but unwanted locales can also be removed from the
archive using localedef. For eglibc, we could use --add-to-archive to force the
creation of an archive to be more consistent with glibc, but it would be
preferable the other way around to use --no-archive with glibc. Unfortunately,
glibc's install-locales does not accept extra options to pass to localedef,
which could be solved by adding a small wrapper script, but I think it is not
worth it since this is not really an issue.

Best regards,
Benoît

[PATCH 4 of 5] scripts/glibc: add partial support for locales

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311851750 -7200
# Node ID ac0f1f93198fca8e2f28b9499e197fa387c67234
# Parent  3b3715d0f1d412df91e472ed563d0cde0afedc6b
scripts/glibc: add partial support for locales

This patch adds partial support for glibc locales.

For now, it only generates the appropriate locales when the host and the target
have the same endianness and uint32_t alignment.

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/scripts/build/libc/glibc.sh b/scripts/build/libc/glibc.sh
--- a/scripts/build/libc/glibc.sh
+++ b/scripts/build/libc/glibc.sh
@@ -63,7 +63,58 @@
 }
 
 # Build and install the libc locales
-# Not yet supported
 do_libc_locales() {
-    :
+    local src_dir="${CT_SRC_DIR}/glibc-${CT_LIBC_VERSION}"
+    local -a extra_config
+
+    mkdir -p "${CT_BUILD_DIR}/build-localedef"
+    cd "${CT_BUILD_DIR}/build-localedef"
+
+    CT_DoLog EXTRA "Configuring C library localedef"
+
+    if [ "${CT_LIBC_EGLIBC_HAS_PKGVERSION_BUGURL}" = "y" ]; then
+        extra_config+=("--with-pkgversion=${CT_PKGVERSION}")
+        [ -n "${CT_TOOLCHAIN_BUGURL}" ] && extra_config+=("--with-bugurl=${CT_TOOLCHAIN_BUGURL}")
+    fi
+
+    CT_DoLog DEBUG "Extra config args passed: '${extra_config[*]}'"
+
+    # ./configure is misled by our tools override wrapper for bash
+    # so just tell it where the real bash is _on_the_target_!
+    # Notes:
+    # - ${ac_cv_path_BASH_SHELL} is only used to set BASH_SHELL
+    # - ${BASH_SHELL}            is only used to set BASH
+    # - ${BASH}                  is only used to set the shebang
+    #                            in two scripts to run on the target
+    # So we can safely bypass bash detection at compile time.
+    # Should this change in a future eglibc release, we'd better
+    # directly mangle the generated scripts _after_ they get built,
+    # or even after they get installed... eglibc is such a sucker...
+    echo "ac_cv_path_BASH_SHELL=/bin/bash" >>config.cache
+
+    # Configure with --prefix the way we want it on the target...
+
+    CT_DoExecLog CFG                                                \
+    CFLAGS="-O2 -fno-stack-protector -U_FORTIFY_SOURCE"             \
+    "${src_dir}/configure"                                          \
+        --prefix=/usr                                               \
+        --cache-file="$(pwd)/config.cache"                          \
+        --without-cvs                                               \
+        --disable-profile                                           \
+        --without-gd                                                \
+        --disable-debug                                             \
+        "${extra_config[@]}"
+
+    CT_DoLog EXTRA "Building C library localedef"
+    CT_DoExecLog ALL make ${JOBSFLAGS}
+
+    # The target's endianness and uint32_t alignment should be passed as options
+    # to localedef, but glibc's localedef does not support these options, which
+    # means that the locale files generated here will be suitable for the target
+    # only if it has the same endianness and uint32_t alignment as the host's.
+
+    CT_DoLog EXTRA "Installing C library locales"
+    CT_DoExecLog ALL make ${JOBSFLAGS}                              \
+                          install_root="${CT_SYSROOT_DIR}"          \
+                          localedata/install-locales
 }

Re: [PATCH 1 of 5] scripts/libc: do not build add-ons by default

[Yann E. MORIN]

Benoît, All,

On Thursday 28 July 2011 21:10:09 Benoît THÉBAUDEAU wrote:
> # HG changeset patch
> # User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> # Date 1311766042 -7200
> # Node ID 0ee44cad2124dcf96979819f0ee4851dd6ef2334
> # Parent  2651341cc084a0b6e5858f666c8a851fa6cbfa92
> scripts/libc: do not build add-ons by default
[--SNIP--]
> This patch disables the building of all add-ons in that case.

You forgot your SoB. Baring that, the patch is OK. Thank you! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [PATCH 2 of 5] libc: create an infrastructure to build and install the libc locales

[Yann E. MORIN]

Benoît, All,

On Thursday 28 July 2011 21:10:10 Benoît THÉBAUDEAU wrote:
> # HG changeset patch
> # User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> # Date 1311780145 -7200
> # Node ID 4fed02058cc3276180d6014899062f83e620eb37
> # Parent  0ee44cad2124dcf96979819f0ee4851dd6ef2334
> libc: create an infrastructure to build and install the libc locales
[--SNIP--]
> Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

Patch looks good.

One point, tough: there is no need to duplicate the two new functions in
each build scripts. The common part exists for a reason: to hold all the
common functions. The specific scripts can override the functions if they
need to.

Then, patch-3 would add such an override for eglibc, and patch-4 for glibc.
And a final patch (4.5 :-)) would remove the now-unneeded common functions.

Granted, this is "not a big deal". But I believe this is The Right Thing To
Do (TM). ;-)

Otherwise, looks good. Thanks! :-)

Regards,
Yann E. MORIN.

> diff --git a/config/libc/glibc-eglibc.in-common b/config/libc/glibc-eglibc.in-common
> --- a/config/libc/glibc-eglibc.in-common
> +++ b/config/libc/glibc-eglibc.in-common
> @@ -144,6 +144,13 @@
>  comment "|  or try again later... :-(                            "
>  endif
>  
> +config LIBC_LOCALES
> +    bool
> +    prompt "Build and install locales"
> +    help
> +      Whether to build and install the libc locale files for the target,
> +      which is required in order to support internationalization.
> +
>  if KERNEL_linux
>  
>  choice LIBC_GLIBC_SUPPORTED_KERNEL
> diff --git a/scripts/build/libc/eglibc.sh b/scripts/build/libc/eglibc.sh
> --- a/scripts/build/libc/eglibc.sh
> +++ b/scripts/build/libc/eglibc.sh
> @@ -95,3 +95,15 @@
>  
>      CT_EndStep
>  }
> +
> +# Extract the files required for the libc locales
> +# Nothing to do
> +do_libc_locales_extract() {
> +    :
> +}
> +
> +# Build and install the libc locales
> +# Not yet supported
> +do_libc_locales() {
> +    :
> +}
> diff --git a/scripts/build/libc/glibc-eglibc.sh-common b/scripts/build/libc/glibc-eglibc.sh-common
> --- a/scripts/build/libc/glibc-eglibc.sh-common
> +++ b/scripts/build/libc/glibc-eglibc.sh-common
> @@ -47,6 +47,10 @@
>      find . -type f -name configure -exec touch {} \; 2>&1 |CT_DoLog ALL
>  
>      CT_Popd
> +
> +    if [ "${CT_LIBC_LOCALES}" = "y" ]; then
> +        do_libc_locales_extract
> +    fi
>  }
>  
>  # Build and install headers and start files
> @@ -294,6 +298,10 @@
>                                "${extra_make_args[@]}"           \
>                                install_root="${CT_SYSROOT_DIR}"  \
>                                install
> +
> +        if [ "${CT_LIBC_LOCALES}" = "y" ]; then
> +            do_libc_locales
> +        fi
>      fi
>  
>      CT_EndStep
> diff --git a/scripts/build/libc/glibc.sh b/scripts/build/libc/glibc.sh
> --- a/scripts/build/libc/glibc.sh
> +++ b/scripts/build/libc/glibc.sh
> @@ -55,3 +55,15 @@
>  do_libc_check_config() {
>      :
>  }
> +
> +# Extract the files required for the libc locales
> +# Nothing to do
> +do_libc_locales_extract() {
> +    :
> +}
> +
> +# Build and install the libc locales
> +# Not yet supported
> +do_libc_locales() {
> +    :
> +}
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

[PATCH] scripts/libc: do not build add-ons by default

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311883771 -7200
# Node ID 424fa2092ace47d4d2f417d40f372293a2f02f26
# Parent  5711f94223290986b14aec6c43c9628acc7e073d
scripts/libc: do not build add-ons by default

Currently, no --enable-add-ons option is passed to libc configure when
"$(do_libc_add_ons_list ,)" is empty, which makes configure automatically search
for present add-ons. In that case, all present add-ons are built, although
no add-on was selected by the user in the config. Moreover, this can make the
configure fail if some non-standard add-ons like eglibc-localedef are present.

This behavior also leads to an inconsistency from a user point of view between
the following cases:
 - LIBC_ADDONS_LIST="", LIBC_GLIBC_USE_PORTS=n and THREADS="none" in the config,
   which makes "$(do_libc_add_ons_list ,)" return "", so all present add-ons
   are built.
 - LIBC_ADDONS_LIST="", LIBC_GLIBC_USE_PORTS=n and THREADS!="none" in the
   config, which makes "$(do_libc_add_ons_list ,)" return the add-on supporting
   the chosen threading implementation, e.g. "nptl", so only this add-on is
   built.

This patch disables the building of all add-ons in that case.

It is still possible to build all present add-ons by adding --enable-add-ons to
LIBC_GLIBC_EXTRA_CONFIG_ARRAY.

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/scripts/build/libc/glibc-eglibc.sh-common b/scripts/build/libc/glibc-eglibc.sh-common
--- a/scripts/build/libc/glibc-eglibc.sh-common
+++ b/scripts/build/libc/glibc-eglibc.sh-common
@@ -142,7 +142,7 @@
     fi
 
     case "$(do_libc_add_ons_list ,)" in
-        "") ;;
+        "") extra_config+=("--enable-add-ons=no");;
         *)  extra_config+=("--enable-add-ons=$(do_libc_add_ons_list ,)");;
     esac
 

Re: [PATCH 3 of 5] scripts/eglibc: add support for locales

[Yann E. MORIN]

Benoît, All,

On Thursday 28 July 2011 21:10:11 Benoît THÉBAUDEAU wrote:
> # HG changeset patch
> # User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> # Date 1311851059 -7200
> # Node ID 3b3715d0f1d412df91e472ed563d0cde0afedc6b
> # Parent  4fed02058cc3276180d6014899062f83e620eb37
> scripts/eglibc: add support for locales

Patch is globally good, but for a few issues, see inlined below.

> Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> 
> diff --git a/scripts/build/libc/eglibc.sh b/scripts/build/libc/eglibc.sh
> --- a/scripts/build/libc/eglibc.sh
> +++ b/scripts/build/libc/eglibc.sh
> @@ -14,6 +14,7 @@
>  # snapshots available.
>  do_libc_get() {
>      local addon
> +    local -a extra_addons
>      local svn_base
>  
>      if [ "${CT_EGLIBC_HTTP}" = "y" ]; then
> @@ -31,7 +32,11 @@
>                "${svn_base}/libc"            \
>                "${CT_EGLIBC_REVISION:-HEAD}"
>  
> -    for addon in $(do_libc_add_ons_list " "); do
> +    if [ "${CT_LIBC_LOCALES}" = "y" ]; then
> +        extra_addons+=("localedef")
> +    fi
> +
> +    for addon in $(do_libc_add_ons_list " ") ${extra_addons[@]}; do

Always quote variable expansion, unless it is strictly required not to. In
the array-variables, expansion of "${foo[@]}" expands to as many entries
there are in the array, not to a single entry, exactly like "$@" does for
args. Similarly, "${foo[*]}" behaves like "$*".

Here, we want a word for each entry in the array 'extra_addons', even if
those entries have embedded spaces (I doubt any sane add-on would ever
have a space in its name, but never ever trust developpers' insanity after
10+ coffees and a 20+ hours stuck in front of a screen! :-])

>          # Never ever try to download these add-ons,
>          # they've always been internal
>          case "${addon}" in
> @@ -97,13 +102,69 @@
>  }
>  
>  # Extract the files required for the libc locales
> -# Nothing to do
>  do_libc_locales_extract() {
> -    :
> +    CT_Extract "eglibc-localedef-${CT_LIBC_VERSION}"
> +    CT_Patch "eglibc" "localedef-${CT_LIBC_VERSION}"
>  }
>  
>  # Build and install the libc locales
> -# Not yet supported
>  do_libc_locales() {
> -    :
> +    local libc_src_dir="${CT_SRC_DIR}/eglibc-${CT_LIBC_VERSION}"
> +    local src_dir="${CT_SRC_DIR}/eglibc-localedef-${CT_LIBC_VERSION}"
> +    local -a extra_config
> +    local -a localedef_opts
> +
> +    mkdir -p "${CT_BUILD_DIR}/build-localedef"
> +    cd "${CT_BUILD_DIR}/build-localedef"
> +
> +    CT_DoLog EXTRA "Configuring C library localedef"
> +
> +    if [ "${CT_LIBC_EGLIBC_HAS_PKGVERSION_BUGURL}" = "y" ]; then
> +        extra_config+=("--with-pkgversion=${CT_PKGVERSION}")
> +        [ -n "${CT_TOOLCHAIN_BUGURL}" ] && extra_config+=("--with-bugurl=${CT_TOOLCHAIN_BUGURL}")
> +    fi
> +
> +    CT_DoLog DEBUG "Extra config args passed: '${extra_config[*]}'"
> +
> +    # ./configure is misled by our tools override wrapper for bash
> +    # so just tell it where the real bash is _on_the_target_!
> +    # Notes:
> +    # - ${ac_cv_path_BASH_SHELL} is only used to set BASH_SHELL
> +    # - ${BASH_SHELL}            is only used to set BASH
> +    # - ${BASH}                  is only used to set the shebang
> +    #                            in two scripts to run on the target
> +    # So we can safely bypass bash detection at compile time.
> +    # Should this change in a future eglibc release, we'd better
> +    # directly mangle the generated scripts _after_ they get built,
> +    # or even after they get installed... eglibc is such a sucker...
> +    echo "ac_cv_path_BASH_SHELL=/bin/bash" >>config.cache
> +
> +    # Configure with --prefix the way we want it on the target...
> +
> +    CT_DoExecLog CFG                                                \
> +    "${src_dir}/configure"                                          \
> +        --prefix=/usr                                               \
> +        --cache-file="$(pwd)/config.cache"                          \
> +        --with-glibc="${libc_src_dir}"                              \
> +        "${extra_config[@]}"
> +
> +    CT_DoLog EXTRA "Building C library localedef"
> +    CT_DoExecLog ALL make ${JOBSFLAGS}
> +
> +    # Set the localedef endianness option
> +    case "${CT_ARCH_BE},${CT_ARCH_LE}" in
> +        y,) localedef_opts+=(--big-endian);;
> +        ,y) localedef_opts+=(--little-endian);;
> +    esac
> +
> +    # Set the localedef option for the target's uint32_t alignment in bytes.
> +    # This is target-specific, but for now, 32-bit alignment should work for all
> +    # supported targets, even 64-bit ones.
> +    localedef_opts+=(--uint32-align=4)
> +
> +    CT_DoLog EXTRA "Installing C library locales"
> +    CT_DoExecLog ALL make ${JOBSFLAGS}                              \
> +                          "LOCALEDEF_OPTS=${localedef_opts[*]}"     \
> +                          install_root="${CT_SYSROOT_DIR}"          \
> +                          install-locales
>  }

Otherwise looks good. Thank you!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [PATCH 4 of 5] scripts/glibc: add partial support for locales

[Yann E. MORIN]

Benoît, All,

On Thursday 28 July 2011 21:10:12 Benoît THÉBAUDEAU wrote:
> # HG changeset patch
> # User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> # Date 1311851750 -7200
> # Node ID ac0f1f93198fca8e2f28b9499e197fa387c67234
> # Parent  3b3715d0f1d412df91e472ed563d0cde0afedc6b
> scripts/glibc: add partial support for locales
> 
> This patch adds partial support for glibc locales.
> 
> For now, it only generates the appropriate locales when the host and the target
> have the same endianness and uint32_t alignment.

It would at least warrant a warning in the menuconfig, such as:

if LIBC_eglibc && LIBC_LOCALES
comment "Warning! this only works if this machine and the target:"
comment "         - are the same bitness"
comment "         - and have the same requirements for the alignment of uint32_t."
comment "         You will have to check by yourself (for now)."
endif # glibc && locales

Plus, see comments inlined below...

> Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> 
> diff --git a/scripts/build/libc/glibc.sh b/scripts/build/libc/glibc.sh
> --- a/scripts/build/libc/glibc.sh
> +++ b/scripts/build/libc/glibc.sh
> @@ -63,7 +63,58 @@
>  }
>  
>  # Build and install the libc locales
> -# Not yet supported
>  do_libc_locales() {
> -    :
> +    local src_dir="${CT_SRC_DIR}/glibc-${CT_LIBC_VERSION}"
> +    local -a extra_config
> +
> +    mkdir -p "${CT_BUILD_DIR}/build-localedef"
> +    cd "${CT_BUILD_DIR}/build-localedef"
> +
> +    CT_DoLog EXTRA "Configuring C library localedef"
> +
> +    if [ "${CT_LIBC_EGLIBC_HAS_PKGVERSION_BUGURL}" = "y" ]; then
> +        extra_config+=("--with-pkgversion=${CT_PKGVERSION}")
> +        [ -n "${CT_TOOLCHAIN_BUGURL}" ] && extra_config+=("--with-bugurl=${CT_TOOLCHAIN_BUGURL}")
> +    fi
> +
> +    CT_DoLog DEBUG "Extra config args passed: '${extra_config[*]}'"
> +
> +    # ./configure is misled by our tools override wrapper for bash
> +    # so just tell it where the real bash is _on_the_target_!
> +    # Notes:
> +    # - ${ac_cv_path_BASH_SHELL} is only used to set BASH_SHELL
> +    # - ${BASH_SHELL}            is only used to set BASH
> +    # - ${BASH}                  is only used to set the shebang
> +    #                            in two scripts to run on the target
> +    # So we can safely bypass bash detection at compile time.
> +    # Should this change in a future eglibc release, we'd better
> +    # directly mangle the generated scripts _after_ they get built,
> +    # or even after they get installed... eglibc is such a sucker...
> +    echo "ac_cv_path_BASH_SHELL=/bin/bash" >>config.cache
> +
> +    # Configure with --prefix the way we want it on the target...
> +
> +    CT_DoExecLog CFG                                                \
> +    CFLAGS="-O2 -fno-stack-protector -U_FORTIFY_SOURCE"             \

Although, we are building for the build machine here, not for target,
-U_FORTIFY_SOURCE should be conditional on ! LIBC_ENABLE_FORTIFIED_BUILD

> +    "${src_dir}/configure"                                          \
> +        --prefix=/usr                                               \
> +        --cache-file="$(pwd)/config.cache"                          \
> +        --without-cvs                                               \
> +        --disable-profile                                           \
> +        --without-gd                                                \
> +        --disable-debug                                             \
> +        "${extra_config[@]}"
> +
> +    CT_DoLog EXTRA "Building C library localedef"
> +    CT_DoExecLog ALL make ${JOBSFLAGS}

He! Wait! Does that mean we do build a complete glibc just for the sake of
having a single binary 'localedef' ? Woosh...

I'm OK with that if that's what it takes to have locales on the target, but
we should definitely try to find a better make target later, if possible.

> +    # The target's endianness and uint32_t alignment should be passed as options
> +    # to localedef, but glibc's localedef does not support these options, which
> +    # means that the locale files generated here will be suitable for the target
> +    # only if it has the same endianness and uint32_t alignment as the host's.
> +
> +    CT_DoLog EXTRA "Installing C library locales"
> +    CT_DoExecLog ALL make ${JOBSFLAGS}                              \
> +                          install_root="${CT_SYSROOT_DIR}"          \
> +                          localedata/install-locales
>  }
> 

Otherwise, looks good.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [PATCH 5 of 5] scripts/functions: force extract folder to archive basename

[Yann E. MORIN]

Benoît, All,

On Thursday 28 July 2011 21:10:13 Benoît THÉBAUDEAU wrote:
> # HG changeset patch
> # User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> # Date 1311867232 -7200
> # Node ID 50c28edc0a0ebe6bf285b6656233365e2ed88df0
> # Parent  ac0f1f93198fca8e2f28b9499e197fa387c67234
> scripts/functions: force extract folder to archive basename
> 
> Some archives like those of the 2011.07 revisions of Linaro GCC contain a folder
> name different from the archive basename, which leads to errors afterwards, e.g.
> when patching. E.g.:
> gcc-linaro-4.5-2011.07.tar.bz2 extracts to gcc-linaro-4.5-2011.07-0/
> 
> This patch changes CT_Extract() to force the extraction of all archives to a
> folder named like the archive basename. E.g.:
> gcc-linaro-4.5-2011.07.tar.bz2 now extracts to gcc-linaro-4.5-2011.07/
> 
> Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> 
> diff --git a/scripts/functions b/scripts/functions
> --- a/scripts/functions
> +++ b/scripts/functions
> @@ -728,10 +728,15 @@
>      CT_Pushd "${nochdir}"
>  
>      CT_DoLog EXTRA "Extracting '${basename}'"
> +    CT_DoExecLog FILE mkdir -p "${basename}"
>      case "${ext}" in
> -        .tar.bz2)     CT_DoExecLog FILE tar xvjf "${full_file}";;
> -        .tar.gz|.tgz) CT_DoExecLog FILE tar xvzf "${full_file}";;
> -        .tar)         CT_DoExecLog FILE tar xvf  "${full_file}";;
> +        .tar.bz2)     { bzcat "${full_file}" \
> +                        |tar --strip-components=1 -C "${basename}" -xvf - ;
> +                      } 2>&1 |CT_DoLog FILE;;

Why don't you use:
  CT_DoExecLog FILE tar --strip-components=1 -C "${basename}" -xvjf "${full_file}";;
...

> +        .tar.gz|.tgz) { gzip -d -c "${full_file}" \
> +                        |tar --strip-components=1 -C "${basename}" -xvf - ;
> +                      } 2>&1 |CT_DoLog FILE;;

... and here with -xvzf ...

> +        .tar)         CT_DoExecLog FILE tar --strip-components=1 -C "${basename}" -xvf "${full_file}";;

... while you are using it here?

>          /.git)        CT_ExtractGit "${basename}" "${@}";;
>          *)            CT_DoLog WARN "Don't know how to handle '${basename}${ext}': unknown extension"
>                        return 1
> 

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [PATCH] scripts/libc: do not build add-ons by default

[Yann E. MORIN]

On Thursday 28 July 2011 22:11:38 Benoît THÉBAUDEAU wrote:
> # HG changeset patch
> # User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> # Date 1311883771 -7200
> # Node ID 424fa2092ace47d4d2f417d40f372293a2f02f26
> # Parent  5711f94223290986b14aec6c43c9628acc7e073d
> scripts/libc: do not build add-ons by default

Applied as #424fa2092ace. Thank you!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

[PATCH 3 of 5] libc/eglibc: add support for locales

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311943340 -7200
# Node ID 09e1f8b35c4a4728aee785c346cdc6e55b08cd24
# Parent  712091002f632e58829dfbb6781938025dba7211
libc/eglibc: add support for locales

This patch adds support for eglibc locales.

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/scripts/build/libc/eglibc.sh b/scripts/build/libc/eglibc.sh
--- a/scripts/build/libc/eglibc.sh
+++ b/scripts/build/libc/eglibc.sh
@@ -14,6 +14,7 @@
 # snapshots available.
 do_libc_get() {
     local addon
+    local -a extra_addons
     local svn_base
 
     if [ "${CT_EGLIBC_HTTP}" = "y" ]; then
@@ -31,7 +32,11 @@
               "${svn_base}/libc"            \
               "${CT_EGLIBC_REVISION:-HEAD}"
 
-    for addon in $(do_libc_add_ons_list " "); do
+    if [ "${CT_LIBC_LOCALES}" = "y" ]; then
+        extra_addons+=("localedef")
+    fi
+
+    for addon in $(do_libc_add_ons_list " ") "${extra_addons[@]}"; do
         # Never ever try to download these add-ons,
         # they've always been internal
         case "${addon}" in
@@ -95,3 +100,71 @@
 
     CT_EndStep
 }
+
+# Extract the files required for the libc locales
+do_libc_locales_extract() {
+    CT_Extract "eglibc-localedef-${CT_LIBC_VERSION}"
+    CT_Patch "eglibc" "localedef-${CT_LIBC_VERSION}"
+}
+
+# Build and install the libc locales
+do_libc_locales() {
+    local libc_src_dir="${CT_SRC_DIR}/eglibc-${CT_LIBC_VERSION}"
+    local src_dir="${CT_SRC_DIR}/eglibc-localedef-${CT_LIBC_VERSION}"
+    local -a extra_config
+    local -a localedef_opts
+
+    mkdir -p "${CT_BUILD_DIR}/build-localedef"
+    cd "${CT_BUILD_DIR}/build-localedef"
+
+    CT_DoLog EXTRA "Configuring C library localedef"
+
+    if [ "${CT_LIBC_EGLIBC_HAS_PKGVERSION_BUGURL}" = "y" ]; then
+        extra_config+=("--with-pkgversion=${CT_PKGVERSION}")
+        [ -n "${CT_TOOLCHAIN_BUGURL}" ] && extra_config+=("--with-bugurl=${CT_TOOLCHAIN_BUGURL}")
+    fi
+
+    CT_DoLog DEBUG "Extra config args passed: '${extra_config[*]}'"
+
+    # ./configure is misled by our tools override wrapper for bash
+    # so just tell it where the real bash is _on_the_target_!
+    # Notes:
+    # - ${ac_cv_path_BASH_SHELL} is only used to set BASH_SHELL
+    # - ${BASH_SHELL}            is only used to set BASH
+    # - ${BASH}                  is only used to set the shebang
+    #                            in two scripts to run on the target
+    # So we can safely bypass bash detection at compile time.
+    # Should this change in a future eglibc release, we'd better
+    # directly mangle the generated scripts _after_ they get built,
+    # or even after they get installed... eglibc is such a sucker...
+    echo "ac_cv_path_BASH_SHELL=/bin/bash" >>config.cache
+
+    # Configure with --prefix the way we want it on the target...
+
+    CT_DoExecLog CFG                                                \
+    "${src_dir}/configure"                                          \
+        --prefix=/usr                                               \
+        --cache-file="$(pwd)/config.cache"                          \
+        --with-glibc="${libc_src_dir}"                              \
+        "${extra_config[@]}"
+
+    CT_DoLog EXTRA "Building C library localedef"
+    CT_DoExecLog ALL make ${JOBSFLAGS}
+
+    # Set the localedef endianness option
+    case "${CT_ARCH_BE},${CT_ARCH_LE}" in
+        y,) localedef_opts+=(--big-endian);;
+        ,y) localedef_opts+=(--little-endian);;
+    esac
+
+    # Set the localedef option for the target's uint32_t alignment in bytes.
+    # This is target-specific, but for now, 32-bit alignment should work for all
+    # supported targets, even 64-bit ones.
+    localedef_opts+=(--uint32-align=4)
+
+    CT_DoLog EXTRA "Installing C library locales"
+    CT_DoExecLog ALL make ${JOBSFLAGS}                              \
+                          "LOCALEDEF_OPTS=${localedef_opts[*]}"     \
+                          install_root="${CT_SYSROOT_DIR}"          \
+                          install-locales
+}

[PATCH 1 of 5] scripts/functions: force extract folder to archive basename

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311937489 -7200
# Node ID 6e1412ba8da9b9b8c100482fce442bf83d520f63
# Parent  424fa2092ace47d4d2f417d40f372293a2f02f26
scripts/functions: force extract folder to archive basename

Some archives like those of the 2011.07 revisions of Linaro GCC contain a folder
name different from the archive basename, which leads to errors afterwards, e.g.
when patching. E.g.:
gcc-linaro-4.5-2011.07.tar.bz2 extracts to gcc-linaro-4.5-2011.07-0/

This patch changes CT_Extract() to force the extraction of all archives to a
folder named like the archive basename. E.g.:
gcc-linaro-4.5-2011.07.tar.bz2 now extracts to gcc-linaro-4.5-2011.07/

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/scripts/functions b/scripts/functions
--- a/scripts/functions
+++ b/scripts/functions
@@ -728,10 +728,11 @@
     CT_Pushd "${nochdir}"
 
     CT_DoLog EXTRA "Extracting '${basename}'"
+    CT_DoExecLog FILE mkdir -p "${basename}"
     case "${ext}" in
-        .tar.bz2)     CT_DoExecLog FILE tar xvjf "${full_file}";;
-        .tar.gz|.tgz) CT_DoExecLog FILE tar xvzf "${full_file}";;
-        .tar)         CT_DoExecLog FILE tar xvf  "${full_file}";;
+        .tar.bz2)     CT_DoExecLog FILE tar --strip-components=1 -C "${basename}" -xvjf "${full_file}";;
+        .tar.gz|.tgz) CT_DoExecLog FILE tar --strip-components=1 -C "${basename}" -xvzf "${full_file}";;
+        .tar)         CT_DoExecLog FILE tar --strip-components=1 -C "${basename}" -xvf "${full_file}";;
         /.git)        CT_ExtractGit "${basename}" "${@}";;
         *)            CT_DoLog WARN "Don't know how to handle '${basename}${ext}': unknown extension"
                       return 1

[PATCH 2 of 5] libc: create an infrastructure to build and install the libc locales

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311938757 -7200
# Node ID 712091002f632e58829dfbb6781938025dba7211
# Parent  6e1412ba8da9b9b8c100482fce442bf83d520f63
libc: create an infrastructure to build and install the libc locales

This patch adds a common glibc/eglibc infrastructure to build and install the
libc locales.

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/config/libc/glibc-eglibc.in-common b/config/libc/glibc-eglibc.in-common
--- a/config/libc/glibc-eglibc.in-common
+++ b/config/libc/glibc-eglibc.in-common
@@ -144,6 +144,13 @@
 comment "|  or try again later... :-(                            "
 endif
 
+config LIBC_LOCALES
+    bool
+    prompt "Build and install locales"
+    help
+      Whether to build and install the libc locale files for the target,
+      which is required in order to support internationalization.
+
 if KERNEL_linux
 
 choice LIBC_GLIBC_SUPPORTED_KERNEL
diff --git a/scripts/build/libc/glibc-eglibc.sh-common b/scripts/build/libc/glibc-eglibc.sh-common
--- a/scripts/build/libc/glibc-eglibc.sh-common
+++ b/scripts/build/libc/glibc-eglibc.sh-common
@@ -47,6 +47,10 @@
     find . -type f -name configure -exec touch {} \; 2>&1 |CT_DoLog ALL
 
     CT_Popd
+
+    if [ "${CT_LIBC_LOCALES}" = "y" ]; then
+        do_libc_locales_extract
+    fi
 }
 
 # Build and install headers and start files
@@ -294,6 +298,10 @@
                               "${extra_make_args[@]}"           \
                               install_root="${CT_SYSROOT_DIR}"  \
                               install
+
+        if [ "${CT_LIBC_LOCALES}" = "y" ]; then
+            do_libc_locales
+        fi
     fi
 
     CT_EndStep
@@ -353,3 +361,15 @@
             ;;
     esac
 }
+
+# Extract the files required for the libc locales
+# Nothing to do by default
+do_libc_locales_extract() {
+    :
+}
+
+# Build and install the libc locales
+# Nothing to do by default
+do_libc_locales() {
+    :
+}

[PATCH 0 of 5] libc internationalization support

[Benoît THÉBAUDEAU]

Hi Yann, all,

This is a new revision taking your comments into account.

Concerning the full build of glibc, see what I said in my patch 0 yesterday.

Best regards,
Benoît

[PATCH 4 of 5] libc/glibc: add partial support for locales

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311946253 -7200
# Node ID b05b2e6bc26037963b24353f25dbd7be27d28b77
# Parent  09e1f8b35c4a4728aee785c346cdc6e55b08cd24
libc/glibc: add partial support for locales

This patch adds partial support for glibc locales.

For now, it only generates the appropriate locales when the host and the target
have the same endianness and uint32_t alignment.

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/config/libc/glibc-eglibc.in-common b/config/libc/glibc-eglibc.in-common
--- a/config/libc/glibc-eglibc.in-common
+++ b/config/libc/glibc-eglibc.in-common
@@ -151,6 +151,15 @@
       Whether to build and install the libc locale files for the target,
       which is required in order to support internationalization.
 
+if LIBC_glibc && LIBC_LOCALES
+comment "WARNING!                                                     "
+comment "|  The built locales will be usable if and only if the build  "
+comment "|  machine and the target:                                    "
+comment "|   - have the same endianness,                               "
+comment "|   - and have the same alignment requirements for uint32_t.  "
+comment "|  You will have to check by yourself (for now).              "
+endif # LIBC_glibc && LIBC_LOCALES
+
 if KERNEL_linux
 
 choice LIBC_GLIBC_SUPPORTED_KERNEL
diff --git a/scripts/build/libc/glibc.sh b/scripts/build/libc/glibc.sh
--- a/scripts/build/libc/glibc.sh
+++ b/scripts/build/libc/glibc.sh
@@ -55,3 +55,73 @@
 do_libc_check_config() {
     :
 }
+
+# Extract the files required for the libc locales
+# Nothing to do
+do_libc_locales_extract() {
+    :
+}
+
+# Build and install the libc locales
+do_libc_locales() {
+    local src_dir="${CT_SRC_DIR}/glibc-${CT_LIBC_VERSION}"
+    local -a extra_config
+    local glibc_cflags
+
+    mkdir -p "${CT_BUILD_DIR}/build-localedef"
+    cd "${CT_BUILD_DIR}/build-localedef"
+
+    CT_DoLog EXTRA "Configuring C library localedef"
+
+    if [ "${CT_LIBC_EGLIBC_HAS_PKGVERSION_BUGURL}" = "y" ]; then
+        extra_config+=("--with-pkgversion=${CT_PKGVERSION}")
+        [ -n "${CT_TOOLCHAIN_BUGURL}" ] && extra_config+=("--with-bugurl=${CT_TOOLCHAIN_BUGURL}")
+    fi
+
+    CT_DoLog DEBUG "Extra config args passed: '${extra_config[*]}'"
+
+    glibc_cflags="-O2 -fno-stack-protector"
+    case "${CT_LIBC_ENABLE_FORTIFIED_BUILD}" in
+        y)  ;;
+        *)  glibc_cflags+=" -U_FORTIFY_SOURCE";;
+    esac
+
+    # ./configure is misled by our tools override wrapper for bash
+    # so just tell it where the real bash is _on_the_target_!
+    # Notes:
+    # - ${ac_cv_path_BASH_SHELL} is only used to set BASH_SHELL
+    # - ${BASH_SHELL}            is only used to set BASH
+    # - ${BASH}                  is only used to set the shebang
+    #                            in two scripts to run on the target
+    # So we can safely bypass bash detection at compile time.
+    # Should this change in a future eglibc release, we'd better
+    # directly mangle the generated scripts _after_ they get built,
+    # or even after they get installed... eglibc is such a sucker...
+    echo "ac_cv_path_BASH_SHELL=/bin/bash" >>config.cache
+
+    # Configure with --prefix the way we want it on the target...
+
+    CT_DoExecLog CFG                                                \
+    CFLAGS="${glibc_cflags}"                                        \
+    "${src_dir}/configure"                                          \
+        --prefix=/usr                                               \
+        --cache-file="$(pwd)/config.cache"                          \
+        --without-cvs                                               \
+        --disable-profile                                           \
+        --without-gd                                                \
+        --disable-debug                                             \
+        "${extra_config[@]}"
+
+    CT_DoLog EXTRA "Building C library localedef"
+    CT_DoExecLog ALL make ${JOBSFLAGS}
+
+    # The target's endianness and uint32_t alignment should be passed as options
+    # to localedef, but glibc's localedef does not support these options, which
+    # means that the locale files generated here will be suitable for the target
+    # only if it has the same endianness and uint32_t alignment as the host's.
+
+    CT_DoLog EXTRA "Installing C library locales"
+    CT_DoExecLog ALL make ${JOBSFLAGS}                              \
+                          install_root="${CT_SYSROOT_DIR}"          \
+                          localedata/install-locales
+}

[PATCH 5 of 5] libc: remove unneeded glibc/eglibc common functions

[Benoît THÉBAUDEAU]

# HG changeset patch
# User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
# Date 1311946258 -7200
# Node ID aacdd4dc7f79d2b85b8d943e51479d1fd235de76
# Parent  b05b2e6bc26037963b24353f25dbd7be27d28b77
libc: remove unneeded glibc/eglibc common functions

do_libc_locales_extract() and do_libc_locales() in glibc-eglibc.sh-common have
been overridden for both glibc and eglibc, so they can now be removed, which
this patch does.

Signed-off-by: "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>

diff --git a/scripts/build/libc/glibc-eglibc.sh-common b/scripts/build/libc/glibc-eglibc.sh-common
--- a/scripts/build/libc/glibc-eglibc.sh-common
+++ b/scripts/build/libc/glibc-eglibc.sh-common
@@ -361,15 +361,3 @@
             ;;
     esac
 }
-
-# Extract the files required for the libc locales
-# Nothing to do by default
-do_libc_locales_extract() {
-    :
-}
-
-# Build and install the libc locales
-# Nothing to do by default
-do_libc_locales() {
-    :
-}

Re: [PATCH 1 of 5] scripts/functions: force extract folder to archive basename

[Yann E. MORIN]

Benoît, All,

On Friday 29 July 2011 15:42:15 Benoît THÉBAUDEAU wrote:
> # HG changeset patch
> # User "Benoît THÉBAUDEAU" <benoit.thebaudeau@advansee.com>
> # Date 1311937489 -7200
> # Node ID 6e1412ba8da9b9b8c100482fce442bf83d520f63
> # Parent  424fa2092ace47d4d2f417d40f372293a2f02f26
> scripts/functions: force extract folder to archive basename

Pushed as #6e1412ba8da9. Thank you!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [PATCH 0 of 5] libc internationalization support

[Yann E. MORIN]

Benoît, All,

On Friday 29 July 2011 15:42:14 Benoît THÉBAUDEAU wrote:
> This is a new revision taking your comments into account.

Thank you. Except for the extract-related patch which I already pushed,
I'll handle the remaining four a bit later (week has been quite busy,
and I'm really exhausted...). But overall, looks good now! Thank you!

> Concerning the full build of glibc, see what I said in my patch 0 yesterday.

Yes, sorry, I totally missed the introductory message. Apologies...

And I totally agree with you analysis. I would just add my own two
thoughts on the subject.

First, if one wants the BIG library for his/her embedded product, one will
most probably be willing to pay the price for the full-blown archive with
all locales in it, so that's not a real issue, I believe.

Second, maintaining a patch against every and all glibc versions just for
the locales is just too much trouble. If the target and host do not have
the same endiannes and/or unit32_t alignment, then it is still possible to
switch to using eglibc instead of plain glibc.

Thank you very much for the thorough explanations!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [PATCH 0 of 5] libc internationalization support

[Benoît THÉBAUDEAU]

Hi Yann, all,

> > This is a new revision taking your comments into account.
>
> Thank you. Except for the extract-related patch which I already
> pushed,
> I'll handle the remaining four a bit later (week has been quite busy,
> and I'm really exhausted...). But overall, looks good now! Thank you!

No problem. But did you forget to apply them for 1.12.0, or did you
want to apply them later?

Best regards,
Benoît

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [PATCH 0 of 5] libc internationalization support

[Yann E. MORIN]

Benoît, All,

On Monday 01 August 2011 131337 Benoît THÉBAUDEAU wrote:
> > I'll handle the remaining four a bit later (week has been quite busy,
> > and I'm really exhausted...). But overall, looks good now! Thank you!
> 
> No problem. But did you forget to apply them for 1.12.0, or did you
> want to apply them later?

Yes, that was on-purpose. It was too short a delay for 1.12.0. They are still
pending in my local tree for some more testing. This WE has been terrible,
getting 1.12.0 out; although my machine is pretty fast, re-building all the
sample takes quite some time.

Thank you again for putting up with me...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +0/33 662376056 | Software  Designer | \ / CAMPAIGN     |   ^                |
| --==< O_o >==-- '------------.-------:  X  AGAINST      |  /e\  There is no  |
| http://ymorin.is-a-geek.org/ | (*_*) | / \ HTML MAIL    |  """  conspiracy.  |
'------------------------------'-------'------------------'--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: [PATCH 0 of 5] libc internationalization support

[Yann E. MORIN]

Benoît, All,

On Friday 29 July 2011 15:42:14 Benoît THÉBAUDEAU wrote:
> This is a new revision taking your comments into account.

Series applied as:
  #45ef0b0660a5
  #4f9efd2c6627
  #7727970d04e9
  #99532252143d

Thank you!

Regards,
Yann E. MORIN. 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq