Two or three stages gcc build?

Two or three stages gcc build?

[Thomas Petazzoni]

Hello Khem,

(This e-mail is Cc'ed to the crossgcc@ and the buildroot@ mailing lists)

We started discussing on IRC about whether a three stage or a two stage
gcc build was needed, but seems like due to timezone issues, we're not
able to discuss simultaneously, so I'm trying to do things by e-mail. I
would like to understand a little bit better since when we can get back
to a two stages gcc build process instead of the three stages build
process that was needed since the introduction of NPTL support.

Currently, what we have in Buildroot is:

 1. Build gcc initial (--without-headers --with-newlib --disable-shared)

 2. Configure uClibc, install headers, crti.o, crtn.o, crt1.o, install
    a fake libc.so and libm.so (just empty C files being compiled as a
    share library)

 3. Build gcc intermediate (no special options passed)

 4. Build uClibc and install it.

 5. Build gcc final

Apparently, this three stage build process is no longer needed. In
OE-Core, you did the following commit:

commit b0faebd1f07e1616004bd19664395932e7c2c48f
Author: Khem Raj <raj.khem@gmail.com>
Date:   Wed Aug 15 23:12:51 2012 -0700

    gcc-cross: Make gcc-cross-initial as the only intermediate gcc stage
    
    Now glibc can be compiled with gcc-cross-initial therefore prepare
    the stage to drop gcc-cross-intermediate
    
    Also drop arm-nolibfloat.patch should not be needed anymore
    half of changes in this patch are meant for OABI which we dont
    use anymore
    
    (From OE-Core rev: 30617bde61a3b0a0944b49a0c9fb7159dacbb19f)
    
    Signed-off-by: Khem Raj <raj.khem@gmail.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

So according to this, the fact that we can go back to a two stages
build seems to be related to glibc. So what happens to uClibc in this
case?

In fe9a576e8d27460468dfe5eac90aad49ab26a8d3 (gcc-cross-intermediate,
gcc-crosssdk-intermediate: Remove) you completely remove the
intermediate recipes, so I assume they are really no longer needed,
even by the uClibc build, since I believe OE-core also supports uClibc
builds (thanks to your work).

Could you explain in more details which version of gcc or glibc/uClibc
made it possible to go back to a 2-stages build? Both Yann E. Morin
(for Crosstool-NG) and myself (for Buildroot) are interested in
understanding this in order to improve those tools. Also, what should
be the two stages now? Just:

 1. Build gcc initial (no C library at all)
 2. Build C library completely
 3. Build gcc final

 ?

Thanks for your insights,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Khem Raj]

Hi Thomas

I am late in reply but here it is.

On Jul 5, 2013, at 2:38 PM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:

> Hello Khem,
> 
> (This e-mail is Cc'ed to the crossgcc@ and the buildroot@ mailing lists)
> 
> We started discussing on IRC about whether a three stage or a two stage
> gcc build was needed, but seems like due to timezone issues, we're not
> able to discuss simultaneously, so I'm trying to do things by e-mail. I
> would like to understand a little bit better since when we can get back
> to a two stages gcc build process instead of the three stages build
> process that was needed since the introduction of NPTL support.
> 
> Currently, what we have in Buildroot is:
> 
> 1. Build gcc initial (--without-headers --with-newlib --disable-shared)
> 
> 2. Configure uClibc, install headers, crti.o, crtn.o, crt1.o, install
>    a fake libc.so and libm.so (just empty C files being compiled as a
>    share library)
> 
> 3. Build gcc intermediate (no special options passed)

One option thats different is that it uses --enable-shared as compared to gcc initial

> 
> 4. Build uClibc and install it.
> 
> 5. Build gcc final
> 
> Apparently, this three stage build process is no longer needed. In
> OE-Core, you did the following commit:
> 
> commit b0faebd1f07e1616004bd19664395932e7c2c48f
> Author: Khem Raj <raj.khem@gmail.com>
> Date:   Wed Aug 15 23:12:51 2012 -0700
> 
>    gcc-cross: Make gcc-cross-initial as the only intermediate gcc stage
> 
>    Now glibc can be compiled with gcc-cross-initial therefore prepare
>    the stage to drop gcc-cross-intermediate
> 
>    Also drop arm-nolibfloat.patch should not be needed anymore
>    half of changes in this patch are meant for OABI which we dont
>    use anymore
> 
>    (From OE-Core rev: 30617bde61a3b0a0944b49a0c9fb7159dacbb19f)
> 
>    Signed-off-by: Khem Raj <raj.khem@gmail.com>
>    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> So according to this, the fact that we can go back to a two stages
> build seems to be related to glibc. So what happens to uClibc in this
> case?
> 
> In fe9a576e8d27460468dfe5eac90aad49ab26a8d3 (gcc-cross-intermediate,
> gcc-crosssdk-intermediate: Remove) you completely remove the
> intermediate recipes, so I assume they are really no longer needed,
> even by the uClibc build, since I believe OE-core also supports uClibc
> builds (thanks to your work).
> 

thats correct. See the code around where limits.h is installed by gcc initial
uclibc was less of an issue, glibc was more until last year when a lot of cleanup
was done especially where build time dependency on libgcc_s and libgcc_eh was averted
in cross gcc and buildroot you can do the same like i have done for OE I can help out


> Could you explain in more details which version of gcc or glibc/uClibc
> made it possible to go back to a 2-stages build? Both Yann E. Morin
> (for Crosstool-NG) and myself (for Buildroot) are interested in
> understanding this in order to improve those tools. Also, what should
> be the two stages now? Just:
> 

0.1 Build cross binutils

> 1. Build gcc initial (no C library at all)

1.1 Build linux-headers
1.2 build libc-headers/fake .so and startup .o files.

> 2. Build C library completely
> 3. Build gcc final

yes pretty much like that

> 
> ?
> 
> Thanks for your insights,
> 
> Thomas
> -- 
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux
> development, consulting, training and support.
> http://free-electrons.com


--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Thomas Petazzoni]

Dear Khem Raj,

On Thu, 5 Sep 2013 10:31:46 -0700, Khem Raj wrote:

> I am late in reply but here it is.

No problem. Thanks for your reply.


> > In fe9a576e8d27460468dfe5eac90aad49ab26a8d3 (gcc-cross-intermediate,
> > gcc-crosssdk-intermediate: Remove) you completely remove the
> > intermediate recipes, so I assume they are really no longer needed,
> > even by the uClibc build, since I believe OE-core also supports uClibc
> > builds (thanks to your work).
> 
> thats correct. See the code around where limits.h is installed by gcc initial
> uclibc was less of an issue, glibc was more until last year when a lot of cleanup
> was done especially where build time dependency on libgcc_s and libgcc_eh was averted
> in cross gcc and buildroot you can do the same like i have done for OE I can help out

In the mean time, I've experimented a bit and actually implemented it.
The patch hasn't been merged yet in Buildroot, but it has been posted:
http://lists.busybox.net/pipermail/buildroot/2013-September/077344.html.

Don't hesitate to have a look and tell me what you think.

Thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Khem Raj]

On Sep 6, 2013, at 5:58 AM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:

> Dear Khem Raj,
> 
> On Thu, 5 Sep 2013 10:31:46 -0700, Khem Raj wrote:
> 
>> I am late in reply but here it is.
> 
> No problem. Thanks for your reply.
> 
> 
>>> In fe9a576e8d27460468dfe5eac90aad49ab26a8d3 (gcc-cross-intermediate,
>>> gcc-crosssdk-intermediate: Remove) you completely remove the
>>> intermediate recipes, so I assume they are really no longer needed,
>>> even by the uClibc build, since I believe OE-core also supports uClibc
>>> builds (thanks to your work).
>> 
>> thats correct. See the code around where limits.h is installed by gcc initial
>> uclibc was less of an issue, glibc was more until last year when a lot of cleanup
>> was done especially where build time dependency on libgcc_s and libgcc_eh was averted
>> in cross gcc and buildroot you can do the same like i have done for OE I can help out
> 
> In the mean time, I've experimented a bit and actually implemented it.
> The patch hasn't been merged yet in Buildroot, but it has been posted:
> http://lists.busybox.net/pipermail/buildroot/2013-September/077344.html.
> 
> Don't hesitate to have a look and tell me what you think.

OK. The patch looks ok to me. one thing

Do you need to support multiple toolchains ? and how far do you want to go back ? I would much rather not
have THREE_STAGE_BUILD options at all.

Otherwise it looks good to me
> 
> Thanks!
> 
> Thomas
> -- 
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux
> development, consulting, training and support.
> http://free-electrons.com


--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Thomas Petazzoni]

Dear Khem Raj,

On Fri, 6 Sep 2013 09:59:34 -0700, Khem Raj wrote:

> > In the mean time, I've experimented a bit and actually implemented it.
> > The patch hasn't been merged yet in Buildroot, but it has been posted:
> > http://lists.busybox.net/pipermail/buildroot/2013-September/077344.html.
> > 
> > Don't hesitate to have a look and tell me what you think.
> 
> OK. The patch looks ok to me. one thing
> 
> Do you need to support multiple toolchains ? and how far do you want to go back ? I would much rather not
> have THREE_STAGE_BUILD options at all.

We do need to support building gcc 4.2.x, because that's the only
version for which thee AVR32 architecture is supported. I'm hoping that
in a year or two we can drop AVR32 support (the architecture has been
deprecated by Atmel), and therefore simplify a few things, but for now,
I believe we need to be able to support older gcc versions than 4.7/4.8.

Thanks for having a look at the patch!

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Martin Guy]

On 07/09/2013, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:
> We do need to support building gcc 4.2.x, because that's the only
> version for which thee AVR32 architecture is supported.

 A few months ago I incorporated and improved the new AVR32 support
for gcc 4.3 and 4.4. Since AVR's dirty patches also affect code
generation on other non-AVR32 processors, I chose to do the
development outside the main branch but the results, a fork of ct-ng,
are visible at https://spaces.atmel.com/gf/project/ct-ng/

Do this in my "spare" time, so donations are welcome at
martinwguy.co.uk/donate.html

Enjoy

   M

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Thomas Petazzoni]

Dear Martin Guy,

On Sat, 7 Sep 2013 13:45:45 +0200, Martin Guy wrote:
> On 07/09/2013, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:
> > We do need to support building gcc 4.2.x, because that's the only
> > version for which thee AVR32 architecture is supported.
> 
>  A few months ago I incorporated and improved the new AVR32 support
> for gcc 4.3 and 4.4. Since AVR's dirty patches also affect code
> generation on other non-AVR32 processors, I chose to do the
> development outside the main branch but the results, a fork of ct-ng,
> are visible at https://spaces.atmel.com/gf/project/ct-ng/

Yes, I have seen that (maybe you announced it already on the ct-ng
mailing list). I personally don't care enough about AVR32 to spend some
time integrating this in Buildroot, but I've Cc'ed Simon Dawson who
might be interested.

That said, even gcc 4.4 is relatively "old" by these days standards, so
I am not sure it would make a huge difference compared to keeping the
existing gcc 4.2.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Martin Guy]

On 07/09/2013, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:
> That said, even gcc 4.4 is relatively "old" by these days standards, so
> I am not sure it would make a huge difference compared to keeping the
> existing gcc 4.2.

Huge, no, if you mean in code size and speed. Just a few percent which
made the product fit into 128KB flash instead of overflowing it.
However it (and Atmel 4.3.2, which is also based on gcc-4.4) generates
correct code in at least one case where the older toolchain generated
wrong code, which, for the customer in question, made the difference
between a working executable and one that crashed at random.

Unsupported targets like AVR32 and my MaverickCrunch FPU stuff stop at
gcc-4.4 because of upheavals in the code-generator structure of GCC
between gcc-4.4 and 4.5, which makes forward-porting patches not as
simple as it was from 4.1 and 4.4, and because no one cares enough
about them to fund the work.

    M

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Martin Guy]

On 07/09/2013, Martin Guy <martinwguy@gmail.com> wrote:
> not as simple as it was from 4.1 and 4.4

"not as simple as it was from 4.1 to 4.4"

   M

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Thomas Petazzoni]

Dear Khem Raj,

On Fri, 6 Sep 2013 09:59:34 -0700, Khem Raj wrote:

> > In the mean time, I've experimented a bit and actually implemented it.
> > The patch hasn't been merged yet in Buildroot, but it has been posted:
> > http://lists.busybox.net/pipermail/buildroot/2013-September/077344.html.
> > 
> > Don't hesitate to have a look and tell me what you think.
> 
> OK. The patch looks ok to me. one thing

After merging the patch, we got a report that it breaks SSP support,
see [1]. After investigation, depending on the architecture, the
__stack_chk_fail symbol maybe be part of the TLS (it's the case on x86,
but on ARM, for example).

However, since we're now only doing gcc-initial -> uclibc -> gcc-final,
and gcc-initial has no thread/TLS support, it breaks the build of
uClibc when SSP support is enabled (in such a case, uClibc is built
with -fstack-protector, which creates references to __stack_chk_fail,
but this symbol isn't available because we don't have TLS support).

Have you already seen this problem? What solution do you suggest?

Thanks!

Thomas

[1]
http://buildroot-busybox.2317881.n4.nabble.com/uClibc-fails-to-build-with-stack-smash-protection-td51478.html
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Two or three stages gcc build?

[Thomas Petazzoni]

Khem,

Any input on the below questions? :-)

Thanks!

Thomas

On Thu, 19 Sep 2013 07:02:55 +0200, Thomas Petazzoni wrote:
> Dear Khem Raj,
> 
> On Fri, 6 Sep 2013 09:59:34 -0700, Khem Raj wrote:
> 
> > > In the mean time, I've experimented a bit and actually implemented it.
> > > The patch hasn't been merged yet in Buildroot, but it has been posted:
> > > http://lists.busybox.net/pipermail/buildroot/2013-September/077344.html.
> > > 
> > > Don't hesitate to have a look and tell me what you think.
> > 
> > OK. The patch looks ok to me. one thing
> 
> After merging the patch, we got a report that it breaks SSP support,
> see [1]. After investigation, depending on the architecture, the
> __stack_chk_fail symbol maybe be part of the TLS (it's the case on x86,
> but on ARM, for example).
> 
> However, since we're now only doing gcc-initial -> uclibc -> gcc-final,
> and gcc-initial has no thread/TLS support, it breaks the build of
> uClibc when SSP support is enabled (in such a case, uClibc is built
> with -fstack-protector, which creates references to __stack_chk_fail,
> but this symbol isn't available because we don't have TLS support).
> 
> Have you already seen this problem? What solution do you suggest?
> 
> Thanks!
> 
> Thomas
> 
> [1]
> http://buildroot-busybox.2317881.n4.nabble.com/uClibc-fails-to-build-with-stack-smash-protection-td51478.html



-- 
Thomas Petazzoni, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

--
For unsubscribe information see http://sourceware.org/lists.html#faq