why build start files before core gcc pass 2 ?

why build start files before core gcc pass 2 ?

[Wang Baisheng]

Dear yan and All,

Before using ct-ng, I built a toolchain as the following steps:

1 build binutils;

2 bare-metal(freestanding) gcc

  a configured with "--with-newlib" not to depend on glibc;
  b configured with "--disable-shared" to build static libgcc, because
shared libgcc will linked start files, crt*.o,  which will be built in
glibc.

3 use the bare-meta gcc builing glibc;

4 glibc is ready, so compile the full gcc.

while studying ct-ng scripts, I found ct-ng add another gcc step:

after building bare-meta gcc, ct-ng install libc headers and build
crt*.o files, then build gcc pass 2, then building glibc using gcc
pass 2, then final gcc.

What I am confused is:

The crt*.o files is linked into executable and shared, so why gcc need
these files in compiling time, why gcc need crt*.o files ?

in docs "How is a toolchain constructed.txt", it said "The start files
are a very few files that gcc needs to be able to turn on thread local
storage (TLS) on an NPTL system." But I can not grasp it.

Thanks.

Regards,
Jacky

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: why build start files before core gcc pass 2 ?

[Wang Baisheng]

In build script gcc.sh, I found a comments:

"In case the threading model is NPTL, we need a shared-capable core gcc".

Yes, shared-capable gcc will need crt*.o, because libgcc*.so will
linked with crt*.o. Accoring build script, in fact, the option
"--disable-shared" is removed when building core gcc pass 2.

So if the anwser is : To support  NPTL, we need  a cross gcc with
"--enable-shared", i.e., building libgcc_.so instead of libgcc.a.
while linking libgcc_s.so, link editor need crt*.o, so build crt*.o
before building core gcc pass 2 ?

If above is true, the question is transfered to : why need  a
shared-capable core gcc to build NPTL ?

Thanks in advance.


2013/1/17 Wang Baisheng <baisheng.wang.cn@gmail.com>:
> Dear yan and All,
>
> Before using ct-ng, I built a toolchain as the following steps:
>
> 1 build binutils;
>
> 2 bare-metal(freestanding) gcc
>
>   a configured with "--with-newlib" not to depend on glibc;
>   b configured with "--disable-shared" to build static libgcc, because
> shared libgcc will linked start files, crt*.o,  which will be built in
> glibc.
>
> 3 use the bare-meta gcc builing glibc;
>
> 4 glibc is ready, so compile the full gcc.
>
> while studying ct-ng scripts, I found ct-ng add another gcc step:
>
> after building bare-meta gcc, ct-ng install libc headers and build
> crt*.o files, then build gcc pass 2, then building glibc using gcc
> pass 2, then final gcc.
>
> What I am confused is:
>
> The crt*.o files is linked into executable and shared, so why gcc need
> these files in compiling time, why gcc need crt*.o files ?
>
> in docs "How is a toolchain constructed.txt", it said "The start files
> are a very few files that gcc needs to be able to turn on thread local
> storage (TLS) on an NPTL system." But I can not grasp it.
>
> Thanks.
>
> Regards,
> Jacky

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: why build start files before core gcc pass 2 ?

[Yann E. MORIN]

Wang, All,

First, asking the same question over-and-over again, on the list or by
private mail, will not make for a faster answer.

Please, be patient. If nobody answers, maybe nobody really knows.

On Thursday 17 January 2013 Wang Baisheng wrote:
> In build script gcc.sh, I found a comments:
> 
> "In case the threading model is NPTL, we need a shared-capable core gcc".
> 
> Yes, shared-capable gcc will need crt*.o, because libgcc*.so will
> linked with crt*.o. Accoring build script, in fact, the option
> "--disable-shared" is removed when building core gcc pass 2.
> 
> So if the anwser is : To support  NPTL, we need  a cross gcc with
> "--enable-shared", i.e., building libgcc_.so instead of libgcc.a.
> while linking libgcc_s.so, link editor need crt*.o, so build crt*.o
> before building core gcc pass 2 ?
> 
> If above is true, the question is transfered to : why need  a
> shared-capable core gcc to build NPTL ?

This is a very complex question, for which I do not have the absolute
answer.

There are two ways to approach this problem: the academic point of view,
and the pragmatic point of view.

First, the pragmatic view point: everybody does it this way: crosstool-NG
openembedded, buildroot, all major distros... So it is a known way of
having an NPTL toolchain, with no need to have to solve new issues no one
will really be interested to investigate, as a working solution already
exists.

That's probably not a satisfiable answer for some, but as it does get the
job done, most of us are just happy with that.

Now, from the academic stand point. NPTL is partly implemented by the
kernel, by the C library, and by the compiler. How these play together
is a bit obscur to me.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'


--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: why build start files before core gcc pass 2 ?

[Esben Haabendal]

"Yann E. MORIN" <yann.morin.1998@free.fr> writes:

> Wang, All,
>
> First, asking the same question over-and-over again, on the list or by
> private mail, will not make for a faster answer.
>
> Please, be patient. If nobody answers, maybe nobody really knows.
>
> On Thursday 17 January 2013 Wang Baisheng wrote:
>> In build script gcc.sh, I found a comments:
>> 
>> "In case the threading model is NPTL, we need a shared-capable core gcc".
>> 
>> Yes, shared-capable gcc will need crt*.o, because libgcc*.so will
>> linked with crt*.o. Accoring build script, in fact, the option
>> "--disable-shared" is removed when building core gcc pass 2.
>> 
>> So if the anwser is : To support  NPTL, we need  a cross gcc with
>> "--enable-shared", i.e., building libgcc_.so instead of libgcc.a.
>> while linking libgcc_s.so, link editor need crt*.o, so build crt*.o
>> before building core gcc pass 2 ?
>> 
>> If above is true, the question is transfered to : why need  a
>> shared-capable core gcc to build NPTL ?
>
> This is a very complex question, for which I do not have the absolute
> answer.
>
> There are two ways to approach this problem: the academic point of view,
> and the pragmatic point of view.
>
> First, the pragmatic view point: everybody does it this way: crosstool-NG
> openembedded, buildroot, all major distros... So it is a known way of
> having an NPTL toolchain, with no need to have to solve new issues no one
> will really be interested to investigate, as a working solution already
> exists.
>
> That's probably not a satisfiable answer for some, but as it does get the
> job done, most of us are just happy with that.
>
> Now, from the academic stand point. NPTL is partly implemented by the
> kernel, by the C library, and by the compiler. How these play together
> is a bit obscur to me.

So if everybody else does something slightly wrong, we will do the same
;-)

FWIW, long ago I refactored the OpenEmbedded gcc and glibc recipes to
get rid of this additional step when doing NPTL.  And it worked fine.
Unfortunately, the work was never merged. As I remember it, based on the
same arguments of people not really understanding.

I think it would be interesting if we could try to see if ct-NG could
get this done right.  It should save other people from spending time
trying to figure out why this is so :-)

/Esben

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: why build start files before core gcc pass 2 ?

[Yann E. MORIN]

Esben, All,

On Tuesday 26 February 2013 Esben Haabendal wrote:
> "Yann E. MORIN" <yann.morin.1998@free.fr> writes:
> > On Thursday 17 January 2013 Wang Baisheng wrote:
[--SNIP--]
> >> If above is true, the question is transfered to : why need  a
> >> shared-capable core gcc to build NPTL ?
> >
> > This is a very complex question, for which I do not have the absolute
> > answer.
> >
> > There are two ways to approach this problem: the academic point of view,
> > and the pragmatic point of view.
> >
> > First, the pragmatic view point: everybody does it this way: crosstool-NG
> > openembedded, buildroot, all major distros... So it is a known way of
> > having an NPTL toolchain, with no need to have to solve new issues no one
> > will really be interested to investigate, as a working solution already
> > exists.
> >
> > That's probably not a satisfiable answer for some, but as it does get the
> > job done, most of us are just happy with that.
> >
> > Now, from the academic stand point. NPTL is partly implemented by the
> > kernel, by the C library, and by the compiler. How these play together
> > is a bit obscur to me.
> 
> So if everybody else does something slightly wrong, we will do the same
> ;-)

Yes. This is the pragmatic approach.

However, if any one comes up with a way to build NPTL, that does involve
only two gcc steps, and is *sane*, then I'm not against integrating it.
It should however be proven to be sound beyond any doubt, though.

> FWIW, long ago I refactored the OpenEmbedded gcc and glibc recipes to
> get rid of this additional step when doing NPTL.  And it worked fine.
> Unfortunately, the work was never merged. As I remember it, based on the
> same arguments of people not really understanding.

Yes, NPTL is a complex matter, and I never said I did understand
everything NPTL. The code is as-is today because it is the way it was
done in the original crosstool. Since it is working, there is no reason
from my point of view to fix it, as it's not broken.

> I think it would be interesting if we could try to see if ct-NG could
> get this done right.  It should save other people from spending time
> trying to figure out why this is so :-)

Well, I'll expect a very strong proof such a change is sound.

For now, it is possible to add additional explanations in the doc.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: why build start files before core gcc pass 2 ?

[Esben Haabendal]

"Yann E. MORIN" <yann.morin.1998@free.fr> writes:

> Esben, All,
>
> On Tuesday 26 February 2013 Esben Haabendal wrote:
>> "Yann E. MORIN" <yann.morin.1998@free.fr> writes:
>> > On Thursday 17 January 2013 Wang Baisheng wrote:
> [--SNIP--]
>> >> If above is true, the question is transfered to : why need  a
>> >> shared-capable core gcc to build NPTL ?
>> >
>> > This is a very complex question, for which I do not have the absolute
>> > answer.
>> >
>> > There are two ways to approach this problem: the academic point of view,
>> > and the pragmatic point of view.
>> >
>> > First, the pragmatic view point: everybody does it this way: crosstool-NG
>> > openembedded, buildroot, all major distros... So it is a known way of
>> > having an NPTL toolchain, with no need to have to solve new issues no one
>> > will really be interested to investigate, as a working solution already
>> > exists.
>> >
>> > That's probably not a satisfiable answer for some, but as it does get the
>> > job done, most of us are just happy with that.
>> >
>> > Now, from the academic stand point. NPTL is partly implemented by the
>> > kernel, by the C library, and by the compiler. How these play together
>> > is a bit obscur to me.
>> 
>> So if everybody else does something slightly wrong, we will do the same
>> ;-)
>
> Yes. This is the pragmatic approach.
>
> However, if any one comes up with a way to build NPTL, that does involve
> only two gcc steps, and is *sane*,

What is the definition of sane here?

> then I'm not against integrating it.  It should however be proven to
> be sound beyond any doubt, though.

Which is AFAIR, same kind of argumentation that caused my work in this
direction to be blocked in OE.  "Prove boyond any doubt" is a really
_HARD_ task here, and in this case, for most people not really feasible.

FWIW, I believe that OE has made some changes in this direction
recently, but I haven't looked into exactly what was changed (as I am
happily using ct-NG in OE-lite).

/Esben

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: why build start files before core gcc pass 2 ?

[Yann E. MORIN]

Esben, All,

On Saturday 02 March 2013 Esben Haabendal wrote:
> "Yann E. MORIN" <yann.morin.1998@free.fr> writes:
> > However, if any one comes up with a way to build NPTL, that does involve
> > only two gcc steps, and is *sane*,
> 
> What is the definition of sane here?

For one, all current samples continue to work as before.

Then, a patchset that brings changes one after the other. It might be
a bit complex to do, since I expect such a change to impact everything
about all at once, but I'd be very reluctant to apply a thousands-lines
long patch in the first place.

Finally, the new code should be commented, and better yet, the whole
new process should be dully documented and explained.

> "Prove boyond any doubt" is a really
> _HARD_ task here, and in this case, for most people not really feasible.

Yes, this is difficult. But I'm not trading a *known working* situation
that is not optimum, for a suposedly-better situation that is not working
(for the sake of not existing so far).

But, as they used to say: show me the code! ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: why build start files before core gcc pass 2 ?

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sat, 2 Mar 2013 16:21:37 +0100, Yann E. MORIN wrote:

> On Saturday 02 March 2013 Esben Haabendal wrote:
> > "Yann E. MORIN" <yann.morin.1998@free.fr> writes:
> > > However, if any one comes up with a way to build NPTL, that does involve
> > > only two gcc steps, and is *sane*,
> > 
> > What is the definition of sane here?
> 
> For one, all current samples continue to work as before.
> 
> Then, a patchset that brings changes one after the other. It might be
> a bit complex to do, since I expect such a change to impact everything
> about all at once, but I'd be very reluctant to apply a thousands-lines
> long patch in the first place.
> 
> Finally, the new code should be commented, and better yet, the whole
> new process should be dully documented and explained.
> 
> > "Prove boyond any doubt" is a really
> > _HARD_ task here, and in this case, for most people not really feasible.
> 
> Yes, this is difficult. But I'm not trading a *known working* situation
> that is not optimum, for a suposedly-better situation that is not working
> (for the sake of not existing so far).
> 
> But, as they used to say: show me the code! ;-)

I believe opinions have started to be a bit strong on this matter. I'm
not sure to understand why you would like this particular change to
provide more proofs that it is absolutely correct than all other
changes that go in Crosstool-NG? For example, the default gcc/binutils
version are regularly bumped, even though nobody has ever provided a
firm and definitive proof that nothing gets broken by the gcc/binutils
bump.

So really, why don't we ask just the same conditions for this
NPTL-related change than for all other changes? Of course, existing
samples should continue to build, and runtime testing on at least a few
platforms should confirm that things still work.

The way you've put things in this thread really doesn't encourage
people to contribute, which I think is quite sad. Especially when the
proposed change has the potential of simplifying the build process,
reducing the amount of code, and the overall build time.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: why build start files before core gcc pass 2 ?

[Yann E. MORIN]

Thomas, Esben, Wang, All,

On Monday 04 March 2013 Thomas Petazzoni wrote:
[--SNIP--]
> I believe opinions have started to be a bit strong on this matter.

Hmm, not on my side, to be clear.

> I'm
> not sure to understand why you would like this particular change to
> provide more proofs that it is absolutely correct than all other
> changes that go in Crosstool-NG? For example, the default gcc/binutils
> version are regularly bumped, even though nobody has ever provided a
> firm and definitive proof that nothing gets broken by the gcc/binutils
> bump.

This is, IMHO, very different than adding a new gcc/binutils/whatever
version. Adding a new version does not remove the existing ones, so it
is still possible to build previously existing configurations.

This, OTOH, would rip existing code/steps, and replace it with alternate
code/steps, so it is not so trivial in the first place to keep existing
configurations working.

> So really, why don't we ask just the same conditions for this
> NPTL-related change than for all other changes?

I am uneasy at changing this, since it really touches the core of the
code. It was quite complex to come up with, and, as this very thread
proves, not well understood.

What I meant is that we now have a working solution, that is mostly agreed
on by different projects. By Esben's own words, this is how OE does it (or
at least, used to do it until very recently). AFAICS, this is also how
Debian does it (even in Wheezy), as this seems to imply:

    gcc-4.7-4.7.2/debian/rules.defs:
    219 ifdef DEB_STAGE
    220   with_cdev := yes
    221   separate_lang := yes
    222   # "stage1" is minimal compiler with static libgcc
    223   # "stage2" is minimal compiler with shared libgcc
    224   ifeq ($(DEB_STAGE),stage1)
    225     with_shared_libgcc := no
    226   endif
    227   ifeq ($(DEB_STAGE),stage2)
    228     with_libgcc := yes
    229     with_shared_libgcc := yes
    230   endif
    231 endif
        [--SNIP--]
    404 ifndef DEB_STAGE
        [--SNIP final gcc--]
    890 endif # ifndef DEB_STAGE

Which means they have three code-paths:
  - one for which DEB_STAGE is set to 'stage1'
  - one for which DEB_STAGE is set to 'stage2'
  - one for which DEB_STAGE is unset

BTW, this is also the way Buildroot does it ;-) and it was added much
later, by people that have a track-record on working on gcc/glibc/uClibc
(ie. Khem Raj, in commit cfbf8abc33d86a0cf5c1bb3e0817a22009b7f301).

> Of course, existing
> samples should continue to build, and runtime testing on at least a few
> platforms should confirm that things still work.

Yes, that's my main concern.

> The way you've put things in this thread really doesn't encourage
> people to contribute, which I think is quite sad. Especially when the
> proposed change has the potential of simplifying the build process,
> reducing the amount of code, and the overall build time.

I'm sorry if anyone took this as an incentive not to contribute. That was
not my intention. Of course I welcome contributions! Even contributions
that touch deeply in the code.

But in this case, I prefer to stand on the conservative side, and wait for
actual code, and very detailed explanations on how that new code works.
Maybe my wording was too strict, and in retrospect, "proven beyond any
doubt" was probably too strong.

Re-reading my mails, I can't see how I was being heated. In my first reply
to Wang, I just explained the state of the affairs: known-working solution
as a pragmatic approach vs. an academic explanation I was not able to
provide:
    http://sourceware.org/ml/crossgcc/2013-01/msg00040.html

Then, more than a month passes, and Esben suggested we look at migrating
crosstool-NG to use a two-pass sequence. To which I replied I did not
completely understand the NPTL requirements, and I would need to be
convinced with hard proof (and code):
    http://sourceware.org/ml/crossgcc/2013-02/msg00057.html
    http://sourceware.org/ml/crossgcc/2013-02/msg00060.html

Esben then questioned what I meant by a "hard proof", to which I replied
that the change be documented and the code commented, and that existing
samples should still build. I fail to see how this could be interpreted
as a refusal of any contribution, and even suggested to submit some code:
    http://sourceware.org/ml/crossgcc/2013-03/msg00002.html
    http://sourceware.org/ml/crossgcc/2013-03/msg00004.html

Now, let me make it clear:
  - I do not completely understand why we need a three-pass sequence,
  - the current NPTL sequence was hard to come with, and is working,
  - I accept it is not necessarily the optimum or best solution, and that
    it might not even be a correct solution, and I accept the current state
    as a pragmatic approach, which is better than nothing,
  - I will gladly review and test a patch(set) that implements a better
    solution; better, as in: [explained-and-documented [correct [faster]].

And a final word: my apologies to anyone who felt offended by my previous
mails.

Thanks for reading! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq