Porting the linker

Porting the linker

[Jean Christophe Beyler]

Dear all,

I'm currently maintaining a port of the binutils to an architecture
that I'm working on. However, I'm wrapping my mind behind what was
done and how it should be done now. My question is :

- My architecture has hierarchy memory (basically 3 levels)

- I am now working on the assembler and the problem is that I see this :

segT reg_section;
segT expr_section;
segT text_section;
segT data_section;
segT bss_section;

- However, technically, I'd have 3 sections for the data (both data
and bss) and 2 sections for text (1 level of the memory can't have
text). How do I explain this to the assembler ?

- Knowing that additionally, I'd need to have a system that overflows
if a smaller level is full

- Is there a correct way to do this without touching the internals of
the binutils that I'm missing or do I have to modify a bit it to make
it compatible to our architecture ?

Currently we modified the internals of the binutils but I'm not sure
this is the best solution.

Thanks for your help and input,
Jc

Re: Porting the linker

[Nick Clifton]

Hi Jc,

> - My architecture has hierarchy memory (basically 3 levels)

Is this sort of like L1 cache, L2 cache and external RAM or something else ?

> - However, technically, I'd have 3 sections for the data (both data
> and bss) and 2 sections for text (1 level of the memory can't have
> text). How do I explain this to the assembler ?

The assembler can support as many sections as you like.  For example the 
.section pseudo-op can be used to create new sections with arbitrary 
names and various different attributes.  Presumably however you want to 
define some "well-known" extra sections that are specific to your port 
of the assembler and which the programmer does not have to specify 
themselves.  This is quite straightforward.  Several ports do it.  For 
example have a look at gas/config/tc-microblaze.c.

> - Knowing that additionally, I'd need to have a system that overflows
> if a smaller level is full

Presumably this is something that you want to happen in the linker, not 
the assembler.

> - Is there a correct way to do this without touching the internals of
> the binutils that I'm missing or do I have to modify a bit it to make
> it compatible to our architecture ?

You are going to have to modify the internals of binutils.  Although 
with luck you should find that all of the changes that you need to make 
can be done via the various target specific hook functions provided by 
bfd and ld.

It feels like your situation is similar to the small data areas 
supported by some architectures.  With most of those however if the 
small area is filled to excess the linker just issues an error message 
and aborts.

Just a thought - it might be easier in the linker to fill the large text 
and data sections first and then, as late as possible, extract portions 
of those sections to go into the smaller text and data sections.

Cheers
   Nick

Re: Porting the linker

[Jean Christophe Beyler]

> It feels like your situation is similar to the small data areas supported by
> some architectures.  With most of those however if the small area is filled
> to excess the linker just issues an error message and aborts.

Yes that is the idea except that we don't want to make it abort.

> Just a thought - it might be easier in the linker to fill the large text and
> data sections first and then, as late as possible, extract portions of those
> sections to go into the smaller text and data sections.

That could be a solution, any hints as to how to do that ?

Thanks a lot,
Jean Christophe Beyler

On Mon, Mar 1, 2010 at 12:24 PM, Nick Clifton <nickc@redhat.com> wrote:
> Hi Jc,
>
>> - My architecture has hierarchy memory (basically 3 levels)
>
> Is this sort of like L1 cache, L2 cache and external RAM or something else ?
>
>> - However, technically, I'd have 3 sections for the data (both data
>> and bss) and 2 sections for text (1 level of the memory can't have
>> text). How do I explain this to the assembler ?
>
> The assembler can support as many sections as you like.  For example the
> .section pseudo-op can be used to create new sections with arbitrary names
> and various different attributes.  Presumably however you want to define
> some "well-known" extra sections that are specific to your port of the
> assembler and which the programmer does not have to specify themselves.
>  This is quite straightforward.  Several ports do it.  For example have a
> look at gas/config/tc-microblaze.c.
>
>> - Knowing that additionally, I'd need to have a system that overflows
>> if a smaller level is full
>
> Presumably this is something that you want to happen in the linker, not the
> assembler.
>
>> - Is there a correct way to do this without touching the internals of
>> the binutils that I'm missing or do I have to modify a bit it to make
>> it compatible to our architecture ?
>
> You are going to have to modify the internals of binutils.  Although with
> luck you should find that all of the changes that you need to make can be
> done via the various target specific hook functions provided by bfd and ld.
>
> It feels like your situation is similar to the small data areas supported by
> some architectures.  With most of those however if the small area is filled
> to excess the linker just issues an error message and aborts.
>
> Just a thought - it might be easier in the linker to fill the large text and
> data sections first and then, as late as possible, extract portions of those
> sections to go into the smaller text and data sections.
>
> Cheers
>  Nick
>
>
>

Re: Porting the linker

[Nick Clifton]

Hi Jean Christophe,

>> Just a thought - it might be easier in the linker to fill the large text and
>> data sections first and then, as late as possible, extract portions of those
>> sections to go into the smaller text and data sections.
>
> That could be a solution, any hints as to how to do that ?

As a guess - how about putting the code in where the section garbage 
collection occurs ?  (Search for "gc_section" in the bfd/ directory for 
lots of examples).  Currently the garbage collection code is used to 
strip out unneeded sections, but maybe it will also prove to be a 
suitable place to move parts of sections around.

Cheers
   Nick

Re: Porting the linker

[Jean Christophe Beyler]

Sorry about the long wait in this subject,

Anyway, I'm back on this and I have a bit more questions :-)

It still isn't clear to me how this should work.

What is clear is that I need a solution to:

- Move variables around in the sections
- It seems to me that if I do this at the latest stage it makes sense

- One idea was to remember all the elements that want to be in the
smaller bank and do what you said: send them off actually to the
biggest memory bank and then bring them back.

- The problem I have is that I have trouble finding two important things:
    - Where exactly to insert that, should it be in the ldwrite
function and do that.

- How to do it:
    (1) I can see the sizes of the sections and I can probably move
all of the section around but how do I go in the section to see what
is in there?

If I can figure that out, then I can probably :

- At the beginning of the linker, store which variables go where
- Then send them all to the biggest memory segment
- Then before ldwrite, actually go back and get one by one and try to
store them into the other smaller segments.

Any ideas on how to do (1) by any chance ?
Thanks in advance,
Jc

On Wed, Mar 10, 2010 at 9:11 AM, Nick Clifton <nickc@redhat.com> wrote:
> Hi Jean Christophe,
>
>>> Just a thought - it might be easier in the linker to fill the large text
>>> and
>>> data sections first and then, as late as possible, extract portions of
>>> those
>>> sections to go into the smaller text and data sections.
>>
>> That could be a solution, any hints as to how to do that ?
>
> As a guess - how about putting the code in where the section garbage
> collection occurs ?  (Search for "gc_section" in the bfd/ directory for lots
> of examples).  Currently the garbage collection code is used to strip out
> unneeded sections, but maybe it will also prove to be a suitable place to
> move parts of sections around.
>
> Cheers
>  Nick
>

Re: Porting the linker

[Jean Christophe Beyler]

I've continued my investigations and, doing what you said to do, I
figured that I could :

- Modify the output segments to point to the bigger segment versions
of them with a function such as this one after the lang_place_orphans
:

void
moveToBiggerSegment (char* segment_suffix)
{
    LANG_FOR_EACH_INPUT_STATEMENT (file)
    {
        asection *s;

        for (s = file->the_bfd->sections; s != NULL; s = s->next)
        {
            if (s->output_section)
            {
                asection *os = s->output_section;
                if (os != NULL)
                {
                    /* Let's move these to bigger segment */
                    static char *whatToMove[] = { ".bss", ".sbss",
".lbss", ".data"};
                    char buf[256];
                    unsigned int i;

                    for (i=0; i < sizeof (whatToMove) / sizeof
(whatToMove[0]); i++)
                    {
                        if (strcmp (os->name, whatToMove[i]) == 0)
                        {
                            sprintf (buf, "%s_%s", whatToMove[i],
segment_suffix);
                            lang_output_section_statement_type *new_sec = NULL;
                            new_sec = lang_output_section_statement_lookup(buf);
                            s->output_section = new_sec->bfd_section;
                        }
                    }
                }
            }
        }
    }
}

(a) However, for a reason I don't know, this doesn't work with archive
files such as what is in newlib. I'll have to see why, I know it has
to do with the moving of .data to .data_segment_suffix...

Second, once I've done that, I've lost what variables have to go into
the smaller zone so I don't know yet how to do that.
The one idea I had was:

For example, let's consider the .bss, instead of sending it to
.bss_segment_suffix, I'll send it to : .bss_temp
And define .bss_temp, that should allow me to then go to all the data
and add them into .bss until I fill that in. Then I'll send the rest
to the bigger zone.

(b) However, I still don't know how to, once segments have been sized
up, see every variable/symbol and which segment it goes into.

Any ideas about (a) or (b) would be appreciated.

Thanks again,
Jean Christophe Beyler

On Wed, Mar 10, 2010 at 9:11 AM, Nick Clifton <nickc@redhat.com> wrote:
> Hi Jean Christophe,
>
>>> Just a thought - it might be easier in the linker to fill the large text
>>> and
>>> data sections first and then, as late as possible, extract portions of
>>> those
>>> sections to go into the smaller text and data sections.
>>
>> That could be a solution, any hints as to how to do that ?
>
> As a guess - how about putting the code in where the section garbage
> collection occurs ?  (Search for "gc_section" in the bfd/ directory for lots
> of examples).  Currently the garbage collection code is used to strip out
> unneeded sections, but maybe it will also prove to be a suitable place to
> move parts of sections around.
>
> Cheers
>  Nick
>

Re: Porting the linker

[Alan Modra]

On Mon, May 17, 2010 at 05:03:11PM -0400, Jean Christophe Beyler wrote:
> - Modify the output segments to point to the bigger segment versions
> of them with a function such as this one after the lang_place_orphans

This isn't sufficient.  You will also need to call init_os on the new
output section.  See lang_add_section, the function that does the work
for map_input_to_output_sections.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Porting the linker

[Jean Christophe Beyler]

Dear all, I've pretty much worked through most problems to achieve
what I want. However, I'm still stuck in redirecting the sections.

I now know when I should overflow a section but I can't make it work.

For example, I can determine that file f1.o should overflow its data
to data_overflow. However, I get
However, I get a linking error : "Bad value"

I think it's because that data_overflow segment already exists in f1.o
and isn't empty so the VMA/LMA get mixed up in this case.

However, I thought of adding a section in the linker script after
data_overflow to handle my "during linking" overflow.

So I would have :
Small memory:
data

Bigger memory:
data_overflow (given by the programmer)
data_overflow_lk (at link time)

I've been looking at doing this change as soon as possible, and I can
determine at the ldlang_add_file stage what I should be overflowing.
Therefore, I tried to do something with bfd_map_over_sections. But I
don't quite understand what I should change during that traversal.

Any ideas or code example that you could show me that would for
example simply do this (or another direction that I'm missing). My
example is in place, my should_overflow works and tells me if I should
overflow that input section to the _overflow_lk section but I can't
seem to do it!

Here's the code extract:

static void
should_overflow_it (bfd *abfd, asection *sec, void *data)
{
  lang_input_statement_type *entry = data;

  if (should_overflow (abfd, sec, entry))
  {
      //Just link this section to output section name + _overflow_lk
  }
}

...

void
ldlang_add_file (lang_input_statement_type *entry)
{
...
  bfd_map_over_sections (entry->the_bfd, mycheck, entry);
  bfd_map_over_sections (entry->the_bfd, section_already_linked, entry);
}

Thanks again,
Jc

On Mon, May 17, 2010 at 5:03 PM, Jean Christophe Beyler
<jean.christophe.beyler@gmail.com> wrote:
> I've continued my investigations and, doing what you said to do, I
> figured that I could :
>
> - Modify the output segments to point to the bigger segment versions
> of them with a function such as this one after the lang_place_orphans
> :
>
> void
> moveToBiggerSegment (char* segment_suffix)
> {
>    LANG_FOR_EACH_INPUT_STATEMENT (file)
>    {
>        asection *s;
>
>        for (s = file->the_bfd->sections; s != NULL; s = s->next)
>        {
>            if (s->output_section)
>            {
>                asection *os = s->output_section;
>                if (os != NULL)
>                {
>                    /* Let's move these to bigger segment */
>                    static char *whatToMove[] = { ".bss", ".sbss",
> ".lbss", ".data"};
>                    char buf[256];
>                    unsigned int i;
>
>                    for (i=0; i < sizeof (whatToMove) / sizeof
> (whatToMove[0]); i++)
>                    {
>                        if (strcmp (os->name, whatToMove[i]) == 0)
>                        {
>                            sprintf (buf, "%s_%s", whatToMove[i],
> segment_suffix);
>                            lang_output_section_statement_type *new_sec = NULL;
>                            new_sec = lang_output_section_statement_lookup(buf);
>                            s->output_section = new_sec->bfd_section;
>                        }
>                    }
>                }
>            }
>        }
>    }
> }
>
> (a) However, for a reason I don't know, this doesn't work with archive
> files such as what is in newlib. I'll have to see why, I know it has
> to do with the moving of .data to .data_segment_suffix...
>
> Second, once I've done that, I've lost what variables have to go into
> the smaller zone so I don't know yet how to do that.
> The one idea I had was:
>
> For example, let's consider the .bss, instead of sending it to
> .bss_segment_suffix, I'll send it to : .bss_temp
> And define .bss_temp, that should allow me to then go to all the data
> and add them into .bss until I fill that in. Then I'll send the rest
> to the bigger zone.
>
> (b) However, I still don't know how to, once segments have been sized
> up, see every variable/symbol and which segment it goes into.
>
> Any ideas about (a) or (b) would be appreciated.
>
> Thanks again,
> Jean Christophe Beyler
>
> On Wed, Mar 10, 2010 at 9:11 AM, Nick Clifton <nickc@redhat.com> wrote:
>> Hi Jean Christophe,
>>
>>>> Just a thought - it might be easier in the linker to fill the large text
>>>> and
>>>> data sections first and then, as late as possible, extract portions of
>>>> those
>>>> sections to go into the smaller text and data sections.
>>>
>>> That could be a solution, any hints as to how to do that ?
>>
>> As a guess - how about putting the code in where the section garbage
>> collection occurs ?  (Search for "gc_section" in the bfd/ directory for lots
>> of examples).  Currently the garbage collection code is used to strip out
>> unneeded sections, but maybe it will also prove to be a suitable place to
>> move parts of sections around.
>>
>> Cheers
>>  Nick
>>
>