RFC: generating a header using the linker (ASCII, ASCIZ commands)

RFC: generating a header using the linker (ASCII, ASCIZ commands)

[Ulf Samuelsson]

[-- Attachment #1: Type: text/plain, Size: 4287 bytes --]

Hi guys,

My very first mailing to the list. I joined due to a problem.

I was involved in an embedded project using the GNU linker.

One of the problems was that we wanted to generate a header,

and for various reasons, it was desirable to have the linker generate 
the header

==============

There are a few problems involved with that.

1: You want to compute a CRC over the text area. The text area should be 
aligned with a flash erase sector.

     We tried several approaches to align things, and gave up. We 
resorted to specify the size in advance.

     I could not see a way to align easily to the flash boundaries and 
add fill-

2: How generate the CRC? It would be good to have such an application in 
binutils.

    I ported the ielfutils from

     IAR has a windows command line program which is open source 
(ielfutils) so
     I ported that to Linux

3. You  want strings in the header. Not just BYTEs, SHORTs and LONGs etc.

     It is of course possible to emulate strings using the above 
functions but not so nice.

==============

I am thinking of adding two commands:

* ASCII     <size> , "string"

* ASCIZ   "string"

The ASCII command would allocate a fixed size area and would fill it 
from the string.

If the string is shorter, then the remainder would be cleared.
If the string is longer, then there would be an error message.

The ASCIZ command could allocate an area large enough to store the string.

===========================

I did a first prototype by patching a Yocto build.

The simplest idea I could think of was to just try to create a BYTE 
record for each byte in the field.
Obviously, creating a single field for the complete string would be better,
but I have never studied the code before, so bear with me.

The Yocto SDK on an U-Boot with a modified linker command file.

I added.

      ASCIZ "My first string"

to the linker command file.

The ASCIZ seems to work. checked U-Boot.bin with hexedit and my string 
was there.

I then tried

      ASCII  20, "My second string"

this ASCII did not work - get a syntax error.

The ASCIZ command is not flawless.

Depending on the size of the string, I can get warnings:

"warning: unsupported relocation type 1051152 at ffad8"

===========================

A summary of the changes are:

ldlex.l

+<WILD>"ASCII"                { RTOKEN(ASCII); }
+<WILD>"ASCIZ"                { RTOKEN(ASCIZ); }

ldgram.y

+%token <token> ALIGN_K BLOCK BIND QUAD SQUAD LONG SHORT BYTE ASCII ASCIZ

+    | ASCII INT ',' NAME
+        {
+          lang_add_string($2.integer, $4);
+        }
+    | ASCIZ NAME
+        {
+          lang_add_stringz($2);
+        }

ldlang.c

+void
+lang_add_string (bfd_vma size, char *s)
+{
+     lang_data_statement_type *new_stmt;
+     bfd_vma stringlen = strlen(s) + 1;    /* Add one for terminating 
'\0' */
+     bfd_vma fill_len = 0;
+     if (size == 0) {  /* Zero terminated string */
+        size = stringlen;
+     } else if (size > stringlen) {    /* Fix Size string */
+        fill_len = size - stringlen;
+     } else if (size > stringlen) {
+        /* We have an error */
+        einfo (_("%P:%pS: warning: string does not fit \"%s\"\n"), 
NULL, s);
+     }
+      /* Add byte expressions until end of string */
+     for (bfd_vma i = 0 ; i < size ; i++) {
+        new_stmt = new_stat (lang_data_statement, stat_ptr);
+        new_stmt->exp = exp_intop(s[i]);
+        new_stmt->type = BYTE;
+     }
+      /* Add byte expressions for filling to the end of the string */
+     for (bfd_vma i = 0 ; i < fill_len ; i++) {
+        new_stmt = new_stat (lang_data_statement, stat_ptr);
+        new_stmt->exp = exp_intop(0);
+        new_stmt->type = BYTE;
+     }

+}

...

+void
+lang_add_stringz (char *s)
+{
+    lang_add_string (0, s);
+}
+

On top of that, then info file is updated.

*So I wonder if this type of functionality is of interest?*

Also, what kind of glaring mistakes did I do for

a) the ASCII statment to result in a syntax error?

b) trigger the unsupported relocation.

Best Regards
Ulf Samuelsson

Re: RFC: generating a header using the linker (ASCII, ASCIZ commands)

[Nick Clifton]

Hi Ulf,

> My very first mailing to the list. 

Well then welcome to the binutils community.


> One of the problems was that we wanted to generate a header,
> and for various reasons, it was desirable to have the linker generate the header

 From the sound of what you have written below it would be
easier to use the assembler to create the header and then just
include it as another object file when you perform your link.
That together with a customised linker script should allow you
to achieve what you want without having to add new code to the
linker.


> The text area should be aligned with a flash erase sector.

This should be possible using a linker script.  You can use the MEMORY
directive to specify the flash memory region(s) and the various section
alignment and assignment directives to put the text section where you
want.


>      I could not see a way to align easily to the flash boundaries and add fill-

Perhaps you could provide an example of what you want to achieve ?
I am pretty sure that it can be done using linker script directives.


> 2: How generate the CRC? It would be good to have such an application in binutils.
> 
>     I ported the ielfutils from
> 
>      IAR has a windows command line program which is open source (ielfutils) so
>      I ported that to Linux

Well if you already have a tool, then that is fine.

It is true that the binutils does not have a tool specifically for the creation
of CRCs, but then that does seem to be a bit of a niche application.

In theory you might be able to use the objcopy command to extract the text section
which you could then pass to a tool to generate the CRC.

Also - could you use the build-id mechanism instead ?  That would create a
special section containing a CRC-like value which your run-time system could
access.


> 3. You  want strings in the header. Not just BYTEs, SHORTs and LONGs etc.
> 
>      It is of course possible to emulate strings using the above functions but not so nice.

Agreed.  Having a couple of string directives supported in the linker
syntax would be nice.


> Depending on the size of the string, I can get warnings: 
> "warning: unsupported relocation type 1051152 at ffad8"

Hmmm, this looks like memory corruption of some kind.

I looked over the code you provided and it looks good, but that
was just a visual check.  I would have to have an actual reproducer
in order to investigate further.


> +     for (bfd_vma i = 0 ; i < size ; i++) {
> +        new_stmt = new_stat (lang_data_statement, stat_ptr);
> +        new_stmt->exp = exp_intop(s[i]);
> +        new_stmt->type = BYTE;
> +     }

FYI - this loop (and the one below) could be simplified to:

   for (bfd_vma i = 0; i < size; i++)
     lang_add_data (BYTYE, exp_intop(s[i]);

Re: RFC: generating a header using the linker (ASCII, ASCIZ commands)

[Ulf Samuelsson]

Den 2023-02-13 kl. 11:09, skrev Nick Clifton:
> Hi Ulf,
>
>> My very first mailing to the list. 
>
> Well then welcome to the binutils community.
>
>
>> One of the problems was that we wanted to generate a header,
>> and for various reasons, it was desirable to have the linker generate 
>> the header
>
> From the sound of what you have written below it would be
> easier to use the assembler to create the header and then just
> include it as another object file when you perform your link.
> That together with a customised linker script should allow you
> to achieve what you want without having to add new code to the
> linker.

One of the problems is time of build.
The header should contain the time when things are linked together, not 
when things are compiled.

Another is that one of the fields is the size of the application which 
is only known at link time.

Doing things in the linker seems to be the cleanest way.

We would probably have the header is in a separate file which is 
included in the linker script.


>> The text area should be aligned with a flash erase sector.
>
> This should be possible using a linker script.  You can use the MEMORY
> directive to specify the flash memory region(s) and the various section
> alignment and assignment directives to put the text section where you
> want.

I have not found a way.

I can put the text where I want it, but I do not not know the size of 
the application until link time.

Assume you have the following flash sectors.

* 64kB (65536 bytes)

* 64kB (65536 bytes)

* 128kB

* n x 256kB

The bootloader resides in the first 64 kB sector and the application 
starts in the second 64kB sector.

If the application <= 65536 bytes, the section should be 64kB and should 
be filled with a FILL value

If the application is 65540 bytes, the section should be 64kB+128kB and 
should be filled with a FILL value

I have not found a way to do this, so my workaround is to specify the 
size to be 192kB.
This sort of works, until it doesn't, and I have to change the setup.

>
>
>>      I could not see a way to align easily to the flash boundaries 
>> and add fill-
>
> Perhaps you could provide an example of what you want to achieve ?
> I am pretty sure that it can be done using linker script directives.
>
>
>> 2: How generate the CRC? It would be good to have such an application 
>> in binutils.
>>
>>     I ported the ielfutils from
>>
>>      IAR has a windows command line program which is open source 
>> (ielfutils) so
>>      I ported that to Linux
>
> Well if you already have a tool, then that is fine.
>
> It is true that the binutils does not have a tool specifically for the 
> creation
> of CRCs, but then that does seem to be a bit of a niche application.

In the linux world, it is certainly a niche thing.

In embedded control, this is considered good practice.
In safety applications, CRC checks are mandatory.

The way I would support CRC is to generate a command file in the linker 
for a tool which inserts the CRC.

Having the CRC generation utility inside binutils simplifies things.

> In theory you might be able to use the objcopy command to extract the 
> text section
> which you could then pass to a tool to generate the CRC.

There are two alternatives.

1. Generate the CRC on the binary (your suggestion).

2. Generate the CRC in the ELF file (implemented in ielfutils).

The latter is neccessary if you want to be able to
debug the application. The debugger downloads from the ELF file, and not 
the binary.

Would you consider adding ielftools to binutils?

This is the IAR license which looks OK to me.

///////////////////////////////// -*- C++ -*- 
/////////////////////////////////
/*
  * Copyright 2007-2017 IAR Systems AB.
  *
  * Permission to use, copy, modify, and distribute this software for any
  * purpose with or without fee is hereby granted, provided that the above
  * copyright notice and this permission notice appear in all copies.
  *
  * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
  * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
  * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
  * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
  * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
  */

I have ported it to linux here

https://github.com/emagii/ielftool.git

Since ielftool was created for the IAR Embedded Workbench, there are some
small things that needs handholding. I had to add fill value == 0x00
to get it to work properly, but after that I managed to use it successfully
on something linked with the binutils linker.

With more exposure, things like that are probably going to get fixed 
quicker.

====

Note that if the linker can call an external tool to compute the CRC
based on the extracted text segment, and insert that into the resulting 
ELF file.
then this is superior to running a utility afterwards.

Working on a an extracted text segment is of course much easier
than working on an ELF file.

> Also - could you use the build-id mechanism instead ?  That would 
> create a
> special section containing a CRC-like value which your run-time system 
> could
> access.

Had a quick look at this and this is how I understand it.

"The build-id is computed over the ELF header"

We need to compute it over the valid flash sections.

We also need to be able to select the SYNDROME and not SHA1.

Some chips (I.E: TMS570xxx) have CRC checking hardware, and using them 
is much faster than
computing the CRC in software. In many systems, you compute CRC before 
you start the application
but in safety critical systems, you need to recompute CRC continuously. 
Then performance is an issue.

>> 3. You  want strings in the header. Not just BYTEs, SHORTs and LONGs 
>> etc.
>>
>>      It is of course possible to emulate strings using the above 
>> functions but not so nice.
>
> Agreed.  Having a couple of string directives supported in the linker
> syntax would be nice.

I tried to implement

     ASCIZ "string"

     ASCII  size, "string"

Only got the ASCIZ to work.

I tried the ASCII and got a syntax error whenever I tested.

I therefore sent a patchset with only the ASCIZ string command

I am a newbie on the code, so if I can get someone to review the 
non-working code
I can perhaps get the ASCII command right.
I think that should be a second step.

>> Depending on the size of the string, I can get warnings: "warning: 
>> unsupported relocation type 1051152 at ffad8"
>
> Hmmm, this looks like memory corruption of some kind.

I tested this while linking U-Boot.
After testing a few things, I think that this is specfic to U-Boot and 
actually not a real problem.

SECTIONS
{
  . = 0x00000000;
  . = ALIGN(4);
  .text :
  {

   /* ASCIZ statement here generates a problem */

   *(.__image_copy_start)

   /* ASCIZ statement here does not generates a problem */

   *(.vectors)
   arch/arm/cpu/armv7/start.o (.text*)
  }

I suspect that there are a lot of object files accessing something in 
the ".__image_copy_start" section
using short relative addressing, and if something is added before the 
section, it breaks.

The same problem occurs when I add BYTE statements at the wrong location.
> I looked over the code you provided and it looks good, but that
> was just a visual check.  I would have to have an actual reproducer
> in order to investigate further.

Does that mean that you will try to build with the patches?

If not, what does it mean?

I provided a patchset for the ASCIZ command to the mailing list on 
2023-02-10 18:44

Unfortunately with  an empty header.

I build using Yocto, and can provide the bbappend with the patches, if 
it is of any help.
>> +     for (bfd_vma i = 0 ; i < size ; i++) {
>> +        new_stmt = new_stat (lang_data_statement, stat_ptr);
>> +        new_stmt->exp = exp_intop(s[i]);
>> +        new_stmt->type = BYTE;
>> +     }
>
> FYI - this loop (and the one below) could be simplified to:
>
>   for (bfd_vma i = 0; i < size; i++)
>     lang_add_data (BYTYE, exp_intop(s[i]);
>
I can update the patch, but perhaps we should wait until you tested the 
first patch.

Best Regards
Ulf Samuelsson

Re: RFC: generating a header using the linker (ASCII, ASCIZ commands)

[Jan Beulich]

On 13.02.2023 12:12, Ulf Samuelsson via Binutils wrote:
> Den 2023-02-13 kl. 11:09, skrev Nick Clifton:
>>> One of the problems was that we wanted to generate a header,
>>> and for various reasons, it was desirable to have the linker generate 
>>> the header
>>
>> From the sound of what you have written below it would be
>> easier to use the assembler to create the header and then just
>> include it as another object file when you perform your link.
>> That together with a customised linker script should allow you
>> to achieve what you want without having to add new code to the
>> linker.
> 
> One of the problems is time of build.
> The header should contain the time when things are linked together, not 
> when things are compiled.
> 
> Another is that one of the fields is the size of the application which 
> is only known at link time.
> 
> Doing things in the linker seems to be the cleanest way.
> 
> We would probably have the header is in a separate file which is 
> included in the linker script.

Otherwise I would be wondering if you're not really defining a new binary
format (potentially an extension of an existing one), at which point it
might be best to also treat it as such in binutils / ld.

Jan

Re: RFC: generating a header using the linker (ASCII, ASCIZ commands)

[Ulf Samuelsson]

Den 2023-02-13 kl. 13:33, skrev Jan Beulich:
> On 13.02.2023 12:12, Ulf Samuelsson via Binutils wrote:
>> Den 2023-02-13 kl. 11:09, skrev Nick Clifton:
>>>> One of the problems was that we wanted to generate a header,
>>>> and for various reasons, it was desirable to have the linker generate
>>>> the header
>>>  From the sound of what you have written below it would be
>>> easier to use the assembler to create the header and then just
>>> include it as another object file when you perform your link.
>>> That together with a customised linker script should allow you
>>> to achieve what you want without having to add new code to the
>>> linker.
>> One of the problems is time of build.
>> The header should contain the time when things are linked together, not
>> when things are compiled.
>>
>> Another is that one of the fields is the size of the application which
>> is only known at link time.
>>
>> Doing things in the linker seems to be the cleanest way.
>>
>> We would probably have the header is in a separate file which is
>> included in the linker script.
> Otherwise I would be wondering if you're not really defining a new binary
> format (potentially an extension of an existing one), at which point it
> might be best to also treat it as such in binutils / ld.

Yes, but each customer has their own header format and they
do not want to make changes to binutils.

Typically, the image is downloaded and checked by a bootloader
which will receive a binary.

You also want to be able to download it with a debugger during development.

I do not think you need a new binary format for this.
Simplifying the creation of the header file is good enough for me.

/Ulf

> Jan

Re: RFC: generating a header using the linker (ASCII, ASCIZ commands)

[Nick Clifton]

Hi Ulf,

>> From the sound of what you have written below it would be
>> easier to use the assembler to create the header 

> One of the problems is time of build.
> The header should contain the time when things are linked together, not when things are compiled.
> 
> Another is that one of the fields is the size of the application which is only known at link time.
> 
> Doing things in the linker seems to be the cleanest way.
> 
> We would probably have the header is in a separate file which is included in the linker script.

If you need to compute information in the header based upon the
actual linked executable then you will not be able to do so from
inside a linker script.  (Well not without modifying the linker,
and I am trying to avoid that).

But - what you can do is post-process the linked file to extract
the information you want, create a separate file containing the
header constructed from this information, and then insert the
header into the executable, without relinking it.  (I am thinking
of objcopy's --add-section command line option here).

Alternatively you could link with a dummy header that is basically
empty and then use a tool like GNU Poke[1] to insert the necessary
bytes and strings afterwards.

[1] http://www.jemarch.net/poke


> Assume you have the following flash sectors.
> 
> * 64kB (65536 bytes)
> * 64kB (65536 bytes)
> * 128kB
> * n x 256kB
> 
> The bootloader resides in the first 64 kB sector and the application starts in the second 64kB sector.
> 
> If the application <= 65536 bytes, the section should be 64kB and should be filled with a FILL value
> 
> If the application is 65540 bytes, the section should be 64kB+128kB and should be filled with a FILL value

Ah yes.  This is a known problem for the linker.  It does not have a mechanism
to distribute section(s) across multiple memory regions, using one until it is full
and then switching to another.

I am sorry, but I do not have an easy workaround for you.  If you are able to try
multiple links, you could start with one that just uses the smallest memory configuration
first, and if that fails move up to bigger configurations, possibly with different
assignments of sections to memory regions afterwards.


> Would you consider adding ielftools to binutils?

I would prefer to leave it as a separate project for now.  I am worried
about bringing in a tool that will only be of use to a very small audience
(maybe just you ?) and having to maintain it.  Of course if you can persuade
more people to request that the sources be integrated then I can take abother
look at the issue...



> Note that if the linker can call an external tool to compute the CRC
> based on the extracted text segment, and insert that into the resulting ELF file.
> then this is superior to running a utility afterwards.

Actually -- there might be a way to do this: A linker plugin.  The linker already
has the architecture to support plugins, and you could create a new one which would
scan the text section, compute a CRC and then insert the value into a header in the
linked image...

Cheers
   Nick

Re: RFC: generating a header using the linker (ASCII, ASCIZ commands)

[Ulf Samuelsson]

[-- Attachment #1: Type: text/plain, Size: 6209 bytes --]


Den 2023-02-13 kl. 15:11, skrev Nick Clifton:
> Hi Ulf,
>
>>> From the sound of what you have written below it would be
>>> easier to use the assembler to create the header 
>
>> One of the problems is time of build.
>> The header should contain the time when things are linked together, 
>> not when things are compiled.
>>
>> Another is that one of the fields is the size of the application 
>> which is only known at link time.
>>
>> Doing things in the linker seems to be the cleanest way.
>>
>> We would probably have the header is in a separate file which is 
>> included in the linker script.
>
> If you need to compute information in the header based upon the
> actual linked executable then you will not be able to do so from
> inside a linker script.  (Well not without modifying the linker,
> and I am trying to avoid that).
>
> But - what you can do is post-process the linked file to extract
> the information you want, create a separate file containing the
> header constructed from this information, and then insert the
> header into the executable, without relinking it.  (I am thinking
> of objcopy's --add-section command line option here).
>
> Alternatively you could link with a dummy header that is basically
> empty and then use a tool like GNU Poke[1] to insert the necessary
> bytes and strings afterwards.

This is what a typical header looks like:

header:

            QUAD(TAG1)                            // 0

ECC:    QUAD(ECC(SYNDROME, START_OF_ECC, .end_of_text))     // 8

START_OFF_ECC:

BUILDTIME:

             QUAD(SECONDS_SINCE_1970)               // 16

ENTRY: LONG(program_entry)                 // 24

SIZE:   LONG(.end_of_text - .start_of_text)                 // 28

            BYTE(VERSION1)                         // 32

            BYTE(VERSION2)                         // 33

            BYTE(VERSION3)                         // 34

            BYTE(VERSION3)                         // 35

            LONG(0)                             // 36

            LONG(0)                             // 40

            LONG(0)                             // 44

APPLICATION:

            ASCII    16, "MyApp"                          // 48

            ASCII    32, "A special app"                        // 64

            ASCII    32, "With Info"                            // 96

HELP

            ASCII    120 "No help available"                     // 120

            QUAD(TAG2)                          // 248

Total Size = 265 byte

Most of the header info is part of the linker file.
The linker needs to compute:
    ECC:

    BUILDTIME:

    ENTRY:

    SIZE:

===
We already discussed ideas for ECC.

For buildtime, we currently added an include file which we create during 
the linking process.

To have a BUILDTIME directive that generates a 64-bit word with "seconds 
since 1970-01-01 00:00"
does not seem to be very difficult - at least on linux.
Parse the call and get the |__time64_t |seconds from the OS.

ENTRY and SIZE is supported right now in the linker.

Best Regards
Ulf Samuelsson

>
> [1] http://www.jemarch.net/poke
>
>
>> Assume you have the following flash sectors.
>>
>> * 64kB (65536 bytes)
>> * 64kB (65536 bytes)
>> * 128kB
>> * n x 256kB
>>
>> The bootloader resides in the first 64 kB sector and the application 
>> starts in the second 64kB sector.
>>
>> If the application <= 65536 bytes, the section should be 64kB and 
>> should be filled with a FILL value
>>
>> If the application is 65540 bytes, the section should be 64kB+128kB 
>> and should be filled with a FILL value
>
> Ah yes.  This is a known problem for the linker.  It does not have a 
> mechanism
> to distribute section(s) across multiple memory regions, using one 
> until it is full
> and then switching to another.

I don't think we need to distribute sections amongs multiple regions.

I just need to say that the PC should be aligned to the next 64kB 
boundary and it
should be filled with my fill value.

Then I need to check the PC and if it is not at the end of a flash segment,
I want to add an additional BYTE(0xFF) and again align to the next segment.

The following would handle application up to 512kB.

     . = ALIGN(64kB)

if (. == 0x30000)

     BYTE(0xFF)

     . = ALIGN(64kB)

end if

if (. == 0x50000)

     BYTE(0xFF)

     . = ALIGN(64kB)

end if


if (. == 0x60000)

     BYTE(0xFF)

     . = ALIGN(64kB)

end if


if (. == 0x70000)

     BYTE(0xFF)

     . = ALIGN(64kB)

end if

>
> I am sorry, but I do not have an easy workaround for you.  If you are 
> able to try
> multiple links, you could start with one that just uses the smallest 
> memory configuration
> first, and if that fails move up to bigger configurations, possibly 
> with different
> assignments of sections to memory regions afterwards.
>
>
>> Would you consider adding ielftools to binutils?
>
> I would prefer to leave it as a separate project for now.  I am worried
> about bringing in a tool that will only be of use to a very small 
> audience
> (maybe just you ?) and having to maintain it.  Of course if you can 
> persuade
> more people to request that the sources be integrated then I can take 
> abother
> look at the issue...
>
>> Note that if the linker can call an external tool to compute the CRC
>> based on the extracted text segment, and insert that into the 
>> resulting ELF file.
>> then this is superior to running a utility afterwards.
>
> Actually -- there might be a way to do this: A linker plugin.  The 
> linker already
> has the architecture to support plugins, and you could create a new 
> one which would
> scan the text section, compute a CRC and then insert the value into a 
> header in the
> linked image...
>
OK, tell me more!

> Cheers
>   Nick
>
>

Re: RFC: generating a header using the linker (ASCII, ASCIZ commands)

[Nick Clifton]

Hi Ulf,

>>> Note that if the linker can call an external tool to compute the CRC
>>> based on the extracted text segment, and insert that into the resulting ELF file.
>>> then this is superior to running a utility afterwards.
>>
>> Actually -- there might be a way to do this: A linker plugin.  The linker already
>> has the architecture to support plugins, and you could create a new one which would
>> scan the text section, compute a CRC and then insert the value into a header in the
>> linked image...
>>
> OK, tell me more!

Sure.  The linker has the ability to run plugins via the use of the "-plugin <name>"
command line option.  Currently there are two known plugins for the linker.  One is
used to process the dependencies of static archives (ld/libdep_plugin.c in the
binutils sources) and one is used to pass LTO enabled object files back to the compiler
for recompilation (lto-plugin/ in the gcc sources).

The big problem with linker plugins however is the total lack of documentation
on how to write them.  Until the day comes when somebody writes a manual the
only documentation is the source code itself.  The place to start is the
include/plugin-api.h file which defines the interface between plugins and the
linker.  Then take a look at the plugin sources mentioned above and the ld/plugin.c
source file which is the linker's side of the coin.

In essence what a plugin does is provide a selection of callback functions to
the linker.  These functions are called at various stages of the link, and can do
pretty much anything.  Since plugins are shared objects, they have full access to
the linker's run-time memory space and can muck about as they please.  Of course
plugins are expected to behave nicely.

Cheers
   Nick

Re: RFC: generating a header using the linker (ASCII, ASCIZ commands)

[Ulf Samuelsson]

Den 2023-02-14 kl. 17:06, skrev Nick Clifton:
> Hi Ulf,
>
>>>> Note that if the linker can call an external tool to compute the CRC
>>>> based on the extracted text segment, and insert that into the 
>>>> resulting ELF file.
>>>> then this is superior to running a utility afterwards.
>>>
>>> Actually -- there might be a way to do this: A linker plugin. The 
>>> linker already
>>> has the architecture to support plugins, and you could create a new 
>>> one which would
>>> scan the text section, compute a CRC and then insert the value into 
>>> a header in the
>>> linked image...
>>>
>> OK, tell me more!
>
> Sure.  The linker has the ability to run plugins via the use of the 
> "-plugin <name>"
> command line option.  Currently there are two known plugins for the 
> linker.  One is
> used to process the dependencies of static archives 
> (ld/libdep_plugin.c in the
> binutils sources) and one is used to pass LTO enabled object files 
> back to the compiler
> for recompilation (lto-plugin/ in the gcc sources).
>
> The big problem with linker plugins however is the total lack of 
> documentation
> on how to write them.  Until the day comes when somebody writes a 
> manual the
> only documentation is the source code itself.  The place to start is the
> include/plugin-api.h file which defines the interface between plugins 
> and the
> linker.  Then take a look at the plugin sources mentioned above and 
> the ld/plugin.c
> source file which is the linker's side of the coin.

I think I can read the code, but:

What I would need help with is to extract the information from the 
linker data.

1) Assume I created a variable, how do I pass the value of that variable 
to the plugin?

2) The plugin needs to run after the linking process is complete, but 
before it is emitted.
     Which callback to use?

3) What is the data structure that contains the linked code, and how do 
I pass it to the plugin.

4) If I have pointers to an expression, is there a function that returns 
the calculated result.

======================

This is my idea of a CRC implementation:

I am thinking of generating a directive "CRC64" with parameters.

CRC64 ISO ',' <startaddress> ',' <endaddress>

CRC64 ECMA ',' <startaddress> ',' <endaddress>

CRC64 POLY '(' INT64 ')'   ','   <startaddress> ',' <endaddress>

Only one such directive is allowed in a linker command file.

The command should:

* Check that the plugin is available

* immediately calculate the table used for CRC generation (2kB of data).

* Reserve room for a 64-bit word in the image (containing the CRC).

* Save the program counter for this word.

* Save <startaddress> and <endaddress> expressions in global pointers

======================

MOTIVATION

Once the linking process is completed, the <startaddress> and 
<endaddress> needs to be computed
and passed to the plugin together with the <pc>.

I think you want your linker command file to look like:

       startaddress = .;

       .text {}

       . = ALIGN(0x10000)

       endaddress = .;

You always specify the startaddress as a parameter.

The second parameter can be the address of:

The byte following the segment:   => CRC64 ISO , <startaddress> , 
<endaddress>

The last byte of the segment:       => CRC64 ISO , <startaddress> , 
<endaddress> - 1

The size of the segment:              =>  CRC64 ISO , <startaddress> , 
<endaddress> - <startaddress>

It seems that the first alternative means less writing for the user, so 
this is what I plan to use.


Best Regards
Ulf Samuelsson


> In essence what a plugin does is provide a selection of callback 
> functions to
> the linker.  These functions are called at various stages of the link, 
> and can do
> pretty much anything.  Since plugins are shared objects, they have 
> full access to
> the linker's run-time memory space and can muck about as they please.  
> Of course
> plugins are expected to behave nicely.
>

> Cheers
>   Nick
>

Re: RFC: generating a header using the linker (CRC calculation)

[Ulf Samuelsson]

[-- Attachment #1: Type: text/plain, Size: 3389 bytes --]

CRC calculation in the linker

> Actually -- there might be a way to do this: A linker plugin.  The 
> linker already
> has the architecture to support plugins, and you could create a new 
> one which would
> scan the text section, compute a CRC and then insert the value into a 
> header in the
> linked image...

There is a CRC library available from a guy named Lammert Bies

This is released under an MIT license.

/*
  * Library: libcrc
  * File:    src/crc64.c
  * Author:  Lammert Bies
  *
  * This file is licensed under the MIT License as stated below
  *
  * Copyright (c) 2016 Lammert Bies
  *
  * Permission is hereby granted, free of charge, to any person 
obtaining a copy
  * of this software and associated documentation files (the 
"Software"), to deal
  * in the Software without restriction, including without limitation 
the rights
  * to use, copy, modify, merge, publish, distribute, sublicense, and/or 
sell
  * copies of the Software, and to permit persons to whom the Software is
  * furnished to do so, subject to the following conditions:
  *
  * The above copyright notice and this permission notice shall be 
included in all
  * copies or substantial portions of the Software.
  *
  * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 
EXPRESS OR
  * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT 
SHALL THE
  * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
  * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 
ARISING FROM,
  * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER 
DEALINGS IN THE
  * SOFTWARE.
  *
  * Description
  * -----------
  * The source file src/crc64.c contains the routines which are needed to
  * calculate a 64 bit CRC value of a sequence of bytes.
  */


The library supports a variety of CRC algorithms, but only 64-bit CRC is 
needed here.

The two enclosed files are (with minor modifications) all that are needed.

When you encounter a CRC, you generate a table based on a polynom..

ECMA:    0x42F0E1EBA9EA3693ull
ISO:        0xD800000000000000ull
custom: <whatever>


| CRC64 ISO ' (' mustbe_exp ',' mustbe_exp ')'
           {
                      init_crc64_tab(0xD800000000000000ull)
                      ecc_start = $4;
                      ecc_end  = $6;
           }

| CRC64 ECMA ' (' mustbe_exp ',' mustbe_exp ')'
           {
                      init_crc64_tab(0x42F0E1EBA9EA3693ull);
                      ecc_start = $4;
                      ecc_end  = $6;
           }
| CRC64 POLY '[' mustbe_exp ']' '(' mustbe_exp ',' mustbe_exp ')'
           {
                      init_crc64_tab($4);
                      ecc_start = $7;
                      ecc_end  = $9;
           }

When it is time to calculate the CRC you either call crc_64 or crc_64_inv
on the area you specified in the CRC64 directive.

The question is if a plugin is the right way or just add it to the 
normal build.

The code is well known, and very unlikely to change.

> Cheers
>   Nick
>
>

[-- Attachment #2: crc64.c --]
[-- Type: text/x-csrc, Size: 2025 bytes --]

#include <stddef.h>
#include <stdint.h>
#include "crc64.h"
uint64_t	crc_tab64[256];
/*
 * void init_crc64_tab( void );
 *
 * For optimal speed, the CRC64 calculation uses a table with pre-calculated
 * bit patterns which are used in the XOR operations in the program. This table
 * is generated during compilation of the library and added to the library as a
 * table with constant values.
 */
void init_crc64_tab( uint64_t poly ) {

	uint64_t i;
	uint64_t j;
	uint64_t c;
	uint64_t crc;

	for (i=0; i<256; i++) {

		crc = 0;
		c   = i << 56;

		for (j=0; j<8; j++) {

			if ( ( crc ^ c ) & 0x8000000000000000ull ) crc = ( crc << 1 ) ^ poly;
			else                                       crc =   crc << 1;

			c = c << 1;
		}

		crc_tab64[i] = crc;
	}

}  /* init_crc64_tab */

/*
 * uint64_t crc_64_ecma( const unsigned char *input_str, size_t num_bytes );
 *
 * The function crc_64() calculates in one pass the 64 bit CRC value
 * for a byte string that is passed to the function together with a parameter
 * indicating the length.
 * This is used for CRC64-ECMA and CRC64-ISO
 */

uint64_t crc_64(const unsigned char *input_str, size_t num_bytes)
{
	uint64_t crc;
	const unsigned char *ptr;
	size_t a;
	crc = CRC_START_64;
	ptr = input_str;
	if ( ptr != NULL ) {
		for (a=0; a<num_bytes; a++) {
		crc = (crc << 8) ^ crc_tab64[ ((crc >> 56) ^ (uint64_t) *ptr++) & 0x00000000000000FFull ];
	}
	return crc;
}  /* crc_64 */


/*
 * The function crc_64_inv() calculates in one pass the CRC64 64 bit CRC
 * value for a byte string that is passed to the function together with a
 * parameter indicating the length.
 * This is used for CRC64-WE
 */

uint64_t crc_64_inv( const unsigned char *input_str, size_t num_bytes ) {

	uint64_t crc;
	const unsigned char *ptr;
	size_t a;

	crc = CRC_START_64_INV;
	ptr = input_str;

	if ( ptr != NULL ) for (a=0; a<num_bytes; a++) {

		crc = (crc << 8) ^ crc_tab64[ ((crc >> 56) ^ (uint64_t) *ptr++) & 0x00000000000000FFull ];
	}

	return crc ^ 0xFFFFFFFFFFFFFFFFull;

}  /* crc_64_inv */


[-- Attachment #3: crc64.h --]
[-- Type: text/x-chdr, Size: 373 bytes --]

#include <stddef.h>
#include <stdint.h>

#define		CRC_POLY_64			0x42F0E1EBA9EA3693ull
#define		CRC_POLY_64_ISO		0xD800000000000000ull

#define		CRC_START_64		0x0000000000000000ull
#define		CRC_START_64_INV	0xFFFFFFFFFFFFFFFFull

uint64_t	crc_64    (const unsigned char *input_str, size_t num_bytes);
uint64_t	crc_64_inv(const unsigned char *input_str, size_t num_bytes);

Re: RFC: generating a header using the linker (CRC calculation)

[Ulf Samuelsson]

[-- Attachment #1: Type: text/plain, Size: 6030 bytes --]

Some argument for including CRC support in the linker.

============

The AUTOSAR organisation publishes requirement for developing code for 
the Automotive industry.

Protecting your code with a CRC is mandatory, and the chosen algorithm 
is CRC64 ECMA.

https://www.autosar.org/fileadmin/standards/classic/19-11/AUTOSAR_SWS_CRCLibrary.pdf

Large companies like Texas Instruments put hardware into their 
microcontrollers
for safetly critical operations liḱe the TMS570LC4375.

https://www.ti.com/lit/pdf/spnu563

This is a big thing in embedded (non-linux/windows control) so I think 
that it is
safe to say that people will be interested in simplifying their development
by creating the CRC inside the linker.

============

That it is important also for Aviation is shown in this presentation:

https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf

============

TÜV are very important in "Functional Safety".
https://www.tuvsud.com/en/services/functional-safety/top-misunderstandings-about-functional-safety

"Some integrity/diagnostic measures are performed at each startup of the 
system (e.g. RAM test, *Flash CRC check*, output tests, etc.). *These 
tests are very important*, as they are e.g. used for argumentation on 
diagnostic measures, or for fault detection time (test interval) in the 
safety analysis. However, sometime the operation condition can change 
(new project, new requirements, etc.). Systems which were regularly 
powered-down, stay powered-up for longer time. For example, modern rail 
systems are nowadays often continuously powered-on, without re-start 
every morning. In such cases the intended diagnostic measures will not 
become effective. If needed by the safety analysis, regular re-start 
shall be therefore explicitly specified in the safety manual, operator 
documentation, etc.

You cannot get a Functional Safety device qualified, if the firmware is 
not protected by a CRC.

The Microcontroller will run the bootloader, which will first perform a 
CRC check on itself,
and then before an application image is started, the CRC check will pass.

============

So there is a significant part of the embedded industry that relies on 
CRC calculations for ensuring that their systems are not broken.

What is important here is the "Hamming distance" which measures the 
quality of detection.

The width of the CRC needs to grow as the program size grows.

Best Regards
Ulf Samuelsson


Den 2023-02-15 kl. 21:07, skrev Ulf Samuelsson via Binutils:
> CRC calculation in the linker
>
>> Actually -- there might be a way to do this: A linker plugin.  The 
>> linker already
>> has the architecture to support plugins, and you could create a new 
>> one which would
>> scan the text section, compute a CRC and then insert the value into a 
>> header in the
>> linked image...
>
> There is a CRC library available from a guy named Lammert Bies
>
> This is released under an MIT license.
>
> /*
>  * Library: libcrc
>  * File:    src/crc64.c
>  * Author:  Lammert Bies
>  *
>  * This file is licensed under the MIT License as stated below
>  *
>  * Copyright (c) 2016 Lammert Bies
>  *
>  * Permission is hereby granted, free of charge, to any person 
> obtaining a copy
>  * of this software and associated documentation files (the 
> "Software"), to deal
>  * in the Software without restriction, including without limitation 
> the rights
>  * to use, copy, modify, merge, publish, distribute, sublicense, 
> and/or sell
>  * copies of the Software, and to permit persons to whom the Software is
>  * furnished to do so, subject to the following conditions:
>  *
>  * The above copyright notice and this permission notice shall be 
> included in all
>  * copies or substantial portions of the Software.
>  *
>  * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 
> EXPRESS OR
>  * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
> MERCHANTABILITY,
>  * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT 
> SHALL THE
>  * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
>  * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 
> ARISING FROM,
>  * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER 
> DEALINGS IN THE
>  * SOFTWARE.
>  *
>  * Description
>  * -----------
>  * The source file src/crc64.c contains the routines which are needed to
>  * calculate a 64 bit CRC value of a sequence of bytes.
>  */
>
>
> The library supports a variety of CRC algorithms, but only 64-bit CRC 
> is needed here.
>
> The two enclosed files are (with minor modifications) all that are 
> needed.
>
> When you encounter a CRC, you generate a table based on a polynom..
>
> ECMA:    0x42F0E1EBA9EA3693ull
> ISO:        0xD800000000000000ull
> custom: <whatever>
>
>
> | CRC64 ISO ' (' mustbe_exp ',' mustbe_exp ')'
>           {
>                      init_crc64_tab(0xD800000000000000ull)
>                      ecc_start = $4;
>                      ecc_end  = $6;
>           }
>
> | CRC64 ECMA ' (' mustbe_exp ',' mustbe_exp ')'
>           {
>                      init_crc64_tab(0x42F0E1EBA9EA3693ull);
>                      ecc_start = $4;
>                      ecc_end  = $6;
>           }
> | CRC64 POLY '[' mustbe_exp ']' '(' mustbe_exp ',' mustbe_exp ')'
>           {
>                      init_crc64_tab($4);
>                      ecc_start = $7;
>                      ecc_end  = $9;
>           }
>
> When it is time to calculate the CRC you either call crc_64 or crc_64_inv
> on the area you specified in the CRC64 directive.
>
> The question is if a plugin is the right way or just add it to the 
> normal build.
>
> The code is well known, and very unlikely to change.
>
>> Cheers
>>   Nick
>>
>>

Re: RFC: generating a header using the linker (CRC calculation)

[Paul Koning]

> On Feb 15, 2023, at 4:01 PM, Ulf Samuelsson via Binutils <binutils@sourceware.org> wrote:
> 
> Some argument for including CRC support in the linker.
> 
> ============
> 
> The AUTOSAR organisation publishes requirement for developing code for the Automotive industry.
> 
> Protecting your code with a CRC is mandatory, and the chosen algorithm is CRC64 ECMA.
> ...
> You cannot get a Functional Safety device qualified, if the firmware is not protected by a CRC.
> ...
> So there is a significant part of the embedded industry that relies on CRC calculations for ensuring that their systems are not broken.
> 
> What is important here is the "Hamming distance" which measures the quality of detection.
> 
> The width of the CRC needs to grow as the program size grows.

Indeed; CRC32 is not suitable for that reason.

But what about other options?  CRC64 is certainly one reasonable integrity check (if the assumption is that errors are unintentional as opposed to the result of active attack).  But SHA-1, SHA-2, or even MD-5 are at least as good.  Digital signatures provide protection against intentional modification.  Does the standard recognize that those other options are also suitable?

	paul

Re: RFC: generating a header using the linker (CRC calculation)

[Ulf Samuelsson]

The AUTOSARS says CRC-64 ECMA.
It is also popular in Linux.
I have not seen anyone using SHA algorithms or MD5 in any embedded project I have worked with to verify the flash contents.

My proposed implementation would make it easy to use the ISO or ECMA variants, but also allow a user defined polynom.

Best Regards
Ulf Samuelsson


> 
> 15 feb. 2023 kl. 22:29 skrev Paul Koning <paulkoning@comcast.net>:
> 
> 
> 
>> On Feb 15, 2023, at 4:01 PM, Ulf Samuelsson via Binutils <binutils@sourceware.org> wrote:
>> 
>> Some argument for including CRC support in the linker.
>> 
>> ============
>> 
>> The AUTOSAR organisation publishes requirement for developing code for the Automotive industry.
>> 
>> Protecting your code with a CRC is mandatory, and the chosen algorithm is CRC64 ECMA.
>> ...
>> You cannot get a Functional Safety device qualified, if the firmware is not protected by a CRC.
>> ...
>> So there is a significant part of the embedded industry that relies on CRC calculations for ensuring that their systems are not broken.
>> 
>> What is important here is the "Hamming distance" which measures the quality of detection.
>> 
>> The width of the CRC needs to grow as the program size grows.
> 
> Indeed; CRC32 is not suitable for that reason.
> 
> But what about other options?  CRC64 is certainly one reasonable integrity check (if the assumption is that errors are unintentional as opposed to the result of active attack).  But SHA-1, SHA-2, or even MD-5 are at least as good.  Digital signatures provide protection against intentional modification.  Does the standard recognize that those other options are also suitable?
> 
>    paul
> 

Re: RFC: generating a header using the linker (CRC calculation)

[Paul Koning]

> On Feb 15, 2023, at 5:08 PM, Ulf Samuelsson <binutils@emagii.com> wrote:
> 
> The AUTOSARS says CRC-64 ECMA.
> It is also popular in Linux.
> I have not seen anyone using SHA algorithms or MD5 in any embedded project I have worked with to verify the flash contents.

I'm not sure I have on anything I worked on, but it's how one would do cryptographic integrity (for example, signed code) when tamper-resistance is required.  It's more likely to show up as a signature on an update kit, which would prevent the installation of false updates.

> My proposed implementation would make it easy to use the ISO or ECMA variants, but also allow a user defined polynom.

That's certainly useful, so long as it is understood that CRC provides no protection whatsoever against deliberate tampering.

	paul

Re: RFC: generating a header using the linker (CRC calculation)

[Ulf Samuelsson]

> 15 feb. 2023 kl. 23:12 skrev Paul Koning <paulkoning@comcast.net>:
> 
> 
> 
>> On Feb 15, 2023, at 5:08 PM, Ulf Samuelsson <binutils@emagii.com> wrote:
>> 
>> The AUTOSARS says CRC-64 ECMA.
>> It is also popular in Linux.
>> I have not seen anyone using SHA algorithms or MD5 in any embedded project I have worked with to verify the flash contents.
> 
> I'm not sure I have on anything I worked on, but it's how one would do cryptographic integrity (for example, signed code) when tamper-resistance is required.  It's more likely to show up as a signature on an update kit, which would prevent the installation of false updates.
> 

Since the CRC check is done on code already flashed into the chip. we are way past the point of installation. 
The latest project I worked on would first download the installation package into an embedded linux system. This would download images to the embedded microcontroller, and all such verifications would be done in linux. 

Best Regards
Ulf Samuelsson

>> My proposed implementation would make it easy to use the ISO or ECMA variants, but also allow a user defined polynom.
> 
> That's certainly useful, so long as it is understood that CRC provides no protection whatsoever against deliberate tampering.
> 
>    paul
>