From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <SRS0=7O1C=63=emagii.com=binutils@sourceware.org>
Received: from emagii.se (www.emagii.com [185.133.207.17])
	by sourceware.org (Postfix) with ESMTPS id C87B83858281
	for <binutils@sourceware.org>; Fri,  3 Mar 2023 15:12:35 +0000 (GMT)
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org C87B83858281
Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=emagii.com
Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=emagii.com
Received: from [10.175.196.145] (84-55-68-216.customers.ownit.se [84.55.68.216])
	by emagii.se (Postfix) with ESMTPSA id 0EDE21201B4;
	Fri,  3 Mar 2023 16:12:31 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=emagii.com;
	s=default; t=1677856354;
	bh=ET1yVKymeWHpoxv+/lLMf6TjSrolkH4SVOdzmzk9djc=; h=Subject:To:From;
	b=Xyq5vTVotwbs8VH/N2DyjPBfmnRokSW842Jw4gDTxqzZ9UM8F1MtIpwBnmyRdY0Ge
	 JNS/romAcV7aDY591E+iaVekWoksta6PDylt3k6l+kOh9TqBV+NZpYrWPLqz0YPUIt
	 xt2/CFpLDq7wfsZmakgBLV9+1/bwqbKZmymxCLe8=
Authentication-Results: emagii.beebytevps.io;
        spf=pass (sender IP is 84.55.68.216) smtp.mailfrom=binutils@emagii.com smtp.helo=[10.175.196.145]
Received-SPF: pass (emagii.beebytevps.io: connection is authenticated)
Content-Type: multipart/alternative;
 boundary="------------7R6sAs3W6H2wbBQaP8yoNlpY"
Message-ID: <cc1fcc89-a693-38e7-3057-c957710cb08b@emagii.com>
Date: Fri, 3 Mar 2023 16:12:30 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.7.1
Subject: Re: [PATCH v11 0/11 Add support for CRC64 generation in linker
Content-Language: en-US
To: binutils@sourceware.org
Cc: nickc@redhat.com
References: <20230303144706.1977061-1-binutils@emagii.com>
From: Ulf Samuelsson <binutils@emagii.com>
In-Reply-To: <20230303144706.1977061-1-binutils@emagii.com>
X-PPP-Message-ID: 
 <167785635432.1868325.1142150187196245790@localhost.localdomain>
X-PPP-Vhost: emagii.com
X-Spam-Status: No, score=-6.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,HTML_MESSAGE,NICE_REPLY_A,SPF_HELO_FAIL,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org
List-Id: <binutils.sourceware.org>

This is a multi-part message in MIME format.
--------------7R6sAs3W6H2wbBQaP8yoNlpY
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

It has been tested on the following targets.

  * aarch64-linux-gnu
  * all
  * mips-elf
  * powerpc64-linux-gnu
  * x86_64-pc-linux-gnu
  * x86_64-w64-mingw32


On 2023-03-03 15:46, Ulf Samuelsson via Binutils wrote:
> Patchset XI
> During final generation of CRC, the output sections will be searched
> for the section that contains the address of the digest
> and this will be the selected section.
>
> The 'DIGEST SECTION "name"' command is removed.
>
> Added multiple test for error conditions when testing DIGEST
>
> Patchset 10
> Forget patchset IX.
> Managed to push the master branch.
> This is the real thing.
>
> Patchset IX
> Endian:
> Default small endian.
> Override with switches: -BE, -LE
> Override in label using [#BE] and [#LE]
>
>    DIGEST "<label>[#BE|#LE]" "CRC32"              (start, end)
>    DIGEST "<label>[#BE|#LE]" "CRC64-ECMA"         (start, end)
>    DIGEST "<label>[#BE|#LE]" "CRC64-GO-ISO"       (start, end)
>    DIGEST "<label>[#BE|#LE]" "CRC64-GO-ISO-R"     (start, end)
>    DIGEST "<label>[#BE|#LE]" "CRC64-WE"           (start, end)
>    DIGEST "<label>[#BE|#LE]" "CRC64-XZ"           (start, end)
>    DIGEST "<label>[#BE|#LE]"  POLY (<params>)     (start, end)
>    DIGEST TABLE "<label>[#BE|#LE]"
>    DIGEST SECTION "<section>"
>
> Improve error messages. Some warnings are now errors.
> Handle reported errors.
> Keep DIGEST SECTION "name", but the whole feature should be deleted.
>
> Patchset VIII
> Memory in testsuite reduced to 16kB ROM and 4 kB RAM
> All situated in the first 64 KB.
>
> Patchset VII
>
> Rebased against master
> Fixed whitespace errors
> Run indent on CRC files
> Moved license to ld directory and renamed to COPYING.<website>
> Added comments to COPYING.www.sunshine2k.de explaining what it is.
>
> Patchset six
> Commands changes so you always get little endian
> but can override this in the linker command file.
>
>    DIGEST "<label>[.BE]" "CRC32"              (start, end)
>    DIGEST "<label>[.BE]" "CRC64-ECMA"         (start, end)
>    DIGEST "<label>[.BE]" "CRC64-GO-ISO"       (start, end)
>    DIGEST "<label>[.BE]" "CRC64-GO-ISO-R"     (start, end)
>    DIGEST "<label>[.BE]" "CRC64-WE"           (start, end)
>    DIGEST "<label>[.BE]" "CRC64-XZ"           (start, end)
>    DIGEST "<label>[.BE]"  POLY (<params>)     (start, end)
>    DIGEST TABLE "<label>[.BE]"
>    DIGEST SECTION "<section>"
>
> Each command has a label, which gets defined (not include ".BE")
>
> It has been built natively as well as with the following setup
> to configure.
>
>    --target=x86_64-pc-linux-gnu
>    --target=x86_64-w64-mingw32
>    --target=aarch64-linux-gnu
>    --target=powerpc64-linux-gnu
>    --target=mips-elf
>    --enable-targets=all --enable-64-bit-bfd
>
> A testsuite for all predefined algorithms are added.
> The testsuite calculates the CRC for the "123456789"
> string which is common.
> The expected checksum value is provided.
> The check ensures that a table is generated, but does not
> check the binary contents of the table.
> It is indirectly checked by using the same table for the
> algorithm
>
> The "DIGEST POLY" commands are checked by using
> the the "CRC32" and "CRC64-ECMA" polynomes and parameters.
>
> There is an online calculator at
> http://www.sunshine2k.de/coding/javascript/crc/crc_js.html
>
> There are no testsuites for the DEBUG ON|OFF, TIMESTAMP and
> "DIGEST SECTION" commands yet.
>
> The testsuite will fail if "DEBUG ON" is executed due to
> verbose output.
>
> The TIMESTAMP output varies with time and is difficult to test.
>
> The DIGEST SECTION command is not tested, because this
> need a little extra discussion.
> I have not seen any other suitable sections being generated.
>
> Since the information athttp://www.sunshine2k.de/  has
> been very useful, their license (MIT) is added.
>
> Code examples for the CRC check routines that needs to be in the
> user source is added to the documentation with copyright statements.
> Should they be there, or is it enough to have the license
> in the source code?
>
> The Makefile.in is manually generated since I had a HDD crash
> and upgraded to a new disk and installed Ubuntu 22.04.
> The autotools are upgraded to 2.71. The build system
> did not like this, so I installed 2.69.
> This changes a lot more than the autotools-2.69 in Ubuntu 18.04
> so I resorted to manual edit of Makefile.in.
>
> Maybe someone should run autotools and generate this automatically
> instead of applying the the manual edit.
>
> Added instructions to build ldint.pdf which is not buildable today.
>
> The system has only been tested on a PC == little endian.
> I do not have a big endian host, but maybe someone could
> test it on such a host.
>
> The way endianess is implemented, it is easy for the user
> to supply a workaround by specifying another endianess.
>
>
>
> Fifth patchset.
>
> +* The linker script has a new command to insert a timestamp
> +  TIMESTAMP
> +  inserts the current time (seconds since Epoch) as a 64-bit value
> +
> +* The linker script syntax has new commands for debugging a linker script
> +  DEBUG ON  turns on debugging
> +  DEBUG OFF turns off debugging
> +
> +* The linker script syntax has new commands for handling CRC-32/64 calculations
> +  on the '.text' section
> +  It uses code fromhttps://www.libcrc.org/
> +
> +  DIGEST "CRC32"              (start, end)
> +  DIGEST "CRC64-ECMA"         (start, end)
> +  DIGEST "CRC64-ISO"          (start, end)
> +  DIGEST "CRC64-WE"           (start, end)
> +  DIGEST POLY (size, polynome)(start, end)
> +  DIGEST POLYI(size, polynome)(start, end) (inversion during CRC calculation)
> +  DIGEST TABLE
> +  DIGEST SECTION "<section>
>
> ran indent on the new files.
>
> The *.d files in the testsuite was edited to
> reduce size and to remove all '(' characters.
> The new linker can link using the testsuite/ld-script/*.t
> linker files, but they do not pass the 'make check-ld'
> test.
>
> Fourth patchset.
> Get rid of binaries in testsuite.
>
> Here is the third patchset for introducing CRC64 generation.
> This has focused on supporting maintainability of the feature.
> In order to do so, CRC32 generation has been introduced to
> detect issues when extending the linker with another algoritm
> for checking integrity of the image.
> This will simplify adding other algorithms like SHA algoritms
> in the future.
>
> In addition, the TIMESTAMP command is used to store the current time
> in seconds since Epoch is stored in the image.
>
> The main routine for calculating CRCs has been broken down
> in subroutines which allow other algorithms.
>
> An algorithm typically needs
> <type> *init_<algo>_tab( <type> poly )
> <type> calc_<algo>_inv( const unsigned char *input_str, size_t num_bytes );
> <type> calc_<algo>    ( const unsigned char *input_str, size_t num_bytes );
> void lang_add_<algo>_syndrome(bool invert, bfd_vma poly)
> void lang_add_<algo>_table(void)
>
> The common functions are:
> * bool get_text_section_contents(void)
>    This reads the '.text' section and its contents
>    Pointers are stored in the global variables
>    asection *text_section
>    char     *text_contents
> * bool set_crc_checksum(bfd_vma crc, bfd_vma addr, bfd_vma size)
>    Stores the CRC at the index addr. The size of the CRC in bytes is specified.
>    Storing something larger that 64-bit is not supported here.
> * bool symbol_lookup(char *name, bfd_vma *val)
>    Gets the value of a symbol into 'val'. Returns false if not found.
>
> To add CRC32 support means adding the following commands
> * CRC32 CRC32                    '(' crc_start ',' crc_end ')'
> * CRC32 POLY  '[' mustbe_exp ']' '(' crc_start ',' crc_end ')'
> * CRC32 POLYI '[' mustbe_exp ']' '(' crc_start ',' crc_end ')'
>
> ================
> Here is the second patchset for introducing CRC64 generation in the linker
> This is mainly looking at indentation and whitespace errors.
> The patches applies cleanly without whitespace error
> except for the last patch which contains the testsuite.
> When the contents of .text is printed out, sometimes
> the last byte of a line in the CRC table is a space.
> This is reported as a whitespace error.
>
> Supports the following new linker commands:
> * CRC64 ECMA                     '(' crc_start ',' crc_end ')'
> * CRC64 ISO                      '(' crc_start ',' crc_end ')'
> * CRC64 POLY  '[' mustbe_exp ']' '(' crc_start ',' crc_end ')'
> * CRC64 POLYI '[' mustbe_exp ']' '(' crc_start ',' crc_end ')'
> * CRC64 TABLE
>
> ECMA  == 0x42F0E1EBA9EA3693
> ISO   == 0xD800000000000000
> POLY  == Allows your own polynome
> POLYI == Allows your own polynome, with inversion during calc
>
> The CRC is calculated from crc_start to crc_end (not included)
>
> The "CRC64 <polynome> command
> * Allows specifying the polynom (ECMA(default), ISO or your own)
> * Allows for inversion in the CRC calculation (CRC64-WE)
> * Allows specifying the area that should be checked.
> * reserves room for the CRC (8 bytes)
> * Declares a symbol ___CRC64___ for the address of the CRC.
> * Declares a symbol ___CRC64_START___ for the first address of the checked area
> * Declares a symbol ___CRC64_END___ for the first address after the checked area
>
> The symbols names are declared in "checksum.h".
> If the names are unsuitable, it is easy to change.
>
> The CRC TABLE command
>    This is used to speed up the CRC calculation.
> * Creates a 2kB table which speeds up the CRC calculation
> * Can insert the 2kB table into the .text section at current location.
> * Declares a symbol ___CRC64_TABLE___ if the table is inserted.
>
> Comments on the code:
>
> The process of calculation involves:
> * Check if CRC is requested
> * Validation of CRC prerequisites
> * get .text section
> * get .text section contents
> * calculate CRC over the area
> * insert the CRC in the correct location
> ====================================
>
> This version also supports the
> * DEBUG ON
> * DEBUG OFF
> turning on/off yydebug
> I find it useful for debugging the build, but it can easly be removed.
>
> This patch contains a lot of debug output, that is enabled by
> #define DEBUG_CRC 1
> This should be removed in the final version.
>
> The ld.texi stuff needs some more work. Not very experienced with that.
>
> Added an entry in NEWS
>
> Added 4 test cases for the different CRC64 polynome commands.
> All testcases generate a CRC table.
>
> The code is using the libcrc released under an MIT license found in
> *https://www.libcrc.org/
> *https://github.com/lammertb/libcrc/tree/v2.0
> Author:  Lammert Bies
> A license for libcrc is added to the patch.
>
> [PATCH v11 01/11] DIGEST: LICENSING
> [PATCH v11 02/11] DIGEST: NEWS
> [PATCH v11 03/11] DIGEST: Documentation
> [PATCH v11 04/11] DIGEST: testsuite
> [PATCH v11 05/11] DIGEST: ldlex.l
> [PATCH v11 06/11] DIGEST: ldgram.y
> [PATCH v11 07/11] DIGEST: ldmain.c
> [PATCH v11 08/11] DIGEST: ldlang.*: add timestamp
> [PATCH v11 09/11] DIGEST: calculation
> [PATCH v11 10/11] DIGEST: Makefile.*
> [PATCH v11 11/11] Build ldint
>
--------------7R6sAs3W6H2wbBQaP8yoNlpY--