From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from emagii.se (www.emagii.com [185.133.207.17]) by sourceware.org (Postfix) with ESMTPS id 1BD6D3858D39 for ; Mon, 6 Mar 2023 10:00:43 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1BD6D3858D39 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=emagii.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=emagii.com Received: from [10.175.196.145] (84-55-68-216.customers.ownit.se [84.55.68.216]) by emagii.se (Postfix) with ESMTPSA id EBAAF12025D; Mon, 6 Mar 2023 11:00:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=emagii.com; s=default; t=1678096841; bh=+FMQNydTn7FM5R3ruymlb/qg4o964s+bmKXq1tK9ifk=; h=Subject:To:From; b=bQab2aTYBtCqisFZxpstMcIE/t/NZpEQK6QSyAzldr0RMQ97krk72UeIA/G6+40aH TNd+pmyp5+PEd/Ef6M9w59EdaMAoh62POMIeVDu/tHVWcmi60UVdOpT+MMSKVUPa4y JdzUOTBUFjKL2C8TBqiqS8H7bpcYRiGZdDL4me7E= Authentication-Results: emagii.beebytevps.io; spf=pass (sender IP is 84.55.68.216) smtp.mailfrom=binutils@emagii.com smtp.helo=[10.175.196.145] Received-SPF: pass (emagii.beebytevps.io: connection is authenticated) Message-ID: <1d074a88-856c-bf0c-524f-6eaa8c268cfa@emagii.com> Date: Mon, 6 Mar 2023 11:00:33 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: [RFC v0 0/1] Add support for CRC64 generation in linker To: Fangrui Song Cc: Nick Clifton , binutils@sourceware.org References: <20230216204006.1977-1-binutils@emagii.com> <8c80d58c-8c43-90c9-9d0b-9a5de91c16ed@emagii.com> <328005c0-a162-764e-47ba-036dce0ecbe4@redhat.com> <8c16e149-2117-56e1-61bf-a499f05a8fb9@emagii.com> Content-Language: en-US From: Ulf Samuelsson In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-PPP-Message-ID: <167809683978.219667.824027104134413765@localhost.localdomain> X-PPP-Vhost: emagii.com X-Spam-Status: No, score=-6.4 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,SPF_HELO_FAIL,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 2023-03-06 08:50, Fangrui Song wrote: > On Fri, Feb 17, 2023 at 4:03 AM Ulf Samuelsson wrote: >> >> Den 2023-02-17 kl. 12:11, skrev Nick Clifton: >>> Hi Ulf, >>> >>>>> Hi Ulf, can you state why a built-in support of ld is needed? If you >>>>> want to embed a checksum, you can use Output Section Data to reserve a >>>>> few bytes in the output, then use a post-link tool to compute the >>>>> checksum and rewrite the reserved bytes. >>>> In my experience, the post link tools usually work on the binary data >>>> and not the ELF file. >>> The objcopy program can do most of this for you though. For example: >>> >>> % objcopy --dump-section .text=contents-of-text a.out >>> % crc32 contents-of-text > crc32-of-text >>> % objcopy --add-section .crc32=crc32-of-text a.out >>> % readelf -x.crc32 a.out >>> Hex dump of section '.crc32': >>> 0x00000000 32323064 37636339 0a 220d7cc9. >>> >>> In this example the crc32 is stored as ascii text, but I am sure that >>> you can find a version of the crc32 program that generates binary output. >> The crc32 generates a 32-bit CRC. Modern microcontrollers require a >> 64-bit CRC. >> >> The second problem is: where is the .crc32 section and its contents? >> The program needs to access the contents, but it is already linked. >> The typical use is a header in front of the program, and the header >> is part of the ".text" area. >> Can you explain how this would work? >> >>> >>>> Another thing is that the post-link tools I have seen are typically >>>> poorly maintained. >>> ...and so you want to move that maintainership burden onto us, yes ? >> The problem with the post-link tools is that they are hard wired to work on >> special use cased. >> Example of problems >> >> * CRC is fixed to be at a certain address >> >> * CRC table is fixed to be at a certain address. >> >> * Works on binaries and not on ELF files >> >> * You have to have one postprocessor for each file format. > Hi Ulf, I think a natural question from other binutils contributors > is: why is the CRC-64 feature so special that it deserves several > keywords dedicated for it in the linker script language. > You can place placeholder content into the CRC-64 section (say, it is > .crc64), compute its value with a post-link program, then update the > content with > objcopy --update-section .crc64=.... exe If you look at the latest patchset (v11), you will find that there is no "CRC-64" keyword. It is replaced by the "DIGEST" keyword which takes a string parameter describing a "known" algorithm, or the POLY keyword which allows you to specify your own algorithm. If someone wants to support additional algorithms like the SHA series it is esaily extended. > If objcopy --update-section somehow doesn't achieve your goal, it may > be worth a feature request or bug, since the operation is generic and > useful for a large number of users, not just your CRC-64 customers. The problems with supporting things in an external application is that you need an application for every conceivable object file format. The ielftools that I have used for this is 11-12000 lines of code of non-trivial code. I do not not know just how many file format are supported but to me, it appears to be at least a dozen. There is no standard for such a tool, so you will find that many companies need to maintain their own tool for this very purpose. The next problem is that the "crc32" application is really only good for very small microcontrollers with a few kB of flash. In order to support the 128kB-MBs of flash available in modern microcontrollers you need a 64 bit CRC. There is no standard application that generates CRC-64. That means someone needs to write an application that is allows standard polynomes as well as custom polynomes. Since the problem is difficult, many resort to just updating the binary, which means that the code is not easily debuggable. You cannot load the ELF file into the debugger and run, because it does not have the checksum. This patchset really allows people worldwide to get rid of millions of lines of code that no longer is needed. ===== The alternative I am proposing is straightforward code The core code, doing the CRC calculation is well known. The libcrc has not changed in 7 years. >> None of these problems affect the linker since it is agnostic on the >> file format >> as long as there is a ".text" section. >> >> The CRC calculation has been stable on www.libcrc.com for 7 years. >> There is no reason for the CRC calculation to change. >> The only chance I can see is a different polynom, but that is already >> supported. >> >> The biggest problem is of course that it slows down the debugging >> because you cannot download from an ELF file - it lacks the CRC. > I am unsure how a linker script extension is more convenient than > using the existing functionality plus a CRC64 calculator and objcopy. > The linker script extension appears to add a lot of code to the linker > script language, which is already quite challenging to maintain. It supports all object file formats and all architectures immediately. It replaces millions of lines of code. Your solution requires that the process knows which section contains the CRC. If someone moves the CRC to a different section, your toolchain is broken. The crc32 application computes the crc of all the .text section, so it cannot be used. but you need to calculate the checsum of only part of the selected section. You also need to insert the checksum at a user selectable part of the section. On top of that, not having to postprocess the object code simplifies the flow. So objcopy and crc32 does not meet the requirements as of today. KEYWORDS. Right now it adds the following keywords "DIGEST", "TABLE", "POLY". In addition the patch has two more features. * Debugging feature adding "DEBUG", "ON", "OFF" * timestamp feature adding "TIMESTAMP" > If you distribute such object files with CRC64, I don't think it harms > debuggability. There is no CRC64 application, and even if it was, it is a fragile solution, because there is no automatic information on where to put the checksum and what area to calculate the checksum on. Adding checksum calculations to the linker is converting a very difficult problem into a really simple one. If you were managing a team to decide where to put the CRC calculation the linker is the obvious place to look at. Best Regards Ulf Samuelsson > >>> >>>> Adding a post-link step seems like a kludge if the linker can provide >>>> the CRC inside the ELF file. >>> But it also keeps things simple. No new code in the linker = no new >>> bugs in the linker. Solving a problem using existing tools = no need >>> for new versions of the linker when the already existing versions will >>> work just fine. >> The problem is that the existing versions *barely* work. >> Every company have to write their own solution. >> It does not support source level debugging. >> >> Supporting it in the linker makes for a much cleaner solution. >> >>> Cheers >>> Nick >>> >> Best Regards >> >> Ulf Samuelsson >> >>