From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) by sourceware.org (Postfix) with ESMTPS id EDDC33857B8E for ; Wed, 13 Mar 2024 15:52:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EDDC33857B8E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org EDDC33857B8E Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::42e ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710345153; cv=none; b=ZvuNxccjlYMFHHVcwwqAk56BiHnCsaDFtU/5qd7/PhhEHIBkHKEo0RSSvpsiLtETlJJ1mlDPBxAEoyhdRnppI2SMu2L1eSnVZUQttLVr9dmC9UPVNGqx72w8WoxQxvxXI9jgeVevOq8Q1q6wJRH7ZaOp1NoJhstevpQPW/TLnzI= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710345153; c=relaxed/simple; bh=G+0Ble4j3/Yrd5N0Aw3BnEkx+4RAvtVqFxWtU5kJUew=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=bt2OnkZ0p5V0ysdjsfAqmByLxoFDyxlQkgk/QYo7wBvQUEypmADGp4ZfwoQr2K8ftqCzoMGk+/c2JIXnSwQZv05mM+uGHwR1vr8ktjjfRs0aQlTFXTo22MMmb+kA9O1JpL4Js/MJ+K7iQViTfZKd6U8dx9QPDjzKqFwXQzOPlgs= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-pf1-x42e.google.com with SMTP id d2e1a72fcca58-6e6aa5c5a6fso61898b3a.0 for ; Wed, 13 Mar 2024 08:52:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710345149; x=1710949949; darn=sourceware.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=gPtIiFL7TmO1lZRaEpJkVivA1QIWdzF9WGeyQi8Qb0c=; b=YzhAQBctK+qQehhF6ZydU86mp2cdiCIVJgYF52nX0jbFzDAoJ1OxZUHkUyvt0VkBus 74iYlLIjTHnzH/fb4aJqtqmCWt71NGbMT2EYYGGvmcymRZBAShPBRcrfvdGuldjEZl+W 7MFrqJ2wBspW1HJZIREQGgFkufceS/vDG5G3X927QJz9H1WhosnF8bIzoOL7AWQUNKh8 Cud95XirTWm9z8gCY+fGa87Ck/gqqxhn+9TUCHGyT8bYNW3bUFi8rjP2ZxS9vu4O1EaI ++ssovHd19c/R9TAe8ytQdDqFkfyXL2QVF559DbK64Hn50NfbKnMzwzwS5CtGq+WlDJp 5UVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710345149; x=1710949949; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gPtIiFL7TmO1lZRaEpJkVivA1QIWdzF9WGeyQi8Qb0c=; b=KZGWe/2onnRIAMJN2yzR6HU0PSncFiOzKLbBD6oEffS8caUdAxkjYa2BqDiy3MEKNq PbcwU5H/ax5Zh/uMEt/laGdoPediIVue60v9gdmxzTuuG+Z9gXqkwdmFE9wAgJgbVnHS 6Bme3oa2o4ptO33gwBil714hERYFP5+nNonxi2nYLvSJZmmHLNr+jDaft7W8yS3a7Otj OWsMNwlfSPHqu022FFHXdM6IuX/emM9G4fnJp48+y1MnRYX2DmjsnfJ5WQRg0HdjUo0m KIxizpP3KE/MjfXSsia2QJhkojhx/fsvuSXu3qyqGjn2cGatVAmu/W2q2Ea2PWAr0Fn+ WZGg== X-Gm-Message-State: AOJu0YzMv0LG3iVt5xJeB+TRD6UEr1R+/rqaweksGqJXynCMjXNyE/IR Pyn3q/cConI9DQKhJZAmmLEKNoZ+2aIgMl60WX2Qsw06vpj1rnPpsor228Mh X-Google-Smtp-Source: AGHT+IFYjNNotsmMYa9J+DQlvAwSbmqYgBGYIGLcVD0dynw36cXLcayTdkr/YeMAyxMooP8CEUeDtw== X-Received: by 2002:a05:6a20:2d06:b0:1a0:a438:f161 with SMTP id g6-20020a056a202d0600b001a0a438f161mr8141117pzl.25.1710345149369; Wed, 13 Mar 2024 08:52:29 -0700 (PDT) Received: from gnu-cfl-3.localdomain ([172.58.89.72]) by smtp.gmail.com with ESMTPSA id b3-20020a056a00114300b006e66666de0dsm8100792pfm.199.2024.03.13.08.52.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Mar 2024 08:52:28 -0700 (PDT) Received: from gnu-cfl-3.. (localhost [IPv6:::1]) by gnu-cfl-3.localdomain (Postfix) with ESMTP id D0AFA740050; Wed, 13 Mar 2024 08:52:27 -0700 (PDT) From: "H.J. Lu" To: binutils@sourceware.org Cc: goldstein.w.n@gmail.com, sam@gentoo.org, amodra@gmail.com Subject: [PATCH v8 0/6] elf: Use mmap to map in section contents Date: Wed, 13 Mar 2024 08:52:21 -0700 Message-ID: <20240313155227.513873-1-hjl.tools@gmail.com> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-3011.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_ABUSEAT,RCVD_IN_DNSWL_NONE,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Changes in v8: 1. Rebase against master branch. 2. Add _bfd_elf_link_mmap_section_contents and _bfd_elf_link_munmap_section_contents. Changes in v7: 1. Don't add the --keep-memory linker option. Changes in v6: 1. Add the --keep-memory linker option and always cache symbol and relocation tables for --keep-memory. 2. Always keep symbol table and relocation info for eh_frame to speedup the Rust binary build by ~300x: https://sourceware.org/bugzilla/show_bug.cgi?id=31466 Changes in v5: 1. Drop 2 patches which have been merged onto master branch. 2. Rename _bfd_elf_mmap_section to _bfd_elf_mmap_section_contents. 3. Rename _bfd_mmap_readonly_tracked, _bfd_mmap_readonly_untracked, _bfd_munmap_readonly_untracked, _bfd_mmap_read_untracked to _bfd_mmap_readonly_persistent, _bfd_mmap_readonly_temporary, _bfd_munmap_readonly_temporary and _bfd_mmap_read_temporary. 4. Drop the setup_group change. 5. Fix a typo. 6. Update comments. Changes in v4: 1. Change don't cache symbol nor relocation tables with mmap to opt-in. Changes in v3: 1. Fix non-mmap build. 2. Change the argument name of bfd_mmap_local from flags to prot since its values are PROT_XXX. Changes in v2: 1. Don't hard-code BFD_JUMP_TABLE_COPY in bfd so that elf-bfd.h can be included in libbfd.c. 2. Change the --with-mmap default to true. 3. Check USE_MMAP instead of HAVE_MMAP. 4. Remove the asize parameter to _bfd_mmap_readonly_tracked. 5. Add contents_addr and contents_size to bfd_elf_section_data. 6. Rename _bfd_link_keep_memory to _bfd_elf_link_keep_memory. --- We can use mmap to map in ELF section contents, instead of copying them into memory by hand. We don't need to cache symbol nor relocation tables if they are mapped in. Data to link the 3.5GB clang executable in LLVM 17 debug build on Linux/x86-64 with 32GB RAM is: stdio mmap improvement user 86.73 87.02 -0.3% system 9.55 9.21 3.6% total 100.40 97.66 0.7% maximum set(GB) 17.34 13.14 24% page faults 4047667 3042877 25% and data to link the 275M cc1plus executable in GCC 14 stage 1 build is: user 5.41 5.44 -0.5% system 0.80 0.76 5% total 6.25 6.26 -0.2% maximum set(MB) 1323 968 27% page faults 323451 236371 27% Data shows that these won't improve the single copy linker performance. But they improve the overall system performance when linker is used by reducing linker memory usage and page faults. They allow more parallel linker jobs on LLVM debug build. Here is a quote from Noah Goldstein: "on a large project they are an extremely large speedup". H.J. Lu (6): elf: Use mmap to map in read-only sections elf: Add _bfd_elf_m[un]map_section_contents elf: Use mmap to map in symbol and relocation tables elf: Don't cache symbol nor relocation tables with mmap elf: Always keep symbol table and relocation info for eh_frame elf: Add _bfd_elf_link_m[un]map_section_contents bfd/bfd-in2.h | 24 ++++- bfd/bfd.c | 17 ++++ bfd/bfdwin.c | 8 +- bfd/cache.c | 7 +- bfd/compress.c | 2 +- bfd/elf-bfd.h | 33 ++++++ bfd/elf-eh-frame.c | 4 +- bfd/elf-sframe.c | 4 +- bfd/elf.c | 236 +++++++++++++++++++++++++++++++++++-------- bfd/elf32-i386.c | 8 +- bfd/elf64-x86-64.c | 12 +-- bfd/elfcode.h | 7 +- bfd/elflink.c | 188 +++++++++++++++++++++++++---------- bfd/elfxx-target.h | 6 +- bfd/elfxx-x86.c | 8 +- bfd/elfxx-x86.h | 1 + bfd/libbfd-in.h | 33 +++++- bfd/libbfd.c | 243 ++++++++++++++++++++++++++++++++++++++++++++- bfd/libbfd.h | 33 +++++- bfd/linker.c | 35 ------- bfd/lynx-core.c | 2 +- bfd/opncls.c | 21 ++++ bfd/section.c | 9 +- bfd/sysdep.h | 4 + 24 files changed, 770 insertions(+), 175 deletions(-) -- 2.44.0