From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw1-x112b.google.com (mail-yw1-x112b.google.com [IPv6:2607:f8b0:4864:20::112b]) by sourceware.org (Postfix) with ESMTPS id 1AEB23858D1E for ; Thu, 28 Mar 2024 13:29:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1AEB23858D1E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1AEB23858D1E Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::112b ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1711632588; cv=none; b=fEN5LOO6OkTpcCHKgN/HrB1Svv/nYdvU+ysZ5bmIyP8Czi/zN7HiJEqtNNO9iCdbmue9qLW8d4zB53/sx5IOFrVttW5nqZuZH0PAF6hlxIXaYJs2LiVamqo8D8BKvQtJK69IyLEmor8A93qR0ms0Q11LP7S5sNouiJrLmbIYHn4= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1711632588; c=relaxed/simple; bh=0F07a7wFllqMLPqSi1E+1zhXWeif6CN67SQ8/0KEpNY=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=FxDjqif7wn/EGesb+CluPviUGSiAiF2XqDESgyULagvzM6qyBS14oTJCrZWhbDFLJ/r2EWIeGjWeT/P27p2CuXHz2ZRoaErLHTI/XhVGbSpvM0MnrAX8kIm6EXHBVU1ODhCQFV6SK/4M9bLIYaWGaLfBKaZVWW9d+i4AyLWsYjA= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-yw1-x112b.google.com with SMTP id 00721157ae682-611248b4805so9957617b3.0 for ; Thu, 28 Mar 2024 06:29:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711632585; x=1712237385; darn=sourceware.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=cnbH5RGXsZ0G0NCO750ZueceZC7gdba1WuzrbQJhzmo=; b=Cj8IPnrNIOgtBrC787yk4aadVcy1iqadI5kL07NVyxFjnXmKgNQuZ5ez/jS5drzVaQ fjojtppUxhpTZvw0sj1oTUlpZPZlOP1+ISkm6c3S495ETbjbyq6zAasQB8XliWQRuBwN xLtqnu8bfe0gyEIloAsbLfrXOrOwi4NZUHAnu9O5XH4tehWCEJ7E9qgm58J7NE/PaMca gcBAmSTz/mSNENHxv1GkoEcFTnvhB929gOS6+QmKQIipZnPRAQLdb77ZZXnlaUFrSo7V hKr07sYhAlEbYmTtuW39ydT1WhwW6+JR7LT9qUq4QQjgaUcgNLhWviWJYYky+43JHfsc 7Dag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711632585; x=1712237385; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cnbH5RGXsZ0G0NCO750ZueceZC7gdba1WuzrbQJhzmo=; b=i+34dO6C88ALY9NxLx/U+y8V7POYbMaX9IcNvG0jOs9MLQkbzGhUFdlC8QoQL34N1X MvL3KfYLGu2geIAlQacfw1175wJWkFyBc5RALPk1ODplMP8LUFdtsJoDYr9wxN9v0mux kQ1EcAQR+34hStFj9j/qrGSIHpIG5Ehl6SphYKdslz1WOKPV1Szkdwodc6PjNiSUj3XR /FEt1x0/tz0kTIUiMePCL9Fvg8Tit/+zG3Vi0Q/TY1YDzfLH6UDNCg5RzYOnaCqqPhFi xbrcmCiI+MhSbJvgoDkb4A2JnBlKBCfw52IoM81HhG3C1kpjoBXcBjVfeFs5QM6LJzVE 1+Ag== X-Gm-Message-State: AOJu0Yyh+I10b7S7IoiatiDKIP3ixOHgJhb2Wib4O4BMs2CQPpfAuG1p VDAm8RJFe2rTcx32cwB6rMre8MgqVJaWHQVEj9LLPPvxifTnOCJCxS7OMRTPzZhH9KzlNMltfbn llxYexo+LP8YUq7punqU6ZfTCrXqY7sOT X-Google-Smtp-Source: AGHT+IH7HaYqCT7fVB6BblEIdBkhJhL3e6APw9HVZpgVUR2E89hpHqHJuGjny+gdffcfLX373qXVnHuknFjaSnZad0U= X-Received: by 2002:a0d:e045:0:b0:610:bc71:43ab with SMTP id j66-20020a0de045000000b00610bc7143abmr2701605ywe.44.1711632585126; Thu, 28 Mar 2024 06:29:45 -0700 (PDT) MIME-Version: 1.0 References: <20240317121912.799372-1-hjl.tools@gmail.com> In-Reply-To: <20240317121912.799372-1-hjl.tools@gmail.com> From: "H.J. Lu" Date: Thu, 28 Mar 2024 06:29:08 -0700 Message-ID: Subject: PING: [PATCH v12 0/6] elf: Use mmap to map in section contents To: Binutils Cc: Noah Goldstein , Sam James , Alan Modra Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-3013.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Sun, Mar 17, 2024 at 5:19=E2=80=AFAM H.J. Lu wrote= : > > Changes in v12: > > 1. Don't call _bfd_elf_link_keep_memory for x86 since it always returns > false. > 2. Make _bfd_elf_link_keep_memory static. > > Changes in v11: > > 1. Update _bfd_mmap_read_temporary to allocate buffer as needed. > 2. Don't allocate buffer when _bfd_mmap_read_temporary is called. > > Changes in v10: > > 1. Malloc a 4 * page size buffer for external relocations for the final > link to avoid mmap/munmap on small relocation sections. > 2. Malloc a buffer for input section contents which are smaller than > 4 * page size or can't be mapped for the final link. > > Changes in v9: > > 1. Use MAP_FAILED for mmap failure. > > Changes in v8: > > 1. Rebase against master branch. > 2. Add _bfd_elf_link_mmap_section_contents and > _bfd_elf_link_munmap_section_contents. > > Changes in v7: > > 1. Don't add the --keep-memory linker option. > > Changes in v6: > > 1. Add the --keep-memory linker option and always cache symbol and > relocation tables for --keep-memory. > 2. Always keep symbol table and relocation info for eh_frame to speedup > the Rust binary build by ~300x: > > https://sourceware.org/bugzilla/show_bug.cgi?id=3D31466 > > Changes in v5: > > 1. Drop 2 patches which have been merged onto master branch. > 2. Rename _bfd_elf_mmap_section to _bfd_elf_mmap_section_contents. > 3. Rename _bfd_mmap_readonly_tracked, _bfd_mmap_readonly_untracked, > _bfd_munmap_readonly_untracked, _bfd_mmap_read_untracked to > _bfd_mmap_readonly_persistent, _bfd_mmap_readonly_temporary, > _bfd_munmap_readonly_temporary and _bfd_mmap_read_temporary. > 4. Drop the setup_group change. > 5. Fix a typo. > 6. Update comments. > > Changes in v4: > > 1. Change don't cache symbol nor relocation tables with mmap to opt-in. > > Changes in v3: > > 1. Fix non-mmap build. > 2. Change the argument name of bfd_mmap_local from flags to prot since > its values are PROT_XXX. > > Changes in v2: > > 1. Don't hard-code BFD_JUMP_TABLE_COPY in bfd so that elf-bfd.h can be > included in libbfd.c. > 2. Change the --with-mmap default to true. > 3. Check USE_MMAP instead of HAVE_MMAP. > 4. Remove the asize parameter to _bfd_mmap_readonly_tracked. > 5. Add contents_addr and contents_size to bfd_elf_section_data. > 6. Rename _bfd_link_keep_memory to _bfd_elf_link_keep_memory. > > --- > We can use mmap to map in ELF section contents, instead of copying them > into memory by hand. We don't need to cache symbol nor relocation tables > if they are mapped in. Data to link the 3.5GB clang executable in LLVM > 17 debug build on Linux/x86-64 with 32GB RAM is: > > stdio mmap improvement > user 86.73 87.02 -0.3% > system 9.55 9.21 3.6% > total 100.40 97.66 0.7% > maximum set(GB) 17.34 13.14 24% > page faults 4047667 3042877 25% > > and data to link the 275M cc1plus executable in GCC 14 stage 1 build is: > > user 5.41 5.44 -0.5% > system 0.80 0.76 5% > total 6.25 6.26 -0.2% > maximum set(MB) 1323 968 27% > page faults 323451 236371 27% > > Data shows that these won't improve the single copy linker performance. > But they improve the overall system performance when linker is used by > reducing linker memory usage and page faults. They allow more parallel > linker jobs on LLVM debug build. > > Here is a quote from Noah Goldstein: "on a large project they are an > extremely large speedup". > > H.J. Lu (6): > elf: Use mmap to map in read-only sections > elf: Add _bfd_elf_m[un]map_section_contents > elf: Use mmap to map in symbol and relocation tables > elf: Don't cache symbol nor relocation tables with mmap > elf: Always keep symbol table and relocation info for eh_frame > elf: Add _bfd_elf_link_m[un]map_section_contents > > bfd/bfd-in2.h | 24 ++++- > bfd/bfd.c | 17 +++ > bfd/bfdwin.c | 8 +- > bfd/cache.c | 7 +- > bfd/compress.c | 2 +- > bfd/elf-bfd.h | 24 +++++ > bfd/elf-eh-frame.c | 4 +- > bfd/elf-sframe.c | 4 +- > bfd/elf.c | 246 ++++++++++++++++++++++++++++++++++-------- > bfd/elf32-i386.c | 8 +- > bfd/elf64-x86-64.c | 12 +-- > bfd/elfcode.h | 7 +- > bfd/elflink.c | 213 ++++++++++++++++++++++++------------ > bfd/elfxx-target.h | 6 +- > bfd/elfxx-x86.c | 8 +- > bfd/elfxx-x86.h | 1 + > bfd/libbfd-in.h | 33 +++++- > bfd/libbfd.c | 262 ++++++++++++++++++++++++++++++++++++++++++++- > bfd/libbfd.h | 33 +++++- > bfd/linker.c | 35 ------ > bfd/lynx-core.c | 2 +- > bfd/opncls.c | 21 ++++ > bfd/section.c | 9 +- > bfd/sysdep.h | 4 + > 24 files changed, 795 insertions(+), 195 deletions(-) > > -- > 2.44.0 > PING. --=20 H.J.