From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by sourceware.org (Postfix) with ESMTPS id 71131385801C for ; Wed, 6 Mar 2024 20:47:36 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 71131385801C Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 71131385801C Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::629 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1709758059; cv=none; b=UrHKXru6hhTTFeEr0v4k1fSDn9lYo8hJdTb52Qoqba2XnT8YbBpe/CkeEz5wkcpETSbXZxwyYfqvvaEdNXysOfKHDE8aO4bUPEf11+IqqxXiqOuWUQ77T8fr9ntABdDqinludSCGUfBLak0dvUAzSTo53dgojq7nB9xD4rlqXE4= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1709758059; c=relaxed/simple; bh=rqEGArxt2nsNWnZY5YJ6yM7GMhm9MJ3r5O2e3dwCG6s=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=d3vs+kSmmDSSCGC9D5M0Lr2PxzK84xmHJYDL6gk6IBMaxrrCSgAYjSKxX8HsuuH0UaT56hKMqDJBgFoynE6Z49EnyomTwPxRzI+oMCAR8DqbENyW2PDSayB8knVOQZUOaWvNDz6uhzqIZwWLcM+SNGSlB+AUwr/q734+vNOpaCo= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1dd10ae77d8so1497915ad.0 for ; Wed, 06 Mar 2024 12:47:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709758055; x=1710362855; darn=sourceware.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=/9KbNAXtRRvSOCaNU/NkjQf6ETL6DwQH0u1I+6TRtzQ=; b=iWWlu+wosxiNmcD8lDMf1c5XZ2GJii/3FFtOvsQER9r2nL/MNb9cHMKcSORmFxMJFT jYP/jy+2x+BtO+ZtDtAjOrhqEHq6bG+qhqjU4C04fFs5A9J+j04JNFeYWL077Iv2Z00Y zC+5UJ52djMV1URCiFY4xr3ZTVp/VUFQC4pb8yxc9Nxr7L69Fo6g9VP5YStL+foqTTiR 0U2VAK1/LvgA0ZkCO13X/FQECaUTDDTKDz8MRafn2AkH1wxIJD/GQRRZu1VwevIWiHhY B8GJ5tI5ina6nWt7RRYI1FFeJHoQi8qGLiDnVz+op8HOQY7k0Z0WNMJNu5Ad4h4QHyKW JrxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709758055; x=1710362855; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/9KbNAXtRRvSOCaNU/NkjQf6ETL6DwQH0u1I+6TRtzQ=; b=F0ezEpk1vknNoZ+XDv8VXUaPd1g9nf0MP/NnkiLXk4DQ4nuAuDgFiIUFWCE1/Ozy76 iL8VpR32BjxycuXWXblC3e2hSwXNfUwNNFJb7WjSLJE5CJVYTj7+0UKzqJ9vdko2UxQA pCAXFyz7y1ys8WK8l+noNLkN+e4JY6yLpJ387jhzdnETYt0LsqIjtAXwXNHnwEBUYafU YyeUqRhjrPI7D6H5R423adao3/ZjYMypReUDlF2SvruKx73Cx1qM91gA3YLK5VT6aWFB ipiafgp849c93h6wK+LjMxxm7uddCZWLGNd94bdg9Z72gJGhbAuvAPR3gMmAlu+r53Oz hWlw== X-Gm-Message-State: AOJu0Yy0zH9VVjxnB0TJBvZ0f6oI5Z0FG6S7HWUj34uYlQah5BRMQ0fO 7ImEaBLcc47Q4pFpvtjBxXVlLnzkR/RO0ATSFnvG0gbbaCVWYjgMCoMxSReW X-Google-Smtp-Source: AGHT+IF+M9ujNzIZYEeDyZ+WS7ivuUyGaTS9r0Zjrxc3azgqhrFl0Tlmxo+dvkQSK0wRjJcVfZapTQ== X-Received: by 2002:a17:902:d4c1:b0:1d8:ab27:d76c with SMTP id o1-20020a170902d4c100b001d8ab27d76cmr7337433plg.51.1709758055466; Wed, 06 Mar 2024 12:47:35 -0800 (PST) Received: from gnu-cfl-3.localdomain ([172.58.89.72]) by smtp.gmail.com with ESMTPSA id n18-20020a170902e55200b001dd3bc79142sm1929518plf.264.2024.03.06.12.47.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Mar 2024 12:47:33 -0800 (PST) Received: from gnu-cfl-3.. (localhost [IPv6:::1]) by gnu-cfl-3.localdomain (Postfix) with ESMTP id 905EA740098; Wed, 6 Mar 2024 12:47:31 -0800 (PST) From: "H.J. Lu" To: binutils@sourceware.org Cc: goldstein.w.n@gmail.com, amodra@gmail.com Subject: [PATCH v3 0/6] elf: Use mmap to map in section contents Date: Wed, 6 Mar 2024 12:47:25 -0800 Message-ID: <20240306204731.1098405-1-hjl.tools@gmail.com> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-3012.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_ABUSEAT,RCVD_IN_DNSWL_NONE,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Changes in v3: 1. Fix non-mmap build. 2. Change the argument name of bfd_mmap_local from flags to prot since its values are PROT_XXX. Changes in v2: 1. Don't hard-code BFD_JUMP_TABLE_COPY in bfd so that elf-bfd.h can be included in libbfd.c. 2. Change the --with-mmap default to true. 3. Check USE_MMAP instead of HAVE_MMAP. 4. Remove the asize parameter to _bfd_mmap_readonly_tracked. 5. Add contents_addr and contents_size to bfd_elf_section_data. 6. Rename _bfd_link_keep_memory to _bfd_elf_link_keep_memory. --- We can use mmap to map in ELF section contents, instead of copying them into memory by hand. We don't need to cache symbol nor relocation tables if they are mapped in. Data to link the 3.5GB clang executable in LLVM 17 debug build on Linux/x86-64 with 32GB RAM is: stdio mmap improvement user 86.73 87.02 -0.3% system 9.55 9.21 3.6% total 100.40 97.66 0.7% maximum set(GB) 17.34 13.14 24% page faults 4047667 3042877 25% and data to link the 275M cc1plus executable in GCC 14 stage 1 build is: user 5.41 5.44 -0.5% system 0.80 0.76 5% total 6.25 6.26 -0.2% maximum set(MB) 1323 968 27% page faults 323451 236371 27% Data shows that these won't improve the single copy linker performance. But they improve the overall system performance when linker is used by reducing linker memory usage and page faults. They allow more parallel linker jobs on LLVM debug build. Here is a quote from Noah Goldstein: "on a large project they are an extremely large speedup". H.J. Lu (6): bfd: Don't hard-code BFD_JUMP_TABLE_COPY bfd: Change the --with-mmap default to true elf: Use mmap to map in read-only sections elf: Add _bfd_elf_mmap_section and _bfd_elf_munmap_section_contents elf: Use mmap to map in symbol and relocation tables elf: Don't cache symbol nor relocation tables with mmap bfd/aout-target.h | 3 + bfd/bfd-in2.h | 30 +++++- bfd/bfd.c | 17 +++ bfd/bfdio.c | 16 +-- bfd/bfdwin.c | 8 +- bfd/cache.c | 11 +- bfd/coff-rs6000.c | 2 + bfd/coffcode.h | 2 + bfd/compress.c | 2 +- bfd/configure | 2 +- bfd/configure.ac | 2 +- bfd/elf-bfd.h | 25 ++++- bfd/elf-eh-frame.c | 4 +- bfd/elf-sframe.c | 4 +- bfd/elf.c | 253 ++++++++++++++++++++++++++++++++++++-------- bfd/elf32-i386.c | 8 +- bfd/elf64-x86-64.c | 12 +-- bfd/elfcode.h | 7 +- bfd/elflink.c | 134 ++++++++++++++++------- bfd/elfxx-target.h | 8 +- bfd/elfxx-x86.c | 7 +- bfd/elfxx-x86.h | 1 + bfd/libbfd-in.h | 33 +++++- bfd/libbfd.c | 249 ++++++++++++++++++++++++++++++++++++++++++- bfd/libbfd.h | 37 ++++++- bfd/libecoff.h | 3 + bfd/linker.c | 35 ------ bfd/lynx-core.c | 2 +- bfd/mach-o-target.c | 1 + bfd/mmo.c | 1 + bfd/opncls.c | 29 ++++- bfd/plugin.c | 1 + bfd/ppcboot.c | 1 + bfd/section.c | 9 +- bfd/som.c | 1 + bfd/targets.c | 2 +- bfd/vms-alpha.c | 1 + 37 files changed, 776 insertions(+), 187 deletions(-) -- 2.44.0