From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw1-x112a.google.com (mail-yw1-x112a.google.com [IPv6:2607:f8b0:4864:20::112a]) by sourceware.org (Postfix) with ESMTPS id E98523857736 for ; Fri, 8 Mar 2024 15:32:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org E98523857736 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org E98523857736 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::112a ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1709911954; cv=none; b=it9eLK/0CrGkEpRci2ERP3XMFM0B7y+XvGh3SyhjmatSo6hEdpfBR8MESU+IjKripBH5gxdci4kkaq2ahl8y3bd8LJ32RmhfY+4fG5paVfCmfipUiX2+jTp3F6MhfKbXPT6V0A5VRyJxD0+x5ih0b1lJTNjPwKyR/JlpIJihb8I= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1709911954; c=relaxed/simple; bh=ehJgqARhV5W/IaNm+FOYc0GELbTIEdalnsuQa3jHDd8=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=YQUPNzlEV37w90JtItT0rFcIJi7CneSPEsrScFvUjWZpe26aoeikO6U5v92uLFAS8OmAAitGHtN95EC8wumnwjkk1jbHmanD0vcNGlGf75Qh2ku1CCRe8AfQILVS3rUKgTaZoPyV1Tzpg4AxAo6MIUBJsPlCAcyfVGK7rFXEvhI= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-yw1-x112a.google.com with SMTP id 00721157ae682-609eb87a847so17051067b3.0 for ; Fri, 08 Mar 2024 07:32:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709911950; x=1710516750; darn=sourceware.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=KkM0myb/sMw+u1Puj6yebVuh+XVgKz3TvuGfIDySWWg=; b=RYlu6Vn8xjwwmIQNWfa50+IF4EmcFw1OA45KuF2CcOIH0F/D8LzPXI6l5GQRkeO/Oz FSvSWEp0mK37rhz92hbdIUGNiFddc9QdCHtcN2g/FpGlVq7+4tP+4Fxg5Z5ymU8UfK7S N4IWR3eQ8yHItCiuaOBbRZQoOpWBKIuN1xjRVUyX7NEAJxtx8CnfElZBxTsPj07wBp26 eThJ4BtqOra+/oEqdl5YEcFlfWrzHloG47bJ2+TXwkpjwyFDYkxnvhrZTnxWHUSiDdt9 pR/rZhSPog0kFGh7OXLyov1ketyhi1vH2U6VscH8eDqx9CkxVepHms91hN942TxrrC/U 3Hhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709911950; x=1710516750; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KkM0myb/sMw+u1Puj6yebVuh+XVgKz3TvuGfIDySWWg=; b=GS5zz287wMAr/bWgb4lQ0mjbfMGAvNc6pyzA4q2XuY5Fc05vYkeR6THaqFu2n8rAXA 5UvOIHbQmFwK6vRAEyfnAbkt4La6p1O3XWy8DaaBKqALddgz7mps+QldOB/aZzCWFq3e qb2c7UUmUX162mcYg/GIG6xmvldKQUknEDArh6C60u5W6OSwQuEaiDSwZsurtsryaQ94 4XGACsuV+vL+QO1PbFFidzMlKvET2crqThqn7YN2IxeA0YUpSgz+yYAV+NbolMQdSEW8 +LAPv7mskKX/v/qfglS1pqrwAtGZG6vB3kEus/RdKNqs+Fv5wafQ+/pe/UZgSG7daDFx t9GA== X-Gm-Message-State: AOJu0YwGpeNQBFYA3Wa2uod2XkpvLZXtPvIh17ZftojznCI5cq0XpfoD 1PZPA1V8jqekhg0UfuOVisfhVgzxzFKYopuxGxrk216EmI2+HxBwOmlYnI/r7WBaYjHzs9XVE0V pEUYzfimTkPtUSWaSGErHIIfxpZ1qzOFYRRs= X-Google-Smtp-Source: AGHT+IEcA2IEpkfLHYQ5L8EN+aqZfSr4jtEnRiatoDKCcpvt8pRCCx+6UD8htdopbDHCdGmjd3LLAUwg4Ln0Z6rk1uk= X-Received: by 2002:a81:91d0:0:b0:609:88c3:b820 with SMTP id i199-20020a8191d0000000b0060988c3b820mr498598ywg.21.1709911950181; Fri, 08 Mar 2024 07:32:30 -0800 (PST) MIME-Version: 1.0 References: <20240306232401.1408530-1-hjl.tools@gmail.com> <20240306232401.1408530-7-hjl.tools@gmail.com> <87il1xy2e3.fsf@gentoo.org> In-Reply-To: <87il1xy2e3.fsf@gentoo.org> From: "H.J. Lu" Date: Fri, 8 Mar 2024 07:31:54 -0800 Message-ID: Subject: Re: [PATCH v4 6/6] elf: Don't cache symbol nor relocation tables with mmap To: Sam James Cc: binutils@sourceware.org, goldstein.w.n@gmail.com, amodra@gmail.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-3020.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,GIT_PATCH_0,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Thu, Mar 7, 2024 at 4:48=E2=80=AFPM Sam James wrote: > > "H.J. Lu" writes: > > > During a "-j 8" LLVM 17 debug build on a machine with 32GB RAM and 16GB > > swap, ld was killed by kernel because of out of memory: > > > > [79437.949336] oom-kill:constraint=3DCONSTRAINT_NONE,nodemask=3D(null),= cpuset=3D/,mems_allowed=3D0,global_oom,task_memcg=3D/user.slice/user-1000.s= lice/session-9.scope,task=3Dld,pid=3D797431,uid=3D1000 > > [79437.949349] Out of memory: Killed process 797431 (ld) total-vm:92196= 00kB, anon-rss:6558156kB, file-rss:1792kB, shmem-rss:0kB, UID:1000 pgtables= :17552kB oom_score_adj:0 > > > > Don't cache symbol nor relocation tables if they are mapped in. Data t= o > > link the 3.5GB clang executable in LLVM 17 debug build on Linux/x86-64 > > with 32GB RAM is: > > > > stdio mmap improvement > > user 86.73 87.02 -0.3% > > system 9.55 9.21 3.6% > > total 100.40 97.66 0.7% > > maximum set(GB) 17.34 13.14 24% > > page faults 4047667 3042877 25% > > > > and data to link the 275M cc1plus executable in GCC 14 stage 1 build is= : > > > > user 5.41 5.44 -0.5% > > system 0.80 0.76 5% > > total 6.25 6.26 -0.2% > > maximum set(MB) 1323 968 27% > > page faults 323451 236371 27% > > > > These improve the overall system performance for parallel build by > > reducing memory usage and page faults. > > > > Also rename _bfd_link_keep_memory to _bfd_elf_link_keep_memory. Since > > the --no-keep-memory linker option causes: > > > > FAIL: MIPS eh-frame 3 > > > > in the linker testsuite for mipsisa32r2el-elf target, this is opt-in by > > each backend. > > > > * elf-bfd.h (_bfd_elf_link_keep_memory): New. > > * elf32-i386.c (elf_i386_scan_relocs): Replace > > _bfd_link_keep_memory with _bfd_elf_link_keep_memory. > > * elf64-x86-64.c (elf_x86_64_scan_relocs): Likewise. > > * elflink.c (_bfd_elf_link_iterate_on_relocs): Likewise. > > (elf_link_add_object_symbols): Likewise. > > (init_reloc_cookie): Likewise. > > (_bfd_elf_link_keep_memory): New. > > * libbfd-in.h (_bfd_link_keep_memory): Removed. > > * linker.c (_bfd_link_keep_memory): Likewise. > > * libbfd.h: Regenerated. > > --- > > bfd/elf-bfd.h | 3 ++ > > bfd/elf32-i386.c | 2 +- > > bfd/elf64-x86-64.c | 2 +- > > bfd/elflink.c | 70 ++++++++++++++++++++++++++++++++++++++-------- > > bfd/libbfd-in.h | 3 -- > > bfd/libbfd.h | 3 -- > > bfd/linker.c | 35 ----------------------- > > 7 files changed, 63 insertions(+), 55 deletions(-) > > > > diff --git a/bfd/elf-bfd.h b/bfd/elf-bfd.h > > index da7c5208017..6ed6f13cba2 100644 > > --- a/bfd/elf-bfd.h > > +++ b/bfd/elf-bfd.h > > @@ -3144,6 +3144,9 @@ extern bool _bfd_elf_mmap_section > > extern void _bfd_elf_munmap_section_contents > > (asection *, void *); > > > > +extern bool _bfd_elf_link_keep_memory > > + (struct bfd_link_info *); > > + > > /* Large common section. */ > > extern asection _bfd_elf_large_com_section; > > > > diff --git a/bfd/elf32-i386.c b/bfd/elf32-i386.c > > index 8aa1533d0ba..606e38b472c 100644 > > --- a/bfd/elf32-i386.c > > +++ b/bfd/elf32-i386.c > > @@ -1932,7 +1932,7 @@ elf_i386_scan_relocs (bfd *abfd, > > > > if (elf_section_data (sec)->this_hdr.contents !=3D contents) > > { > > - if (!converted && !_bfd_link_keep_memory (info)) > > + if (!converted && !_bfd_elf_link_keep_memory (info)) > > _bfd_elf_munmap_section_contents (sec, contents); > > else > > { > > diff --git a/bfd/elf64-x86-64.c b/bfd/elf64-x86-64.c > > index f1e06040a5a..e52fa932ffc 100644 > > --- a/bfd/elf64-x86-64.c > > +++ b/bfd/elf64-x86-64.c > > @@ -2590,7 +2590,7 @@ elf_x86_64_scan_relocs (bfd *abfd, struct bfd_lin= k_info *info, > > > > if (elf_section_data (sec)->this_hdr.contents !=3D contents) > > { > > - if (!converted && !_bfd_link_keep_memory (info)) > > + if (!converted && !_bfd_elf_link_keep_memory (info)) > > _bfd_elf_munmap_section_contents (sec, contents); > > else > > { > > diff --git a/bfd/elflink.c b/bfd/elflink.c > > index 4602fb3d10f..d7f5fe90017 100644 > > --- a/bfd/elflink.c > > +++ b/bfd/elflink.c > > @@ -4205,10 +4205,9 @@ _bfd_elf_link_iterate_on_relocs > > || bfd_is_abs_section (o->output_section)) > > continue; > > > > - internal_relocs =3D _bfd_elf_link_info_read_relocs (abfd, info, > > - o, NULL, > > - NULL, > > - _bfd_link_kee= p_memory (info)); > > + internal_relocs =3D _bfd_elf_link_info_read_relocs > > + (abfd, info, o, NULL, NULL, > > + _bfd_elf_link_keep_memory (info)); > > if (internal_relocs =3D=3D NULL) > > return false; > > > > @@ -5574,10 +5573,9 @@ elf_link_add_object_symbols (bfd *abfd, struct b= fd_link_info *info) > > && (s->flags & SEC_DEBUGGING) !=3D 0)) > > continue; > > > > - internal_relocs =3D _bfd_elf_link_info_read_relocs (abfd, info, > > - s, NULL, > > - NULL, > > - _bfd_link_kee= p_memory (info)); > > + internal_relocs =3D _bfd_elf_link_info_read_relocs > > + (abfd, info, s, NULL, NULL, > > + _bfd_elf_link_keep_memory (info)); > > if (internal_relocs =3D=3D NULL) > > goto error_free_vers; > > > > @@ -13630,7 +13628,7 @@ init_reloc_cookie (struct elf_reloc_cookie *coo= kie, > > info->callbacks->einfo (_("%P%X: can not read symbols: %E\n")); > > return false; > > } > > - if (_bfd_link_keep_memory (info) ) > > + if (_bfd_elf_link_keep_memory (info) ) > > { > > symtab_hdr->contents =3D (bfd_byte *) cookie->locsyms; > > info->cache_size +=3D (cookie->locsymcount > > @@ -13667,9 +13665,9 @@ init_reloc_cookie_rels (struct elf_reloc_cookie= *cookie, > > } > > else > > { > > - cookie->rels =3D _bfd_elf_link_info_read_relocs (abfd, info, sec= , > > - NULL, NULL, > > - _bfd_link_keep_memor= y (info)); > > + cookie->rels =3D _bfd_elf_link_info_read_relocs > > + (abfd, info, sec, NULL, NULL, > > + _bfd_elf_link_keep_memory (info)); > > if (cookie->rels =3D=3D NULL) > > return false; > > cookie->rel =3D cookie->rels; > > @@ -15599,3 +15597,51 @@ _bfd_elf_add_dynamic_tags (bfd *output_bfd, st= ruct bfd_link_info *info, > > > > return true; > > } > > + > > +/* Return false if linker should avoid caching relocation information > > + and symbol tables of input files in memory. */ > > + > > +bool > > +_bfd_elf_link_keep_memory (struct bfd_link_info *info) > > +{ > > +#ifdef USE_MMAP > > + /* Don't cache symbol nor relocation tables if they are mapped in. > > + NB: Since the --no-keep-memory linker option causes > > + > > + FAIL: MIPS eh-frame 3 > > + > > + in the linker testsuite for mipsisa32r2el-elf target, this is > > + opt-in by each backend. */ > > Please add a reference to PR31458. Will change it to /* Don't cache symbol nor relocation tables if they are mapped in. NB: Since the --no-keep-memory linker option causes: https://sourceware.org/bugzilla/show_bug.cgi?id=3D31458 this is opt-in by each backend. */ > > + const struct elf_backend_data *bed > > + =3D get_elf_backend_data (info->output_bfd); > > + if (bed->use_mmap) > > + return false; > > +#endif > > + bfd *abfd; > > + bfd_size_type size; > > + > > + if (!info->keep_memory) > > + return false; > > + > > + if (info->max_cache_size =3D=3D (bfd_size_type) -1) > > + return true; > > + > > + abfd =3D info->input_bfds; > > + size =3D info->cache_size; > > + do > > + { > > + if (size >=3D info->max_cache_size) > > + { > > + /* Over the limit. Reduce the memory usage. */ > > + info->keep_memory =3D false; > > + return false; > > + } > > + if (!abfd) > > + break; > > + size +=3D abfd->alloc_size; > > + abfd =3D abfd->link.next; > > + } > > + while (1); > > + > > + return true; > > +} > > diff --git a/bfd/libbfd-in.h b/bfd/libbfd-in.h > > index effe1b86b53..f89de00b551 100644 > > --- a/bfd/libbfd-in.h > > +++ b/bfd/libbfd-in.h > > @@ -848,9 +848,6 @@ extern bfd_byte * _bfd_write_unsigned_leb128 > > extern struct bfd_link_info *_bfd_get_link_info (bfd *) > > ATTRIBUTE_HIDDEN; > > > > -extern bool _bfd_link_keep_memory (struct bfd_link_info *) > > - ATTRIBUTE_HIDDEN; > > - > > extern uintptr_t _bfd_pagesize ATTRIBUTE_HIDDEN; > > extern uintptr_t _bfd_pagesize_m1 ATTRIBUTE_HIDDEN; > > extern uintptr_t _bfd_minimum_mmap_size ATTRIBUTE_HIDDEN; > > diff --git a/bfd/libbfd.h b/bfd/libbfd.h > > index c80f5a86ed1..3ed2e55a34b 100644 > > --- a/bfd/libbfd.h > > +++ b/bfd/libbfd.h > > @@ -854,9 +854,6 @@ extern bfd_byte * _bfd_write_unsigned_leb128 > > extern struct bfd_link_info *_bfd_get_link_info (bfd *) > > ATTRIBUTE_HIDDEN; > > > > -extern bool _bfd_link_keep_memory (struct bfd_link_info *) > > - ATTRIBUTE_HIDDEN; > > - > > extern uintptr_t _bfd_pagesize ATTRIBUTE_HIDDEN; > > extern uintptr_t _bfd_pagesize_m1 ATTRIBUTE_HIDDEN; > > extern uintptr_t _bfd_minimum_mmap_size ATTRIBUTE_HIDDEN; > > diff --git a/bfd/linker.c b/bfd/linker.c > > index 36cca9624c2..eb42a78b622 100644 > > --- a/bfd/linker.c > > +++ b/bfd/linker.c > > @@ -3556,38 +3556,3 @@ _bfd_nolink_bfd_define_start_stop (struct bfd_li= nk_info *info ATTRIBUTE_UNUSED, > > { > > return (struct bfd_link_hash_entry *) _bfd_ptr_bfd_null_error (sec->= owner); > > } > > - > > -/* Return false if linker should avoid caching relocation infomation > > - and symbol tables of input files in memory. */ > > - > > -bool > > -_bfd_link_keep_memory (struct bfd_link_info * info) > > -{ > > - bfd *abfd; > > - bfd_size_type size; > > - > > - if (!info->keep_memory) > > - return false; > > - > > - if (info->max_cache_size =3D=3D (bfd_size_type) -1) > > - return true; > > - > > - abfd =3D info->input_bfds; > > - size =3D info->cache_size; > > - do > > - { > > - if (size >=3D info->max_cache_size) > > - { > > - /* Over the limit. Reduce the memory usage. */ > > - info->keep_memory =3D false; > > - return false; > > - } > > - if (!abfd) > > - break; > > - size +=3D abfd->alloc_size; > > - abfd =3D abfd->link.next; > > - } > > - while (1); > > - > > - return true; > > -} --=20 H.J.