From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 48) id D56033858D35; Sat, 18 Nov 2023 18:48:04 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D56033858D35 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1700333284; bh=B352AcKggbyVzgce0QWdPCMdK5G02WPDdbwSExrlk44=; h=From:To:Subject:Date:From; b=BFYvfz4kimX4gVS8Ci/vyisYW1tYxFIY26eLdEGwMh0mE/zwPzfoBlrSdiVx0llcS +VNsZs3S+wP0pBVueqLyLept5wATDN0FDwtTBObaubu2ZyYYdZXVHohlui1WsTSX/X 0JVoCdi8nw8KJJJf17Yk6HMnK0nVLZpN9rG6t3Fc= From: "jyescas at google dot com" To: glibc-bugs@sourceware.org Subject: [Bug dynamic-link/31076] New: Extra struct vm_area_struct with ---p created when PAGE_SIZE < max-page-size Date: Sat, 18 Nov 2023 18:48:04 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: glibc X-Bugzilla-Component: dynamic-link X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: jyescas at google dot com X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: P2 X-Bugzilla-Assigned-To: unassigned at sourceware dot org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version bug_status bug_severity priority component assigned_to reporter target_milestone attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 List-Id: https://sourceware.org/bugzilla/show_bug.cgi?id=3D31076 Bug ID: 31076 Summary: Extra struct vm_area_struct with ---p created when PAGE_SIZE < max-page-size Product: glibc Version: unspecified Status: UNCONFIRMED Severity: normal Priority: P2 Component: dynamic-link Assignee: unassigned at sourceware dot org Reporter: jyescas at google dot com Target Milestone: --- Created attachment 15224 --> https://sourceware.org/bugzilla/attachment.cgi?id=3D15224&action=3Ded= it Contains the C files, headers and makefile to reproduce the issue. # The story of the extra struct vm_area_struct and the loader ## Problem When the page size is `4096` and shared libraries and binaries are `16384/65536` elf aligned with the flag `-Wl,-z,max-page-size=3D[16384|6553= 6]` the dynamic linker (loader) creates an extra vm_area_struct for each segmen= t. This happens because the loader allocates a memory area big enough to map the shared library and then mprotect every segment at page size boundaries instead of a `p_align` boundary. The size of `struct vm_area_struct` is 152 bytes in the `Linux 6.3.13` kern= el. The increase in the number of vm_area_struct is an issue for low end memory devices (mobile devices, etc). We have seen an increase of 30MBs of unreclaimable kernel memory on those devices due the extra vm_area_struct. ## Linux and gcc details ``` $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.1 LTS Release: 20.04 Codename: focal $ gcc --version gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 ``` ## How to reproduce it (See attachments) In a `4096` page size kernel, compile a shared library with 16384 or 65536 = elf alignment. In the `sample` directory, there are these files: ``` $ tree sample/ sample/ =E2=94=9C=E2=94=80=E2=94=80 arithmetic.c =E2=94=9C=E2=94=80=E2=94=80 arithmetic.h =E2=94=9C=E2=94=80=E2=94=80 main.c =E2=94=94=E2=94=80=E2=94=80 Makefile ``` To compile the shared library and main, run: ``` $ cd sample # Build shared library and main $ make build ``` To load and execute a program, there are 2 possible ways: 1. Ask the Linux kernel to load the program and dynamic linker, once that t= he dynamic linker is loaded, it will load the shared libraries, in this case `libarithmetic.so`. ``` $ make run ``` 2. Ask the kernel to load the dynamic linker and then the dynamic linker wi= ll load the program and shared libraries, in this case `libarithmetic.so`. ``` $ make runloader ``` ### Linux kernel loads the program and dynamic linker When the `Linux kernel` loads the program and dynamic linker, the dynamic linker loads the shared libraries, we can see that an extra vm_area_struct is used with the permissions `---p` in the shared libraries that are 16384/65536 elf aligned. ``` $ make run LD_LIBRARY_PATH=3D./build ./build/main # In another console, run $ cat /proc/1717808/maps 564d8e90f000-564d8e910000 r--p 00000000 08:05 2519504 /sample/build/= main 564d8e91f000-564d8e920000 r-xp 00010000 08:05 2519504 /sample/build/= main 564d8e92f000-564d8e930000 r--p 00020000 08:05 2519504 /sample/build/= main 564d8e93f000-564d8e940000 r--p 00020000 08:05 2519504 /sample/build/= main 564d8e940000-564d8e941000 rw-p 00021000 08:05 2519504 /sample/build/= main 564d9055d000-564d9057e000 rw-p 00000000 00:00 0 [heap] 7ff6417ba000-7ff6417bd000 rw-p 00000000 00:00 0 7ff6417bd000-7ff6417df000 r--p 00000000 08:05 2623619=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7ff6417df000-7ff641957000 r-xp 00022000 08:05 2623619=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7ff641957000-7ff6419a5000 r--p 0019a000 08:05 2623619=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7ff6419a5000-7ff6419a9000 r--p 001e7000 08:05 2623619=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7ff6419a9000-7ff6419ab000 rw-p 001eb000 08:05 2623619=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7ff6419ab000-7ff6419af000 rw-p 00000000 00:00 0 7ff6419c0000-7ff6419c1000 r--p 00000000 08:05 2519503=20=20=20=20=20=20=20 /sample/build/libarithmetic.so 7ff6419c1000-7ff6419d0000 ---p 00001000 08:05 2519503=20=20=20=20=20=20=20 /sample/build/libarithmetic.so 7ff6419d0000-7ff6419d1000 r-xp 00010000 08:05 2519503=20=20=20=20=20=20=20 /sample/build/libarithmetic.so 7ff6419d1000-7ff6419e0000 ---p 00011000 08:05 2519503=20=20=20=20=20=20=20 /sample/build/libarithmetic.so 7ff6419e0000-7ff6419e1000 r--p 00020000 08:05 2519503=20=20=20=20=20=20=20 /sample/build/libarithmetic.so 7ff6419e1000-7ff6419f0000 ---p 00021000 08:05 2519503=20=20=20=20=20=20=20 /sample/build/libarithmetic.so 7ff6419f0000-7ff6419f1000 r--p 00020000 08:05 2519503=20=20=20=20=20=20=20 /sample/build/libarithmetic.so 7ff6419f1000-7ff6419f2000 rw-p 00021000 08:05 2519503=20=20=20=20=20=20=20 /sample/build/libarithmetic.so 7ff6419f2000-7ff6419f4000 rw-p 00000000 00:00 0 7ff6419f4000-7ff6419f5000 r--p 00000000 08:05 2623613=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7ff6419f5000-7ff641a18000 r-xp 00001000 08:05 2623613=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7ff641a18000-7ff641a20000 r--p 00024000 08:05 2623613=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7ff641a21000-7ff641a22000 r--p 0002c000 08:05 2623613=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7ff641a22000-7ff641a23000 rw-p 0002d000 08:05 2623613=20=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7ff641a23000-7ff641a24000 rw-p 00000000 00:00 0 7ffdfd748000-7ffdfd769000 rw-p 00000000 00:00 0 [stack] 7ffdfd77c000-7ffdfd77f000 r--p 00000000 00:00 0 [vvar] 7ffdfd77f000-7ffdfd780000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ``` > Note that in `main` there is not a vm_area_struct with `---p` permissions= due > the Linux kernel loads the program. ### Linux kernel ONLY loads the dynamic linker When the Linux kernel `ONLY` loads the dynamic linker, and the dynamic link= er loads the shared libraries and program, we can see that an extra vm_area_st= ruct is used with the permissions `---p` in the shared libraries and program that are 16384/65536 elf aligned. ``` $ make runloader LD_LIBRARY_PATH=3D./build /lib64/ld-linux-x86-64.so.2 ./build/main # In another console, run $ cat /proc/1711495/maps 555557048000-555557069000 rw-p 00000000 00:00 0 [heap] 7f58a13c0000-7f58a13c3000 rw-p 00000000 00:00 0 7f58a13c3000-7f58a13e5000 r--p 00000000 08:05 2623619=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f58a13e5000-7f58a155d000 r-xp 00022000 08:05 2623619=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f58a155d000-7f58a15ab000 r--p 0019a000 08:05 2623619=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f58a15ab000-7f58a15af000 r--p 001e7000 08:05 2623619=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f58a15af000-7f58a15b1000 rw-p 001eb000 08:05 2623619=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f58a15b1000-7f58a15b5000 rw-p 00000000 00:00 0 7f58a15c6000-7f58a15c7000 r--p 00000000 08:05 2519503=20=20=20=20=20=20 /sample/build/libarithmetic.so 7f58a15c7000-7f58a15d6000 ---p 00001000 08:05 2519503=20=20=20=20=20=20 /sample/build/libarithmetic.so 7f58a15d6000-7f58a15d7000 r-xp 00010000 08:05 2519503=20=20=20=20=20=20 /sample/build/libarithmetic.so 7f58a15d7000-7f58a15e6000 ---p 00011000 08:05 2519503=20=20=20=20=20=20 /sample/build/libarithmetic.so 7f58a15e6000-7f58a15e7000 r--p 00020000 08:05 2519503=20=20=20=20=20=20 /sample/build/libarithmetic.so 7f58a15e7000-7f58a15f6000 ---p 00021000 08:05 2519503=20=20=20=20=20=20 /sample/build/libarithmetic.so 7f58a15f6000-7f58a15f7000 r--p 00020000 08:05 2519503=20=20=20=20=20=20 /sample/build/libarithmetic.so 7f58a15f7000-7f58a15f8000 rw-p 00021000 08:05 2519503=20=20=20=20=20=20 /sample/build/libarithmetic.so 7f58a15f8000-7f58a15fa000 rw-p 00000000 00:00 0 7f58a15fa000-7f58a15fb000 r--p 00000000 08:05 2519504 /sample/build/m= ain 7f58a15fb000-7f58a160a000 ---p 00001000 08:05 2519504 /sample/build/m= ain 7f58a160a000-7f58a160b000 r-xp 00010000 08:05 2519504 /sample/build/m= ain 7f58a160b000-7f58a161a000 ---p 00011000 08:05 2519504 /sample/build/m= ain 7f58a161a000-7f58a161b000 r--p 00020000 08:05 2519504 /sample/build/m= ain 7f58a161b000-7f58a162a000 ---p 00021000 08:05 2519504 /sample/build/m= ain 7f58a162a000-7f58a162b000 r--p 00020000 08:05 2519504 /sample/build/m= ain 7f58a162b000-7f58a162c000 rw-p 00021000 08:05 2519504 /sample/build/m= ain 7f58a162c000-7f58a162d000 r--p 00000000 08:05 2623613=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f58a162d000-7f58a1650000 r-xp 00001000 08:05 2623613=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f58a1650000-7f58a1658000 r--p 00024000 08:05 2623613=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f58a1659000-7f58a165a000 r--p 0002c000 08:05 2623613=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f58a165a000-7f58a165b000 rw-p 0002d000 08:05 2623613=20=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f58a165b000-7f58a165c000 rw-p 00000000 00:00 0 7ffe19e5c000-7ffe19e7d000 rw-p 00000000 00:00 0 [stack] 7ffe19f7e000-7ffe19f81000 r--p 00000000 00:00 0 [vvar] 7ffe19f81000-7ffe19f82000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ``` ### Linux kernel page size is 4096 and the elf alignment is 4096 When a shared library and executable are linked using the flag `-Wl,-z,max-page-size=3D4096` and loaded by the dynamic linker, we can see that there is `NOT` an extra vm_area_struct with permissions `---p`. ``` $ make runloader LD_LIBRARY_PATH=3D./build /lib64/ld-linux-x86-64.so.2 ./build/main # In another console, run $ cat /proc/1721958/maps 555556cef000-555556d10000 rw-p 00000000 00:00 0 [heap] 7f70bb246000-7f70bb249000 rw-p 00000000 00:00 0 7f70bb249000-7f70bb26b000 r--p 00000000 08:05 2623619=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f70bb26b000-7f70bb3e3000 r-xp 00022000 08:05 2623619=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f70bb3e3000-7f70bb431000 r--p 0019a000 08:05 2623619=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f70bb431000-7f70bb435000 r--p 001e7000 08:05 2623619=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f70bb435000-7f70bb437000 rw-p 001eb000 08:05 2623619=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/libc-2.31.so 7f70bb437000-7f70bb43b000 rw-p 00000000 00:00 0 7f70bb44c000-7f70bb44d000 r--p 00000000 08:05 2519503=20=20=20=20=20 /sample/build/libarithmetic.so 7f70bb44d000-7f70bb44e000 r-xp 00001000 08:05 2519503=20=20=20=20=20 /sample/build/libarithmetic.so 7f70bb44e000-7f70bb44f000 r--p 00002000 08:05 2519503=20=20=20=20=20 /sample/build/libarithmetic.so 7f70bb44f000-7f70bb450000 r--p 00002000 08:05 2519503=20=20=20=20=20 /sample/build/libarithmetic.so 7f70bb450000-7f70bb451000 rw-p 00003000 08:05 2519503=20=20=20=20=20 /sample/build/libarithmetic.so 7f70bb451000-7f70bb453000 rw-p 00000000 00:00 0 7f70bb453000-7f70bb454000 r--p 00000000 08:05 2519504 /sample/build/ma= in 7f70bb454000-7f70bb455000 r-xp 00001000 08:05 2519504 /sample/build/ma= in 7f70bb455000-7f70bb456000 r--p 00002000 08:05 2519504 /sample/build/ma= in 7f70bb456000-7f70bb457000 r--p 00002000 08:05 2519504 /sample/build/ma= in 7f70bb457000-7f70bb458000 rw-p 00003000 08:05 2519504 /sample/build/ma= in 7f70bb458000-7f70bb459000 r--p 00000000 08:05 2623613=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f70bb459000-7f70bb47c000 r-xp 00001000 08:05 2623613=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f70bb47c000-7f70bb484000 r--p 00024000 08:05 2623613=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f70bb485000-7f70bb486000 r--p 0002c000 08:05 2623613=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f70bb486000-7f70bb487000 rw-p 0002d000 08:05 2623613=20=20=20=20=20 /usr/lib/x86_64-linux-gnu/ld-2.31.so 7f70bb487000-7f70bb488000 rw-p 00000000 00:00 0 7ffe42e67000-7ffe42e88000 rw-p 00000000 00:00 0 [stack] 7ffe42ebe000-7ffe42ec1000 r--p 00000000 00:00 0 [vvar] 7ffe42ec1000-7ffe42ec2000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ``` ## Solution 1 - Modify the dynamic linker When the dynamic linker loads the shared libraries, extend the `vm_area_str= uct` to be at a `p_align` boundary. This is done by passing to `mprotect()` the right address space ranges. ## Solution 2 - Modify the static linker When the static linker creates the program segments, make sure that the `p_filesz` and `p_memsz` are extended to a `p_align` boundary. This can be = done by inserting a new ELF section at every PT_LOAD ELF segment that will serve= as padding. --=20 You are receiving this mail because: You are on the CC list for the bug.=