public inbox for glibc-bugs@sourceware.org
help / color / mirror / Atom feed
From: "jyescas at google dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug dynamic-link/31076] New: Extra struct vm_area_struct with ---p created when PAGE_SIZE < max-page-size
Date: Sat, 18 Nov 2023 18:48:04 +0000 [thread overview]
Message-ID: <bug-31076-131@http.sourceware.org/bugzilla/> (raw)
https://sourceware.org/bugzilla/show_bug.cgi?id=31076
Bug ID: 31076
Summary: Extra struct vm_area_struct with ---p created when
PAGE_SIZE < max-page-size
Product: glibc
Version: unspecified
Status: UNCONFIRMED
Severity: normal
Priority: P2
Component: dynamic-link
Assignee: unassigned at sourceware dot org
Reporter: jyescas at google dot com
Target Milestone: ---
Created attachment 15224
--> https://sourceware.org/bugzilla/attachment.cgi?id=15224&action=edit
Contains the C files, headers and makefile to reproduce the issue.
# The story of the extra struct vm_area_struct and the loader
## Problem
When the page size is `4096` and shared libraries and binaries are
`16384/65536` elf aligned with the flag `-Wl,-z,max-page-size=[16384|65536]`
the dynamic linker (loader) creates an extra vm_area_struct for each segment.
This happens because the loader allocates a memory area big enough to
map the shared library and then mprotect every segment at
page size boundaries instead of a `p_align` boundary.
The size of `struct vm_area_struct` is 152 bytes in the `Linux 6.3.13` kernel.
The increase in the number of vm_area_struct is an issue for low end memory
devices (mobile devices, etc). We have seen an increase of 30MBs of
unreclaimable kernel memory on those devices due the extra vm_area_struct.
## Linux and gcc details
```
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
```
## How to reproduce it (See attachments)
In a `4096` page size kernel, compile a shared library with 16384 or 65536 elf
alignment.
In the `sample` directory, there are these files:
```
$ tree sample/
sample/
├── arithmetic.c
├── arithmetic.h
├── main.c
└── Makefile
```
To compile the shared library and main, run:
```
$ cd sample
# Build shared library and main
$ make build
```
To load and execute a program, there are 2 possible ways:
1. Ask the Linux kernel to load the program and dynamic linker, once that the
dynamic linker is loaded, it will load the shared libraries, in this case
`libarithmetic.so`.
```
$ make run
```
2. Ask the kernel to load the dynamic linker and then the dynamic linker will
load the program and shared libraries, in this case `libarithmetic.so`.
```
$ make runloader
```
### Linux kernel loads the program and dynamic linker
When the `Linux kernel` loads the program and dynamic linker, the dynamic
linker
loads the shared libraries, we can see that an extra vm_area_struct is used
with
the permissions `---p` in the shared libraries that are 16384/65536 elf
aligned.
```
$ make run
LD_LIBRARY_PATH=./build ./build/main
# In another console, run
$ cat /proc/1717808/maps
564d8e90f000-564d8e910000 r--p 00000000 08:05 2519504 /sample/build/main
564d8e91f000-564d8e920000 r-xp 00010000 08:05 2519504 /sample/build/main
564d8e92f000-564d8e930000 r--p 00020000 08:05 2519504 /sample/build/main
564d8e93f000-564d8e940000 r--p 00020000 08:05 2519504 /sample/build/main
564d8e940000-564d8e941000 rw-p 00021000 08:05 2519504 /sample/build/main
564d9055d000-564d9057e000 rw-p 00000000 00:00 0 [heap]
7ff6417ba000-7ff6417bd000 rw-p 00000000 00:00 0
7ff6417bd000-7ff6417df000 r--p 00000000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7ff6417df000-7ff641957000 r-xp 00022000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7ff641957000-7ff6419a5000 r--p 0019a000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7ff6419a5000-7ff6419a9000 r--p 001e7000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7ff6419a9000-7ff6419ab000 rw-p 001eb000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7ff6419ab000-7ff6419af000 rw-p 00000000 00:00 0
7ff6419c0000-7ff6419c1000 r--p 00000000 08:05 2519503
/sample/build/libarithmetic.so
7ff6419c1000-7ff6419d0000 ---p 00001000 08:05 2519503
/sample/build/libarithmetic.so
7ff6419d0000-7ff6419d1000 r-xp 00010000 08:05 2519503
/sample/build/libarithmetic.so
7ff6419d1000-7ff6419e0000 ---p 00011000 08:05 2519503
/sample/build/libarithmetic.so
7ff6419e0000-7ff6419e1000 r--p 00020000 08:05 2519503
/sample/build/libarithmetic.so
7ff6419e1000-7ff6419f0000 ---p 00021000 08:05 2519503
/sample/build/libarithmetic.so
7ff6419f0000-7ff6419f1000 r--p 00020000 08:05 2519503
/sample/build/libarithmetic.so
7ff6419f1000-7ff6419f2000 rw-p 00021000 08:05 2519503
/sample/build/libarithmetic.so
7ff6419f2000-7ff6419f4000 rw-p 00000000 00:00 0
7ff6419f4000-7ff6419f5000 r--p 00000000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7ff6419f5000-7ff641a18000 r-xp 00001000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7ff641a18000-7ff641a20000 r--p 00024000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7ff641a21000-7ff641a22000 r--p 0002c000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7ff641a22000-7ff641a23000 rw-p 0002d000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7ff641a23000-7ff641a24000 rw-p 00000000 00:00 0
7ffdfd748000-7ffdfd769000 rw-p 00000000 00:00 0 [stack]
7ffdfd77c000-7ffdfd77f000 r--p 00000000 00:00 0 [vvar]
7ffdfd77f000-7ffdfd780000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
```
> Note that in `main` there is not a vm_area_struct with `---p` permissions due
> the Linux kernel loads the program.
### Linux kernel ONLY loads the dynamic linker
When the Linux kernel `ONLY` loads the dynamic linker, and the dynamic linker
loads the shared libraries and program, we can see that an extra vm_area_struct
is used with the permissions `---p` in the shared libraries and program that
are 16384/65536 elf aligned.
```
$ make runloader
LD_LIBRARY_PATH=./build /lib64/ld-linux-x86-64.so.2 ./build/main
# In another console, run
$ cat /proc/1711495/maps
555557048000-555557069000 rw-p 00000000 00:00 0 [heap]
7f58a13c0000-7f58a13c3000 rw-p 00000000 00:00 0
7f58a13c3000-7f58a13e5000 r--p 00000000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f58a13e5000-7f58a155d000 r-xp 00022000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f58a155d000-7f58a15ab000 r--p 0019a000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f58a15ab000-7f58a15af000 r--p 001e7000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f58a15af000-7f58a15b1000 rw-p 001eb000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f58a15b1000-7f58a15b5000 rw-p 00000000 00:00 0
7f58a15c6000-7f58a15c7000 r--p 00000000 08:05 2519503
/sample/build/libarithmetic.so
7f58a15c7000-7f58a15d6000 ---p 00001000 08:05 2519503
/sample/build/libarithmetic.so
7f58a15d6000-7f58a15d7000 r-xp 00010000 08:05 2519503
/sample/build/libarithmetic.so
7f58a15d7000-7f58a15e6000 ---p 00011000 08:05 2519503
/sample/build/libarithmetic.so
7f58a15e6000-7f58a15e7000 r--p 00020000 08:05 2519503
/sample/build/libarithmetic.so
7f58a15e7000-7f58a15f6000 ---p 00021000 08:05 2519503
/sample/build/libarithmetic.so
7f58a15f6000-7f58a15f7000 r--p 00020000 08:05 2519503
/sample/build/libarithmetic.so
7f58a15f7000-7f58a15f8000 rw-p 00021000 08:05 2519503
/sample/build/libarithmetic.so
7f58a15f8000-7f58a15fa000 rw-p 00000000 00:00 0
7f58a15fa000-7f58a15fb000 r--p 00000000 08:05 2519504 /sample/build/main
7f58a15fb000-7f58a160a000 ---p 00001000 08:05 2519504 /sample/build/main
7f58a160a000-7f58a160b000 r-xp 00010000 08:05 2519504 /sample/build/main
7f58a160b000-7f58a161a000 ---p 00011000 08:05 2519504 /sample/build/main
7f58a161a000-7f58a161b000 r--p 00020000 08:05 2519504 /sample/build/main
7f58a161b000-7f58a162a000 ---p 00021000 08:05 2519504 /sample/build/main
7f58a162a000-7f58a162b000 r--p 00020000 08:05 2519504 /sample/build/main
7f58a162b000-7f58a162c000 rw-p 00021000 08:05 2519504 /sample/build/main
7f58a162c000-7f58a162d000 r--p 00000000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f58a162d000-7f58a1650000 r-xp 00001000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f58a1650000-7f58a1658000 r--p 00024000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f58a1659000-7f58a165a000 r--p 0002c000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f58a165a000-7f58a165b000 rw-p 0002d000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f58a165b000-7f58a165c000 rw-p 00000000 00:00 0
7ffe19e5c000-7ffe19e7d000 rw-p 00000000 00:00 0 [stack]
7ffe19f7e000-7ffe19f81000 r--p 00000000 00:00 0 [vvar]
7ffe19f81000-7ffe19f82000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
```
### Linux kernel page size is 4096 and the elf alignment is 4096
When a shared library and executable are linked using the flag
`-Wl,-z,max-page-size=4096`
and loaded by the dynamic linker, we can see that there is `NOT` an extra
vm_area_struct with
permissions `---p`.
```
$ make runloader
LD_LIBRARY_PATH=./build /lib64/ld-linux-x86-64.so.2 ./build/main
# In another console, run
$ cat /proc/1721958/maps
555556cef000-555556d10000 rw-p 00000000 00:00 0 [heap]
7f70bb246000-7f70bb249000 rw-p 00000000 00:00 0
7f70bb249000-7f70bb26b000 r--p 00000000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f70bb26b000-7f70bb3e3000 r-xp 00022000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f70bb3e3000-7f70bb431000 r--p 0019a000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f70bb431000-7f70bb435000 r--p 001e7000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f70bb435000-7f70bb437000 rw-p 001eb000 08:05 2623619
/usr/lib/x86_64-linux-gnu/libc-2.31.so
7f70bb437000-7f70bb43b000 rw-p 00000000 00:00 0
7f70bb44c000-7f70bb44d000 r--p 00000000 08:05 2519503
/sample/build/libarithmetic.so
7f70bb44d000-7f70bb44e000 r-xp 00001000 08:05 2519503
/sample/build/libarithmetic.so
7f70bb44e000-7f70bb44f000 r--p 00002000 08:05 2519503
/sample/build/libarithmetic.so
7f70bb44f000-7f70bb450000 r--p 00002000 08:05 2519503
/sample/build/libarithmetic.so
7f70bb450000-7f70bb451000 rw-p 00003000 08:05 2519503
/sample/build/libarithmetic.so
7f70bb451000-7f70bb453000 rw-p 00000000 00:00 0
7f70bb453000-7f70bb454000 r--p 00000000 08:05 2519504 /sample/build/main
7f70bb454000-7f70bb455000 r-xp 00001000 08:05 2519504 /sample/build/main
7f70bb455000-7f70bb456000 r--p 00002000 08:05 2519504 /sample/build/main
7f70bb456000-7f70bb457000 r--p 00002000 08:05 2519504 /sample/build/main
7f70bb457000-7f70bb458000 rw-p 00003000 08:05 2519504 /sample/build/main
7f70bb458000-7f70bb459000 r--p 00000000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f70bb459000-7f70bb47c000 r-xp 00001000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f70bb47c000-7f70bb484000 r--p 00024000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f70bb485000-7f70bb486000 r--p 0002c000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f70bb486000-7f70bb487000 rw-p 0002d000 08:05 2623613
/usr/lib/x86_64-linux-gnu/ld-2.31.so
7f70bb487000-7f70bb488000 rw-p 00000000 00:00 0
7ffe42e67000-7ffe42e88000 rw-p 00000000 00:00 0 [stack]
7ffe42ebe000-7ffe42ec1000 r--p 00000000 00:00 0 [vvar]
7ffe42ec1000-7ffe42ec2000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
```
## Solution 1 - Modify the dynamic linker
When the dynamic linker loads the shared libraries, extend the `vm_area_struct`
to be at a `p_align` boundary. This is done by passing to `mprotect()` the
right address space ranges.
## Solution 2 - Modify the static linker
When the static linker creates the program segments, make sure that the
`p_filesz` and `p_memsz` are extended to a `p_align` boundary. This can be done
by inserting a new ELF section at every PT_LOAD ELF segment that will serve as
padding.
--
You are receiving this mail because:
You are on the CC list for the bug.
next reply other threads:[~2023-11-18 18:48 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-18 18:48 jyescas at google dot com [this message]
2023-11-18 18:50 ` [Bug dynamic-link/31076] " jyescas at google dot com
2023-11-21 13:42 ` carlos at redhat dot com
2023-11-22 0:39 ` i at maskray dot me
2023-11-22 4:33 ` kaleshsingh at google dot com
2023-11-22 18:19 ` jyescas at google dot com
2023-11-23 11:42 ` sam at gentoo dot org
2023-11-24 17:40 ` adhemerval.zanella at linaro dot org
2023-11-27 15:11 ` fweimer at redhat dot com
2023-11-27 15:22 ` fweimer at redhat dot com
2023-11-27 16:27 ` adhemerval.zanella at linaro dot org
2023-11-27 17:19 ` fweimer at redhat dot com
2023-11-27 17:39 ` adhemerval.zanella at linaro dot org
2023-11-27 17:45 ` fweimer at redhat dot com
2023-11-27 17:58 ` adhemerval.zanella at linaro dot org
2023-11-27 19:47 ` jyescas at google dot com
2023-11-27 19:55 ` jyescas at google dot com
2023-11-28 8:48 ` rprichard at google dot com
2023-11-28 18:59 ` kaleshsingh at google dot com
2023-11-28 23:58 ` jyescas at google dot com
2023-12-02 17:08 ` i at maskray dot me
2023-12-06 11:57 ` fweimer at redhat dot com
2023-12-07 5:11 ` i at maskray dot me
2023-12-07 9:30 ` fweimer at redhat dot com
2023-12-08 3:22 ` i at maskray dot me
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bug-31076-131@http.sourceware.org/bugzilla/ \
--to=sourceware-bugzilla@sourceware.org \
--cc=glibc-bugs@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).