From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from forward502b.mail.yandex.net (forward502b.mail.yandex.net [178.154.239.146]) by sourceware.org (Postfix) with ESMTPS id 2D9F23858D39 for ; Mon, 3 Apr 2023 10:43:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2D9F23858D39 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=yandex.ru Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=yandex.ru Received: from mail-nwsmtp-smtp-production-main-17.iva.yp-c.yandex.net (mail-nwsmtp-smtp-production-main-17.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:79ae:0:640:3507:0]) by forward502b.mail.yandex.net (Yandex) with ESMTP id 7F9665ECF9; Mon, 3 Apr 2023 13:43:27 +0300 (MSK) Received: by mail-nwsmtp-smtp-production-main-17.iva.yp-c.yandex.net (smtp/Yandex) with ESMTPSA id OhKe95IWx0U0-7BB9QqEW; Mon, 03 Apr 2023 13:43:26 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1680518607; bh=QZVPJ+5fw6H2XZfa7S+WH5k92rDHulcNJxhoh2BY4jU=; h=In-Reply-To:From:Date:References:To:Subject:Message-ID; b=g+mOKIGjtVSMWfKFqK8Ss5Hew9N/Y6dE87BZIMCizrYs+ISe/MVbvNiGaRE/l81JP /rn1OY40uioR7s2hvDH2M/9mgAZgN3w7u3V7U61LMFACfWqfanYgOhLAwexUGo3pnP 1a4MucSpsuJyNdmyOyYanCd8k2UI529QumCOCPxk= Authentication-Results: mail-nwsmtp-smtp-production-main-17.iva.yp-c.yandex.net; dkim=pass header.i=@yandex.ru Message-ID: <298b04a6-3055-b89b-59c1-4cfbe955848e@yandex.ru> Date: Mon, 3 Apr 2023 15:43:24 +0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: [PATCH v9 0/13] implement dlmem() function Content-Language: en-US To: Szabolcs Nagy , Adhemerval Zanella Netto , libc-alpha@sourceware.org, janderson@rice.edu, Carlos O'Donell , Rich Felker References: <20230318165110.3672749-1-stsp2@yandex.ru> <2f3a10fa-4f79-7f9a-6407-d227dbf31935@yandex.ru> From: stsp In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,BODY_8BITS,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,TXREP,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi, 03.04.2023 15:04, Szabolcs Nagy пишет: > "preserving" the "destination mapping" (?) > a new requirement.. Indeed, which is the main point of that patch. New requirement - new API. Briefly, if the premap callback established some MAP_SHARED mapping, then it will remain that way after dlmem() finished working. But that's an optional behavior, activated by a flag. Normal tasks (like dlopen_with_offset()) need nothing like that. See the full details by an URL from my prev mail about the use-case for such a thing. > the segments of an elf file can be mmapped in many ways, > there is no such thing as initial mmap (there need not > be a mapping that covers all segments and there is no > ordering requirement for the mappings or which mapping > is file backed etc) your callback exposes a particular > behaviour. I think this is a misunderstanding. Even before my patch there is this code in dl-load.c:     /* Length of the sections to be loaded.  */     maplength = loadcmds[nloadcmds - 1].allocend - loadcmds[0].mapstart; So it calculates the maximum length to map, which is then mapped in 1 go in _dl_map_segment(). There is also this comment (again, w/o any patches of mine):          So we map the first segment without MAP_FIXED, but with its          extent increased to cover all the segments.  Then we remove          access from excess portion, and there is known sufficient space          there to remap from the later segments. So I am working on an existing frame-work only. > and libc apis are exposed to not use locks that have > lock ordering requirement wrt the dynamic linker > internal lock. (e.g. we could add such a lock in mmap > or shm_open since they are not required to be as-safe. > so your callback can deadlock in a future glibc). Well you probably can't add such a lock into mmap(), as the loader calls mmap() anyway. Premap callback just does the same thing. If you can add some lock to shm_open(), then shm_open() can simply be done before calling dlmem(). The only thing to care here, is probably ftruncate(). Do you really think some lock can be added to ftruncate()? > and it's not just locks but any unusual libc internal > state that becomes observable via the callback. > (the callback runs when the dynamic linker may be in > inconsistent internal state. why do you assume that > shm_open or other libc apis work in that case? this > is a new requirement that has to be documented.) Mainly mmap(), and the loader does mmap()s by himself, so I quite assume they work from the callback. > it does not matter if it's 99% or 99.999%, what matters > is that if we expose an api then that has to work > *forever* under all potential usage and we have to > keep *supporting* it with all the api&abi constraints. Would it be acceptable to settle on an agreement to not add a lock to ftruncate() that would prevent its use from a premap callback? If its not possible, of course there is still an option to document the premap callback in a way so it will have to do the syscalls directly. Which doesn't look like a very good choice, but possible. > giving examples is not a *proof* that the api works. The example was needed to demonstrate that premap is not needed to map segments. >> Unfortunately my setup is much more >> complicated than that: I need to force >> the mapping under 4Gb AND I need to >> force it into the shared buffer which I >> then mmap() to another address... > you didnt explain this in the rationale of the patches. I mentioned such a use-case actually in the rationale, here's the quote: More so, if DLMEM_DONTREPLACE flag is used, then the mapping established by the premap callback, will not be replaced with the file-backed mapping. In that case dlmem() have to use memcpy(), which is likely even faster than mmaps() but doesn't end up with the proper /proc/self/map_files or /proc/self/maps entries. So for example if the premap callback uses MAP_SHARED, then with the use of the DLMEM_DONTREPLACE flag you can get your solib relocated into a shared memory buffer. Should I make is more verbose? > but it seems what you want is not suitable for libc. > you have to write your own loader to do this. I very much wish to do that! But so far I failed with a few naive attempts, like building the patched glibc statically, or loading custom libc.so into a separate name-space... I don't know what interface should be exposed by glibc to let my own loader to create a new link-map and bind symbols to it. Maybe even none, maybe I can do that via r_debug for example. So I very much wish to work in that direction, if you think it is a more appropriate solution that can be accepted into glibc (if any changes are at all needed). However, dlmem() can be used for Android's dlopen_with_offset(), which was also rejected from glibc. With dlmem() its just a dozen of lines that never need to be in glibc itself. So for example the problem with dlopen_with_offset() could be easily settled with my dlmem().