From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by sourceware.org (Postfix) with ESMTPS id 3991D3858D37; Mon, 10 Oct 2022 06:18:02 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3991D3858D37 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-ed1-x532.google.com with SMTP id y100so14499634ede.6; Sun, 09 Oct 2022 23:18:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=LPvGKWq9TD+BvveNFkPZVkQUKOcgbBD0OUtpTAjte4s=; b=XbfopST2JUMIf5O+ZF5zboz4uNeZpB53UhQ9kBCsuw+cqAkVFhIA5UKEEQpUTBLr5W EH2tNu78YQAhBu9heTnD+4dslIP9yrHXxIliixzldxltzWwSahhaSJxQ7VEfh/k4UQT2 D5viKoipceN1eKAk9CfEFQ0jHanXCF18kxHeHeBr3SnUeugjsXQQVlMOuOaKc5fREqfH qS7MYTRJuZKox9HBf1yivlq1G1HRQTH51PVp5ZyymQF/KQ+wjp3iXmejuMNPicnvSBBs CFSBxzhm23qxgx1p39BfSWoakurEeiozZVMxPvA8Tk8XD/kbQs8laon/HzUmgARbV2qk VnMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LPvGKWq9TD+BvveNFkPZVkQUKOcgbBD0OUtpTAjte4s=; b=Cg8H5Iz6UyAVa2GlgTzLBZyNiANN1xcPIMVMyAJFyTJgPZM1shFY9gl9ABemVbw5H3 vNG+MTqf1Jjxl31VwZRynyDxkanGQtmpv32G4wXWzQrNh4VP+5d867mOPjqxyuLI0kHQ yu9lMSFAgmaNCickXWnDnM+x0qk3OwiNJfFM/BTx/VgTeg8P3n3HfMtDacvbjX8Eo0D7 OWc41cYcHS0jo03oyuXDttJmTxGYpFVK+9CrgHJuVjbDTDsBuPWtz8X/b4BPhIHMYhGI /ak72MnFesibKvh48B/mmzW0hY8QX5sdMJYJlQHhfRklt2f9LXK2N0ElqNh/PTx2MWC/ GMRQ== X-Gm-Message-State: ACrzQf3debFL/eQsv4xkbtBB99atOexja2lStlKPz0k3W5xPDZQqW4q8 SpOvseb/xpDk9W6bxzJT5WP9IQSk5Qz3Nnae/RY= X-Google-Smtp-Source: AMsMyM5BxTNPIHLzVfGwZAN5elY9PvpmrzdtqYIoHFd0e87pvLyZicMBQGKZa2fgwy6LrBC/A8C7VPLy/6uDf2xwyLc= X-Received: by 2002:a05:6402:550c:b0:443:7d15:d57f with SMTP id fi12-20020a056402550c00b004437d15d57fmr16407646edb.147.1665382680320; Sun, 09 Oct 2022 23:18:00 -0700 (PDT) MIME-Version: 1.0 References: <20221007155452.1299670-1-jwakely@redhat.com> In-Reply-To: <20221007155452.1299670-1-jwakely@redhat.com> From: Richard Biener Date: Mon, 10 Oct 2022 08:17:48 +0200 Message-ID: Subject: Re: [PATCH] libstdc++: Allow emergency EH alloc pool size to be tuned [PR68606] To: Jonathan Wakely Cc: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,GIT_PATCH_0,KAM_SHORT,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Fri, Oct 7, 2022 at 5:55 PM Jonathan Wakely via Gcc-patches wrote: > > This needs a little more documentation (see the TODO in the manual), > rather than just the comments in the source. This isn't final, but I > think it's the direction I want to take. > > -- >8 -- > > Implement a long-standing request to support tuning the size of the > emergency buffer for allocating exceptions after malloc fails, or to > disable that buffer entirely. > > It's now possible to disable the dynamic allocation of the buffer and > use a fixed-size static buffer, via --enable-libstdcxx-static-eh-pool. > This is a built-time choice that is baked into libstdc++ and so affects > all code linked against that build of libstdc++. > > The size of the pool can be set by --with-libstdcxx-eh-pool-obj-count=N > which is measured in units of sizeof(void*) not bytes. A given exception > type such as std::system_error depends on the target, so giving a size > in bytes wouldn't be portable across 16/32/64-bit targets. > > When libstdc++ is configured to use a dynamic buffer, the size of that > buffer can now be tuned at runtime by setting the GLIBCXX_TUNABLES > environment variable (c.f. PR libstdc++/88264). The number of exceptions > to reserve space for is controlled by the "glibcxx.eh_pool.obj_count" > and "glibcxx.eh_pool.obj_size" tunables. The pool will be sized to be > able to allocate obj_count exceptions of size obj_size*sizeof(void*) and > obj_count "dependent" exceptions rethrown by std::rethrow_exception. > > With the ability to tune the buffer size, we can reduce the default pool > size. Most users never need to throw 1kB exceptions in parallel from > hundreds of threads after malloc is OOM. But does it hurt? Back in time when I reworked the allocator to be less wasteful the whole point was to allow more exceptions to be in-flight during OOM shutdown of a process with many threads. So if we reduce the default buffer size that should be documented in changes.html, maybe with a hint how to restore the old buffer size (configury flags required or runtime ENV setting)? Otherwise looks OK to me. Thanks, Richard. > The users who do need that can > use the tunables to select larger sizes. The old default can be chosen > with: > > For 64-bit: > GLIBCXX_TUNABLES="glibcxx.eh_pool.obj_count=64:glibcxx.eh_pool.obj_size=112" > For 32-bit: > GLIBCXX_TUNABLES="glibcxx.eh_pool.obj_count=32:glibcxx.eh_pool.obj_size=112" > > libstdc++-v3/ChangeLog: > > PR libstdc++/68606 > * Makefile.in: Regenerate. > * acinclude.m4 (GLIBCXX_EMERGENCY_EH_ALLOC): New macro. > * configure: Regenerate. > * configure.ac: Use GLIBCXX_EMERGENCY_EH_ALLOC. > * crossconfig.m4: Check for secure_getenv. > * doc/Makefile.in: Regenerate. > * doc/xml/manual/configure.xml: Document new configure options. > * doc/xml/manual/using_exceptions.xml: Document emergency > buffer. > * doc/html/*: Regenerate. > * include/Makefile.in: Regenerate. > * libsupc++/Makefile.am: Use EH_POOL_FLAGS. > * libsupc++/Makefile.in: Regenerate. > * libsupc++/eh_alloc.cc (EMERGENCY_OBJ_SIZE): Define in units > of sizeof(void*) not including the ABI's exception header. > (EMERGENCY_OBJ_COUNT): Define as target-independent calculation > based on word size. > (MAX_OBJ_COUNT): Define macro for upper limit on pool size. > (pool) [_GLIBCXX_EH_POOL_STATIC]: Use fixed-size buffer. > (pool::buffer_size_in_bytes): New static member function. > (pool::pool): Parse GLIBCXX_TUNABLES environment variable to set > pool size at runtime. > (pool::in_pool): Use std::less for total order. > (__freeres) [_GLIBCXX_EH_POOL_STATIC]: Do nothing. > (__cxa_allocate_exception, __cxa_free_exception) > (__cxa_allocate_dependent_exception) > (__cxa_free_dependent_exception): Add [[unlikely]] attributes. > * po/Makefile.in: Regenerate. > * python/Makefile.in: Regenerate. > * src/Makefile.in: Regenerate. > * src/c++11/Makefile.in: Regenerate. > * src/c++17/Makefile.in: Regenerate. > * src/c++20/Makefile.in: Regenerate. > * src/c++98/Makefile.in: Regenerate. > * src/filesystem/Makefile.in: Regenerate. > * src/libbacktrace/Makefile.in: Regenerate. > * testsuite/Makefile.in: Regenerate. > --- > libstdc++-v3/Makefile.in | 1 + > libstdc++-v3/acinclude.m4 | 45 ++++ > libstdc++-v3/configure | 67 +++++- > libstdc++-v3/configure.ac | 3 + > libstdc++-v3/crossconfig.m4 | 1 + > libstdc++-v3/doc/Makefile.in | 1 + > libstdc++-v3/doc/html/index.html | 2 +- > libstdc++-v3/doc/html/manual/configure.html | 10 +- > libstdc++-v3/doc/html/manual/index.html | 2 +- > libstdc++-v3/doc/html/manual/intro.html | 2 +- > libstdc++-v3/doc/html/manual/using.html | 2 +- > .../doc/html/manual/using_exceptions.html | 53 +++- > libstdc++-v3/doc/xml/manual/configure.xml | 23 ++ > .../doc/xml/manual/using_exceptions.xml | 48 ++++ > libstdc++-v3/include/Makefile.in | 1 + > libstdc++-v3/libsupc++/Makefile.am | 2 +- > libstdc++-v3/libsupc++/Makefile.in | 3 +- > libstdc++-v3/libsupc++/eh_alloc.cc | 226 +++++++++++++----- > libstdc++-v3/po/Makefile.in | 1 + > libstdc++-v3/python/Makefile.in | 1 + > libstdc++-v3/src/Makefile.in | 1 + > libstdc++-v3/src/c++11/Makefile.in | 1 + > libstdc++-v3/src/c++17/Makefile.in | 1 + > libstdc++-v3/src/c++20/Makefile.in | 1 + > libstdc++-v3/src/c++98/Makefile.in | 1 + > libstdc++-v3/src/filesystem/Makefile.in | 1 + > libstdc++-v3/src/libbacktrace/Makefile.in | 1 + > libstdc++-v3/testsuite/Makefile.in | 1 + > 28 files changed, 414 insertions(+), 88 deletions(-) > > diff --git a/libstdc++-v3/Makefile.in b/libstdc++-v3/Makefile.in > index 58a0acdcc1b..a7c2b60678b 100644 > --- a/libstdc++-v3/Makefile.in > +++ b/libstdc++-v3/Makefile.in > @@ -240,6 +240,7 @@ ECHO_C = @ECHO_C@ > ECHO_N = @ECHO_N@ > ECHO_T = @ECHO_T@ > EGREP = @EGREP@ > +EH_POOL_FLAGS = @EH_POOL_FLAGS@ > ERROR_CONSTANTS_SRCDIR = @ERROR_CONSTANTS_SRCDIR@ > EXEEXT = @EXEEXT@ > EXTRA_CFLAGS = @EXTRA_CFLAGS@ > diff --git a/libstdc++-v3/acinclude.m4 b/libstdc++-v3/acinclude.m4 > index 719eab15c77..6523ba9a816 100644 > --- a/libstdc++-v3/acinclude.m4 > +++ b/libstdc++-v3/acinclude.m4 > @@ -5092,6 +5092,51 @@ BACKTRACE_CPPFLAGS="$BACKTRACE_CPPFLAGS -DBACKTRACE_ELF_SIZE=$elfsize" > GLIBCXX_CONDITIONAL(ENABLE_BACKTRACE, [test "$enable_libstdcxx_backtrace" = yes]) > ]) > > +dnl > +dnl Allow the emergency EH pool to be configured. > +dnl > +dnl --enable-libstdcxx-static-eh-pool will cause a fixed-size static buffer > +dnl to be used for allocating exceptions after malloc fails. The default is > +dnl to allocate a buffer using malloc > +dnl > +dnl --with-libstdcxx-eh-pool-obj-count=N will set the default size for the > +dnl buffer. For a static buffer that size is fixed permanently. For a dynamic > +dnl buffer it's the default, but it can be overridden from the environment. > +dnl > +dnl To set the default to approximately the same values as GCC 12, > +dnl use --with-libstdcxx-eh-pool-obj-count=94 for 32-bit targets, > +dnl and --with-libstdcxx-eh-pool-obj-count=252 for 64-bit targets. > +dnl > +dnl Defines: > +dnl _GLIBCXX_EH_POOL_STATIC if a fixed-size static buffer should be used > +dnl instead of allocating a buffer on startup. > +dnl _GLIBCXX_EH_POOL_NOBJS to override the default EMERGENCY_OBJ_COUNT value. > +dnl > +AC_DEFUN([GLIBCXX_EMERGENCY_EH_ALLOC], [ > + eh_pool_static= > + eh_pool_nobjs= > + AC_ARG_ENABLE([libstdcxx-static-eh-pool], > + AC_HELP_STRING([--enable-libstdcxx-static-eh-pool], > + [use a fixed-size static buffer for allocating exceptions if malloc fails]), > + [if test "${enableval}" = yes; then > + eh_pool_static="-D_GLIBCXX_EH_POOL_STATIC" > + AC_MSG_NOTICE([EH pool using static buffer]) > + fi],) > + > + AC_ARG_WITH([libstdcxx-eh-pool-obj-count], > + AC_HELP_STRING([--with-libstdcxx-eh-pool-obj-count], > + [the number of exceptions that can be allocated from the pool if malloc fails]), > + [if test "${withval}" -ge 0 2>/dev/null; then > + eh_pool_obj_count="-D_GLIBCXX_EH_POOL_NOBJS=${withval}" > + AC_MSG_NOTICE([EH pool object count: ${withval}]) > + else > + AC_MSG_ERROR([EH pool obj count must be a non-negative integer: $withval]) > + fi],) > + > + EH_POOL_FLAGS="$eh_pool_static $eh_pool_obj_count" > + AC_SUBST(EH_POOL_FLAGS) > +]) > + > # Macros from the top-level gcc directory. > m4_include([../config/gc++filt.m4]) > m4_include([../config/tls.m4]) > diff --git a/libstdc++-v3/configure.ac b/libstdc++-v3/configure.ac > index c05fcdda7e9..42c453099f2 100644 > --- a/libstdc++-v3/configure.ac > +++ b/libstdc++-v3/configure.ac > @@ -538,6 +538,9 @@ GLIBCXX_CHECK_SIZE_T_MANGLING > # Check which release added std::exception_ptr for the target > GLIBCXX_CHECK_EXCEPTION_PTR_SYMVER > > +# For libsupc++/eh_alloc.cc defaults. > +GLIBCXX_EMERGENCY_EH_ALLOC > + > # Define documentation rules conditionally. > > # See if makeinfo has been installed and is modern enough > diff --git a/libstdc++-v3/crossconfig.m4 b/libstdc++-v3/crossconfig.m4 > index 130f47fb1d4..b3269cb88e0 100644 > --- a/libstdc++-v3/crossconfig.m4 > +++ b/libstdc++-v3/crossconfig.m4 > @@ -187,6 +187,7 @@ case "${host}" in > AC_CHECK_FUNCS(timespec_get) > AC_CHECK_FUNCS(sockatmark) > AC_CHECK_FUNCS(uselocale) > + AC_CHECK_FUNCS(secure_getenv) > AM_ICONV > ;; > *-mingw32*) > diff --git a/libstdc++-v3/doc/xml/manual/configure.xml b/libstdc++-v3/doc/xml/manual/configure.xml > index 8c26acc95a7..c6c5981968d 100644 > --- a/libstdc++-v3/doc/xml/manual/configure.xml > +++ b/libstdc++-v3/doc/xml/manual/configure.xml > @@ -366,6 +366,11 @@ > > > > + --disable-libstdcxx-hosted > + > + This is an alias for --disable-hosted-libstdcxx. > + > + > --disable-libstdcxx-verbose > > > @@ -446,6 +451,24 @@ > > > > + --enable-libstdcxx-static-eh-pool > + > + Use a fixed-size static buffer for the emergency exception handling > + pool (see ). The default > + is to allocate the pool on program startup using malloc. > + With this option, a static buffer will be provided by libstdc++ instead. > + This does not change the library ABI. > + > + > + > + --with-libstdcxx-eh-pool-obj-count=NUM > + > + Set the size of the emergency exception handling pool. NUM is the > + number of simultaneous allocated exceptions to support. > + This does not change the library ABI. > + > + > + > > > > diff --git a/libstdc++-v3/doc/xml/manual/using_exceptions.xml b/libstdc++-v3/doc/xml/manual/using_exceptions.xml > index 32bff85549f..d17a915f237 100644 > --- a/libstdc++-v3/doc/xml/manual/using_exceptions.xml > +++ b/libstdc++-v3/doc/xml/manual/using_exceptions.xml > @@ -188,6 +188,54 @@ exception neutrality and exception safety. > > > > +
Memory allocation > + > + > + When the program throws an exception the runtime will obtain storage for > + a __cxa_exception header and the thrown object itself. > + Libstdc++ will try to use malloc to obtain storage, > + but provides an emergency buffer to be used if malloc fails, > + as described by the + xlink:href="https://itanium-cxx-abi.github.io/cxx-abi/abi-eh.html#imp-emergency">Itanium > + exception handling ABI. > + > + > + > + Contrary to the ABI, the libstdc++ emergency buffer is not always 64kB, > + and does not always allocate 1kB chunks. The buffer is used as a pool for > + variable-sized allocations, so that it doesn't waste space for smaller > + exception objects, such as std::bad_alloc. > + The total size of the buffer is scaled appropriately for the target, > + specifically it depends on sizeof(void*), so that a 64-bit > + system uses a larger pool than a 32-bit system. This is done because for > + 32-bit systems the exception objects (and the exception header) require > + less space, and core counts and thread counts are typically lower as well. > + > + > + > + By default, libstdc++ will use malloc to allocate the buffer > + on program startup. > + libstdc++ with the > + --enable-libstdcxx-static-eh-pool option will make it > + use a static buffer instead of using malloc. > + The default buffer size is chosen automatically, but can be overridden > + by configuring with --with-libstdcxx-eh-pool-obj-count=NUM, > + where NUM is the number of simultaneous allocations that should be > + supported. The size of the pool will be sufficient for NUM exceptions > + of 6 * sizeof(void*) bytes, plus another NUM exceptions > + captured in std::exception_ptr and rethrown using > + std::rethrow_exception. That default size applies whether > + the buffer is reserved as static storage or is allocated dynamically. > + For a dynamic buffer, the default size can also be changed at runtime, > + per-process, via the GLIBCXX_TUNABLES environment variable. > + > + > + > + TODO document GLIBCXX_TUNABLES > + > + > +
> + >
Doing without > > > diff --git a/libstdc++-v3/libsupc++/Makefile.am b/libstdc++-v3/libsupc++/Makefile.am > index 65b5c1a87fd..cde9fbdb1eb 100644 > --- a/libstdc++-v3/libsupc++/Makefile.am > +++ b/libstdc++-v3/libsupc++/Makefile.am > @@ -139,7 +139,7 @@ atomicity.cc: ${atomicity_file} > # as the occasion call for it. > AM_CXXFLAGS = \ > $(glibcxx_lt_pic_flag) $(glibcxx_compiler_shared_flag) \ > - $(XTEMPLATE_FLAGS) $(FREESTANDING_FLAGS) \ > + $(XTEMPLATE_FLAGS) $(FREESTANDING_FLAGS) $(EH_POOL_FLAGS) \ > $(WARN_CXXFLAGS) $(OPTIMIZE_CXXFLAGS) $(CONFIG_CXXFLAGS) > > AM_MAKEFLAGS = \ > diff --git a/libstdc++-v3/libsupc++/eh_alloc.cc b/libstdc++-v3/libsupc++/eh_alloc.cc > index 68f319869f9..32378d586ad 100644 > --- a/libstdc++-v3/libsupc++/eh_alloc.cc > +++ b/libstdc++-v3/libsupc++/eh_alloc.cc > @@ -25,16 +25,45 @@ > // This is derived from the C++ ABI for IA-64. Where we diverge > // for cross-architecture compatibility are noted with "@@@". > > -#include > -#include > -#if _GLIBCXX_HOSTED > -#include > -#endif > -#include > -#include > +#include // std::exception > +#include // std::terminate > +#include // std::malloc, std::free, std::strtoul > +#include // INT_MAX > +#include // std::less > #include "unwind-cxx.h" > -#include > -#include > +#if _GLIBCXX_HOSTED > +# include // std::string_view > +# include // std::strchr, std::memset > +# include // __gnu_cxx::__mutex, __gnu_cxx::__scoped_lock > +#endif > + > +// We use an emergency buffer used for exceptions when malloc fails. > +// If _GLIBCXX_EH_POOL_STATIC is defined (e.g. by configure) then we use > +// a fixed-size static buffer. Otherwise, allocate on startup using malloc. > +// > +// The size of the buffer is N * (S * P + R + D), where: > +// N == The number of objects to reserve space for. > +// Defaults to EMERGENCY_OBJ_COUNT, defined below. > +// S == Estimated size of exception objects to account for. > +// This size is in units of sizeof(void*) not bytes. > +// Defaults to EMERGENCY_OBJ_SIZE, defined below. > +// P == sizeof(void*). > +// R == sizeof(__cxa_refcounted_exception). > +// D == sizeof(__cxa_dependent_exception). > +// > +// This provides space for N thrown exceptions of S words each, and an > +// additional N dependent exceptions from std::rethrow_exception. > +// > +// The calculation allows values of N and S to be target-independent, > +// as the size will be scaled by the size of basic types on the target, > +// and space for the C++ exception header (__cxa_refcounted_exception) > +// is added automatically. > +// > +// For a dynamically allocated buffer, N and S can be set from the environment. > +// Setting N=0 will disable the emergency buffer. > +// The GLIBCXX_TUNABLES environment variable will be checked for the following: > +// - Tunable glibcxx.eh_pool.obj_count overrides EMERGENCY_OBJ_COUNT. > +// - Tunable glibcxx.eh_pool.obj_size overrides EMERGENCY_OBJ_SIZE. > > #if _GLIBCXX_HOSTED > using std::free; > @@ -50,46 +79,61 @@ extern "C" void *memset (void *, int, std::size_t); > > using namespace __cxxabiv1; > > -// ??? How to control these parameters. > +// Assume that 6 * sizeof(void*) is a reasonable exception object size. > +// Throwing very many large objects will exhaust the pool more quickly. > +// N.B. sizeof(std::bad_alloc) == sizeof(void*) > +// and sizeof(std::runtime_error) == 2 * sizeof(void*) > +// and sizeof(std::system_error) == 4 * sizeof(void*). > +#define EMERGENCY_OBJ_SIZE 6 > > -// Guess from the size of basic types how large a buffer is reasonable. > -// Note that the basic c++ exception header has 13 pointers and 2 ints, > -// so on a system with PSImode pointers we're talking about 56 bytes > -// just for overhead. > - > -#if INT_MAX == 32767 > -# define EMERGENCY_OBJ_SIZE 128 > -# define EMERGENCY_OBJ_COUNT 16 > -#elif !defined (_GLIBCXX_LLP64) && LONG_MAX == 2147483647 > -# define EMERGENCY_OBJ_SIZE 512 > -# define EMERGENCY_OBJ_COUNT 32 > +#ifdef __GTHREADS > +// Assume that the number of concurrent exception objects scales with the > +// processor word size, i.e., 16-bit systems are not likely to have hundreds > +// or threads all simultaneously throwing on OOM conditions. > +# define EMERGENCY_OBJ_COUNT (8 * __SIZEOF_POINTER__) > +# define MAX_OBJ_COUNT (16 << __SIZEOF_POINTER__) > #else > -# define EMERGENCY_OBJ_SIZE 1024 > -# define EMERGENCY_OBJ_COUNT 64 > +# define EMERGENCY_OBJ_COUNT 4 > +# define MAX_OBJ_COUNT 64 > #endif > > -#ifndef __GTHREADS > -# undef EMERGENCY_OBJ_COUNT > -# define EMERGENCY_OBJ_COUNT 4 > +// This can be set by configure. > +#ifdef _GLIBCXX_EH_POOL_NOBJS > +# if _GLIBCXX_EH_POOL_NOBJS > MAX_OBJ_COUNT > +# warning "_GLIBCXX_EH_POOL_NOBJS value is too large; ignoring it" > +# else > +# undef EMERGENCY_OBJ_COUNT > +# define EMERGENCY_OBJ_COUNT _GLIBCXX_EH_POOL_NOBJS > +# endif > #endif > > namespace __gnu_cxx > { > - void __freeres(); > + void __freeres() noexcept; > } > > namespace > { > + static constexpr std::size_t > + buffer_size_in_bytes(std::size_t obj_count, std::size_t obj_size) noexcept > + { > + // N * (S * P + R + D) > + constexpr std::size_t P = sizeof(void*); > + constexpr std::size_t R = sizeof(__cxa_refcounted_exception); > + constexpr std::size_t D = sizeof(__cxa_dependent_exception); > + return obj_count * (obj_size * P + R + D); > + } > + > // A fixed-size heap, variable size object allocator > class pool > { > public: > - pool(); > + pool() noexcept; > > - _GLIBCXX_NODISCARD void *allocate (std::size_t); > - void free (void *); > + _GLIBCXX_NODISCARD void *allocate (std::size_t) noexcept; > + void free (void *) noexcept; > > - bool in_pool (void *); > + bool in_pool (void *) const noexcept; > > private: > struct free_entry { > @@ -101,33 +145,87 @@ namespace > char data[] __attribute__((aligned)); > }; > > +#ifdef __GTHREADS > // A single mutex controlling emergency allocations. > __gnu_cxx::__mutex emergency_mutex; > + using __scoped_lock = __gnu_cxx::__scoped_lock; > +#else > + int emergency_mutex = 0; > + struct __scoped_lock { explicit __scoped_lock(int) { } }; > +#endif > > // The free-list > - free_entry *first_free_entry; > + free_entry *first_free_entry = nullptr; > // The arena itself - we need to keep track of these only > // to implement in_pool. > - char *arena; > - std::size_t arena_size; > +#ifdef _GLIBCXX_EH_POOL_STATIC > + static constexpr std::size_t arena_size > + = buffer_size_in_bytes(EMERGENCY_OBJ_COUNT, EMERGENCY_OBJ_SIZE); > + alignas(void*) char arena[std::max(arena_size, sizeof(free_entry))]; > +#else > + char *arena = nullptr; > + std::size_t arena_size = 0; > +#endif > > - friend void __gnu_cxx::__freeres(); > + friend void __gnu_cxx::__freeres() noexcept; > }; > > - pool::pool() > + pool::pool() noexcept > { > - // Allocate the arena - we could add a GLIBCXX_EH_ARENA_SIZE environment > - // to make this tunable. > - arena_size = (EMERGENCY_OBJ_SIZE * EMERGENCY_OBJ_COUNT > - + EMERGENCY_OBJ_COUNT * sizeof (__cxa_dependent_exception)); > +#ifndef _GLIBCXX_EH_POOL_STATIC > + int obj_size = EMERGENCY_OBJ_SIZE; > + int obj_count = EMERGENCY_OBJ_COUNT; > + > +#if _GLIBCXX_HOSTED > +#if _GLIBCXX_HAVE_SECURE_GETENV > + const char* str = ::secure_getenv("GLIBCXX_TUNABLES"); > +#else > + const char* str = std::getenv("GLIBCXX_TUNABLES"); > +#endif > + const std::string_view ns_name = "glibcxx.eh_pool"; > + std::pair tunables[]{ > + {"obj_size", 0}, {"obj_count", obj_count} > + }; > + while (str) > + { > + if (*str == ':') > + ++str; > + > + if (!ns_name.compare(0, ns_name.size(), str, ns_name.size()) > + && str[ns_name.size()] == '.') > + { > + str += ns_name.size() + 1; > + for (auto& t : tunables) > + if (!t.first.compare(0, t.first.size(), str, t.first.size()) > + && str[t.first.size()] == '=') > + { > + str += t.first.size() + 1; > + char* end; > + unsigned long val = strtoul(str, &end, 0); > + if ((*end == ':' || *end == '\0') && val <= INT_MAX) > + t.second = val; > + str = end; > + break; > + } > + } > + str = strchr(str, ':'); > + } > + obj_count = std::min(tunables[1].second, MAX_OBJ_COUNT); // Can be zero. > + if (tunables[0].second != 0) > + obj_size = tunables[0].second; > +#endif // HOSTED > + > + arena_size = buffer_size_in_bytes(obj_count, obj_size); > + if (arena_size == 0) > + return; > arena = (char *)malloc (arena_size); > if (!arena) > { > // If the allocation failed go without an emergency pool. > arena_size = 0; > - first_free_entry = NULL; > return; > } > +#endif // STATIC > > // Populate the free-list with a single entry covering the whole arena > first_free_entry = reinterpret_cast (arena); > @@ -136,7 +234,7 @@ namespace > first_free_entry->next = NULL; > } > > - void *pool::allocate (std::size_t size) > + void *pool::allocate (std::size_t size) noexcept > { > __gnu_cxx::__scoped_lock sentry(emergency_mutex); > // We need an additional size_t member plus the padding to > @@ -188,9 +286,9 @@ namespace > return &x->data; > } > > - void pool::free (void *data) > + void pool::free (void *data) noexcept > { > - __gnu_cxx::__scoped_lock sentry(emergency_mutex); > + __scoped_lock sentry(emergency_mutex); > allocated_entry *e = reinterpret_cast > (reinterpret_cast (data) - offsetof (allocated_entry, data)); > std::size_t sz = e->size; > @@ -252,11 +350,10 @@ namespace > } > } > > - bool pool::in_pool (void *ptr) > + inline bool pool::in_pool (void *ptr) const noexcept > { > - char *p = reinterpret_cast (ptr); > - return (p > arena > - && p < arena + arena_size); > + std::less less; > + return less(ptr, arena + arena_size) && less(arena, ptr); > } > > pool emergency_pool; > @@ -264,29 +361,32 @@ namespace > > namespace __gnu_cxx > { > + __attribute__((cold)) > void > - __freeres() > + __freeres() noexcept > { > +#ifndef _GLIBCXX_EH_POOL_STATIC > if (emergency_pool.arena) > { > ::free(emergency_pool.arena); > emergency_pool.arena = 0; > } > +#endif > } > } > > extern "C" void * > -__cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) _GLIBCXX_NOTHROW > +__cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) noexcept > { > void *ret; > > thrown_size += sizeof (__cxa_refcounted_exception); > ret = malloc (thrown_size); > > - if (!ret) > + if (!ret) [[__unlikely__]] > ret = emergency_pool.allocate (thrown_size); > > - if (!ret) > + if (!ret) [[__unlikely__]] > std::terminate (); > > memset (ret, 0, sizeof (__cxa_refcounted_exception)); > @@ -296,10 +396,10 @@ __cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) _GLIBCXX_NOTHROW > > > extern "C" void > -__cxxabiv1::__cxa_free_exception(void *vptr) _GLIBCXX_NOTHROW > +__cxxabiv1::__cxa_free_exception(void *vptr) noexcept > { > char *ptr = (char *) vptr - sizeof (__cxa_refcounted_exception); > - if (emergency_pool.in_pool (ptr)) > + if (emergency_pool.in_pool (ptr)) [[__unlikely__]] > emergency_pool.free (ptr); > else > free (ptr); > @@ -307,31 +407,29 @@ __cxxabiv1::__cxa_free_exception(void *vptr) _GLIBCXX_NOTHROW > > > extern "C" __cxa_dependent_exception* > -__cxxabiv1::__cxa_allocate_dependent_exception() _GLIBCXX_NOTHROW > +__cxxabiv1::__cxa_allocate_dependent_exception() noexcept > { > - __cxa_dependent_exception *ret; > + void *ret; > > - ret = static_cast<__cxa_dependent_exception*> > - (malloc (sizeof (__cxa_dependent_exception))); > + ret = malloc (sizeof (__cxa_dependent_exception)); > > - if (!ret) > - ret = static_cast <__cxa_dependent_exception*> > - (emergency_pool.allocate (sizeof (__cxa_dependent_exception))); > + if (!ret) [[__unlikely__]] > + ret = emergency_pool.allocate (sizeof (__cxa_dependent_exception)); > > if (!ret) > std::terminate (); > > memset (ret, 0, sizeof (__cxa_dependent_exception)); > > - return ret; > + return static_cast<__cxa_dependent_exception*>(ret); > } > > > extern "C" void > __cxxabiv1::__cxa_free_dependent_exception > - (__cxa_dependent_exception *vptr) _GLIBCXX_NOTHROW > + (__cxa_dependent_exception *vptr) noexcept > { > - if (emergency_pool.in_pool (vptr)) > + if (emergency_pool.in_pool (vptr)) [[__unlikely__]] > emergency_pool.free (vptr); > else > free (vptr); >