From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id 5F3613948A5C for ; Fri, 7 Oct 2022 15:54:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 5F3613948A5C Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665158095; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=g4cQZ7soUfufeA5hTd5p5J2CKcT40buT9ENtK+EoQOo=; b=fwlaZdpVjNo/COItjLwhXoeguCnoYOAjcVMF2FVY/LeW1Ifw9blI7ilbcDkQgODFQCwXxL NyYpeFLncInFFxu7SsqDBPNFg3i9Gg6CkD0ZjM8in50GaAxpATOuCHpYuf6I1cic6FHNv0 jK06LkFoMifU5/yT6IUbh2zbdTl1U1g= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-633-s7XNql6IMheCMMAlrVSzqQ-1; Fri, 07 Oct 2022 11:54:54 -0400 X-MC-Unique: s7XNql6IMheCMMAlrVSzqQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BD6351C05ADA; Fri, 7 Oct 2022 15:54:53 +0000 (UTC) Received: from localhost (unknown [10.33.36.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id 671CC2166B4D; Fri, 7 Oct 2022 15:54:53 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH] libstdc++: Allow emergency EH alloc pool size to be tuned [PR68606] Date: Fri, 7 Oct 2022 16:54:52 +0100 Message-Id: <20221007155452.1299670-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-12.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,KAM_SHORT,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: This needs a little more documentation (see the TODO in the manual), rather than just the comments in the source. This isn't final, but I think it's the direction I want to take. -- >8 -- Implement a long-standing request to support tuning the size of the emergency buffer for allocating exceptions after malloc fails, or to disable that buffer entirely. It's now possible to disable the dynamic allocation of the buffer and use a fixed-size static buffer, via --enable-libstdcxx-static-eh-pool. This is a built-time choice that is baked into libstdc++ and so affects all code linked against that build of libstdc++. The size of the pool can be set by --with-libstdcxx-eh-pool-obj-count=N which is measured in units of sizeof(void*) not bytes. A given exception type such as std::system_error depends on the target, so giving a size in bytes wouldn't be portable across 16/32/64-bit targets. When libstdc++ is configured to use a dynamic buffer, the size of that buffer can now be tuned at runtime by setting the GLIBCXX_TUNABLES environment variable (c.f. PR libstdc++/88264). The number of exceptions to reserve space for is controlled by the "glibcxx.eh_pool.obj_count" and "glibcxx.eh_pool.obj_size" tunables. The pool will be sized to be able to allocate obj_count exceptions of size obj_size*sizeof(void*) and obj_count "dependent" exceptions rethrown by std::rethrow_exception. With the ability to tune the buffer size, we can reduce the default pool size. Most users never need to throw 1kB exceptions in parallel from hundreds of threads after malloc is OOM. The users who do need that can use the tunables to select larger sizes. The old default can be chosen with: For 64-bit: GLIBCXX_TUNABLES="glibcxx.eh_pool.obj_count=64:glibcxx.eh_pool.obj_size=112" For 32-bit: GLIBCXX_TUNABLES="glibcxx.eh_pool.obj_count=32:glibcxx.eh_pool.obj_size=112" libstdc++-v3/ChangeLog: PR libstdc++/68606 * Makefile.in: Regenerate. * acinclude.m4 (GLIBCXX_EMERGENCY_EH_ALLOC): New macro. * configure: Regenerate. * configure.ac: Use GLIBCXX_EMERGENCY_EH_ALLOC. * crossconfig.m4: Check for secure_getenv. * doc/Makefile.in: Regenerate. * doc/xml/manual/configure.xml: Document new configure options. * doc/xml/manual/using_exceptions.xml: Document emergency buffer. * doc/html/*: Regenerate. * include/Makefile.in: Regenerate. * libsupc++/Makefile.am: Use EH_POOL_FLAGS. * libsupc++/Makefile.in: Regenerate. * libsupc++/eh_alloc.cc (EMERGENCY_OBJ_SIZE): Define in units of sizeof(void*) not including the ABI's exception header. (EMERGENCY_OBJ_COUNT): Define as target-independent calculation based on word size. (MAX_OBJ_COUNT): Define macro for upper limit on pool size. (pool) [_GLIBCXX_EH_POOL_STATIC]: Use fixed-size buffer. (pool::buffer_size_in_bytes): New static member function. (pool::pool): Parse GLIBCXX_TUNABLES environment variable to set pool size at runtime. (pool::in_pool): Use std::less for total order. (__freeres) [_GLIBCXX_EH_POOL_STATIC]: Do nothing. (__cxa_allocate_exception, __cxa_free_exception) (__cxa_allocate_dependent_exception) (__cxa_free_dependent_exception): Add [[unlikely]] attributes. * po/Makefile.in: Regenerate. * python/Makefile.in: Regenerate. * src/Makefile.in: Regenerate. * src/c++11/Makefile.in: Regenerate. * src/c++17/Makefile.in: Regenerate. * src/c++20/Makefile.in: Regenerate. * src/c++98/Makefile.in: Regenerate. * src/filesystem/Makefile.in: Regenerate. * src/libbacktrace/Makefile.in: Regenerate. * testsuite/Makefile.in: Regenerate. --- libstdc++-v3/Makefile.in | 1 + libstdc++-v3/acinclude.m4 | 45 ++++ libstdc++-v3/configure | 67 +++++- libstdc++-v3/configure.ac | 3 + libstdc++-v3/crossconfig.m4 | 1 + libstdc++-v3/doc/Makefile.in | 1 + libstdc++-v3/doc/html/index.html | 2 +- libstdc++-v3/doc/html/manual/configure.html | 10 +- libstdc++-v3/doc/html/manual/index.html | 2 +- libstdc++-v3/doc/html/manual/intro.html | 2 +- libstdc++-v3/doc/html/manual/using.html | 2 +- .../doc/html/manual/using_exceptions.html | 53 +++- libstdc++-v3/doc/xml/manual/configure.xml | 23 ++ .../doc/xml/manual/using_exceptions.xml | 48 ++++ libstdc++-v3/include/Makefile.in | 1 + libstdc++-v3/libsupc++/Makefile.am | 2 +- libstdc++-v3/libsupc++/Makefile.in | 3 +- libstdc++-v3/libsupc++/eh_alloc.cc | 226 +++++++++++++----- libstdc++-v3/po/Makefile.in | 1 + libstdc++-v3/python/Makefile.in | 1 + libstdc++-v3/src/Makefile.in | 1 + libstdc++-v3/src/c++11/Makefile.in | 1 + libstdc++-v3/src/c++17/Makefile.in | 1 + libstdc++-v3/src/c++20/Makefile.in | 1 + libstdc++-v3/src/c++98/Makefile.in | 1 + libstdc++-v3/src/filesystem/Makefile.in | 1 + libstdc++-v3/src/libbacktrace/Makefile.in | 1 + libstdc++-v3/testsuite/Makefile.in | 1 + 28 files changed, 414 insertions(+), 88 deletions(-) diff --git a/libstdc++-v3/Makefile.in b/libstdc++-v3/Makefile.in index 58a0acdcc1b..a7c2b60678b 100644 --- a/libstdc++-v3/Makefile.in +++ b/libstdc++-v3/Makefile.in @@ -240,6 +240,7 @@ ECHO_C = @ECHO_C@ ECHO_N = @ECHO_N@ ECHO_T = @ECHO_T@ EGREP = @EGREP@ +EH_POOL_FLAGS = @EH_POOL_FLAGS@ ERROR_CONSTANTS_SRCDIR = @ERROR_CONSTANTS_SRCDIR@ EXEEXT = @EXEEXT@ EXTRA_CFLAGS = @EXTRA_CFLAGS@ diff --git a/libstdc++-v3/acinclude.m4 b/libstdc++-v3/acinclude.m4 index 719eab15c77..6523ba9a816 100644 --- a/libstdc++-v3/acinclude.m4 +++ b/libstdc++-v3/acinclude.m4 @@ -5092,6 +5092,51 @@ BACKTRACE_CPPFLAGS="$BACKTRACE_CPPFLAGS -DBACKTRACE_ELF_SIZE=$elfsize" GLIBCXX_CONDITIONAL(ENABLE_BACKTRACE, [test "$enable_libstdcxx_backtrace" = yes]) ]) +dnl +dnl Allow the emergency EH pool to be configured. +dnl +dnl --enable-libstdcxx-static-eh-pool will cause a fixed-size static buffer +dnl to be used for allocating exceptions after malloc fails. The default is +dnl to allocate a buffer using malloc +dnl +dnl --with-libstdcxx-eh-pool-obj-count=N will set the default size for the +dnl buffer. For a static buffer that size is fixed permanently. For a dynamic +dnl buffer it's the default, but it can be overridden from the environment. +dnl +dnl To set the default to approximately the same values as GCC 12, +dnl use --with-libstdcxx-eh-pool-obj-count=94 for 32-bit targets, +dnl and --with-libstdcxx-eh-pool-obj-count=252 for 64-bit targets. +dnl +dnl Defines: +dnl _GLIBCXX_EH_POOL_STATIC if a fixed-size static buffer should be used +dnl instead of allocating a buffer on startup. +dnl _GLIBCXX_EH_POOL_NOBJS to override the default EMERGENCY_OBJ_COUNT value. +dnl +AC_DEFUN([GLIBCXX_EMERGENCY_EH_ALLOC], [ + eh_pool_static= + eh_pool_nobjs= + AC_ARG_ENABLE([libstdcxx-static-eh-pool], + AC_HELP_STRING([--enable-libstdcxx-static-eh-pool], + [use a fixed-size static buffer for allocating exceptions if malloc fails]), + [if test "${enableval}" = yes; then + eh_pool_static="-D_GLIBCXX_EH_POOL_STATIC" + AC_MSG_NOTICE([EH pool using static buffer]) + fi],) + + AC_ARG_WITH([libstdcxx-eh-pool-obj-count], + AC_HELP_STRING([--with-libstdcxx-eh-pool-obj-count], + [the number of exceptions that can be allocated from the pool if malloc fails]), + [if test "${withval}" -ge 0 2>/dev/null; then + eh_pool_obj_count="-D_GLIBCXX_EH_POOL_NOBJS=${withval}" + AC_MSG_NOTICE([EH pool object count: ${withval}]) + else + AC_MSG_ERROR([EH pool obj count must be a non-negative integer: $withval]) + fi],) + + EH_POOL_FLAGS="$eh_pool_static $eh_pool_obj_count" + AC_SUBST(EH_POOL_FLAGS) +]) + # Macros from the top-level gcc directory. m4_include([../config/gc++filt.m4]) m4_include([../config/tls.m4]) diff --git a/libstdc++-v3/configure.ac b/libstdc++-v3/configure.ac index c05fcdda7e9..42c453099f2 100644 --- a/libstdc++-v3/configure.ac +++ b/libstdc++-v3/configure.ac @@ -538,6 +538,9 @@ GLIBCXX_CHECK_SIZE_T_MANGLING # Check which release added std::exception_ptr for the target GLIBCXX_CHECK_EXCEPTION_PTR_SYMVER +# For libsupc++/eh_alloc.cc defaults. +GLIBCXX_EMERGENCY_EH_ALLOC + # Define documentation rules conditionally. # See if makeinfo has been installed and is modern enough diff --git a/libstdc++-v3/crossconfig.m4 b/libstdc++-v3/crossconfig.m4 index 130f47fb1d4..b3269cb88e0 100644 --- a/libstdc++-v3/crossconfig.m4 +++ b/libstdc++-v3/crossconfig.m4 @@ -187,6 +187,7 @@ case "${host}" in AC_CHECK_FUNCS(timespec_get) AC_CHECK_FUNCS(sockatmark) AC_CHECK_FUNCS(uselocale) + AC_CHECK_FUNCS(secure_getenv) AM_ICONV ;; *-mingw32*) diff --git a/libstdc++-v3/doc/xml/manual/configure.xml b/libstdc++-v3/doc/xml/manual/configure.xml index 8c26acc95a7..c6c5981968d 100644 --- a/libstdc++-v3/doc/xml/manual/configure.xml +++ b/libstdc++-v3/doc/xml/manual/configure.xml @@ -366,6 +366,11 @@ + --disable-libstdcxx-hosted + + This is an alias for --disable-hosted-libstdcxx. + + --disable-libstdcxx-verbose @@ -446,6 +451,24 @@ + --enable-libstdcxx-static-eh-pool + + Use a fixed-size static buffer for the emergency exception handling + pool (see ). The default + is to allocate the pool on program startup using malloc. + With this option, a static buffer will be provided by libstdc++ instead. + This does not change the library ABI. + + + + --with-libstdcxx-eh-pool-obj-count=NUM + + Set the size of the emergency exception handling pool. NUM is the + number of simultaneous allocated exceptions to support. + This does not change the library ABI. + + + diff --git a/libstdc++-v3/doc/xml/manual/using_exceptions.xml b/libstdc++-v3/doc/xml/manual/using_exceptions.xml index 32bff85549f..d17a915f237 100644 --- a/libstdc++-v3/doc/xml/manual/using_exceptions.xml +++ b/libstdc++-v3/doc/xml/manual/using_exceptions.xml @@ -188,6 +188,54 @@ exception neutrality and exception safety. +
Memory allocation + + + When the program throws an exception the runtime will obtain storage for + a __cxa_exception header and the thrown object itself. + Libstdc++ will try to use malloc to obtain storage, + but provides an emergency buffer to be used if malloc fails, + as described by the Itanium + exception handling ABI. + + + + Contrary to the ABI, the libstdc++ emergency buffer is not always 64kB, + and does not always allocate 1kB chunks. The buffer is used as a pool for + variable-sized allocations, so that it doesn't waste space for smaller + exception objects, such as std::bad_alloc. + The total size of the buffer is scaled appropriately for the target, + specifically it depends on sizeof(void*), so that a 64-bit + system uses a larger pool than a 32-bit system. This is done because for + 32-bit systems the exception objects (and the exception header) require + less space, and core counts and thread counts are typically lower as well. + + + + By default, libstdc++ will use malloc to allocate the buffer + on program startup. + libstdc++ with the + --enable-libstdcxx-static-eh-pool option will make it + use a static buffer instead of using malloc. + The default buffer size is chosen automatically, but can be overridden + by configuring with --with-libstdcxx-eh-pool-obj-count=NUM, + where NUM is the number of simultaneous allocations that should be + supported. The size of the pool will be sufficient for NUM exceptions + of 6 * sizeof(void*) bytes, plus another NUM exceptions + captured in std::exception_ptr and rethrown using + std::rethrow_exception. That default size applies whether + the buffer is reserved as static storage or is allocated dynamically. + For a dynamic buffer, the default size can also be changed at runtime, + per-process, via the GLIBCXX_TUNABLES environment variable. + + + + TODO document GLIBCXX_TUNABLES + + +
+
Doing without diff --git a/libstdc++-v3/libsupc++/Makefile.am b/libstdc++-v3/libsupc++/Makefile.am index 65b5c1a87fd..cde9fbdb1eb 100644 --- a/libstdc++-v3/libsupc++/Makefile.am +++ b/libstdc++-v3/libsupc++/Makefile.am @@ -139,7 +139,7 @@ atomicity.cc: ${atomicity_file} # as the occasion call for it. AM_CXXFLAGS = \ $(glibcxx_lt_pic_flag) $(glibcxx_compiler_shared_flag) \ - $(XTEMPLATE_FLAGS) $(FREESTANDING_FLAGS) \ + $(XTEMPLATE_FLAGS) $(FREESTANDING_FLAGS) $(EH_POOL_FLAGS) \ $(WARN_CXXFLAGS) $(OPTIMIZE_CXXFLAGS) $(CONFIG_CXXFLAGS) AM_MAKEFLAGS = \ diff --git a/libstdc++-v3/libsupc++/eh_alloc.cc b/libstdc++-v3/libsupc++/eh_alloc.cc index 68f319869f9..32378d586ad 100644 --- a/libstdc++-v3/libsupc++/eh_alloc.cc +++ b/libstdc++-v3/libsupc++/eh_alloc.cc @@ -25,16 +25,45 @@ // This is derived from the C++ ABI for IA-64. Where we diverge // for cross-architecture compatibility are noted with "@@@". -#include -#include -#if _GLIBCXX_HOSTED -#include -#endif -#include -#include +#include // std::exception +#include // std::terminate +#include // std::malloc, std::free, std::strtoul +#include // INT_MAX +#include // std::less #include "unwind-cxx.h" -#include -#include +#if _GLIBCXX_HOSTED +# include // std::string_view +# include // std::strchr, std::memset +# include // __gnu_cxx::__mutex, __gnu_cxx::__scoped_lock +#endif + +// We use an emergency buffer used for exceptions when malloc fails. +// If _GLIBCXX_EH_POOL_STATIC is defined (e.g. by configure) then we use +// a fixed-size static buffer. Otherwise, allocate on startup using malloc. +// +// The size of the buffer is N * (S * P + R + D), where: +// N == The number of objects to reserve space for. +// Defaults to EMERGENCY_OBJ_COUNT, defined below. +// S == Estimated size of exception objects to account for. +// This size is in units of sizeof(void*) not bytes. +// Defaults to EMERGENCY_OBJ_SIZE, defined below. +// P == sizeof(void*). +// R == sizeof(__cxa_refcounted_exception). +// D == sizeof(__cxa_dependent_exception). +// +// This provides space for N thrown exceptions of S words each, and an +// additional N dependent exceptions from std::rethrow_exception. +// +// The calculation allows values of N and S to be target-independent, +// as the size will be scaled by the size of basic types on the target, +// and space for the C++ exception header (__cxa_refcounted_exception) +// is added automatically. +// +// For a dynamically allocated buffer, N and S can be set from the environment. +// Setting N=0 will disable the emergency buffer. +// The GLIBCXX_TUNABLES environment variable will be checked for the following: +// - Tunable glibcxx.eh_pool.obj_count overrides EMERGENCY_OBJ_COUNT. +// - Tunable glibcxx.eh_pool.obj_size overrides EMERGENCY_OBJ_SIZE. #if _GLIBCXX_HOSTED using std::free; @@ -50,46 +79,61 @@ extern "C" void *memset (void *, int, std::size_t); using namespace __cxxabiv1; -// ??? How to control these parameters. +// Assume that 6 * sizeof(void*) is a reasonable exception object size. +// Throwing very many large objects will exhaust the pool more quickly. +// N.B. sizeof(std::bad_alloc) == sizeof(void*) +// and sizeof(std::runtime_error) == 2 * sizeof(void*) +// and sizeof(std::system_error) == 4 * sizeof(void*). +#define EMERGENCY_OBJ_SIZE 6 -// Guess from the size of basic types how large a buffer is reasonable. -// Note that the basic c++ exception header has 13 pointers and 2 ints, -// so on a system with PSImode pointers we're talking about 56 bytes -// just for overhead. - -#if INT_MAX == 32767 -# define EMERGENCY_OBJ_SIZE 128 -# define EMERGENCY_OBJ_COUNT 16 -#elif !defined (_GLIBCXX_LLP64) && LONG_MAX == 2147483647 -# define EMERGENCY_OBJ_SIZE 512 -# define EMERGENCY_OBJ_COUNT 32 +#ifdef __GTHREADS +// Assume that the number of concurrent exception objects scales with the +// processor word size, i.e., 16-bit systems are not likely to have hundreds +// or threads all simultaneously throwing on OOM conditions. +# define EMERGENCY_OBJ_COUNT (8 * __SIZEOF_POINTER__) +# define MAX_OBJ_COUNT (16 << __SIZEOF_POINTER__) #else -# define EMERGENCY_OBJ_SIZE 1024 -# define EMERGENCY_OBJ_COUNT 64 +# define EMERGENCY_OBJ_COUNT 4 +# define MAX_OBJ_COUNT 64 #endif -#ifndef __GTHREADS -# undef EMERGENCY_OBJ_COUNT -# define EMERGENCY_OBJ_COUNT 4 +// This can be set by configure. +#ifdef _GLIBCXX_EH_POOL_NOBJS +# if _GLIBCXX_EH_POOL_NOBJS > MAX_OBJ_COUNT +# warning "_GLIBCXX_EH_POOL_NOBJS value is too large; ignoring it" +# else +# undef EMERGENCY_OBJ_COUNT +# define EMERGENCY_OBJ_COUNT _GLIBCXX_EH_POOL_NOBJS +# endif #endif namespace __gnu_cxx { - void __freeres(); + void __freeres() noexcept; } namespace { + static constexpr std::size_t + buffer_size_in_bytes(std::size_t obj_count, std::size_t obj_size) noexcept + { + // N * (S * P + R + D) + constexpr std::size_t P = sizeof(void*); + constexpr std::size_t R = sizeof(__cxa_refcounted_exception); + constexpr std::size_t D = sizeof(__cxa_dependent_exception); + return obj_count * (obj_size * P + R + D); + } + // A fixed-size heap, variable size object allocator class pool { public: - pool(); + pool() noexcept; - _GLIBCXX_NODISCARD void *allocate (std::size_t); - void free (void *); + _GLIBCXX_NODISCARD void *allocate (std::size_t) noexcept; + void free (void *) noexcept; - bool in_pool (void *); + bool in_pool (void *) const noexcept; private: struct free_entry { @@ -101,33 +145,87 @@ namespace char data[] __attribute__((aligned)); }; +#ifdef __GTHREADS // A single mutex controlling emergency allocations. __gnu_cxx::__mutex emergency_mutex; + using __scoped_lock = __gnu_cxx::__scoped_lock; +#else + int emergency_mutex = 0; + struct __scoped_lock { explicit __scoped_lock(int) { } }; +#endif // The free-list - free_entry *first_free_entry; + free_entry *first_free_entry = nullptr; // The arena itself - we need to keep track of these only // to implement in_pool. - char *arena; - std::size_t arena_size; +#ifdef _GLIBCXX_EH_POOL_STATIC + static constexpr std::size_t arena_size + = buffer_size_in_bytes(EMERGENCY_OBJ_COUNT, EMERGENCY_OBJ_SIZE); + alignas(void*) char arena[std::max(arena_size, sizeof(free_entry))]; +#else + char *arena = nullptr; + std::size_t arena_size = 0; +#endif - friend void __gnu_cxx::__freeres(); + friend void __gnu_cxx::__freeres() noexcept; }; - pool::pool() + pool::pool() noexcept { - // Allocate the arena - we could add a GLIBCXX_EH_ARENA_SIZE environment - // to make this tunable. - arena_size = (EMERGENCY_OBJ_SIZE * EMERGENCY_OBJ_COUNT - + EMERGENCY_OBJ_COUNT * sizeof (__cxa_dependent_exception)); +#ifndef _GLIBCXX_EH_POOL_STATIC + int obj_size = EMERGENCY_OBJ_SIZE; + int obj_count = EMERGENCY_OBJ_COUNT; + +#if _GLIBCXX_HOSTED +#if _GLIBCXX_HAVE_SECURE_GETENV + const char* str = ::secure_getenv("GLIBCXX_TUNABLES"); +#else + const char* str = std::getenv("GLIBCXX_TUNABLES"); +#endif + const std::string_view ns_name = "glibcxx.eh_pool"; + std::pair tunables[]{ + {"obj_size", 0}, {"obj_count", obj_count} + }; + while (str) + { + if (*str == ':') + ++str; + + if (!ns_name.compare(0, ns_name.size(), str, ns_name.size()) + && str[ns_name.size()] == '.') + { + str += ns_name.size() + 1; + for (auto& t : tunables) + if (!t.first.compare(0, t.first.size(), str, t.first.size()) + && str[t.first.size()] == '=') + { + str += t.first.size() + 1; + char* end; + unsigned long val = strtoul(str, &end, 0); + if ((*end == ':' || *end == '\0') && val <= INT_MAX) + t.second = val; + str = end; + break; + } + } + str = strchr(str, ':'); + } + obj_count = std::min(tunables[1].second, MAX_OBJ_COUNT); // Can be zero. + if (tunables[0].second != 0) + obj_size = tunables[0].second; +#endif // HOSTED + + arena_size = buffer_size_in_bytes(obj_count, obj_size); + if (arena_size == 0) + return; arena = (char *)malloc (arena_size); if (!arena) { // If the allocation failed go without an emergency pool. arena_size = 0; - first_free_entry = NULL; return; } +#endif // STATIC // Populate the free-list with a single entry covering the whole arena first_free_entry = reinterpret_cast (arena); @@ -136,7 +234,7 @@ namespace first_free_entry->next = NULL; } - void *pool::allocate (std::size_t size) + void *pool::allocate (std::size_t size) noexcept { __gnu_cxx::__scoped_lock sentry(emergency_mutex); // We need an additional size_t member plus the padding to @@ -188,9 +286,9 @@ namespace return &x->data; } - void pool::free (void *data) + void pool::free (void *data) noexcept { - __gnu_cxx::__scoped_lock sentry(emergency_mutex); + __scoped_lock sentry(emergency_mutex); allocated_entry *e = reinterpret_cast (reinterpret_cast (data) - offsetof (allocated_entry, data)); std::size_t sz = e->size; @@ -252,11 +350,10 @@ namespace } } - bool pool::in_pool (void *ptr) + inline bool pool::in_pool (void *ptr) const noexcept { - char *p = reinterpret_cast (ptr); - return (p > arena - && p < arena + arena_size); + std::less less; + return less(ptr, arena + arena_size) && less(arena, ptr); } pool emergency_pool; @@ -264,29 +361,32 @@ namespace namespace __gnu_cxx { + __attribute__((cold)) void - __freeres() + __freeres() noexcept { +#ifndef _GLIBCXX_EH_POOL_STATIC if (emergency_pool.arena) { ::free(emergency_pool.arena); emergency_pool.arena = 0; } +#endif } } extern "C" void * -__cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) _GLIBCXX_NOTHROW +__cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) noexcept { void *ret; thrown_size += sizeof (__cxa_refcounted_exception); ret = malloc (thrown_size); - if (!ret) + if (!ret) [[__unlikely__]] ret = emergency_pool.allocate (thrown_size); - if (!ret) + if (!ret) [[__unlikely__]] std::terminate (); memset (ret, 0, sizeof (__cxa_refcounted_exception)); @@ -296,10 +396,10 @@ __cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) _GLIBCXX_NOTHROW extern "C" void -__cxxabiv1::__cxa_free_exception(void *vptr) _GLIBCXX_NOTHROW +__cxxabiv1::__cxa_free_exception(void *vptr) noexcept { char *ptr = (char *) vptr - sizeof (__cxa_refcounted_exception); - if (emergency_pool.in_pool (ptr)) + if (emergency_pool.in_pool (ptr)) [[__unlikely__]] emergency_pool.free (ptr); else free (ptr); @@ -307,31 +407,29 @@ __cxxabiv1::__cxa_free_exception(void *vptr) _GLIBCXX_NOTHROW extern "C" __cxa_dependent_exception* -__cxxabiv1::__cxa_allocate_dependent_exception() _GLIBCXX_NOTHROW +__cxxabiv1::__cxa_allocate_dependent_exception() noexcept { - __cxa_dependent_exception *ret; + void *ret; - ret = static_cast<__cxa_dependent_exception*> - (malloc (sizeof (__cxa_dependent_exception))); + ret = malloc (sizeof (__cxa_dependent_exception)); - if (!ret) - ret = static_cast <__cxa_dependent_exception*> - (emergency_pool.allocate (sizeof (__cxa_dependent_exception))); + if (!ret) [[__unlikely__]] + ret = emergency_pool.allocate (sizeof (__cxa_dependent_exception)); if (!ret) std::terminate (); memset (ret, 0, sizeof (__cxa_dependent_exception)); - return ret; + return static_cast<__cxa_dependent_exception*>(ret); } extern "C" void __cxxabiv1::__cxa_free_dependent_exception - (__cxa_dependent_exception *vptr) _GLIBCXX_NOTHROW + (__cxa_dependent_exception *vptr) noexcept { - if (emergency_pool.in_pool (vptr)) + if (emergency_pool.in_pool (vptr)) [[__unlikely__]] emergency_pool.free (vptr); else free (vptr);