* [PATCH 1/2] Add framework for tunables
2016-08-15 20:05 [PATCHv4 0/2] tunables for glibc Siddhesh Poyarekar
2016-08-15 20:05 ` [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable Siddhesh Poyarekar
@ 2016-08-15 20:05 ` Siddhesh Poyarekar
2016-08-23 8:08 ` [PING][PATCHv4 0/2] tunables for glibc Siddhesh Poyarekar
2016-08-23 19:08 ` [PATCHv4 " H.J. Lu
3 siblings, 0 replies; 9+ messages in thread
From: Siddhesh Poyarekar @ 2016-08-15 20:05 UTC (permalink / raw)
To: libc-alpha; +Cc: carlos, fweimer
The tunables framework allows us to uniformly manage and expose global
variables inside glibc as switches to users. tunables/README has
instructions for glibc developers to add new tunables.
Tunables support can be enabled by passing the --enable-tunables
configure flag to the configure script. This patch only adds a
framework and does not pose any limitations on how tunable values are
read from the user. It also adds environment variables used in malloc
behaviour tweaking to the tunables framework as a PoC of the
compatibility interface.
* manual/install.texi: Add --enable-tunables option.
* INSTALL: Regenerate.
* Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
(before-compile): Generate dl-tunable-list.h early.
* config.h.in: Add HAVE_TUNABLES.
* config.make.in: Add have-tunables.
* configure.ac: Add --enable-tunables option.
* configure: Regenerate.
* malloc/arena.c [HAVE_TUNABLES]: Include dl-tunables.h.
Define TUNABLE_NAMESPACE.
(DL_TUNABLE_CALLBACK (set_mallopt_check)): New function.
(DL_TUNABLE_CALLBACK_FNDECL): New macro. Use it to define
callback functions.
(ptmalloc_init): Set tunable values.
* malloc/malloc.c (do_set_mallopt_check,
do_set_mmap_threshold, do_set_mmaps_max, do_set_top_pad,
do_set_perturb_byte, do_set_trim_threshold, do_set_arena_max,
do_set_arena_test): New functions.
(__libc_mallopt): Use them.
* malloc/tst-malloc-usable-static.c: New test case.
* malloc/Makefile (tests-static): Add it.
* csu/init-first.c [HAVE_TUNABLES]: Include dl-tunables.h.
(__libc_init_first) [!SHARED]: Initialize tunables for static
binaries.
* scripts/gen-tunables.awk: New file.
* README.tunables: New file.
* elf/Makefile (dl-routines): Add dl-tunables.
* elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
namespace.
* elf/dl-sysdep.c [HAVE_TUNABLES]: Include dl-tunables.h
(_dl_sysdep_start): Call __TUNABLES_INIT.
* elf/dl-tunable-list.h: New auto-generated file.
* elf/dl-tunables.c: New file.
* elf/dl-tunables.h: New file.
* elf/dl-tunables.list: New file.
* elf/dl-tunable-types.h: New file.
* elf/rtld.c [HAVE_TUNABLES]: Include dl-tunables.h
(process_envvars): Call __tunables_init.
* sysdeps/mach/hurd/dl-sysdep.c [HAVE_TUNABLES]: Include
dl-tunables.h.
(_dl_sysdep_start): Call __TUNABLES_INIT.
* sysdeps/mach/hurd/i386/init-first.c [HAVE_TUNABLES]: Include
dl-tunables.h.
(_dl_sysdep_start): Call __TUNABLES_INIT.
---
INSTALL | 5 +
Makeconfig | 16 ++++
README.tunables | 84 ++++++++++++++++
config.h.in | 3 +
config.make.in | 1 +
configure | 16 ++++
configure.ac | 10 ++
csu/init-first.c | 7 ++
elf/Makefile | 5 +
elf/Versions | 3 +
elf/dl-sysdep.c | 8 ++
elf/dl-tunable-types.h | 46 +++++++++
elf/dl-tunables.c | 185 ++++++++++++++++++++++++++++++++++++
elf/dl-tunables.h | 78 +++++++++++++++
elf/dl-tunables.list | 69 ++++++++++++++
malloc/Makefile | 3 +
malloc/arena.c | 54 +++++++++++
malloc/malloc.c | 126 +++++++++++++++++-------
malloc/tst-malloc-usable-static.c | 1 +
manual/install.texi | 5 +
scripts/gen-tunables.awk | 157 ++++++++++++++++++++++++++++++
sysdeps/mach/hurd/dl-sysdep.c | 8 ++
sysdeps/mach/hurd/i386/init-first.c | 8 ++
23 files changed, 864 insertions(+), 34 deletions(-)
create mode 100644 README.tunables
create mode 100644 elf/dl-tunable-types.h
create mode 100644 elf/dl-tunables.c
create mode 100644 elf/dl-tunables.h
create mode 100644 elf/dl-tunables.list
create mode 100644 malloc/tst-malloc-usable-static.c
create mode 100644 scripts/gen-tunables.awk
diff --git a/INSTALL b/INSTALL
index ec3445f..43edee4 100644
--- a/INSTALL
+++ b/INSTALL
@@ -158,6 +158,11 @@ will be used, and CFLAGS sets optimization options for the compiler.
By default for x86_64, the GNU C Library is built with vector math
library. Use this option to disable vector math library.
+'--enable-tunables'
+ Tunables support allows additional library parameters to be
+ customized at runtime. This is an experimental feature and affects
+ startup time and is thus disabled by default.
+
'--build=BUILD-SYSTEM'
'--host=HOST-SYSTEM'
These options are for cross-compiling. If you specify both options
diff --git a/Makeconfig b/Makeconfig
index 03fd89c..ba1180f 100644
--- a/Makeconfig
+++ b/Makeconfig
@@ -883,6 +883,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \
$(foreach lib,$(libof-$(basename $(@F))) \
$(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \
$(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F)))
+
+ifeq (yes,$(have-tunables))
+CPPFLAGS += -DTOP_NAMESPACE=glibc
+endif
+
override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \
$(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \
$(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \
@@ -1055,6 +1060,17 @@ $(common-objpfx)libc-modules.stmp: $(..)scripts/gen-libc-modules.awk \
endif
+# Build the tunables list header early since it could be used by any module in
+# glibc.
+ifeq (yes,$(have-tunables))
+before-compile += $(common-objpfx)dl-tunable-list.h
+
+$(common-objpfx)dl-tunable-list.h: $(..)scripts/gen-tunables.awk \
+ $(..)elf/dl-tunables.list
+ $(AWK) -f $^ > $@.tmp
+ mv $@{.tmp,}
+endif
+
common-generated += libc-modules.h libc-modules.stmp
# The name under which the run-time dynamic linker is installed.
diff --git a/README.tunables b/README.tunables
new file mode 100644
index 0000000..3d9c86a
--- /dev/null
+++ b/README.tunables
@@ -0,0 +1,84 @@
+ TUNABLE FRAMEWORK
+ =================
+
+Tunables is a feature in the GNU C Library that allows application authors and
+distribution maintainers to alter the runtime library behaviour to match their
+workload.
+
+The tunable framework allows modules within glibc to register variables that
+may be tweaked through an environment variable or an API call. It aims to
+enforce a strict namespace rule to bring consistency to naming of these tunable
+environment variables across the project. This document is a guide for glibc
+developers to add tunables to the framework.
+
+ADDING A NEW TUNABLE
+--------------------
+
+The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in
+distributions for specific tunables if they want to add their own tunables.
+Downstream implementations are discouraged from using the 'glibc' namespace for
+tunables they don't already have consensus to push upstream.
+
+There are two steps to adding a tunable:
+
+1. Add a tunable ID:
+
+Modules that wish to use the tunables interface must define the
+TUNABLE_NAMESPACE macro. Following this, for each tunable you want to
+add, make an entry in elf/dl-tunables.list. The format of the file is as
+follows:
+
+TOP_NAMESPACE {
+ NAMESPACE1 {
+ TUNABLE1 {
+ # tunable attributes, one per line
+ }
+ # A tunable with default attributes, i.e. string variable.
+ TUNABLE2
+ TUNABLE3 {
+ # its attributes
+ }
+ }
+ NAMESPACE2 {
+ ...
+ }
+}
+
+The list of allowed attributes are:
+
+- type: Data type. Defaults to STRING. Allowed types are:
+ INT_32, SIZE_T and STRING.
+
+- minval: Optional minimum acceptable value. For a string type
+ this is the minimum length of the value.
+
+- maxval: Optional maximum acceptable value. For a string type
+ this is the maximum length of the value.
+
+- env_alias: An alias environment variable
+
+- is_secure: Specify whether the tunable should be read for setuid
+ binaries. True allows the tunable to be read for
+ setuid binaries while false disables it. Note that
+ even if this is set as true and the value is read, it
+ may not be used if it does not validate against the
+ acceptable values or is not considered safe by the
+ module.
+
+2. Call either the TUNABLE_SET_VALUE and pass into it the tunable name and a
+ pointer to the variable that should be set with the tunable value.
+ If additional work needs to be done after setting the value, use the
+ TUNABLE_SET_VALUE_WITH_CALLBACK instead and additionally pass a pointer to
+ the function that should be called if the tunable value has been set.
+
+FUTURE WORK
+-----------
+
+The framework currently only allows a one-time initialization of variables
+through environment variables and in some cases, modification of variables via
+an API call. A future goals for this project include:
+
+- Setting system-wide and user-wide defaults for tunables through some
+ mechanism like a configuration file.
+
+- Allow tweaking of some tunables at runtime
diff --git a/config.h.in b/config.h.in
index 856ef6a..f8be48e 100644
--- a/config.h.in
+++ b/config.h.in
@@ -240,4 +240,7 @@
/* PowerPC32 uses fctidz for floating point to long long conversions. */
#define HAVE_PPC_FCTIDZ 0
+/* Build glibc with tunables support. */
+#define HAVE_TUNABLES 0
+
#endif
diff --git a/config.make.in b/config.make.in
index 95c6f36..8ca9579 100644
--- a/config.make.in
+++ b/config.make.in
@@ -91,6 +91,7 @@ use-nscd = @use_nscd@
build-hardcoded-path-in-tests= @hardcoded_path_in_tests@
build-pt-chown = @build_pt_chown@
enable-lock-elision = @enable_lock_elision@
+have-tunables = @have_tunables@
# Build tools.
CC = @CC@
diff --git a/configure b/configure
index 17625e1..b184205 100755
--- a/configure
+++ b/configure
@@ -660,6 +660,7 @@ multi_arch
base_machine
add_on_subdirs
add_ons
+have_tunables
build_pt_chown
build_nscd
link_obsolete_rpc
@@ -774,6 +775,7 @@ enable_systemtap
enable_build_nscd
enable_nscd
enable_pt_chown
+enable_tunables
enable_mathvec
with_cpu
'
@@ -1441,6 +1443,7 @@ Optional Features:
--disable-build-nscd disable building and installing the nscd daemon
--disable-nscd library functions will not contact the nscd daemon
--enable-pt_chown Enable building and installing pt_chown
+ --enable-tunables Enable tunables support
--enable-mathvec Enable building and installing mathvec [default
depends on architecture]
@@ -3647,6 +3650,19 @@ if test "$build_pt_chown" = yes; then
fi
+# Check whether --enable-tunables was given.
+if test "${enable_tunables+set}" = set; then :
+ enableval=$enable_tunables; have_tunables=$enableval
+else
+ have_tunables=no
+fi
+
+
+if test "$have_tunables" = yes; then
+ $as_echo "#define HAVE_TUNABLES 1" >>confdefs.h
+
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/configure.ac b/configure.ac
index 33bcd62..ece902a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -395,6 +395,16 @@ if test "$build_pt_chown" = yes; then
AC_DEFINE(HAVE_PT_CHOWN)
fi
+AC_ARG_ENABLE([tunables],
+ [AS_HELP_STRING([--enable-tunables],
+ [Enable tunables support])],
+ [have_tunables=$enableval],
+ [have_tunables=no])
+AC_SUBST(have_tunables)
+if test "$have_tunables" = yes; then
+ AC_DEFINE(HAVE_TUNABLES)
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/csu/init-first.c b/csu/init-first.c
index 77c6e1c..a35f997 100644
--- a/csu/init-first.c
+++ b/csu/init-first.c
@@ -28,6 +28,9 @@
#include <libc-internal.h>
#include <ldsodefs.h>
+#if HAVE_TUNABLES
+# include <elf/dl-tunables.h>
+#endif
/* Set nonzero if we have to be prepared for more than one libc being
used in the process. Safe assumption if initializer never runs. */
@@ -74,6 +77,10 @@ _init (int argc, char **argv, char **envp)
#ifndef SHARED
__libc_init_secure ();
+#if HAVE_TUNABLES
+ __tunables_init (envp);
+#endif
+
/* First the initialization which normally would be done by the
dynamic linker. */
_dl_non_dynamic_init ();
diff --git a/elf/Makefile b/elf/Makefile
index 593403c..48989e7 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -35,6 +35,11 @@ dl-routines = $(addprefix dl-,load lookup object reloc deps hwcaps \
ifeq (yes,$(use-ldconfig))
dl-routines += dl-cache
endif
+
+ifeq (yes,$(have-tunables))
+dl-routines += dl-tunables
+endif
+
all-dl-routines = $(dl-routines) $(sysdep-dl-routines)
# But they are absent from the shared libc, because that code is in ld.so.
elide-routines.os = $(all-dl-routines) dl-support enbl-secure dl-origin \
diff --git a/elf/Versions b/elf/Versions
index 23deda9..1734697 100644
--- a/elf/Versions
+++ b/elf/Versions
@@ -64,5 +64,8 @@ ld {
# Pointer protection.
__pointer_chk_guard;
+
+ # Set value of a tunable.
+ __tunable_set_val;
}
}
diff --git a/elf/dl-sysdep.c b/elf/dl-sysdep.c
index eaa7155..68f30f2 100644
--- a/elf/dl-sysdep.c
+++ b/elf/dl-sysdep.c
@@ -44,6 +44,10 @@
#include <hp-timing.h>
#include <tls.h>
+#if HAVE_TUNABLES
+# include <dl-tunables.h>
+#endif
+
extern char **_environ attribute_hidden;
extern char _end[] attribute_hidden;
@@ -219,6 +223,10 @@ _dl_sysdep_start (void **start_argptr,
}
#endif
+#if HAVE_TUNABLES
+ __tunables_init (_environ);
+#endif
+
#ifdef DL_SYSDEP_INIT
DL_SYSDEP_INIT;
#endif
diff --git a/elf/dl-tunable-types.h b/elf/dl-tunable-types.h
new file mode 100644
index 0000000..d1591b6
--- /dev/null
+++ b/elf/dl-tunable-types.h
@@ -0,0 +1,46 @@
+/* Tunable type information.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLE_TYPES_H_
+# define _TUNABLE_TYPES_H_
+#include <stddef.h>
+
+typedef void (*tunable_callback_t) (void *);
+
+typedef enum
+{
+ TUNABLE_TYPE_INT_32,
+ TUNABLE_TYPE_SIZE_T,
+ TUNABLE_TYPE_STRING
+} tunable_type_code_t;
+
+typedef struct
+{
+ tunable_type_code_t type_code;
+ int64_t min;
+ int64_t max;
+} tunable_type_t;
+
+typedef union
+{
+ int64_t numval;
+ const char *strval;
+} tunable_val_t;
+
+#endif
diff --git a/elf/dl-tunables.c b/elf/dl-tunables.c
new file mode 100644
index 0000000..b8f8fc0
--- /dev/null
+++ b/elf/dl-tunables.c
@@ -0,0 +1,185 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <stdlib.h>
+
+#define TUNABLES_INTERNAL 1
+#include "dl-tunables.h"
+
+/* Compare environment names, bounded by the name hardcoded in glibc. */
+static bool
+is_name (const char *orig, const char *envname)
+{
+ for (;*orig != '\0' && *envname != '\0'; envname++, orig++)
+ if (*orig != *envname)
+ break;
+
+ /* The ENVNAME is immediately followed by a value. */
+ if (*orig == '\0' && *envname == '=')
+ return true;
+ else
+ return false;
+}
+
+static char **
+get_next_env (char **envp, char **name, size_t *namelen, char **val)
+{
+ while (envp != NULL && *envp != NULL)
+ {
+ char *envline = *envp;
+ int len = 0;
+
+ while (envline[len] != '\0' && envline[len] != '=')
+ len++;
+
+ /* Just the name and no value, go to the next one. */
+ if (envline[len] == '\0')
+ continue;
+
+ *name = envline;
+ *namelen = len;
+ *val = &envline[len + 1];
+
+ return ++envp;
+ }
+
+ return NULL;
+}
+
+/* If the value does not validate, don't bother initializing the internal type
+ and also take care to clear the recorded string value in STRVAL. */
+#define RETURN_IF_INVALID_RANGE(__cur, __val) \
+({ \
+ __typeof ((__cur)->type.min) min = (__cur)->type.min; \
+ __typeof ((__cur)->type.max) max = (__cur)->type.max; \
+ if (min != max && ((__val) < min || (__val) > max)) \
+ return; \
+})
+
+/* Validate range of the input value and initialize the tunable CUR if it looks
+ good. */
+static void
+tunable_initialize (tunable_t *cur, const char *strval)
+{
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ int32_t val = (int32_t) __strtoul_internal (strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ cur->strval = strval;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ size_t val = (size_t) __strtoul_internal (strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ cur->strval = strval;
+ break;
+ }
+ case TUNABLE_TYPE_STRING:
+ {
+ cur->val.strval = cur->strval = strval;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+}
+
+/* Initialize the tunables list from the environment. For now we only use the
+ ENV_ALIAS to find values. Later we will also use the tunable names to find
+ values. */
+void
+__tunables_init (char **envp)
+{
+ char *envname = NULL;
+ char *envval = NULL;
+ size_t len = 0;
+
+ while ((envp = get_next_env (envp, &envname, &len, &envval)) != NULL)
+ {
+ for (int i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
+ {
+ tunable_t *cur = &tunable_list[i];
+
+ /* Skip over tunables that have either been set already or should be
+ skipped. */
+ if (cur->strval != NULL || cur->env_alias == NULL
+ || (__libc_enable_secure && !cur->is_secure))
+ continue;
+
+ const char *name = cur->env_alias;
+
+ /* We have a match. Initialize and move on to the next line. */
+ if (is_name (name, envname))
+ {
+ tunable_initialize (cur, envval);
+ break;
+ }
+ }
+ }
+}
+
+/* Set the tunable value. This is called by the module that the tunable exists
+ in. */
+void
+__tunable_set_val (tunable_id_t id, void *valp, tunable_callback_t callback)
+{
+ tunable_t *cur = &tunable_list[id];
+
+ /* Don't do anything if our tunable was not set during initialization or if
+ it failed validation. */
+ if (cur->strval == NULL)
+ return;
+
+ if (valp == NULL)
+ goto cb;
+
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ *((int32_t *) valp) = (int32_t) cur->val.numval;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ *((size_t *) valp) = (size_t) cur->val.numval;
+ break;
+ }
+ case TUNABLE_TYPE_STRING:
+ {
+ *((const char **)valp) = cur->val.strval;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+
+cb:
+ if (callback)
+ callback (&cur->val);
+}
diff --git a/elf/dl-tunables.h b/elf/dl-tunables.h
new file mode 100644
index 0000000..10a644b
--- /dev/null
+++ b/elf/dl-tunables.h
@@ -0,0 +1,78 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLES_H_
+# define _TUNABLES_H_
+# include <stddef.h>
+# include "dl-tunable-types.h"
+
+/* A tunable. */
+struct _tunable
+{
+ const char *name; /* Internal name of the tunable. */
+ tunable_type_t type; /* Data type of the tunable. */
+ tunable_val_t val; /* The value. */
+ const char *strval; /* The string containing the value,
+ points into envp. */
+ bool is_secure; /* Whether the tunable must be read
+ even for setuid binaries. Note that
+ even if the tunable is read, it may
+ not get used by the target module if
+ the value is considered unsafe. */
+ /* Compatibility elements. */
+ const char *env_alias; /* The compatibility environment
+ variable name. */
+};
+
+typedef struct _tunable tunable_t;
+
+/* Full name for a tunable is top_ns.tunable_ns.id. */
+#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id
+
+# define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)
+# define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id
+
+#include "dl-tunable-list.h"
+
+extern void __tunables_init (char **);
+extern void __tunable_set_val (tunable_id_t, void *, tunable_callback_t);
+
+/* Check if the tunable has been set to a non-default value and if it is, copy
+ it over into __VAL. */
+# define TUNABLE_SET_VAL(__id,__val) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ NULL); \
+})
+
+/* Same as TUNABLE_SET_VAL, but also set the callback function to __CB and call
+ it. */
+# define TUNABLE_SET_VAL_WITH_CALLBACK(__id,__val,__cb) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ DL_TUNABLE_CALLBACK (__cb)); \
+})
+
+/* Namespace sanity for callback functions. Use this macro to keep the
+ namespace of the modules clean. */
+#define DL_TUNABLE_CALLBACK(__name) _dl_tunable_ ## __name
+#endif
diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list
new file mode 100644
index 0000000..11504c4
--- /dev/null
+++ b/elf/dl-tunables.list
@@ -0,0 +1,69 @@
+# Copyright (C) 2016 Free Software Foundation, Inc.
+# This file is part of the GNU C Library.
+
+# The GNU C Library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+
+# The GNU C Library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+
+# You should have received a copy of the GNU Lesser General Public
+# License along with the GNU C Library; if not, see
+# <http://www.gnu.org/licenses/>.
+
+# Allowed attributes for tunables:
+#
+# type: Defaults to STRING
+# minval: Optional minimum acceptable value
+# maxval: Optional maximum acceptable value
+# env_alias: An alias environment variable
+# is_secure: Specify whether the environment variable should be read for
+# setuid binaries.
+
+glibc {
+ malloc {
+ check {
+ type: INT_32
+ minval: 0
+ maxval: 3
+ env_alias: MALLOC_CHECK_
+ is_secure: true
+ }
+ top_pad {
+ type: SIZE_T
+ env_alias: MALLOC_TOP_PAD_
+ }
+ perturb {
+ type: INT_32
+ minval: 0
+ maxval: 0xff
+ env_alias: MALLOC_PERTURB_
+ }
+ mmap_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_MMAP_THRESHOLD_
+ }
+ trim_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_TRIM_THRESHOLD_
+ }
+ mmap_max {
+ type: INT_32
+ env_alias: MALLOC_MMAP_MAX_
+ }
+ arena_max {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_MAX
+ minval: 1
+ }
+ arena_test {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_TEST
+ minval: 1
+ }
+ }
+}
diff --git a/malloc/Makefile b/malloc/Makefile
index 4d5c81d..de9aafe 100644
--- a/malloc/Makefile
+++ b/malloc/Makefile
@@ -31,6 +31,8 @@ tests := mallocbug tst-malloc tst-valloc tst-calloc tst-obstack \
tst-malloc-backtrace tst-malloc-thread-exit \
tst-malloc-thread-fail tst-malloc-fork-deadlock \
tst-mallocfork2
+tests-static := tst-malloc-usable-static
+tests += $(tests-static)
test-srcs = tst-mtrace
routines = malloc morecore mcheck mtrace obstack \
@@ -138,6 +140,7 @@ endif
tst-mcheck-ENV = MALLOC_CHECK_=3
tst-malloc-usable-ENV = MALLOC_CHECK_=3
+tst-malloc-usable-static-ENV = $(tst-malloc-usable-ENV)
# Uncomment this for test releases. For public releases it is too expensive.
#CPPFLAGS-malloc.o += -DMALLOC_DEBUG=1
diff --git a/malloc/arena.c b/malloc/arena.c
index 4e16593..ddca8a7 100644
--- a/malloc/arena.c
+++ b/malloc/arena.c
@@ -19,6 +19,11 @@
#include <stdbool.h>
+#if HAVE_TUNABLES
+# define TUNABLE_NAMESPACE malloc
+# include <elf/dl-tunables.h>
+#endif
+
/* Compile-time constants. */
#define HEAP_MIN_SIZE (32 * 1024)
@@ -204,6 +209,34 @@ __malloc_fork_unlock_child (void)
mutex_init (&list_lock);
}
+#if HAVE_TUNABLES
+static inline int do_set_mallopt_check (int32_t value);
+void
+DL_TUNABLE_CALLBACK (set_mallopt_check) (void *valp)
+{
+ int32_t value = *(int32_t *) valp;
+ do_set_mallopt_check (value);
+ if (check_action != 0)
+ __malloc_check_init ();
+}
+
+# define DL_TUNABLE_CALLBACK_FNDECL(__name, __type) \
+static inline int do_ ## __name (__type value); \
+void \
+DL_TUNABLE_CALLBACK (__name) (void *valp) \
+{ \
+ __type value = *(__type *) valp; \
+ do_ ## __name (value); \
+}
+
+DL_TUNABLE_CALLBACK_FNDECL (set_mmap_threshold, size_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_mmaps_max, int32_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_top_pad, size_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_perturb_byte, int32_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_trim_threshold, size_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_arena_max, size_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_arena_test, size_t)
+#else
/* Initialization routine. */
#include <string.h>
extern char **_environ;
@@ -238,6 +271,7 @@ next_env_entry (char ***position)
return result;
}
+#endif
#ifdef SHARED
@@ -272,6 +306,24 @@ ptmalloc_init (void)
#endif
thread_arena = &main_arena;
+
+#if HAVE_TUNABLES
+ /* Ensure initialization/consolidation and do it under a lock so that a
+ thread attempting to use the arena in parallel waits on us till we
+ finish. */
+ mutex_lock (&main_arena.mutex);
+ malloc_consolidate (&main_arena);
+
+ TUNABLE_SET_VAL_WITH_CALLBACK (check, NULL, set_mallopt_check);
+ TUNABLE_SET_VAL_WITH_CALLBACK (top_pad, NULL, set_top_pad);
+ TUNABLE_SET_VAL_WITH_CALLBACK (perturb, NULL, set_perturb_byte);
+ TUNABLE_SET_VAL_WITH_CALLBACK (mmap_threshold, NULL, set_mmap_threshold);
+ TUNABLE_SET_VAL_WITH_CALLBACK (trim_threshold, NULL, set_trim_threshold);
+ TUNABLE_SET_VAL_WITH_CALLBACK (mmap_max, NULL, set_mmaps_max);
+ TUNABLE_SET_VAL_WITH_CALLBACK (arena_max, NULL, set_arena_max);
+ TUNABLE_SET_VAL_WITH_CALLBACK (arena_test, NULL, set_arena_test);
+ mutex_unlock (&main_arena.mutex);
+#else
const char *s = NULL;
if (__glibc_likely (_environ != NULL))
{
@@ -340,6 +392,8 @@ ptmalloc_init (void)
if (check_action != 0)
__malloc_check_init ();
}
+#endif
+
#if HAVE_MALLOC_INIT_HOOK
void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook);
if (hook != NULL)
diff --git a/malloc/malloc.c b/malloc/malloc.c
index bb52b3e..002c916 100644
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -4724,6 +4724,90 @@ __malloc_stats (void)
/*
------------------------------ mallopt ------------------------------
*/
+static inline int
+__always_inline
+do_set_mallopt_check (int32_t value)
+{
+ LIBC_PROBE (memory_mallopt_check_action, 2, value, check_action);
+ check_action = value;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_mmap_threshold (size_t value)
+{
+ if (value <= HEAP_MAX_SIZE / 2)
+ {
+ LIBC_PROBE (memory_mallopt_mmap_threshold, 3, value, mp_.mmap_threshold,
+ mp_.no_dyn_threshold);
+ mp_.mmap_threshold = value;
+ mp_.no_dyn_threshold = 1;
+ return 1;
+ }
+ return 0;
+}
+
+static inline int
+__always_inline
+do_set_mmaps_max (int32_t value)
+{
+ LIBC_PROBE (memory_mallopt_mmap_max, 3, value, mp_.n_mmaps_max,
+ mp_.no_dyn_threshold);
+ mp_.n_mmaps_max = value;
+ mp_.no_dyn_threshold = 1;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_top_pad (size_t value)
+{
+ LIBC_PROBE (memory_mallopt_top_pad, 3, value, mp_.top_pad,
+ mp_.no_dyn_threshold);
+ mp_.top_pad = value;
+ mp_.no_dyn_threshold = 1;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_perturb_byte (int32_t value)
+{
+ LIBC_PROBE (memory_mallopt_perturb, 2, value, perturb_byte);
+ perturb_byte = value;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_trim_threshold (size_t value)
+{
+ LIBC_PROBE (memory_mallopt_trim_threshold, 3, value, mp_.trim_threshold,
+ mp_.no_dyn_threshold);
+ mp_.trim_threshold = value;
+ mp_.no_dyn_threshold = 1;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_arena_max (size_t value)
+{
+ LIBC_PROBE (memory_mallopt_arena_max, 2, value, mp_.arena_max);
+ mp_.arena_max = value;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_arena_test (size_t value)
+{
+ LIBC_PROBE (memory_mallopt_arena_test, 2, value, mp_.arena_test);
+ mp_.arena_test = value;
+ return 1;
+}
+
int
__libc_mallopt (int param_number, int value)
@@ -4752,63 +4836,37 @@ __libc_mallopt (int param_number, int value)
break;
case M_TRIM_THRESHOLD:
- LIBC_PROBE (memory_mallopt_trim_threshold, 3, value,
- mp_.trim_threshold, mp_.no_dyn_threshold);
- mp_.trim_threshold = value;
- mp_.no_dyn_threshold = 1;
+ do_set_trim_threshold (value);
break;
case M_TOP_PAD:
- LIBC_PROBE (memory_mallopt_top_pad, 3, value,
- mp_.top_pad, mp_.no_dyn_threshold);
- mp_.top_pad = value;
- mp_.no_dyn_threshold = 1;
+ do_set_top_pad (value);
break;
case M_MMAP_THRESHOLD:
- /* Forbid setting the threshold too high. */
- if ((unsigned long) value > HEAP_MAX_SIZE / 2)
- res = 0;
- else
- {
- LIBC_PROBE (memory_mallopt_mmap_threshold, 3, value,
- mp_.mmap_threshold, mp_.no_dyn_threshold);
- mp_.mmap_threshold = value;
- mp_.no_dyn_threshold = 1;
- }
+ res = do_set_mmap_threshold (value);
break;
case M_MMAP_MAX:
- LIBC_PROBE (memory_mallopt_mmap_max, 3, value,
- mp_.n_mmaps_max, mp_.no_dyn_threshold);
- mp_.n_mmaps_max = value;
- mp_.no_dyn_threshold = 1;
+ do_set_mmaps_max (value);
break;
case M_CHECK_ACTION:
- LIBC_PROBE (memory_mallopt_check_action, 2, value, check_action);
- check_action = value;
+ do_set_mallopt_check (value);
break;
case M_PERTURB:
- LIBC_PROBE (memory_mallopt_perturb, 2, value, perturb_byte);
- perturb_byte = value;
+ do_set_perturb_byte (value);
break;
case M_ARENA_TEST:
if (value > 0)
- {
- LIBC_PROBE (memory_mallopt_arena_test, 2, value, mp_.arena_test);
- mp_.arena_test = value;
- }
+ do_set_arena_test (value);
break;
case M_ARENA_MAX:
if (value > 0)
- {
- LIBC_PROBE (memory_mallopt_arena_max, 2, value, mp_.arena_max);
- mp_.arena_max = value;
- }
+ do_set_arena_test (value);
break;
}
(void) mutex_unlock (&av->mutex);
diff --git a/malloc/tst-malloc-usable-static.c b/malloc/tst-malloc-usable-static.c
new file mode 100644
index 0000000..8907db0
--- /dev/null
+++ b/malloc/tst-malloc-usable-static.c
@@ -0,0 +1 @@
+#include <malloc/tst-malloc-usable.c>
diff --git a/manual/install.texi b/manual/install.texi
index 79ee45f..065b082 100644
--- a/manual/install.texi
+++ b/manual/install.texi
@@ -189,6 +189,11 @@ configure with @option{--disable-werror}.
By default for x86_64, @theglibc{} is built with vector math library.
Use this option to disable vector math library.
+@item --enable-tunables
+Tunables support allows additional library parameters to be customized at
+runtime. This is an experimental feature and affects startup time and is thus
+disabled by default.
+
@item --build=@var{build-system}
@itemx --host=@var{host-system}
These options are for cross-compiling. If you specify both options and
diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
new file mode 100644
index 0000000..b65b5a4
--- /dev/null
+++ b/scripts/gen-tunables.awk
@@ -0,0 +1,157 @@
+# Generate dl-tunable-list.h from dl-tunables.list
+
+BEGIN {
+ tunable=""
+ ns=""
+ top_ns=""
+}
+
+# Skip over blank lines and comments.
+/^#/ {
+ next
+}
+
+/^[ \t]*$/ {
+ next
+}
+
+# Beginning of either a top namespace, tunable namespace or a tunable, decided
+# on the current value of TUNABLE, NS or TOP_NS.
+$2 == "{" {
+ if (top_ns == "") {
+ top_ns = $1
+ }
+ else if (ns == "") {
+ ns = $1
+ }
+ else if (tunable == "") {
+ tunable = $1
+ }
+ else {
+ printf ("Unexpected occurrence of '{': %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+
+ next
+}
+
+# End of either a top namespace, tunable namespace or a tunable.
+$1 == "}" {
+ if (tunable != "") {
+ # Tunables definition ended, now fill in default attributes.
+ if (!types[top_ns][ns][tunable]) {
+ types[top_ns][ns][tunable] = "STRING"
+ }
+ if (!minvals[top_ns][ns][tunable]) {
+ minvals[top_ns][ns][tunable] = "0"
+ }
+ if (!maxvals[top_ns][ns][tunable]) {
+ maxvals[top_ns][ns][tunable] = "0"
+ }
+ if (!env_alias[top_ns][ns][tunable]) {
+ env_alias[top_ns][ns][tunable] = "NULL"
+ }
+ if (!is_secure[top_ns][ns][tunable]) {
+ is_secure[top_ns][ns][tunable] = "false"
+ }
+
+ tunable = ""
+ }
+ else if (ns != "") {
+ ns = ""
+ }
+ else if (top_ns != "") {
+ top_ns = ""
+ }
+ else {
+ printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+ next
+}
+
+# Everything else, which could either be a tunable without any attributes or a
+# tunable attribute.
+{
+ if (ns == "") {
+ printf("Line %d: Invalid tunable outside a namespace: %s\n", NR, $0)
+ exit 1
+ }
+
+ if (tunable == "") {
+ # We encountered a tunable without any attributes, so note it with a
+ # default.
+ types[top_ns][ns][$1] = "STRING"
+ next
+ }
+
+ # Otherwise, we have encountered a tunable attribute.
+ split($0, arr, ":")
+ attr = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[1])
+ val = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[2])
+
+ if (attr == "type") {
+ types[top_ns][ns][tunable] = val
+ }
+ else if (attr == "minval") {
+ minvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "maxval") {
+ maxvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "env_alias") {
+ env_alias[top_ns][ns][tunable] = sprintf("\"%s\"", val)
+ }
+ else if (attr == "is_secure") {
+ if (val == "true" || val == "false") {
+ is_secure[top_ns][ns][tunable] = val
+ }
+ else {
+ printf("Line %d: Invalid value (%s) for is_secure: %s, ", NR, val,
+ $0)
+ print("Allowed values are 'true' or 'false'")
+ exit 1
+ }
+ }
+}
+
+END {
+ if (ns != "") {
+ print "Unterminated namespace. Is a closing brace missing?"
+ exit 1
+ }
+
+ print "/* AUTOGENERATED by gen-tunables.awk. */"
+ print "#ifndef _TUNABLES_H_"
+ print "# error \"Do not include this file directly.\""
+ print "# error \"Include tunables.h instead.\""
+ print "#endif"
+
+ # Now, the enum names
+ print "\ntypedef enum"
+ print "{"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, m);
+ }
+ }
+ }
+ print "} tunable_id_t;\n"
+
+ # Finally, the tunable list.
+ print "\n#ifdef TUNABLES_INTERNAL"
+ print "static tunable_t tunable_list[] = {"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" {TUNABLE_NAME_S(%s, %s, %s)", t, n, m)
+ printf (", {TUNABLE_TYPE_%s, %s, %s}, {.numval = 0}, NULL, %s, %s},\n",
+ types[t][n][m], minvals[t][n][m], maxvals[t][n][m],
+ is_secure[t][n][m], env_alias[t][n][m]);
+ }
+ }
+ }
+ print "};"
+ print "#endif"
+}
diff --git a/sysdeps/mach/hurd/dl-sysdep.c b/sysdeps/mach/hurd/dl-sysdep.c
index 76140cf..d2b7422 100644
--- a/sysdeps/mach/hurd/dl-sysdep.c
+++ b/sysdeps/mach/hurd/dl-sysdep.c
@@ -44,6 +44,10 @@
#include <dl-machine.h>
#include <dl-procinfo.h>
+#if HAVE_TUNABLES
+# include <dl-tunables.h>
+#endif
+
extern void __mach_init (void);
extern int _dl_argc;
@@ -143,6 +147,10 @@ _dl_sysdep_start (void **start_argptr,
__libc_enable_secure = _dl_hurd_data->flags & EXEC_SECURE;
+#if HAVE_TUNABLES
+ __tunables_init (_environ);
+#endif
+
if (_dl_hurd_data->flags & EXEC_STACK_ARGS &&
_dl_hurd_data->user_entry == 0)
_dl_hurd_data->user_entry = (vm_address_t) ENTRY_POINT;
diff --git a/sysdeps/mach/hurd/i386/init-first.c b/sysdeps/mach/hurd/i386/init-first.c
index 7a905c2..7723479 100644
--- a/sysdeps/mach/hurd/i386/init-first.c
+++ b/sysdeps/mach/hurd/i386/init-first.c
@@ -31,6 +31,10 @@
#include <ldsodefs.h>
#include <fpu_control.h>
+#if HAVE_TUNABLES
+# include <dl-tunables.h>
+#endif
+
extern void __mach_init (void);
extern void __init_misc (int, char **, char **);
#ifdef USE_NONOPTION_FLAGS
@@ -83,6 +87,10 @@ posixland_init (int argc, char **argv, char **envp)
__libc_argv = argv;
__environ = envp;
+#if HAVE_TUNABLES
+ __tunables_init (__environ);
+#endif
+
#ifndef SHARED
_dl_non_dynamic_init ();
#endif
--
2.7.4
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCHv4 0/2] tunables for glibc
@ 2016-08-15 20:05 Siddhesh Poyarekar
2016-08-15 20:05 ` [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable Siddhesh Poyarekar
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Siddhesh Poyarekar @ 2016-08-15 20:05 UTC (permalink / raw)
To: libc-alpha; +Cc: carlos, fweimer
Hi,
After another episode of disappearance, here is an updated iteration of the
tunables patch with suggestions from Florian, Carlos and H. J. incorporated.
Changes since the last version:
- Fixed mallopt behaviour using a set of intermediate functions that the
callback and __libc_mallopt use. I did not make them into direct callbacks
because the callbacks are built only when tunables is enabled.
- Loads of comment and documentation fixes based on Carlos' and Florian's
suggestions
- Call __init_tunables even earlier to ensure that tunables are initialized
before get_cpu_features is called.
Siddhesh Poyarekar (2):
Add framework for tunables
Initialize tunable list with the GLIBC_TUNABLES environment variable
INSTALL | 5 +
Makeconfig | 16 ++
README.tunables | 84 +++++++++
config.h.in | 3 +
config.make.in | 1 +
configure | 16 ++
configure.ac | 10 +
csu/init-first.c | 7 +
elf/Makefile | 5 +
elf/Versions | 3 +
elf/dl-sysdep.c | 8 +
elf/dl-tunable-types.h | 46 +++++
elf/dl-tunables.c | 284 +++++++++++++++++++++++++++++
elf/dl-tunables.h | 78 ++++++++
elf/dl-tunables.list | 69 +++++++
malloc/Makefile | 7 +-
malloc/arena.c | 54 ++++++
malloc/malloc.c | 126 +++++++++----
malloc/tst-malloc-usable-static-tunables.c | 1 +
malloc/tst-malloc-usable-static.c | 1 +
malloc/tst-malloc-usable-tunables.c | 1 +
manual/install.texi | 5 +
scripts/gen-tunables.awk | 157 ++++++++++++++++
sysdeps/mach/hurd/dl-sysdep.c | 8 +
sysdeps/mach/hurd/i386/init-first.c | 8 +
25 files changed, 968 insertions(+), 35 deletions(-)
create mode 100644 README.tunables
create mode 100644 elf/dl-tunable-types.h
create mode 100644 elf/dl-tunables.c
create mode 100644 elf/dl-tunables.h
create mode 100644 elf/dl-tunables.list
create mode 100644 malloc/tst-malloc-usable-static-tunables.c
create mode 100644 malloc/tst-malloc-usable-static.c
create mode 100644 malloc/tst-malloc-usable-tunables.c
create mode 100644 scripts/gen-tunables.awk
--
2.7.4
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable
2016-08-15 20:05 [PATCHv4 0/2] tunables for glibc Siddhesh Poyarekar
@ 2016-08-15 20:05 ` Siddhesh Poyarekar
2016-08-15 20:05 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
` (2 subsequent siblings)
3 siblings, 0 replies; 9+ messages in thread
From: Siddhesh Poyarekar @ 2016-08-15 20:05 UTC (permalink / raw)
To: libc-alpha; +Cc: carlos, fweimer
Read tunables values from the users using the GLIBC_TUNABLES
environment variable. The value of this variable is a colon-separated
list of name=value pairs. So a typical string would look like this:
GLIBC_TUNABLES=glibc.malloc.mmap_threshold=2048:glibc.malloc.trim_threshold=1024
* elf/dl-tunables.c: Include mman.h and libc-internals.h.
(tunables_strdup): New function.
(parse_tunables): New function.
(GLIBC_TUNABLES): New macro.
(__tunables_init): Use the new functions and macro.
* malloc/tst-malloc-usable-tunables.c: New test case.
* malloc/tst-malloc-usable-static-tunables.c: New test case.
* malloc/Makefile (tests, tests-static): Add tests.
---
elf/dl-tunables.c | 99 ++++++++++++++++++++++++++++++
malloc/Makefile | 6 +-
malloc/tst-malloc-usable-static-tunables.c | 1 +
malloc/tst-malloc-usable-tunables.c | 1 +
4 files changed, 105 insertions(+), 2 deletions(-)
create mode 100644 malloc/tst-malloc-usable-static-tunables.c
create mode 100644 malloc/tst-malloc-usable-tunables.c
diff --git a/elf/dl-tunables.c b/elf/dl-tunables.c
index b8f8fc0..ff71e9d 100644
--- a/elf/dl-tunables.c
+++ b/elf/dl-tunables.c
@@ -22,10 +22,14 @@
#include <stdbool.h>
#include <unistd.h>
#include <stdlib.h>
+#include <sys/mman.h>
+#include <libc-internal.h>
#define TUNABLES_INTERNAL 1
#include "dl-tunables.h"
+#define GLIBC_TUNABLES "GLIBC_TUNABLES"
+
/* Compare environment names, bounded by the name hardcoded in glibc. */
static bool
is_name (const char *orig, const char *envname)
@@ -41,6 +45,27 @@ is_name (const char *orig, const char *envname)
return false;
}
+static char *tunables_strdup (const char *in)
+{
+ size_t i = 0;
+
+ while (in[i++]);
+
+ char *out = __mmap (NULL, ALIGN_UP (i, __getpagesize ()),
+ PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1,
+ 0);
+
+ if (out == MAP_FAILED)
+ return NULL;
+
+ i--;
+
+ while (i-- > 0)
+ out[i] = in[i];
+
+ return out;
+}
+
static char **
get_next_env (char **envp, char **name, size_t *namelen, char **val)
{
@@ -109,6 +134,72 @@ tunable_initialize (tunable_t *cur, const char *strval)
}
}
+static void
+parse_tunables (char *tunestr)
+{
+ if (tunestr == NULL || *tunestr == '\0')
+ return;
+
+ char *p = tunestr;
+
+ while (true)
+ {
+ char *name = p;
+ size_t len = 0;
+
+ /* First, find where the name ends. */
+ while (p[len] != '=' && p[len] != ':' && p[len] != '\0')
+ len++;
+
+ /* If we reach the end of the string before getting a valid name-value
+ pair, bail out. */
+ if (p[len] == '\0')
+ return;
+
+ /* We did not find a valid name-value pair before encountering the
+ colon. */
+ if (p[len]== ':')
+ {
+ p += len + 1;
+ continue;
+ }
+
+ p += len + 1;
+
+ char *value = p;
+ len = 0;
+
+ while (p[len] != ':' && p[len] != '\0')
+ len++;
+
+ char end = p[len];
+ p[len] = '\0';
+
+ /* Add the tunable if it exists. */
+ for (size_t i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
+ {
+ tunable_t *cur = &tunable_list[i];
+
+ /* If we are in a secure context (AT_SECURE) then ignore the tunable
+ unless it is explicitly marked as secure. Tunable values take
+ precendence over their envvar aliases. */
+ if (__libc_enable_secure && !cur->is_secure)
+ continue;
+
+ if (is_name (cur->name, name))
+ {
+ tunable_initialize (cur, value);
+ break;
+ }
+ }
+
+ if (end == ':')
+ p += len + 1;
+ else
+ return;
+ }
+}
+
/* Initialize the tunables list from the environment. For now we only use the
ENV_ALIAS to find values. Later we will also use the tunable names to find
values. */
@@ -121,6 +212,14 @@ __tunables_init (char **envp)
while ((envp = get_next_env (envp, &envname, &len, &envval)) != NULL)
{
+ if (is_name (GLIBC_TUNABLES, envname))
+ {
+ char *val = tunables_strdup (envval);
+ if (val != NULL)
+ parse_tunables (val);
+ continue;
+ }
+
for (int i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
{
tunable_t *cur = &tunable_list[i];
diff --git a/malloc/Makefile b/malloc/Makefile
index de9aafe..599ae9d 100644
--- a/malloc/Makefile
+++ b/malloc/Makefile
@@ -30,8 +30,8 @@ tests := mallocbug tst-malloc tst-valloc tst-calloc tst-obstack \
tst-pvalloc tst-memalign tst-mallopt tst-scratch_buffer \
tst-malloc-backtrace tst-malloc-thread-exit \
tst-malloc-thread-fail tst-malloc-fork-deadlock \
- tst-mallocfork2
-tests-static := tst-malloc-usable-static
+ tst-mallocfork2 tst-malloc-usable-tunables
+tests-static := tst-malloc-usable-static tst-malloc-usable-static-tunables
tests += $(tests-static)
test-srcs = tst-mtrace
@@ -141,6 +141,8 @@ endif
tst-mcheck-ENV = MALLOC_CHECK_=3
tst-malloc-usable-ENV = MALLOC_CHECK_=3
tst-malloc-usable-static-ENV = $(tst-malloc-usable-ENV)
+tst-malloc-usable-tunables-ENV = GLIBC_TUNABLES=glibc.malloc.check=3
+tst-malloc-usable-static-tunables-ENV = $(tst-malloc-usable-tunables-ENV)
# Uncomment this for test releases. For public releases it is too expensive.
#CPPFLAGS-malloc.o += -DMALLOC_DEBUG=1
diff --git a/malloc/tst-malloc-usable-static-tunables.c b/malloc/tst-malloc-usable-static-tunables.c
new file mode 100644
index 0000000..8907db0
--- /dev/null
+++ b/malloc/tst-malloc-usable-static-tunables.c
@@ -0,0 +1 @@
+#include <malloc/tst-malloc-usable.c>
diff --git a/malloc/tst-malloc-usable-tunables.c b/malloc/tst-malloc-usable-tunables.c
new file mode 100644
index 0000000..8907db0
--- /dev/null
+++ b/malloc/tst-malloc-usable-tunables.c
@@ -0,0 +1 @@
+#include <malloc/tst-malloc-usable.c>
--
2.7.4
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PING][PATCHv4 0/2] tunables for glibc
2016-08-15 20:05 [PATCHv4 0/2] tunables for glibc Siddhesh Poyarekar
2016-08-15 20:05 ` [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable Siddhesh Poyarekar
2016-08-15 20:05 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
@ 2016-08-23 8:08 ` Siddhesh Poyarekar
2016-08-23 19:08 ` [PATCHv4 " H.J. Lu
3 siblings, 0 replies; 9+ messages in thread
From: Siddhesh Poyarekar @ 2016-08-23 8:08 UTC (permalink / raw)
To: libc-alpha; +Cc: carlos, fweimer
Ping!
On Tuesday 16 August 2016 01:34 AM, Siddhesh Poyarekar wrote:
> Hi,
>
> After another episode of disappearance, here is an updated iteration of the
> tunables patch with suggestions from Florian, Carlos and H. J. incorporated.
> Changes since the last version:
>
> - Fixed mallopt behaviour using a set of intermediate functions that the
> callback and __libc_mallopt use. I did not make them into direct callbacks
> because the callbacks are built only when tunables is enabled.
>
> - Loads of comment and documentation fixes based on Carlos' and Florian's
> suggestions
>
> - Call __init_tunables even earlier to ensure that tunables are initialized
> before get_cpu_features is called.
>
> Siddhesh Poyarekar (2):
> Add framework for tunables
> Initialize tunable list with the GLIBC_TUNABLES environment variable
>
> INSTALL | 5 +
> Makeconfig | 16 ++
> README.tunables | 84 +++++++++
> config.h.in | 3 +
> config.make.in | 1 +
> configure | 16 ++
> configure.ac | 10 +
> csu/init-first.c | 7 +
> elf/Makefile | 5 +
> elf/Versions | 3 +
> elf/dl-sysdep.c | 8 +
> elf/dl-tunable-types.h | 46 +++++
> elf/dl-tunables.c | 284 +++++++++++++++++++++++++++++
> elf/dl-tunables.h | 78 ++++++++
> elf/dl-tunables.list | 69 +++++++
> malloc/Makefile | 7 +-
> malloc/arena.c | 54 ++++++
> malloc/malloc.c | 126 +++++++++----
> malloc/tst-malloc-usable-static-tunables.c | 1 +
> malloc/tst-malloc-usable-static.c | 1 +
> malloc/tst-malloc-usable-tunables.c | 1 +
> manual/install.texi | 5 +
> scripts/gen-tunables.awk | 157 ++++++++++++++++
> sysdeps/mach/hurd/dl-sysdep.c | 8 +
> sysdeps/mach/hurd/i386/init-first.c | 8 +
> 25 files changed, 968 insertions(+), 35 deletions(-)
> create mode 100644 README.tunables
> create mode 100644 elf/dl-tunable-types.h
> create mode 100644 elf/dl-tunables.c
> create mode 100644 elf/dl-tunables.h
> create mode 100644 elf/dl-tunables.list
> create mode 100644 malloc/tst-malloc-usable-static-tunables.c
> create mode 100644 malloc/tst-malloc-usable-static.c
> create mode 100644 malloc/tst-malloc-usable-tunables.c
> create mode 100644 scripts/gen-tunables.awk
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCHv4 0/2] tunables for glibc
2016-08-15 20:05 [PATCHv4 0/2] tunables for glibc Siddhesh Poyarekar
` (2 preceding siblings ...)
2016-08-23 8:08 ` [PING][PATCHv4 0/2] tunables for glibc Siddhesh Poyarekar
@ 2016-08-23 19:08 ` H.J. Lu
2016-09-07 6:25 ` Siddhesh Poyarekar
3 siblings, 1 reply; 9+ messages in thread
From: H.J. Lu @ 2016-08-23 19:08 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library, Carlos O'Donell, Florian Weimer
On Mon, Aug 15, 2016 at 1:04 PM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> Hi,
>
> After another episode of disappearance, here is an updated iteration of the
> tunables patch with suggestions from Florian, Carlos and H. J. incorporated.
> Changes since the last version:
>
> - Fixed mallopt behaviour using a set of intermediate functions that the
> callback and __libc_mallopt use. I did not make them into direct callbacks
> because the callbacks are built only when tunables is enabled.
>
> - Loads of comment and documentation fixes based on Carlos' and Florian's
> suggestions
>
> - Call __init_tunables even earlier to ensure that tunables are initialized
> before get_cpu_features is called.
>
__tunables_init is called too late for static executables:
Breakpoint 2, __libc_start_main (main=0x400400 <main>, argc=1,
argv=0x7fffffffde18, init=0x401d10 <__libc_csu_init>,
fini=0x401da0 <__libc_csu_fini>, rtld_fini=0x0, stack_end=0x7fffffffde08)
at ../sysdeps/x86/libc-start.c:37
37 init_cpu_features (&_dl_x86_cpu_features);
(gdb) bt
#0 __libc_start_main (main=0x400400 <main>, argc=1, argv=0x7fffffffde18,
init=0x401d10 <__libc_csu_init>, fini=0x401da0 <__libc_csu_fini>,
rtld_fini=0x0, stack_end=0x7fffffffde08) at ../sysdeps/x86/libc-start.c:37
#1 0x0000000000400aea in _start () at ../sysdeps/x86_64/start.S:120
(gdb) c
Continuing.
Breakpoint 3, __tunables_init (envp=envp@entry=0x7fffffffde28)
at dl-tunables.c:208
208 {
(gdb) bt
#0 __tunables_init (envp=envp@entry=0x7fffffffde28) at dl-tunables.c:208
#1 0x0000000000447ceb in __libc_init_first (argc=argc@entry=1,
argv=argv@entry=0x7fffffffde18, envp=0x7fffffffde28)
at ../csu/init-first.c:81
#2 0x00000000004013f8 in generic_start_main (main=main@entry=0x400400 <main>,
argc=argc@entry=1, argv=argv@entry=0x7fffffffde18,
init=init@entry=0x401d10 <__libc_csu_init>,
fini=fini@entry=0x401da0 <__libc_csu_fini>, rtld_fini=rtld_fini@entry=0x0,
stack_end=0x7fffffffde08) at ../csu/libc-start.c:225
#3 0x0000000000401702 in __libc_start_main (main=0x400400 <main>, argc=1,
argv=0x7fffffffde18, init=0x401d10 <__libc_csu_init>,
fini=0x401da0 <__libc_csu_fini>, rtld_fini=0x0, stack_end=0x7fffffffde08)
at ../sysdeps/x86/libc-start.c:38
#4 0x0000000000400aea in _start () at ../sysdeps/x86_64/start.S:120
(gdb)
It needs to be called before init_cpu_features on x86.
--
H.J.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCHv4 0/2] tunables for glibc
2016-08-23 19:08 ` [PATCHv4 " H.J. Lu
@ 2016-09-07 6:25 ` Siddhesh Poyarekar
2016-09-07 14:45 ` H.J. Lu
0 siblings, 1 reply; 9+ messages in thread
From: Siddhesh Poyarekar @ 2016-09-07 6:25 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library, Carlos O'Donell, Florian Weimer
On Wednesday 24 August 2016 12:37 AM, H.J. Lu wrote:
> It needs to be called before init_cpu_features on x86.
I've been thinking about this and it seems that init_cpu_features is
called too early and I can't call __tunables_init any earlier without
impacting what it can read, i.e. __libc_enable_secure for example.
So one approach to solving this could be to find a place in init-first
where init_cpu_features can be called early enough and still later than
__tunables_init. I haven't done a thorough analysis but it seems it can
be done.
However a cleaner approach seems to me to be to keep the cpu features
stuff separate from the overriding and then modify the ifunc resolvers
to use the overrides if necessary. That is, let the cpu features show
what the cpu is actually capable of and have a separate overriding
structure that is initialized later (via __init_tunables or before ifunc
resolution) and used when necessary, i.e. not used for AT_SECURE
executables or any other condition that may arise in future.
Siddhesh
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCHv4 0/2] tunables for glibc
2016-09-07 6:25 ` Siddhesh Poyarekar
@ 2016-09-07 14:45 ` H.J. Lu
2016-09-07 14:51 ` Florian Weimer
2016-09-10 15:30 ` Siddhesh Poyarekar
0 siblings, 2 replies; 9+ messages in thread
From: H.J. Lu @ 2016-09-07 14:45 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library, Carlos O'Donell, Florian Weimer
On Tue, Sep 6, 2016 at 11:25 PM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> On Wednesday 24 August 2016 12:37 AM, H.J. Lu wrote:
>> It needs to be called before init_cpu_features on x86.
>
> I've been thinking about this and it seems that init_cpu_features is
> called too early and I can't call __tunables_init any earlier without
> impacting what it can read, i.e. __libc_enable_secure for example.
>
> So one approach to solving this could be to find a place in init-first
> where init_cpu_features can be called early enough and still later than
> __tunables_init. I haven't done a thorough analysis but it seems it can
> be done.
>
> However a cleaner approach seems to me to be to keep the cpu features
> stuff separate from the overriding and then modify the ifunc resolvers
> to use the overrides if necessary. That is, let the cpu features show
> what the cpu is actually capable of and have a separate overriding
> structure that is initialized later (via __init_tunables or before ifunc
> resolution) and used when necessary, i.e. not used for AT_SECURE
> executables or any other condition that may arise in future.
The whole point of cpu features is it is initialized as early as possible
so that it can be used for ifunc. I don't think __init_tunables is suitable
for ifunc, which doesn't have any security implications on x86.
--
H.J.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCHv4 0/2] tunables for glibc
2016-09-07 14:45 ` H.J. Lu
@ 2016-09-07 14:51 ` Florian Weimer
2016-09-10 15:30 ` Siddhesh Poyarekar
1 sibling, 0 replies; 9+ messages in thread
From: Florian Weimer @ 2016-09-07 14:51 UTC (permalink / raw)
To: H.J. Lu, Siddhesh Poyarekar; +Cc: GNU C Library, Carlos O'Donell
On 09/07/2016 04:44 PM, H.J. Lu wrote:
> The whole point of cpu features is it is initialized as early as possible
> so that it can be used for ifunc. I don't think __init_tunables is suitable
> for ifunc, which doesn't have any security implications on x86.
Disabling CPU features for crypto acceleration may introduce timing
attacks and information leakage (especially if the fallback
implementation uses lookup tables).
(It's not relevant for glibc at present, I think.)
Florian
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCHv4 0/2] tunables for glibc
2016-09-07 14:45 ` H.J. Lu
2016-09-07 14:51 ` Florian Weimer
@ 2016-09-10 15:30 ` Siddhesh Poyarekar
1 sibling, 0 replies; 9+ messages in thread
From: Siddhesh Poyarekar @ 2016-09-10 15:30 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library, Carlos O'Donell, Florian Weimer
On Wednesday 07 September 2016 08:14 PM, H.J. Lu wrote:
>> However a cleaner approach seems to me to be to keep the cpu features
>> stuff separate from the overriding and then modify the ifunc resolvers
>> to use the overrides if necessary. That is, let the cpu features show
>> what the cpu is actually capable of and have a separate overriding
>> structure that is initialized later (via __init_tunables or before ifunc
>> resolution) and used when necessary, i.e. not used for AT_SECURE
>> executables or any other condition that may arise in future.
>
> The whole point of cpu features is it is initialized as early as possible
> so that it can be used for ifunc. I don't think __init_tunables is suitable
> for ifunc, which doesn't have any security implications on x86.
It is fine that cpu features (and its overrides) be initialized before
ifunc. My question is whether we can delay ifunc (and hence cpu
features initialization) enough that tunables can influence them. Not
only that, ifunc delaying may help other things, like the ability to use
more functions in the resolver to do more complicated things, especially
when they're used outside of glibc.
The latter seems to have use cases, but it is not clear to me yet how
feasible it is.
Siddhesh
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-09-10 15:30 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-15 20:05 [PATCHv4 0/2] tunables for glibc Siddhesh Poyarekar
2016-08-15 20:05 ` [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable Siddhesh Poyarekar
2016-08-15 20:05 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-08-23 8:08 ` [PING][PATCHv4 0/2] tunables for glibc Siddhesh Poyarekar
2016-08-23 19:08 ` [PATCHv4 " H.J. Lu
2016-09-07 6:25 ` Siddhesh Poyarekar
2016-09-07 14:45 ` H.J. Lu
2016-09-07 14:51 ` Florian Weimer
2016-09-10 15:30 ` Siddhesh Poyarekar
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).