* [PATCH 1/2] Add framework for tunables
2016-07-02 17:13 [PATCH 0/2] tunables for glibc Siddhesh Poyarekar
@ 2016-07-02 17:13 ` Siddhesh Poyarekar
2016-07-03 14:30 ` H.J. Lu
` (2 more replies)
2016-07-02 17:13 ` [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable Siddhesh Poyarekar
2016-07-03 0:24 ` [PATCH 0/2] tunables for glibc H.J. Lu
2 siblings, 3 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-02 17:13 UTC (permalink / raw)
To: libc-alpha
The tunables framework allows us to uniformly manage and expose global
variables inside glibc as switches to users. README.tunables has
instructions for glibc developers to add new tunables.
Tunables support can be enabled by passing the --enable-tunables
configure flag to the configure script. This patch only adds a
framework and does not pose any limitations on how tunable values are
read from the user. It also adds environment variables used in malloc
behaviour tweaking to the tunables framework as a PoC of the
compatibility interface.
* manual/install.texi: Add --enable-tunables option.
* INSTALL: Regenerate.
* Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
(before-compile): Generate dl-tunable-list.h early.
* config.h.in: Add BUILD_TUNABLES.
* config.make.in: Add build-tunables.
* configure.ac: Add --enable-tunables option.
* configure: Regenerate.
* malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
Define TUNABLE_NAMESPACE.
(DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
(ptmalloc_init): Set tunable values.
* malloc/tst-malloc-usable-static.c: New test case.
* csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
(__libc_init_first) [!SHARED]: Initialize tunables for static
binaries.
* scripts/gen-tunables.awk: New file.
* README.tunables: New file.
* elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
namespace.
* elf/dl-tunable-list.h: New auto-generated file.
* elf/dl-tunables.c: New file.
* elf/dl-tunables.h: New file.
* elf/dl-tunables.list: New file.
* elf/dl-tunable-types.h: New file.
* elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
(process_envvars): Call __tunables_init.
---
INSTALL | 6 ++
Makeconfig | 16 ++++
README.tunables | 74 ++++++++++++++++
config.h.in | 3 +
config.make.in | 1 +
configure | 16 ++++
configure.ac | 10 +++
csu/init-first.c | 7 ++
elf/Makefile | 5 ++
elf/Versions | 3 +
elf/dl-tunable-types.h | 45 ++++++++++
elf/dl-tunables.c | 172 ++++++++++++++++++++++++++++++++++++++
elf/dl-tunables.h | 76 +++++++++++++++++
elf/dl-tunables.list | 50 +++++++++++
elf/rtld.c | 8 ++
malloc/Makefile | 3 +
malloc/arena.c | 35 ++++++++
malloc/tst-malloc-usable-static.c | 1 +
manual/install.texi | 5 ++
scripts/gen-tunables.awk | 157 ++++++++++++++++++++++++++++++++++
20 files changed, 693 insertions(+)
create mode 100644 README.tunables
create mode 100644 elf/dl-tunable-types.h
create mode 100644 elf/dl-tunables.c
create mode 100644 elf/dl-tunables.h
create mode 100644 elf/dl-tunables.list
create mode 100644 malloc/tst-malloc-usable-static.c
create mode 100644 scripts/gen-tunables.awk
diff --git a/INSTALL b/INSTALL
index ec3445f..90dc481 100644
--- a/INSTALL
+++ b/INSTALL
@@ -158,6 +158,12 @@ will be used, and CFLAGS sets optimization options for the compiler.
By default for x86_64, the GNU C Library is built with vector math
library. Use this option to disable vector math library.
+'--enable-tunables'
+ Tunables support allows additional library parameters to be
+ customized at runtime for each application. This is an
+ experimental feature and affects startup time and is thus disabled
+ by default.
+
'--build=BUILD-SYSTEM'
'--host=HOST-SYSTEM'
These options are for cross-compiling. If you specify both options
diff --git a/Makeconfig b/Makeconfig
index 901e253..70ab097 100644
--- a/Makeconfig
+++ b/Makeconfig
@@ -883,6 +883,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \
$(foreach lib,$(libof-$(basename $(@F))) \
$(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \
$(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F)))
+
+ifeq (yes,$(build-tunables))
+CPPFLAGS += -DTOP_NAMESPACE=glibc
+endif
+
override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \
$(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \
$(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \
@@ -1054,6 +1059,17 @@ $(common-objpfx)libc-modules.stmp: $(..)scripts/gen-libc-modules.awk \
endif
+# Build the tunables list header early since it could be used by any module in
+# glibc.
+ifeq (yes,$(build-tunables))
+before-compile += $(common-objpfx)dl-tunable-list.h
+
+$(common-objpfx)dl-tunable-list.h: $(..)scripts/gen-tunables.awk \
+ $(..)elf/dl-tunables.list
+ $(AWK) -f $^ > $@.tmp
+ mv $@{.tmp,}
+endif
+
common-generated += libc-modules.h libc-modules.stmp
# The name under which the run-time dynamic linker is installed.
diff --git a/README.tunables b/README.tunables
new file mode 100644
index 0000000..8890249
--- /dev/null
+++ b/README.tunables
@@ -0,0 +1,74 @@
+ TUNABLE FRAMEWORK
+ =================
+
+The tunable framework allows modules within glibc to register variables that
+may be tweaked through an environment variable or an API call. It aims to
+enforce a strict namespace rule to bring consistency to naming of these tunable
+environment variables across the project.
+
+ADDING A NEW TUNABLE
+--------------------
+
+The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in
+distributions for specific tunables if they want to add their own tunables.
+Downstream implementations are discouraged from using the 'glibc' namespace for
+tunables they don't already have consensus to push upstream.
+
+There are two steps to adding a tunable:
+
+1. Add a tunable ID:
+
+Modules that wish to use the tunables interface must define the
+TUNABLE_NAMESPACE macro. Following this, for each tunable you want to
+add, make an entry in elf/dl-tunables.list. The format of the file is as
+follows:
+
+TOP_NAMESPACE {
+ NAMESPACE1 {
+ TUNABLE1 {
+ # tunable attributes, one per line
+ }
+ # A tunable with default attributes, i.e. string variable.
+ TUNABLE2
+ TUNABLE3 {
+ # its attributes
+ }
+ }
+ NAMESPACE2 {
+ ...
+ }
+}
+
+The list of allowed attributes are:
+
+- type: Data type. Defaults to STRING. Allowed types are:
+ INT_32, SIZE_T. Note that as of now the STRING type is
+ neither defined nor supported since no tunables need
+ it. It will be in future though, when needed.
+
+- minval: Optional minimum acceptable value
+
+- maxval: Optional maximum acceptable value
+
+- env_alias: An alias environment variable
+
+- secure_env_alias: Specify whether the environment variable should be read
+ for setuid binaries.
+
+2. Call either the TUNABLE_SET_VALUE and pass into it the tunable name and a
+ pointer to the variable that should be set with the tunable value.
+ If additional work needs to be done after setting the value, use the
+ TUNABLE_SET_VALUE_WITH_CALLBACK instead and additionally pass a pointer to
+ the function that should be called if the tunable value has been set.
+
+FUTURE WORK
+-----------
+
+The framework currently only allows a one-time initialization of variables
+through environment variables and in some cases, modification of variables via
+an API call. A future goals for this project include:
+
+- Setting system-wide and user-wide defaults for tunables through some
+ mechanism like a configuration file.
+
+- Allow tweaking of some tunables at runtime
diff --git a/config.h.in b/config.h.in
index 856ef6a..6112782 100644
--- a/config.h.in
+++ b/config.h.in
@@ -240,4 +240,7 @@
/* PowerPC32 uses fctidz for floating point to long long conversions. */
#define HAVE_PPC_FCTIDZ 0
+/* Build glibc with tunables support. */
+#define BUILD_TUNABLES 0
+
#endif
diff --git a/config.make.in b/config.make.in
index 95c6f36..653ef6f 100644
--- a/config.make.in
+++ b/config.make.in
@@ -91,6 +91,7 @@ use-nscd = @use_nscd@
build-hardcoded-path-in-tests= @hardcoded_path_in_tests@
build-pt-chown = @build_pt_chown@
enable-lock-elision = @enable_lock_elision@
+build-tunables = @build_tunables@
# Build tools.
CC = @CC@
diff --git a/configure b/configure
index 19a4829..1534ba7 100755
--- a/configure
+++ b/configure
@@ -659,6 +659,7 @@ multi_arch
base_machine
add_on_subdirs
add_ons
+build_tunables
build_pt_chown
build_nscd
link_obsolete_rpc
@@ -773,6 +774,7 @@ enable_systemtap
enable_build_nscd
enable_nscd
enable_pt_chown
+enable_tunables
enable_mathvec
with_cpu
'
@@ -1440,6 +1442,7 @@ Optional Features:
--disable-build-nscd disable building and installing the nscd daemon
--disable-nscd library functions will not contact the nscd daemon
--enable-pt_chown Enable building and installing pt_chown
+ --enable-tunables Enable tunables support
--enable-mathvec Enable building and installing mathvec [default
depends on architecture]
@@ -3646,6 +3649,19 @@ if test "$build_pt_chown" = yes; then
fi
+# Check whether --enable-tunables was given.
+if test "${enable_tunables+set}" = set; then :
+ enableval=$enable_tunables; build_tunables=$enableval
+else
+ build_tunables=no
+fi
+
+
+if test "$build_tunables" = yes; then
+ $as_echo "#define BUILD_TUNABLES 1" >>confdefs.h
+
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/configure.ac b/configure.ac
index 123f0d2..b2aaf9b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -395,6 +395,16 @@ if test "$build_pt_chown" = yes; then
AC_DEFINE(HAVE_PT_CHOWN)
fi
+AC_ARG_ENABLE([tunables],
+ [AS_HELP_STRING([--enable-tunables],
+ [Enable tunables support])],
+ [build_tunables=$enableval],
+ [build_tunables=no])
+AC_SUBST(build_tunables)
+if test "$build_tunables" = yes; then
+ AC_DEFINE(BUILD_TUNABLES)
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/csu/init-first.c b/csu/init-first.c
index 77c6e1c..7427121 100644
--- a/csu/init-first.c
+++ b/csu/init-first.c
@@ -28,6 +28,9 @@
#include <libc-internal.h>
#include <ldsodefs.h>
+#if BUILD_TUNABLES
+# include <elf/dl-tunables.h>
+#endif
/* Set nonzero if we have to be prepared for more than one libc being
used in the process. Safe assumption if initializer never runs. */
@@ -74,6 +77,10 @@ _init (int argc, char **argv, char **envp)
#ifndef SHARED
__libc_init_secure ();
+#if BUILD_TUNABLES
+ __tunables_init (envp);
+#endif
+
/* First the initialization which normally would be done by the
dynamic linker. */
_dl_non_dynamic_init ();
diff --git a/elf/Makefile b/elf/Makefile
index 210dde9..c634d120 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -35,6 +35,11 @@ dl-routines = $(addprefix dl-,load lookup object reloc deps hwcaps \
ifeq (yes,$(use-ldconfig))
dl-routines += dl-cache
endif
+
+ifeq (yes,$(build-tunables))
+dl-routines += dl-tunables
+endif
+
all-dl-routines = $(dl-routines) $(sysdep-dl-routines)
# But they are absent from the shared libc, because that code is in ld.so.
elide-routines.os = $(all-dl-routines) dl-support enbl-secure dl-origin \
diff --git a/elf/Versions b/elf/Versions
index 23deda9..1734697 100644
--- a/elf/Versions
+++ b/elf/Versions
@@ -64,5 +64,8 @@ ld {
# Pointer protection.
__pointer_chk_guard;
+
+ # Set value of a tunable.
+ __tunable_set_val;
}
}
diff --git a/elf/dl-tunable-types.h b/elf/dl-tunable-types.h
new file mode 100644
index 0000000..ecbb197
--- /dev/null
+++ b/elf/dl-tunable-types.h
@@ -0,0 +1,45 @@
+/* Tunable type information.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLE_TYPES_H_
+# define _TUNABLE_TYPES_H_
+#include <stddef.h>
+
+typedef void (*tunable_callback_t) (void);
+
+typedef enum
+{
+ TUNABLE_TYPE_INT_32,
+ TUNABLE_TYPE_SIZE_T
+} tunable_type_code_t;
+
+typedef struct
+{
+ tunable_type_code_t type_code;
+ int64_t min;
+ int64_t max;
+} tunable_type_t;
+
+typedef union
+{
+ int64_t numval;
+ const char *strval;
+} tunable_val_t;
+
+#endif
diff --git a/elf/dl-tunables.c b/elf/dl-tunables.c
new file mode 100644
index 0000000..8a26dfc
--- /dev/null
+++ b/elf/dl-tunables.c
@@ -0,0 +1,172 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <stdlib.h>
+
+#define TUNABLES_INTERNAL 1
+#include "dl-tunables.h"
+
+/* Compare environment names, bounded by the name hardcoded in glibc. */
+static bool
+is_name (const char *orig, const char *envname)
+{
+ for (;*orig != '\0' && *envname != '\0'; envname++, orig++)
+ if (*orig != *envname)
+ break;
+
+ /* The ENVNAME is immediately followed by a value. */
+ if (*orig == '\0' && *envname == '=')
+ return true;
+ else
+ return false;
+}
+
+static bool
+get_next_env (char ***envp, char **name, size_t *namelen, char **val)
+{
+ char **ev = *envp;
+
+ while (ev != NULL && *ev != '\0')
+ {
+ char *envline = *ev;
+ int len = 0;
+
+ while (envline[len] != '\0' && envline[len] != '=')
+ len++;
+
+ /* Just the name and no value, go to the next one. */
+ if (envline[len] == '\0')
+ continue;
+
+ *name = envline;
+ *namelen = len;
+ *val = &envline[len + 1];
+ *envp = ++ev;
+
+ return true;
+ }
+
+ return false;
+}
+
+
+#define RETURN_IF_INVALID_RANGE(__cur, __val) \
+({ \
+ __typeof ((__cur)->type.min) min = (__cur)->type.min; \
+ __typeof ((__cur)->type.max) max = (__cur)->type.max; \
+ if (min != max && ((__val) < min || (__val) > max)) \
+ return; \
+})
+
+/* Validate range of the input value and initialize the tunable CUR if it looks
+ good. */
+static void
+tunable_initialize (tunable_t *cur)
+{
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ int32_t val = (int32_t) __strtoul_internal (cur->strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ size_t val = (size_t) __strtoul_internal (cur->strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+}
+
+/* Initialize the tunables list from the environment. For now we only use the
+ ENV_ALIAS to find values. Later we will also use the tunable names to find
+ values. */
+void
+__tunables_init (char **envp)
+{
+ char *envname = NULL;
+ char *envval = NULL;
+ size_t len = 0;
+
+ while (get_next_env (&envp, &envname, &len, &envval))
+ {
+ for (int i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
+ {
+ tunable_t *cur = &tunable_list[i];
+
+ /* Skip over tunables that have either been set already or should be
+ skipped. */
+ if (cur->strval != NULL || cur->env_alias == NULL
+ || (__libc_enable_secure && !cur->secure_env_alias))
+ continue;
+
+ const char *name = cur->env_alias;
+
+ /* We have a match. Initialize and move on to the next line. */
+ if (is_name (name, envname))
+ {
+ cur->strval = envval;
+ tunable_initialize (cur);
+ break;
+ }
+ }
+ }
+}
+
+/* Set the tunable value. This is called by the module that the tunable exists
+ in. */
+void
+__tunable_set_val (tunable_id_t id, void *valp, void (*callback) (void))
+{
+ tunable_t *cur = &tunable_list[id];
+
+ /* Sanity check: don't do anything if our tunable was not set during
+ initialization. */
+ if (cur->strval == NULL)
+ return;
+
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ *((int32_t *) valp) = (int32_t) cur->val.numval;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ *((size_t *) valp) = (size_t) cur->val.numval;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+
+ if (callback)
+ callback ();
+}
diff --git a/elf/dl-tunables.h b/elf/dl-tunables.h
new file mode 100644
index 0000000..93bcad7
--- /dev/null
+++ b/elf/dl-tunables.h
@@ -0,0 +1,76 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLES_H_
+# define _TUNABLES_H_
+# include <stddef.h>
+# include "dl-tunable-types.h"
+
+/* A tunable. */
+struct _tunable
+{
+ const char *name; /* Internal name of the tunable. */
+ tunable_type_t type; /* Data type of the tunable. */
+ tunable_val_t val; /* The value. */
+ const char *strval; /* The string containing the value,
+ points into envp. */
+ /* Compatibility elements. */
+ const char *env_alias; /* The compatibility environment
+ variable name. */
+ bool secure_env_alias; /* Whether the environment variable
+ must be read even for setuid
+ binaries. */
+};
+
+typedef struct _tunable tunable_t;
+
+/* Full name for a tunable is top_ns.tunable_ns.id. */
+#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id
+
+# define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)
+# define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id
+
+#include "dl-tunable-list.h"
+
+extern void __tunables_init (char **);
+extern void __tunable_set_val (tunable_id_t, void *, void (*) (void));
+
+/* Check if the tunable has been set to a non-default value and if it is, copy
+ it over into __VAL. */
+# define TUNABLE_SET_VAL(__id,__val) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ NULL); \
+})
+
+/* Same as TUNABLE_SET_VAL, but also set the callback function to __CB and call
+ it. */
+# define TUNABLE_SET_VAL_WITH_CALLBACK(__id,__val,__cb) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ DL_TUNABLE_CALLBACK (__cb)); \
+})
+
+/* Namespace sanity for callback functions. Use this macro to keep the
+ namespace of the modules clean. */
+#define DL_TUNABLE_CALLBACK(__name) _dl_tunable_ ## __name
+#endif
diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list
new file mode 100644
index 0000000..342e3d5
--- /dev/null
+++ b/elf/dl-tunables.list
@@ -0,0 +1,50 @@
+# Allowed attributes for tunables:
+#
+# type: Defaults to STRING
+# minval: Optional minimum acceptable value
+# maxval: Optional maximum acceptable value
+# env_alias: An alias environment variable
+# secure_env_alias: Specify whether the environment variable should be read for
+# setuid binaries.
+
+glibc {
+ malloc {
+ check {
+ type: INT_32
+ minval: 0
+ maxval: 3
+ env_alias: MALLOC_CHECK_
+ secure_env_alias: true
+ }
+ top_pad {
+ type: SIZE_T
+ env_alias: MALLOC_TOP_PAD_
+ }
+ perturb {
+ type: INT_32
+ minval: 0
+ maxval: 0xff
+ env_alias: MALLOC_PERTURB_
+ }
+ mmap_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_MMAP_THRESHOLD_
+ }
+ trim_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_TRIM_THRESHOLD_
+ }
+ mmap_max {
+ type: INT_32
+ env_alias: MALLOC_MMAP_MAX_
+ }
+ arena_max {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_MAX
+ }
+ arena_test {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_TEST
+ }
+ }
+}
diff --git a/elf/rtld.c b/elf/rtld.c
index 647661c..263723a 100644
--- a/elf/rtld.c
+++ b/elf/rtld.c
@@ -44,6 +44,10 @@
#include <assert.h>
+#if BUILD_TUNABLES
+# include <dl-tunables.h>
+#endif
+
/* Avoid PLT use for our local calls at startup. */
extern __typeof (__mempcpy) __mempcpy attribute_hidden;
@@ -2346,6 +2350,10 @@ process_envvars (enum mode *modep)
enum mode mode = normal;
char *debug_output = NULL;
+#if BUILD_TUNABLES
+ __tunables_init (_environ);
+#endif
+
/* This is the default place for profiling data file. */
GLRO(dl_profile_output)
= &"/var/tmp\0/var/profile"[__libc_enable_secure ? 9 : 0];
diff --git a/malloc/Makefile b/malloc/Makefile
index fa1730e..34816f2 100644
--- a/malloc/Makefile
+++ b/malloc/Makefile
@@ -31,6 +31,8 @@ tests := mallocbug tst-malloc tst-valloc tst-calloc tst-obstack \
tst-malloc-backtrace tst-malloc-thread-exit \
tst-malloc-thread-fail tst-malloc-fork-deadlock \
tst-mallocfork2
+tests-static := tst-malloc-usable-static
+tests += $(tests-static)
test-srcs = tst-mtrace
routines = malloc morecore mcheck mtrace obstack \
@@ -138,6 +140,7 @@ endif
tst-mcheck-ENV = MALLOC_CHECK_=3
tst-malloc-usable-ENV = MALLOC_CHECK_=3
+tst-malloc-usable-static-ENV = MALLOC_CHECK_=3
# Uncomment this for test releases. For public releases it is too expensive.
#CPPFLAGS-malloc.o += -DMALLOC_DEBUG=1
diff --git a/malloc/arena.c b/malloc/arena.c
index 229783f..9cc7099 100644
--- a/malloc/arena.c
+++ b/malloc/arena.c
@@ -19,6 +19,11 @@
#include <stdbool.h>
+#if BUILD_TUNABLES
+# define TUNABLE_NAMESPACE malloc
+# include <elf/dl-tunables.h>
+#endif
+
/* Compile-time constants. */
#define HEAP_MIN_SIZE (32 * 1024)
@@ -204,6 +209,15 @@ __malloc_fork_unlock_child (void)
mutex_init (&list_lock);
}
+#if BUILD_TUNABLES
+void
+DL_TUNABLE_CALLBACK (set_mallopt_check) (void)
+{
+ if (check_action != 0)
+ __malloc_check_init ();
+}
+
+#else
/* Initialization routine. */
#include <string.h>
extern char **_environ;
@@ -238,6 +252,7 @@ next_env_entry (char ***position)
return result;
}
+#endif
#ifdef SHARED
@@ -272,6 +287,24 @@ ptmalloc_init (void)
#endif
thread_arena = &main_arena;
+
+#if BUILD_TUNABLES
+ /* Ensure initialization/consolidation and do it under a lock so that a
+ thread attempting to use the arena in parallel waits on us till we
+ finish. */
+ mutex_lock (&main_arena.mutex);
+ malloc_consolidate (&main_arena);
+
+ TUNABLE_SET_VAL_WITH_CALLBACK (check, &check_action, set_mallopt_check);
+ TUNABLE_SET_VAL (top_pad, &mp_.top_pad);
+ TUNABLE_SET_VAL (perturb, &perturb_byte);
+ TUNABLE_SET_VAL (mmap_threshold, &mp_.mmap_threshold);
+ TUNABLE_SET_VAL (trim_threshold, &mp_.trim_threshold);
+ TUNABLE_SET_VAL (mmap_max, &mp_.n_mmaps_max);
+ TUNABLE_SET_VAL (arena_max, &mp_.arena_max);
+ TUNABLE_SET_VAL (arena_test, &mp_.arena_test);
+ mutex_unlock (&main_arena.mutex);
+#else
const char *s = NULL;
if (__glibc_likely (_environ != NULL))
{
@@ -340,6 +373,8 @@ ptmalloc_init (void)
if (check_action != 0)
__malloc_check_init ();
}
+#endif
+
#if HAVE_MALLOC_INIT_HOOK
void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook);
if (hook != NULL)
diff --git a/malloc/tst-malloc-usable-static.c b/malloc/tst-malloc-usable-static.c
new file mode 100644
index 0000000..8907db0
--- /dev/null
+++ b/malloc/tst-malloc-usable-static.c
@@ -0,0 +1 @@
+#include <malloc/tst-malloc-usable.c>
diff --git a/manual/install.texi b/manual/install.texi
index 79ee45f..f9b4784 100644
--- a/manual/install.texi
+++ b/manual/install.texi
@@ -189,6 +189,11 @@ configure with @option{--disable-werror}.
By default for x86_64, @theglibc{} is built with vector math library.
Use this option to disable vector math library.
+@item --enable-tunables
+Tunables support allows additional library parameters to be customized at
+runtime for each application. This is an experimental feature and affects
+startup time and is thus disabled by default.
+
@item --build=@var{build-system}
@itemx --host=@var{host-system}
These options are for cross-compiling. If you specify both options and
diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
new file mode 100644
index 0000000..1b930e7
--- /dev/null
+++ b/scripts/gen-tunables.awk
@@ -0,0 +1,157 @@
+# Generate dl-tunable-list.h from dl-tunables.list
+
+BEGIN {
+ tunable=""
+ ns=""
+ top_ns=""
+}
+
+# Skip over blank lines and comments.
+/^#/ {
+ next
+}
+
+/^[ \t]*$/ {
+ next
+}
+
+# Beginning of either a top namespace, tunable namespace or a tunable, decided
+# on the current value of TUNABLE, NS or TOP_NS.
+$2 == "{" {
+ if (top_ns == "") {
+ top_ns = $1
+ }
+ else if (ns == "") {
+ ns = $1
+ }
+ else if (tunable == "") {
+ tunable = $1
+ }
+ else {
+ printf ("Unexpected occurrence of '{': %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+
+ next
+}
+
+# End of either a top namespace, tunable namespace or a tunable.
+$1 == "}" {
+ if (tunable != "") {
+ # Tunables definition ended, now fill in default attributes.
+ if (!types[top_ns][ns][tunable]) {
+ types[top_ns][ns][tunable] = "STRING"
+ }
+ if (!minvals[top_ns][ns][tunable]) {
+ minvals[top_ns][ns][tunable] = "0"
+ }
+ if (!maxvals[top_ns][ns][tunable]) {
+ maxvals[top_ns][ns][tunable] = "0"
+ }
+ if (!env_alias[top_ns][ns][tunable]) {
+ env_alias[top_ns][ns][tunable] = "NULL"
+ }
+ if (!secure_env_alias[top_ns][ns][tunable]) {
+ secure_env_alias[top_ns][ns][tunable] = "false"
+ }
+
+ tunable = ""
+ }
+ else if (ns != "") {
+ ns = ""
+ }
+ else if (top_ns != "") {
+ top_ns = ""
+ }
+ else {
+ printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+ next
+}
+
+# Everything else, which could either be a tunable without any attributes or a
+# tunable attribute.
+{
+ if (ns == "") {
+ printf("Line %d: Invalid tunable outside a namespace: %s\n", NR, $0)
+ exit 1
+ }
+
+ if (tunable == "") {
+ # We encountered a tunable without any attributes, so note it with a
+ # default.
+ types[top_ns][ns][$1] = "STRING"
+ next
+ }
+
+ # Otherwise, we have encountered a tunable attribute.
+ split($0, arr, ":")
+ attr = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[1])
+ val = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[2])
+
+ if (attr == "type") {
+ types[top_ns][ns][tunable] = val
+ }
+ else if (attr == "minval") {
+ minvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "maxval") {
+ maxvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "env_alias") {
+ env_alias[top_ns][ns][tunable] = sprintf("\"%s\"", val)
+ }
+ else if (attr == "secure_env_alias") {
+ if (val == "true" || val == "false") {
+ secure_env_alias[top_ns][ns][tunable] = val
+ }
+ else {
+ printf("Line %d: Invalid value (%s) for secure_env_alias: %s, ", NR, val,
+ $0)
+ print("Allowed values are 'true' or 'false'")
+ exit 1
+ }
+ }
+}
+
+END {
+ if (ns != "") {
+ print "Unterminated namespace. Is a closing brace missing?"
+ exit 1
+ }
+
+ print "/* AUTOGENERATED by gen-tunables.awk. */"
+ print "#ifndef _TUNABLES_H_"
+ print "# error \"Do not include this file directly.\""
+ print "# error \"Include tunables.h instead.\""
+ print "#endif"
+
+ # Now, the enum names
+ print "\ntypedef enum"
+ print "{"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, m);
+ }
+ }
+ }
+ print "} tunable_id_t;\n"
+
+ # Finally, the tunable list.
+ print "\n#ifdef TUNABLES_INTERNAL"
+ print "static tunable_t tunable_list[] = {"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" {TUNABLE_NAME_S(%s, %s, %s)", t, n, m)
+ printf (", {TUNABLE_TYPE_%s, %s, %s}, {.numval = 0}, NULL, %s, %s},\n",
+ types[t][n][m], minvals[t][n][m], maxvals[t][n][m],
+ env_alias[t][n][m], secure_env_alias[t][n][m]);
+ }
+ }
+ }
+ print "};"
+ print "#endif"
+}
--
2.5.5
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable
2016-07-02 17:13 [PATCH 0/2] tunables for glibc Siddhesh Poyarekar
2016-07-02 17:13 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
@ 2016-07-02 17:13 ` Siddhesh Poyarekar
2016-07-03 14:39 ` H.J. Lu
2016-07-03 15:53 ` H.J. Lu
2016-07-03 0:24 ` [PATCH 0/2] tunables for glibc H.J. Lu
2 siblings, 2 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-02 17:13 UTC (permalink / raw)
To: libc-alpha
Read tunables values from the users using the GLIBC_TUNABLES
environment variable. The value of this variable is a colon-separated
list of name=value pairs. So a typical string would look like this:
GLIBC_TUNABLES=glibc.malloc.mmap_threshold=2048:glibc.malloc.trim_threshold=1024
* elf/dl-tunables.c: Include string.h.
(parse_tunables): New function.
(GLIBC_TUNABLES): New macro.
(__tunables_init): Use it.
---
elf/dl-tunables.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 69 insertions(+)
diff --git a/elf/dl-tunables.c b/elf/dl-tunables.c
index 8a26dfc..04d1d52 100644
--- a/elf/dl-tunables.c
+++ b/elf/dl-tunables.c
@@ -22,10 +22,13 @@
#include <stdbool.h>
#include <unistd.h>
#include <stdlib.h>
+#include <string.h>
#define TUNABLES_INTERNAL 1
#include "dl-tunables.h"
+#define GLIBC_TUNABLES "GLIBC_TUNABLES"
+
/* Compare environment names, bounded by the name hardcoded in glibc. */
static bool
is_name (const char *orig, const char *envname)
@@ -104,6 +107,66 @@ tunable_initialize (tunable_t *cur)
}
}
+static void
+parse_tunables (char *tunestr)
+{
+ if (tunestr == NULL || *tunestr == '\0')
+ return;
+
+ char *p = tunestr;
+
+ while (true)
+ {
+ char *name = p;
+ size_t len = 0;
+
+ /* First, find where the name ends. */
+ while (p[len] != '=' && p[len] != ':' && p[len] != '\0')
+ len++;
+
+ /* If we reach the end of the string before getting a valid name-value
+ pair, bail out. */
+ if (p[len] == '\0')
+ return;
+
+ /* We did not find a valid name-value pair before encountering the
+ colon. */
+ if (p[len]== ':')
+ {
+ p += len + 1;
+ continue;
+ }
+
+ p += len + 1;
+
+ char *value = p;
+ len = 0;
+
+ while (p[len] != ':' && p[len] != '\0')
+ len++;
+
+ char end = p[len];
+ p[len] = '\0';
+
+ /* Add the tunable if it exists. */
+ for (size_t i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
+ {
+ if (is_name (tunable_list[i].name, name))
+ {
+ /* Tunable values take precedence over the env_alias. */
+ tunable_list[i].strval = value;
+ tunable_initialize (&tunable_list[i]);
+ break;
+ }
+ }
+
+ if (end == ':')
+ p += len + 1;
+ else
+ return;
+ }
+}
+
/* Initialize the tunables list from the environment. For now we only use the
ENV_ALIAS to find values. Later we will also use the tunable names to find
values. */
@@ -116,6 +179,12 @@ __tunables_init (char **envp)
while (get_next_env (&envp, &envname, &len, &envval))
{
+ if (is_name (GLIBC_TUNABLES, envname))
+ {
+ parse_tunables (strdup (envval));
+ continue;
+ }
+
for (int i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
{
tunable_t *cur = &tunable_list[i];
--
2.5.5
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH 0/2] tunables for glibc
@ 2016-07-02 17:13 Siddhesh Poyarekar
2016-07-02 17:13 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
` (2 more replies)
0 siblings, 3 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-02 17:13 UTC (permalink / raw)
To: libc-alpha
Hi,
Here's another stab at the tunables patchset. This patch now gets tunables
working with static binaries as well.
The tunables framework aims to provide a simple and consistent interface for
the library to expose tuning switches in a much more coherent manner to
userspace. In its initial form, the framework will allow users to tweak
certain parts of the library using an environment variable (or set of
environment variables, depending on where the consensus sends us). In future,
this could be extended to per-user or even systemwide tuning switches.
The first patch implements a bare framework that moves the environment
variables in the malloc module to the tunables framework without adding any new
user-visible functionality. The environment variables that were implemented
earlier should work as is.
The second patch adds support for a GLIBC_TUNABLES environment variable, which
users can use to set up tunables instead of using the old environment
variables. The alternative approach to this is to expose namespace-consistent
environment variables for each tunable for userspace to take advantage of
instead of the earlier environment variables.
Siddhesh Poyarekar (2):
Add framework for tunables
Initialize tunable list with the GLIBC_TUNABLES environment variable
INSTALL | 6 +
Makeconfig | 16 +++
README.tunables | 74 ++++++++++++
config.h.in | 3 +
config.make.in | 1 +
configure | 16 +++
configure.ac | 10 ++
csu/init-first.c | 7 ++
elf/Makefile | 5 +
elf/Versions | 3 +
elf/dl-tunable-types.h | 45 +++++++
elf/dl-tunables.c | 241 ++++++++++++++++++++++++++++++++++++++
elf/dl-tunables.h | 76 ++++++++++++
elf/dl-tunables.list | 50 ++++++++
elf/rtld.c | 8 ++
malloc/Makefile | 3 +
malloc/arena.c | 35 ++++++
malloc/tst-malloc-usable-static.c | 1 +
manual/install.texi | 5 +
scripts/gen-tunables.awk | 157 +++++++++++++++++++++++++
20 files changed, 762 insertions(+)
create mode 100644 README.tunables
create mode 100644 elf/dl-tunable-types.h
create mode 100644 elf/dl-tunables.c
create mode 100644 elf/dl-tunables.h
create mode 100644 elf/dl-tunables.list
create mode 100644 malloc/tst-malloc-usable-static.c
create mode 100644 scripts/gen-tunables.awk
--
2.5.5
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 0/2] tunables for glibc
2016-07-02 17:13 [PATCH 0/2] tunables for glibc Siddhesh Poyarekar
2016-07-02 17:13 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-07-02 17:13 ` [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable Siddhesh Poyarekar
@ 2016-07-03 0:24 ` H.J. Lu
2016-07-03 3:08 ` Siddhesh Poyarekar
2 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-03 0:24 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Sat, Jul 2, 2016 at 10:12 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> Hi,
>
> Here's another stab at the tunables patchset. This patch now gets tunables
> working with static binaries as well.
>
> The tunables framework aims to provide a simple and consistent interface for
> the library to expose tuning switches in a much more coherent manner to
> userspace. In its initial form, the framework will allow users to tweak
> certain parts of the library using an environment variable (or set of
> environment variables, depending on where the consensus sends us). In future,
> this could be extended to per-user or even systemwide tuning switches.
>
> The first patch implements a bare framework that moves the environment
> variables in the malloc module to the tunables framework without adding any new
> user-visible functionality. The environment variables that were implemented
> earlier should work as is.
>
> The second patch adds support for a GLIBC_TUNABLES environment variable, which
> users can use to set up tunables instead of using the old environment
> variables. The alternative approach to this is to expose namespace-consistent
> environment variables for each tunable for userspace to take advantage of
> instead of the earlier environment variables.
>
> Siddhesh Poyarekar (2):
> Add framework for tunables
> Initialize tunable list with the GLIBC_TUNABLES environment variable
>
> INSTALL | 6 +
> Makeconfig | 16 +++
> README.tunables | 74 ++++++++++++
> config.h.in | 3 +
> config.make.in | 1 +
> configure | 16 +++
> configure.ac | 10 ++
> csu/init-first.c | 7 ++
> elf/Makefile | 5 +
> elf/Versions | 3 +
> elf/dl-tunable-types.h | 45 +++++++
> elf/dl-tunables.c | 241 ++++++++++++++++++++++++++++++++++++++
> elf/dl-tunables.h | 76 ++++++++++++
> elf/dl-tunables.list | 50 ++++++++
> elf/rtld.c | 8 ++
> malloc/Makefile | 3 +
> malloc/arena.c | 35 ++++++
> malloc/tst-malloc-usable-static.c | 1 +
> manual/install.texi | 5 +
> scripts/gen-tunables.awk | 157 +++++++++++++++++++++++++
> 20 files changed, 762 insertions(+)
> create mode 100644 README.tunables
> create mode 100644 elf/dl-tunable-types.h
> create mode 100644 elf/dl-tunables.c
> create mode 100644 elf/dl-tunables.h
> create mode 100644 elf/dl-tunables.list
> create mode 100644 malloc/tst-malloc-usable-static.c
> create mode 100644 scripts/gen-tunables.awk
>
Can you push you new patches to siddhesh/tunables branch so
that I can give it a try?
Thanks.
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 0/2] tunables for glibc
2016-07-03 0:24 ` [PATCH 0/2] tunables for glibc H.J. Lu
@ 2016-07-03 3:08 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-03 3:08 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Sat, Jul 02, 2016 at 05:24:40PM -0700, H.J. Lu wrote:
> Can you push you new patches to siddhesh/tunables branch so
> that I can give it a try?
Oops, I thought I had already done that. I've pushed it now.
Thanks,
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-02 17:13 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
@ 2016-07-03 14:30 ` H.J. Lu
2016-07-03 17:08 ` Siddhesh Poyarekar
2016-07-03 14:44 ` H.J. Lu
2016-07-03 15:13 ` H.J. Lu
2 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-03 14:30 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Sat, Jul 2, 2016 at 10:12 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> The tunables framework allows us to uniformly manage and expose global
> variables inside glibc as switches to users. README.tunables has
> instructions for glibc developers to add new tunables.
>
> Tunables support can be enabled by passing the --enable-tunables
> configure flag to the configure script. This patch only adds a
> framework and does not pose any limitations on how tunable values are
> read from the user. It also adds environment variables used in malloc
> behaviour tweaking to the tunables framework as a PoC of the
> compatibility interface.
>
> * manual/install.texi: Add --enable-tunables option.
> * INSTALL: Regenerate.
> * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
> (before-compile): Generate dl-tunable-list.h early.
> * config.h.in: Add BUILD_TUNABLES.
> * config.make.in: Add build-tunables.
> * configure.ac: Add --enable-tunables option.
> * configure: Regenerate.
> * malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
> Define TUNABLE_NAMESPACE.
> (DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
> (ptmalloc_init): Set tunable values.
> * malloc/tst-malloc-usable-static.c: New test case.
> * csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
> (__libc_init_first) [!SHARED]: Initialize tunables for static
> binaries.
> * scripts/gen-tunables.awk: New file.
> * README.tunables: New file.
> * elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
> namespace.
> * elf/dl-tunable-list.h: New auto-generated file.
> * elf/dl-tunables.c: New file.
> * elf/dl-tunables.h: New file.
> * elf/dl-tunables.list: New file.
> * elf/dl-tunable-types.h: New file.
> * elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
> (process_envvars): Call __tunables_init.
>
> diff --git a/csu/init-first.c b/csu/init-first.c
> index 77c6e1c..7427121 100644
> --- a/csu/init-first.c
> +++ b/csu/init-first.c
> @@ -28,6 +28,9 @@
> #include <libc-internal.h>
>
> #include <ldsodefs.h>
> +#if BUILD_TUNABLES
> +# include <elf/dl-tunables.h>
> +#endif
>
> /* Set nonzero if we have to be prepared for more than one libc being
> used in the process. Safe assumption if initializer never runs. */
> @@ -74,6 +77,10 @@ _init (int argc, char **argv, char **envp)
> #ifndef SHARED
> __libc_init_secure ();
>
> +#if BUILD_TUNABLES
> + __tunables_init (envp);
> +#endif
> +
> /* First the initialization which normally would be done by the
> dynamic linker. */
> _dl_non_dynamic_init ();
Shouldn't __tunables_init be called from _dl_non_dynamic_init?
> diff --git a/elf/rtld.c b/elf/rtld.c
> index 647661c..263723a 100644
> --- a/elf/rtld.c
> +++ b/elf/rtld.c
> @@ -44,6 +44,10 @@
>
> #include <assert.h>
>
> +#if BUILD_TUNABLES
> +# include <dl-tunables.h>
> +#endif
> +
> /* Avoid PLT use for our local calls at startup. */
> extern __typeof (__mempcpy) __mempcpy attribute_hidden;
>
> @@ -2346,6 +2350,10 @@ process_envvars (enum mode *modep)
> enum mode mode = normal;
> char *debug_output = NULL;
>
> +#if BUILD_TUNABLES
> + __tunables_init (_environ);
> +#endif
> +
Shouldn't __tunables_init be called from _dl_sysdep_start?
> /* This is the default place for profiling data file. */
> GLRO(dl_profile_output)
> = &"/var/tmp\0/var/profile"[__libc_enable_secure ? 9 : 0];
I
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable
2016-07-02 17:13 ` [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable Siddhesh Poyarekar
@ 2016-07-03 14:39 ` H.J. Lu
2016-07-03 17:15 ` Siddhesh Poyarekar
2016-07-03 15:53 ` H.J. Lu
1 sibling, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-03 14:39 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Sat, Jul 2, 2016 at 10:12 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> Read tunables values from the users using the GLIBC_TUNABLES
> environment variable. The value of this variable is a colon-separated
> list of name=value pairs. So a typical string would look like this:
>
> GLIBC_TUNABLES=glibc.malloc.mmap_threshold=2048:glibc.malloc.trim_threshold=1024
>
> * elf/dl-tunables.c: Include string.h.
> (parse_tunables): New function.
> (GLIBC_TUNABLES): New macro.
> (__tunables_init): Use it.
> ---
> elf/dl-tunables.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 69 insertions(+)
>
> diff --git a/elf/dl-tunables.c b/elf/dl-tunables.c
> index 8a26dfc..04d1d52 100644
> --- a/elf/dl-tunables.c
> +++ b/elf/dl-tunables.c
> @@ -22,10 +22,13 @@
> #include <stdbool.h>
> #include <unistd.h>
> #include <stdlib.h>
> +#include <string.h>
>
> #define TUNABLES_INTERNAL 1
> #include "dl-tunables.h"
>
> +#define GLIBC_TUNABLES "GLIBC_TUNABLES"
> +
> /* Compare environment names, bounded by the name hardcoded in glibc. */
> static bool
> is_name (const char *orig, const char *envname)
> @@ -104,6 +107,66 @@ tunable_initialize (tunable_t *cur)
> }
> }
>
> +static void
> +parse_tunables (char *tunestr)
> +{
> + if (tunestr == NULL || *tunestr == '\0')
> + return;
> +
> + char *p = tunestr;
> +
> + while (true)
> + {
> + char *name = p;
> + size_t len = 0;
> +
> + /* First, find where the name ends. */
> + while (p[len] != '=' && p[len] != ':' && p[len] != '\0')
> + len++;
> +
> + /* If we reach the end of the string before getting a valid name-value
> + pair, bail out. */
> + if (p[len] == '\0')
> + return;
> +
> + /* We did not find a valid name-value pair before encountering the
> + colon. */
> + if (p[len]== ':')
> + {
> + p += len + 1;
> + continue;
> + }
> +
> + p += len + 1;
> +
> + char *value = p;
> + len = 0;
> +
> + while (p[len] != ':' && p[len] != '\0')
> + len++;
> +
> + char end = p[len];
> + p[len] = '\0';
> +
> + /* Add the tunable if it exists. */
> + for (size_t i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
> + {
> + if (is_name (tunable_list[i].name, name))
Can you add the length of name to tunable_list field and check
if the length matches first?
> + {
> + /* Tunable values take precedence over the env_alias. */
> + tunable_list[i].strval = value;
> + tunable_initialize (&tunable_list[i]);
> + break;
> + }
> + }
> +
> + if (end == ':')
> + p += len + 1;
> + else
> + return;
> + }
> +}
> +
> /* Initialize the tunables list from the environment. For now we only use the
> ENV_ALIAS to find values. Later we will also use the tunable names to find
> values. */
> @@ -116,6 +179,12 @@ __tunables_init (char **envp)
>
> while (get_next_env (&envp, &envname, &len, &envval))
> {
> + if (is_name (GLIBC_TUNABLES, envname))
> + {
> + parse_tunables (strdup (envval));
> + continue;
> + }
> +
> for (int i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
> {
> tunable_t *cur = &tunable_list[i];
> --
> 2.5.5
>
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-02 17:13 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-07-03 14:30 ` H.J. Lu
@ 2016-07-03 14:44 ` H.J. Lu
2016-07-03 17:14 ` Siddhesh Poyarekar
2016-07-03 15:13 ` H.J. Lu
2 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-03 14:44 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Sat, Jul 2, 2016 at 10:12 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> The tunables framework allows us to uniformly manage and expose global
> variables inside glibc as switches to users. README.tunables has
> instructions for glibc developers to add new tunables.
>
> Tunables support can be enabled by passing the --enable-tunables
> configure flag to the configure script. This patch only adds a
> framework and does not pose any limitations on how tunable values are
> read from the user. It also adds environment variables used in malloc
> behaviour tweaking to the tunables framework as a PoC of the
> compatibility interface.
>
> * manual/install.texi: Add --enable-tunables option.
> * INSTALL: Regenerate.
> * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
> (before-compile): Generate dl-tunable-list.h early.
> * config.h.in: Add BUILD_TUNABLES.
> * config.make.in: Add build-tunables.
> * configure.ac: Add --enable-tunables option.
> * configure: Regenerate.
> * malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
> Define TUNABLE_NAMESPACE.
> (DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
> (ptmalloc_init): Set tunable values.
> * malloc/tst-malloc-usable-static.c: New test case.
> * csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
> (__libc_init_first) [!SHARED]: Initialize tunables for static
> binaries.
> * scripts/gen-tunables.awk: New file.
> * README.tunables: New file.
> * elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
> namespace.
> * elf/dl-tunable-list.h: New auto-generated file.
> * elf/dl-tunables.c: New file.
> * elf/dl-tunables.h: New file.
> * elf/dl-tunables.list: New file.
> * elf/dl-tunable-types.h: New file.
> * elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
> (process_envvars): Call __tunables_init.
> +
> +/* Compare environment names, bounded by the name hardcoded in glibc. */
> +static bool
> +is_name (const char *orig, const char *envname)
> +{
> + for (;*orig != '\0' && *envname != '\0'; envname++, orig++)
> + if (*orig != *envname)
> + break;
> +
> + /* The ENVNAME is immediately followed by a value. */
> + if (*orig == '\0' && *envname == '=')
> + return true;
> + else
> + return false;
> +}
Can you add the length parameter to is_name and make is_name
processor specific so that misaligned short/int/long long load can
be used, similar to equal in
https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86/cpu-features.c;h=3ae44cfb86e39b41c43903e403325541476b37bf;hb=refs/heads/hjl/ifunc/master
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-02 17:13 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-07-03 14:30 ` H.J. Lu
2016-07-03 14:44 ` H.J. Lu
@ 2016-07-03 15:13 ` H.J. Lu
2016-07-03 17:43 ` H.J. Lu
2 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-03 15:13 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Sat, Jul 2, 2016 at 10:12 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> The tunables framework allows us to uniformly manage and expose global
> variables inside glibc as switches to users. README.tunables has
> instructions for glibc developers to add new tunables.
>
> Tunables support can be enabled by passing the --enable-tunables
> configure flag to the configure script. This patch only adds a
> framework and does not pose any limitations on how tunable values are
> read from the user. It also adds environment variables used in malloc
> behaviour tweaking to the tunables framework as a PoC of the
> compatibility interface.
>
> * manual/install.texi: Add --enable-tunables option.
> * INSTALL: Regenerate.
> * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
> (before-compile): Generate dl-tunable-list.h early.
> * config.h.in: Add BUILD_TUNABLES.
> * config.make.in: Add build-tunables.
> * configure.ac: Add --enable-tunables option.
> * configure: Regenerate.
> * malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
> Define TUNABLE_NAMESPACE.
> (DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
> (ptmalloc_init): Set tunable values.
> * malloc/tst-malloc-usable-static.c: New test case.
> * csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
> (__libc_init_first) [!SHARED]: Initialize tunables for static
> binaries.
> * scripts/gen-tunables.awk: New file.
> * README.tunables: New file.
> * elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
> namespace.
> * elf/dl-tunable-list.h: New auto-generated file.
> * elf/dl-tunables.c: New file.
> * elf/dl-tunables.h: New file.
> * elf/dl-tunables.list: New file.
> * elf/dl-tunable-types.h: New file.
> * elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
> (process_envvars): Call __tunables_init.
> ---
> INSTALL | 6 ++
> Makeconfig | 16 ++++
> README.tunables | 74 ++++++++++++++++
> config.h.in | 3 +
> config.make.in | 1 +
> configure | 16 ++++
> configure.ac | 10 +++
> csu/init-first.c | 7 ++
> elf/Makefile | 5 ++
> elf/Versions | 3 +
> elf/dl-tunable-types.h | 45 ++++++++++
> elf/dl-tunables.c | 172 ++++++++++++++++++++++++++++++++++++++
> elf/dl-tunables.h | 76 +++++++++++++++++
> elf/dl-tunables.list | 50 +++++++++++
> elf/rtld.c | 8 ++
> malloc/Makefile | 3 +
> malloc/arena.c | 35 ++++++++
> malloc/tst-malloc-usable-static.c | 1 +
> manual/install.texi | 5 ++
> scripts/gen-tunables.awk | 157 ++++++++++++++++++++++++++++++++++
> 20 files changed, 693 insertions(+)
> create mode 100644 README.tunables
> create mode 100644 elf/dl-tunable-types.h
> create mode 100644 elf/dl-tunables.c
> create mode 100644 elf/dl-tunables.h
> create mode 100644 elf/dl-tunables.list
> create mode 100644 malloc/tst-malloc-usable-static.c
> create mode 100644 scripts/gen-tunables.awk
>
> +/* Same as TUNABLE_SET_VAL, but also set the callback function to __CB and call
> + it. */
> +# define TUNABLE_SET_VAL_WITH_CALLBACK(__id,__val,__cb) \
> +({ \
> + __tunable_set_val \
> + (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
> + DL_TUNABLE_CALLBACK (__cb)); \
> +})
> +
> +/* Namespace sanity for callback functions. Use this macro to keep the
> + namespace of the modules clean. */
> +#define DL_TUNABLE_CALLBACK(__name) _dl_tunable_ ## __name
> +#endif
For IFUNC, or any processor specific tunables, a different callback is
needed:
void
ifunc_callback (const char *p)
{
...
}
where p is the string after "GLIBC_IFUNC=".
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable
2016-07-02 17:13 ` [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable Siddhesh Poyarekar
2016-07-03 14:39 ` H.J. Lu
@ 2016-07-03 15:53 ` H.J. Lu
2016-07-03 17:18 ` Siddhesh Poyarekar
1 sibling, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-03 15:53 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Sat, Jul 2, 2016 at 10:12 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> Read tunables values from the users using the GLIBC_TUNABLES
> environment variable. The value of this variable is a colon-separated
> list of name=value pairs. So a typical string would look like this:
>
> GLIBC_TUNABLES=glibc.malloc.mmap_threshold=2048:glibc.malloc.trim_threshold=1024
>
> * elf/dl-tunables.c: Include string.h.
> (parse_tunables): New function.
> (GLIBC_TUNABLES): New macro.
> (__tunables_init): Use it.
I didn't see any tests for GLIBC_TUNABLES=glibc.malloc and I don't think
it works for static executable:
gdb) bt
#0 __tunable_set_val (id=glibc_malloc_check, valp=0x6b67d0 <check_action>,
callback=0x414460 <_dl_tunable_set_mallopt_check>) at dl-tunables.c:221
#1 0x0000000000413cda in ptmalloc_init () at arena.c:299
#2 0x0000000000414dfe in ptmalloc_init () at hooks.c:29
#3 malloc_hook_ini (sz=66, caller=<optimized out>) at hooks.c:31
#4 0x0000000000418b2a in __strdup (
s=0x7fffffffef66
"glibc.malloc.mmap_threshold=2048:glibc.malloc.trim_threshold=1024")
at strdup.c:42
#5 0x0000000000437e01 in __tunables_init (envp=0x7fffffffe088,
envp@entry=0x7fffffffdeb8) at dl-tunables.c:184
#6 0x000000000043908b in __libc_init_first (argc=argc@entry=1,
argv=argv@entry=0x7fffffffdea8, envp=0x7fffffffdeb8)
at ../csu/init-first.c:81
#7 0x00000000004012d5 in generic_start_main (main=0x400550 <main>, argc=1,
argv=argv@entry=0x7fffffffdea8,
init=init@entry=0x401b20 <__libc_csu_init>,
fini=0x401bb0 <__libc_csu_fini>, rtld_fini=0x0, stack_end=0x7fffffffde98)
at ../csu/libc-start.c:225
#8 0x00000000004015c7 in __libc_start_main (main=<optimized out>,
argc=<optimized out>, argv=0x7fffffffdea8,
init=0x401b20 <__libc_csu_init>, fini=0x401bb0 <__libc_csu_fini>,
rtld_fini=<optimized out>, stack_end=0x7fffffffde98)
at ../sysdeps/x86/libc-start.c:38
---Type <return> to continue, or q <return> to quit---
#9 0x0000000000400e1a in _start () at ../sysdeps/x86_64/start.S:120
if (is_name (GLIBC_TUNABLES, envname))
{
parse_tunables (strdup (envval));
^^^^^^^^^ Which calls malloc and
other library functions.
continue;
}
Since tunable code is called very early, it can't use any C library functions.
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-03 14:30 ` H.J. Lu
@ 2016-07-03 17:08 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-03 17:08 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Sun, Jul 03, 2016 at 07:30:30AM -0700, H.J. Lu wrote:
> Shouldn't __tunables_init be called from _dl_non_dynamic_init?
>
<snip>
>
> Shouldn't __tunables_init be called from _dl_sysdep_start?
It needs to be earlier. Eventually I intend to put dynamic linker
envvars also into the tunables.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-03 14:44 ` H.J. Lu
@ 2016-07-03 17:14 ` Siddhesh Poyarekar
2016-07-03 17:41 ` H.J. Lu
0 siblings, 1 reply; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-03 17:14 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Sun, Jul 03, 2016 at 07:44:25AM -0700, H.J. Lu wrote:
> > +
> > +/* Compare environment names, bounded by the name hardcoded in glibc. */
> > +static bool
> > +is_name (const char *orig, const char *envname)
> > +{
> > + for (;*orig != '\0' && *envname != '\0'; envname++, orig++)
> > + if (*orig != *envname)
> > + break;
> > +
> > + /* The ENVNAME is immediately followed by a value. */
> > + if (*orig == '\0' && *envname == '=')
> > + return true;
> > + else
> > + return false;
> > +}
>
> Can you add the length parameter to is_name and make is_name
> processor specific so that misaligned short/int/long long load can
> be used, similar to equal in
Will it really make that much of an impact? These will be really
short comparisons (less than 16 bytes in most cases) and will really
only happen once during startup.
Maybe we could do it as an addon patch if the tunables initialization
really turns out to be that much of a bottleneck.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable
2016-07-03 14:39 ` H.J. Lu
@ 2016-07-03 17:15 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-03 17:15 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Sun, Jul 03, 2016 at 07:38:54AM -0700, H.J. Lu wrote:
> Can you add the length of name to tunable_list field and check
> if the length matches first?
Similar to passing the length in is_name, can we do this as an addon
if we see a significant performance impact?
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable
2016-07-03 15:53 ` H.J. Lu
@ 2016-07-03 17:18 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-03 17:18 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Sun, Jul 03, 2016 at 08:53:12AM -0700, H.J. Lu wrote:
> I didn't see any tests for GLIBC_TUNABLES=glibc.malloc and I don't think
> it works for static executable:
Hmm, right, I need to fix this. I had manually run the
tst-malloc-usable patch with GLIBC_TUNABLES but didn't do it for
static case again. I'll fix this up, write those tests and repost.
Thanks,
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-03 17:14 ` Siddhesh Poyarekar
@ 2016-07-03 17:41 ` H.J. Lu
0 siblings, 0 replies; 49+ messages in thread
From: H.J. Lu @ 2016-07-03 17:41 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Sun, Jul 3, 2016 at 10:13 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> On Sun, Jul 03, 2016 at 07:44:25AM -0700, H.J. Lu wrote:
>> > +
>> > +/* Compare environment names, bounded by the name hardcoded in glibc. */
>> > +static bool
>> > +is_name (const char *orig, const char *envname)
>> > +{
>> > + for (;*orig != '\0' && *envname != '\0'; envname++, orig++)
>> > + if (*orig != *envname)
>> > + break;
>> > +
>> > + /* The ENVNAME is immediately followed by a value. */
>> > + if (*orig == '\0' && *envname == '=')
>> > + return true;
>> > + else
>> > + return false;
>> > +}
>>
>> Can you add the length parameter to is_name and make is_name
>> processor specific so that misaligned short/int/long long load can
>> be used, similar to equal in
>
> Will it really make that much of an impact? These will be really
> short comparisons (less than 16 bytes in most cases) and will really
> only happen once during startup.
>
> Maybe we could do it as an addon patch if the tunables initialization
> really turns out to be that much of a bottleneck.
>
Sure.
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-03 15:13 ` H.J. Lu
@ 2016-07-03 17:43 ` H.J. Lu
2016-07-03 18:20 ` Siddhesh Poyarekar
0 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-03 17:43 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Sun, Jul 3, 2016 at 8:13 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Sat, Jul 2, 2016 at 10:12 AM, Siddhesh Poyarekar
> <siddhesh@sourceware.org> wrote:
>> The tunables framework allows us to uniformly manage and expose global
>> variables inside glibc as switches to users. README.tunables has
>> instructions for glibc developers to add new tunables.
>>
>> Tunables support can be enabled by passing the --enable-tunables
>> configure flag to the configure script. This patch only adds a
>> framework and does not pose any limitations on how tunable values are
>> read from the user. It also adds environment variables used in malloc
>> behaviour tweaking to the tunables framework as a PoC of the
>> compatibility interface.
>>
>> * manual/install.texi: Add --enable-tunables option.
>> * INSTALL: Regenerate.
>> * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
>> (before-compile): Generate dl-tunable-list.h early.
>> * config.h.in: Add BUILD_TUNABLES.
>> * config.make.in: Add build-tunables.
>> * configure.ac: Add --enable-tunables option.
>> * configure: Regenerate.
>> * malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
>> Define TUNABLE_NAMESPACE.
>> (DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
>> (ptmalloc_init): Set tunable values.
>> * malloc/tst-malloc-usable-static.c: New test case.
>> * csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
>> (__libc_init_first) [!SHARED]: Initialize tunables for static
>> binaries.
>> * scripts/gen-tunables.awk: New file.
>> * README.tunables: New file.
>> * elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
>> namespace.
>> * elf/dl-tunable-list.h: New auto-generated file.
>> * elf/dl-tunables.c: New file.
>> * elf/dl-tunables.h: New file.
>> * elf/dl-tunables.list: New file.
>> * elf/dl-tunable-types.h: New file.
>> * elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
>> (process_envvars): Call __tunables_init.
>
>
>> +/* Same as TUNABLE_SET_VAL, but also set the callback function to __CB and call
>> + it. */
>> +# define TUNABLE_SET_VAL_WITH_CALLBACK(__id,__val,__cb) \
>> +({ \
>> + __tunable_set_val \
>> + (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
>> + DL_TUNABLE_CALLBACK (__cb)); \
>> +})
>> +
>> +/* Namespace sanity for callback functions. Use this macro to keep the
>> + namespace of the modules clean. */
>> +#define DL_TUNABLE_CALLBACK(__name) _dl_tunable_ ## __name
>> +#endif
>
> For IFUNC, or any processor specific tunables, a different callback is
> needed:
>
> void
> ifunc_callback (const char *p)
> {
> ...
> }
>
> where p is the string after "GLIBC_IFUNC=".
>
> H.J.
This should be an option for __tunables_init not to parse the
string, but instead pass the string to callback directly.
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-03 17:43 ` H.J. Lu
@ 2016-07-03 18:20 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-03 18:20 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Sun, Jul 03, 2016 at 10:43:36AM -0700, H.J. Lu wrote:
> This should be an option for __tunables_init not to parse the
> string, but instead pass the string to callback directly.
If the tunable type is string then right now it won't do any parsing,
just assign the value in the tunable_val_t. I could modify the
callback to:
void (*callback) (void *)
That way the value is sent back via the callback regardless of the
type. The callback can then cast the value appropriately and use it.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH 1/2] Add framework for tunables
2016-08-15 20:05 [PATCHv4 " Siddhesh Poyarekar
@ 2016-08-15 20:05 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-08-15 20:05 UTC (permalink / raw)
To: libc-alpha; +Cc: carlos, fweimer
The tunables framework allows us to uniformly manage and expose global
variables inside glibc as switches to users. tunables/README has
instructions for glibc developers to add new tunables.
Tunables support can be enabled by passing the --enable-tunables
configure flag to the configure script. This patch only adds a
framework and does not pose any limitations on how tunable values are
read from the user. It also adds environment variables used in malloc
behaviour tweaking to the tunables framework as a PoC of the
compatibility interface.
* manual/install.texi: Add --enable-tunables option.
* INSTALL: Regenerate.
* Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
(before-compile): Generate dl-tunable-list.h early.
* config.h.in: Add HAVE_TUNABLES.
* config.make.in: Add have-tunables.
* configure.ac: Add --enable-tunables option.
* configure: Regenerate.
* malloc/arena.c [HAVE_TUNABLES]: Include dl-tunables.h.
Define TUNABLE_NAMESPACE.
(DL_TUNABLE_CALLBACK (set_mallopt_check)): New function.
(DL_TUNABLE_CALLBACK_FNDECL): New macro. Use it to define
callback functions.
(ptmalloc_init): Set tunable values.
* malloc/malloc.c (do_set_mallopt_check,
do_set_mmap_threshold, do_set_mmaps_max, do_set_top_pad,
do_set_perturb_byte, do_set_trim_threshold, do_set_arena_max,
do_set_arena_test): New functions.
(__libc_mallopt): Use them.
* malloc/tst-malloc-usable-static.c: New test case.
* malloc/Makefile (tests-static): Add it.
* csu/init-first.c [HAVE_TUNABLES]: Include dl-tunables.h.
(__libc_init_first) [!SHARED]: Initialize tunables for static
binaries.
* scripts/gen-tunables.awk: New file.
* README.tunables: New file.
* elf/Makefile (dl-routines): Add dl-tunables.
* elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
namespace.
* elf/dl-sysdep.c [HAVE_TUNABLES]: Include dl-tunables.h
(_dl_sysdep_start): Call __TUNABLES_INIT.
* elf/dl-tunable-list.h: New auto-generated file.
* elf/dl-tunables.c: New file.
* elf/dl-tunables.h: New file.
* elf/dl-tunables.list: New file.
* elf/dl-tunable-types.h: New file.
* elf/rtld.c [HAVE_TUNABLES]: Include dl-tunables.h
(process_envvars): Call __tunables_init.
* sysdeps/mach/hurd/dl-sysdep.c [HAVE_TUNABLES]: Include
dl-tunables.h.
(_dl_sysdep_start): Call __TUNABLES_INIT.
* sysdeps/mach/hurd/i386/init-first.c [HAVE_TUNABLES]: Include
dl-tunables.h.
(_dl_sysdep_start): Call __TUNABLES_INIT.
---
INSTALL | 5 +
Makeconfig | 16 ++++
README.tunables | 84 ++++++++++++++++
config.h.in | 3 +
config.make.in | 1 +
configure | 16 ++++
configure.ac | 10 ++
csu/init-first.c | 7 ++
elf/Makefile | 5 +
elf/Versions | 3 +
elf/dl-sysdep.c | 8 ++
elf/dl-tunable-types.h | 46 +++++++++
elf/dl-tunables.c | 185 ++++++++++++++++++++++++++++++++++++
elf/dl-tunables.h | 78 +++++++++++++++
elf/dl-tunables.list | 69 ++++++++++++++
malloc/Makefile | 3 +
malloc/arena.c | 54 +++++++++++
malloc/malloc.c | 126 +++++++++++++++++-------
malloc/tst-malloc-usable-static.c | 1 +
manual/install.texi | 5 +
scripts/gen-tunables.awk | 157 ++++++++++++++++++++++++++++++
sysdeps/mach/hurd/dl-sysdep.c | 8 ++
sysdeps/mach/hurd/i386/init-first.c | 8 ++
23 files changed, 864 insertions(+), 34 deletions(-)
create mode 100644 README.tunables
create mode 100644 elf/dl-tunable-types.h
create mode 100644 elf/dl-tunables.c
create mode 100644 elf/dl-tunables.h
create mode 100644 elf/dl-tunables.list
create mode 100644 malloc/tst-malloc-usable-static.c
create mode 100644 scripts/gen-tunables.awk
diff --git a/INSTALL b/INSTALL
index ec3445f..43edee4 100644
--- a/INSTALL
+++ b/INSTALL
@@ -158,6 +158,11 @@ will be used, and CFLAGS sets optimization options for the compiler.
By default for x86_64, the GNU C Library is built with vector math
library. Use this option to disable vector math library.
+'--enable-tunables'
+ Tunables support allows additional library parameters to be
+ customized at runtime. This is an experimental feature and affects
+ startup time and is thus disabled by default.
+
'--build=BUILD-SYSTEM'
'--host=HOST-SYSTEM'
These options are for cross-compiling. If you specify both options
diff --git a/Makeconfig b/Makeconfig
index 03fd89c..ba1180f 100644
--- a/Makeconfig
+++ b/Makeconfig
@@ -883,6 +883,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \
$(foreach lib,$(libof-$(basename $(@F))) \
$(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \
$(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F)))
+
+ifeq (yes,$(have-tunables))
+CPPFLAGS += -DTOP_NAMESPACE=glibc
+endif
+
override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \
$(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \
$(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \
@@ -1055,6 +1060,17 @@ $(common-objpfx)libc-modules.stmp: $(..)scripts/gen-libc-modules.awk \
endif
+# Build the tunables list header early since it could be used by any module in
+# glibc.
+ifeq (yes,$(have-tunables))
+before-compile += $(common-objpfx)dl-tunable-list.h
+
+$(common-objpfx)dl-tunable-list.h: $(..)scripts/gen-tunables.awk \
+ $(..)elf/dl-tunables.list
+ $(AWK) -f $^ > $@.tmp
+ mv $@{.tmp,}
+endif
+
common-generated += libc-modules.h libc-modules.stmp
# The name under which the run-time dynamic linker is installed.
diff --git a/README.tunables b/README.tunables
new file mode 100644
index 0000000..3d9c86a
--- /dev/null
+++ b/README.tunables
@@ -0,0 +1,84 @@
+ TUNABLE FRAMEWORK
+ =================
+
+Tunables is a feature in the GNU C Library that allows application authors and
+distribution maintainers to alter the runtime library behaviour to match their
+workload.
+
+The tunable framework allows modules within glibc to register variables that
+may be tweaked through an environment variable or an API call. It aims to
+enforce a strict namespace rule to bring consistency to naming of these tunable
+environment variables across the project. This document is a guide for glibc
+developers to add tunables to the framework.
+
+ADDING A NEW TUNABLE
+--------------------
+
+The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in
+distributions for specific tunables if they want to add their own tunables.
+Downstream implementations are discouraged from using the 'glibc' namespace for
+tunables they don't already have consensus to push upstream.
+
+There are two steps to adding a tunable:
+
+1. Add a tunable ID:
+
+Modules that wish to use the tunables interface must define the
+TUNABLE_NAMESPACE macro. Following this, for each tunable you want to
+add, make an entry in elf/dl-tunables.list. The format of the file is as
+follows:
+
+TOP_NAMESPACE {
+ NAMESPACE1 {
+ TUNABLE1 {
+ # tunable attributes, one per line
+ }
+ # A tunable with default attributes, i.e. string variable.
+ TUNABLE2
+ TUNABLE3 {
+ # its attributes
+ }
+ }
+ NAMESPACE2 {
+ ...
+ }
+}
+
+The list of allowed attributes are:
+
+- type: Data type. Defaults to STRING. Allowed types are:
+ INT_32, SIZE_T and STRING.
+
+- minval: Optional minimum acceptable value. For a string type
+ this is the minimum length of the value.
+
+- maxval: Optional maximum acceptable value. For a string type
+ this is the maximum length of the value.
+
+- env_alias: An alias environment variable
+
+- is_secure: Specify whether the tunable should be read for setuid
+ binaries. True allows the tunable to be read for
+ setuid binaries while false disables it. Note that
+ even if this is set as true and the value is read, it
+ may not be used if it does not validate against the
+ acceptable values or is not considered safe by the
+ module.
+
+2. Call either the TUNABLE_SET_VALUE and pass into it the tunable name and a
+ pointer to the variable that should be set with the tunable value.
+ If additional work needs to be done after setting the value, use the
+ TUNABLE_SET_VALUE_WITH_CALLBACK instead and additionally pass a pointer to
+ the function that should be called if the tunable value has been set.
+
+FUTURE WORK
+-----------
+
+The framework currently only allows a one-time initialization of variables
+through environment variables and in some cases, modification of variables via
+an API call. A future goals for this project include:
+
+- Setting system-wide and user-wide defaults for tunables through some
+ mechanism like a configuration file.
+
+- Allow tweaking of some tunables at runtime
diff --git a/config.h.in b/config.h.in
index 856ef6a..f8be48e 100644
--- a/config.h.in
+++ b/config.h.in
@@ -240,4 +240,7 @@
/* PowerPC32 uses fctidz for floating point to long long conversions. */
#define HAVE_PPC_FCTIDZ 0
+/* Build glibc with tunables support. */
+#define HAVE_TUNABLES 0
+
#endif
diff --git a/config.make.in b/config.make.in
index 95c6f36..8ca9579 100644
--- a/config.make.in
+++ b/config.make.in
@@ -91,6 +91,7 @@ use-nscd = @use_nscd@
build-hardcoded-path-in-tests= @hardcoded_path_in_tests@
build-pt-chown = @build_pt_chown@
enable-lock-elision = @enable_lock_elision@
+have-tunables = @have_tunables@
# Build tools.
CC = @CC@
diff --git a/configure b/configure
index 17625e1..b184205 100755
--- a/configure
+++ b/configure
@@ -660,6 +660,7 @@ multi_arch
base_machine
add_on_subdirs
add_ons
+have_tunables
build_pt_chown
build_nscd
link_obsolete_rpc
@@ -774,6 +775,7 @@ enable_systemtap
enable_build_nscd
enable_nscd
enable_pt_chown
+enable_tunables
enable_mathvec
with_cpu
'
@@ -1441,6 +1443,7 @@ Optional Features:
--disable-build-nscd disable building and installing the nscd daemon
--disable-nscd library functions will not contact the nscd daemon
--enable-pt_chown Enable building and installing pt_chown
+ --enable-tunables Enable tunables support
--enable-mathvec Enable building and installing mathvec [default
depends on architecture]
@@ -3647,6 +3650,19 @@ if test "$build_pt_chown" = yes; then
fi
+# Check whether --enable-tunables was given.
+if test "${enable_tunables+set}" = set; then :
+ enableval=$enable_tunables; have_tunables=$enableval
+else
+ have_tunables=no
+fi
+
+
+if test "$have_tunables" = yes; then
+ $as_echo "#define HAVE_TUNABLES 1" >>confdefs.h
+
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/configure.ac b/configure.ac
index 33bcd62..ece902a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -395,6 +395,16 @@ if test "$build_pt_chown" = yes; then
AC_DEFINE(HAVE_PT_CHOWN)
fi
+AC_ARG_ENABLE([tunables],
+ [AS_HELP_STRING([--enable-tunables],
+ [Enable tunables support])],
+ [have_tunables=$enableval],
+ [have_tunables=no])
+AC_SUBST(have_tunables)
+if test "$have_tunables" = yes; then
+ AC_DEFINE(HAVE_TUNABLES)
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/csu/init-first.c b/csu/init-first.c
index 77c6e1c..a35f997 100644
--- a/csu/init-first.c
+++ b/csu/init-first.c
@@ -28,6 +28,9 @@
#include <libc-internal.h>
#include <ldsodefs.h>
+#if HAVE_TUNABLES
+# include <elf/dl-tunables.h>
+#endif
/* Set nonzero if we have to be prepared for more than one libc being
used in the process. Safe assumption if initializer never runs. */
@@ -74,6 +77,10 @@ _init (int argc, char **argv, char **envp)
#ifndef SHARED
__libc_init_secure ();
+#if HAVE_TUNABLES
+ __tunables_init (envp);
+#endif
+
/* First the initialization which normally would be done by the
dynamic linker. */
_dl_non_dynamic_init ();
diff --git a/elf/Makefile b/elf/Makefile
index 593403c..48989e7 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -35,6 +35,11 @@ dl-routines = $(addprefix dl-,load lookup object reloc deps hwcaps \
ifeq (yes,$(use-ldconfig))
dl-routines += dl-cache
endif
+
+ifeq (yes,$(have-tunables))
+dl-routines += dl-tunables
+endif
+
all-dl-routines = $(dl-routines) $(sysdep-dl-routines)
# But they are absent from the shared libc, because that code is in ld.so.
elide-routines.os = $(all-dl-routines) dl-support enbl-secure dl-origin \
diff --git a/elf/Versions b/elf/Versions
index 23deda9..1734697 100644
--- a/elf/Versions
+++ b/elf/Versions
@@ -64,5 +64,8 @@ ld {
# Pointer protection.
__pointer_chk_guard;
+
+ # Set value of a tunable.
+ __tunable_set_val;
}
}
diff --git a/elf/dl-sysdep.c b/elf/dl-sysdep.c
index eaa7155..68f30f2 100644
--- a/elf/dl-sysdep.c
+++ b/elf/dl-sysdep.c
@@ -44,6 +44,10 @@
#include <hp-timing.h>
#include <tls.h>
+#if HAVE_TUNABLES
+# include <dl-tunables.h>
+#endif
+
extern char **_environ attribute_hidden;
extern char _end[] attribute_hidden;
@@ -219,6 +223,10 @@ _dl_sysdep_start (void **start_argptr,
}
#endif
+#if HAVE_TUNABLES
+ __tunables_init (_environ);
+#endif
+
#ifdef DL_SYSDEP_INIT
DL_SYSDEP_INIT;
#endif
diff --git a/elf/dl-tunable-types.h b/elf/dl-tunable-types.h
new file mode 100644
index 0000000..d1591b6
--- /dev/null
+++ b/elf/dl-tunable-types.h
@@ -0,0 +1,46 @@
+/* Tunable type information.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLE_TYPES_H_
+# define _TUNABLE_TYPES_H_
+#include <stddef.h>
+
+typedef void (*tunable_callback_t) (void *);
+
+typedef enum
+{
+ TUNABLE_TYPE_INT_32,
+ TUNABLE_TYPE_SIZE_T,
+ TUNABLE_TYPE_STRING
+} tunable_type_code_t;
+
+typedef struct
+{
+ tunable_type_code_t type_code;
+ int64_t min;
+ int64_t max;
+} tunable_type_t;
+
+typedef union
+{
+ int64_t numval;
+ const char *strval;
+} tunable_val_t;
+
+#endif
diff --git a/elf/dl-tunables.c b/elf/dl-tunables.c
new file mode 100644
index 0000000..b8f8fc0
--- /dev/null
+++ b/elf/dl-tunables.c
@@ -0,0 +1,185 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <stdlib.h>
+
+#define TUNABLES_INTERNAL 1
+#include "dl-tunables.h"
+
+/* Compare environment names, bounded by the name hardcoded in glibc. */
+static bool
+is_name (const char *orig, const char *envname)
+{
+ for (;*orig != '\0' && *envname != '\0'; envname++, orig++)
+ if (*orig != *envname)
+ break;
+
+ /* The ENVNAME is immediately followed by a value. */
+ if (*orig == '\0' && *envname == '=')
+ return true;
+ else
+ return false;
+}
+
+static char **
+get_next_env (char **envp, char **name, size_t *namelen, char **val)
+{
+ while (envp != NULL && *envp != NULL)
+ {
+ char *envline = *envp;
+ int len = 0;
+
+ while (envline[len] != '\0' && envline[len] != '=')
+ len++;
+
+ /* Just the name and no value, go to the next one. */
+ if (envline[len] == '\0')
+ continue;
+
+ *name = envline;
+ *namelen = len;
+ *val = &envline[len + 1];
+
+ return ++envp;
+ }
+
+ return NULL;
+}
+
+/* If the value does not validate, don't bother initializing the internal type
+ and also take care to clear the recorded string value in STRVAL. */
+#define RETURN_IF_INVALID_RANGE(__cur, __val) \
+({ \
+ __typeof ((__cur)->type.min) min = (__cur)->type.min; \
+ __typeof ((__cur)->type.max) max = (__cur)->type.max; \
+ if (min != max && ((__val) < min || (__val) > max)) \
+ return; \
+})
+
+/* Validate range of the input value and initialize the tunable CUR if it looks
+ good. */
+static void
+tunable_initialize (tunable_t *cur, const char *strval)
+{
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ int32_t val = (int32_t) __strtoul_internal (strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ cur->strval = strval;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ size_t val = (size_t) __strtoul_internal (strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ cur->strval = strval;
+ break;
+ }
+ case TUNABLE_TYPE_STRING:
+ {
+ cur->val.strval = cur->strval = strval;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+}
+
+/* Initialize the tunables list from the environment. For now we only use the
+ ENV_ALIAS to find values. Later we will also use the tunable names to find
+ values. */
+void
+__tunables_init (char **envp)
+{
+ char *envname = NULL;
+ char *envval = NULL;
+ size_t len = 0;
+
+ while ((envp = get_next_env (envp, &envname, &len, &envval)) != NULL)
+ {
+ for (int i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
+ {
+ tunable_t *cur = &tunable_list[i];
+
+ /* Skip over tunables that have either been set already or should be
+ skipped. */
+ if (cur->strval != NULL || cur->env_alias == NULL
+ || (__libc_enable_secure && !cur->is_secure))
+ continue;
+
+ const char *name = cur->env_alias;
+
+ /* We have a match. Initialize and move on to the next line. */
+ if (is_name (name, envname))
+ {
+ tunable_initialize (cur, envval);
+ break;
+ }
+ }
+ }
+}
+
+/* Set the tunable value. This is called by the module that the tunable exists
+ in. */
+void
+__tunable_set_val (tunable_id_t id, void *valp, tunable_callback_t callback)
+{
+ tunable_t *cur = &tunable_list[id];
+
+ /* Don't do anything if our tunable was not set during initialization or if
+ it failed validation. */
+ if (cur->strval == NULL)
+ return;
+
+ if (valp == NULL)
+ goto cb;
+
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ *((int32_t *) valp) = (int32_t) cur->val.numval;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ *((size_t *) valp) = (size_t) cur->val.numval;
+ break;
+ }
+ case TUNABLE_TYPE_STRING:
+ {
+ *((const char **)valp) = cur->val.strval;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+
+cb:
+ if (callback)
+ callback (&cur->val);
+}
diff --git a/elf/dl-tunables.h b/elf/dl-tunables.h
new file mode 100644
index 0000000..10a644b
--- /dev/null
+++ b/elf/dl-tunables.h
@@ -0,0 +1,78 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLES_H_
+# define _TUNABLES_H_
+# include <stddef.h>
+# include "dl-tunable-types.h"
+
+/* A tunable. */
+struct _tunable
+{
+ const char *name; /* Internal name of the tunable. */
+ tunable_type_t type; /* Data type of the tunable. */
+ tunable_val_t val; /* The value. */
+ const char *strval; /* The string containing the value,
+ points into envp. */
+ bool is_secure; /* Whether the tunable must be read
+ even for setuid binaries. Note that
+ even if the tunable is read, it may
+ not get used by the target module if
+ the value is considered unsafe. */
+ /* Compatibility elements. */
+ const char *env_alias; /* The compatibility environment
+ variable name. */
+};
+
+typedef struct _tunable tunable_t;
+
+/* Full name for a tunable is top_ns.tunable_ns.id. */
+#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id
+
+# define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)
+# define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id
+
+#include "dl-tunable-list.h"
+
+extern void __tunables_init (char **);
+extern void __tunable_set_val (tunable_id_t, void *, tunable_callback_t);
+
+/* Check if the tunable has been set to a non-default value and if it is, copy
+ it over into __VAL. */
+# define TUNABLE_SET_VAL(__id,__val) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ NULL); \
+})
+
+/* Same as TUNABLE_SET_VAL, but also set the callback function to __CB and call
+ it. */
+# define TUNABLE_SET_VAL_WITH_CALLBACK(__id,__val,__cb) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ DL_TUNABLE_CALLBACK (__cb)); \
+})
+
+/* Namespace sanity for callback functions. Use this macro to keep the
+ namespace of the modules clean. */
+#define DL_TUNABLE_CALLBACK(__name) _dl_tunable_ ## __name
+#endif
diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list
new file mode 100644
index 0000000..11504c4
--- /dev/null
+++ b/elf/dl-tunables.list
@@ -0,0 +1,69 @@
+# Copyright (C) 2016 Free Software Foundation, Inc.
+# This file is part of the GNU C Library.
+
+# The GNU C Library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+
+# The GNU C Library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+
+# You should have received a copy of the GNU Lesser General Public
+# License along with the GNU C Library; if not, see
+# <http://www.gnu.org/licenses/>.
+
+# Allowed attributes for tunables:
+#
+# type: Defaults to STRING
+# minval: Optional minimum acceptable value
+# maxval: Optional maximum acceptable value
+# env_alias: An alias environment variable
+# is_secure: Specify whether the environment variable should be read for
+# setuid binaries.
+
+glibc {
+ malloc {
+ check {
+ type: INT_32
+ minval: 0
+ maxval: 3
+ env_alias: MALLOC_CHECK_
+ is_secure: true
+ }
+ top_pad {
+ type: SIZE_T
+ env_alias: MALLOC_TOP_PAD_
+ }
+ perturb {
+ type: INT_32
+ minval: 0
+ maxval: 0xff
+ env_alias: MALLOC_PERTURB_
+ }
+ mmap_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_MMAP_THRESHOLD_
+ }
+ trim_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_TRIM_THRESHOLD_
+ }
+ mmap_max {
+ type: INT_32
+ env_alias: MALLOC_MMAP_MAX_
+ }
+ arena_max {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_MAX
+ minval: 1
+ }
+ arena_test {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_TEST
+ minval: 1
+ }
+ }
+}
diff --git a/malloc/Makefile b/malloc/Makefile
index 4d5c81d..de9aafe 100644
--- a/malloc/Makefile
+++ b/malloc/Makefile
@@ -31,6 +31,8 @@ tests := mallocbug tst-malloc tst-valloc tst-calloc tst-obstack \
tst-malloc-backtrace tst-malloc-thread-exit \
tst-malloc-thread-fail tst-malloc-fork-deadlock \
tst-mallocfork2
+tests-static := tst-malloc-usable-static
+tests += $(tests-static)
test-srcs = tst-mtrace
routines = malloc morecore mcheck mtrace obstack \
@@ -138,6 +140,7 @@ endif
tst-mcheck-ENV = MALLOC_CHECK_=3
tst-malloc-usable-ENV = MALLOC_CHECK_=3
+tst-malloc-usable-static-ENV = $(tst-malloc-usable-ENV)
# Uncomment this for test releases. For public releases it is too expensive.
#CPPFLAGS-malloc.o += -DMALLOC_DEBUG=1
diff --git a/malloc/arena.c b/malloc/arena.c
index 4e16593..ddca8a7 100644
--- a/malloc/arena.c
+++ b/malloc/arena.c
@@ -19,6 +19,11 @@
#include <stdbool.h>
+#if HAVE_TUNABLES
+# define TUNABLE_NAMESPACE malloc
+# include <elf/dl-tunables.h>
+#endif
+
/* Compile-time constants. */
#define HEAP_MIN_SIZE (32 * 1024)
@@ -204,6 +209,34 @@ __malloc_fork_unlock_child (void)
mutex_init (&list_lock);
}
+#if HAVE_TUNABLES
+static inline int do_set_mallopt_check (int32_t value);
+void
+DL_TUNABLE_CALLBACK (set_mallopt_check) (void *valp)
+{
+ int32_t value = *(int32_t *) valp;
+ do_set_mallopt_check (value);
+ if (check_action != 0)
+ __malloc_check_init ();
+}
+
+# define DL_TUNABLE_CALLBACK_FNDECL(__name, __type) \
+static inline int do_ ## __name (__type value); \
+void \
+DL_TUNABLE_CALLBACK (__name) (void *valp) \
+{ \
+ __type value = *(__type *) valp; \
+ do_ ## __name (value); \
+}
+
+DL_TUNABLE_CALLBACK_FNDECL (set_mmap_threshold, size_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_mmaps_max, int32_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_top_pad, size_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_perturb_byte, int32_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_trim_threshold, size_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_arena_max, size_t)
+DL_TUNABLE_CALLBACK_FNDECL (set_arena_test, size_t)
+#else
/* Initialization routine. */
#include <string.h>
extern char **_environ;
@@ -238,6 +271,7 @@ next_env_entry (char ***position)
return result;
}
+#endif
#ifdef SHARED
@@ -272,6 +306,24 @@ ptmalloc_init (void)
#endif
thread_arena = &main_arena;
+
+#if HAVE_TUNABLES
+ /* Ensure initialization/consolidation and do it under a lock so that a
+ thread attempting to use the arena in parallel waits on us till we
+ finish. */
+ mutex_lock (&main_arena.mutex);
+ malloc_consolidate (&main_arena);
+
+ TUNABLE_SET_VAL_WITH_CALLBACK (check, NULL, set_mallopt_check);
+ TUNABLE_SET_VAL_WITH_CALLBACK (top_pad, NULL, set_top_pad);
+ TUNABLE_SET_VAL_WITH_CALLBACK (perturb, NULL, set_perturb_byte);
+ TUNABLE_SET_VAL_WITH_CALLBACK (mmap_threshold, NULL, set_mmap_threshold);
+ TUNABLE_SET_VAL_WITH_CALLBACK (trim_threshold, NULL, set_trim_threshold);
+ TUNABLE_SET_VAL_WITH_CALLBACK (mmap_max, NULL, set_mmaps_max);
+ TUNABLE_SET_VAL_WITH_CALLBACK (arena_max, NULL, set_arena_max);
+ TUNABLE_SET_VAL_WITH_CALLBACK (arena_test, NULL, set_arena_test);
+ mutex_unlock (&main_arena.mutex);
+#else
const char *s = NULL;
if (__glibc_likely (_environ != NULL))
{
@@ -340,6 +392,8 @@ ptmalloc_init (void)
if (check_action != 0)
__malloc_check_init ();
}
+#endif
+
#if HAVE_MALLOC_INIT_HOOK
void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook);
if (hook != NULL)
diff --git a/malloc/malloc.c b/malloc/malloc.c
index bb52b3e..002c916 100644
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -4724,6 +4724,90 @@ __malloc_stats (void)
/*
------------------------------ mallopt ------------------------------
*/
+static inline int
+__always_inline
+do_set_mallopt_check (int32_t value)
+{
+ LIBC_PROBE (memory_mallopt_check_action, 2, value, check_action);
+ check_action = value;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_mmap_threshold (size_t value)
+{
+ if (value <= HEAP_MAX_SIZE / 2)
+ {
+ LIBC_PROBE (memory_mallopt_mmap_threshold, 3, value, mp_.mmap_threshold,
+ mp_.no_dyn_threshold);
+ mp_.mmap_threshold = value;
+ mp_.no_dyn_threshold = 1;
+ return 1;
+ }
+ return 0;
+}
+
+static inline int
+__always_inline
+do_set_mmaps_max (int32_t value)
+{
+ LIBC_PROBE (memory_mallopt_mmap_max, 3, value, mp_.n_mmaps_max,
+ mp_.no_dyn_threshold);
+ mp_.n_mmaps_max = value;
+ mp_.no_dyn_threshold = 1;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_top_pad (size_t value)
+{
+ LIBC_PROBE (memory_mallopt_top_pad, 3, value, mp_.top_pad,
+ mp_.no_dyn_threshold);
+ mp_.top_pad = value;
+ mp_.no_dyn_threshold = 1;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_perturb_byte (int32_t value)
+{
+ LIBC_PROBE (memory_mallopt_perturb, 2, value, perturb_byte);
+ perturb_byte = value;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_trim_threshold (size_t value)
+{
+ LIBC_PROBE (memory_mallopt_trim_threshold, 3, value, mp_.trim_threshold,
+ mp_.no_dyn_threshold);
+ mp_.trim_threshold = value;
+ mp_.no_dyn_threshold = 1;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_arena_max (size_t value)
+{
+ LIBC_PROBE (memory_mallopt_arena_max, 2, value, mp_.arena_max);
+ mp_.arena_max = value;
+ return 1;
+}
+
+static inline int
+__always_inline
+do_set_arena_test (size_t value)
+{
+ LIBC_PROBE (memory_mallopt_arena_test, 2, value, mp_.arena_test);
+ mp_.arena_test = value;
+ return 1;
+}
+
int
__libc_mallopt (int param_number, int value)
@@ -4752,63 +4836,37 @@ __libc_mallopt (int param_number, int value)
break;
case M_TRIM_THRESHOLD:
- LIBC_PROBE (memory_mallopt_trim_threshold, 3, value,
- mp_.trim_threshold, mp_.no_dyn_threshold);
- mp_.trim_threshold = value;
- mp_.no_dyn_threshold = 1;
+ do_set_trim_threshold (value);
break;
case M_TOP_PAD:
- LIBC_PROBE (memory_mallopt_top_pad, 3, value,
- mp_.top_pad, mp_.no_dyn_threshold);
- mp_.top_pad = value;
- mp_.no_dyn_threshold = 1;
+ do_set_top_pad (value);
break;
case M_MMAP_THRESHOLD:
- /* Forbid setting the threshold too high. */
- if ((unsigned long) value > HEAP_MAX_SIZE / 2)
- res = 0;
- else
- {
- LIBC_PROBE (memory_mallopt_mmap_threshold, 3, value,
- mp_.mmap_threshold, mp_.no_dyn_threshold);
- mp_.mmap_threshold = value;
- mp_.no_dyn_threshold = 1;
- }
+ res = do_set_mmap_threshold (value);
break;
case M_MMAP_MAX:
- LIBC_PROBE (memory_mallopt_mmap_max, 3, value,
- mp_.n_mmaps_max, mp_.no_dyn_threshold);
- mp_.n_mmaps_max = value;
- mp_.no_dyn_threshold = 1;
+ do_set_mmaps_max (value);
break;
case M_CHECK_ACTION:
- LIBC_PROBE (memory_mallopt_check_action, 2, value, check_action);
- check_action = value;
+ do_set_mallopt_check (value);
break;
case M_PERTURB:
- LIBC_PROBE (memory_mallopt_perturb, 2, value, perturb_byte);
- perturb_byte = value;
+ do_set_perturb_byte (value);
break;
case M_ARENA_TEST:
if (value > 0)
- {
- LIBC_PROBE (memory_mallopt_arena_test, 2, value, mp_.arena_test);
- mp_.arena_test = value;
- }
+ do_set_arena_test (value);
break;
case M_ARENA_MAX:
if (value > 0)
- {
- LIBC_PROBE (memory_mallopt_arena_max, 2, value, mp_.arena_max);
- mp_.arena_max = value;
- }
+ do_set_arena_test (value);
break;
}
(void) mutex_unlock (&av->mutex);
diff --git a/malloc/tst-malloc-usable-static.c b/malloc/tst-malloc-usable-static.c
new file mode 100644
index 0000000..8907db0
--- /dev/null
+++ b/malloc/tst-malloc-usable-static.c
@@ -0,0 +1 @@
+#include <malloc/tst-malloc-usable.c>
diff --git a/manual/install.texi b/manual/install.texi
index 79ee45f..065b082 100644
--- a/manual/install.texi
+++ b/manual/install.texi
@@ -189,6 +189,11 @@ configure with @option{--disable-werror}.
By default for x86_64, @theglibc{} is built with vector math library.
Use this option to disable vector math library.
+@item --enable-tunables
+Tunables support allows additional library parameters to be customized at
+runtime. This is an experimental feature and affects startup time and is thus
+disabled by default.
+
@item --build=@var{build-system}
@itemx --host=@var{host-system}
These options are for cross-compiling. If you specify both options and
diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
new file mode 100644
index 0000000..b65b5a4
--- /dev/null
+++ b/scripts/gen-tunables.awk
@@ -0,0 +1,157 @@
+# Generate dl-tunable-list.h from dl-tunables.list
+
+BEGIN {
+ tunable=""
+ ns=""
+ top_ns=""
+}
+
+# Skip over blank lines and comments.
+/^#/ {
+ next
+}
+
+/^[ \t]*$/ {
+ next
+}
+
+# Beginning of either a top namespace, tunable namespace or a tunable, decided
+# on the current value of TUNABLE, NS or TOP_NS.
+$2 == "{" {
+ if (top_ns == "") {
+ top_ns = $1
+ }
+ else if (ns == "") {
+ ns = $1
+ }
+ else if (tunable == "") {
+ tunable = $1
+ }
+ else {
+ printf ("Unexpected occurrence of '{': %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+
+ next
+}
+
+# End of either a top namespace, tunable namespace or a tunable.
+$1 == "}" {
+ if (tunable != "") {
+ # Tunables definition ended, now fill in default attributes.
+ if (!types[top_ns][ns][tunable]) {
+ types[top_ns][ns][tunable] = "STRING"
+ }
+ if (!minvals[top_ns][ns][tunable]) {
+ minvals[top_ns][ns][tunable] = "0"
+ }
+ if (!maxvals[top_ns][ns][tunable]) {
+ maxvals[top_ns][ns][tunable] = "0"
+ }
+ if (!env_alias[top_ns][ns][tunable]) {
+ env_alias[top_ns][ns][tunable] = "NULL"
+ }
+ if (!is_secure[top_ns][ns][tunable]) {
+ is_secure[top_ns][ns][tunable] = "false"
+ }
+
+ tunable = ""
+ }
+ else if (ns != "") {
+ ns = ""
+ }
+ else if (top_ns != "") {
+ top_ns = ""
+ }
+ else {
+ printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+ next
+}
+
+# Everything else, which could either be a tunable without any attributes or a
+# tunable attribute.
+{
+ if (ns == "") {
+ printf("Line %d: Invalid tunable outside a namespace: %s\n", NR, $0)
+ exit 1
+ }
+
+ if (tunable == "") {
+ # We encountered a tunable without any attributes, so note it with a
+ # default.
+ types[top_ns][ns][$1] = "STRING"
+ next
+ }
+
+ # Otherwise, we have encountered a tunable attribute.
+ split($0, arr, ":")
+ attr = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[1])
+ val = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[2])
+
+ if (attr == "type") {
+ types[top_ns][ns][tunable] = val
+ }
+ else if (attr == "minval") {
+ minvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "maxval") {
+ maxvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "env_alias") {
+ env_alias[top_ns][ns][tunable] = sprintf("\"%s\"", val)
+ }
+ else if (attr == "is_secure") {
+ if (val == "true" || val == "false") {
+ is_secure[top_ns][ns][tunable] = val
+ }
+ else {
+ printf("Line %d: Invalid value (%s) for is_secure: %s, ", NR, val,
+ $0)
+ print("Allowed values are 'true' or 'false'")
+ exit 1
+ }
+ }
+}
+
+END {
+ if (ns != "") {
+ print "Unterminated namespace. Is a closing brace missing?"
+ exit 1
+ }
+
+ print "/* AUTOGENERATED by gen-tunables.awk. */"
+ print "#ifndef _TUNABLES_H_"
+ print "# error \"Do not include this file directly.\""
+ print "# error \"Include tunables.h instead.\""
+ print "#endif"
+
+ # Now, the enum names
+ print "\ntypedef enum"
+ print "{"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, m);
+ }
+ }
+ }
+ print "} tunable_id_t;\n"
+
+ # Finally, the tunable list.
+ print "\n#ifdef TUNABLES_INTERNAL"
+ print "static tunable_t tunable_list[] = {"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" {TUNABLE_NAME_S(%s, %s, %s)", t, n, m)
+ printf (", {TUNABLE_TYPE_%s, %s, %s}, {.numval = 0}, NULL, %s, %s},\n",
+ types[t][n][m], minvals[t][n][m], maxvals[t][n][m],
+ is_secure[t][n][m], env_alias[t][n][m]);
+ }
+ }
+ }
+ print "};"
+ print "#endif"
+}
diff --git a/sysdeps/mach/hurd/dl-sysdep.c b/sysdeps/mach/hurd/dl-sysdep.c
index 76140cf..d2b7422 100644
--- a/sysdeps/mach/hurd/dl-sysdep.c
+++ b/sysdeps/mach/hurd/dl-sysdep.c
@@ -44,6 +44,10 @@
#include <dl-machine.h>
#include <dl-procinfo.h>
+#if HAVE_TUNABLES
+# include <dl-tunables.h>
+#endif
+
extern void __mach_init (void);
extern int _dl_argc;
@@ -143,6 +147,10 @@ _dl_sysdep_start (void **start_argptr,
__libc_enable_secure = _dl_hurd_data->flags & EXEC_SECURE;
+#if HAVE_TUNABLES
+ __tunables_init (_environ);
+#endif
+
if (_dl_hurd_data->flags & EXEC_STACK_ARGS &&
_dl_hurd_data->user_entry == 0)
_dl_hurd_data->user_entry = (vm_address_t) ENTRY_POINT;
diff --git a/sysdeps/mach/hurd/i386/init-first.c b/sysdeps/mach/hurd/i386/init-first.c
index 7a905c2..7723479 100644
--- a/sysdeps/mach/hurd/i386/init-first.c
+++ b/sysdeps/mach/hurd/i386/init-first.c
@@ -31,6 +31,10 @@
#include <ldsodefs.h>
#include <fpu_control.h>
+#if HAVE_TUNABLES
+# include <dl-tunables.h>
+#endif
+
extern void __mach_init (void);
extern void __init_misc (int, char **, char **);
#ifdef USE_NONOPTION_FLAGS
@@ -83,6 +87,10 @@ posixland_init (int argc, char **argv, char **envp)
__libc_argv = argv;
__environ = envp;
+#if HAVE_TUNABLES
+ __tunables_init (__environ);
+#endif
+
#ifndef SHARED
_dl_non_dynamic_init ();
#endif
--
2.7.4
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-11 19:06 ` Carlos O'Donell
@ 2016-07-14 11:27 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-14 11:27 UTC (permalink / raw)
To: Carlos O'Donell; +Cc: libc-alpha
On Mon, Jul 11, 2016 at 03:06:34PM -0400, Carlos O'Donell wrote:
> > +The list of allowed attributes are:
> > +
> > +- type: Data type. Defaults to STRING. Allowed types are:
> > + INT_32, SIZE_T. Note that as of now the STRING type is
> > + neither defined nor supported since no tunables need
> > + it. It will be in future though, when needed.
>
> OK.
>
> I expect we can add more types as time goes on.
Yes, and update this document when we do, which reminds me to update
the fact that STRING is supported.
> > +
> > +- minval: Optional minimum acceptable value
> > +
> > +- maxval: Optional maximum acceptable value
> > +
>
> I assume valid only for numeric types and such values as are valid for the type?
>
> Or does minval/maxval indicate minlength/maxlength for a string as an easy form
> of filtering?
It was only for numbers but I can overload that to mean string size
limits too. As for string size checks, I'll add with the first string
tunable that actually needs it.
> > +- env_alias: An alias environment variable
> > +
> > +- secure_env_alias: Specify whether the environment variable should be read
> > + for setuid binaries.
>
> We should make a few things clear here:
>
> * The tunable may still be ignored in AT_SECURE if the non-default value
> fails to pass some level of runtime checking e.g. value could break the
> application.
>
> * The 'secure_env_alias' should really mean "Specify whether the environment
> variable should be read for setuid binaries, 0 (default) means they are not
> read and 1 means they are. Even if the variable is read it may be ignored."
OK, I'll extend the meaning of that for the tunable in general, not
just its alias. That way, the meaning does not change between the
alias and the tunable.
> Are there any restrictions on the callback? I assume the foreign function (callback)
> is called from _any_ context so you have to be very very very careful about what
> functions you call?
The callback is currently only called from one context: the module
initialization. In future it may get called later, i.e. when a
tunable is changed at runtime. I prefer to think of the latter case
later if that is OK.
> Can we unit test the interface to make sure the API works as designed?
>
> https://sourceware.org/ml/libc-alpha/2016-07/msg00110.html
Let me see if I can do that, but not as part of this patch.
> Can we pass back a void* to return success/failure like the
> pthreads API does? I can see a short term need to know if the
> foreign function call failed.
I can, but can you elaborate on why we would need that? The call is
effectively going to happen in the same context in which the callback
is defined, i.e. we won't have a case where the callback is in one
module and the initialization is in another.
> > +#define RETURN_IF_INVALID_RANGE(__cur, __val) \
> > +({ \
> > + __typeof ((__cur)->type.min) min = (__cur)->type.min; \
> > + __typeof ((__cur)->type.max) max = (__cur)->type.max; \
> > + if (min != max && ((__val) < min || (__val) > max)) \
> > + return; \
> > +})
>
> As mentioned above, might be nice to have min/max limit string min
> and max length.
I'll add string validation when we have an actual tunable that needs it.
> Add "May still be ignored if value is dangerous."
How about "May still be ignored if the value does not meet criteria
for allowed values for the tunable."? That seems more precise since
we don't really have a way to identify dangerous values beyond the
validation we have in place.
> Two things are wrong.
>
> Missing systemtap probes, _and_ wrong settings.
I skipped the probes on purpose thinking that it doesn't make sense to
hit them anyway, but I guess it is required since it changes
behaviour. I'll add it.
>
> For example setting top_pad should disable dynamic thresholding.
Yeah, I missed that. I'll review all of the options and fix up.
>
> Therefore you need to do the following:
>
> (a) Take __libc_mallopt from malloc/malloc.c and split out the
> setting of each variable into a "callback"
>
> (b) Have __libc_mallopt call the callbacks.
>
> (c) Have TUNABLE_SET_VAL call the callback via the tunables
> mechanism.
>
> That way you have code in one place for what has to happen
> when setting the variable.
That's a good idea.
Thanks,
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-13 18:03 ` H.J. Lu
@ 2016-07-14 6:57 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-14 6:57 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Wed, Jul 13, 2016 at 11:03:02AM -0700, H.J. Lu wrote:
> Another thing, please add TUNABLE_WITH_CALLBACK:
>
> TUNABLE_WITH_CALLBACK (ifunc, ifunc);
>
> so that I can have a callback without setting a value.
You can set the val as NULL, I'll add a NULL check.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-13 17:59 ` H.J. Lu
@ 2016-07-13 18:03 ` H.J. Lu
2016-07-14 6:57 ` Siddhesh Poyarekar
0 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-13 18:03 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Wed, Jul 13, 2016 at 10:59 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Wed, Jul 13, 2016 at 10:55 AM, Siddhesh Poyarekar
> <siddhesh@sourceware.org> wrote:
>> On Wed, Jul 13, 2016 at 10:37:56AM -0700, H.J. Lu wrote:
>>> This scheme doesn't work well for IFUNC. cpu_features must be
>>> updated very early after init_cpu_features is called. But __tunables_init
>>> is called too late. Can it be moved before init_cpu_features so that
>>> cpu_features can be updated at the end of init_cpu_features?
>>
>> Sure, I'll move it further up.
>
> Please make sure that it is called before init_cpu_features for both
> static and dynamic executables.
>
Another thing, please add TUNABLE_WITH_CALLBACK:
TUNABLE_WITH_CALLBACK (ifunc, ifunc);
so that I can have a callback without setting a value.
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-13 17:56 ` Siddhesh Poyarekar
@ 2016-07-13 17:59 ` H.J. Lu
2016-07-13 18:03 ` H.J. Lu
0 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-13 17:59 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Wed, Jul 13, 2016 at 10:55 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> On Wed, Jul 13, 2016 at 10:37:56AM -0700, H.J. Lu wrote:
>> This scheme doesn't work well for IFUNC. cpu_features must be
>> updated very early after init_cpu_features is called. But __tunables_init
>> is called too late. Can it be moved before init_cpu_features so that
>> cpu_features can be updated at the end of init_cpu_features?
>
> Sure, I'll move it further up.
Please make sure that it is called before init_cpu_features for both
static and dynamic executables.
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-13 17:38 ` H.J. Lu
@ 2016-07-13 17:56 ` Siddhesh Poyarekar
2016-07-13 17:59 ` H.J. Lu
0 siblings, 1 reply; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-13 17:56 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Wed, Jul 13, 2016 at 10:37:56AM -0700, H.J. Lu wrote:
> This scheme doesn't work well for IFUNC. cpu_features must be
> updated very early after init_cpu_features is called. But __tunables_init
> is called too late. Can it be moved before init_cpu_features so that
> cpu_features can be updated at the end of init_cpu_features?
Sure, I'll move it further up.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-13 14:49 ` Siddhesh Poyarekar
@ 2016-07-13 17:38 ` H.J. Lu
2016-07-13 17:56 ` Siddhesh Poyarekar
0 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-13 17:38 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Wed, Jul 13, 2016 at 7:49 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> On Wed, Jul 13, 2016 at 07:43:22AM -0700, H.J. Lu wrote:
>> On x86-64, I got
>>
>> FAIL: malloc/tst-malloc-usable-static-tunables
>> FAIL: malloc/tst-malloc-usable-tunables
>>
>> [hjl@gnu-6 build-x86_64-linux]$ cat malloc/tst-malloc-usable-static-tunables.out
>> malloc_usable_size: expected 7 but got 24
>> [hjl@gnu-6 build-x86_64-linux]$ cat malloc/tst-malloc-usable-tunables.out
>> malloc_usable_size: expected 7 but got 24
>> [hjl@gnu-6 build-x86_64-linux]$
>>
>> I used
>>
>> BUILD_CC="gcc" CC="gcc" CXX="g++" CFLAGS="-O3 -g"
>> /export/gnu/import/git/sources/glibc/configure --prefix=/usr
>> --without-cvs --without-selinux --target=x86_64-linux
>> --build=x86_64-linux --host=x86_64-linux
>> --enable-hardcoded-path-in-tests --enable-large-benchtests
>
> I test on x86_64, so that's odd, maybe due to
> --enable-hardcoded-path-in-tests. I'll take a look.
>
This scheme doesn't work well for IFUNC. cpu_features must be
updated very early after init_cpu_features is called. But __tunables_init
is called too late. Can it be moved before init_cpu_features so that
cpu_features can be updated at the end of init_cpu_features?
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-13 14:43 ` H.J. Lu
@ 2016-07-13 14:49 ` Siddhesh Poyarekar
2016-07-13 17:38 ` H.J. Lu
0 siblings, 1 reply; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-13 14:49 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Wed, Jul 13, 2016 at 07:43:22AM -0700, H.J. Lu wrote:
> On x86-64, I got
>
> FAIL: malloc/tst-malloc-usable-static-tunables
> FAIL: malloc/tst-malloc-usable-tunables
>
> [hjl@gnu-6 build-x86_64-linux]$ cat malloc/tst-malloc-usable-static-tunables.out
> malloc_usable_size: expected 7 but got 24
> [hjl@gnu-6 build-x86_64-linux]$ cat malloc/tst-malloc-usable-tunables.out
> malloc_usable_size: expected 7 but got 24
> [hjl@gnu-6 build-x86_64-linux]$
>
> I used
>
> BUILD_CC="gcc" CC="gcc" CXX="g++" CFLAGS="-O3 -g"
> /export/gnu/import/git/sources/glibc/configure --prefix=/usr
> --without-cvs --without-selinux --target=x86_64-linux
> --build=x86_64-linux --host=x86_64-linux
> --enable-hardcoded-path-in-tests --enable-large-benchtests
I test on x86_64, so that's odd, maybe due to
--enable-hardcoded-path-in-tests. I'll take a look.
Thanks,
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-11 16:41 ` Siddhesh Poyarekar
@ 2016-07-13 14:43 ` H.J. Lu
2016-07-13 14:49 ` Siddhesh Poyarekar
0 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-13 14:43 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Mon, Jul 11, 2016 at 9:41 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> On Mon, Jul 11, 2016 at 09:25:07AM -0700, H.J. Lu wrote:
>> Did you update siddhesh/tunables branch? It still has the old patches.
>
> I have now, sorry I keep forgetting.
On x86-64, I got
FAIL: malloc/tst-malloc-usable-static-tunables
FAIL: malloc/tst-malloc-usable-tunables
[hjl@gnu-6 build-x86_64-linux]$ cat malloc/tst-malloc-usable-static-tunables.out
malloc_usable_size: expected 7 but got 24
[hjl@gnu-6 build-x86_64-linux]$ cat malloc/tst-malloc-usable-tunables.out
malloc_usable_size: expected 7 but got 24
[hjl@gnu-6 build-x86_64-linux]$
I used
BUILD_CC="gcc" CC="gcc" CXX="g++" CFLAGS="-O3 -g"
/export/gnu/import/git/sources/glibc/configure --prefix=/usr
--without-cvs --without-selinux --target=x86_64-linux
--build=x86_64-linux --host=x86_64-linux
--enable-hardcoded-path-in-tests --enable-large-benchtests
to configure glibc.
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-11 16:39 ` Siddhesh Poyarekar
2016-07-11 19:13 ` DJ Delorie
@ 2016-07-12 9:00 ` Florian Weimer
1 sibling, 0 replies; 49+ messages in thread
From: Florian Weimer @ 2016-07-12 9:00 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: libc-alpha
On 07/11/2016 06:39 PM, Siddhesh Poyarekar wrote:
>>> > > +- secure_env_alias: Specify whether the environment variable should be read
>>> > > + for setuid binaries.
>> >
>> > This needs to describe the exact meaning of the choices, and the default.
>> > I'm not sure if the current/name values are intuitive.
> It specifies if the env_alias should be read even for setuid binaries.
> This is currently only useful for MALLOC_CHECK_ since it is read even
> for setuid binaries. Would env_alias_is_secure be clearer?
No, because I think the sense is reverted:
+ check {
+ type: INT_32
+ minval: 0
+ maxval: 3
+ env_alias: MALLOC_CHECK_
+ secure_env_alias: true
+ }
That's why I said you need to document what the options are and what
they do in AT_SECURE mode, and the default value.
Thanks,
Florian
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-11 16:05 ` Florian Weimer
2016-07-11 16:39 ` Siddhesh Poyarekar
@ 2016-07-12 8:51 ` Florian Weimer
1 sibling, 0 replies; 49+ messages in thread
From: Florian Weimer @ 2016-07-12 8:51 UTC (permalink / raw)
To: Siddhesh Poyarekar, libc-alpha
On 07/11/2016 06:05 PM, Florian Weimer wrote:
> This is indeed closer to what I had in mind. But what I really want is
> something that can patch the tunables directly into a place which malloc
> can use. My hope is that one day, we have a tunable descriptor inside
> libc.so, and (rarely written) range in the .data segment which contains
> the tunable variables. Code which relies on tunables directly access
> the tunable range (using relaxed MO loads), and some external tool can
> use the tunables descriptor to update the tunables at run time.
Just to clarify: I don't see this an obstacle to applying the present
patch. I merely hope we can agree this is a direction we want to move in.
The main problem I see with the patch is the lack of equivalence with
the mallopt-based approach.
Thanks,
Florian
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-11 16:39 ` Siddhesh Poyarekar
@ 2016-07-11 19:13 ` DJ Delorie
2016-07-12 9:00 ` Florian Weimer
1 sibling, 0 replies; 49+ messages in thread
From: DJ Delorie @ 2016-07-11 19:13 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: libc-alpha
Siddhesh Poyarekar <siddhesh@sourceware.org> writes:
>> You do not need __ prefixes on macro arguments.
>
> I did it out of habit to avoid namespace collisions, but it's not
> necessary here so I'll remove it.
I've been burned by unprotected parameter names in headers too (although
not macros); it's a good habit to include the underscores, even if not
needed.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-09 18:49 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
` (2 preceding siblings ...)
2016-07-11 16:56 ` Paul E. Murphy
@ 2016-07-11 19:06 ` Carlos O'Donell
2016-07-14 11:27 ` Siddhesh Poyarekar
3 siblings, 1 reply; 49+ messages in thread
From: Carlos O'Donell @ 2016-07-11 19:06 UTC (permalink / raw)
To: Siddhesh Poyarekar, libc-alpha
On 07/09/2016 02:37 PM, Siddhesh Poyarekar wrote:
> The tunables framework allows us to uniformly manage and expose global
> variables inside glibc as switches to users. tunables/README has
> instructions for glibc developers to add new tunables.
>
> Tunables support can be enabled by passing the --enable-tunables
> configure flag to the configure script. This patch only adds a
> framework and does not pose any limitations on how tunable values are
> read from the user. It also adds environment variables used in malloc
> behaviour tweaking to the tunables framework as a PoC of the
> compatibility interface.
Thank you for doing this work and posting these patches.
Overall the patch looks great.
There is are only two high-level things:
* Callback should be able to return error. I have some immediate
needs for that. You can ignore this comment if you want.
* Malloc tunables are wrong. See my comments.
> * manual/install.texi: Add --enable-tunables option.
> * INSTALL: Regenerate.
> * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
> (before-compile): Generate dl-tunable-list.h early.
> * config.h.in: Add BUILD_TUNABLES.
> * config.make.in: Add build-tunables.
> * configure.ac: Add --enable-tunables option.
> * configure: Regenerate.
> * malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
> Define TUNABLE_NAMESPACE.
> (DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
> (ptmalloc_init): Set tunable values.
> * malloc/tst-malloc-usable-static.c: New test case.
> * malloc/Makefile (tests-static): Add it.
> * csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
> (__libc_init_first) [!SHARED]: Initialize tunables for static
> binaries.
> * scripts/gen-tunables.awk: New file.
> * README.tunables: New file.
> * elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
> namespace.
> * elf/dl-tunable-list.h: New auto-generated file.
> * elf/dl-tunables.c: New file.
> * elf/dl-tunables.h: New file.
> * elf/dl-tunables.list: New file.
> * elf/dl-tunable-types.h: New file.
> * elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
> (process_envvars): Call __tunables_init.
> ---
> INSTALL | 6 ++
> Makeconfig | 16 ++++
> README.tunables | 74 ++++++++++++++++
> config.h.in | 3 +
> config.make.in | 1 +
> configure | 16 ++++
> configure.ac | 10 +++
> csu/init-first.c | 7 ++
> elf/Makefile | 5 ++
> elf/Versions | 3 +
> elf/dl-tunable-types.h | 46 ++++++++++
> elf/dl-tunables.c | 182 ++++++++++++++++++++++++++++++++++++++
> elf/dl-tunables.h | 76 ++++++++++++++++
> elf/dl-tunables.list | 50 +++++++++++
> elf/rtld.c | 8 ++
> malloc/Makefile | 3 +
> malloc/arena.c | 35 ++++++++
> malloc/tst-malloc-usable-static.c | 1 +
> manual/install.texi | 5 ++
> scripts/gen-tunables.awk | 157 ++++++++++++++++++++++++++++++++
> 20 files changed, 704 insertions(+)
> create mode 100644 README.tunables
> create mode 100644 elf/dl-tunable-types.h
> create mode 100644 elf/dl-tunables.c
> create mode 100644 elf/dl-tunables.h
> create mode 100644 elf/dl-tunables.list
> create mode 100644 malloc/tst-malloc-usable-static.c
> create mode 100644 scripts/gen-tunables.awk
>
> diff --git a/INSTALL b/INSTALL
> index ec3445f..90dc481 100644
> --- a/INSTALL
> +++ b/INSTALL
> @@ -158,6 +158,12 @@ will be used, and CFLAGS sets optimization options for the compiler.
> By default for x86_64, the GNU C Library is built with vector math
> library. Use this option to disable vector math library.
>
> +'--enable-tunables'
> + Tunables support allows additional library parameters to be
> + customized at runtime for each application. This is an
s/for each application//g
> + experimental feature and affects startup time and is thus disabled
> + by default.
> +
> '--build=BUILD-SYSTEM'
> '--host=HOST-SYSTEM'
> These options are for cross-compiling. If you specify both options
> diff --git a/Makeconfig b/Makeconfig
> index 58bd3b3..732b870 100644
> --- a/Makeconfig
> +++ b/Makeconfig
> @@ -895,6 +895,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \
> $(foreach lib,$(libof-$(basename $(@F))) \
> $(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \
> $(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F)))
> +
> +ifeq (yes,$(build-tunables))
> +CPPFLAGS += -DTOP_NAMESPACE=glibc
> +endif
> +
OK.
> override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \
> $(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \
> $(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \
> @@ -1067,6 +1072,17 @@ $(common-objpfx)libc-modules.stmp: $(..)scripts/gen-libc-modules.awk \
>
> endif
>
> +# Build the tunables list header early since it could be used by any module in
> +# glibc.
> +ifeq (yes,$(build-tunables))
> +before-compile += $(common-objpfx)dl-tunable-list.h
> +
> +$(common-objpfx)dl-tunable-list.h: $(..)scripts/gen-tunables.awk \
> + $(..)elf/dl-tunables.list
> + $(AWK) -f $^ > $@.tmp
> + mv $@{.tmp,}
> +endif
OK. I like the header list.
> +
> common-generated += libc-modules.h libc-modules.stmp
>
> # The name under which the run-time dynamic linker is installed.
> diff --git a/README.tunables b/README.tunables
> new file mode 100644
> index 0000000..8890249
> --- /dev/null
> +++ b/README.tunables
> @@ -0,0 +1,74 @@
> + TUNABLE FRAMEWORK
> + =================
> +
> +The tunable framework allows modules within glibc to register variables that
> +may be tweaked through an environment variable or an API call. It aims to
> +enforce a strict namespace rule to bring consistency to naming of these tunable
> +environment variables across the project.
The reason for tunables is to allow application authors to alter
the runtime library behaviour to match their workload. We should
say that up front since that notion will guide the choices we make.
See "Why?":
https://sourceware.org/glibc/wiki/TuningLibraryRuntimeBehavior
> +
> +ADDING A NEW TUNABLE
> +--------------------
> +
> +The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in
> +distributions for specific tunables if they want to add their own tunables.
> +Downstream implementations are discouraged from using the 'glibc' namespace for
> +tunables they don't already have consensus to push upstream.
> +
> +There are two steps to adding a tunable:
> +
> +1. Add a tunable ID:
> +
> +Modules that wish to use the tunables interface must define the
> +TUNABLE_NAMESPACE macro. Following this, for each tunable you want to
> +add, make an entry in elf/dl-tunables.list. The format of the file is as
> +follows:
> +
> +TOP_NAMESPACE {
> + NAMESPACE1 {
> + TUNABLE1 {
> + # tunable attributes, one per line
> + }
> + # A tunable with default attributes, i.e. string variable.
> + TUNABLE2
> + TUNABLE3 {
> + # its attributes
> + }
> + }
> + NAMESPACE2 {
> + ...
> + }
> +}
> +
> +The list of allowed attributes are:
> +
> +- type: Data type. Defaults to STRING. Allowed types are:
> + INT_32, SIZE_T. Note that as of now the STRING type is
> + neither defined nor supported since no tunables need
> + it. It will be in future though, when needed.
OK.
I expect we can add more types as time goes on.
> +
> +- minval: Optional minimum acceptable value
> +
> +- maxval: Optional maximum acceptable value
> +
I assume valid only for numeric types and such values as are valid for the type?
Or does minval/maxval indicate minlength/maxlength for a string as an easy form
of filtering?
> +- env_alias: An alias environment variable
> +
> +- secure_env_alias: Specify whether the environment variable should be read
> + for setuid binaries.
We should make a few things clear here:
* The tunable may still be ignored in AT_SECURE if the non-default value
fails to pass some level of runtime checking e.g. value could break the
application.
* The 'secure_env_alias' should really mean "Specify whether the environment
variable should be read for setuid binaries, 0 (default) means they are not
read and 1 means they are. Even if the variable is read it may be ignored."
> +
> +2. Call either the TUNABLE_SET_VALUE and pass into it the tunable name and a
> + pointer to the variable that should be set with the tunable value.
> + If additional work needs to be done after setting the value, use the
> + TUNABLE_SET_VALUE_WITH_CALLBACK instead and additionally pass a pointer to
> + the function that should be called if the tunable value has been set.
> +
Are there any restrictions on the callback? I assume the foreign function (callback)
is called from _any_ context so you have to be very very very careful about what
functions you call?
> +FUTURE WORK
> +-----------
> +
> +The framework currently only allows a one-time initialization of variables
> +through environment variables and in some cases, modification of variables via
> +an API call. A future goals for this project include:
> +
> +- Setting system-wide and user-wide defaults for tunables through some
> + mechanism like a configuration file.
> +
> +- Allow tweaking of some tunables at runtime
OK.
> diff --git a/config.h.in b/config.h.in
> index 856ef6a..6112782 100644
> --- a/config.h.in
> +++ b/config.h.in
> @@ -240,4 +240,7 @@
> /* PowerPC32 uses fctidz for floating point to long long conversions. */
> #define HAVE_PPC_FCTIDZ 0
>
> +/* Build glibc with tunables support. */
> +#define BUILD_TUNABLES 0
> +
Suggest 'HAVE_TUNABLES' following the other HAVE_* defines.
> #endif
> diff --git a/config.make.in b/config.make.in
> index 95c6f36..653ef6f 100644
> --- a/config.make.in
> +++ b/config.make.in
> @@ -91,6 +91,7 @@ use-nscd = @use_nscd@
> build-hardcoded-path-in-tests= @hardcoded_path_in_tests@
> build-pt-chown = @build_pt_chown@
> enable-lock-elision = @enable_lock_elision@
> +build-tunables = @build_tunables@
Since tunables is less about "building" and more about "having"
them, again total bikeshed, have-tunables.
>
> # Build tools.
> CC = @CC@
> diff --git a/configure b/configure
> index 19a4829..1534ba7 100755
> --- a/configure
> +++ b/configure
> @@ -659,6 +659,7 @@ multi_arch
> base_machine
> add_on_subdirs
> add_ons
> +build_tunables
> build_pt_chown
> build_nscd
> link_obsolete_rpc
> @@ -773,6 +774,7 @@ enable_systemtap
> enable_build_nscd
> enable_nscd
> enable_pt_chown
> +enable_tunables
> enable_mathvec
> with_cpu
> '
> @@ -1440,6 +1442,7 @@ Optional Features:
> --disable-build-nscd disable building and installing the nscd daemon
> --disable-nscd library functions will not contact the nscd daemon
> --enable-pt_chown Enable building and installing pt_chown
> + --enable-tunables Enable tunables support
> --enable-mathvec Enable building and installing mathvec [default
> depends on architecture]
>
> @@ -3646,6 +3649,19 @@ if test "$build_pt_chown" = yes; then
>
> fi
>
> +# Check whether --enable-tunables was given.
> +if test "${enable_tunables+set}" = set; then :
> + enableval=$enable_tunables; build_tunables=$enableval
> +else
> + build_tunables=no
> +fi
> +
> +
> +if test "$build_tunables" = yes; then
> + $as_echo "#define BUILD_TUNABLES 1" >>confdefs.h
> +
> +fi
> +
> # The abi-tags file uses a fairly simplistic model for name recognition that
> # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
> # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
> diff --git a/configure.ac b/configure.ac
> index 123f0d2..b2aaf9b 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -395,6 +395,16 @@ if test "$build_pt_chown" = yes; then
> AC_DEFINE(HAVE_PT_CHOWN)
> fi
>
> +AC_ARG_ENABLE([tunables],
> + [AS_HELP_STRING([--enable-tunables],
> + [Enable tunables support])],
> + [build_tunables=$enableval],
> + [build_tunables=no])
> +AC_SUBST(build_tunables)
> +if test "$build_tunables" = yes; then
> + AC_DEFINE(BUILD_TUNABLES)
> +fi
> +
> # The abi-tags file uses a fairly simplistic model for name recognition that
> # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
> # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
> diff --git a/csu/init-first.c b/csu/init-first.c
> index 77c6e1c..7427121 100644
> --- a/csu/init-first.c
> +++ b/csu/init-first.c
> @@ -28,6 +28,9 @@
> #include <libc-internal.h>
>
> #include <ldsodefs.h>
> +#if BUILD_TUNABLES
> +# include <elf/dl-tunables.h>
> +#endif
OK.
>
> /* Set nonzero if we have to be prepared for more than one libc being
> used in the process. Safe assumption if initializer never runs. */
> @@ -74,6 +77,10 @@ _init (int argc, char **argv, char **envp)
> #ifndef SHARED
> __libc_init_secure ();
>
> +#if BUILD_TUNABLES
> + __tunables_init (envp);
OK, very early initialization.
> +#endif
> +
> /* First the initialization which normally would be done by the
> dynamic linker. */
> _dl_non_dynamic_init ();
> diff --git a/elf/Makefile b/elf/Makefile
> index 210dde9..c634d120 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -35,6 +35,11 @@ dl-routines = $(addprefix dl-,load lookup object reloc deps hwcaps \
> ifeq (yes,$(use-ldconfig))
> dl-routines += dl-cache
> endif
> +
> +ifeq (yes,$(build-tunables))
> +dl-routines += dl-tunables
> +endif
OK.
Can we unit test the interface to make sure the API works as designed?
https://sourceware.org/ml/libc-alpha/2016-07/msg00110.html
> +
> all-dl-routines = $(dl-routines) $(sysdep-dl-routines)
> # But they are absent from the shared libc, because that code is in ld.so.
> elide-routines.os = $(all-dl-routines) dl-support enbl-secure dl-origin \
> diff --git a/elf/Versions b/elf/Versions
> index 23deda9..1734697 100644
> --- a/elf/Versions
> +++ b/elf/Versions
> @@ -64,5 +64,8 @@ ld {
>
> # Pointer protection.
> __pointer_chk_guard;
> +
> + # Set value of a tunable.
> + __tunable_set_val;
OK.
> }
> }
> diff --git a/elf/dl-tunable-types.h b/elf/dl-tunable-types.h
> new file mode 100644
> index 0000000..d1591b6
> --- /dev/null
> +++ b/elf/dl-tunable-types.h
> @@ -0,0 +1,46 @@
> +/* Tunable type information.
> +
> + Copyright (C) 2016 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <http://www.gnu.org/licenses/>. */
> +
> +#ifndef _TUNABLE_TYPES_H_
> +# define _TUNABLE_TYPES_H_
> +#include <stddef.h>
> +
> +typedef void (*tunable_callback_t) (void *);
Can we pass back a void* to return success/failure like the
pthreads API does? I can see a short term need to know if the
foreign function call failed.
> +
> +typedef enum
> +{
> + TUNABLE_TYPE_INT_32,
> + TUNABLE_TYPE_SIZE_T,
> + TUNABLE_TYPE_STRING
> +} tunable_type_code_t;
OK.
> +
> +typedef struct
> +{
> + tunable_type_code_t type_code;
> + int64_t min;
> + int64_t max;
> +} tunable_type_t;
OK.
> +
> +typedef union
> +{
> + int64_t numval;
> + const char *strval;
> +} tunable_val_t;
> +
> +#endif
> diff --git a/elf/dl-tunables.c b/elf/dl-tunables.c
> new file mode 100644
> index 0000000..ee8c2f0
> --- /dev/null
> +++ b/elf/dl-tunables.c
> @@ -0,0 +1,182 @@
> +/* The tunable framework. See the README to know how to use the tunable in
> + a glibc module.
> +
> + Copyright (C) 2016 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <http://www.gnu.org/licenses/>. */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <unistd.h>
> +#include <stdlib.h>
> +
> +#define TUNABLES_INTERNAL 1
> +#include "dl-tunables.h"
> +
> +/* Compare environment names, bounded by the name hardcoded in glibc. */
> +static bool
> +is_name (const char *orig, const char *envname)
> +{
> + for (;*orig != '\0' && *envname != '\0'; envname++, orig++)
> + if (*orig != *envname)
> + break;
> +
> + /* The ENVNAME is immediately followed by a value. */
> + if (*orig == '\0' && *envname == '=')
> + return true;
> + else
> + return false;
> +}
OK.
> +
> +static bool
> +get_next_env (char ***envp, char **name, size_t *namelen, char **val)
> +{
> + char **ev = *envp;
> +
> + while (ev != NULL && *ev != '\0')
> + {
> + char *envline = *ev;
> + int len = 0;
> +
> + while (envline[len] != '\0' && envline[len] != '=')
> + len++;
> +
> + /* Just the name and no value, go to the next one. */
> + if (envline[len] == '\0')
> + continue;
> +
> + *name = envline;
> + *namelen = len;
> + *val = &envline[len + 1];
> + *envp = ++ev;
> +
> + return true;
> + }
> +
> + return false;
> +}
OK.
> +
> +
> +#define RETURN_IF_INVALID_RANGE(__cur, __val) \
> +({ \
> + __typeof ((__cur)->type.min) min = (__cur)->type.min; \
> + __typeof ((__cur)->type.max) max = (__cur)->type.max; \
> + if (min != max && ((__val) < min || (__val) > max)) \
> + return; \
> +})
As mentioned above, might be nice to have min/max limit string min
and max length.
> +
> +/* Validate range of the input value and initialize the tunable CUR if it looks
> + good. */
> +static void
> +tunable_initialize (tunable_t *cur)
> +{
> + switch (cur->type.type_code)
> + {
> + case TUNABLE_TYPE_INT_32:
> + {
> + int32_t val = (int32_t) __strtoul_internal (cur->strval, NULL, 0, 0);
> + RETURN_IF_INVALID_RANGE (cur, val);
> + cur->val.numval = (int64_t) val;
> + break;
> + }
> + case TUNABLE_TYPE_SIZE_T:
> + {
> + size_t val = (size_t) __strtoul_internal (cur->strval, NULL, 0, 0);
> + RETURN_IF_INVALID_RANGE (cur, val);
> + cur->val.numval = (int64_t) val;
> + break;
> + }
> + case TUNABLE_TYPE_STRING:
> + {
> + cur->val.strval = cur->strval;
> + break;
> + }
> + default:
> + __builtin_unreachable ();
> + }
> +}
OK.
> +
> +/* Initialize the tunables list from the environment. For now we only use the
> + ENV_ALIAS to find values. Later we will also use the tunable names to find
> + values. */
> +void
> +__tunables_init (char **envp)
> +{
> + char *envname = NULL;
> + char *envval = NULL;
> + size_t len = 0;
> +
> + while (get_next_env (&envp, &envname, &len, &envval))
> + {
> + for (int i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
> + {
> + tunable_t *cur = &tunable_list[i];
> +
> + /* Skip over tunables that have either been set already or should be
> + skipped. */
> + if (cur->strval != NULL || cur->env_alias == NULL
> + || (__libc_enable_secure && !cur->secure_env_alias))
> + continue;
> +
> + const char *name = cur->env_alias;
> +
> + /* We have a match. Initialize and move on to the next line. */
> + if (is_name (name, envname))
> + {
> + cur->strval = envval;
> + tunable_initialize (cur);
> + break;
> + }
> + }
> + }
> +}
> +
OK. We can accelerate this a bit if we used the TOP_NAMESPACE[0] char
to reject env entries.
> +/* Set the tunable value. This is called by the module that the tunable exists
> + in. */
> +void
> +__tunable_set_val (tunable_id_t id, void *valp, tunable_callback_t callback)
> +{
> + tunable_t *cur = &tunable_list[id];
> +
> + /* Sanity check: don't do anything if our tunable was not set during
> + initialization. */
> + if (cur->strval == NULL)
> + return;
> +
> + switch (cur->type.type_code)
> + {
> + case TUNABLE_TYPE_INT_32:
> + {
> + *((int32_t *) valp) = (int32_t) cur->val.numval;
> + break;
> + }
> + case TUNABLE_TYPE_SIZE_T:
> + {
> + *((size_t *) valp) = (size_t) cur->val.numval;
> + break;
> + }
> + case TUNABLE_TYPE_STRING:
> + {
> + *((const char **)valp) = cur->strval;
> + break;
> + }
> + default:
> + __builtin_unreachable ();
> + }
> +
> + if (callback)
> + callback (&cur->val);
> +}
OK.
> diff --git a/elf/dl-tunables.h b/elf/dl-tunables.h
> new file mode 100644
> index 0000000..c4ae97d
> --- /dev/null
> +++ b/elf/dl-tunables.h
> @@ -0,0 +1,76 @@
> +/* The tunable framework. See the README to know how to use the tunable in
> + a glibc module.
> +
> + Copyright (C) 2016 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <http://www.gnu.org/licenses/>. */
> +
> +#ifndef _TUNABLES_H_
> +# define _TUNABLES_H_
> +# include <stddef.h>
> +# include "dl-tunable-types.h"
> +
> +/* A tunable. */
> +struct _tunable
> +{
> + const char *name; /* Internal name of the tunable. */
> + tunable_type_t type; /* Data type of the tunable. */
> + tunable_val_t val; /* The value. */
> + const char *strval; /* The string containing the value,
> + points into envp. */
> + /* Compatibility elements. */
> + const char *env_alias; /* The compatibility environment
> + variable name. */
> + bool secure_env_alias; /* Whether the environment variable
> + must be read even for setuid
> + binaries. */
Add "May still be ignored if value is dangerous."
> +};
> +
> +typedef struct _tunable tunable_t;
> +
> +/* Full name for a tunable is top_ns.tunable_ns.id. */
> +#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id
> +
> +# define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)
> +# define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id
> +
> +#include "dl-tunable-list.h"
> +
> +extern void __tunables_init (char **);
> +extern void __tunable_set_val (tunable_id_t, void *, tunable_callback_t);
> +
> +/* Check if the tunable has been set to a non-default value and if it is, copy
> + it over into __VAL. */
> +# define TUNABLE_SET_VAL(__id,__val) \
> +({ \
> + __tunable_set_val \
> + (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
> + NULL); \
> +})
> +
> +/* Same as TUNABLE_SET_VAL, but also set the callback function to __CB and call
> + it. */
> +# define TUNABLE_SET_VAL_WITH_CALLBACK(__id,__val,__cb) \
> +({ \
> + __tunable_set_val \
> + (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
> + DL_TUNABLE_CALLBACK (__cb)); \
> +})
> +
> +/* Namespace sanity for callback functions. Use this macro to keep the
> + namespace of the modules clean. */
> +#define DL_TUNABLE_CALLBACK(__name) _dl_tunable_ ## __name
> +#endif
OK.
> diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list
> new file mode 100644
> index 0000000..342e3d5
> --- /dev/null
> +++ b/elf/dl-tunables.list
> @@ -0,0 +1,50 @@
> +# Allowed attributes for tunables:
> +#
> +# type: Defaults to STRING
> +# minval: Optional minimum acceptable value
> +# maxval: Optional maximum acceptable value
> +# env_alias: An alias environment variable
> +# secure_env_alias: Specify whether the environment variable should be read for
> +# setuid binaries.
> +
> +glibc {
> + malloc {
> + check {
> + type: INT_32
> + minval: 0
> + maxval: 3
> + env_alias: MALLOC_CHECK_
> + secure_env_alias: true
> + }
> + top_pad {
> + type: SIZE_T
> + env_alias: MALLOC_TOP_PAD_
> + }
> + perturb {
> + type: INT_32
> + minval: 0
> + maxval: 0xff
> + env_alias: MALLOC_PERTURB_
> + }
> + mmap_threshold {
> + type: SIZE_T
> + env_alias: MALLOC_MMAP_THRESHOLD_
> + }
> + trim_threshold {
> + type: SIZE_T
> + env_alias: MALLOC_TRIM_THRESHOLD_
> + }
> + mmap_max {
> + type: INT_32
> + env_alias: MALLOC_MMAP_MAX_
> + }
> + arena_max {
> + type: SIZE_T
> + env_alias: MALLOC_ARENA_MAX
> + }
> + arena_test {
> + type: SIZE_T
> + env_alias: MALLOC_ARENA_TEST
> + }
> + }
> +}
Nice. We need M_MXFAST on the list! ;-)
> diff --git a/elf/rtld.c b/elf/rtld.c
> index 647661c..263723a 100644
> --- a/elf/rtld.c
> +++ b/elf/rtld.c
> @@ -44,6 +44,10 @@
>
> #include <assert.h>
>
> +#if BUILD_TUNABLES
> +# include <dl-tunables.h>
> +#endif
> +
> /* Avoid PLT use for our local calls at startup. */
> extern __typeof (__mempcpy) __mempcpy attribute_hidden;
>
> @@ -2346,6 +2350,10 @@ process_envvars (enum mode *modep)
> enum mode mode = normal;
> char *debug_output = NULL;
>
> +#if BUILD_TUNABLES
> + __tunables_init (_environ);
> +#endif
OK.
> +
> /* This is the default place for profiling data file. */
> GLRO(dl_profile_output)
> = &"/var/tmp\0/var/profile"[__libc_enable_secure ? 9 : 0];
> diff --git a/malloc/Makefile b/malloc/Makefile
> index fa1730e..23ab2f4 100644
> --- a/malloc/Makefile
> +++ b/malloc/Makefile
> @@ -31,6 +31,8 @@ tests := mallocbug tst-malloc tst-valloc tst-calloc tst-obstack \
> tst-malloc-backtrace tst-malloc-thread-exit \
> tst-malloc-thread-fail tst-malloc-fork-deadlock \
> tst-mallocfork2
> +tests-static := tst-malloc-usable-static
> +tests += $(tests-static)
OK.
> test-srcs = tst-mtrace
>
> routines = malloc morecore mcheck mtrace obstack \
> @@ -138,6 +140,7 @@ endif
>
> tst-mcheck-ENV = MALLOC_CHECK_=3
> tst-malloc-usable-ENV = MALLOC_CHECK_=3
> +tst-malloc-usable-static-ENV = $(tst-malloc-usable-ENV)
>
> # Uncomment this for test releases. For public releases it is too expensive.
> #CPPFLAGS-malloc.o += -DMALLOC_DEBUG=1
> diff --git a/malloc/arena.c b/malloc/arena.c
> index 229783f..7de4caf 100644
> --- a/malloc/arena.c
> +++ b/malloc/arena.c
> @@ -19,6 +19,11 @@
>
> #include <stdbool.h>
>
> +#if BUILD_TUNABLES
> +# define TUNABLE_NAMESPACE malloc
> +# include <elf/dl-tunables.h>
> +#endif
> +
> /* Compile-time constants. */
>
> #define HEAP_MIN_SIZE (32 * 1024)
> @@ -204,6 +209,15 @@ __malloc_fork_unlock_child (void)
> mutex_init (&list_lock);
> }
>
> +#if BUILD_TUNABLES
> +void
> +DL_TUNABLE_CALLBACK (set_mallopt_check) (void *unused)
> +{
> + if (check_action != 0)
> + __malloc_check_init ();
> +}
> +
> +#else
> /* Initialization routine. */
> #include <string.h>
> extern char **_environ;
> @@ -238,6 +252,7 @@ next_env_entry (char ***position)
>
> return result;
> }
> +#endif
>
>
> #ifdef SHARED
> @@ -272,6 +287,24 @@ ptmalloc_init (void)
> #endif
>
> thread_arena = &main_arena;
> +
> +#if BUILD_TUNABLES
> + /* Ensure initialization/consolidation and do it under a lock so that a
> + thread attempting to use the arena in parallel waits on us till we
> + finish. */
> + mutex_lock (&main_arena.mutex);
> + malloc_consolidate (&main_arena);
> +
> + TUNABLE_SET_VAL_WITH_CALLBACK (check, &check_action, set_mallopt_check);
> + TUNABLE_SET_VAL (top_pad, &mp_.top_pad);
> + TUNABLE_SET_VAL (perturb, &perturb_byte);
> + TUNABLE_SET_VAL (mmap_threshold, &mp_.mmap_threshold);
> + TUNABLE_SET_VAL (trim_threshold, &mp_.trim_threshold);
> + TUNABLE_SET_VAL (mmap_max, &mp_.n_mmaps_max);
> + TUNABLE_SET_VAL (arena_max, &mp_.arena_max);
> + TUNABLE_SET_VAL (arena_test, &mp_.arena_test);
Two things are wrong.
Missing systemtap probes, _and_ wrong settings.
For example setting top_pad should disable dynamic thresholding.
Therefore you need to do the following:
(a) Take __libc_mallopt from malloc/malloc.c and split out the
setting of each variable into a "callback"
(b) Have __libc_mallopt call the callbacks.
(c) Have TUNABLE_SET_VAL call the callback via the tunables
mechanism.
That way you have code in one place for what has to happen
when setting the variable.
> + mutex_unlock (&main_arena.mutex);
> +#else
> const char *s = NULL;
> if (__glibc_likely (_environ != NULL))
> {
> @@ -340,6 +373,8 @@ ptmalloc_init (void)
> if (check_action != 0)
> __malloc_check_init ();
> }
> +#endif
> +
> #if HAVE_MALLOC_INIT_HOOK
> void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook);
> if (hook != NULL)
OK.
> diff --git a/malloc/tst-malloc-usable-static.c b/malloc/tst-malloc-usable-static.c
> new file mode 100644
> index 0000000..8907db0
> --- /dev/null
> +++ b/malloc/tst-malloc-usable-static.c
> @@ -0,0 +1 @@
> +#include <malloc/tst-malloc-usable.c>
> diff --git a/manual/install.texi b/manual/install.texi
> index 79ee45f..f9b4784 100644
> --- a/manual/install.texi
> +++ b/manual/install.texi
> @@ -189,6 +189,11 @@ configure with @option{--disable-werror}.
> By default for x86_64, @theglibc{} is built with vector math library.
> Use this option to disable vector math library.
>
> +@item --enable-tunables
> +Tunables support allows additional library parameters to be customized at
> +runtime for each application. This is an experimental feature and affects
> +startup time and is thus disabled by default.
> +
> @item --build=@var{build-system}
> @itemx --host=@var{host-system}
> These options are for cross-compiling. If you specify both options and
> diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
> new file mode 100644
> index 0000000..1b930e7
> --- /dev/null
> +++ b/scripts/gen-tunables.awk
> @@ -0,0 +1,157 @@
> +# Generate dl-tunable-list.h from dl-tunables.list
> +
> +BEGIN {
> + tunable=""
> + ns=""
> + top_ns=""
> +}
> +
> +# Skip over blank lines and comments.
> +/^#/ {
> + next
> +}
> +
> +/^[ \t]*$/ {
> + next
> +}
> +
> +# Beginning of either a top namespace, tunable namespace or a tunable, decided
> +# on the current value of TUNABLE, NS or TOP_NS.
> +$2 == "{" {
> + if (top_ns == "") {
> + top_ns = $1
> + }
> + else if (ns == "") {
> + ns = $1
> + }
> + else if (tunable == "") {
> + tunable = $1
> + }
> + else {
> + printf ("Unexpected occurrence of '{': %s:%d\n", FILENAME, FNR)
> + exit 1
> + }
> +
> + next
> +}
> +
> +# End of either a top namespace, tunable namespace or a tunable.
> +$1 == "}" {
> + if (tunable != "") {
> + # Tunables definition ended, now fill in default attributes.
> + if (!types[top_ns][ns][tunable]) {
> + types[top_ns][ns][tunable] = "STRING"
> + }
> + if (!minvals[top_ns][ns][tunable]) {
> + minvals[top_ns][ns][tunable] = "0"
> + }
> + if (!maxvals[top_ns][ns][tunable]) {
> + maxvals[top_ns][ns][tunable] = "0"
> + }
> + if (!env_alias[top_ns][ns][tunable]) {
> + env_alias[top_ns][ns][tunable] = "NULL"
> + }
> + if (!secure_env_alias[top_ns][ns][tunable]) {
> + secure_env_alias[top_ns][ns][tunable] = "false"
> + }
> +
> + tunable = ""
> + }
> + else if (ns != "") {
> + ns = ""
> + }
> + else if (top_ns != "") {
> + top_ns = ""
> + }
> + else {
> + printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR)
> + exit 1
> + }
> + next
> +}
> +
> +# Everything else, which could either be a tunable without any attributes or a
> +# tunable attribute.
> +{
> + if (ns == "") {
> + printf("Line %d: Invalid tunable outside a namespace: %s\n", NR, $0)
> + exit 1
> + }
> +
> + if (tunable == "") {
> + # We encountered a tunable without any attributes, so note it with a
> + # default.
> + types[top_ns][ns][$1] = "STRING"
> + next
> + }
> +
> + # Otherwise, we have encountered a tunable attribute.
> + split($0, arr, ":")
> + attr = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[1])
> + val = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[2])
> +
> + if (attr == "type") {
> + types[top_ns][ns][tunable] = val
> + }
> + else if (attr == "minval") {
> + minvals[top_ns][ns][tunable] = val
> + }
> + else if (attr == "maxval") {
> + maxvals[top_ns][ns][tunable] = val
> + }
> + else if (attr == "env_alias") {
> + env_alias[top_ns][ns][tunable] = sprintf("\"%s\"", val)
> + }
> + else if (attr == "secure_env_alias") {
> + if (val == "true" || val == "false") {
> + secure_env_alias[top_ns][ns][tunable] = val
> + }
> + else {
> + printf("Line %d: Invalid value (%s) for secure_env_alias: %s, ", NR, val,
> + $0)
> + print("Allowed values are 'true' or 'false'")
> + exit 1
> + }
> + }
> +}
> +
> +END {
> + if (ns != "") {
> + print "Unterminated namespace. Is a closing brace missing?"
> + exit 1
> + }
> +
> + print "/* AUTOGENERATED by gen-tunables.awk. */"
> + print "#ifndef _TUNABLES_H_"
> + print "# error \"Do not include this file directly.\""
> + print "# error \"Include tunables.h instead.\""
> + print "#endif"
> +
> + # Now, the enum names
> + print "\ntypedef enum"
> + print "{"
> + for (t in types) {
> + for (n in types[t]) {
> + for (m in types[t][n]) {
> + printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, m);
> + }
> + }
> + }
> + print "} tunable_id_t;\n"
> +
> + # Finally, the tunable list.
> + print "\n#ifdef TUNABLES_INTERNAL"
> + print "static tunable_t tunable_list[] = {"
> + for (t in types) {
> + for (n in types[t]) {
> + for (m in types[t][n]) {
> + printf (" {TUNABLE_NAME_S(%s, %s, %s)", t, n, m)
> + printf (", {TUNABLE_TYPE_%s, %s, %s}, {.numval = 0}, NULL, %s, %s},\n",
> + types[t][n][m], minvals[t][n][m], maxvals[t][n][m],
> + env_alias[t][n][m], secure_env_alias[t][n][m]);
> + }
> + }
> + }
> + print "};"
> + print "#endif"
> +}
>
OK.
Cheers,
Carlos.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-11 16:56 ` Paul E. Murphy
@ 2016-07-11 17:28 ` H.J. Lu
0 siblings, 0 replies; 49+ messages in thread
From: H.J. Lu @ 2016-07-11 17:28 UTC (permalink / raw)
To: Paul E. Murphy; +Cc: Siddhesh Poyarekar, GNU C Library
On Mon, Jul 11, 2016 at 9:56 AM, Paul E. Murphy
<murphyp@linux.vnet.ibm.com> wrote:
> I am still reviewing the patch, but the question which pops into
> my head is whether this is sufficient to initialize tunables
> used early on via an init section.
.init section isn't sufficient for IFUNC.
> As implied above, my hope is that the initial patches will enable
> trivial addition tunables for TLE. I'll port my old patches and
> see how it goes nevertheless.
>
>
> On 07/09/2016 01:37 PM, Siddhesh Poyarekar wrote:
>> The tunables framework allows us to uniformly manage and expose global
>> variables inside glibc as switches to users. tunables/README has
>> instructions for glibc developers to add new tunables.
>>
>> Tunables support can be enabled by passing the --enable-tunables
>> configure flag to the configure script. This patch only adds a
>> framework and does not pose any limitations on how tunable values are
>> read from the user. It also adds environment variables used in malloc
>> behaviour tweaking to the tunables framework as a PoC of the
>> compatibility interface.
>>
>> * manual/install.texi: Add --enable-tunables option.
>> * INSTALL: Regenerate.
>> * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
>> (before-compile): Generate dl-tunable-list.h early.
>> * config.h.in: Add BUILD_TUNABLES.
>> * config.make.in: Add build-tunables.
>> * configure.ac: Add --enable-tunables option.
>> * configure: Regenerate.
>> * malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
>> Define TUNABLE_NAMESPACE.
>> (DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
>> (ptmalloc_init): Set tunable values.
>> * malloc/tst-malloc-usable-static.c: New test case.
>> * malloc/Makefile (tests-static): Add it.
>> * csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
>> (__libc_init_first) [!SHARED]: Initialize tunables for static
>> binaries.
>> * scripts/gen-tunables.awk: New file.
>> * README.tunables: New file.
>> * elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
>> namespace.
>> * elf/dl-tunable-list.h: New auto-generated file.
>> * elf/dl-tunables.c: New file.
>> * elf/dl-tunables.h: New file.
>> * elf/dl-tunables.list: New file.
>> * elf/dl-tunable-types.h: New file.
>> * elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
>> (process_envvars): Call __tunables_init.
>
--
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-09 18:49 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-07-11 16:05 ` Florian Weimer
2016-07-11 16:25 ` H.J. Lu
@ 2016-07-11 16:56 ` Paul E. Murphy
2016-07-11 17:28 ` H.J. Lu
2016-07-11 19:06 ` Carlos O'Donell
3 siblings, 1 reply; 49+ messages in thread
From: Paul E. Murphy @ 2016-07-11 16:56 UTC (permalink / raw)
To: Siddhesh Poyarekar, libc-alpha
I am still reviewing the patch, but the question which pops into
my head is whether this is sufficient to initialize tunables
used early on via an init section.
As implied above, my hope is that the initial patches will enable
trivial addition tunables for TLE. I'll port my old patches and
see how it goes nevertheless.
On 07/09/2016 01:37 PM, Siddhesh Poyarekar wrote:
> The tunables framework allows us to uniformly manage and expose global
> variables inside glibc as switches to users. tunables/README has
> instructions for glibc developers to add new tunables.
>
> Tunables support can be enabled by passing the --enable-tunables
> configure flag to the configure script. This patch only adds a
> framework and does not pose any limitations on how tunable values are
> read from the user. It also adds environment variables used in malloc
> behaviour tweaking to the tunables framework as a PoC of the
> compatibility interface.
>
> * manual/install.texi: Add --enable-tunables option.
> * INSTALL: Regenerate.
> * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
> (before-compile): Generate dl-tunable-list.h early.
> * config.h.in: Add BUILD_TUNABLES.
> * config.make.in: Add build-tunables.
> * configure.ac: Add --enable-tunables option.
> * configure: Regenerate.
> * malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
> Define TUNABLE_NAMESPACE.
> (DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
> (ptmalloc_init): Set tunable values.
> * malloc/tst-malloc-usable-static.c: New test case.
> * malloc/Makefile (tests-static): Add it.
> * csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
> (__libc_init_first) [!SHARED]: Initialize tunables for static
> binaries.
> * scripts/gen-tunables.awk: New file.
> * README.tunables: New file.
> * elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
> namespace.
> * elf/dl-tunable-list.h: New auto-generated file.
> * elf/dl-tunables.c: New file.
> * elf/dl-tunables.h: New file.
> * elf/dl-tunables.list: New file.
> * elf/dl-tunable-types.h: New file.
> * elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
> (process_envvars): Call __tunables_init.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-11 16:25 ` H.J. Lu
@ 2016-07-11 16:41 ` Siddhesh Poyarekar
2016-07-13 14:43 ` H.J. Lu
0 siblings, 1 reply; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-11 16:41 UTC (permalink / raw)
To: H.J. Lu; +Cc: GNU C Library
On Mon, Jul 11, 2016 at 09:25:07AM -0700, H.J. Lu wrote:
> Did you update siddhesh/tunables branch? It still has the old patches.
I have now, sorry I keep forgetting.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-11 16:05 ` Florian Weimer
@ 2016-07-11 16:39 ` Siddhesh Poyarekar
2016-07-11 19:13 ` DJ Delorie
2016-07-12 9:00 ` Florian Weimer
2016-07-12 8:51 ` Florian Weimer
1 sibling, 2 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-11 16:39 UTC (permalink / raw)
To: Florian Weimer; +Cc: libc-alpha
On Mon, Jul 11, 2016 at 06:05:09PM +0200, Florian Weimer wrote:
> On 07/09/2016 08:37 PM, Siddhesh Poyarekar wrote:
>
> > +ifeq (yes,$(build-tunables))
> > +CPPFLAGS += -DTOP_NAMESPACE=glibc
> > +endif
>
> Should probably be TUNABLE_NAMESPACE or TUNABLE_ROOT.
OK.
>
> > +- secure_env_alias: Specify whether the environment variable should be read
> > + for setuid binaries.
>
> This needs to describe the exact meaning of the choices, and the default.
> I'm not sure if the current/name values are intuitive.
It specifies if the env_alias should be read even for setuid binaries.
This is currently only useful for MALLOC_CHECK_ since it is read even
for setuid binaries. Would env_alias_is_secure be clearer?
> > +/* Compare environment names, bounded by the name hardcoded in glibc. */
> > +static bool
> > +is_name (const char *orig, const char *envname)
>
> I don't understand this comment.
The comparison loop is bound by the name that is hardcoded into the
tunables_list structure, i.e. either tunable.name or
tunable.env_alias.
> > +static bool
> > +get_next_env (char ***envp, char **name, size_t *namelen, char **val)
>
> Can you get rid of one indirection somehow?
I had picked this up verbatim from existing code. I'll see what I can
do.
>
> > +{
> > + char **ev = *envp;
> > +
> > + while (ev != NULL && *ev != '\0')
>
> *ev has type char *, so NULL is better than '\0'. I'm surprised GCC does
> not warn about this.
Ugh, yes, I'll fix it.
> > +# define TUNABLE_SET_VAL(__id,__val) \
> > +({ \
> > + __tunable_set_val \
> > + (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
> > + NULL); \
> > +})
>
> You do not need __ prefixes on macro arguments.
I did it out of habit to avoid namespace collisions, but it's not
necessary here so I'll remove it.
> This doesn't mirror what mallopt does internally, so it alters behavior with
> BUILD_TUNABLES. In particular, the mmap threshold is still dynamically
> adjusted after that, I think.
Thanks, I missed that. I'll fix it.
> This is indeed closer to what I had in mind. But what I really want is
> something that can patch the tunables directly into a place which malloc can
I plan to do it via the tunables structure in the second pass, with
TUNABLE_SET_VAL_WITH_PTR that does the same thing as TUNABLE_SET_VAL
and then stores the pointer (&mp_.mmap_threshold) and callback in the
tunables structure to allow for a live update.
> use. My hope is that one day, we have a tunable descriptor inside libc.so,
> and (rarely written) range in the .data segment which contains the tunable
> variables. Code which relies on tunables directly access the tunable range
> (using relaxed MO loads), and some external tool can use the tunables
> descriptor to update the tunables at run time.
This is an interesting idea, but it will still have to live inside
ld.so if it needs to make IFUNC decisions.
> > diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
> > new file mode 100644
> > index 0000000..1b930e7
> > --- /dev/null
> > +++ b/scripts/gen-tunables.awk
> > @@ -0,0 +1,157 @@
> > +# Generate dl-tunable-list.h from dl-tunables.list
>
> Please add the copyright header because the file is non-trivial.
OK.
> > + print "static tunable_t tunable_list[] = {"
>
> There's a const missing there. Due to function pointer, this should really
> go into .relro.
OK.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-09 18:49 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-07-11 16:05 ` Florian Weimer
@ 2016-07-11 16:25 ` H.J. Lu
2016-07-11 16:41 ` Siddhesh Poyarekar
2016-07-11 16:56 ` Paul E. Murphy
2016-07-11 19:06 ` Carlos O'Donell
3 siblings, 1 reply; 49+ messages in thread
From: H.J. Lu @ 2016-07-11 16:25 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Sat, Jul 9, 2016 at 11:37 AM, Siddhesh Poyarekar
<siddhesh@sourceware.org> wrote:
> The tunables framework allows us to uniformly manage and expose global
> variables inside glibc as switches to users. tunables/README has
> instructions for glibc developers to add new tunables.
>
> Tunables support can be enabled by passing the --enable-tunables
> configure flag to the configure script. This patch only adds a
> framework and does not pose any limitations on how tunable values are
> read from the user. It also adds environment variables used in malloc
> behaviour tweaking to the tunables framework as a PoC of the
> compatibility interface.
>
> * manual/install.texi: Add --enable-tunables option.
> * INSTALL: Regenerate.
> * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
> (before-compile): Generate dl-tunable-list.h early.
> * config.h.in: Add BUILD_TUNABLES.
> * config.make.in: Add build-tunables.
> * configure.ac: Add --enable-tunables option.
> * configure: Regenerate.
> * malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
> Define TUNABLE_NAMESPACE.
> (DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
> (ptmalloc_init): Set tunable values.
> * malloc/tst-malloc-usable-static.c: New test case.
> * malloc/Makefile (tests-static): Add it.
> * csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
> (__libc_init_first) [!SHARED]: Initialize tunables for static
> binaries.
> * scripts/gen-tunables.awk: New file.
> * README.tunables: New file.
> * elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
> namespace.
> * elf/dl-tunable-list.h: New auto-generated file.
> * elf/dl-tunables.c: New file.
> * elf/dl-tunables.h: New file.
> * elf/dl-tunables.list: New file.
> * elf/dl-tunable-types.h: New file.
> * elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
> (process_envvars): Call __tunables_init.
Did you update siddhesh/tunables branch? It still has the old patches.
H.J.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-07-09 18:49 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
@ 2016-07-11 16:05 ` Florian Weimer
2016-07-11 16:39 ` Siddhesh Poyarekar
2016-07-12 8:51 ` Florian Weimer
2016-07-11 16:25 ` H.J. Lu
` (2 subsequent siblings)
3 siblings, 2 replies; 49+ messages in thread
From: Florian Weimer @ 2016-07-11 16:05 UTC (permalink / raw)
To: Siddhesh Poyarekar, libc-alpha
On 07/09/2016 08:37 PM, Siddhesh Poyarekar wrote:
> +ifeq (yes,$(build-tunables))
> +CPPFLAGS += -DTOP_NAMESPACE=glibc
> +endif
Should probably be TUNABLE_NAMESPACE or TUNABLE_ROOT.
> +- secure_env_alias: Specify whether the environment variable should be read
> + for setuid binaries.
This needs to describe the exact meaning of the choices, and the
default. I'm not sure if the current/name values are intuitive.
> +/* Compare environment names, bounded by the name hardcoded in glibc. */
> +static bool
> +is_name (const char *orig, const char *envname)
I don't understand this comment.
> +static bool
> +get_next_env (char ***envp, char **name, size_t *namelen, char **val)
Can you get rid of one indirection somehow?
> +{
> + char **ev = *envp;
> +
> + while (ev != NULL && *ev != '\0')
*ev has type char *, so NULL is better than '\0'. I'm surprised GCC
does not warn about this.
> +# define TUNABLE_SET_VAL(__id,__val) \
> +({ \
> + __tunable_set_val \
> + (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
> + NULL); \
> +})
You do not need __ prefixes on macro arguments.
> +#if BUILD_TUNABLES
> + /* Ensure initialization/consolidation and do it under a lock so that a
> + thread attempting to use the arena in parallel waits on us till we
> + finish. */
> + mutex_lock (&main_arena.mutex);
> + malloc_consolidate (&main_arena);
> +
> + TUNABLE_SET_VAL_WITH_CALLBACK (check, &check_action, set_mallopt_check);
> + TUNABLE_SET_VAL (top_pad, &mp_.top_pad);
> + TUNABLE_SET_VAL (perturb, &perturb_byte);
> + TUNABLE_SET_VAL (mmap_threshold, &mp_.mmap_threshold);
> + TUNABLE_SET_VAL (trim_threshold, &mp_.trim_threshold);
> + TUNABLE_SET_VAL (mmap_max, &mp_.n_mmaps_max);
> + TUNABLE_SET_VAL (arena_max, &mp_.arena_max);
> + TUNABLE_SET_VAL (arena_test, &mp_.arena_test);
This doesn't mirror what mallopt does internally, so it alters behavior
with BUILD_TUNABLES. In particular, the mmap threshold is still
dynamically adjusted after that, I think.
This is indeed closer to what I had in mind. But what I really want is
something that can patch the tunables directly into a place which malloc
can use. My hope is that one day, we have a tunable descriptor inside
libc.so, and (rarely written) range in the .data segment which contains
the tunable variables. Code which relies on tunables directly access
the tunable range (using relaxed MO loads), and some external tool can
use the tunables descriptor to update the tunables at run time.
> diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
> new file mode 100644
> index 0000000..1b930e7
> --- /dev/null
> +++ b/scripts/gen-tunables.awk
> @@ -0,0 +1,157 @@
> +# Generate dl-tunable-list.h from dl-tunables.list
Please add the copyright header because the file is non-trivial.
> + print "static tunable_t tunable_list[] = {"
There's a const missing there. Due to function pointer, this should
really go into .relro.
Thanks,
Florian
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH 1/2] Add framework for tunables
2016-07-09 18:49 [PATCHv3 0/2] tunables for glibc Siddhesh Poyarekar
@ 2016-07-09 18:49 ` Siddhesh Poyarekar
2016-07-11 16:05 ` Florian Weimer
` (3 more replies)
0 siblings, 4 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-09 18:49 UTC (permalink / raw)
To: libc-alpha
The tunables framework allows us to uniformly manage and expose global
variables inside glibc as switches to users. tunables/README has
instructions for glibc developers to add new tunables.
Tunables support can be enabled by passing the --enable-tunables
configure flag to the configure script. This patch only adds a
framework and does not pose any limitations on how tunable values are
read from the user. It also adds environment variables used in malloc
behaviour tweaking to the tunables framework as a PoC of the
compatibility interface.
* manual/install.texi: Add --enable-tunables option.
* INSTALL: Regenerate.
* Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
(before-compile): Generate dl-tunable-list.h early.
* config.h.in: Add BUILD_TUNABLES.
* config.make.in: Add build-tunables.
* configure.ac: Add --enable-tunables option.
* configure: Regenerate.
* malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
Define TUNABLE_NAMESPACE.
(DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
(ptmalloc_init): Set tunable values.
* malloc/tst-malloc-usable-static.c: New test case.
* malloc/Makefile (tests-static): Add it.
* csu/init-first.c [BUILD_TUNABLES]: Include dl-tunables.h.
(__libc_init_first) [!SHARED]: Initialize tunables for static
binaries.
* scripts/gen-tunables.awk: New file.
* README.tunables: New file.
* elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
namespace.
* elf/dl-tunable-list.h: New auto-generated file.
* elf/dl-tunables.c: New file.
* elf/dl-tunables.h: New file.
* elf/dl-tunables.list: New file.
* elf/dl-tunable-types.h: New file.
* elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
(process_envvars): Call __tunables_init.
---
INSTALL | 6 ++
Makeconfig | 16 ++++
README.tunables | 74 ++++++++++++++++
config.h.in | 3 +
config.make.in | 1 +
configure | 16 ++++
configure.ac | 10 +++
csu/init-first.c | 7 ++
elf/Makefile | 5 ++
elf/Versions | 3 +
elf/dl-tunable-types.h | 46 ++++++++++
elf/dl-tunables.c | 182 ++++++++++++++++++++++++++++++++++++++
elf/dl-tunables.h | 76 ++++++++++++++++
elf/dl-tunables.list | 50 +++++++++++
elf/rtld.c | 8 ++
malloc/Makefile | 3 +
malloc/arena.c | 35 ++++++++
malloc/tst-malloc-usable-static.c | 1 +
manual/install.texi | 5 ++
scripts/gen-tunables.awk | 157 ++++++++++++++++++++++++++++++++
20 files changed, 704 insertions(+)
create mode 100644 README.tunables
create mode 100644 elf/dl-tunable-types.h
create mode 100644 elf/dl-tunables.c
create mode 100644 elf/dl-tunables.h
create mode 100644 elf/dl-tunables.list
create mode 100644 malloc/tst-malloc-usable-static.c
create mode 100644 scripts/gen-tunables.awk
diff --git a/INSTALL b/INSTALL
index ec3445f..90dc481 100644
--- a/INSTALL
+++ b/INSTALL
@@ -158,6 +158,12 @@ will be used, and CFLAGS sets optimization options for the compiler.
By default for x86_64, the GNU C Library is built with vector math
library. Use this option to disable vector math library.
+'--enable-tunables'
+ Tunables support allows additional library parameters to be
+ customized at runtime for each application. This is an
+ experimental feature and affects startup time and is thus disabled
+ by default.
+
'--build=BUILD-SYSTEM'
'--host=HOST-SYSTEM'
These options are for cross-compiling. If you specify both options
diff --git a/Makeconfig b/Makeconfig
index 58bd3b3..732b870 100644
--- a/Makeconfig
+++ b/Makeconfig
@@ -895,6 +895,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \
$(foreach lib,$(libof-$(basename $(@F))) \
$(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \
$(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F)))
+
+ifeq (yes,$(build-tunables))
+CPPFLAGS += -DTOP_NAMESPACE=glibc
+endif
+
override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \
$(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \
$(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \
@@ -1067,6 +1072,17 @@ $(common-objpfx)libc-modules.stmp: $(..)scripts/gen-libc-modules.awk \
endif
+# Build the tunables list header early since it could be used by any module in
+# glibc.
+ifeq (yes,$(build-tunables))
+before-compile += $(common-objpfx)dl-tunable-list.h
+
+$(common-objpfx)dl-tunable-list.h: $(..)scripts/gen-tunables.awk \
+ $(..)elf/dl-tunables.list
+ $(AWK) -f $^ > $@.tmp
+ mv $@{.tmp,}
+endif
+
common-generated += libc-modules.h libc-modules.stmp
# The name under which the run-time dynamic linker is installed.
diff --git a/README.tunables b/README.tunables
new file mode 100644
index 0000000..8890249
--- /dev/null
+++ b/README.tunables
@@ -0,0 +1,74 @@
+ TUNABLE FRAMEWORK
+ =================
+
+The tunable framework allows modules within glibc to register variables that
+may be tweaked through an environment variable or an API call. It aims to
+enforce a strict namespace rule to bring consistency to naming of these tunable
+environment variables across the project.
+
+ADDING A NEW TUNABLE
+--------------------
+
+The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in
+distributions for specific tunables if they want to add their own tunables.
+Downstream implementations are discouraged from using the 'glibc' namespace for
+tunables they don't already have consensus to push upstream.
+
+There are two steps to adding a tunable:
+
+1. Add a tunable ID:
+
+Modules that wish to use the tunables interface must define the
+TUNABLE_NAMESPACE macro. Following this, for each tunable you want to
+add, make an entry in elf/dl-tunables.list. The format of the file is as
+follows:
+
+TOP_NAMESPACE {
+ NAMESPACE1 {
+ TUNABLE1 {
+ # tunable attributes, one per line
+ }
+ # A tunable with default attributes, i.e. string variable.
+ TUNABLE2
+ TUNABLE3 {
+ # its attributes
+ }
+ }
+ NAMESPACE2 {
+ ...
+ }
+}
+
+The list of allowed attributes are:
+
+- type: Data type. Defaults to STRING. Allowed types are:
+ INT_32, SIZE_T. Note that as of now the STRING type is
+ neither defined nor supported since no tunables need
+ it. It will be in future though, when needed.
+
+- minval: Optional minimum acceptable value
+
+- maxval: Optional maximum acceptable value
+
+- env_alias: An alias environment variable
+
+- secure_env_alias: Specify whether the environment variable should be read
+ for setuid binaries.
+
+2. Call either the TUNABLE_SET_VALUE and pass into it the tunable name and a
+ pointer to the variable that should be set with the tunable value.
+ If additional work needs to be done after setting the value, use the
+ TUNABLE_SET_VALUE_WITH_CALLBACK instead and additionally pass a pointer to
+ the function that should be called if the tunable value has been set.
+
+FUTURE WORK
+-----------
+
+The framework currently only allows a one-time initialization of variables
+through environment variables and in some cases, modification of variables via
+an API call. A future goals for this project include:
+
+- Setting system-wide and user-wide defaults for tunables through some
+ mechanism like a configuration file.
+
+- Allow tweaking of some tunables at runtime
diff --git a/config.h.in b/config.h.in
index 856ef6a..6112782 100644
--- a/config.h.in
+++ b/config.h.in
@@ -240,4 +240,7 @@
/* PowerPC32 uses fctidz for floating point to long long conversions. */
#define HAVE_PPC_FCTIDZ 0
+/* Build glibc with tunables support. */
+#define BUILD_TUNABLES 0
+
#endif
diff --git a/config.make.in b/config.make.in
index 95c6f36..653ef6f 100644
--- a/config.make.in
+++ b/config.make.in
@@ -91,6 +91,7 @@ use-nscd = @use_nscd@
build-hardcoded-path-in-tests= @hardcoded_path_in_tests@
build-pt-chown = @build_pt_chown@
enable-lock-elision = @enable_lock_elision@
+build-tunables = @build_tunables@
# Build tools.
CC = @CC@
diff --git a/configure b/configure
index 19a4829..1534ba7 100755
--- a/configure
+++ b/configure
@@ -659,6 +659,7 @@ multi_arch
base_machine
add_on_subdirs
add_ons
+build_tunables
build_pt_chown
build_nscd
link_obsolete_rpc
@@ -773,6 +774,7 @@ enable_systemtap
enable_build_nscd
enable_nscd
enable_pt_chown
+enable_tunables
enable_mathvec
with_cpu
'
@@ -1440,6 +1442,7 @@ Optional Features:
--disable-build-nscd disable building and installing the nscd daemon
--disable-nscd library functions will not contact the nscd daemon
--enable-pt_chown Enable building and installing pt_chown
+ --enable-tunables Enable tunables support
--enable-mathvec Enable building and installing mathvec [default
depends on architecture]
@@ -3646,6 +3649,19 @@ if test "$build_pt_chown" = yes; then
fi
+# Check whether --enable-tunables was given.
+if test "${enable_tunables+set}" = set; then :
+ enableval=$enable_tunables; build_tunables=$enableval
+else
+ build_tunables=no
+fi
+
+
+if test "$build_tunables" = yes; then
+ $as_echo "#define BUILD_TUNABLES 1" >>confdefs.h
+
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/configure.ac b/configure.ac
index 123f0d2..b2aaf9b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -395,6 +395,16 @@ if test "$build_pt_chown" = yes; then
AC_DEFINE(HAVE_PT_CHOWN)
fi
+AC_ARG_ENABLE([tunables],
+ [AS_HELP_STRING([--enable-tunables],
+ [Enable tunables support])],
+ [build_tunables=$enableval],
+ [build_tunables=no])
+AC_SUBST(build_tunables)
+if test "$build_tunables" = yes; then
+ AC_DEFINE(BUILD_TUNABLES)
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/csu/init-first.c b/csu/init-first.c
index 77c6e1c..7427121 100644
--- a/csu/init-first.c
+++ b/csu/init-first.c
@@ -28,6 +28,9 @@
#include <libc-internal.h>
#include <ldsodefs.h>
+#if BUILD_TUNABLES
+# include <elf/dl-tunables.h>
+#endif
/* Set nonzero if we have to be prepared for more than one libc being
used in the process. Safe assumption if initializer never runs. */
@@ -74,6 +77,10 @@ _init (int argc, char **argv, char **envp)
#ifndef SHARED
__libc_init_secure ();
+#if BUILD_TUNABLES
+ __tunables_init (envp);
+#endif
+
/* First the initialization which normally would be done by the
dynamic linker. */
_dl_non_dynamic_init ();
diff --git a/elf/Makefile b/elf/Makefile
index 210dde9..c634d120 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -35,6 +35,11 @@ dl-routines = $(addprefix dl-,load lookup object reloc deps hwcaps \
ifeq (yes,$(use-ldconfig))
dl-routines += dl-cache
endif
+
+ifeq (yes,$(build-tunables))
+dl-routines += dl-tunables
+endif
+
all-dl-routines = $(dl-routines) $(sysdep-dl-routines)
# But they are absent from the shared libc, because that code is in ld.so.
elide-routines.os = $(all-dl-routines) dl-support enbl-secure dl-origin \
diff --git a/elf/Versions b/elf/Versions
index 23deda9..1734697 100644
--- a/elf/Versions
+++ b/elf/Versions
@@ -64,5 +64,8 @@ ld {
# Pointer protection.
__pointer_chk_guard;
+
+ # Set value of a tunable.
+ __tunable_set_val;
}
}
diff --git a/elf/dl-tunable-types.h b/elf/dl-tunable-types.h
new file mode 100644
index 0000000..d1591b6
--- /dev/null
+++ b/elf/dl-tunable-types.h
@@ -0,0 +1,46 @@
+/* Tunable type information.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLE_TYPES_H_
+# define _TUNABLE_TYPES_H_
+#include <stddef.h>
+
+typedef void (*tunable_callback_t) (void *);
+
+typedef enum
+{
+ TUNABLE_TYPE_INT_32,
+ TUNABLE_TYPE_SIZE_T,
+ TUNABLE_TYPE_STRING
+} tunable_type_code_t;
+
+typedef struct
+{
+ tunable_type_code_t type_code;
+ int64_t min;
+ int64_t max;
+} tunable_type_t;
+
+typedef union
+{
+ int64_t numval;
+ const char *strval;
+} tunable_val_t;
+
+#endif
diff --git a/elf/dl-tunables.c b/elf/dl-tunables.c
new file mode 100644
index 0000000..ee8c2f0
--- /dev/null
+++ b/elf/dl-tunables.c
@@ -0,0 +1,182 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <stdlib.h>
+
+#define TUNABLES_INTERNAL 1
+#include "dl-tunables.h"
+
+/* Compare environment names, bounded by the name hardcoded in glibc. */
+static bool
+is_name (const char *orig, const char *envname)
+{
+ for (;*orig != '\0' && *envname != '\0'; envname++, orig++)
+ if (*orig != *envname)
+ break;
+
+ /* The ENVNAME is immediately followed by a value. */
+ if (*orig == '\0' && *envname == '=')
+ return true;
+ else
+ return false;
+}
+
+static bool
+get_next_env (char ***envp, char **name, size_t *namelen, char **val)
+{
+ char **ev = *envp;
+
+ while (ev != NULL && *ev != '\0')
+ {
+ char *envline = *ev;
+ int len = 0;
+
+ while (envline[len] != '\0' && envline[len] != '=')
+ len++;
+
+ /* Just the name and no value, go to the next one. */
+ if (envline[len] == '\0')
+ continue;
+
+ *name = envline;
+ *namelen = len;
+ *val = &envline[len + 1];
+ *envp = ++ev;
+
+ return true;
+ }
+
+ return false;
+}
+
+
+#define RETURN_IF_INVALID_RANGE(__cur, __val) \
+({ \
+ __typeof ((__cur)->type.min) min = (__cur)->type.min; \
+ __typeof ((__cur)->type.max) max = (__cur)->type.max; \
+ if (min != max && ((__val) < min || (__val) > max)) \
+ return; \
+})
+
+/* Validate range of the input value and initialize the tunable CUR if it looks
+ good. */
+static void
+tunable_initialize (tunable_t *cur)
+{
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ int32_t val = (int32_t) __strtoul_internal (cur->strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ size_t val = (size_t) __strtoul_internal (cur->strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ break;
+ }
+ case TUNABLE_TYPE_STRING:
+ {
+ cur->val.strval = cur->strval;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+}
+
+/* Initialize the tunables list from the environment. For now we only use the
+ ENV_ALIAS to find values. Later we will also use the tunable names to find
+ values. */
+void
+__tunables_init (char **envp)
+{
+ char *envname = NULL;
+ char *envval = NULL;
+ size_t len = 0;
+
+ while (get_next_env (&envp, &envname, &len, &envval))
+ {
+ for (int i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
+ {
+ tunable_t *cur = &tunable_list[i];
+
+ /* Skip over tunables that have either been set already or should be
+ skipped. */
+ if (cur->strval != NULL || cur->env_alias == NULL
+ || (__libc_enable_secure && !cur->secure_env_alias))
+ continue;
+
+ const char *name = cur->env_alias;
+
+ /* We have a match. Initialize and move on to the next line. */
+ if (is_name (name, envname))
+ {
+ cur->strval = envval;
+ tunable_initialize (cur);
+ break;
+ }
+ }
+ }
+}
+
+/* Set the tunable value. This is called by the module that the tunable exists
+ in. */
+void
+__tunable_set_val (tunable_id_t id, void *valp, tunable_callback_t callback)
+{
+ tunable_t *cur = &tunable_list[id];
+
+ /* Sanity check: don't do anything if our tunable was not set during
+ initialization. */
+ if (cur->strval == NULL)
+ return;
+
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ *((int32_t *) valp) = (int32_t) cur->val.numval;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ *((size_t *) valp) = (size_t) cur->val.numval;
+ break;
+ }
+ case TUNABLE_TYPE_STRING:
+ {
+ *((const char **)valp) = cur->strval;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+
+ if (callback)
+ callback (&cur->val);
+}
diff --git a/elf/dl-tunables.h b/elf/dl-tunables.h
new file mode 100644
index 0000000..c4ae97d
--- /dev/null
+++ b/elf/dl-tunables.h
@@ -0,0 +1,76 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLES_H_
+# define _TUNABLES_H_
+# include <stddef.h>
+# include "dl-tunable-types.h"
+
+/* A tunable. */
+struct _tunable
+{
+ const char *name; /* Internal name of the tunable. */
+ tunable_type_t type; /* Data type of the tunable. */
+ tunable_val_t val; /* The value. */
+ const char *strval; /* The string containing the value,
+ points into envp. */
+ /* Compatibility elements. */
+ const char *env_alias; /* The compatibility environment
+ variable name. */
+ bool secure_env_alias; /* Whether the environment variable
+ must be read even for setuid
+ binaries. */
+};
+
+typedef struct _tunable tunable_t;
+
+/* Full name for a tunable is top_ns.tunable_ns.id. */
+#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id
+
+# define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)
+# define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id
+
+#include "dl-tunable-list.h"
+
+extern void __tunables_init (char **);
+extern void __tunable_set_val (tunable_id_t, void *, tunable_callback_t);
+
+/* Check if the tunable has been set to a non-default value and if it is, copy
+ it over into __VAL. */
+# define TUNABLE_SET_VAL(__id,__val) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ NULL); \
+})
+
+/* Same as TUNABLE_SET_VAL, but also set the callback function to __CB and call
+ it. */
+# define TUNABLE_SET_VAL_WITH_CALLBACK(__id,__val,__cb) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ DL_TUNABLE_CALLBACK (__cb)); \
+})
+
+/* Namespace sanity for callback functions. Use this macro to keep the
+ namespace of the modules clean. */
+#define DL_TUNABLE_CALLBACK(__name) _dl_tunable_ ## __name
+#endif
diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list
new file mode 100644
index 0000000..342e3d5
--- /dev/null
+++ b/elf/dl-tunables.list
@@ -0,0 +1,50 @@
+# Allowed attributes for tunables:
+#
+# type: Defaults to STRING
+# minval: Optional minimum acceptable value
+# maxval: Optional maximum acceptable value
+# env_alias: An alias environment variable
+# secure_env_alias: Specify whether the environment variable should be read for
+# setuid binaries.
+
+glibc {
+ malloc {
+ check {
+ type: INT_32
+ minval: 0
+ maxval: 3
+ env_alias: MALLOC_CHECK_
+ secure_env_alias: true
+ }
+ top_pad {
+ type: SIZE_T
+ env_alias: MALLOC_TOP_PAD_
+ }
+ perturb {
+ type: INT_32
+ minval: 0
+ maxval: 0xff
+ env_alias: MALLOC_PERTURB_
+ }
+ mmap_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_MMAP_THRESHOLD_
+ }
+ trim_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_TRIM_THRESHOLD_
+ }
+ mmap_max {
+ type: INT_32
+ env_alias: MALLOC_MMAP_MAX_
+ }
+ arena_max {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_MAX
+ }
+ arena_test {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_TEST
+ }
+ }
+}
diff --git a/elf/rtld.c b/elf/rtld.c
index 647661c..263723a 100644
--- a/elf/rtld.c
+++ b/elf/rtld.c
@@ -44,6 +44,10 @@
#include <assert.h>
+#if BUILD_TUNABLES
+# include <dl-tunables.h>
+#endif
+
/* Avoid PLT use for our local calls at startup. */
extern __typeof (__mempcpy) __mempcpy attribute_hidden;
@@ -2346,6 +2350,10 @@ process_envvars (enum mode *modep)
enum mode mode = normal;
char *debug_output = NULL;
+#if BUILD_TUNABLES
+ __tunables_init (_environ);
+#endif
+
/* This is the default place for profiling data file. */
GLRO(dl_profile_output)
= &"/var/tmp\0/var/profile"[__libc_enable_secure ? 9 : 0];
diff --git a/malloc/Makefile b/malloc/Makefile
index fa1730e..23ab2f4 100644
--- a/malloc/Makefile
+++ b/malloc/Makefile
@@ -31,6 +31,8 @@ tests := mallocbug tst-malloc tst-valloc tst-calloc tst-obstack \
tst-malloc-backtrace tst-malloc-thread-exit \
tst-malloc-thread-fail tst-malloc-fork-deadlock \
tst-mallocfork2
+tests-static := tst-malloc-usable-static
+tests += $(tests-static)
test-srcs = tst-mtrace
routines = malloc morecore mcheck mtrace obstack \
@@ -138,6 +140,7 @@ endif
tst-mcheck-ENV = MALLOC_CHECK_=3
tst-malloc-usable-ENV = MALLOC_CHECK_=3
+tst-malloc-usable-static-ENV = $(tst-malloc-usable-ENV)
# Uncomment this for test releases. For public releases it is too expensive.
#CPPFLAGS-malloc.o += -DMALLOC_DEBUG=1
diff --git a/malloc/arena.c b/malloc/arena.c
index 229783f..7de4caf 100644
--- a/malloc/arena.c
+++ b/malloc/arena.c
@@ -19,6 +19,11 @@
#include <stdbool.h>
+#if BUILD_TUNABLES
+# define TUNABLE_NAMESPACE malloc
+# include <elf/dl-tunables.h>
+#endif
+
/* Compile-time constants. */
#define HEAP_MIN_SIZE (32 * 1024)
@@ -204,6 +209,15 @@ __malloc_fork_unlock_child (void)
mutex_init (&list_lock);
}
+#if BUILD_TUNABLES
+void
+DL_TUNABLE_CALLBACK (set_mallopt_check) (void *unused)
+{
+ if (check_action != 0)
+ __malloc_check_init ();
+}
+
+#else
/* Initialization routine. */
#include <string.h>
extern char **_environ;
@@ -238,6 +252,7 @@ next_env_entry (char ***position)
return result;
}
+#endif
#ifdef SHARED
@@ -272,6 +287,24 @@ ptmalloc_init (void)
#endif
thread_arena = &main_arena;
+
+#if BUILD_TUNABLES
+ /* Ensure initialization/consolidation and do it under a lock so that a
+ thread attempting to use the arena in parallel waits on us till we
+ finish. */
+ mutex_lock (&main_arena.mutex);
+ malloc_consolidate (&main_arena);
+
+ TUNABLE_SET_VAL_WITH_CALLBACK (check, &check_action, set_mallopt_check);
+ TUNABLE_SET_VAL (top_pad, &mp_.top_pad);
+ TUNABLE_SET_VAL (perturb, &perturb_byte);
+ TUNABLE_SET_VAL (mmap_threshold, &mp_.mmap_threshold);
+ TUNABLE_SET_VAL (trim_threshold, &mp_.trim_threshold);
+ TUNABLE_SET_VAL (mmap_max, &mp_.n_mmaps_max);
+ TUNABLE_SET_VAL (arena_max, &mp_.arena_max);
+ TUNABLE_SET_VAL (arena_test, &mp_.arena_test);
+ mutex_unlock (&main_arena.mutex);
+#else
const char *s = NULL;
if (__glibc_likely (_environ != NULL))
{
@@ -340,6 +373,8 @@ ptmalloc_init (void)
if (check_action != 0)
__malloc_check_init ();
}
+#endif
+
#if HAVE_MALLOC_INIT_HOOK
void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook);
if (hook != NULL)
diff --git a/malloc/tst-malloc-usable-static.c b/malloc/tst-malloc-usable-static.c
new file mode 100644
index 0000000..8907db0
--- /dev/null
+++ b/malloc/tst-malloc-usable-static.c
@@ -0,0 +1 @@
+#include <malloc/tst-malloc-usable.c>
diff --git a/manual/install.texi b/manual/install.texi
index 79ee45f..f9b4784 100644
--- a/manual/install.texi
+++ b/manual/install.texi
@@ -189,6 +189,11 @@ configure with @option{--disable-werror}.
By default for x86_64, @theglibc{} is built with vector math library.
Use this option to disable vector math library.
+@item --enable-tunables
+Tunables support allows additional library parameters to be customized at
+runtime for each application. This is an experimental feature and affects
+startup time and is thus disabled by default.
+
@item --build=@var{build-system}
@itemx --host=@var{host-system}
These options are for cross-compiling. If you specify both options and
diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
new file mode 100644
index 0000000..1b930e7
--- /dev/null
+++ b/scripts/gen-tunables.awk
@@ -0,0 +1,157 @@
+# Generate dl-tunable-list.h from dl-tunables.list
+
+BEGIN {
+ tunable=""
+ ns=""
+ top_ns=""
+}
+
+# Skip over blank lines and comments.
+/^#/ {
+ next
+}
+
+/^[ \t]*$/ {
+ next
+}
+
+# Beginning of either a top namespace, tunable namespace or a tunable, decided
+# on the current value of TUNABLE, NS or TOP_NS.
+$2 == "{" {
+ if (top_ns == "") {
+ top_ns = $1
+ }
+ else if (ns == "") {
+ ns = $1
+ }
+ else if (tunable == "") {
+ tunable = $1
+ }
+ else {
+ printf ("Unexpected occurrence of '{': %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+
+ next
+}
+
+# End of either a top namespace, tunable namespace or a tunable.
+$1 == "}" {
+ if (tunable != "") {
+ # Tunables definition ended, now fill in default attributes.
+ if (!types[top_ns][ns][tunable]) {
+ types[top_ns][ns][tunable] = "STRING"
+ }
+ if (!minvals[top_ns][ns][tunable]) {
+ minvals[top_ns][ns][tunable] = "0"
+ }
+ if (!maxvals[top_ns][ns][tunable]) {
+ maxvals[top_ns][ns][tunable] = "0"
+ }
+ if (!env_alias[top_ns][ns][tunable]) {
+ env_alias[top_ns][ns][tunable] = "NULL"
+ }
+ if (!secure_env_alias[top_ns][ns][tunable]) {
+ secure_env_alias[top_ns][ns][tunable] = "false"
+ }
+
+ tunable = ""
+ }
+ else if (ns != "") {
+ ns = ""
+ }
+ else if (top_ns != "") {
+ top_ns = ""
+ }
+ else {
+ printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+ next
+}
+
+# Everything else, which could either be a tunable without any attributes or a
+# tunable attribute.
+{
+ if (ns == "") {
+ printf("Line %d: Invalid tunable outside a namespace: %s\n", NR, $0)
+ exit 1
+ }
+
+ if (tunable == "") {
+ # We encountered a tunable without any attributes, so note it with a
+ # default.
+ types[top_ns][ns][$1] = "STRING"
+ next
+ }
+
+ # Otherwise, we have encountered a tunable attribute.
+ split($0, arr, ":")
+ attr = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[1])
+ val = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[2])
+
+ if (attr == "type") {
+ types[top_ns][ns][tunable] = val
+ }
+ else if (attr == "minval") {
+ minvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "maxval") {
+ maxvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "env_alias") {
+ env_alias[top_ns][ns][tunable] = sprintf("\"%s\"", val)
+ }
+ else if (attr == "secure_env_alias") {
+ if (val == "true" || val == "false") {
+ secure_env_alias[top_ns][ns][tunable] = val
+ }
+ else {
+ printf("Line %d: Invalid value (%s) for secure_env_alias: %s, ", NR, val,
+ $0)
+ print("Allowed values are 'true' or 'false'")
+ exit 1
+ }
+ }
+}
+
+END {
+ if (ns != "") {
+ print "Unterminated namespace. Is a closing brace missing?"
+ exit 1
+ }
+
+ print "/* AUTOGENERATED by gen-tunables.awk. */"
+ print "#ifndef _TUNABLES_H_"
+ print "# error \"Do not include this file directly.\""
+ print "# error \"Include tunables.h instead.\""
+ print "#endif"
+
+ # Now, the enum names
+ print "\ntypedef enum"
+ print "{"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, m);
+ }
+ }
+ }
+ print "} tunable_id_t;\n"
+
+ # Finally, the tunable list.
+ print "\n#ifdef TUNABLES_INTERNAL"
+ print "static tunable_t tunable_list[] = {"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" {TUNABLE_NAME_S(%s, %s, %s)", t, n, m)
+ printf (", {TUNABLE_TYPE_%s, %s, %s}, {.numval = 0}, NULL, %s, %s},\n",
+ types[t][n][m], minvals[t][n][m], maxvals[t][n][m],
+ env_alias[t][n][m], secure_env_alias[t][n][m]);
+ }
+ }
+ }
+ print "};"
+ print "#endif"
+}
--
2.5.5
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH 1/2] Add framework for tunables
2016-07-01 18:31 [PATCH 0/2] Tunables for glibc Siddhesh Poyarekar
@ 2016-07-01 18:32 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-07-01 18:32 UTC (permalink / raw)
To: libc-alpha
The tunables framework allows us to uniformly manage and expose global
variables inside glibc as switches to users. tunables/README has
instructions for glibc developers to add new tunables.
Tunables support can be enabled by passing the --enable-tunables
configure flag to the configure script. This patch only adds a
framework and does not pose any limitations on how tunable values are
read from the user. It also adds environment variables used in malloc
behaviour tweaking to the tunables framework as a PoC of the
compatibility interface.
* manual/install.texi: Add --enable-tunables option.
* INSTALL: Regenerate.
* Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
(before-compile): Generate dl-tunable-list.h early.
* config.h.in: Add BUILD_TUNABLES.
* config.make.in: Add build-tunables.
* configure.ac: Add --enable-tunables option.
* configure: Regenerate.
* malloc/arena.c [BUILD_TUNABLES]: Include dl-tunables.h.
Define TUNABLE_NAMESPACE.
(DL_TUNABLE_CALLBACK(set_mallopt_check)): New function.
(ptmalloc_init): Set tunable values.
* scripts/gen-tunables.awk: New file.
* README.tunables: New file.
* elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE
namespace.
* elf/dl-tunable-list.h: New auto-generated file.
* elf/dl-tunables.c: New file.
* elf/dl-tunables.h: New file.
* elf/dl-tunables.list: New file.
* elf/dl-tunable-types.h: New file.
* elf/rtld.c [BUILD_TUNABLES]: Include dl-tunables.h
(process_envvars): Call __tunables_init.
---
INSTALL | 6 ++
Makeconfig | 16 +++++
README.tunables | 74 ++++++++++++++++++++
config.h.in | 3 +
config.make.in | 1 +
configure | 16 +++++
configure.ac | 10 +++
elf/Makefile | 5 ++
elf/Versions | 3 +
elf/dl-tunable-types.h | 45 +++++++++++++
elf/dl-tunables.c | 172 +++++++++++++++++++++++++++++++++++++++++++++++
elf/dl-tunables.h | 76 +++++++++++++++++++++
elf/dl-tunables.list | 50 ++++++++++++++
elf/rtld.c | 8 +++
malloc/arena.c | 28 ++++++++
manual/install.texi | 5 ++
scripts/gen-tunables.awk | 157 ++++++++++++++++++++++++++++++++++++++++++
17 files changed, 675 insertions(+)
create mode 100644 README.tunables
create mode 100644 elf/dl-tunable-types.h
create mode 100644 elf/dl-tunables.c
create mode 100644 elf/dl-tunables.h
create mode 100644 elf/dl-tunables.list
create mode 100644 scripts/gen-tunables.awk
diff --git a/INSTALL b/INSTALL
index ec3445f..90dc481 100644
--- a/INSTALL
+++ b/INSTALL
@@ -158,6 +158,12 @@ will be used, and CFLAGS sets optimization options for the compiler.
By default for x86_64, the GNU C Library is built with vector math
library. Use this option to disable vector math library.
+'--enable-tunables'
+ Tunables support allows additional library parameters to be
+ customized at runtime for each application. This is an
+ experimental feature and affects startup time and is thus disabled
+ by default.
+
'--build=BUILD-SYSTEM'
'--host=HOST-SYSTEM'
These options are for cross-compiling. If you specify both options
diff --git a/Makeconfig b/Makeconfig
index 901e253..70ab097 100644
--- a/Makeconfig
+++ b/Makeconfig
@@ -883,6 +883,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \
$(foreach lib,$(libof-$(basename $(@F))) \
$(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \
$(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F)))
+
+ifeq (yes,$(build-tunables))
+CPPFLAGS += -DTOP_NAMESPACE=glibc
+endif
+
override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \
$(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \
$(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \
@@ -1054,6 +1059,17 @@ $(common-objpfx)libc-modules.stmp: $(..)scripts/gen-libc-modules.awk \
endif
+# Build the tunables list header early since it could be used by any module in
+# glibc.
+ifeq (yes,$(build-tunables))
+before-compile += $(common-objpfx)dl-tunable-list.h
+
+$(common-objpfx)dl-tunable-list.h: $(..)scripts/gen-tunables.awk \
+ $(..)elf/dl-tunables.list
+ $(AWK) -f $^ > $@.tmp
+ mv $@{.tmp,}
+endif
+
common-generated += libc-modules.h libc-modules.stmp
# The name under which the run-time dynamic linker is installed.
diff --git a/README.tunables b/README.tunables
new file mode 100644
index 0000000..8890249
--- /dev/null
+++ b/README.tunables
@@ -0,0 +1,74 @@
+ TUNABLE FRAMEWORK
+ =================
+
+The tunable framework allows modules within glibc to register variables that
+may be tweaked through an environment variable or an API call. It aims to
+enforce a strict namespace rule to bring consistency to naming of these tunable
+environment variables across the project.
+
+ADDING A NEW TUNABLE
+--------------------
+
+The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in
+distributions for specific tunables if they want to add their own tunables.
+Downstream implementations are discouraged from using the 'glibc' namespace for
+tunables they don't already have consensus to push upstream.
+
+There are two steps to adding a tunable:
+
+1. Add a tunable ID:
+
+Modules that wish to use the tunables interface must define the
+TUNABLE_NAMESPACE macro. Following this, for each tunable you want to
+add, make an entry in elf/dl-tunables.list. The format of the file is as
+follows:
+
+TOP_NAMESPACE {
+ NAMESPACE1 {
+ TUNABLE1 {
+ # tunable attributes, one per line
+ }
+ # A tunable with default attributes, i.e. string variable.
+ TUNABLE2
+ TUNABLE3 {
+ # its attributes
+ }
+ }
+ NAMESPACE2 {
+ ...
+ }
+}
+
+The list of allowed attributes are:
+
+- type: Data type. Defaults to STRING. Allowed types are:
+ INT_32, SIZE_T. Note that as of now the STRING type is
+ neither defined nor supported since no tunables need
+ it. It will be in future though, when needed.
+
+- minval: Optional minimum acceptable value
+
+- maxval: Optional maximum acceptable value
+
+- env_alias: An alias environment variable
+
+- secure_env_alias: Specify whether the environment variable should be read
+ for setuid binaries.
+
+2. Call either the TUNABLE_SET_VALUE and pass into it the tunable name and a
+ pointer to the variable that should be set with the tunable value.
+ If additional work needs to be done after setting the value, use the
+ TUNABLE_SET_VALUE_WITH_CALLBACK instead and additionally pass a pointer to
+ the function that should be called if the tunable value has been set.
+
+FUTURE WORK
+-----------
+
+The framework currently only allows a one-time initialization of variables
+through environment variables and in some cases, modification of variables via
+an API call. A future goals for this project include:
+
+- Setting system-wide and user-wide defaults for tunables through some
+ mechanism like a configuration file.
+
+- Allow tweaking of some tunables at runtime
diff --git a/config.h.in b/config.h.in
index 856ef6a..6112782 100644
--- a/config.h.in
+++ b/config.h.in
@@ -240,4 +240,7 @@
/* PowerPC32 uses fctidz for floating point to long long conversions. */
#define HAVE_PPC_FCTIDZ 0
+/* Build glibc with tunables support. */
+#define BUILD_TUNABLES 0
+
#endif
diff --git a/config.make.in b/config.make.in
index 95c6f36..653ef6f 100644
--- a/config.make.in
+++ b/config.make.in
@@ -91,6 +91,7 @@ use-nscd = @use_nscd@
build-hardcoded-path-in-tests= @hardcoded_path_in_tests@
build-pt-chown = @build_pt_chown@
enable-lock-elision = @enable_lock_elision@
+build-tunables = @build_tunables@
# Build tools.
CC = @CC@
diff --git a/configure b/configure
index 19a4829..1534ba7 100755
--- a/configure
+++ b/configure
@@ -659,6 +659,7 @@ multi_arch
base_machine
add_on_subdirs
add_ons
+build_tunables
build_pt_chown
build_nscd
link_obsolete_rpc
@@ -773,6 +774,7 @@ enable_systemtap
enable_build_nscd
enable_nscd
enable_pt_chown
+enable_tunables
enable_mathvec
with_cpu
'
@@ -1440,6 +1442,7 @@ Optional Features:
--disable-build-nscd disable building and installing the nscd daemon
--disable-nscd library functions will not contact the nscd daemon
--enable-pt_chown Enable building and installing pt_chown
+ --enable-tunables Enable tunables support
--enable-mathvec Enable building and installing mathvec [default
depends on architecture]
@@ -3646,6 +3649,19 @@ if test "$build_pt_chown" = yes; then
fi
+# Check whether --enable-tunables was given.
+if test "${enable_tunables+set}" = set; then :
+ enableval=$enable_tunables; build_tunables=$enableval
+else
+ build_tunables=no
+fi
+
+
+if test "$build_tunables" = yes; then
+ $as_echo "#define BUILD_TUNABLES 1" >>confdefs.h
+
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/configure.ac b/configure.ac
index 123f0d2..b2aaf9b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -395,6 +395,16 @@ if test "$build_pt_chown" = yes; then
AC_DEFINE(HAVE_PT_CHOWN)
fi
+AC_ARG_ENABLE([tunables],
+ [AS_HELP_STRING([--enable-tunables],
+ [Enable tunables support])],
+ [build_tunables=$enableval],
+ [build_tunables=no])
+AC_SUBST(build_tunables)
+if test "$build_tunables" = yes; then
+ AC_DEFINE(BUILD_TUNABLES)
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/elf/Makefile b/elf/Makefile
index 210dde9..c634d120 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -35,6 +35,11 @@ dl-routines = $(addprefix dl-,load lookup object reloc deps hwcaps \
ifeq (yes,$(use-ldconfig))
dl-routines += dl-cache
endif
+
+ifeq (yes,$(build-tunables))
+dl-routines += dl-tunables
+endif
+
all-dl-routines = $(dl-routines) $(sysdep-dl-routines)
# But they are absent from the shared libc, because that code is in ld.so.
elide-routines.os = $(all-dl-routines) dl-support enbl-secure dl-origin \
diff --git a/elf/Versions b/elf/Versions
index 23deda9..1734697 100644
--- a/elf/Versions
+++ b/elf/Versions
@@ -64,5 +64,8 @@ ld {
# Pointer protection.
__pointer_chk_guard;
+
+ # Set value of a tunable.
+ __tunable_set_val;
}
}
diff --git a/elf/dl-tunable-types.h b/elf/dl-tunable-types.h
new file mode 100644
index 0000000..ecbb197
--- /dev/null
+++ b/elf/dl-tunable-types.h
@@ -0,0 +1,45 @@
+/* Tunable type information.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLE_TYPES_H_
+# define _TUNABLE_TYPES_H_
+#include <stddef.h>
+
+typedef void (*tunable_callback_t) (void);
+
+typedef enum
+{
+ TUNABLE_TYPE_INT_32,
+ TUNABLE_TYPE_SIZE_T
+} tunable_type_code_t;
+
+typedef struct
+{
+ tunable_type_code_t type_code;
+ int64_t min;
+ int64_t max;
+} tunable_type_t;
+
+typedef union
+{
+ int64_t numval;
+ const char *strval;
+} tunable_val_t;
+
+#endif
diff --git a/elf/dl-tunables.c b/elf/dl-tunables.c
new file mode 100644
index 0000000..8a26dfc
--- /dev/null
+++ b/elf/dl-tunables.c
@@ -0,0 +1,172 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <stdlib.h>
+
+#define TUNABLES_INTERNAL 1
+#include "dl-tunables.h"
+
+/* Compare environment names, bounded by the name hardcoded in glibc. */
+static bool
+is_name (const char *orig, const char *envname)
+{
+ for (;*orig != '\0' && *envname != '\0'; envname++, orig++)
+ if (*orig != *envname)
+ break;
+
+ /* The ENVNAME is immediately followed by a value. */
+ if (*orig == '\0' && *envname == '=')
+ return true;
+ else
+ return false;
+}
+
+static bool
+get_next_env (char ***envp, char **name, size_t *namelen, char **val)
+{
+ char **ev = *envp;
+
+ while (ev != NULL && *ev != '\0')
+ {
+ char *envline = *ev;
+ int len = 0;
+
+ while (envline[len] != '\0' && envline[len] != '=')
+ len++;
+
+ /* Just the name and no value, go to the next one. */
+ if (envline[len] == '\0')
+ continue;
+
+ *name = envline;
+ *namelen = len;
+ *val = &envline[len + 1];
+ *envp = ++ev;
+
+ return true;
+ }
+
+ return false;
+}
+
+
+#define RETURN_IF_INVALID_RANGE(__cur, __val) \
+({ \
+ __typeof ((__cur)->type.min) min = (__cur)->type.min; \
+ __typeof ((__cur)->type.max) max = (__cur)->type.max; \
+ if (min != max && ((__val) < min || (__val) > max)) \
+ return; \
+})
+
+/* Validate range of the input value and initialize the tunable CUR if it looks
+ good. */
+static void
+tunable_initialize (tunable_t *cur)
+{
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ int32_t val = (int32_t) __strtoul_internal (cur->strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ size_t val = (size_t) __strtoul_internal (cur->strval, NULL, 0, 0);
+ RETURN_IF_INVALID_RANGE (cur, val);
+ cur->val.numval = (int64_t) val;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+}
+
+/* Initialize the tunables list from the environment. For now we only use the
+ ENV_ALIAS to find values. Later we will also use the tunable names to find
+ values. */
+void
+__tunables_init (char **envp)
+{
+ char *envname = NULL;
+ char *envval = NULL;
+ size_t len = 0;
+
+ while (get_next_env (&envp, &envname, &len, &envval))
+ {
+ for (int i = 0; i < sizeof (tunable_list) / sizeof (tunable_t); i++)
+ {
+ tunable_t *cur = &tunable_list[i];
+
+ /* Skip over tunables that have either been set already or should be
+ skipped. */
+ if (cur->strval != NULL || cur->env_alias == NULL
+ || (__libc_enable_secure && !cur->secure_env_alias))
+ continue;
+
+ const char *name = cur->env_alias;
+
+ /* We have a match. Initialize and move on to the next line. */
+ if (is_name (name, envname))
+ {
+ cur->strval = envval;
+ tunable_initialize (cur);
+ break;
+ }
+ }
+ }
+}
+
+/* Set the tunable value. This is called by the module that the tunable exists
+ in. */
+void
+__tunable_set_val (tunable_id_t id, void *valp, void (*callback) (void))
+{
+ tunable_t *cur = &tunable_list[id];
+
+ /* Sanity check: don't do anything if our tunable was not set during
+ initialization. */
+ if (cur->strval == NULL)
+ return;
+
+ switch (cur->type.type_code)
+ {
+ case TUNABLE_TYPE_INT_32:
+ {
+ *((int32_t *) valp) = (int32_t) cur->val.numval;
+ break;
+ }
+ case TUNABLE_TYPE_SIZE_T:
+ {
+ *((size_t *) valp) = (size_t) cur->val.numval;
+ break;
+ }
+ default:
+ __builtin_unreachable ();
+ }
+
+ if (callback)
+ callback ();
+}
diff --git a/elf/dl-tunables.h b/elf/dl-tunables.h
new file mode 100644
index 0000000..93bcad7
--- /dev/null
+++ b/elf/dl-tunables.h
@@ -0,0 +1,76 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _TUNABLES_H_
+# define _TUNABLES_H_
+# include <stddef.h>
+# include "dl-tunable-types.h"
+
+/* A tunable. */
+struct _tunable
+{
+ const char *name; /* Internal name of the tunable. */
+ tunable_type_t type; /* Data type of the tunable. */
+ tunable_val_t val; /* The value. */
+ const char *strval; /* The string containing the value,
+ points into envp. */
+ /* Compatibility elements. */
+ const char *env_alias; /* The compatibility environment
+ variable name. */
+ bool secure_env_alias; /* Whether the environment variable
+ must be read even for setuid
+ binaries. */
+};
+
+typedef struct _tunable tunable_t;
+
+/* Full name for a tunable is top_ns.tunable_ns.id. */
+#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id
+
+# define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)
+# define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id
+
+#include "dl-tunable-list.h"
+
+extern void __tunables_init (char **);
+extern void __tunable_set_val (tunable_id_t, void *, void (*) (void));
+
+/* Check if the tunable has been set to a non-default value and if it is, copy
+ it over into __VAL. */
+# define TUNABLE_SET_VAL(__id,__val) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ NULL); \
+})
+
+/* Same as TUNABLE_SET_VAL, but also set the callback function to __CB and call
+ it. */
+# define TUNABLE_SET_VAL_WITH_CALLBACK(__id,__val,__cb) \
+({ \
+ __tunable_set_val \
+ (TUNABLE_ENUM_NAME (TOP_NAMESPACE, TUNABLE_NAMESPACE, __id), (__val), \
+ DL_TUNABLE_CALLBACK (__cb)); \
+})
+
+/* Namespace sanity for callback functions. Use this macro to keep the
+ namespace of the modules clean. */
+#define DL_TUNABLE_CALLBACK(__name) _dl_tunable_ ## __name
+#endif
diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list
new file mode 100644
index 0000000..342e3d5
--- /dev/null
+++ b/elf/dl-tunables.list
@@ -0,0 +1,50 @@
+# Allowed attributes for tunables:
+#
+# type: Defaults to STRING
+# minval: Optional minimum acceptable value
+# maxval: Optional maximum acceptable value
+# env_alias: An alias environment variable
+# secure_env_alias: Specify whether the environment variable should be read for
+# setuid binaries.
+
+glibc {
+ malloc {
+ check {
+ type: INT_32
+ minval: 0
+ maxval: 3
+ env_alias: MALLOC_CHECK_
+ secure_env_alias: true
+ }
+ top_pad {
+ type: SIZE_T
+ env_alias: MALLOC_TOP_PAD_
+ }
+ perturb {
+ type: INT_32
+ minval: 0
+ maxval: 0xff
+ env_alias: MALLOC_PERTURB_
+ }
+ mmap_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_MMAP_THRESHOLD_
+ }
+ trim_threshold {
+ type: SIZE_T
+ env_alias: MALLOC_TRIM_THRESHOLD_
+ }
+ mmap_max {
+ type: INT_32
+ env_alias: MALLOC_MMAP_MAX_
+ }
+ arena_max {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_MAX
+ }
+ arena_test {
+ type: SIZE_T
+ env_alias: MALLOC_ARENA_TEST
+ }
+ }
+}
diff --git a/elf/rtld.c b/elf/rtld.c
index 647661c..263723a 100644
--- a/elf/rtld.c
+++ b/elf/rtld.c
@@ -44,6 +44,10 @@
#include <assert.h>
+#if BUILD_TUNABLES
+# include <dl-tunables.h>
+#endif
+
/* Avoid PLT use for our local calls at startup. */
extern __typeof (__mempcpy) __mempcpy attribute_hidden;
@@ -2346,6 +2350,10 @@ process_envvars (enum mode *modep)
enum mode mode = normal;
char *debug_output = NULL;
+#if BUILD_TUNABLES
+ __tunables_init (_environ);
+#endif
+
/* This is the default place for profiling data file. */
GLRO(dl_profile_output)
= &"/var/tmp\0/var/profile"[__libc_enable_secure ? 9 : 0];
diff --git a/malloc/arena.c b/malloc/arena.c
index 229783f..c962d4e 100644
--- a/malloc/arena.c
+++ b/malloc/arena.c
@@ -19,6 +19,11 @@
#include <stdbool.h>
+#if BUILD_TUNABLES
+# define TUNABLE_NAMESPACE malloc
+# include <elf/dl-tunables.h>
+#endif
+
/* Compile-time constants. */
#define HEAP_MIN_SIZE (32 * 1024)
@@ -204,6 +209,15 @@ __malloc_fork_unlock_child (void)
mutex_init (&list_lock);
}
+#if BUILD_TUNABLES
+void
+DL_TUNABLE_CALLBACK (set_mallopt_check) (void)
+{
+ if (check_action != 0)
+ __malloc_check_init ();
+}
+
+#else
/* Initialization routine. */
#include <string.h>
extern char **_environ;
@@ -238,6 +252,7 @@ next_env_entry (char ***position)
return result;
}
+#endif
#ifdef SHARED
@@ -272,6 +287,17 @@ ptmalloc_init (void)
#endif
thread_arena = &main_arena;
+
+#if BUILD_TUNABLES
+ TUNABLE_SET_VAL_WITH_CALLBACK (check, &check_action, set_mallopt_check);
+ TUNABLE_SET_VAL (top_pad, &mp_.top_pad);
+ TUNABLE_SET_VAL (perturb, &perturb_byte);
+ TUNABLE_SET_VAL (mmap_threshold, &mp_.mmap_threshold);
+ TUNABLE_SET_VAL (trim_threshold, &mp_.trim_threshold);
+ TUNABLE_SET_VAL (mmap_max, &mp_.n_mmaps_max);
+ TUNABLE_SET_VAL (arena_max, &mp_.arena_max);
+ TUNABLE_SET_VAL (arena_test, &mp_.arena_test);
+#else
const char *s = NULL;
if (__glibc_likely (_environ != NULL))
{
@@ -340,6 +366,8 @@ ptmalloc_init (void)
if (check_action != 0)
__malloc_check_init ();
}
+#endif
+
#if HAVE_MALLOC_INIT_HOOK
void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook);
if (hook != NULL)
diff --git a/manual/install.texi b/manual/install.texi
index 79ee45f..f9b4784 100644
--- a/manual/install.texi
+++ b/manual/install.texi
@@ -189,6 +189,11 @@ configure with @option{--disable-werror}.
By default for x86_64, @theglibc{} is built with vector math library.
Use this option to disable vector math library.
+@item --enable-tunables
+Tunables support allows additional library parameters to be customized at
+runtime for each application. This is an experimental feature and affects
+startup time and is thus disabled by default.
+
@item --build=@var{build-system}
@itemx --host=@var{host-system}
These options are for cross-compiling. If you specify both options and
diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
new file mode 100644
index 0000000..1b930e7
--- /dev/null
+++ b/scripts/gen-tunables.awk
@@ -0,0 +1,157 @@
+# Generate dl-tunable-list.h from dl-tunables.list
+
+BEGIN {
+ tunable=""
+ ns=""
+ top_ns=""
+}
+
+# Skip over blank lines and comments.
+/^#/ {
+ next
+}
+
+/^[ \t]*$/ {
+ next
+}
+
+# Beginning of either a top namespace, tunable namespace or a tunable, decided
+# on the current value of TUNABLE, NS or TOP_NS.
+$2 == "{" {
+ if (top_ns == "") {
+ top_ns = $1
+ }
+ else if (ns == "") {
+ ns = $1
+ }
+ else if (tunable == "") {
+ tunable = $1
+ }
+ else {
+ printf ("Unexpected occurrence of '{': %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+
+ next
+}
+
+# End of either a top namespace, tunable namespace or a tunable.
+$1 == "}" {
+ if (tunable != "") {
+ # Tunables definition ended, now fill in default attributes.
+ if (!types[top_ns][ns][tunable]) {
+ types[top_ns][ns][tunable] = "STRING"
+ }
+ if (!minvals[top_ns][ns][tunable]) {
+ minvals[top_ns][ns][tunable] = "0"
+ }
+ if (!maxvals[top_ns][ns][tunable]) {
+ maxvals[top_ns][ns][tunable] = "0"
+ }
+ if (!env_alias[top_ns][ns][tunable]) {
+ env_alias[top_ns][ns][tunable] = "NULL"
+ }
+ if (!secure_env_alias[top_ns][ns][tunable]) {
+ secure_env_alias[top_ns][ns][tunable] = "false"
+ }
+
+ tunable = ""
+ }
+ else if (ns != "") {
+ ns = ""
+ }
+ else if (top_ns != "") {
+ top_ns = ""
+ }
+ else {
+ printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+ next
+}
+
+# Everything else, which could either be a tunable without any attributes or a
+# tunable attribute.
+{
+ if (ns == "") {
+ printf("Line %d: Invalid tunable outside a namespace: %s\n", NR, $0)
+ exit 1
+ }
+
+ if (tunable == "") {
+ # We encountered a tunable without any attributes, so note it with a
+ # default.
+ types[top_ns][ns][$1] = "STRING"
+ next
+ }
+
+ # Otherwise, we have encountered a tunable attribute.
+ split($0, arr, ":")
+ attr = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[1])
+ val = gensub(/^[ \t]+|[ \t]+$/, "", "g", arr[2])
+
+ if (attr == "type") {
+ types[top_ns][ns][tunable] = val
+ }
+ else if (attr == "minval") {
+ minvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "maxval") {
+ maxvals[top_ns][ns][tunable] = val
+ }
+ else if (attr == "env_alias") {
+ env_alias[top_ns][ns][tunable] = sprintf("\"%s\"", val)
+ }
+ else if (attr == "secure_env_alias") {
+ if (val == "true" || val == "false") {
+ secure_env_alias[top_ns][ns][tunable] = val
+ }
+ else {
+ printf("Line %d: Invalid value (%s) for secure_env_alias: %s, ", NR, val,
+ $0)
+ print("Allowed values are 'true' or 'false'")
+ exit 1
+ }
+ }
+}
+
+END {
+ if (ns != "") {
+ print "Unterminated namespace. Is a closing brace missing?"
+ exit 1
+ }
+
+ print "/* AUTOGENERATED by gen-tunables.awk. */"
+ print "#ifndef _TUNABLES_H_"
+ print "# error \"Do not include this file directly.\""
+ print "# error \"Include tunables.h instead.\""
+ print "#endif"
+
+ # Now, the enum names
+ print "\ntypedef enum"
+ print "{"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, m);
+ }
+ }
+ }
+ print "} tunable_id_t;\n"
+
+ # Finally, the tunable list.
+ print "\n#ifdef TUNABLES_INTERNAL"
+ print "static tunable_t tunable_list[] = {"
+ for (t in types) {
+ for (n in types[t]) {
+ for (m in types[t][n]) {
+ printf (" {TUNABLE_NAME_S(%s, %s, %s)", t, n, m)
+ printf (", {TUNABLE_TYPE_%s, %s, %s}, {.numval = 0}, NULL, %s, %s},\n",
+ types[t][n][m], minvals[t][n][m], maxvals[t][n][m],
+ env_alias[t][n][m], secure_env_alias[t][n][m]);
+ }
+ }
+ }
+ print "};"
+ print "#endif"
+}
--
2.5.5
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 19:02 ` Joseph Myers
@ 2016-01-12 2:18 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-01-12 2:18 UTC (permalink / raw)
To: Joseph Myers; +Cc: libc-alpha, roland, carlos, Paul E. Murphy, Andi Kleen
On Mon, Jan 11, 2016 at 07:02:46PM +0000, Joseph Myers wrote:
> On Mon, 11 Jan 2016, Siddhesh Poyarekar wrote:
>
> > +# * False positive: tunable_register, compat_tunables_init_envvars,
> > +# tunables_init. These functions are in the glibc internal namespace.
> > @whitelist = qw(stdin stdout stderr re_syntax_options matherr matherrf
> > - matherrl);
> > + matherrl tunable_register compat_tunables_init_envvars tunables_init);
>
> I don't understand this comment. None of those names are in an internal
> namespace. All those functions should be renamed to start with __.
Oh, I meant that they're in the GLIBC_PRIVATE namespace. In any case,
I'll remove these aliases and make libc_hidden_def instead to get
__GI___* for internal use.
Thanks,
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 18:11 ` Zack Weinberg
@ 2016-01-12 2:15 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-01-12 2:15 UTC (permalink / raw)
To: Zack Weinberg; +Cc: GNU C Library
On Mon, Jan 11, 2016 at 01:11:19PM -0500, Zack Weinberg wrote:
> You have it backward: git (and many other tools) think there *should*
> be a newline at the end of every text file. (That is, \n is a line
> *terminator*, not a line *separator.*)
That's possible. I'll fix it up and if the git commit hook rejects
it, I'll revert it.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 11:17 Siddhesh Poyarekar
2016-01-11 16:34 ` Andi Kleen
2016-01-11 16:39 ` Paul E. Murphy
@ 2016-01-11 19:02 ` Joseph Myers
2016-01-12 2:18 ` Siddhesh Poyarekar
2 siblings, 1 reply; 49+ messages in thread
From: Joseph Myers @ 2016-01-11 19:02 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: libc-alpha, roland, carlos, Paul E. Murphy, Andi Kleen
On Mon, 11 Jan 2016, Siddhesh Poyarekar wrote:
> +# * False positive: tunable_register, compat_tunables_init_envvars,
> +# tunables_init. These functions are in the glibc internal namespace.
> @whitelist = qw(stdin stdout stderr re_syntax_options matherr matherrf
> - matherrl);
> + matherrl tunable_register compat_tunables_init_envvars tunables_init);
I don't understand this comment. None of those names are in an internal
namespace. All those functions should be renamed to start with __.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 17:10 ` Siddhesh Poyarekar
@ 2016-01-11 18:11 ` Zack Weinberg
2016-01-12 2:15 ` Siddhesh Poyarekar
0 siblings, 1 reply; 49+ messages in thread
From: Zack Weinberg @ 2016-01-11 18:11 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: GNU C Library
On Mon, Jan 11, 2016 at 12:10 PM, Siddhesh Poyarekar
<sid@reserved-bit.com> wrote:
> On Mon, Jan 11, 2016 at 06:07:45PM +0100, Andreas Schwab wrote:
>> Siddhesh Poyarekar <sid@reserved-bit.com> writes:
>>
>> > No, that is deliberate, to avoid adding a newline to the end of the
>> > file.
>>
>> Why do you want no newline at the end of a source file?
>
> Doesn't git throw an error when there is a newline at the end of a
> file? I distinctly remember git do that.
You have it backward: git (and many other tools) think there *should*
be a newline at the end of every text file. (That is, \n is a line
*terminator*, not a line *separator.*)
zw
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 17:07 ` Andreas Schwab
@ 2016-01-11 17:10 ` Siddhesh Poyarekar
2016-01-11 18:11 ` Zack Weinberg
0 siblings, 1 reply; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-01-11 17:10 UTC (permalink / raw)
To: Andreas Schwab; +Cc: Paul E. Murphy, libc-alpha, roland, carlos, Andi Kleen
On Mon, Jan 11, 2016 at 06:07:45PM +0100, Andreas Schwab wrote:
> Siddhesh Poyarekar <sid@reserved-bit.com> writes:
>
> > No, that is deliberate, to avoid adding a newline to the end of the
> > file.
>
> Why do you want no newline at the end of a source file?
Doesn't git throw an error when there is a newline at the end of a
file? I distinctly remember git do that.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 16:54 ` Siddhesh Poyarekar
@ 2016-01-11 17:07 ` Andreas Schwab
2016-01-11 17:10 ` Siddhesh Poyarekar
0 siblings, 1 reply; 49+ messages in thread
From: Andreas Schwab @ 2016-01-11 17:07 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: Paul E. Murphy, libc-alpha, roland, carlos, Andi Kleen
Siddhesh Poyarekar <sid@reserved-bit.com> writes:
> No, that is deliberate, to avoid adding a newline to the end of the
> file.
Why do you want no newline at the end of a source file?
Andreas.
--
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 16:34 ` Andi Kleen
@ 2016-01-11 16:55 ` Siddhesh Poyarekar
0 siblings, 0 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-01-11 16:55 UTC (permalink / raw)
To: Andi Kleen; +Cc: libc-alpha, roland, carlos, Paul E. Murphy
On Mon, Jan 11, 2016 at 05:34:42PM +0100, Andi Kleen wrote:
> > * manual/install.texi: Add --enable-tunables option.
>
> This needs much more description in the manual, how to actually
> tune.
Yes, for that I will add a new section to the manual once we have
consensus on how to actually tune. That is, it will be part of the
second patch once we have consensus on the approach for it.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 16:39 ` Paul E. Murphy
@ 2016-01-11 16:54 ` Siddhesh Poyarekar
2016-01-11 17:07 ` Andreas Schwab
0 siblings, 1 reply; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-01-11 16:54 UTC (permalink / raw)
To: Paul E. Murphy; +Cc: libc-alpha, roland, carlos, Andi Kleen
On Mon, Jan 11, 2016 at 10:39:32AM -0600, Paul E. Murphy wrote:
> I'm guessing this should be 'print' not 'printf'?
No, that is deliberate, to avoid adding a newline to the end of the
file.
Siddhesh
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 11:17 Siddhesh Poyarekar
2016-01-11 16:34 ` Andi Kleen
@ 2016-01-11 16:39 ` Paul E. Murphy
2016-01-11 16:54 ` Siddhesh Poyarekar
2016-01-11 19:02 ` Joseph Myers
2 siblings, 1 reply; 49+ messages in thread
From: Paul E. Murphy @ 2016-01-11 16:39 UTC (permalink / raw)
To: Siddhesh Poyarekar, libc-alpha; +Cc: roland, carlos, Andi Kleen
On 01/11/2016 05:16 AM, Siddhesh Poyarekar wrote:
> diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
...
> + print "} tunable_id_t;\n"
> + print "#ifdef TUNABLES_INTERNAL"
> + print "static tunable_t tunable_list[] = {"
> + for (t in val) {
> + for (n in val[t]) {
> + for (c in val[t][n]) {
> + printf (" {TUNABLE_NAME_S(%s, %s, %s), NULL, NULL, false},\n",
> + t, n, val[t][n][c], t, n, val[t][n][c]);
> + }
> + }
> + }
> + print "};"
> + printf "#endif"
I'm guessing this should be 'print' not 'printf'?
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH 1/2] Add framework for tunables
2016-01-11 11:17 Siddhesh Poyarekar
@ 2016-01-11 16:34 ` Andi Kleen
2016-01-11 16:55 ` Siddhesh Poyarekar
2016-01-11 16:39 ` Paul E. Murphy
2016-01-11 19:02 ` Joseph Myers
2 siblings, 1 reply; 49+ messages in thread
From: Andi Kleen @ 2016-01-11 16:34 UTC (permalink / raw)
To: Siddhesh Poyarekar; +Cc: libc-alpha, roland, carlos, Paul E. Murphy, Andi Kleen
> * manual/install.texi: Add --enable-tunables option.
This needs much more description in the manual, how to actually
tune.
-Andi
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH 1/2] Add framework for tunables
@ 2016-01-11 11:17 Siddhesh Poyarekar
2016-01-11 16:34 ` Andi Kleen
` (2 more replies)
0 siblings, 3 replies; 49+ messages in thread
From: Siddhesh Poyarekar @ 2016-01-11 11:17 UTC (permalink / raw)
To: libc-alpha; +Cc: roland, carlos, Paul E. Murphy, Andi Kleen
The tunables framework allows us to uniformly manage and expose global
variables inside glibc as switches to users. tunables/README has
instructions for glibc developers to add new tunables.
Tunables support can be enabled by passing the --enable-tunables
configure flag to the configure script. This patch only adds a
framework and does not pose any limitations on how tunable values are
read from the user. It also adds environment variables used in malloc
behaviour tweaking to the tunables framework as a PoC of the
compatibility interface.
* manual/install.texi: Add --enable-tunables option.
* INSTALL: Regenerate.
* Makeconfig (CPPFLAGS): Define TOP_NAMESPACE.
(all-subdirs): Add tunables.
* config.h.in: Add BUILD_TUNABLES.
* config.make.in: Add build-tunables.
* configure.ac: Add --enable-tunables option.
* configure: Regenerate.
* conform/linknamespace.pl (whitelist): Add tunable_register,
compat_tunables_init_envvars and tunables_init.
* csu/init-first.c: Include tunables.h.
(_init): Initialize tunables.
* nptl/nptl-init.c [BUILD_TUNABLES]: Include tunables.h.
(__pthread_initialize_minimal_internal): Accept argc, argv and
envp. Initialize tunables.
* csu/libc-start.c (LIBC_START_MAIN): Adjust.
* malloc/arena.c [BUILD_TUNABLES]: Include tunables.h. Define
TUNABLE_NAMESPACE.
(tunable_set_mallopt_top_pad, tunable_set_mallopt_perturb,
tunable_set_mallopt_mmap_max, tunable_set_mallopt_arena_max,
tunable_set_mallopt_arena_test,
tunable_set_mallopt_trim_threshold,
tunable_set_mallopt_mmap_threshold,
tunable_set_mallopt_check): New functions.
(ptmalloc_init): Use them and register tunables.
* scripts/gen-tunables.awk: New file.
* tunables/Makefile: New file.
* tunables/README: New file.
* tunables/Versions: New file.
* tunables/tunable-list.h: New auto-generated file.
* tunables/tunables.c: New file.
* tunables/tunables.h: New file.
* tunables/tunables.list: New file.
---
INSTALL | 6 ++
Makeconfig | 9 +++
config.h.in | 3 +
config.make.in | 1 +
configure | 16 ++++++
configure.ac | 10 ++++
conform/linknamespace.pl | 4 +-
csu/init-first.c | 7 +++
csu/libc-start.c | 4 +-
malloc/arena.c | 86 +++++++++++++++++++++++++++
manual/install.texi | 5 ++
nptl/nptl-init.c | 9 ++-
scripts/gen-tunables.awk | 84 +++++++++++++++++++++++++++
tunables/Makefile | 33 +++++++++++
tunables/README | 93 ++++++++++++++++++++++++++++++
tunables/Versions | 8 +++
tunables/tunable-list.h | 32 +++++++++++
tunables/tunables.c | 147 +++++++++++++++++++++++++++++++++++++++++++++++
tunables/tunables.h | 111 +++++++++++++++++++++++++++++++++++
tunables/tunables.list | 12 ++++
20 files changed, 676 insertions(+), 4 deletions(-)
create mode 100644 scripts/gen-tunables.awk
create mode 100644 tunables/Makefile
create mode 100644 tunables/README
create mode 100644 tunables/Versions
create mode 100644 tunables/tunable-list.h
create mode 100644 tunables/tunables.c
create mode 100644 tunables/tunables.h
create mode 100644 tunables/tunables.list
diff --git a/INSTALL b/INSTALL
index c70ea9f..dd06e2e 100644
--- a/INSTALL
+++ b/INSTALL
@@ -164,6 +164,12 @@ will be used, and CFLAGS sets optimization options for the compiler.
By default for x86_64, the GNU C Library is built with vector math
library. Use this option to disable vector math library.
+'--enable-tunables'
+ Tunables support allows additional library parameters to be
+ customized at runtime for each application. This is an
+ experimental feature and affects startup time and is thus disabled
+ by default.
+
'--build=BUILD-SYSTEM'
'--host=HOST-SYSTEM'
These options are for cross-compiling. If you specify both options
diff --git a/Makeconfig b/Makeconfig
index 87a22e8..8ac199b 100644
--- a/Makeconfig
+++ b/Makeconfig
@@ -883,6 +883,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \
$(foreach lib,$(libof-$(basename $(@F))) \
$(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \
$(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F)))
+
+ifeq (yes,$(build-tunables))
+CPPFLAGS += -DTOP_NAMESPACE=glibc
+endif
+
override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \
$(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \
$(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \
@@ -1098,6 +1103,10 @@ all-subdirs = csu assert ctype locale intl catgets math setjmp signal \
crypt localedata timezone rt conform debug mathvec \
$(add-on-subdirs) dlfcn elf
+ifeq (yes,$(build-tunables))
+all-subdirs += tunables
+endif
+
ifndef avoid-generated
# sysd-sorted itself will contain rules making the sysd-sorted target
# depend on Depend files. But if you just added a Depend file to an
diff --git a/config.h.in b/config.h.in
index ec9c8bc..9b07580 100644
--- a/config.h.in
+++ b/config.h.in
@@ -231,4 +231,7 @@
/* PowerPC32 uses fctidz for floating point to long long conversions. */
#define HAVE_PPC_FCTIDZ 0
+/* Build glibc with tunables support. */
+#define BUILD_TUNABLES 0
+
#endif
diff --git a/config.make.in b/config.make.in
index 05ed6ec..b158f51 100644
--- a/config.make.in
+++ b/config.make.in
@@ -90,6 +90,7 @@ use-nscd = @use_nscd@
build-hardcoded-path-in-tests= @hardcoded_path_in_tests@
build-pt-chown = @build_pt_chown@
enable-lock-elision = @enable_lock_elision@
+build-tunables = @build_tunables@
# Build tools.
CC = @CC@
diff --git a/configure b/configure
index aa05d49..6fef02c 100755
--- a/configure
+++ b/configure
@@ -659,6 +659,7 @@ multi_arch
base_machine
add_on_subdirs
add_ons
+build_tunables
build_pt_chown
build_nscd
link_obsolete_rpc
@@ -773,6 +774,7 @@ enable_systemtap
enable_build_nscd
enable_nscd
enable_pt_chown
+enable_tunables
enable_mathvec
with_cpu
'
@@ -1440,6 +1442,7 @@ Optional Features:
--disable-build-nscd disable building and installing the nscd daemon
--disable-nscd library functions will not contact the nscd daemon
--enable-pt_chown Enable building and installing pt_chown
+ --enable-tunables Enable tunables support
--enable-mathvec Enable building and installing mathvec [default
depends on architecture]
@@ -3642,6 +3645,19 @@ if test "$build_pt_chown" = yes; then
fi
+# Check whether --enable-tunables was given.
+if test "${enable_tunables+set}" = set; then :
+ enableval=$enable_tunables; build_tunables=$enableval
+else
+ build_tunables=no
+fi
+
+
+if test "$build_tunables" = yes; then
+ $as_echo "#define BUILD_TUNABLES 1" >>confdefs.h
+
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/configure.ac b/configure.ac
index ee7a3f1..41507ac 100644
--- a/configure.ac
+++ b/configure.ac
@@ -392,6 +392,16 @@ if test "$build_pt_chown" = yes; then
AC_DEFINE(HAVE_PT_CHOWN)
fi
+AC_ARG_ENABLE([tunables],
+ [AS_HELP_STRING([--enable-tunables],
+ [Enable tunables support])],
+ [build_tunables=$enableval],
+ [build_tunables=no])
+AC_SUBST(build_tunables)
+if test "$build_tunables" = yes; then
+ AC_DEFINE(BUILD_TUNABLES)
+fi
+
# The abi-tags file uses a fairly simplistic model for name recognition that
# can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a
# $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell.
diff --git a/conform/linknamespace.pl b/conform/linknamespace.pl
index d6a1850..4e8e287 100644
--- a/conform/linknamespace.pl
+++ b/conform/linknamespace.pl
@@ -50,8 +50,10 @@ close (STDSYMS) || die ("close $stdsyms_file: $!\n");
# * False positive: matherr only used conditionally. matherrf/matherrl are used
# by IA64 too for the same reason.
#
+# * False positive: tunable_register, compat_tunables_init_envvars,
+# tunables_init. These functions are in the glibc internal namespace.
@whitelist = qw(stdin stdout stderr re_syntax_options matherr matherrf
- matherrl);
+ matherrl tunable_register compat_tunables_init_envvars tunables_init);
foreach my $sym (@whitelist) {
$stdsyms{$sym} = 1;
}
diff --git a/csu/init-first.c b/csu/init-first.c
index 77c6e1c..f86a029 100644
--- a/csu/init-first.c
+++ b/csu/init-first.c
@@ -26,6 +26,9 @@
#include <sys/param.h>
#include <sys/types.h>
#include <libc-internal.h>
+#if BUILD_TUNABLES
+# include <tunables/tunables.h>
+#endif
#include <ldsodefs.h>
@@ -96,6 +99,10 @@ _init (int argc, char **argv, char **envp)
#if defined SHARED && !defined NO_CTORS_DTORS_SECTIONS
__libc_global_ctors ();
#endif
+
+#if BUILD_TUNABLES
+ tunables_init (__environ);
+#endif
}
/* This function is defined here so that if this file ever gets into
diff --git a/csu/libc-start.c b/csu/libc-start.c
index f4aa01a..7629fec 100644
--- a/csu/libc-start.c
+++ b/csu/libc-start.c
@@ -32,7 +32,7 @@ extern int __libc_multiple_libcs;
#include <tls.h>
#ifndef SHARED
# include <dl-osinfo.h>
-extern void __pthread_initialize_minimal (void);
+extern void __pthread_initialize_minimal (int, char **, char **);
# ifndef THREAD_SET_STACK_GUARD
/* Only exported for architectures that don't store the stack guard canary
in thread local area. */
@@ -193,7 +193,7 @@ LIBC_START_MAIN (int (*main) (int, char **, char ** MAIN_AUXVEC_DECL),
/* Initialize the thread library at least a bit since the libgcc
functions are using thread functions if these are available and
we need to setup errno. */
- __pthread_initialize_minimal ();
+ __pthread_initialize_minimal (argc, argv, __environ);
/* Set up the stack checker's canary. */
uintptr_t stack_chk_guard = _dl_setup_stack_chk_guard (_dl_random);
diff --git a/malloc/arena.c b/malloc/arena.c
index 1edb4d4..f773b10 100644
--- a/malloc/arena.c
+++ b/malloc/arena.c
@@ -19,6 +19,11 @@
#include <stdbool.h>
+#if BUILD_TUNABLES
+# define TUNABLE_NAMESPACE malloc
+# include <tunables/tunables.h>
+#endif
+
/* Compile-time constants. */
#define HEAP_MIN_SIZE (32 * 1024)
@@ -332,6 +337,61 @@ ptmalloc_unlock_all2 (void)
# endif
#endif /* !NO_THREADS */
+#if BUILD_TUNABLES
+static void
+tunable_set_mallopt_top_pad (const char *val)
+{
+ __libc_mallopt (M_TOP_PAD, atoi (val));
+}
+
+static void
+tunable_set_mallopt_perturb (const char *val)
+{
+ __libc_mallopt (M_PERTURB, atoi (val));
+}
+
+static void
+tunable_set_mallopt_mmap_max (const char *val)
+{
+ __libc_mallopt (M_MMAP_MAX, atoi (val));
+}
+
+static void
+tunable_set_mallopt_arena_max (const char *val)
+{
+ __libc_mallopt (M_ARENA_MAX, atoi (val));
+}
+
+static void
+tunable_set_mallopt_arena_test (const char *val)
+{
+ __libc_mallopt (M_ARENA_TEST, atoi (val));
+}
+
+static void
+tunable_set_mallopt_trim_threshold (const char *val)
+{
+ __libc_mallopt (M_TRIM_THRESHOLD, atoi (val));
+}
+
+static void
+tunable_set_mallopt_mmap_threshold (const char *val)
+{
+ __libc_mallopt (M_MMAP_THRESHOLD, atoi (val));
+}
+
+static void
+tunable_set_mallopt_check (const char *val)
+{
+ if (val[0])
+ {
+ __libc_mallopt (M_CHECK_ACTION, (int) (val[0] - '0'));
+ if (check_action != 0)
+ __malloc_check_init ();
+ }
+}
+
+#else
/* Initialization routine. */
#include <string.h>
extern char **_environ;
@@ -366,6 +426,7 @@ next_env_entry (char ***position)
return result;
}
+#endif
#ifdef SHARED
@@ -401,6 +462,29 @@ ptmalloc_init (void)
thread_arena = &main_arena;
thread_atfork (ptmalloc_lock_all, ptmalloc_unlock_all, ptmalloc_unlock_all2);
+
+#if BUILD_TUNABLES
+ COMPAT_TUNABLES_NAMESPACE_BEGIN (8);
+
+ COMPAT_TUNABLE_REGISTER_SECURE (check, "MALLOC_CHECK_",
+ tunable_set_mallopt_check);
+ COMPAT_TUNABLE_REGISTER (top_pad, "MALLOC_TOP_PAD_",
+ tunable_set_mallopt_top_pad);
+ COMPAT_TUNABLE_REGISTER (perturb, "MALLOC_PERTURB_",
+ tunable_set_mallopt_perturb);
+ COMPAT_TUNABLE_REGISTER (mmap_threshold, "MALLOC_MMAP_THRESHOLD_",
+ tunable_set_mallopt_mmap_threshold);
+ COMPAT_TUNABLE_REGISTER (trim_threshold, "MALLOC_TRIM_THRESHOLD_",
+ tunable_set_mallopt_trim_threshold);
+ COMPAT_TUNABLE_REGISTER (mmap_max, "MALLOC_MMAP_MAX_",
+ tunable_set_mallopt_mmap_max);
+ COMPAT_TUNABLE_REGISTER (arena_max, "MALLOC_ARENA_MAX",
+ tunable_set_mallopt_arena_max);
+ COMPAT_TUNABLE_REGISTER (arena_test, "MALLOC_ARENA_TEST",
+ tunable_set_mallopt_arena_test);
+
+ COMPAT_TUNABLES_NAMESPACE_INIT ();
+#else
const char *s = NULL;
if (__glibc_likely (_environ != NULL))
{
@@ -469,6 +553,8 @@ ptmalloc_init (void)
if (check_action != 0)
__malloc_check_init ();
}
+#endif
+
void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook);
if (hook != NULL)
(*hook)();
diff --git a/manual/install.texi b/manual/install.texi
index de9d270..95392ea 100644
--- a/manual/install.texi
+++ b/manual/install.texi
@@ -195,6 +195,11 @@ configure with @option{--disable-werror}.
By default for x86_64, @theglibc{} is built with vector math library.
Use this option to disable vector math library.
+@item --enable-tunables
+Tunables support allows additional library parameters to be customized at
+runtime for each application. This is an experimental feature and affects
+startup time and is thus disabled by default.
+
@item --build=@var{build-system}
@itemx --host=@var{host-system}
These options are for cross-compiling. If you specify both options and
diff --git a/nptl/nptl-init.c b/nptl/nptl-init.c
index bdbdfed..be12d98 100644
--- a/nptl/nptl-init.c
+++ b/nptl/nptl-init.c
@@ -38,6 +38,9 @@
#include <kernel-features.h>
#include <libc-internal.h>
#include <pthread-pids.h>
+#if BUILD_TUNABLES
+# include <tunables/tunables.h>
+#endif
#ifndef TLS_MULTIPLE_THREADS_IN_TCB
/* Pointer to the corresponding variable in libc. */
@@ -297,7 +300,7 @@ extern void **__libc_dl_error_tsd (void) __attribute__ ((const));
static bool __nptl_initial_report_events __attribute_used__;
void
-__pthread_initialize_minimal_internal (void)
+__pthread_initialize_minimal_internal (int argc, char **argv, char **envp)
{
#ifndef SHARED
/* Unlike in the dynamically linked case the dynamic linker has not
@@ -311,6 +314,10 @@ __pthread_initialize_minimal_internal (void)
__asm __volatile ("");
#endif
+#if BUILD_TUNABLES
+ tunables_init (envp);
+#endif
+
/* Minimal initialization of the thread descriptor. */
struct pthread *pd = THREAD_SELF;
__pthread_initialize_pids (pd);
diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk
new file mode 100644
index 0000000..5a65fc9
--- /dev/null
+++ b/scripts/gen-tunables.awk
@@ -0,0 +1,84 @@
+# Generate tunable-list.h from tunables.list
+
+BEGIN {
+ ns=""
+ top_ns=""
+}
+
+$2 == "{" {
+ if (top_ns == "") {
+ top_ns = $1
+ }
+ else if (ns == "") {
+ ns = $1
+ count = 0
+ }
+ else {
+ printf ("Unexpected occurrence of '{' inside a namespace: %s:%d\n",
+ FILENAME, FNR)
+ exit 1
+ }
+
+ next
+}
+
+$1 == "}" {
+ if (ns != "") {
+ ns = ""
+ }
+ else if (top_ns != "") {
+ top_ns = ""
+ }
+ else {
+ printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR)
+ exit 1
+ }
+ next
+}
+
+{
+ if (ns == "") {
+ print "Invalid tunable outside a namespace"
+ exit 1
+ }
+ val[top_ns][ns][count] = $1
+ count = count + 1
+}
+
+END {
+ if (ns != "") {
+ print "Unterminated namespace. Is a closing brace missing?"
+ exit 1
+ }
+
+ print "/* Print a full tunable enum name. */"
+ print "#include <stddef.h>"
+ print "#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)"
+ print "#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id\n"
+
+ print "/* Full name for a tunable is top_ns.tunable_ns.id. */"
+ print "#define TUNABLE_NAME_S(top,ns,id) #top \".\" #ns \".\" #id\n"
+
+ print "typedef enum"
+ print "{"
+ for (t in val) {
+ for (n in val[t]) {
+ for (c in val[t][n]) {
+ printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, val[t][n][c]);
+ }
+ }
+ }
+ print "} tunable_id_t;\n"
+ print "#ifdef TUNABLES_INTERNAL"
+ print "static tunable_t tunable_list[] = {"
+ for (t in val) {
+ for (n in val[t]) {
+ for (c in val[t][n]) {
+ printf (" {TUNABLE_NAME_S(%s, %s, %s), NULL, NULL, false},\n",
+ t, n, val[t][n][c], t, n, val[t][n][c]);
+ }
+ }
+ }
+ print "};"
+ printf "#endif"
+}
diff --git a/tunables/Makefile b/tunables/Makefile
new file mode 100644
index 0000000..44439f6
--- /dev/null
+++ b/tunables/Makefile
@@ -0,0 +1,33 @@
+# Copyright (C) 2016 Free Software Foundation, Inc.
+# This file is part of the GNU C Library.
+
+# The GNU C Library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+
+# The GNU C Library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+
+# You should have received a copy of the GNU Lesser General Public
+# License along with the GNU C Library; if not, see
+# <http://www.gnu.org/licenses/>.
+
+#
+# Makefile for tunables.
+#
+subdir := tunables
+
+include ../Makeconfig
+
+routines = tunables
+
+$(objpfx)tunables.os: tunable-list.h
+
+tunable-list.h: $(..)scripts/gen-tunables.awk tunables.list
+ $(AWK) -f $^ > $@.tmp
+ mv $@{.tmp,}
+
+include ../Rules
diff --git a/tunables/README b/tunables/README
new file mode 100644
index 0000000..02a3f7a
--- /dev/null
+++ b/tunables/README
@@ -0,0 +1,93 @@
+ TUNABLE FRAMEWORK
+ =================
+
+The tunable framework allows modules within glibc to register variables that
+may be tweaked through an environment variable or an API call. It aims to
+enforce a strict namespace rule to bring consistency to naming of these tunable
+environment variables across the project.
+
+ADDING A NEW TUNABLE
+--------------------
+
+The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in
+distributions for specific tunables if they want to add their own tunables.
+Downstream implementations are discouraged from using the 'glibc' namespace for
+tunables they don't already have consensus to push upstream.
+
+There are two steps to adding a tunable:
+
+1. Add a tunable ID:
+
+Modules that wish to use the tunables interface must define the
+TUNABLE_NAMESPACE macro. Following this, for each tunable you want to
+register, call the TUNABLE_REGISTER macro.
+
+The TUNABLE_REGISTER macro takes the following arguments:
+
+- id: The short name of the tunable. It will be concatenated with
+ the TOP_NAMESPACE and TUNABLE_NAMESPACE to build the full ID of
+ the tunable.
+
+- setter: A function that accepts a string input and initializes the
+ tunable variable. The prototype of the function should be as
+ follows:
+
+ void setter_func (const char *)
+
+2. Add to tunables.list:
+
+This file nests tunable ids inside their tunable namespace within curly braces
+and multiple such namespaces are in turn nested inside their top namespaces in
+a similar manner. The malloc and resolv snippets for example could look like
+this:
+
+glibc {
+ malloc {
+ check
+ top_pad
+ }
+ resolv {
+ secure_dns
+ }
+}
+
+Add your tunable in this hierarchy using the proper nesting for your namespace
+and top namespace. If the top namespace is not glibc, create a new snippet
+with its own hierarchy, separate from the glibc hierarchy.
+
+ADDING A LEGACY TUNABLE
+-----------------------
+
+One may add tunables for variables that are currently controlled by users with
+environment variables. Multiple such variables should be defined together for
+better performance by using the COMPAT_TUNABLES_NAMESPACE_BEGIN macro, which
+accepts a numeric parameter with the number of tunables you intend to register
+in the block.
+
+Individual tunables are then registered using the COMPAT_TUNABLE_REGISTER and
+COMPAT_TUNABLE_REGISTER_SECURE macros. They take the following parameters:
+
+- id: The tunable ID, similar to the normal way of adding a tunable
+
+- env: The name of the legacy environment variable so that one may
+ read its value and use it.
+
+- setter: The setter function, similar to the normal way of adding a
+ tunable.
+
+COMPAT_TUNABLE_REGISTER reads the environment variable only for non-setuid
+binaries, whereas COMPAT_TUNABLE_REGISTER_SECURE always attempts to read the
+environment variable. Use COMPAT_TUNABLE_REGISTER_SECURE with care.
+
+You will also have to add the tunable id to tunables/tunables.list.
+
+DO NOT ADD NEW TUNABLES USING THIS MECHANISM!
+
+FUTURE WORK
+-----------
+
+The framework currently only allows a one-time initialization of variables
+through environment variables and in some cases, modification of variables via
+an API call. A future goal for this project is to allow tweaking of some
+values in a running process, possibly through some kind of shared memory
+mechanism.
diff --git a/tunables/Versions b/tunables/Versions
new file mode 100644
index 0000000..00a75c1
--- /dev/null
+++ b/tunables/Versions
@@ -0,0 +1,8 @@
+# Exports from tunables should only be in the GLIBC_PRIVATE namespace.
+libc {
+ GLIBC_PRIVATE {
+ compat_tunables_init_envvars;
+ tunables_init;
+ tunable_register;
+ }
+}
diff --git a/tunables/tunable-list.h b/tunables/tunable-list.h
new file mode 100644
index 0000000..2b1caa4
--- /dev/null
+++ b/tunables/tunable-list.h
@@ -0,0 +1,32 @@
+/* Print a full tunable enum name. */
+#include <stddef.h>
+#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)
+#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id
+
+/* Full name for a tunable is top_ns.tunable_ns.id. */
+#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id
+
+typedef enum
+{
+ TUNABLE_ENUM_NAME(glibc, malloc, check),
+ TUNABLE_ENUM_NAME(glibc, malloc, top_pad),
+ TUNABLE_ENUM_NAME(glibc, malloc, perturb),
+ TUNABLE_ENUM_NAME(glibc, malloc, mmap_threshold),
+ TUNABLE_ENUM_NAME(glibc, malloc, trim_threshold),
+ TUNABLE_ENUM_NAME(glibc, malloc, mmap_max),
+ TUNABLE_ENUM_NAME(glibc, malloc, arena_max),
+ TUNABLE_ENUM_NAME(glibc, malloc, arena_test),
+} tunable_id_t;
+
+#ifdef TUNABLES_INTERNAL
+static tunable_t tunable_list[] = {
+ {TUNABLE_NAME_S(glibc, malloc, check), NULL, NULL, false},
+ {TUNABLE_NAME_S(glibc, malloc, top_pad), NULL, NULL, false},
+ {TUNABLE_NAME_S(glibc, malloc, perturb), NULL, NULL, false},
+ {TUNABLE_NAME_S(glibc, malloc, mmap_threshold), NULL, NULL, false},
+ {TUNABLE_NAME_S(glibc, malloc, trim_threshold), NULL, NULL, false},
+ {TUNABLE_NAME_S(glibc, malloc, mmap_max), NULL, NULL, false},
+ {TUNABLE_NAME_S(glibc, malloc, arena_max), NULL, NULL, false},
+ {TUNABLE_NAME_S(glibc, malloc, arena_test), NULL, NULL, false},
+};
+#endif
diff --git a/tunables/tunables.c b/tunables/tunables.c
new file mode 100644
index 0000000..2ae1050
--- /dev/null
+++ b/tunables/tunables.c
@@ -0,0 +1,147 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <stddef.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <sys/param.h>
+
+extern char **__environ;
+
+#define TUNABLES_INTERNAL 1
+#include "tunables.h"
+
+static int
+t_strncmp (const char *a, const char *b, size_t len)
+{
+ size_t i = 0;
+
+ for (i = 0; i < len && *a != '\0' && *b != '\0'; i++, a++, b++)
+ if (*a != *b)
+ return *a - *b;
+
+ /* If we weren't limited by LEN, then one of the strings terminated. */
+ if (i < len)
+ return *a - *b;
+
+ return 0;
+}
+
+static bool
+get_next_env (char ***envp, char **name, size_t *namelen, char **val)
+{
+ char **ev = *envp;
+
+ while (ev != NULL && *ev != '\0')
+ {
+ char *envline = *ev;
+ int len = 0;
+
+ while (envline[len] != '\0' && envline[len] != '=')
+ len++;
+
+ /* Just the name and no value, go to the next one. */
+ if (envline[len] == '\0')
+ continue;
+
+ *name = envline;
+ *namelen = len;
+ *val = &envline[len + 1];
+ *envp = ++ev;
+
+ return true;
+ }
+
+ return false;
+}
+
+/* This is where tunables will be read in from either an environment variable,
+ a set of environment variables or some other source and then initialized.
+ Caller should pass it the environment variable; __environ may not be
+ reliable if it is called earlier than libc.so initialization. */
+void
+__tunables_init (char **envp)
+{
+ /* Empty for now. */
+}
+strong_alias (__tunables_init, tunables_init)
+
+/* Initialize all tunables using its legacy environment variable values whose
+ names are passed in ENVVARS. */
+void
+__compat_tunables_init_envvars (struct compat_tunable_env *envvars, int count)
+{
+ /* Traverse through the environment to find environment variables we may need
+ to set. */
+ char **envp = __environ;
+ char *envname = NULL;
+ char *envval = NULL;
+ size_t len = 0;
+
+ while (get_next_env (&envp, &envname, &len, &envval))
+ {
+ int init_count = 0;
+ for (int i = 0; i < count; i++)
+ {
+ tunable_id_t t = envvars[i].id;
+ tunable_t *cur = &tunable_list[t];
+
+ /* Skip over tunables that have already been initialized. */
+ if (cur->initialized)
+ {
+ init_count++;
+ continue;
+ }
+
+ const char *name = envvars[i].env;
+
+ /* We have a match. Initialize and move on to the next line. */
+ if (t_strncmp (envname, name, len) == 0)
+ {
+ cur->val = envval;
+ cur->set (cur->val);
+ cur->initialized = true;
+ break;
+ }
+ }
+
+ /* All of the tunable envvars have been initialized. */
+ if (count == init_count)
+ break;
+ }
+}
+strong_alias (__compat_tunables_init_envvars, compat_tunables_init_envvars)
+
+/* Initialize a tunable and set its value. */
+void
+__tunable_register (tunable_id_t id, tunable_setter_t set_func)
+{
+ tunable_t *cur = &tunable_list[id];
+
+ cur->set = set_func;
+ if (cur->val != NULL)
+ {
+ set_func (cur->val);
+ cur->initialized = true;
+ }
+}
+strong_alias (__tunable_register, tunable_register)
diff --git a/tunables/tunables.h b/tunables/tunables.h
new file mode 100644
index 0000000..095eda2
--- /dev/null
+++ b/tunables/tunables.h
@@ -0,0 +1,111 @@
+/* The tunable framework. See the README to know how to use the tunable in
+ a glibc module.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <stdbool.h>
+
+typedef void (*tunable_setter_t) (const char *);
+
+/* A tunable. */
+struct _tunable
+{
+ const char *name;
+ const char *val;
+ tunable_setter_t set;
+ bool initialized;
+};
+
+typedef struct _tunable tunable_t;
+
+#include "tunable-list.h"
+
+struct compat_tunable_env
+{
+ tunable_id_t id;
+ const char *env;
+};
+
+/* Avoid a PLT reference when called from within libc.so. */
+#if IS_IN (libc) && !defined TUNABLES_INTERNAL
+# define compat_tunables_init_envvars __compat_tunables_init_envvars
+# define tunable_register __tunable_register
+# define tunables_init __tunables_init
+#endif
+
+extern void compat_tunables_init_envvars (struct compat_tunable_env *, int);
+extern void tunable_register (tunable_id_t, tunable_setter_t);
+extern void tunables_init (char **);
+
+/* Register a tunable ID in namespace NS inside the glibc top namespace with
+ SET as the setter function. */
+#define TUNABLE_REGISTER(ns,id,set) \
+ TUNABLE_REGISTER_FULL (TOP_NAMESPACE, ns, id, set)
+
+/* Register a tunable ID in namespace NS inside the TNS top namespace with
+ SET as the setter function. */
+#define TUNABLE_REGISTER_FULL(tns,ns,id,set) \
+ tunable_register (TUNABLE_ENUM_NAME (tns, ns, id), (set))
+
+\f
+
+/* COMPATIBILITY INTERFACE. These macros should be used to define tunables for
+ variables that are currently being controlled using environment variables.
+ New tunables must not use this interface. */
+
+#define _ADD_COMPAT_TUNABLE_ENV(__id,__env) \
+({ \
+ envvars[envvars_cnt].id = TUNABLE_ENUM_NAME (TOP_NAMESPACE, \
+ TUNABLE_NAMESPACE, __id); \
+ envvars[envvars_cnt++].env = (__env); \
+})
+
+/* Start registering tunables in the current namespace. */
+#define COMPAT_TUNABLES_NAMESPACE_BEGIN(size) \
+ { \
+ struct compat_tunable_env envvars[size]; \
+ int envvars_cnt = 0;
+
+/* Register a tunable. This macro validates that the call is OK and then calls
+ tunable_init to do the real work of adding the tunable and setting its value
+ based on its environment variable(s). */
+#define COMPAT_TUNABLE_REGISTER(id,env,set) \
+({ \
+ assert (envvars_cnt < (sizeof (envvars) \
+ / sizeof (struct compat_tunable_env))); \
+ if (!__libc_enable_secure) \
+ _ADD_COMPAT_TUNABLE_ENV (id, env); \
+ TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \
+})
+
+/* Does exactly the same thing as TUNABLE_REGISTER, except that it allows the
+ tunable to look for environment variable values even for setuid binaries.
+ This is a separate macro and not just another parameter in TUNABLE_REGISTER
+ to avoid accidentally setting a secure flag where it is not required. */
+#define COMPAT_TUNABLE_REGISTER_SECURE(id,env,set) \
+({ \
+ assert (envvars_cnt < (sizeof (envvars) \
+ / sizeof (struct compat_tunable_env))); \
+ _ADD_COMPAT_TUNABLE_ENV(id, env); \
+ TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \
+})
+
+/* Initialize tunables in the namespace. */
+#define COMPAT_TUNABLES_NAMESPACE_INIT() \
+ compat_tunables_init_envvars (envvars, envvars_cnt); \
+ }
diff --git a/tunables/tunables.list b/tunables/tunables.list
new file mode 100644
index 0000000..f1335cc
--- /dev/null
+++ b/tunables/tunables.list
@@ -0,0 +1,12 @@
+glibc {
+ malloc {
+ check
+ top_pad
+ perturb
+ mmap_threshold
+ trim_threshold
+ mmap_max
+ arena_max
+ arena_test
+ }
+}
--
2.5.0
^ permalink raw reply [flat|nested] 49+ messages in thread
end of thread, other threads:[~2016-08-15 20:05 UTC | newest]
Thread overview: 49+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-07-02 17:13 [PATCH 0/2] tunables for glibc Siddhesh Poyarekar
2016-07-02 17:13 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-07-03 14:30 ` H.J. Lu
2016-07-03 17:08 ` Siddhesh Poyarekar
2016-07-03 14:44 ` H.J. Lu
2016-07-03 17:14 ` Siddhesh Poyarekar
2016-07-03 17:41 ` H.J. Lu
2016-07-03 15:13 ` H.J. Lu
2016-07-03 17:43 ` H.J. Lu
2016-07-03 18:20 ` Siddhesh Poyarekar
2016-07-02 17:13 ` [PATCH 2/2] Initialize tunable list with the GLIBC_TUNABLES environment variable Siddhesh Poyarekar
2016-07-03 14:39 ` H.J. Lu
2016-07-03 17:15 ` Siddhesh Poyarekar
2016-07-03 15:53 ` H.J. Lu
2016-07-03 17:18 ` Siddhesh Poyarekar
2016-07-03 0:24 ` [PATCH 0/2] tunables for glibc H.J. Lu
2016-07-03 3:08 ` Siddhesh Poyarekar
-- strict thread matches above, loose matches on Subject: below --
2016-08-15 20:05 [PATCHv4 " Siddhesh Poyarekar
2016-08-15 20:05 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-07-09 18:49 [PATCHv3 0/2] tunables for glibc Siddhesh Poyarekar
2016-07-09 18:49 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-07-11 16:05 ` Florian Weimer
2016-07-11 16:39 ` Siddhesh Poyarekar
2016-07-11 19:13 ` DJ Delorie
2016-07-12 9:00 ` Florian Weimer
2016-07-12 8:51 ` Florian Weimer
2016-07-11 16:25 ` H.J. Lu
2016-07-11 16:41 ` Siddhesh Poyarekar
2016-07-13 14:43 ` H.J. Lu
2016-07-13 14:49 ` Siddhesh Poyarekar
2016-07-13 17:38 ` H.J. Lu
2016-07-13 17:56 ` Siddhesh Poyarekar
2016-07-13 17:59 ` H.J. Lu
2016-07-13 18:03 ` H.J. Lu
2016-07-14 6:57 ` Siddhesh Poyarekar
2016-07-11 16:56 ` Paul E. Murphy
2016-07-11 17:28 ` H.J. Lu
2016-07-11 19:06 ` Carlos O'Donell
2016-07-14 11:27 ` Siddhesh Poyarekar
2016-07-01 18:31 [PATCH 0/2] Tunables for glibc Siddhesh Poyarekar
2016-07-01 18:32 ` [PATCH 1/2] Add framework for tunables Siddhesh Poyarekar
2016-01-11 11:17 Siddhesh Poyarekar
2016-01-11 16:34 ` Andi Kleen
2016-01-11 16:55 ` Siddhesh Poyarekar
2016-01-11 16:39 ` Paul E. Murphy
2016-01-11 16:54 ` Siddhesh Poyarekar
2016-01-11 17:07 ` Andreas Schwab
2016-01-11 17:10 ` Siddhesh Poyarekar
2016-01-11 18:11 ` Zack Weinberg
2016-01-12 2:15 ` Siddhesh Poyarekar
2016-01-11 19:02 ` Joseph Myers
2016-01-12 2:18 ` Siddhesh Poyarekar
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).