* [PATCH 01/11] support: Add support_copy_file
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
@ 2020-11-09 18:40 ` Florian Weimer
2020-11-26 16:43 ` Adhemerval Zanella
2020-11-09 18:40 ` [PATCH 02/11] elf: Introduce enum opt_format in the ldconfig implementation Florian Weimer
` (11 subsequent siblings)
12 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:40 UTC (permalink / raw)
To: libc-alpha
---
support/Makefile | 1 +
support/support.h | 5 +++++
support/support_copy_file.c | 43 +++++++++++++++++++++++++++++++++++++
3 files changed, 49 insertions(+)
create mode 100644 support/support_copy_file.c
diff --git a/support/Makefile b/support/Makefile
index 4154863511..f5f59bf8d2 100644
--- a/support/Makefile
+++ b/support/Makefile
@@ -46,6 +46,7 @@ libsupport-routines = \
support_capture_subprocess \
support_capture_subprocess_check \
support_chroot \
+ support_copy_file \
support_copy_file_range \
support_descriptor_supports_holes \
support_descriptors \
diff --git a/support/support.h b/support/support.h
index 905b5a76d8..abda3a69f1 100644
--- a/support/support.h
+++ b/support/support.h
@@ -119,6 +119,11 @@ extern const char support_install_rootsbindir[];
/* Corresponds to the install's compiled locale directory. */
extern const char support_complocaledir_prefix[];
+/* Copies the file at the path FROM to TO. If TO does not exist, it
+ is created. If TO is a regular file, it is truncated before
+ copying. The file mode is copied, but the permissions are not. */
+extern void support_copy_file (const char *from, const char *to);
+
extern ssize_t support_copy_file_range (int, off64_t *, int, off64_t *,
size_t, unsigned int);
diff --git a/support/support_copy_file.c b/support/support_copy_file.c
new file mode 100644
index 0000000000..bf8f83803a
--- /dev/null
+++ b/support/support_copy_file.c
@@ -0,0 +1,43 @@
+/* Copy a file from one path to another.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <fcntl.h>
+#include <support/check.h>
+#include <support/support.h>
+#include <support/xunistd.h>
+
+void
+support_copy_file (const char *from, const char *to)
+{
+ struct stat64 st;
+ xstat (from, &st);
+ int fd_from = xopen (from, O_RDONLY, 0);
+ mode_t mode = st.st_mode & 0777;
+ int fd_to = xopen (to, O_WRONLY | O_TRUNC | O_CREAT, mode);
+ ssize_t ret = support_copy_file_range (fd_from, NULL, fd_to, NULL,
+ st.st_size, 0);
+ if (ret < 0)
+ FAIL_EXIT1 ("copying from \"%s\" to \"%s\": %m", from, to);
+ if (ret != st.st_size)
+ FAIL_EXIT1 ("copying from \"%s\" to \"%s\": only %zd of %llu bytes copied",
+ from, to, ret, (unsigned long long int) st.st_size);
+ if (fchmod (fd_to, mode) < 0)
+ FAIL_EXIT1 ("fchmod on %s: %m", to);
+ xclose (fd_to);
+ xclose (fd_from);
+}
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 01/11] support: Add support_copy_file
2020-11-09 18:40 ` [PATCH 01/11] support: Add support_copy_file Florian Weimer
@ 2020-11-26 16:43 ` Adhemerval Zanella
2020-11-26 17:22 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-26 16:43 UTC (permalink / raw)
To: libc-alpha
LGTM with a couple nits below.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
On 09/11/2020 15:40, Florian Weimer via Libc-alpha wrote:
> ---
> support/Makefile | 1 +
> support/support.h | 5 +++++
> support/support_copy_file.c | 43 +++++++++++++++++++++++++++++++++++++
> 3 files changed, 49 insertions(+)
> create mode 100644 support/support_copy_file.c
>
> diff --git a/support/Makefile b/support/Makefile
> index 4154863511..f5f59bf8d2 100644
> --- a/support/Makefile
> +++ b/support/Makefile
> @@ -46,6 +46,7 @@ libsupport-routines = \
> support_capture_subprocess \
> support_capture_subprocess_check \
> support_chroot \
> + support_copy_file \
> support_copy_file_range \
> support_descriptor_supports_holes \
> support_descriptors \
> diff --git a/support/support.h b/support/support.h
> index 905b5a76d8..abda3a69f1 100644
> --- a/support/support.h
> +++ b/support/support.h
> @@ -119,6 +119,11 @@ extern const char support_install_rootsbindir[];
> /* Corresponds to the install's compiled locale directory. */
> extern const char support_complocaledir_prefix[];
>
> +/* Copies the file at the path FROM to TO. If TO does not exist, it
> + is created. If TO is a regular file, it is truncated before
> + copying. The file mode is copied, but the permissions are not. */
> +extern void support_copy_file (const char *from, const char *to);
> +
> extern ssize_t support_copy_file_range (int, off64_t *, int, off64_t *,
> size_t, unsigned int);
>
Ok.
> diff --git a/support/support_copy_file.c b/support/support_copy_file.c
> new file mode 100644
> index 0000000000..bf8f83803a
> --- /dev/null
> +++ b/support/support_copy_file.c
> @@ -0,0 +1,43 @@
> +/* Copy a file from one path to another.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <fcntl.h>
> +#include <support/check.h>
> +#include <support/support.h>
> +#include <support/xunistd.h>
> +
> +void
> +support_copy_file (const char *from, const char *to)
> +{
> + struct stat64 st;
> + xstat (from, &st);
> + int fd_from = xopen (from, O_RDONLY, 0);
> + mode_t mode = st.st_mode & 0777;
> + int fd_to = xopen (to, O_WRONLY | O_TRUNC | O_CREAT, mode);
> + ssize_t ret = support_copy_file_range (fd_from, NULL, fd_to, NULL,
> + st.st_size, 0);
> + if (ret < 0)
> + FAIL_EXIT1 ("copying from \"%s\" to \"%s\": %m", from, to);
> + if (ret != st.st_size)
> + FAIL_EXIT1 ("copying from \"%s\" to \"%s\": only %zd of %llu bytes copied",
This line seems too long.
> + from, to, ret, (unsigned long long int) st.st_size);
> + if (fchmod (fd_to, mode) < 0)
> + FAIL_EXIT1 ("fchmod on %s: %m", to);
Maybe put 'to' within quotations as well?
> + xclose (fd_to);
> + xclose (fd_from);
> +}
>
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 01/11] support: Add support_copy_file
2020-11-26 16:43 ` Adhemerval Zanella
@ 2020-11-26 17:22 ` Florian Weimer
0 siblings, 0 replies; 58+ messages in thread
From: Florian Weimer @ 2020-11-26 17:22 UTC (permalink / raw)
To: Adhemerval Zanella via Libc-alpha
* Adhemerval Zanella via Libc-alpha:
>> + if (ret != st.st_size)
>> + FAIL_EXIT1 ("copying from \"%s\" to \"%s\": only %zd of %llu bytes copied",
>
> This line seems too long.
It's 79 characters long, which is the maximum according to:
<https://sourceware.org/glibc/wiki/Style_and_Conventions#A79-Column_Lines>
>> + from, to, ret, (unsigned long long int) st.st_size);
>> + if (fchmod (fd_to, mode) < 0)
>> + FAIL_EXIT1 ("fchmod on %s: %m", to);
>
> Maybe put 'to' within quotations as well?
Thanks, fixed. Will push if it still builds.
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 02/11] elf: Introduce enum opt_format in the ldconfig implementation
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
2020-11-09 18:40 ` [PATCH 01/11] support: Add support_copy_file Florian Weimer
@ 2020-11-09 18:40 ` Florian Weimer
2020-11-26 16:44 ` Adhemerval Zanella
2020-11-26 18:54 ` [PATCH 12/11] s390x: Add Add glibc-hwcaps support Florian Weimer
2020-11-09 18:40 ` [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH Florian Weimer
` (10 subsequent siblings)
12 siblings, 2 replies; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:40 UTC (permalink / raw)
To: libc-alpha
---
elf/cache.c | 24 ++++++++++++------------
elf/ldconfig.c | 9 ++++-----
sysdeps/generic/ldconfig.h | 9 ++++++++-
3 files changed, 24 insertions(+), 18 deletions(-)
diff --git a/elf/cache.c b/elf/cache.c
index 1eb1455883..c241c17ef9 100644
--- a/elf/cache.c
+++ b/elf/cache.c
@@ -321,13 +321,13 @@ save_cache (const char *cache_name)
struct cache_file *file_entries = NULL;
size_t file_entries_size = 0;
- if (opt_format != 2)
+ if (opt_format != opt_format_new)
{
/* struct cache_file_new is 64-bit aligned on some arches while
only 32-bit aligned on other arches. Duplicate last old
cache entry so that new cache in ld.so.cache can be used by
both. */
- if (opt_format != 0)
+ if (opt_format != opt_format_old)
cache_entry_old_count = (cache_entry_old_count + 1) & ~1;
/* And the list of all entries in the old format. */
@@ -345,7 +345,7 @@ save_cache (const char *cache_name)
struct cache_file_new *file_entries_new = NULL;
size_t file_entries_new_size = 0;
- if (opt_format != 0)
+ if (opt_format != opt_format_old)
{
/* And the list of all entries in the new format. */
file_entries_new_size = sizeof (struct cache_file_new)
@@ -370,7 +370,7 @@ save_cache (const char *cache_name)
table, we have to adjust all string indices for this so that
old libc5/glibc 2 dynamic linkers just ignore them. */
unsigned int str_offset;
- if (opt_format != 0)
+ if (opt_format != opt_format_old)
str_offset = file_entries_new_size;
else
str_offset = 0;
@@ -385,13 +385,13 @@ save_cache (const char *cache_name)
entry = entry->next, ++idx_new)
{
/* First the library. */
- if (opt_format != 2 && entry->hwcap == 0)
+ if (opt_format != opt_format_new && entry->hwcap == 0)
{
file_entries->libs[idx_old].flags = entry->flags;
/* XXX: Actually we can optimize here and remove duplicates. */
file_entries->libs[idx_old].key = str_offset + pad;
}
- if (opt_format != 0)
+ if (opt_format != opt_format_old)
{
/* We could subtract file_entries_new_size from str_offset -
not doing so makes the code easier, the string table
@@ -407,9 +407,9 @@ save_cache (const char *cache_name)
str = mempcpy (str, entry->lib, len);
str_offset += len;
/* Then the path. */
- if (opt_format != 2 && entry->hwcap == 0)
+ if (opt_format != opt_format_new && entry->hwcap == 0)
file_entries->libs[idx_old].value = str_offset + pad;
- if (opt_format != 0)
+ if (opt_format != opt_format_old)
file_entries_new->libs[idx_new].value = str_offset;
len = strlen (entry->path) + 1;
str = mempcpy (str, entry->path, len);
@@ -420,7 +420,7 @@ save_cache (const char *cache_name)
}
/* Duplicate last old cache entry if needed. */
- if (opt_format != 2
+ if (opt_format != opt_format_new
&& idx_old < cache_entry_old_count)
file_entries->libs[idx_old] = file_entries->libs[idx_old - 1];
@@ -438,16 +438,16 @@ save_cache (const char *cache_name)
temp_name);
/* Write contents. */
- if (opt_format != 2)
+ if (opt_format != opt_format_new)
{
if (write (fd, file_entries, file_entries_size)
!= (ssize_t) file_entries_size)
error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
}
- if (opt_format != 0)
+ if (opt_format != opt_format_old)
{
/* Align cache. */
- if (opt_format != 2)
+ if (opt_format != opt_format_new)
{
char zero[pad];
memset (zero, '\0', pad);
diff --git a/elf/ldconfig.c b/elf/ldconfig.c
index 3768267bac..006198fe59 100644
--- a/elf/ldconfig.c
+++ b/elf/ldconfig.c
@@ -100,8 +100,7 @@ static int opt_print_cache;
int opt_verbose;
/* Format to support. */
-/* 0: only libc5/glibc2; 1: both; 2: only glibc 2.2. */
-int opt_format = 2;
+enum opt_format opt_format = opt_format_new;
/* Build cache. */
static int opt_build_cache = 1;
@@ -281,11 +280,11 @@ parse_opt (int key, char *arg, struct argp_state *state)
break;
case 'c':
if (strcmp (arg, "old") == 0)
- opt_format = 0;
+ opt_format = opt_format_old;
else if (strcmp (arg, "compat") == 0)
- opt_format = 1;
+ opt_format = opt_format_compat;
else if (strcmp (arg, "new") == 0)
- opt_format = 2;
+ opt_format = opt_format_new;
break;
default:
return ARGP_ERR_UNKNOWN;
diff --git a/sysdeps/generic/ldconfig.h b/sysdeps/generic/ldconfig.h
index b64aab0064..cfec9d4668 100644
--- a/sysdeps/generic/ldconfig.h
+++ b/sysdeps/generic/ldconfig.h
@@ -90,7 +90,14 @@ extern char *chroot_canon (const char *chroot, const char *name);
/* Declared in ldconfig.c. */
extern int opt_verbose;
-extern int opt_format;
+enum opt_format
+ {
+ opt_format_old = 0, /* Use struct cache_file. */
+ opt_format_compat = 1, /* Use both, old format followed by new. */
+ opt_format_new = 2, /* Use struct cache_file_new. */
+ };
+
+extern enum opt_format opt_format;
/* Prototypes for a few program-wide used functions. */
#include <programs/xmalloc.h>
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 02/11] elf: Introduce enum opt_format in the ldconfig implementation
2020-11-09 18:40 ` [PATCH 02/11] elf: Introduce enum opt_format in the ldconfig implementation Florian Weimer
@ 2020-11-26 16:44 ` Adhemerval Zanella
2020-11-26 18:54 ` [PATCH 12/11] s390x: Add Add glibc-hwcaps support Florian Weimer
1 sibling, 0 replies; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-26 16:44 UTC (permalink / raw)
To: libc-alpha
LGTM, thanks.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
On 09/11/2020 15:40, Florian Weimer via Libc-alpha wrote:
> ---
> elf/cache.c | 24 ++++++++++++------------
> elf/ldconfig.c | 9 ++++-----
> sysdeps/generic/ldconfig.h | 9 ++++++++-
> 3 files changed, 24 insertions(+), 18 deletions(-)
>
> diff --git a/elf/cache.c b/elf/cache.c
> index 1eb1455883..c241c17ef9 100644
> --- a/elf/cache.c
> +++ b/elf/cache.c
> @@ -321,13 +321,13 @@ save_cache (const char *cache_name)
> struct cache_file *file_entries = NULL;
> size_t file_entries_size = 0;
>
> - if (opt_format != 2)
> + if (opt_format != opt_format_new)
> {
> /* struct cache_file_new is 64-bit aligned on some arches while
> only 32-bit aligned on other arches. Duplicate last old
> cache entry so that new cache in ld.so.cache can be used by
> both. */
> - if (opt_format != 0)
> + if (opt_format != opt_format_old)
> cache_entry_old_count = (cache_entry_old_count + 1) & ~1;
>
> /* And the list of all entries in the old format. */
> @@ -345,7 +345,7 @@ save_cache (const char *cache_name)
> struct cache_file_new *file_entries_new = NULL;
> size_t file_entries_new_size = 0;
>
> - if (opt_format != 0)
> + if (opt_format != opt_format_old)
> {
> /* And the list of all entries in the new format. */
> file_entries_new_size = sizeof (struct cache_file_new)
> @@ -370,7 +370,7 @@ save_cache (const char *cache_name)
> table, we have to adjust all string indices for this so that
> old libc5/glibc 2 dynamic linkers just ignore them. */
> unsigned int str_offset;
> - if (opt_format != 0)
> + if (opt_format != opt_format_old)
> str_offset = file_entries_new_size;
> else
> str_offset = 0;
> @@ -385,13 +385,13 @@ save_cache (const char *cache_name)
> entry = entry->next, ++idx_new)
> {
> /* First the library. */
> - if (opt_format != 2 && entry->hwcap == 0)
> + if (opt_format != opt_format_new && entry->hwcap == 0)
> {
> file_entries->libs[idx_old].flags = entry->flags;
> /* XXX: Actually we can optimize here and remove duplicates. */
> file_entries->libs[idx_old].key = str_offset + pad;
> }
> - if (opt_format != 0)
> + if (opt_format != opt_format_old)
> {
> /* We could subtract file_entries_new_size from str_offset -
> not doing so makes the code easier, the string table
> @@ -407,9 +407,9 @@ save_cache (const char *cache_name)
> str = mempcpy (str, entry->lib, len);
> str_offset += len;
> /* Then the path. */
> - if (opt_format != 2 && entry->hwcap == 0)
> + if (opt_format != opt_format_new && entry->hwcap == 0)
> file_entries->libs[idx_old].value = str_offset + pad;
> - if (opt_format != 0)
> + if (opt_format != opt_format_old)
> file_entries_new->libs[idx_new].value = str_offset;
> len = strlen (entry->path) + 1;
> str = mempcpy (str, entry->path, len);
> @@ -420,7 +420,7 @@ save_cache (const char *cache_name)
> }
>
> /* Duplicate last old cache entry if needed. */
> - if (opt_format != 2
> + if (opt_format != opt_format_new
> && idx_old < cache_entry_old_count)
> file_entries->libs[idx_old] = file_entries->libs[idx_old - 1];
>
> @@ -438,16 +438,16 @@ save_cache (const char *cache_name)
> temp_name);
>
> /* Write contents. */
> - if (opt_format != 2)
> + if (opt_format != opt_format_new)
> {
> if (write (fd, file_entries, file_entries_size)
> != (ssize_t) file_entries_size)
> error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
> }
> - if (opt_format != 0)
> + if (opt_format != opt_format_old)
> {
> /* Align cache. */
> - if (opt_format != 2)
> + if (opt_format != opt_format_new)
> {
> char zero[pad];
> memset (zero, '\0', pad);
> diff --git a/elf/ldconfig.c b/elf/ldconfig.c
> index 3768267bac..006198fe59 100644
> --- a/elf/ldconfig.c
> +++ b/elf/ldconfig.c
> @@ -100,8 +100,7 @@ static int opt_print_cache;
> int opt_verbose;
>
> /* Format to support. */
> -/* 0: only libc5/glibc2; 1: both; 2: only glibc 2.2. */
> -int opt_format = 2;
> +enum opt_format opt_format = opt_format_new;
>
> /* Build cache. */
> static int opt_build_cache = 1;
> @@ -281,11 +280,11 @@ parse_opt (int key, char *arg, struct argp_state *state)
> break;
> case 'c':
> if (strcmp (arg, "old") == 0)
> - opt_format = 0;
> + opt_format = opt_format_old;
> else if (strcmp (arg, "compat") == 0)
> - opt_format = 1;
> + opt_format = opt_format_compat;
> else if (strcmp (arg, "new") == 0)
> - opt_format = 2;
> + opt_format = opt_format_new;
> break;
> default:
> return ARGP_ERR_UNKNOWN;
> diff --git a/sysdeps/generic/ldconfig.h b/sysdeps/generic/ldconfig.h
> index b64aab0064..cfec9d4668 100644
> --- a/sysdeps/generic/ldconfig.h
> +++ b/sysdeps/generic/ldconfig.h
> @@ -90,7 +90,14 @@ extern char *chroot_canon (const char *chroot, const char *name);
> /* Declared in ldconfig.c. */
> extern int opt_verbose;
>
> -extern int opt_format;
> +enum opt_format
> + {
> + opt_format_old = 0, /* Use struct cache_file. */
> + opt_format_compat = 1, /* Use both, old format followed by new. */
> + opt_format_new = 2, /* Use struct cache_file_new. */
> + };
> +
> +extern enum opt_format opt_format;
>
> /* Prototypes for a few program-wide used functions. */
> #include <programs/xmalloc.h>
>
^ permalink raw reply [flat|nested] 58+ messages in thread* [PATCH 12/11] s390x: Add Add glibc-hwcaps support
2020-11-09 18:40 ` [PATCH 02/11] elf: Introduce enum opt_format in the ldconfig implementation Florian Weimer
2020-11-26 16:44 ` Adhemerval Zanella
@ 2020-11-26 18:54 ` Florian Weimer
2020-12-07 8:16 ` Florian Weimer
1 sibling, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-26 18:54 UTC (permalink / raw)
To: libc-alpha
Subdirectories z13, z14, z15 can be selected, mostly based on the
level of support for vector instructions.
Compared to the earlier version, this adds tests, slightly more
elaborate AT_HWCAP tests, and _dl_hwcaps_subdirs_build_bitmask is used.
I'm open to using different selection logic. I couldn't quite figure
out what GCC-generated code actually requires when using, say,
-march=z14.
The tests pass on a z15 and a zEC12 system, both with z/VM.
The patch is also available on the fw/glibc-hwcaps branch. The
elf/tst-glibc-hwcaps-cache.script patch won't apply against the posted
series.
Thanks,
Florian
---
elf/Makefile | 2 +-
elf/tst-glibc-hwcaps-cache.script | 10 ++++
sysdeps/s390/s390-64/Makefile | 39 ++++++++++++++++
sysdeps/s390/s390-64/dl-hwcaps-subdirs.c | 53 ++++++++++++++++++++++
sysdeps/s390/s390-64/tst-glibc-hwcaps.c | 78 ++++++++++++++++++++++++++++++++
5 files changed, 181 insertions(+), 1 deletion(-)
diff --git a/elf/Makefile b/elf/Makefile
index 37d3af593c..2b8458a18f 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -1828,7 +1828,7 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
# Most likely search subdirectories, for each supported architecture.
# Used to obtain test coverage wide test coverage.
-glibc-hwcaps-first-subdirs-for-tests = power9 x86-64-v2
+glibc-hwcaps-first-subdirs-for-tests = power9 x86-64-v2 z13
# The test modules are parameterized by preprocessor macros.
LDFLAGS-libmarkermod1-1.so += -Wl,-soname,libmarkermod1.so
diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
index 6a4675f9bd..19b06d0adc 100644
--- a/elf/tst-glibc-hwcaps-cache.script
+++ b/elf/tst-glibc-hwcaps-cache.script
@@ -11,6 +11,16 @@ mkdirp 0770 $L/glibc-hwcaps/power10
cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/power9/libmarkermod3.so
cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/power10/libmarkermod3.so
+mkdirp 0770 $L/glibc-hwcaps/z13
+cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/z13/libmarkermod2.so
+mkdirp 0770 $L/glibc-hwcaps/z14
+cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/z14/libmarkermod3.so
+cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/z14/libmarkermod3.so
+mkdirp 0770 $L/glibc-hwcaps/z15
+cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/z15/libmarkermod4.so
+cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/z15/libmarkermod4.so
+cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/z15/libmarkermod4.so
+
mkdirp 0770 $L/glibc-hwcaps/x86-64-v2
cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod2.so
mkdirp 0770 $L/glibc-hwcaps/x86-64-v3
diff --git a/sysdeps/s390/s390-64/Makefile b/sysdeps/s390/s390-64/Makefile
index b4d793bb3d..e5da26871c 100644
--- a/sysdeps/s390/s390-64/Makefile
+++ b/sysdeps/s390/s390-64/Makefile
@@ -6,4 +6,43 @@ ifeq ($(subdir),elf)
CFLAGS-rtld.c += -Wno-uninitialized -Wno-unused
CFLAGS-dl-load.c += -Wno-unused
CFLAGS-dl-reloc.c += -Wno-unused
+
+$(objpfx)tst-glibc-hwcaps: $(objpfx)libmarkermod2-1.so \
+ $(objpfx)libmarkermod3-1.so $(objpfx)libmarkermod4-1.so
+$(objpfx)tst-glibc-hwcaps.out: \
+ $(objpfx)libmarkermod2.so \
+ $(objpfx)glibc-hwcaps/z13/libmarkermod2.so \
+ $(objpfx)libmarkermod3.so \
+ $(objpfx)glibc-hwcaps/z13/libmarkermod3.so \
+ $(objpfx)glibc-hwcaps/z14/libmarkermod3.so \
+ $(objpfx)libmarkermod4.so \
+ $(objpfx)glibc-hwcaps/z13/libmarkermod4.so \
+ $(objpfx)glibc-hwcaps/z14/libmarkermod4.so \
+ $(objpfx)glibc-hwcaps/z15/libmarkermod4.so \
+
+$(objpfx)glibc-hwcaps/z13/libmarkermod2.so: $(objpfx)libmarkermod2-2.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/z13/libmarkermod3.so: $(objpfx)libmarkermod3-2.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/z14/libmarkermod3.so: $(objpfx)libmarkermod3-3.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/z13/libmarkermod4.so: $(objpfx)libmarkermod4-2.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/z14/libmarkermod4.so: $(objpfx)libmarkermod4-3.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/z15/libmarkermod4.so: $(objpfx)libmarkermod4-4.so
+ $(make-target-directory)
+ cp $< $@
+
+ifeq (no,$(build-hardcoded-path-in-tests))
+# This is an ld.so.cache test, and RPATH/RUNPATH in the executable
+# interferes with its test objectives.
+tests-container += tst-glibc-hwcaps-cache
endif
+
+endif # $(subdir) == elf
diff --git a/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c b/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
new file mode 100644
index 0000000000..fa8d2ce1f1
--- /dev/null
+++ b/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
@@ -0,0 +1,53 @@
+/* Architecture-specific glibc-hwcaps subdirectories. s390x version.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <dl-hwcaps.h>
+#include <ldsodefs.h>
+
+const char _dl_hwcaps_subdirs[] = "z15:z14:z13";
+enum { subdirs_count = 3 }; /* Number of components in _dl_hwcaps_subdirs. */
+
+uint32_t
+_dl_hwcaps_subdirs_active (void)
+{
+ int active = 0;
+
+ /* Test in reverse preference order. */
+
+ /* z13. */
+ if (!(GLRO (dl_hwcap) & HWCAP_S390_VX))
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+ ++active;
+
+ /* z14. */
+ if (!((GLRO (dl_hwcap) & HWCAP_S390_VXD)
+ && (GLRO (dl_hwcap) & HWCAP_S390_VXE)
+ && (GLRO (dl_hwcap) & HWCAP_S390_GS)))
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+ ++active;
+
+ /* z15. */
+ if (!((GLRO (dl_hwcap) & HWCAP_S390_VXRS_EXT2)
+ && (GLRO (dl_hwcap) & HWCAP_S390_VXRS_PDE)
+ && (GLRO (dl_hwcap) & HWCAP_S390_SORT)
+ && (GLRO (dl_hwcap) & HWCAP_S390_DFLT)))
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+ ++active;
+
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+}
diff --git a/sysdeps/s390/s390-64/tst-glibc-hwcaps.c b/sysdeps/s390/s390-64/tst-glibc-hwcaps.c
new file mode 100644
index 0000000000..39f56d0c81
--- /dev/null
+++ b/sysdeps/s390/s390-64/tst-glibc-hwcaps.c
@@ -0,0 +1,78 @@
+/* glibc-hwcaps subdirectory test. s390x version.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <stdio.h>
+#include <string.h>
+#include <support/check.h>
+#include <sys/auxv.h>
+#include <sys/param.h>
+
+extern int marker2 (void);
+extern int marker3 (void);
+extern int marker4 (void);
+
+/* Return the POWER level, 8 for the baseline. */
+static int
+compute_level (void)
+{
+ const char *platform = (const char *) getauxval (AT_PLATFORM);
+
+ int result;
+ if (sscanf (platform, "arch%d", &result) == 1)
+ return result;
+
+ /* The arch* versions refer to the edition of the Principles of
+ Operation, and they are off by two when compared with the recent
+ product names. (The code below should not be considered an
+ accurate mapping to Principles of Operation editions for earlier
+ AT_PLATFORM strings). */
+ if (strcmp (platform, "z900") == 0)
+ return 5;
+ if (strcmp (platform, "z990") == 0)
+ return 6;
+ if (strcmp (platform, "z9-109") == 0)
+ return 7;
+ if (strcmp (platform, "z10") == 0)
+ return 8;
+ if (strcmp (platform, "z196") == 0)
+ return 9;
+ if (strcmp (platform, "zEC12") == 0)
+ return 10;
+ if (strcmp (platform, "z13") == 0)
+ return 11;
+ if (strcmp (platform, "z14") == 0)
+ return 12;
+ if (strcmp (platform, "z15") == 0)
+ return 13;
+ printf ("warning: unrecognized AT_PLATFORM value: %s\n", platform);
+ /* Assume that the new platform supports z15. */
+ return 13;
+}
+
+static int
+do_test (void)
+{
+ int level = compute_level ();
+ printf ("info: detected architecture level: arch%d\n", level);
+ TEST_COMPARE (marker2 (), MIN (level - 9, 2));
+ TEST_COMPARE (marker3 (), MIN (level - 9, 3));
+ TEST_COMPARE (marker4 (), MIN (level - 9, 4));
+ return 0;
+}
+
+#include <support/test-driver.c>
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 12/11] s390x: Add Add glibc-hwcaps support
2020-11-26 18:54 ` [PATCH 12/11] s390x: Add Add glibc-hwcaps support Florian Weimer
@ 2020-12-07 8:16 ` Florian Weimer
2020-12-08 15:47 ` Stefan Liebler
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-12-07 8:16 UTC (permalink / raw)
To: Florian Weimer via Libc-alpha
* Florian Weimer via Libc-alpha:
> diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
> index 6a4675f9bd..19b06d0adc 100644
> --- a/elf/tst-glibc-hwcaps-cache.script
> +++ b/elf/tst-glibc-hwcaps-cache.script
> @@ -11,6 +11,16 @@ mkdirp 0770 $L/glibc-hwcaps/power10
> cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/power9/libmarkermod3.so
> cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/power10/libmarkermod3.so
>
> +mkdirp 0770 $L/glibc-hwcaps/z13
> +cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/z13/libmarkermod2.so
> +mkdirp 0770 $L/glibc-hwcaps/z14
> +cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/z14/libmarkermod3.so
> +cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/z14/libmarkermod3.so
> +mkdirp 0770 $L/glibc-hwcaps/z15
> +cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/z15/libmarkermod4.so
> +cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/z15/libmarkermod4.so
> +cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/z15/libmarkermod4.so
> +
> mkdirp 0770 $L/glibc-hwcaps/x86-64-v2
> cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod2.so
> mkdirp 0770 $L/glibc-hwcaps/x86-64-v3
Stefan pointed out that this should be:
mkdirp 0770 $L/glibc-hwcaps/z13
cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/z13/libmarkermod2.so
mkdirp 0770 $L/glibc-hwcaps/z14
cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/z13/libmarkermod3.so
cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/z14/libmarkermod3.so
mkdirp 0770 $L/glibc-hwcaps/z15
cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/z13/libmarkermod4.so
cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/z14/libmarkermod4.so
cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/z15/libmarkermod4.so
I have fixed it on the fw/glibc-hwcaps branch.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH 12/11] s390x: Add Add glibc-hwcaps support
2020-12-07 8:16 ` Florian Weimer
@ 2020-12-08 15:47 ` Stefan Liebler
2020-12-09 18:52 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Stefan Liebler @ 2020-12-08 15:47 UTC (permalink / raw)
To: libc-alpha; +Cc: Florian Weimer
On 12/7/20 9:16 AM, Florian Weimer via Libc-alpha wrote:
> * Florian Weimer via Libc-alpha:
>
>> diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
>> index 6a4675f9bd..19b06d0adc 100644
>> --- a/elf/tst-glibc-hwcaps-cache.script
>> +++ b/elf/tst-glibc-hwcaps-cache.script
>> @@ -11,6 +11,16 @@ mkdirp 0770 $L/glibc-hwcaps/power10
>> cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/power9/libmarkermod3.so
>> cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/power10/libmarkermod3.so
>>
>> +mkdirp 0770 $L/glibc-hwcaps/z13
>> +cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/z13/libmarkermod2.so
>> +mkdirp 0770 $L/glibc-hwcaps/z14
>> +cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/z14/libmarkermod3.so
>> +cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/z14/libmarkermod3.so
>> +mkdirp 0770 $L/glibc-hwcaps/z15
>> +cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/z15/libmarkermod4.so
>> +cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/z15/libmarkermod4.so
>> +cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/z15/libmarkermod4.so
>> +
>> mkdirp 0770 $L/glibc-hwcaps/x86-64-v2
>> cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod2.so
>> mkdirp 0770 $L/glibc-hwcaps/x86-64-v3
>
> Stefan pointed out that this should be:
>
> mkdirp 0770 $L/glibc-hwcaps/z13
> cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/z13/libmarkermod2.so
> mkdirp 0770 $L/glibc-hwcaps/z14
> cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/z13/libmarkermod3.so
> cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/z14/libmarkermod3.so
> mkdirp 0770 $L/glibc-hwcaps/z15
> cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/z13/libmarkermod4.so
> cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/z14/libmarkermod4.so
> cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/z15/libmarkermod4.so
>
> I have fixed it on the fw/glibc-hwcaps branch.
>
> Thanks,
> Florian
>
Hi Florian,
I've had a look to your patches. Can you please adjust some lines. Then
this patch is okay for s390x:
- The commit subject-line contains "Add Add"
- If e.g. a machine newer-than-z15 does not have HWCAP_S390_SORT, then
it would fall back to z14:
diff --git a/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
b/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
index fa8d2ce1f1..3673808a45 100644
--- a/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
+++ b/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
@@ -41,11 +41,12 @@ _dl_hwcaps_subdirs_active (void)
return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
++active;
- /* z15. */
+ /* z15.
+ Note: We do not list HWCAP_S390_SORT and HWCAP_S390_DFLT here as,
+ according to the Principles of Operation, those may be replaced or
removed
+ in future. */
if (!((GLRO (dl_hwcap) & HWCAP_S390_VXRS_EXT2)
- && (GLRO (dl_hwcap) & HWCAP_S390_VXRS_PDE)
- && (GLRO (dl_hwcap) & HWCAP_S390_SORT)
- && (GLRO (dl_hwcap) & HWCAP_S390_DFLT)))
+ && (GLRO (dl_hwcap) & HWCAP_S390_VXRS_PDE)))
return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
++active;
- I've asked the kernel-guys regarding AT_PLATFORM: The list is
complete, but it never contains archXYZ. This is only available for
binutils/gcc -march=archXYZ.
- If running e.g. on z196 or older, tst-glibc-hwcaps will always fail,
as level would be <= 9 which leads to fails:
TEST_COMPARE (marker2 (), MIN (level - 9, 2));
Therefore compute_level should always return the baseline for older
platforms.
- If we are running on z13 or newer and the kernel was booted with novx,
then AT_PLATFORM is z13 or newer, but _dl_hwcaps_subdirs_active will
return zero and the _dl_hwcaps_subdirs are not searched as HWCAP_S390_VX
and all the other VX.. flags are not set. This leads to a test fail.
diff --git a/sysdeps/s390/s390-64/tst-glibc-hwcaps.c
b/sysdeps/s390/s390-64/tst-glibc-hwcaps.c
index 39f56d0c81..690f0d5fab 100644
--- a/sysdeps/s390/s390-64/tst-glibc-hwcaps.c
+++ b/sysdeps/s390/s390-64/tst-glibc-hwcaps.c
@@ -26,33 +26,37 @@ extern int marker2 (void);
extern int marker3 (void);
extern int marker4 (void);
-/* Return the POWER level, 8 for the baseline. */
+/* Return the arch level, 10 for the baseline libmarkermod*.so's. */
static int
compute_level (void)
{
const char *platform = (const char *) getauxval (AT_PLATFORM);
- int result;
- if (sscanf (platform, "arch%d", &result) == 1)
- return result;
-
/* The arch* versions refer to the edition of the Principles of
Operation, and they are off by two when compared with the recent
product names. (The code below should not be considered an
accurate mapping to Principles of Operation editions for earlier
AT_PLATFORM strings). */
if (strcmp (platform, "z900") == 0)
- return 5;
+ return 10;
if (strcmp (platform, "z990") == 0)
- return 6;
+ return 10;
if (strcmp (platform, "z9-109") == 0)
- return 7;
+ return 10;
if (strcmp (platform, "z10") == 0)
- return 8;
+ return 10;
if (strcmp (platform, "z196") == 0)
- return 9;
+ return 10;
if (strcmp (platform, "zEC12") == 0)
return 10;
+
+ /* If we are running on z13 or newer and the kernel was booted with novx,
+ then AT_PLATFORM is z13 or newer, but _dl_hwcaps_subdirs_active will
+ return zero and the _dl_hwcaps_subdirs are not searched. */
+ const unsigned long int hwcap = getauxval (AT_HWCAP);
+ if ((hwcap & HWCAP_S390_VX) == 0)
+ return 10;
+
if (strcmp (platform, "z13") == 0)
return 11;
if (strcmp (platform, "z14") == 0)
I've also recognized that if build with gcc 6.5.0, I'll get test-fails:
elf/tst-glibc-hwcaps-cache
elf/tst-glibc-hwcaps-prepend-cache
elf/tst-ldconfig-X
elf/tst-ldconfig-bad-aux-cache
elf/tst-ldconfig-ld_so_conf-update
elf/tst-stringtable
It seems as it always fails with "String table is too large".
I've debugged elf/tst-stringtable into elf/stringtable:185:
else if (__builtin_add_overflow (previous->offset,
previous->length + 1,
¤t->offset))
error (EXIT_FAILURE, 0, _("String table is too large"));
It seems as gcc 6.5.0 __builtin_add_overflow is buggy:
al %r12,12(%r9)
st %r12,12(%r7)
jhe 0x1002762 <stringtable_finalize+274>
larl %r4,0x10050d6
lghi %r3,0
lghi %r2,1
brasl %r14,0x1001bf8 <error@plt>
=> The jump should jump away in case of no overflow.
(gdb) p *previous (== r9)
$2 = {next = 0x0, length = 3, offset = 7, string = 0x3fffdc0b800 "999"}
(gdb) p *current (== r7)
$3 = {next = 0x0, length = 10, offset = 11, string = 0x3fffdfbf5c0
"prefix/899"}
(gdb) p *(uint32_t*) (12 + $r9)
$1 = 7
(gdb) i r r12 cc
r12 0x3ff0000000b 4393751543819
cc 0x1 1
=> cc is 1, but "jhe" jumps away for cc==0 or cc==2.
Resulting Condition Code (of al instruction):
0 Result zero; no carry
1 Result not zero; no carry
2 Result zero; carry
3 Result not zero; carry
If build with other GCC versions, I don't see this bug.
Have you recognized such a bug on other archs?
While viewing your previous patches, I've found the following "if" in
commit "elf: Add glibc-hwcaps subdirectory support to ld.so cache
processing":
elf/dl-hwcaps.c:
static void
+sort_priorities_by_name (void)
+{
...
+ int cmp = memcmp (current->name, previous->name, to_compare);
+ if (cmp >= 0
+ || (cmp == 0 && current->name_length >= previous->name_length))
+ break;
Is this condition intended? The second part "cmp == 0" will never be
evaluated as in this case, the first part "cmp >= 0" is already true.
Thanks,
Stefan
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 12/11] s390x: Add Add glibc-hwcaps support
2020-12-08 15:47 ` Stefan Liebler
@ 2020-12-09 18:52 ` Florian Weimer
2020-12-10 10:22 ` Stefan Liebler
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-12-09 18:52 UTC (permalink / raw)
To: Stefan Liebler; +Cc: libc-alpha
* Stefan Liebler:
> I've had a look to your patches. Can you please adjust some lines. Then
> this patch is okay for s390x:
> - The commit subject-line contains "Add Add"
Oops, fixed.
> - If e.g. a machine newer-than-z15 does not have HWCAP_S390_SORT, then
> it would fall back to z14:
>
> diff --git a/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
> b/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
> index fa8d2ce1f1..3673808a45 100644
> --- a/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
> +++ b/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
> @@ -41,11 +41,12 @@ _dl_hwcaps_subdirs_active (void)
> return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> ++active;
>
> - /* z15. */
> + /* z15.
> + Note: We do not list HWCAP_S390_SORT and HWCAP_S390_DFLT here as,
> + according to the Principles of Operation, those may be replaced or
> removed
> + in future. */
> if (!((GLRO (dl_hwcap) & HWCAP_S390_VXRS_EXT2)
> - && (GLRO (dl_hwcap) & HWCAP_S390_VXRS_PDE)
> - && (GLRO (dl_hwcap) & HWCAP_S390_SORT)
> - && (GLRO (dl_hwcap) & HWCAP_S390_DFLT)))
> + && (GLRO (dl_hwcap) & HWCAP_S390_VXRS_PDE)))
> return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> ++active;
Does -march=z15 imply SORT and DFLT? That bit wasn't clear to me.
If it does not, we should net test for it.
> - I've asked the kernel-guys regarding AT_PLATFORM: The list is
> complete, but it never contains archXYZ. This is only available for
> binutils/gcc -march=archXYZ.
Okay, I went with the current kernel sources and did not check the
history there.
> - If running e.g. on z196 or older, tst-glibc-hwcaps will always fail,
> as level would be <= 9 which leads to fails:
> TEST_COMPARE (marker2 (), MIN (level - 9, 2));
> Therefore compute_level should always return the baseline for older
> platforms.
Oh, good point.
> - If we are running on z13 or newer and the kernel was booted with novx,
> then AT_PLATFORM is z13 or newer, but _dl_hwcaps_subdirs_active will
> return zero and the _dl_hwcaps_subdirs are not searched as HWCAP_S390_VX
> and all the other VX.. flags are not set. This leads to a test fail.
That's quite bad because it breaks the existing AT_PLATFORM subdirectory
for z13 and newer. (I expect the reason to have such a directory is to
put vectorized code there.) Given that this is an unsupported
configuration for glibc, maybe the test failure is acceptable. On the
other hand, if we deprecate the old mechanism (separate discussion),
this becomes a supported configuration, so adapting the test makes
sense.
> I've also recognized that if build with gcc 6.5.0, I'll get test-fails:
> elf/tst-glibc-hwcaps-cache
> elf/tst-glibc-hwcaps-prepend-cache
> elf/tst-ldconfig-X
> elf/tst-ldconfig-bad-aux-cache
> elf/tst-ldconfig-ld_so_conf-update
> elf/tst-stringtable
>
> It seems as it always fails with "String table is too large".
> I've debugged elf/tst-stringtable into elf/stringtable:185:
> else if (__builtin_add_overflow (previous->offset,
> previous->length + 1,
> ¤t->offset))
> error (EXIT_FAILURE, 0, _("String table is too large"));
>
> It seems as gcc 6.5.0 __builtin_add_overflow is buggy:
Sorry, I do not recall a discussion of this particular bug. I do not
see changes s390.md that would correspond to this (but then I'm not GCC
developer …).
I haven't seen such a failure on other architectures.
> While viewing your previous patches, I've found the following "if" in
> commit "elf: Add glibc-hwcaps subdirectory support to ld.so cache
> processing":
> elf/dl-hwcaps.c:
> static void
> +sort_priorities_by_name (void)
> +{
> ...
> + int cmp = memcmp (current->name, previous->name, to_compare);
> + if (cmp >= 0
> + || (cmp == 0 && current->name_length >= previous->name_length))
> + break;
> Is this condition intended? The second part "cmp == 0" will never be
> evaluated as in this case, the first part "cmp >= 0" is already true.
Thanks, I've pushed a fix for that.
I'll post an update of the s390x patch soon.
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 12/11] s390x: Add Add glibc-hwcaps support
2020-12-09 18:52 ` Florian Weimer
@ 2020-12-10 10:22 ` Stefan Liebler
2020-12-10 14:54 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Stefan Liebler @ 2020-12-10 10:22 UTC (permalink / raw)
To: Florian Weimer; +Cc: libc-alpha
On 12/9/20 7:52 PM, Florian Weimer wrote:
> * Stefan Liebler:
>
>> I've had a look to your patches. Can you please adjust some lines. Then
>> this patch is okay for s390x:
>> - The commit subject-line contains "Add Add"
>
> Oops, fixed.
>
>> - If e.g. a machine newer-than-z15 does not have HWCAP_S390_SORT, then
>> it would fall back to z14:
>>
>> diff --git a/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
>> b/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
>> index fa8d2ce1f1..3673808a45 100644
>> --- a/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
>> +++ b/sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
>> @@ -41,11 +41,12 @@ _dl_hwcaps_subdirs_active (void)
>> return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
>> ++active;
>>
>> - /* z15. */
>> + /* z15.
>> + Note: We do not list HWCAP_S390_SORT and HWCAP_S390_DFLT here as,
>> + according to the Principles of Operation, those may be replaced or
>> removed
>> + in future. */
>> if (!((GLRO (dl_hwcap) & HWCAP_S390_VXRS_EXT2)
>> - && (GLRO (dl_hwcap) & HWCAP_S390_VXRS_PDE)
>> - && (GLRO (dl_hwcap) & HWCAP_S390_SORT)
>> - && (GLRO (dl_hwcap) & HWCAP_S390_DFLT)))
>> + && (GLRO (dl_hwcap) & HWCAP_S390_VXRS_PDE)))
>> return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
>> ++active;
>
> Does -march=z15 imply SORT and DFLT? That bit wasn't clear to me.
> If it does not, we should net test for it.
No, the corresponding instructions are not emitted by gcc -march=z15.
E.g. dfltcc is manually added to zlib. There is a runtime check if this
facility is available.
>
>> - I've asked the kernel-guys regarding AT_PLATFORM: The list is
>> complete, but it never contains archXYZ. This is only available for
>> binutils/gcc -march=archXYZ.
>
> Okay, I went with the current kernel sources and did not check the
> history there.
>
>> - If running e.g. on z196 or older, tst-glibc-hwcaps will always fail,
>> as level would be <= 9 which leads to fails:
>> TEST_COMPARE (marker2 (), MIN (level - 9, 2));
>> Therefore compute_level should always return the baseline for older
>> platforms.
>
> Oh, good point.
>
>> - If we are running on z13 or newer and the kernel was booted with novx,
>> then AT_PLATFORM is z13 or newer, but _dl_hwcaps_subdirs_active will
>> return zero and the _dl_hwcaps_subdirs are not searched as HWCAP_S390_VX
>> and all the other VX.. flags are not set. This leads to a test fail.
>
> That's quite bad because it breaks the existing AT_PLATFORM subdirectory
> for z13 and newer. (I expect the reason to have such a directory is to
> put vectorized code there.) Given that this is an unsupported
> configuration for glibc, maybe the test failure is acceptable. On the
> other hand, if we deprecate the old mechanism (separate discussion),
> this becomes a supported configuration, so adapting the test makes
> sense.
I assume novx is usually used only for testing purposes. If e.g. RHEL 8
is booted with novx, it would fail on the first vector instruction as
the ALS includes those.
Currently I know, that the dfp-subdirectory was used for libdfp in the
past. Now the distros require a minimum architecture level which always
has support for dfp and the base libdfp library is built with hardware
dfp instructions.
There are also different flavors for libatlas. As far as I know, those
are located in the corresponding vector-hwcap subdirectories instead of
z13/z14/z15.
Do you have an outlook when you plan to deprecate/remove the old mechanism?
>
>> I've also recognized that if build with gcc 6.5.0, I'll get test-fails:
>> elf/tst-glibc-hwcaps-cache
>> elf/tst-glibc-hwcaps-prepend-cache
>> elf/tst-ldconfig-X
>> elf/tst-ldconfig-bad-aux-cache
>> elf/tst-ldconfig-ld_so_conf-update
>> elf/tst-stringtable
>>
>> It seems as it always fails with "String table is too large".
>> I've debugged elf/tst-stringtable into elf/stringtable:185:
>> else if (__builtin_add_overflow (previous->offset,
>> previous->length + 1,
>> ¤t->offset))
>> error (EXIT_FAILURE, 0, _("String table is too large"));
>>
>> It seems as gcc 6.5.0 __builtin_add_overflow is buggy:
>
> Sorry, I do not recall a discussion of this particular bug. I do not
> see changes s390.md that would correspond to this (but then I'm not GCC
> developer …).
>
> I haven't seen such a failure on other architectures.
I assume, there won't be a further gcc 6 version in future which could
fix it. As ldconfig would fail with "String table is too large" if build
with gcc 6.5, would you recommend to use gcc 7.1 as minumum on
s390/s390x? Currently gcc 6.2 and newer is required in <glibc>/configure.ac.
>
>> While viewing your previous patches, I've found the following "if" in
>> commit "elf: Add glibc-hwcaps subdirectory support to ld.so cache
>> processing":
>> elf/dl-hwcaps.c:
>> static void
>> +sort_priorities_by_name (void)
>> +{
>> ...
>> + int cmp = memcmp (current->name, previous->name, to_compare);
>> + if (cmp >= 0
>> + || (cmp == 0 && current->name_length >= previous->name_length))
>> + break;
>> Is this condition intended? The second part "cmp == 0" will never be
>> evaluated as in this case, the first part "cmp >= 0" is already true.
>
> Thanks, I've pushed a fix for that.
>
> I'll post an update of the s390x patch soon.
>
> Florian
>
Thanks,
Stefan
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 12/11] s390x: Add Add glibc-hwcaps support
2020-12-10 10:22 ` Stefan Liebler
@ 2020-12-10 14:54 ` Florian Weimer
0 siblings, 0 replies; 58+ messages in thread
From: Florian Weimer @ 2020-12-10 14:54 UTC (permalink / raw)
To: Stefan Liebler via Libc-alpha; +Cc: Stefan Liebler
* Stefan Liebler via Libc-alpha:
>> That's quite bad because it breaks the existing AT_PLATFORM subdirectory
>> for z13 and newer. (I expect the reason to have such a directory is to
>> put vectorized code there.) Given that this is an unsupported
>> configuration for glibc, maybe the test failure is acceptable. On the
>> other hand, if we deprecate the old mechanism (separate discussion),
>> this becomes a supported configuration, so adapting the test makes
>> sense.
> I assume novx is usually used only for testing purposes. If e.g. RHEL 8
> is booted with novx, it would fail on the first vector instruction as
> the ALS includes those.
>
> Currently I know, that the dfp-subdirectory was used for libdfp in the
> past. Now the distros require a minimum architecture level which always
> has support for dfp and the base libdfp library is built with hardware
> dfp instructions.
>
> There are also different flavors for libatlas. As far as I know, those
> are located in the corresponding vector-hwcap subdirectories instead of
> z13/z14/z15.
>
> Do you have an outlook when you plan to deprecate/remove the old mechanism?
I'll start a separate thread for this topic once I've dealt with some
urgent priority work on my end.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
2020-11-09 18:40 ` [PATCH 01/11] support: Add support_copy_file Florian Weimer
2020-11-09 18:40 ` [PATCH 02/11] elf: Introduce enum opt_format in the ldconfig implementation Florian Weimer
@ 2020-11-09 18:40 ` Florian Weimer
2020-11-26 20:29 ` Adhemerval Zanella
2020-11-09 18:40 ` [PATCH 04/11] elf: Add endianness markup to ld.so.cache Florian Weimer
` (9 subsequent siblings)
12 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:40 UTC (permalink / raw)
To: libc-alpha
This hacks non-power-set processing into _dl_important_hwcaps.
Once the legacy hwcaps handling goes away, the subdirectory
handling needs to be reworked, but it is premature to do this
while both approaches are still supported.
---
elf/Makefile | 67 ++++++++++++++--
elf/dl-hwcaps-subdirs.c | 29 +++++++
elf/dl-hwcaps.c | 138 +++++++++++++++++++++++++++-----
elf/dl-hwcaps.h | 105 +++++++++++++++++++++++++
elf/dl-hwcaps_split.c | 77 ++++++++++++++++++
elf/dl-load.c | 7 +-
elf/dl-main.h | 11 ++-
elf/dl-support.c | 5 +-
elf/dl-usage.c | 68 +++++++++++++++-
elf/markermodMARKER-VALUE.c | 29 +++++++
elf/rtld.c | 18 +++++
elf/tst-dl-hwcaps_split.c | 139 +++++++++++++++++++++++++++++++++
elf/tst-glibc-hwcaps-mask.c | 31 ++++++++
elf/tst-glibc-hwcaps-prepend.c | 32 ++++++++
elf/tst-glibc-hwcaps.c | 28 +++++++
sysdeps/generic/ldsodefs.h | 20 +++--
16 files changed, 771 insertions(+), 33 deletions(-)
create mode 100644 elf/dl-hwcaps-subdirs.c
create mode 100644 elf/dl-hwcaps_split.c
create mode 100644 elf/markermodMARKER-VALUE.c
create mode 100644 elf/tst-dl-hwcaps_split.c
create mode 100644 elf/tst-glibc-hwcaps-mask.c
create mode 100644 elf/tst-glibc-hwcaps-prepend.c
create mode 100644 elf/tst-glibc-hwcaps.c
diff --git a/elf/Makefile b/elf/Makefile
index f10cc59e7c..d2f7e99863 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -59,7 +59,8 @@ elide-routines.os = $(all-dl-routines) dl-support enbl-secure dl-origin \
# ld.so uses those routines, plus some special stuff for being the program
# interpreter and operating independent of libc.
rtld-routines = rtld $(all-dl-routines) dl-sysdep dl-environ dl-minimal \
- dl-error-minimal dl-conflict dl-hwcaps dl-usage
+ dl-error-minimal dl-conflict dl-hwcaps dl-hwcaps_split dl-hwcaps-subdirs \
+ dl-usage
all-rtld-routines = $(rtld-routines) $(sysdep-rtld-routines)
CFLAGS-dl-runtime.c += -fexceptions -fasynchronous-unwind-tables
@@ -210,14 +211,14 @@ tests += restest1 preloadtest loadfail multiload origtest resolvfail \
tst-filterobj tst-filterobj-dlopen tst-auxobj tst-auxobj-dlopen \
tst-audit14 tst-audit15 tst-audit16 \
tst-single_threaded tst-single_threaded-pthread \
- tst-tls-ie tst-tls-ie-dlmopen \
- argv0test
+ tst-tls-ie tst-tls-ie-dlmopen argv0test \
+ tst-glibc-hwcaps tst-glibc-hwcaps-prepend tst-glibc-hwcaps-mask
# reldep9
tests-internal += loadtest unload unload2 circleload1 \
neededtest neededtest2 neededtest3 neededtest4 \
tst-tls3 tst-tls6 tst-tls7 tst-tls8 tst-dlmopen2 \
tst-ptrguard1 tst-stackguard1 tst-libc_dlvsym \
- tst-create_format1 tst-tls-surplus
+ tst-create_format1 tst-tls-surplus tst-dl-hwcaps_split
tests-container += tst-pldd tst-dlopen-tlsmodid-container \
tst-dlopen-self-container
test-srcs = tst-pathopt
@@ -329,7 +330,10 @@ modules-names = testobj1 testobj2 testobj3 testobj4 testobj5 testobj6 \
tst-single_threaded-mod3 tst-single_threaded-mod4 \
tst-tls-ie-mod0 tst-tls-ie-mod1 tst-tls-ie-mod2 \
tst-tls-ie-mod3 tst-tls-ie-mod4 tst-tls-ie-mod5 \
- tst-tls-ie-mod6
+ tst-tls-ie-mod6 libmarkermod1-1 libmarkermod1-2 libmarkermod1-3 \
+ libmarkermod2-1 libmarkermod2-2 \
+ libmarkermod3-1 libmarkermod3-2 libmarkermod3-3 \
+ libmarkermod4-1 libmarkermod4-2 libmarkermod4-3 libmarkermod4-4 \
# Most modules build with _ISOMAC defined, but those filtered out
# depend on internal headers.
@@ -1812,3 +1816,56 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
'$(test-wrapper-env)' '$(run_program_env)' \
'$(rpath-link)' 'test-argv0' > $@; \
$(evaluate-test)
+
+# Most likely search subdirectories, for each supported architecture.
+# Used to obtain test coverage wide test coverage.
+glibc-hwcaps-first-subdirs-for-tests =
+
+# The test modules are parameterized by preprocessor macros.
+LDFLAGS-libmarkermod1-1.so += -Wl,-soname,libmarkermod1.so
+LDFLAGS-libmarkermod2-1.so += -Wl,-soname,libmarkermod2.so
+LDFLAGS-libmarkermod3-1.so += -Wl,-soname,libmarkermod3.so
+LDFLAGS-libmarkermod4-1.so += -Wl,-soname,libmarkermod4.so
+$(objpfx)libmarkermod%.os : markermodMARKER-VALUE.c
+ $(compile-command.c) \
+ -DMARKER=marker$(firstword $(subst -, ,$*)) \
+ -DVALUE=$(lastword $(subst -, ,$*))
+$(objpfx)libmarkermod1.so: $(objpfx)libmarkermod1-1.so
+ cp $< $@
+$(objpfx)libmarkermod2.so: $(objpfx)libmarkermod2-1.so
+ cp $< $@
+$(objpfx)libmarkermod3.so: $(objpfx)libmarkermod3-1.so
+ cp $< $@
+$(objpfx)libmarkermod4.so: $(objpfx)libmarkermod4-1.so
+ cp $< $@
+
+# tst-glibc-hwcaps-prepend checks that --glibc-hwcaps-prepend is
+# preferred over auto-detected subdirectories.
+$(objpfx)tst-glibc-hwcaps-prepend: $(objpfx)libmarkermod1-1.so
+$(objpfx)glibc-hwcaps/prepend-markermod1/libmarkermod1.so: \
+ $(objpfx)libmarkermod1-2.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/%/libmarkermod1.so: $(objpfx)libmarkermod1-3.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)tst-glibc-hwcaps-prepend.out: \
+ $(objpfx)tst-glibc-hwcaps-prepend $(objpfx)libmarkermod1.so \
+ $(patsubst %,$(objpfx)glibc-hwcaps/%/libmarkermod1.so,prepend-markermod1 \
+ $(glibc-hwcaps-first-subdirs-for-tests))
+ $(test-wrapper) $(rtld-prefix) \
+ --glibc-hwcaps-prepend prepend-markermod1 \
+ $< > $@; \
+ $(evaluate-test)
+
+# tst-glibc-hwcaps-mask checks that --glibc-hwcaps-mask can be used to
+# suppress all auto-detected subdirectories.
+$(objpfx)tst-glibc-hwcaps-mask: $(objpfx)libmarkermod1-1.so
+$(objpfx)tst-glibc-hwcaps-mask.out: \
+ $(objpfx)tst-glibc-hwcaps-mask $(objpfx)libmarkermod1.so \
+ $(patsubst %,$(objpfx)glibc-hwcaps/%/libmarkermod1.so,\
+ $(glibc-hwcaps-first-subdirs-for-tests))
+ $(test-wrapper) $(rtld-prefix) \
+ --glibc-hwcaps-mask does-not-exist \
+ $< > $@; \
+ $(evaluate-test)
diff --git a/elf/dl-hwcaps-subdirs.c b/elf/dl-hwcaps-subdirs.c
new file mode 100644
index 0000000000..60c6d59731
--- /dev/null
+++ b/elf/dl-hwcaps-subdirs.c
@@ -0,0 +1,29 @@
+/* Architecture-specific glibc-hwcaps subdirectories. Generic version.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <dl-hwcaps.h>
+
+/* In the generic version, there are no subdirectories defined. */
+
+const char _dl_hwcaps_subdirs[] = "";
+
+uint32_t
+_dl_hwcaps_subdirs_active (void)
+{
+ return 0;
+}
diff --git a/elf/dl-hwcaps.c b/elf/dl-hwcaps.c
index 44dbac099f..f611f3a1a6 100644
--- a/elf/dl-hwcaps.c
+++ b/elf/dl-hwcaps.c
@@ -26,20 +26,97 @@
#include <dl-procinfo.h>
#include <dl-hwcaps.h>
+/* This is the result of counting the substrings in a colon-separated
+ hwcaps string. */
+struct hwcaps_counts
+{
+ /* Number of substrings. */
+ size_t count;
+
+ /* Sum of the individual substring lengths (without separators or
+ null terminators). */
+ size_t total_length;
+
+ /* Maximum length of an individual substring. */
+ size_t maximum_length;
+};
+
+/* Update *COUNTS according to the contents of HWCAPS. Skip over
+ entries whose bit is not set in MASK. */
+static void
+update_hwcaps_counts (struct hwcaps_counts *counts, const char *hwcaps,
+ uint32_t bitmask, const char *mask)
+{
+ struct dl_hwcaps_split_masked sp;
+ _dl_hwcaps_split_masked_init (&sp, hwcaps, bitmask, mask);
+ while (_dl_hwcaps_split_masked (&sp))
+ {
+ ++counts->count;
+ counts->total_length += sp.split.length;
+ if (sp.split.length > counts->maximum_length)
+ counts->maximum_length = sp.split.length;
+ }
+}
+
+/* State for copy_hwcaps. Must be initialized to point to
+ the storage areas for the array and the strings themselves. */
+struct copy_hwcaps
+{
+ struct r_strlenpair *next_pair;
+ char *next_string;
+};
+
+/* Copy HWCAPS into the string pairs and strings, advancing *TARGET.
+ Skip over entries whose bit is not set in MASK. */
+static void
+copy_hwcaps (struct copy_hwcaps *target, const char *hwcaps,
+ uint32_t bitmask, const char *mask)
+{
+ struct dl_hwcaps_split_masked sp;
+ _dl_hwcaps_split_masked_init (&sp, hwcaps, bitmask, mask);
+ while (_dl_hwcaps_split_masked (&sp))
+ {
+ target->next_pair->str = target->next_string;
+ char *slash = __mempcpy (__mempcpy (target->next_string,
+ GLIBC_HWCAPS_PREFIX,
+ strlen (GLIBC_HWCAPS_PREFIX)),
+ sp.split.segment, sp.split.length);
+ *slash = '/';
+ target->next_pair->len
+ = strlen (GLIBC_HWCAPS_PREFIX) + sp.split.length + 1;
+ ++target->next_pair;
+ target->next_string = slash + 1;
+ }
+}
+
/* Return an array of useful/necessary hardware capability names. */
const struct r_strlenpair *
-_dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
+_dl_important_hwcaps (const char *glibc_hwcaps_prepend,
+ const char *glibc_hwcaps_mask,
+ size_t *sz, size_t *max_capstrlen)
{
uint64_t hwcap_mask = GET_HWCAP_MASK();
/* Determine how many important bits are set. */
uint64_t masked = GLRO(dl_hwcap) & hwcap_mask;
size_t cnt = GLRO (dl_platform) != NULL;
size_t n, m;
- size_t total;
struct r_strlenpair *result;
struct r_strlenpair *rp;
char *cp;
+ /* glibc-hwcaps subdirectories. These are exempted from the power
+ set construction below. */
+ uint32_t hwcaps_subdirs_active = _dl_hwcaps_subdirs_active ();
+ struct hwcaps_counts hwcaps_counts = { 0, };
+ update_hwcaps_counts (&hwcaps_counts, glibc_hwcaps_prepend, -1, NULL);
+ update_hwcaps_counts (&hwcaps_counts, _dl_hwcaps_subdirs,
+ hwcaps_subdirs_active, glibc_hwcaps_mask);
+
+ /* Each hwcaps subdirectory has a GLIBC_HWCAPS_PREFIX string prefix
+ and a "/" suffix once stored in the result. */
+ size_t total = (hwcaps_counts.count * (strlen (GLIBC_HWCAPS_PREFIX) + 1)
+ + hwcaps_counts.total_length);
+
/* Count the number of bits set in the masked value. */
for (n = 0; (~((1ULL << n) - 1) & masked) != 0; ++n)
if ((masked & (1ULL << n)) != 0)
@@ -74,10 +151,10 @@ _dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
/* Determine the total size of all strings together. */
if (cnt == 1)
- total = temp[0].len + 1;
+ total += temp[0].len + 1;
else
{
- total = temp[0].len + temp[cnt - 1].len + 2;
+ total += temp[0].len + temp[cnt - 1].len + 2;
if (cnt > 2)
{
total <<= 1;
@@ -94,26 +171,48 @@ _dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
}
}
- /* The result structure: we use a very compressed way to store the
- various combinations of capability names. */
- *sz = 1 << cnt;
- result = (struct r_strlenpair *) malloc (*sz * sizeof (*result) + total);
- if (result == NULL)
+ *sz = hwcaps_counts.count + (1 << cnt);
+
+ /* This is the overall result, including both glibc-hwcaps
+ subdirectories and the legacy hwcaps subdirectories using the
+ power set construction. */
+ struct r_strlenpair *overall_result
+ = malloc (*sz * sizeof (*result) + total);
+ if (overall_result == NULL)
_dl_signal_error (ENOMEM, NULL, NULL,
N_("cannot create capability list"));
+ /* Fill in the glibc-hwcaps subdirectories. */
+ {
+ struct copy_hwcaps target;
+ target.next_pair = overall_result;
+ target.next_string = (char *) (overall_result + *sz);
+ copy_hwcaps (&target, glibc_hwcaps_prepend, -1, NULL);
+ copy_hwcaps (&target, _dl_hwcaps_subdirs,
+ hwcaps_subdirs_active, glibc_hwcaps_mask);
+ /* Set up the write target for the power set construction. */
+ result = target.next_pair;
+ cp = target.next_string;
+ }
+
+
+ /* Power set construction begins here. We use a very compressed way
+ to store the various combinations of capability names. */
+
if (cnt == 1)
{
- result[0].str = (char *) (result + *sz);
+ result[0].str = cp;
result[0].len = temp[0].len + 1;
- result[1].str = (char *) (result + *sz);
+ result[1].str = cp;
result[1].len = 0;
- cp = __mempcpy ((char *) (result + *sz), temp[0].str, temp[0].len);
+ cp = __mempcpy (cp, temp[0].str, temp[0].len);
*cp = '/';
- *sz = 2;
- *max_capstrlen = result[0].len;
+ if (result[0].len > hwcaps_counts.maximum_length)
+ *max_capstrlen = result[0].len;
+ else
+ *max_capstrlen = hwcaps_counts.maximum_length;
- return result;
+ return overall_result;
}
/* Fill in the information. This follows the following scheme
@@ -124,7 +223,7 @@ _dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
#3: 0, 3 1001
This allows the representation of all possible combinations of
capability names in the string. First generate the strings. */
- result[1].str = result[0].str = cp = (char *) (result + *sz);
+ result[1].str = result[0].str = cp;
#define add(idx) \
cp = __mempcpy (__mempcpy (cp, temp[idx].str, temp[idx].len), "/", 1);
if (cnt == 2)
@@ -191,7 +290,10 @@ _dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
while (--n != 0);
/* The maximum string length. */
- *max_capstrlen = result[0].len;
+ if (result[0].len > hwcaps_counts.maximum_length)
+ *max_capstrlen = result[0].len;
+ else
+ *max_capstrlen = hwcaps_counts.maximum_length;
- return result;
+ return overall_result;
}
diff --git a/elf/dl-hwcaps.h b/elf/dl-hwcaps.h
index b66da59b89..ab39d8a46d 100644
--- a/elf/dl-hwcaps.h
+++ b/elf/dl-hwcaps.h
@@ -16,6 +16,11 @@
License along with the GNU C Library; if not, see
<https://www.gnu.org/licenses/>. */
+#ifndef _DL_HWCAPS_H
+#define _DL_HWCAPS_H
+
+#include <stdint.h>
+
#include <elf/dl-tunables.h>
#if HAVE_TUNABLES
@@ -28,3 +33,103 @@
# define GET_HWCAP_MASK() (0)
# endif
#endif
+
+#define GLIBC_HWCAPS_SUBDIRECTORY "glibc-hwcaps"
+#define GLIBC_HWCAPS_PREFIX GLIBC_HWCAPS_SUBDIRECTORY "/"
+
+/* Used by _dl_hwcaps_split below, to split strings at ':'
+ separators. */
+struct dl_hwcaps_split
+{
+ const char *segment; /* Start of the current segment. */
+ size_t length; /* Number of bytes until ':' or NUL. */
+};
+
+/* Prepare *S to parse SUBJECT, for future _dl_hwcaps_split calls. If
+ SUBJECT is NULL, it is treated as the empty string. */
+static inline void
+_dl_hwcaps_split_init (struct dl_hwcaps_split *s, const char *subject)
+{
+ s->segment = subject;
+ /* The initial call to _dl_hwcaps_split will not skip anything. */
+ s->length = 0;
+}
+
+/* Extract the next non-empty string segment, up to ':' or the null
+ terminator. Return true if one more segment was found, or false if
+ the end of the string was reached. On success, S->segment is the
+ start of the segment found, and S->length is its length.
+ (Typically, S->segment[S->length] is not null.) */
+_Bool _dl_hwcaps_split (struct dl_hwcaps_split *s) attribute_hidden;
+
+/* Similar to dl_hwcaps_split, but with bit-based and name-based
+ masking. */
+struct dl_hwcaps_split_masked
+{
+ struct dl_hwcaps_split split;
+
+ /* For used by the iterator implementation. */
+ const char *mask;
+ uint32_t bitmask;
+};
+
+/* Prepare *S for iteration with _dl_hwcaps_split_masked. Only HWCAP
+ names in SUBJECT whose bit is set in BITMASK and whose name is in
+ MASK will be returned. SUBJECT must not contain empty HWCAP names.
+ If MASK is NULL, no name-based masking is applied. Likewise for
+ BITMASK if BITMASK is -1 (infinite number of bits). */
+static inline void
+_dl_hwcaps_split_masked_init (struct dl_hwcaps_split_masked *s,
+ const char *subject,
+ uint32_t bitmask, const char *mask)
+{
+ _dl_hwcaps_split_init (&s->split, subject);
+ s->bitmask = bitmask;
+ s->mask = mask;
+}
+
+/* Like _dl_hwcaps_split, but apply masking. */
+_Bool _dl_hwcaps_split_masked (struct dl_hwcaps_split_masked *s)
+ attribute_hidden;
+
+/* Returns true if the colon-separated HWCAP list HWCAPS contains the
+ capability NAME (with length NAME_LENGTH). If HWCAPS is NULL, the
+ function returns true. */
+_Bool _dl_hwcaps_contains (const char *hwcaps, const char *name,
+ size_t name_length) attribute_hidden;
+
+/* Colon-separated string of glibc-hwcaps subdirectories, without the
+ "glibc-hwcaps/" prefix. The most preferred subdirectory needs to
+ be listed first. Up to 32 subdirectories are supported, limited by
+ the width of the uint32_t mask. */
+extern const char _dl_hwcaps_subdirs[] attribute_hidden;
+
+/* Returns a bitmap of active subdirectories in _dl_hwcaps_subdirs.
+ Bit 0 (the LSB) corresponds to the first substring in
+ _dl_hwcaps_subdirs, bit 1 to the second substring, and so on.
+ There is no direct correspondence between HWCAP bitmasks and this
+ bitmask. */
+uint32_t _dl_hwcaps_subdirs_active (void) attribute_hidden;
+
+/* Returns a bitmask that marks the last ACTIVE subdirectories in a
+ _dl_hwcaps_subdirs_active string (containing SUBDIRS directories in
+ total) as active. Intended for use in _dl_hwcaps_subdirs_active
+ implementations (if a contiguous tail of the list in
+ _dl_hwcaps_subdirs is selected). */
+static inline uint32_t
+_dl_hwcaps_subdirs_build_bitmask (int subdirs, int active)
+{
+ /* Leading subdirectories that are not active. */
+ int inactive = subdirs - active;
+ if (inactive == 32)
+ return 0;
+
+ uint32_t mask;
+ if (subdirs != 32)
+ mask = (1U << subdirs) - 1;
+ else
+ mask = -1;
+ return mask ^ ((1U << inactive) - 1);
+}
+
+#endif /* _DL_HWCAPS_H */
diff --git a/elf/dl-hwcaps_split.c b/elf/dl-hwcaps_split.c
new file mode 100644
index 0000000000..95225e9f40
--- /dev/null
+++ b/elf/dl-hwcaps_split.c
@@ -0,0 +1,77 @@
+/* Hardware capability support for run-time dynamic loader. String splitting.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <dl-hwcaps.h>
+#include <stdbool.h>
+#include <string.h>
+
+_Bool
+_dl_hwcaps_split (struct dl_hwcaps_split *s)
+{
+ if (s->segment == NULL)
+ return false;
+
+ /* Skip over the previous segment. */
+ s->segment += s->length;
+
+ /* Consume delimiters. This also avoids returning an empty
+ segment. */
+ while (*s->segment == ':')
+ ++s->segment;
+ if (*s->segment == '\0')
+ return false;
+
+ /* This could use strchrnul, but we would have to link the function
+ into ld.so for that. */
+ const char *colon = strchr (s->segment, ':');
+ if (colon == NULL)
+ s->length = strlen (s->segment);
+ else
+ s->length = colon - s->segment;
+ return true;
+}
+
+_Bool
+_dl_hwcaps_split_masked (struct dl_hwcaps_split_masked *s)
+{
+ while (true)
+ {
+ if (!_dl_hwcaps_split (&s->split))
+ return false;
+ bool active = s->bitmask & 1;
+ s->bitmask >>= 1;
+ if (active && _dl_hwcaps_contains (s->mask,
+ s->split.segment, s->split.length))
+ return true;
+ }
+}
+
+_Bool
+_dl_hwcaps_contains (const char *hwcaps, const char *name, size_t name_length)
+{
+ if (hwcaps == NULL)
+ return true;
+
+ struct dl_hwcaps_split split;
+ _dl_hwcaps_split_init (&split, hwcaps);
+ while (_dl_hwcaps_split (&split))
+ if (split.length == name_length
+ && memcmp (split.segment, name, name_length) == 0)
+ return true;
+ return false;
+}
diff --git a/elf/dl-load.c b/elf/dl-load.c
index f3201e7c14..9020f1646f 100644
--- a/elf/dl-load.c
+++ b/elf/dl-load.c
@@ -682,7 +682,9 @@ cache_rpath (struct link_map *l,
void
-_dl_init_paths (const char *llp, const char *source)
+_dl_init_paths (const char *llp, const char *source,
+ const char *glibc_hwcaps_prepend,
+ const char *glibc_hwcaps_mask)
{
size_t idx;
const char *strp;
@@ -697,7 +699,8 @@ _dl_init_paths (const char *llp, const char *source)
#ifdef SHARED
/* Get the capabilities. */
- capstr = _dl_important_hwcaps (&ncapstr, &max_capstrlen);
+ capstr = _dl_important_hwcaps (glibc_hwcaps_prepend, glibc_hwcaps_mask,
+ &ncapstr, &max_capstrlen);
#endif
/* First set up the rest of the default search directory entries. */
diff --git a/elf/dl-main.h b/elf/dl-main.h
index b51256d3b4..566713a0d1 100644
--- a/elf/dl-main.h
+++ b/elf/dl-main.h
@@ -84,6 +84,14 @@ struct dl_main_state
/* The preload list passed as a command argument. */
const char *preloadarg;
+ /* Additional glibc-hwcaps subdirectories to search first.
+ Colon-separated list. */
+ const char *glibc_hwcaps_prepend;
+
+ /* Mask for the internal glibc-hwcaps subdirectories.
+ Colon-separated list. */
+ const char *glibc_hwcaps_mask;
+
enum rtld_mode mode;
/* True if any of the debugging options is enabled. */
@@ -98,7 +106,8 @@ struct dl_main_state
static inline void
call_init_paths (const struct dl_main_state *state)
{
- _dl_init_paths (state->library_path, state->library_path_source);
+ _dl_init_paths (state->library_path, state->library_path_source,
+ state->glibc_hwcaps_prepend, state->glibc_hwcaps_mask);
}
/* Print ld.so usage information and exit. */
diff --git a/elf/dl-support.c b/elf/dl-support.c
index afbc94df54..3264262f4e 100644
--- a/elf/dl-support.c
+++ b/elf/dl-support.c
@@ -323,7 +323,10 @@ _dl_non_dynamic_init (void)
/* Initialize the data structures for the search paths for shared
objects. */
- _dl_init_paths (getenv ("LD_LIBRARY_PATH"), "LD_LIBRARY_PATH");
+ _dl_init_paths (getenv ("LD_LIBRARY_PATH"), "LD_LIBRARY_PATH",
+ /* No glibc-hwcaps selection support in statically
+ linked binaries. */
+ NULL, NULL);
/* Remember the last search directory added at startup. */
_dl_init_all_dirs = GL(dl_all_dirs);
diff --git a/elf/dl-usage.c b/elf/dl-usage.c
index 796ad38b43..e22a9c3942 100644
--- a/elf/dl-usage.c
+++ b/elf/dl-usage.c
@@ -83,7 +83,7 @@ print_search_path_for_help (struct dl_main_state *state)
{
if (__rtld_search_dirs.dirs == NULL)
/* The run-time search paths have not yet been initialized. */
- _dl_init_paths (state->library_path, state->library_path_source);
+ call_init_paths (state);
_dl_printf ("\nShared library search path:\n");
@@ -132,6 +132,67 @@ print_hwcap_1_finish (bool *first)
_dl_printf (")\n");
}
+/* Print the header for print_hwcaps_subdirectories. */
+static void
+print_hwcaps_subdirectories_header (bool *nothing_printed)
+{
+ if (*nothing_printed)
+ {
+ _dl_printf ("\n\
+Subdirectories of glibc-hwcaps directories, in priority order:\n");
+ *nothing_printed = false;
+ }
+}
+
+/* Print the HWCAP name itself, indented. */
+static void
+print_hwcaps_subdirectories_name (const struct dl_hwcaps_split *split)
+{
+ _dl_write (STDOUT_FILENO, " ", 2);
+ _dl_write (STDOUT_FILENO, split->segment, split->length);
+}
+
+/* Print the list of recognized glibc-hwcaps subdirectories. */
+static void
+print_hwcaps_subdirectories (const struct dl_main_state *state)
+{
+ bool nothing_printed = true;
+ struct dl_hwcaps_split split;
+
+ /* The prepended glibc-hwcaps subdirectories. */
+ _dl_hwcaps_split_init (&split, state->glibc_hwcaps_prepend);
+ while (_dl_hwcaps_split (&split))
+ {
+ print_hwcaps_subdirectories_header (¬hing_printed);
+ print_hwcaps_subdirectories_name (&split);
+ bool first = true;
+ print_hwcap_1 (&first, true, "searched");
+ print_hwcap_1_finish (&first);
+ }
+
+ /* The built-in glibc-hwcaps subdirectories. Do the filtering
+ manually, so that more precise diagnostics are possible. */
+ uint32_t mask = _dl_hwcaps_subdirs_active ();
+ _dl_hwcaps_split_init (&split, _dl_hwcaps_subdirs);
+ while (_dl_hwcaps_split (&split))
+ {
+ print_hwcaps_subdirectories_header (¬hing_printed);
+ print_hwcaps_subdirectories_name (&split);
+ bool first = true;
+ print_hwcap_1 (&first, mask & 1, "supported");
+ bool listed = _dl_hwcaps_contains (state->glibc_hwcaps_mask,
+ split.segment, split.length);
+ print_hwcap_1 (&first, !listed, "masked");
+ print_hwcap_1 (&first, (mask & 1) && listed, "searched");
+ print_hwcap_1_finish (&first);
+ mask >>= 1;
+ }
+
+ if (nothing_printed)
+ _dl_printf ("\n\
+No subdirectories of glibc-hwcaps directories are searched.\n");
+}
+
/* Write a list of hwcap subdirectories to standard output. See
_dl_important_hwcaps in dl-hwcaps.c. */
static void
@@ -186,6 +247,10 @@ setting environment variables (which would be inherited by subprocesses).\n\
--inhibit-cache Do not use " LD_SO_CACHE "\n\
--library-path PATH use given PATH instead of content of the environment\n\
variable LD_LIBRARY_PATH\n\
+ --glibc-hwcaps-prepend LIST\n\
+ search glibc-hwcaps subdirectories in LIST\n\
+ --glibc-hwcaps-mask LIST\n\
+ only search built-in subdirectories if in LIST\n\
--inhibit-rpath LIST ignore RUNPATH and RPATH information in object names\n\
in LIST\n\
--audit LIST use objects named in LIST as auditors\n\
@@ -198,6 +263,7 @@ This program interpreter self-identifies as: " RTLD "\n\
",
argv0);
print_search_path_for_help (state);
+ print_hwcaps_subdirectories (state);
print_legacy_hwcap_directories ();
_exit (EXIT_SUCCESS);
}
diff --git a/elf/markermodMARKER-VALUE.c b/elf/markermodMARKER-VALUE.c
new file mode 100644
index 0000000000..99bdcf71a4
--- /dev/null
+++ b/elf/markermodMARKER-VALUE.c
@@ -0,0 +1,29 @@
+/* Source file template for building shared objects with marker functions.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+/* MARKER and VALUE must be set on the compiler command line. */
+
+#ifndef MARKER
+# error MARKER not defined
+#endif
+
+int
+MARKER (void)
+{
+ return VALUE;
+}
diff --git a/elf/rtld.c b/elf/rtld.c
index 5d117d0d2c..e3ec66c030 100644
--- a/elf/rtld.c
+++ b/elf/rtld.c
@@ -289,6 +289,8 @@ dl_main_state_init (struct dl_main_state *state)
state->library_path_source = NULL;
state->preloadlist = NULL;
state->preloadarg = NULL;
+ state->glibc_hwcaps_prepend = NULL;
+ state->glibc_hwcaps_mask = NULL;
state->mode = rtld_mode_normal;
state->any_debug = false;
state->version_info = false;
@@ -1244,6 +1246,22 @@ dl_main (const ElfW(Phdr) *phdr,
{
argv0 = _dl_argv[2];
+ _dl_skip_args += 2;
+ _dl_argc -= 2;
+ _dl_argv += 2;
+ }
+ else if (strcmp (_dl_argv[1], "--glibc-hwcaps-prepend") == 0
+ && _dl_argc > 2)
+ {
+ state.glibc_hwcaps_prepend = _dl_argv[2];
+ _dl_skip_args += 2;
+ _dl_argc -= 2;
+ _dl_argv += 2;
+ }
+ else if (strcmp (_dl_argv[1], "--glibc-hwcaps-mask") == 0
+ && _dl_argc > 2)
+ {
+ state.glibc_hwcaps_mask = _dl_argv[2];
_dl_skip_args += 2;
_dl_argc -= 2;
_dl_argv += 2;
diff --git a/elf/tst-dl-hwcaps_split.c b/elf/tst-dl-hwcaps_split.c
new file mode 100644
index 0000000000..929c99a23b
--- /dev/null
+++ b/elf/tst-dl-hwcaps_split.c
@@ -0,0 +1,139 @@
+/* Unit tests for dl-hwcaps.c.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <array_length.h>
+#include <dl-hwcaps.h>
+#include <string.h>
+#include <support/check.h>
+
+static void
+check_split_masked (const char *input, int32_t bitmask, const char *mask,
+ const char *expected[], size_t expected_length)
+{
+ struct dl_hwcaps_split_masked split;
+ _dl_hwcaps_split_masked_init (&split, input, bitmask, mask);
+ size_t index = 0;
+ while (_dl_hwcaps_split_masked (&split))
+ {
+ TEST_VERIFY_EXIT (index < expected_length);
+ TEST_COMPARE_BLOB (expected[index], strlen (expected[index]),
+ split.split.segment, split.split.length);
+ ++index;
+ }
+ TEST_COMPARE (index, expected_length);
+}
+
+static void
+check_split (const char *input,
+ const char *expected[], size_t expected_length)
+{
+ struct dl_hwcaps_split split;
+ _dl_hwcaps_split_init (&split, input);
+ size_t index = 0;
+ while (_dl_hwcaps_split (&split))
+ {
+ TEST_VERIFY_EXIT (index < expected_length);
+ TEST_COMPARE_BLOB (expected[index], strlen (expected[index]),
+ split.segment, split.length);
+ ++index;
+ }
+ TEST_COMPARE (index, expected_length);
+
+ /* Reuse the test cases with masking that does not actually remove
+ anything. */
+ check_split_masked (input, -1, NULL, expected, expected_length);
+ check_split_masked (input, -1, input, expected, expected_length);
+}
+
+static int
+do_test (void)
+{
+ /* Splitting tests, without masking. */
+ check_split (NULL, NULL, 0);
+ check_split ("", NULL, 0);
+ check_split (":", NULL, 0);
+ check_split ("::", NULL, 0);
+
+ {
+ const char *expected[] = { "first" };
+ check_split ("first", expected, array_length (expected));
+ check_split (":first", expected, array_length (expected));
+ check_split ("first:", expected, array_length (expected));
+ check_split (":first:", expected, array_length (expected));
+ }
+
+ {
+ const char *expected[] = { "first", "second" };
+ check_split ("first:second", expected, array_length (expected));
+ check_split ("first::second", expected, array_length (expected));
+ check_split (":first:second", expected, array_length (expected));
+ check_split ("first:second:", expected, array_length (expected));
+ check_split (":first:second:", expected, array_length (expected));
+ }
+
+ /* Splitting tests with masking. */
+ {
+ const char *expected[] = { "first" };
+ check_split_masked ("first", 3, "first:second",
+ expected, array_length (expected));
+ check_split_masked ("first:second", 3, "first:",
+ expected, array_length (expected));
+ check_split_masked ("first:second", 1, NULL,
+ expected, array_length (expected));
+ }
+ {
+ const char *expected[] = { "second" };
+ check_split_masked ("first:second", 3, "second",
+ expected, array_length (expected));
+ check_split_masked ("first:second:third", -1, "second:",
+ expected, array_length (expected));
+ check_split_masked ("first:second", 2, NULL,
+ expected, array_length (expected));
+ check_split_masked ("first:second:third", 2, "first:second",
+ expected, array_length (expected));
+ }
+
+ /* Tests for _dl_hwcaps_contains. */
+ TEST_VERIFY (_dl_hwcaps_contains (NULL, "first", strlen ("first")));
+ TEST_VERIFY (_dl_hwcaps_contains (NULL, "", 0));
+ TEST_VERIFY (! _dl_hwcaps_contains ("", "first", strlen ("first")));
+ TEST_VERIFY (! _dl_hwcaps_contains ("firs", "first", strlen ("first")));
+ TEST_VERIFY (_dl_hwcaps_contains ("firs", "first", strlen ("first") - 1));
+ for (int i = 0; i < strlen ("first"); ++i)
+ TEST_VERIFY (! _dl_hwcaps_contains ("first", "first", i));
+ TEST_VERIFY (_dl_hwcaps_contains ("first", "first", strlen ("first")));
+ TEST_VERIFY (_dl_hwcaps_contains ("first:", "first", strlen ("first")));
+ TEST_VERIFY (_dl_hwcaps_contains ("first:second",
+ "first", strlen ("first")));
+ TEST_VERIFY (_dl_hwcaps_contains (":first:second", "first",
+ strlen ("first")));
+ TEST_VERIFY (_dl_hwcaps_contains ("first:second", "second",
+ strlen ("second")));
+ TEST_VERIFY (_dl_hwcaps_contains ("first:second:", "second",
+ strlen ("second")));
+ for (int i = 0; i < strlen ("second"); ++i)
+ TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "sec", i));
+
+ return 0;
+}
+
+#include <support/test-driver.c>
+
+/* Rebuild the sources here because the object file is built for
+ inclusion into the dynamic loader. */
+#include "dl-hwcaps_split.c"
diff --git a/elf/tst-glibc-hwcaps-mask.c b/elf/tst-glibc-hwcaps-mask.c
new file mode 100644
index 0000000000..27b09b358c
--- /dev/null
+++ b/elf/tst-glibc-hwcaps-mask.c
@@ -0,0 +1,31 @@
+/* Test that --glibc-hwcaps-mask works.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <support/check.h>
+
+extern int marker1 (void);
+
+static int
+do_test (void)
+{
+ /* The marker1 function in elf/markermod1.so returns 1. */
+ TEST_COMPARE (marker1 (), 1);
+ return 0;
+}
+
+#include <support/test-driver.c>
diff --git a/elf/tst-glibc-hwcaps-prepend.c b/elf/tst-glibc-hwcaps-prepend.c
new file mode 100644
index 0000000000..57d7319f14
--- /dev/null
+++ b/elf/tst-glibc-hwcaps-prepend.c
@@ -0,0 +1,32 @@
+/* Test that --glibc-hwcaps-prepend works.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <support/check.h>
+
+extern int marker1 (void);
+
+static int
+do_test (void)
+{
+ /* The marker1 function in
+ glibc-hwcaps/prepend-markermod1/markermod1.so returns 2. */
+ TEST_COMPARE (marker1 (), 2);
+ return 0;
+}
+
+#include <support/test-driver.c>
diff --git a/elf/tst-glibc-hwcaps.c b/elf/tst-glibc-hwcaps.c
new file mode 100644
index 0000000000..28f47cf891
--- /dev/null
+++ b/elf/tst-glibc-hwcaps.c
@@ -0,0 +1,28 @@
+/* Stub test for glibc-hwcaps.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <stdio.h>
+
+static int
+do_test (void)
+{
+ puts ("info: generic tst-glibc-hwcaps (tests nothing)");
+ return 0;
+}
+
+#include <support/test-driver.c>
diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h
index 382eeb9be0..0b2babc70c 100644
--- a/sysdeps/generic/ldsodefs.h
+++ b/sysdeps/generic/ldsodefs.h
@@ -1047,8 +1047,13 @@ extern struct r_debug *_dl_debug_initialize (ElfW(Addr) ldbase, Lmid_t ns)
attribute_hidden;
/* Initialize the basic data structure for the search paths. SOURCE
- is either "LD_LIBRARY_PATH" or "--library-path". */
-extern void _dl_init_paths (const char *library_path, const char *source)
+ is either "LD_LIBRARY_PATH" or "--library-path".
+ GLIBC_HWCAPS_PREPEND adds additional glibc-hwcaps subdirectories to
+ search. GLIBC_HWCAPS_MASK is used to filter the built-in
+ subdirectories if not NULL. */
+extern void _dl_init_paths (const char *library_path, const char *source,
+ const char *glibc_hwcaps_prepend,
+ const char *glibc_hwcaps_mask)
attribute_hidden;
/* Gather the information needed to install the profiling tables and start
@@ -1072,9 +1077,14 @@ extern void _dl_show_auxv (void) attribute_hidden;
extern char *_dl_next_ld_env_entry (char ***position) attribute_hidden;
/* Return an array with the names of the important hardware
- capabilities. The length of the array is written to *SZ, and the
- maximum of all strings length is written to *MAX_CAPSTRLEN. */
-const struct r_strlenpair *_dl_important_hwcaps (size_t *sz,
+ capabilities. PREPEND is a colon-separated list of glibc-hwcaps
+ directories to search first. MASK is a colon-separated list used
+ to filter the built-in glibc-hwcaps subdirectories. The length of
+ the array is written to *SZ, and the maximum of all strings length
+ is written to *MAX_CAPSTRLEN. */
+const struct r_strlenpair *_dl_important_hwcaps (const char *prepend,
+ const char *mask,
+ size_t *sz,
size_t *max_capstrlen)
attribute_hidden;
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
2020-11-09 18:40 ` [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH Florian Weimer
@ 2020-11-26 20:29 ` Adhemerval Zanella
2020-11-27 19:49 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-26 20:29 UTC (permalink / raw)
To: libc-alpha
On 09/11/2020 15:40, Florian Weimer via Libc-alpha wrote:
> This hacks non-power-set processing into _dl_important_hwcaps.
> Once the legacy hwcaps handling goes away, the subdirectory
> handling needs to be reworked, but it is premature to do this
> while both approaches are still supported.
Could you extend the commit message with the extra options you are adding
on ld.so and their expected semantic? I would be good to have some
documentation at least on the commit message.
Patch looks goods in general, some nits below.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
> elf/Makefile | 67 ++++++++++++++--
> elf/dl-hwcaps-subdirs.c | 29 +++++++
> elf/dl-hwcaps.c | 138 +++++++++++++++++++++++++++-----
> elf/dl-hwcaps.h | 105 +++++++++++++++++++++++++
> elf/dl-hwcaps_split.c | 77 ++++++++++++++++++
> elf/dl-load.c | 7 +-
> elf/dl-main.h | 11 ++-
> elf/dl-support.c | 5 +-
> elf/dl-usage.c | 68 +++++++++++++++-
> elf/markermodMARKER-VALUE.c | 29 +++++++
> elf/rtld.c | 18 +++++
> elf/tst-dl-hwcaps_split.c | 139 +++++++++++++++++++++++++++++++++
> elf/tst-glibc-hwcaps-mask.c | 31 ++++++++
> elf/tst-glibc-hwcaps-prepend.c | 32 ++++++++
> elf/tst-glibc-hwcaps.c | 28 +++++++
> sysdeps/generic/ldsodefs.h | 20 +++--
> 16 files changed, 771 insertions(+), 33 deletions(-)
> create mode 100644 elf/dl-hwcaps-subdirs.c
> create mode 100644 elf/dl-hwcaps_split.c
> create mode 100644 elf/markermodMARKER-VALUE.c
> create mode 100644 elf/tst-dl-hwcaps_split.c
> create mode 100644 elf/tst-glibc-hwcaps-mask.c
> create mode 100644 elf/tst-glibc-hwcaps-prepend.c
> create mode 100644 elf/tst-glibc-hwcaps.c
>
> diff --git a/elf/Makefile b/elf/Makefile
> index f10cc59e7c..d2f7e99863 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -59,7 +59,8 @@ elide-routines.os = $(all-dl-routines) dl-support enbl-secure dl-origin \
> # ld.so uses those routines, plus some special stuff for being the program
> # interpreter and operating independent of libc.
> rtld-routines = rtld $(all-dl-routines) dl-sysdep dl-environ dl-minimal \
> - dl-error-minimal dl-conflict dl-hwcaps dl-usage
> + dl-error-minimal dl-conflict dl-hwcaps dl-hwcaps_split dl-hwcaps-subdirs \
> + dl-usage
> all-rtld-routines = $(rtld-routines) $(sysdep-rtld-routines)
>
> CFLAGS-dl-runtime.c += -fexceptions -fasynchronous-unwind-tables
Ok.
> @@ -210,14 +211,14 @@ tests += restest1 preloadtest loadfail multiload origtest resolvfail \
> tst-filterobj tst-filterobj-dlopen tst-auxobj tst-auxobj-dlopen \
> tst-audit14 tst-audit15 tst-audit16 \
> tst-single_threaded tst-single_threaded-pthread \
> - tst-tls-ie tst-tls-ie-dlmopen \
> - argv0test
> + tst-tls-ie tst-tls-ie-dlmopen argv0test \
> + tst-glibc-hwcaps tst-glibc-hwcaps-prepend tst-glibc-hwcaps-mask
> # reldep9
> tests-internal += loadtest unload unload2 circleload1 \
> neededtest neededtest2 neededtest3 neededtest4 \
> tst-tls3 tst-tls6 tst-tls7 tst-tls8 tst-dlmopen2 \
> tst-ptrguard1 tst-stackguard1 tst-libc_dlvsym \
> - tst-create_format1 tst-tls-surplus
> + tst-create_format1 tst-tls-surplus tst-dl-hwcaps_split
> tests-container += tst-pldd tst-dlopen-tlsmodid-container \
> tst-dlopen-self-container
> test-srcs = tst-pathopt
Ok.
> @@ -329,7 +330,10 @@ modules-names = testobj1 testobj2 testobj3 testobj4 testobj5 testobj6 \
> tst-single_threaded-mod3 tst-single_threaded-mod4 \
> tst-tls-ie-mod0 tst-tls-ie-mod1 tst-tls-ie-mod2 \
> tst-tls-ie-mod3 tst-tls-ie-mod4 tst-tls-ie-mod5 \
> - tst-tls-ie-mod6
> + tst-tls-ie-mod6 libmarkermod1-1 libmarkermod1-2 libmarkermod1-3 \
> + libmarkermod2-1 libmarkermod2-2 \
> + libmarkermod3-1 libmarkermod3-2 libmarkermod3-3 \
> + libmarkermod4-1 libmarkermod4-2 libmarkermod4-3 libmarkermod4-4 \
>
> # Most modules build with _ISOMAC defined, but those filtered out
> # depend on internal headers.
Ok.
> @@ -1812,3 +1816,56 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
> '$(test-wrapper-env)' '$(run_program_env)' \
> '$(rpath-link)' 'test-argv0' > $@; \
> $(evaluate-test)
> +
> +# Most likely search subdirectories, for each supported architecture.
> +# Used to obtain test coverage wide test coverage.
This comments sounds strange, specially the second line.
> +glibc-hwcaps-first-subdirs-for-tests =
> +
> +# The test modules are parameterized by preprocessor macros.
> +LDFLAGS-libmarkermod1-1.so += -Wl,-soname,libmarkermod1.so
> +LDFLAGS-libmarkermod2-1.so += -Wl,-soname,libmarkermod2.so
> +LDFLAGS-libmarkermod3-1.so += -Wl,-soname,libmarkermod3.so
> +LDFLAGS-libmarkermod4-1.so += -Wl,-soname,libmarkermod4.so
> +$(objpfx)libmarkermod%.os : markermodMARKER-VALUE.c
> + $(compile-command.c) \
> + -DMARKER=marker$(firstword $(subst -, ,$*)) \
> + -DVALUE=$(lastword $(subst -, ,$*))
> +$(objpfx)libmarkermod1.so: $(objpfx)libmarkermod1-1.so
> + cp $< $@
> +$(objpfx)libmarkermod2.so: $(objpfx)libmarkermod2-1.so
> + cp $< $@
> +$(objpfx)libmarkermod3.so: $(objpfx)libmarkermod3-1.so
> + cp $< $@
> +$(objpfx)libmarkermod4.so: $(objpfx)libmarkermod4-1.so
> + cp $< $@
> +
Ok.
> +# tst-glibc-hwcaps-prepend checks that --glibc-hwcaps-prepend is
> +# preferred over auto-detected subdirectories.
> +$(objpfx)tst-glibc-hwcaps-prepend: $(objpfx)libmarkermod1-1.so
> +$(objpfx)glibc-hwcaps/prepend-markermod1/libmarkermod1.so: \
> + $(objpfx)libmarkermod1-2.so
> + $(make-target-directory)
> + cp $< $@
> +$(objpfx)glibc-hwcaps/%/libmarkermod1.so: $(objpfx)libmarkermod1-3.so
> + $(make-target-directory)
> + cp $< $@
> +$(objpfx)tst-glibc-hwcaps-prepend.out: \
> + $(objpfx)tst-glibc-hwcaps-prepend $(objpfx)libmarkermod1.so \
> + $(patsubst %,$(objpfx)glibc-hwcaps/%/libmarkermod1.so,prepend-markermod1 \
> + $(glibc-hwcaps-first-subdirs-for-tests))
> + $(test-wrapper) $(rtld-prefix) \
> + --glibc-hwcaps-prepend prepend-markermod1 \
> + $< > $@; \
> + $(evaluate-test)
> +
Ok.
> +# tst-glibc-hwcaps-mask checks that --glibc-hwcaps-mask can be used to
> +# suppress all auto-detected subdirectories.
> +$(objpfx)tst-glibc-hwcaps-mask: $(objpfx)libmarkermod1-1.so
> +$(objpfx)tst-glibc-hwcaps-mask.out: \
> + $(objpfx)tst-glibc-hwcaps-mask $(objpfx)libmarkermod1.so \
> + $(patsubst %,$(objpfx)glibc-hwcaps/%/libmarkermod1.so,\
> + $(glibc-hwcaps-first-subdirs-for-tests))
> + $(test-wrapper) $(rtld-prefix) \
> + --glibc-hwcaps-mask does-not-exist \
> + $< > $@; \
> + $(evaluate-test)
Ok.
> diff --git a/elf/dl-hwcaps-subdirs.c b/elf/dl-hwcaps-subdirs.c
> new file mode 100644
> index 0000000000..60c6d59731
> --- /dev/null
> +++ b/elf/dl-hwcaps-subdirs.c
> @@ -0,0 +1,29 @@
> +/* Architecture-specific glibc-hwcaps subdirectories. Generic version.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <dl-hwcaps.h>
> +
> +/* In the generic version, there are no subdirectories defined. */
> +
> +const char _dl_hwcaps_subdirs[] = "";
> +
> +uint32_t
> +_dl_hwcaps_subdirs_active (void)
> +{
> + return 0;
> +}
Ok.
> diff --git a/elf/dl-hwcaps.c b/elf/dl-hwcaps.c
> index 44dbac099f..f611f3a1a6 100644
> --- a/elf/dl-hwcaps.c
> +++ b/elf/dl-hwcaps.c
> @@ -26,20 +26,97 @@
> #include <dl-procinfo.h>
> #include <dl-hwcaps.h>
>
> +/* This is the result of counting the substrings in a colon-separated
> + hwcaps string. */
> +struct hwcaps_counts
> +{
> + /* Number of substrings. */
> + size_t count;
> +
> + /* Sum of the individual substring lengths (without separators or
> + null terminators). */
> + size_t total_length;
> +
> + /* Maximum length of an individual substring. */
> + size_t maximum_length;
> +};
> +
Ok.
> +/* Update *COUNTS according to the contents of HWCAPS. Skip over
> + entries whose bit is not set in MASK. */
> +static void
> +update_hwcaps_counts (struct hwcaps_counts *counts, const char *hwcaps,
> + uint32_t bitmask, const char *mask)
> +{
> + struct dl_hwcaps_split_masked sp;
> + _dl_hwcaps_split_masked_init (&sp, hwcaps, bitmask, mask);
> + while (_dl_hwcaps_split_masked (&sp))
> + {
> + ++counts->count;
> + counts->total_length += sp.split.length;
> + if (sp.split.length > counts->maximum_length)
> + counts->maximum_length = sp.split.length;
> + }
> +}
> +
Ok.
> +/* State for copy_hwcaps. Must be initialized to point to
> + the storage areas for the array and the strings themselves. */
> +struct copy_hwcaps
> +{
> + struct r_strlenpair *next_pair;
> + char *next_string;
> +};
> +
> +/* Copy HWCAPS into the string pairs and strings, advancing *TARGET.
> + Skip over entries whose bit is not set in MASK. */
> +static void
> +copy_hwcaps (struct copy_hwcaps *target, const char *hwcaps,
> + uint32_t bitmask, const char *mask)
> +{
> + struct dl_hwcaps_split_masked sp;
> + _dl_hwcaps_split_masked_init (&sp, hwcaps, bitmask, mask);
> + while (_dl_hwcaps_split_masked (&sp))
> + {
> + target->next_pair->str = target->next_string;
> + char *slash = __mempcpy (__mempcpy (target->next_string,
> + GLIBC_HWCAPS_PREFIX,
> + strlen (GLIBC_HWCAPS_PREFIX)),
> + sp.split.segment, sp.split.length);
> + *slash = '/';
> + target->next_pair->len
> + = strlen (GLIBC_HWCAPS_PREFIX) + sp.split.length + 1;
> + ++target->next_pair;
> + target->next_string = slash + 1;
> + }
> +}
> +
Ok.
> /* Return an array of useful/necessary hardware capability names. */
> const struct r_strlenpair *
> -_dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
> +_dl_important_hwcaps (const char *glibc_hwcaps_prepend,
> + const char *glibc_hwcaps_mask,
> + size_t *sz, size_t *max_capstrlen)
> {
> uint64_t hwcap_mask = GET_HWCAP_MASK();
> /* Determine how many important bits are set. */
> uint64_t masked = GLRO(dl_hwcap) & hwcap_mask;
> size_t cnt = GLRO (dl_platform) != NULL;
> size_t n, m;
> - size_t total;
> struct r_strlenpair *result;
> struct r_strlenpair *rp;
> char *cp;
>
> + /* glibc-hwcaps subdirectories. These are exempted from the power
> + set construction below. */
> + uint32_t hwcaps_subdirs_active = _dl_hwcaps_subdirs_active ();
> + struct hwcaps_counts hwcaps_counts = { 0, };
> + update_hwcaps_counts (&hwcaps_counts, glibc_hwcaps_prepend, -1, NULL);
> + update_hwcaps_counts (&hwcaps_counts, _dl_hwcaps_subdirs,
> + hwcaps_subdirs_active, glibc_hwcaps_mask);
> +
> + /* Each hwcaps subdirectory has a GLIBC_HWCAPS_PREFIX string prefix
> + and a "/" suffix once stored in the result. */
> + size_t total = (hwcaps_counts.count * (strlen (GLIBC_HWCAPS_PREFIX) + 1)
> + + hwcaps_counts.total_length);
> +
> /* Count the number of bits set in the masked value. */
> for (n = 0; (~((1ULL << n) - 1) & masked) != 0; ++n)
> if ((masked & (1ULL << n)) != 0)
Ok.
> @@ -74,10 +151,10 @@ _dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
>
> /* Determine the total size of all strings together. */
> if (cnt == 1)
> - total = temp[0].len + 1;
> + total += temp[0].len + 1;
> else
> {
> - total = temp[0].len + temp[cnt - 1].len + 2;
> + total += temp[0].len + temp[cnt - 1].len + 2;
> if (cnt > 2)
> {
> total <<= 1;
Ok.
> @@ -94,26 +171,48 @@ _dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
> }
> }
>
> - /* The result structure: we use a very compressed way to store the
> - various combinations of capability names. */
> - *sz = 1 << cnt;
> - result = (struct r_strlenpair *) malloc (*sz * sizeof (*result) + total);
> - if (result == NULL)
> + *sz = hwcaps_counts.count + (1 << cnt);
> +
> + /* This is the overall result, including both glibc-hwcaps
> + subdirectories and the legacy hwcaps subdirectories using the
> + power set construction. */
> + struct r_strlenpair *overall_result
> + = malloc (*sz * sizeof (*result) + total);
> + if (overall_result == NULL)
> _dl_signal_error (ENOMEM, NULL, NULL,
> N_("cannot create capability list"));
>
> + /* Fill in the glibc-hwcaps subdirectories. */
> + {
> + struct copy_hwcaps target;
> + target.next_pair = overall_result;
> + target.next_string = (char *) (overall_result + *sz);
> + copy_hwcaps (&target, glibc_hwcaps_prepend, -1, NULL);
> + copy_hwcaps (&target, _dl_hwcaps_subdirs,
> + hwcaps_subdirs_active, glibc_hwcaps_mask);
> + /* Set up the write target for the power set construction. */
> + result = target.next_pair;
> + cp = target.next_string;
> + }
> +
> +
> + /* Power set construction begins here. We use a very compressed way
> + to store the various combinations of capability names. */
> +
> if (cnt == 1)
> {
> - result[0].str = (char *) (result + *sz);
> + result[0].str = cp;
> result[0].len = temp[0].len + 1;
> - result[1].str = (char *) (result + *sz);
> + result[1].str = cp;
> result[1].len = 0;
> - cp = __mempcpy ((char *) (result + *sz), temp[0].str, temp[0].len);
> + cp = __mempcpy (cp, temp[0].str, temp[0].len);
> *cp = '/';
> - *sz = 2;
> - *max_capstrlen = result[0].len;
> + if (result[0].len > hwcaps_counts.maximum_length)
> + *max_capstrlen = result[0].len;
> + else
> + *max_capstrlen = hwcaps_counts.maximum_length;
>
> - return result;
> + return overall_result;
> }
>
> /* Fill in the information. This follows the following scheme
Ok.
> @@ -124,7 +223,7 @@ _dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
> #3: 0, 3 1001
> This allows the representation of all possible combinations of
> capability names in the string. First generate the strings. */
> - result[1].str = result[0].str = cp = (char *) (result + *sz);
> + result[1].str = result[0].str = cp;
> #define add(idx) \
> cp = __mempcpy (__mempcpy (cp, temp[idx].str, temp[idx].len), "/", 1);
> if (cnt == 2)
> @@ -191,7 +290,10 @@ _dl_important_hwcaps (size_t *sz, size_t *max_capstrlen)
> while (--n != 0);
>
> /* The maximum string length. */
> - *max_capstrlen = result[0].len;
> + if (result[0].len > hwcaps_counts.maximum_length)
> + *max_capstrlen = result[0].len;
> + else
> + *max_capstrlen = hwcaps_counts.maximum_length;
>
> - return result;
> + return overall_result;
> }
Ok.
> diff --git a/elf/dl-hwcaps.h b/elf/dl-hwcaps.h
> index b66da59b89..ab39d8a46d 100644
> --- a/elf/dl-hwcaps.h
> +++ b/elf/dl-hwcaps.h
> @@ -16,6 +16,11 @@
> License along with the GNU C Library; if not, see
> <https://www.gnu.org/licenses/>. */
>
> +#ifndef _DL_HWCAPS_H
> +#define _DL_HWCAPS_H
> +
> +#include <stdint.h>
> +
> #include <elf/dl-tunables.h>
>
> #if HAVE_TUNABLES
> @@ -28,3 +33,103 @@
> # define GET_HWCAP_MASK() (0)
> # endif
> #endif
> +
> +#define GLIBC_HWCAPS_SUBDIRECTORY "glibc-hwcaps"
> +#define GLIBC_HWCAPS_PREFIX GLIBC_HWCAPS_SUBDIRECTORY "/"
> +
> +/* Used by _dl_hwcaps_split below, to split strings at ':'
> + separators. */
> +struct dl_hwcaps_split
> +{
> + const char *segment; /* Start of the current segment. */
> + size_t length; /* Number of bytes until ':' or NUL. */
> +};
> +
> +/* Prepare *S to parse SUBJECT, for future _dl_hwcaps_split calls. If
> + SUBJECT is NULL, it is treated as the empty string. */
> +static inline void
> +_dl_hwcaps_split_init (struct dl_hwcaps_split *s, const char *subject)
> +{
> + s->segment = subject;
> + /* The initial call to _dl_hwcaps_split will not skip anything. */
> + s->length = 0;
> +}
Ok.
> +
> +/* Extract the next non-empty string segment, up to ':' or the null
> + terminator. Return true if one more segment was found, or false if
> + the end of the string was reached. On success, S->segment is the
> + start of the segment found, and S->length is its length.
> + (Typically, S->segment[S->length] is not null.) */
> +_Bool _dl_hwcaps_split (struct dl_hwcaps_split *s) attribute_hidden;
> +
> +/* Similar to dl_hwcaps_split, but with bit-based and name-based
> + masking. */
> +struct dl_hwcaps_split_masked
> +{
> + struct dl_hwcaps_split split;
> +
> + /* For used by the iterator implementation. */
> + const char *mask;
> + uint32_t bitmask;
> +};
> +
> +/* Prepare *S for iteration with _dl_hwcaps_split_masked. Only HWCAP
> + names in SUBJECT whose bit is set in BITMASK and whose name is in
> + MASK will be returned. SUBJECT must not contain empty HWCAP names.
> + If MASK is NULL, no name-based masking is applied. Likewise for
> + BITMASK if BITMASK is -1 (infinite number of bits). */
> +static inline void
> +_dl_hwcaps_split_masked_init (struct dl_hwcaps_split_masked *s,
> + const char *subject,
> + uint32_t bitmask, const char *mask)
> +{
> + _dl_hwcaps_split_init (&s->split, subject);
> + s->bitmask = bitmask;
> + s->mask = mask;
> +}
> +
Ok.
> +/* Like _dl_hwcaps_split, but apply masking. */
> +_Bool _dl_hwcaps_split_masked (struct dl_hwcaps_split_masked *s)
> + attribute_hidden;
> +
> +/* Returns true if the colon-separated HWCAP list HWCAPS contains the
> + capability NAME (with length NAME_LENGTH). If HWCAPS is NULL, the
> + function returns true. */
> +_Bool _dl_hwcaps_contains (const char *hwcaps, const char *name,
> + size_t name_length) attribute_hidden;
I find it confusing that an null hwcap result true while an empty
relies on name. Do we need that NULL to be handled special here?
> +
> +/* Colon-separated string of glibc-hwcaps subdirectories, without the
> + "glibc-hwcaps/" prefix. The most preferred subdirectory needs to
> + be listed first. Up to 32 subdirectories are supported, limited by
> + the width of the uint32_t mask. */
> +extern const char _dl_hwcaps_subdirs[] attribute_hidden;
> +
> +/* Returns a bitmap of active subdirectories in _dl_hwcaps_subdirs.
> + Bit 0 (the LSB) corresponds to the first substring in
> + _dl_hwcaps_subdirs, bit 1 to the second substring, and so on.
> + There is no direct correspondence between HWCAP bitmasks and this
> + bitmask. */
> +uint32_t _dl_hwcaps_subdirs_active (void) attribute_hidden;
> +
> +/* Returns a bitmask that marks the last ACTIVE subdirectories in a
> + _dl_hwcaps_subdirs_active string (containing SUBDIRS directories in
> + total) as active. Intended for use in _dl_hwcaps_subdirs_active
> + implementations (if a contiguous tail of the list in
> + _dl_hwcaps_subdirs is selected). */
> +static inline uint32_t
> +_dl_hwcaps_subdirs_build_bitmask (int subdirs, int active)
> +{
Should it any assert to check if subdirs and/or active is within the
expected range?
> + /* Leading subdirectories that are not active. */
> + int inactive = subdirs - active;
> + if (inactive == 32)
> + return 0;
> +
> + uint32_t mask;
> + if (subdirs != 32)
> + mask = (1U << subdirs) - 1;
> + else
> + mask = -1;
> + return mask ^ ((1U << inactive) - 1);
> +}
> +
> +#endif /* _DL_HWCAPS_H */
> diff --git a/elf/dl-hwcaps_split.c b/elf/dl-hwcaps_split.c
> new file mode 100644
> index 0000000000..95225e9f40
> --- /dev/null
> +++ b/elf/dl-hwcaps_split.c
> @@ -0,0 +1,77 @@
> +/* Hardware capability support for run-time dynamic loader. String splitting.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <dl-hwcaps.h>
> +#include <stdbool.h>
> +#include <string.h>
> +
> +_Bool
> +_dl_hwcaps_split (struct dl_hwcaps_split *s)
> +{
> + if (s->segment == NULL)
> + return false;
> +
> + /* Skip over the previous segment. */
> + s->segment += s->length;
> +
> + /* Consume delimiters. This also avoids returning an empty
> + segment. */
> + while (*s->segment == ':')
> + ++s->segment;
> + if (*s->segment == '\0')
> + return false;
> +
> + /* This could use strchrnul, but we would have to link the function
> + into ld.so for that. */
> + const char *colon = strchr (s->segment, ':');
> + if (colon == NULL)
> + s->length = strlen (s->segment);
> + else
> + s->length = colon - s->segment;
> + return true;
> +}
> +
Ok.
> +_Bool
> +_dl_hwcaps_split_masked (struct dl_hwcaps_split_masked *s)
> +{
> + while (true)
> + {
> + if (!_dl_hwcaps_split (&s->split))
> + return false;
> + bool active = s->bitmask & 1;
> + s->bitmask >>= 1;
> + if (active && _dl_hwcaps_contains (s->mask,
> + s->split.segment, s->split.length))
> + return true;
> + }
> +}
> +
Ok.
> +_Bool
> +_dl_hwcaps_contains (const char *hwcaps, const char *name, size_t name_length)
> +{
> + if (hwcaps == NULL)
> + return true;
> +
> + struct dl_hwcaps_split split;
> + _dl_hwcaps_split_init (&split, hwcaps);
> + while (_dl_hwcaps_split (&split))
> + if (split.length == name_length
> + && memcmp (split.segment, name, name_length) == 0)
> + return true;
> + return false;
> +}
Ok.
> diff --git a/elf/dl-load.c b/elf/dl-load.c
> index f3201e7c14..9020f1646f 100644
> --- a/elf/dl-load.c
> +++ b/elf/dl-load.c
> @@ -682,7 +682,9 @@ cache_rpath (struct link_map *l,
>
>
> void
> -_dl_init_paths (const char *llp, const char *source)
> +_dl_init_paths (const char *llp, const char *source,
> + const char *glibc_hwcaps_prepend,
> + const char *glibc_hwcaps_mask)
> {
> size_t idx;
> const char *strp;
> @@ -697,7 +699,8 @@ _dl_init_paths (const char *llp, const char *source)
>
> #ifdef SHARED
> /* Get the capabilities. */
> - capstr = _dl_important_hwcaps (&ncapstr, &max_capstrlen);
> + capstr = _dl_important_hwcaps (glibc_hwcaps_prepend, glibc_hwcaps_mask,
> + &ncapstr, &max_capstrlen);
> #endif
>
> /* First set up the rest of the default search directory entries. */
Ok.
> diff --git a/elf/dl-main.h b/elf/dl-main.h
> index b51256d3b4..566713a0d1 100644
> --- a/elf/dl-main.h
> +++ b/elf/dl-main.h
> @@ -84,6 +84,14 @@ struct dl_main_state
> /* The preload list passed as a command argument. */
> const char *preloadarg;
>
> + /* Additional glibc-hwcaps subdirectories to search first.
> + Colon-separated list. */
> + const char *glibc_hwcaps_prepend;
> +
> + /* Mask for the internal glibc-hwcaps subdirectories.
> + Colon-separated list. */
> + const char *glibc_hwcaps_mask;
> +
> enum rtld_mode mode;
>
> /* True if any of the debugging options is enabled. */
Ok.
> @@ -98,7 +106,8 @@ struct dl_main_state
> static inline void
> call_init_paths (const struct dl_main_state *state)
> {
> - _dl_init_paths (state->library_path, state->library_path_source);
> + _dl_init_paths (state->library_path, state->library_path_source,
> + state->glibc_hwcaps_prepend, state->glibc_hwcaps_mask);
> }
>
> /* Print ld.so usage information and exit. */
Ok.
> diff --git a/elf/dl-support.c b/elf/dl-support.c
> index afbc94df54..3264262f4e 100644
> --- a/elf/dl-support.c
> +++ b/elf/dl-support.c
> @@ -323,7 +323,10 @@ _dl_non_dynamic_init (void)
>
> /* Initialize the data structures for the search paths for shared
> objects. */
> - _dl_init_paths (getenv ("LD_LIBRARY_PATH"), "LD_LIBRARY_PATH");
> + _dl_init_paths (getenv ("LD_LIBRARY_PATH"), "LD_LIBRARY_PATH",
> + /* No glibc-hwcaps selection support in statically
> + linked binaries. */
> + NULL, NULL);
>
> /* Remember the last search directory added at startup. */
> _dl_init_all_dirs = GL(dl_all_dirs);
Ok.
> diff --git a/elf/dl-usage.c b/elf/dl-usage.c
> index 796ad38b43..e22a9c3942 100644
> --- a/elf/dl-usage.c
> +++ b/elf/dl-usage.c
> @@ -83,7 +83,7 @@ print_search_path_for_help (struct dl_main_state *state)
> {
> if (__rtld_search_dirs.dirs == NULL)
> /* The run-time search paths have not yet been initialized. */
> - _dl_init_paths (state->library_path, state->library_path_source);
> + call_init_paths (state);
>
> _dl_printf ("\nShared library search path:\n");
>
> @@ -132,6 +132,67 @@ print_hwcap_1_finish (bool *first)
> _dl_printf (")\n");
> }
>
> +/* Print the header for print_hwcaps_subdirectories. */
> +static void
> +print_hwcaps_subdirectories_header (bool *nothing_printed)
> +{
> + if (*nothing_printed)
> + {
> + _dl_printf ("\n\
> +Subdirectories of glibc-hwcaps directories, in priority order:\n");
> + *nothing_printed = false;
> + }
> +}
> +
> +/* Print the HWCAP name itself, indented. */
> +static void
> +print_hwcaps_subdirectories_name (const struct dl_hwcaps_split *split)
> +{
> + _dl_write (STDOUT_FILENO, " ", 2);
> + _dl_write (STDOUT_FILENO, split->segment, split->length);
> +}
> +
> +/* Print the list of recognized glibc-hwcaps subdirectories. */
> +static void
> +print_hwcaps_subdirectories (const struct dl_main_state *state)
> +{
> + bool nothing_printed = true;
> + struct dl_hwcaps_split split;
> +
> + /* The prepended glibc-hwcaps subdirectories. */
> + _dl_hwcaps_split_init (&split, state->glibc_hwcaps_prepend);
> + while (_dl_hwcaps_split (&split))
> + {
> + print_hwcaps_subdirectories_header (¬hing_printed);
> + print_hwcaps_subdirectories_name (&split);
> + bool first = true;
> + print_hwcap_1 (&first, true, "searched");
> + print_hwcap_1_finish (&first);
> + }
> +
> + /* The built-in glibc-hwcaps subdirectories. Do the filtering
> + manually, so that more precise diagnostics are possible. */
> + uint32_t mask = _dl_hwcaps_subdirs_active ();
> + _dl_hwcaps_split_init (&split, _dl_hwcaps_subdirs);
> + while (_dl_hwcaps_split (&split))
> + {
> + print_hwcaps_subdirectories_header (¬hing_printed);
> + print_hwcaps_subdirectories_name (&split);
> + bool first = true;
> + print_hwcap_1 (&first, mask & 1, "supported");
> + bool listed = _dl_hwcaps_contains (state->glibc_hwcaps_mask,
> + split.segment, split.length);
> + print_hwcap_1 (&first, !listed, "masked");
> + print_hwcap_1 (&first, (mask & 1) && listed, "searched");
> + print_hwcap_1_finish (&first);
> + mask >>= 1;
> + }
> +
> + if (nothing_printed)
> + _dl_printf ("\n\
> +No subdirectories of glibc-hwcaps directories are searched.\n");
> +}
> +
> /* Write a list of hwcap subdirectories to standard output. See
> _dl_important_hwcaps in dl-hwcaps.c. */
> static void
> @@ -186,6 +247,10 @@ setting environment variables (which would be inherited by subprocesses).\n\
> --inhibit-cache Do not use " LD_SO_CACHE "\n\
> --library-path PATH use given PATH instead of content of the environment\n\
> variable LD_LIBRARY_PATH\n\
> + --glibc-hwcaps-prepend LIST\n\
> + search glibc-hwcaps subdirectories in LIST\n\
> + --glibc-hwcaps-mask LIST\n\
> + only search built-in subdirectories if in LIST\n\
> --inhibit-rpath LIST ignore RUNPATH and RPATH information in object names\n\
> in LIST\n\
> --audit LIST use objects named in LIST as auditors\n\
> @@ -198,6 +263,7 @@ This program interpreter self-identifies as: " RTLD "\n\
> ",
> argv0);
> print_search_path_for_help (state);
> + print_hwcaps_subdirectories (state);
> print_legacy_hwcap_directories ();
> _exit (EXIT_SUCCESS);
> }
Do we really need to export the new options with the '--glibc' prefix?
The loader options are an implementation detail (same for the internal
variable names).
> diff --git a/elf/markermodMARKER-VALUE.c b/elf/markermodMARKER-VALUE.c
> new file mode 100644
> index 0000000000..99bdcf71a4
> --- /dev/null
> +++ b/elf/markermodMARKER-VALUE.c
> @@ -0,0 +1,29 @@
> +/* Source file template for building shared objects with marker functions.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +/* MARKER and VALUE must be set on the compiler command line. */
> +
> +#ifndef MARKER
> +# error MARKER not defined
> +#endif
Maybe also add a check for the VALUE?
> +
> +int
> +MARKER (void)
> +{
> + return VALUE;
> +}
> diff --git a/elf/rtld.c b/elf/rtld.c
> index 5d117d0d2c..e3ec66c030 100644
> --- a/elf/rtld.c
> +++ b/elf/rtld.c
> @@ -289,6 +289,8 @@ dl_main_state_init (struct dl_main_state *state)
> state->library_path_source = NULL;
> state->preloadlist = NULL;
> state->preloadarg = NULL;
> + state->glibc_hwcaps_prepend = NULL;
> + state->glibc_hwcaps_mask = NULL;
> state->mode = rtld_mode_normal;
> state->any_debug = false;
> state->version_info = false;
> @@ -1244,6 +1246,22 @@ dl_main (const ElfW(Phdr) *phdr,
> {
> argv0 = _dl_argv[2];
>
> + _dl_skip_args += 2;
> + _dl_argc -= 2;
> + _dl_argv += 2;
> + }
> + else if (strcmp (_dl_argv[1], "--glibc-hwcaps-prepend") == 0
> + && _dl_argc > 2)
> + {
> + state.glibc_hwcaps_prepend = _dl_argv[2];
> + _dl_skip_args += 2;
> + _dl_argc -= 2;
> + _dl_argv += 2;
> + }
> + else if (strcmp (_dl_argv[1], "--glibc-hwcaps-mask") == 0
> + && _dl_argc > 2)
> + {
> + state.glibc_hwcaps_mask = _dl_argv[2];
> _dl_skip_args += 2;
> _dl_argc -= 2;
> _dl_argv += 2;
Ok.
> diff --git a/elf/tst-dl-hwcaps_split.c b/elf/tst-dl-hwcaps_split.c
> new file mode 100644
> index 0000000000..929c99a23b
> --- /dev/null
> +++ b/elf/tst-dl-hwcaps_split.c
> @@ -0,0 +1,139 @@
> +/* Unit tests for dl-hwcaps.c.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <array_length.h>
> +#include <dl-hwcaps.h>
> +#include <string.h>
> +#include <support/check.h>
> +
> +static void
> +check_split_masked (const char *input, int32_t bitmask, const char *mask,
> + const char *expected[], size_t expected_length)
> +{
> + struct dl_hwcaps_split_masked split;
> + _dl_hwcaps_split_masked_init (&split, input, bitmask, mask);
> + size_t index = 0;
> + while (_dl_hwcaps_split_masked (&split))
> + {
> + TEST_VERIFY_EXIT (index < expected_length);
> + TEST_COMPARE_BLOB (expected[index], strlen (expected[index]),
> + split.split.segment, split.split.length);
> + ++index;
> + }
> + TEST_COMPARE (index, expected_length);
> +}
> +
> +static void
> +check_split (const char *input,
> + const char *expected[], size_t expected_length)
> +{
> + struct dl_hwcaps_split split;
> + _dl_hwcaps_split_init (&split, input);
> + size_t index = 0;
> + while (_dl_hwcaps_split (&split))
> + {
> + TEST_VERIFY_EXIT (index < expected_length);
> + TEST_COMPARE_BLOB (expected[index], strlen (expected[index]),
> + split.segment, split.length);
> + ++index;
> + }
> + TEST_COMPARE (index, expected_length);
> +
> + /* Reuse the test cases with masking that does not actually remove
> + anything. */
> + check_split_masked (input, -1, NULL, expected, expected_length);
> + check_split_masked (input, -1, input, expected, expected_length);
> +}
> +
> +static int
> +do_test (void)
> +{
> + /* Splitting tests, without masking. */
> + check_split (NULL, NULL, 0);
> + check_split ("", NULL, 0);
> + check_split (":", NULL, 0);
> + check_split ("::", NULL, 0);
Ok.
> +
> + {
> + const char *expected[] = { "first" };
> + check_split ("first", expected, array_length (expected));
> + check_split (":first", expected, array_length (expected));
> + check_split ("first:", expected, array_length (expected));
> + check_split (":first:", expected, array_length (expected));
> + }
> +
Ok.
> + {
> + const char *expected[] = { "first", "second" };
> + check_split ("first:second", expected, array_length (expected));
> + check_split ("first::second", expected, array_length (expected));
> + check_split (":first:second", expected, array_length (expected));
> + check_split ("first:second:", expected, array_length (expected));
> + check_split (":first:second:", expected, array_length (expected));
> + }
> +
Ok.
> + /* Splitting tests with masking. */
> + {
> + const char *expected[] = { "first" };
> + check_split_masked ("first", 3, "first:second",
> + expected, array_length (expected));
> + check_split_masked ("first:second", 3, "first:",
> + expected, array_length (expected));
> + check_split_masked ("first:second", 1, NULL,
> + expected, array_length (expected));
> + }
Ok.
> + {
> + const char *expected[] = { "second" };
> + check_split_masked ("first:second", 3, "second",
> + expected, array_length (expected));
> + check_split_masked ("first:second:third", -1, "second:",
> + expected, array_length (expected));
> + check_split_masked ("first:second", 2, NULL,
> + expected, array_length (expected));
> + check_split_masked ("first:second:third", 2, "first:second",
> + expected, array_length (expected));
> + }
> +
Ok.
> + /* Tests for _dl_hwcaps_contains. */
> + TEST_VERIFY (_dl_hwcaps_contains (NULL, "first", strlen ("first")));
> + TEST_VERIFY (_dl_hwcaps_contains (NULL, "", 0));
> + TEST_VERIFY (! _dl_hwcaps_contains ("", "first", strlen ("first")));
> + TEST_VERIFY (! _dl_hwcaps_contains ("firs", "first", strlen ("first")));
> + TEST_VERIFY (_dl_hwcaps_contains ("firs", "first", strlen ("first") - 1));
> + for (int i = 0; i < strlen ("first"); ++i)
> + TEST_VERIFY (! _dl_hwcaps_contains ("first", "first", i));
> + TEST_VERIFY (_dl_hwcaps_contains ("first", "first", strlen ("first")));
> + TEST_VERIFY (_dl_hwcaps_contains ("first:", "first", strlen ("first")));
> + TEST_VERIFY (_dl_hwcaps_contains ("first:second",
> + "first", strlen ("first")));
> + TEST_VERIFY (_dl_hwcaps_contains (":first:second", "first",
> + strlen ("first")));
> + TEST_VERIFY (_dl_hwcaps_contains ("first:second", "second",
> + strlen ("second")));
> + TEST_VERIFY (_dl_hwcaps_contains ("first:second:", "second",
> + strlen ("second")));
> + for (int i = 0; i < strlen ("second"); ++i)
> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "sec", i));
Maybe add some tests with multiple ':'.
> +
> + return 0;
> +}
> +
> +#include <support/test-driver.c>
> +
> +/* Rebuild the sources here because the object file is built for
> + inclusion into the dynamic loader. */
> +#include "dl-hwcaps_split.c"
Ok.
> diff --git a/elf/tst-glibc-hwcaps-mask.c b/elf/tst-glibc-hwcaps-mask.c
> new file mode 100644
> index 0000000000..27b09b358c
> --- /dev/null
> +++ b/elf/tst-glibc-hwcaps-mask.c
> @@ -0,0 +1,31 @@
> +/* Test that --glibc-hwcaps-mask works.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <support/check.h>
> +
> +extern int marker1 (void);
> +
> +static int
> +do_test (void)
> +{
> + /* The marker1 function in elf/markermod1.so returns 1. */
> + TEST_COMPARE (marker1 (), 1);
> + return 0;
> +}
> +
> +#include <support/test-driver.c>
Ok.
> diff --git a/elf/tst-glibc-hwcaps-prepend.c b/elf/tst-glibc-hwcaps-prepend.c
> new file mode 100644
> index 0000000000..57d7319f14
> --- /dev/null
> +++ b/elf/tst-glibc-hwcaps-prepend.c
> @@ -0,0 +1,32 @@
> +/* Test that --glibc-hwcaps-prepend works.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <support/check.h>
> +
> +extern int marker1 (void);
> +
> +static int
> +do_test (void)
> +{
> + /* The marker1 function in
> + glibc-hwcaps/prepend-markermod1/markermod1.so returns 2. */
> + TEST_COMPARE (marker1 (), 2);
> + return 0;
> +}
> +
> +#include <support/test-driver.c>
Ok.
> diff --git a/elf/tst-glibc-hwcaps.c b/elf/tst-glibc-hwcaps.c
> new file mode 100644
> index 0000000000..28f47cf891
> --- /dev/null
> +++ b/elf/tst-glibc-hwcaps.c
> @@ -0,0 +1,28 @@
> +/* Stub test for glibc-hwcaps.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <stdio.h>
> +
> +static int
> +do_test (void)
> +{
> + puts ("info: generic tst-glibc-hwcaps (tests nothing)");
> + return 0;
> +}
> +
> +#include <support/test-driver.c>
Ok.
> diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h
> index 382eeb9be0..0b2babc70c 100644
> --- a/sysdeps/generic/ldsodefs.h
> +++ b/sysdeps/generic/ldsodefs.h
> @@ -1047,8 +1047,13 @@ extern struct r_debug *_dl_debug_initialize (ElfW(Addr) ldbase, Lmid_t ns)
> attribute_hidden;
>
> /* Initialize the basic data structure for the search paths. SOURCE
> - is either "LD_LIBRARY_PATH" or "--library-path". */
> -extern void _dl_init_paths (const char *library_path, const char *source)
> + is either "LD_LIBRARY_PATH" or "--library-path".
> + GLIBC_HWCAPS_PREPEND adds additional glibc-hwcaps subdirectories to
> + search. GLIBC_HWCAPS_MASK is used to filter the built-in
> + subdirectories if not NULL. */
> +extern void _dl_init_paths (const char *library_path, const char *source,
> + const char *glibc_hwcaps_prepend,
> + const char *glibc_hwcaps_mask)
> attribute_hidden;
>
> /* Gather the information needed to install the profiling tables and start
Ok.
> @@ -1072,9 +1077,14 @@ extern void _dl_show_auxv (void) attribute_hidden;
> extern char *_dl_next_ld_env_entry (char ***position) attribute_hidden;
>
> /* Return an array with the names of the important hardware
> - capabilities. The length of the array is written to *SZ, and the
> - maximum of all strings length is written to *MAX_CAPSTRLEN. */
> -const struct r_strlenpair *_dl_important_hwcaps (size_t *sz,
> + capabilities. PREPEND is a colon-separated list of glibc-hwcaps
> + directories to search first. MASK is a colon-separated list used
> + to filter the built-in glibc-hwcaps subdirectories. The length of
> + the array is written to *SZ, and the maximum of all strings length
> + is written to *MAX_CAPSTRLEN. */
> +const struct r_strlenpair *_dl_important_hwcaps (const char *prepend,
> + const char *mask,
> + size_t *sz,
> size_t *max_capstrlen)
> attribute_hidden;
>
>
Ok.
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
2020-11-26 20:29 ` Adhemerval Zanella
@ 2020-11-27 19:49 ` Florian Weimer
2020-11-30 18:59 ` Adhemerval Zanella
2020-12-04 12:25 ` Matheus Castanho
0 siblings, 2 replies; 58+ messages in thread
From: Florian Weimer @ 2020-11-27 19:49 UTC (permalink / raw)
To: Adhemerval Zanella via Libc-alpha
* Adhemerval Zanella via Libc-alpha:
> On 09/11/2020 15:40, Florian Weimer via Libc-alpha wrote:
>> This hacks non-power-set processing into _dl_important_hwcaps.
>> Once the legacy hwcaps handling goes away, the subdirectory
>> handling needs to be reworked, but it is premature to do this
>> while both approaches are still supported.
>
> Could you extend the commit message with the extra options you are adding
> on ld.so and their expected semantic? I would be good to have some
> documentation at least on the commit message.
I came up with this:
ld.so supports two new arguments, --glibc-hwcaps-prepend and
--glibc-hwcaps-mask. Each accepts a colon-separated list of
glibc-hwcaps subdirectory names. The prepend option adds additional
subdirectories that are searched first, in the specified order. The
mask option restricts the automatically selected subdirectories to
those listed in the option argument.
I'll also try to come up with a way to document ld.so invocations in the
manual, but this will take some time (and probably some command line
interface improvements).
>> @@ -1812,3 +1816,56 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
>> '$(test-wrapper-env)' '$(run_program_env)' \
>> '$(rpath-link)' 'test-argv0' > $@; \
>> $(evaluate-test)
>> +
>> +# Most likely search subdirectories, for each supported architecture.
>> +# Used to obtain test coverage wide test coverage.
>
> This comments sounds strange, specially the second line.
Eh, that was quite garbled. I
# A list containing the name of the most likely searched subdirectory
# of the glibc-hwcaps directory, for each supported architecture (in
# other words, the oldest hardware level recognized by the
# glibc-hwcaps mechanism for this architecture). Used to obtain test
# coverage for some glibc-hwcaps tests for the widest possible range
# of systems.
glibc-hwcaps-first-subdirs-for-tests =
>> +/* Like _dl_hwcaps_split, but apply masking. */
>> +_Bool _dl_hwcaps_split_masked (struct dl_hwcaps_split_masked *s)
>> + attribute_hidden;
>> +
>> +/* Returns true if the colon-separated HWCAP list HWCAPS contains the
>> + capability NAME (with length NAME_LENGTH). If HWCAPS is NULL, the
>> + function returns true. */
>> +_Bool _dl_hwcaps_contains (const char *hwcaps, const char *name,
>> + size_t name_length) attribute_hidden;
>
> I find it confusing that an null hwcap result true while an empty
> relies on name. Do we need that NULL to be handled special here?
NULL means there is no mask. It denotes the universe in set terms.
The caller assumes this. The empty set can be written as "".
>> +/* Colon-separated string of glibc-hwcaps subdirectories, without the
>> +/* Returns a bitmask that marks the last ACTIVE subdirectories in a
>> + _dl_hwcaps_subdirs_active string (containing SUBDIRS directories in
>> + total) as active. Intended for use in _dl_hwcaps_subdirs_active
>> + implementations (if a contiguous tail of the list in
>> + _dl_hwcaps_subdirs is selected). */
>> +static inline uint32_t
>> +_dl_hwcaps_subdirs_build_bitmask (int subdirs, int active)
>> +{
>
> Should it any assert to check if subdirs and/or active is within the
> expected range?
We discussed this before, I think. I think GCC should warn if the
numbers are out of range because the result is undefined. I filed GCC
PR 97424 for this. GCC knows about the undefined nature and exploits it
(but not as agressively as Clang), but does not warn about it because
this issue occurs very often in code that is eventually optimized away
completely.
>> /* Write a list of hwcap subdirectories to standard output. See
>> _dl_important_hwcaps in dl-hwcaps.c. */
>> static void
>> @@ -186,6 +247,10 @@ setting environment variables (which would be inherited by subprocesses).\n\
>> --inhibit-cache Do not use " LD_SO_CACHE "\n\
>> --library-path PATH use given PATH instead of content of the environment\n\
>> variable LD_LIBRARY_PATH\n\
>> + --glibc-hwcaps-prepend LIST\n\
>> + search glibc-hwcaps subdirectories in LIST\n\
>> + --glibc-hwcaps-mask LIST\n\
>> + only search built-in subdirectories if in LIST\n\
>> --inhibit-rpath LIST ignore RUNPATH and RPATH information in object names\n\
>> in LIST\n\
>> --audit LIST use objects named in LIST as auditors\n\
>> @@ -198,6 +263,7 @@ This program interpreter self-identifies as: " RTLD "\n\
>> ",
>> argv0);
>> print_search_path_for_help (state);
>> + print_hwcaps_subdirectories (state);
>> print_legacy_hwcap_directories ();
>> _exit (EXIT_SUCCESS);
>> }
>
> Do we really need to export the new options with the '--glibc' prefix?
> The loader options are an implementation detail (same for the internal
> variable names).
I did that to match the glibc-hwcaps directory name. We already have
LD_HWCAP_MASK, and --hwcaps-mask looks like it would be related to that.
>> +#ifndef MARKER
>> +# error MARKER not defined
>> +#endif
>
> Maybe also add a check for the VALUE?
Compilation will fail anyway if VALUE is not defined:
>> +
>> +int
>> +MARKER (void)
>> +{
>> + return VALUE;
>> +}
>> + /* Tests for _dl_hwcaps_contains. */
>> + TEST_VERIFY (_dl_hwcaps_contains (NULL, "first", strlen ("first")));
>> + TEST_VERIFY (_dl_hwcaps_contains (NULL, "", 0));
>> + TEST_VERIFY (! _dl_hwcaps_contains ("", "first", strlen ("first")));
>> + TEST_VERIFY (! _dl_hwcaps_contains ("firs", "first", strlen ("first")));
>> + TEST_VERIFY (_dl_hwcaps_contains ("firs", "first", strlen ("first") - 1));
>> + for (int i = 0; i < strlen ("first"); ++i)
>> + TEST_VERIFY (! _dl_hwcaps_contains ("first", "first", i));
>> + TEST_VERIFY (_dl_hwcaps_contains ("first", "first", strlen ("first")));
>> + TEST_VERIFY (_dl_hwcaps_contains ("first:", "first", strlen ("first")));
>> + TEST_VERIFY (_dl_hwcaps_contains ("first:second",
>> + "first", strlen ("first")));
>> + TEST_VERIFY (_dl_hwcaps_contains (":first:second", "first",
>> + strlen ("first")));
>> + TEST_VERIFY (_dl_hwcaps_contains ("first:second", "second",
>> + strlen ("second")));
>> + TEST_VERIFY (_dl_hwcaps_contains ("first:second:", "second",
>> + strlen ("second")));
>> + for (int i = 0; i < strlen ("second"); ++i)
>> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "sec", i));
>
> Maybe add some tests with multiple ':'.
Good idea, I came up with this:
@@ -126,8 +126,17 @@ do_test (void)
strlen ("second")));
TEST_VERIFY (_dl_hwcaps_contains ("first:second:", "second",
strlen ("second")));
+ TEST_VERIFY (_dl_hwcaps_contains ("first::second:", "second",
+ strlen ("second")));
+ TEST_VERIFY (_dl_hwcaps_contains ("first:second::", "second",
+ strlen ("second")));
for (int i = 0; i < strlen ("second"); ++i)
- TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "sec", i));
+ {
+ TEST_VERIFY (!_dl_hwcaps_contains ("first:second", "second", i));
+ TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "second", i));
+ TEST_VERIFY (!_dl_hwcaps_contains ("first:second::", "second", i));
+ TEST_VERIFY (!_dl_hwcaps_contains ("first::second", "second", i));
+ }
return 0;
}
i == 0 in the loop handles empty strings.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
2020-11-27 19:49 ` Florian Weimer
@ 2020-11-30 18:59 ` Adhemerval Zanella
2020-12-04 12:25 ` Matheus Castanho
1 sibling, 0 replies; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-30 18:59 UTC (permalink / raw)
To: Florian Weimer, Adhemerval Zanella via Libc-alpha
On 27/11/2020 16:49, Florian Weimer wrote:
> * Adhemerval Zanella via Libc-alpha:
>
>> On 09/11/2020 15:40, Florian Weimer via Libc-alpha wrote:
>>> This hacks non-power-set processing into _dl_important_hwcaps.
>>> Once the legacy hwcaps handling goes away, the subdirectory
>>> handling needs to be reworked, but it is premature to do this
>>> while both approaches are still supported.
>>
>> Could you extend the commit message with the extra options you are adding
>> on ld.so and their expected semantic? I would be good to have some
>> documentation at least on the commit message.
>
> I came up with this:
>
> ld.so supports two new arguments, --glibc-hwcaps-prepend and
> --glibc-hwcaps-mask. Each accepts a colon-separated list of
> glibc-hwcaps subdirectory names. The prepend option adds additional
> subdirectories that are searched first, in the specified order. The
> mask option restricts the automatically selected subdirectories to
> those listed in the option argument.
Looks good, I would also add an example usage (maybe based on how
tst-glibc-hwcaps-prepend-cache is organized).
>
> I'll also try to come up with a way to document ld.so invocations in the
> manual, but this will take some time (and probably some command line
> interface improvements).
Ack.
>
>>> @@ -1812,3 +1816,56 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
>>> '$(test-wrapper-env)' '$(run_program_env)' \
>>> '$(rpath-link)' 'test-argv0' > $@; \
>>> $(evaluate-test)
>>> +
>>> +# Most likely search subdirectories, for each supported architecture.
>>> +# Used to obtain test coverage wide test coverage.
>>
>> This comments sounds strange, specially the second line.
>
> Eh, that was quite garbled. I
>
> # A list containing the name of the most likely searched subdirectory
> # of the glibc-hwcaps directory, for each supported architecture (in
> # other words, the oldest hardware level recognized by the
> # glibc-hwcaps mechanism for this architecture). Used to obtain test
> # coverage for some glibc-hwcaps tests for the widest possible range
> # of systems.
> glibc-hwcaps-first-subdirs-for-tests =
>
It sounds better.
>>> +/* Like _dl_hwcaps_split, but apply masking. */
>>> +_Bool _dl_hwcaps_split_masked (struct dl_hwcaps_split_masked *s)
>>> + attribute_hidden;
>>> +
>>> +/* Returns true if the colon-separated HWCAP list HWCAPS contains the
>>> + capability NAME (with length NAME_LENGTH). If HWCAPS is NULL, the
>>> + function returns true. */
>>> +_Bool _dl_hwcaps_contains (const char *hwcaps, const char *name,
>>> + size_t name_length) attribute_hidden;
>>
>> I find it confusing that an null hwcap result true while an empty
>> relies on name. Do we need that NULL to be handled special here?
>
> NULL means there is no mask. It denotes the universe in set terms.
> The caller assumes this. The empty set can be written as "".
Alright.
>
>>> +/* Colon-separated string of glibc-hwcaps subdirectories, without the
>>> +/* Returns a bitmask that marks the last ACTIVE subdirectories in a
>>> + _dl_hwcaps_subdirs_active string (containing SUBDIRS directories in
>>> + total) as active. Intended for use in _dl_hwcaps_subdirs_active
>>> + implementations (if a contiguous tail of the list in
>>> + _dl_hwcaps_subdirs is selected). */
>>> +static inline uint32_t
>>> +_dl_hwcaps_subdirs_build_bitmask (int subdirs, int active)
>>> +{
>>
>> Should it any assert to check if subdirs and/or active is within the
>> expected range?
>
> We discussed this before, I think.
I can't recall, but sorry if I am bring it again.
> I think GCC should warn if the
> numbers are out of range because the result is undefined. I filed GCC
> PR 97424 for this. GCC knows about the undefined nature and exploits it
> (but not as agressively as Clang), but does not warn about it because
> this issue occurs very often in code that is eventually optimized away
> completely.
It seems that gcc will warn with the newer options -Wanalyzer-shift-count-negative
and -Wanalyzer-shift-count-overflow, but it seems it won't be enabled as default
(it requires -fanalyzer). It also seems that __builtin_warning is still WIP,
so it might take even longer to enable it.
I still think an assert here would improve the code, although at least
since we can get them properly integrated on glibc.
>
>>> /* Write a list of hwcap subdirectories to standard output. See
>>> _dl_important_hwcaps in dl-hwcaps.c. */
>>> static void
>>> @@ -186,6 +247,10 @@ setting environment variables (which would be inherited by subprocesses).\n\
>>> --inhibit-cache Do not use " LD_SO_CACHE "\n\
>>> --library-path PATH use given PATH instead of content of the environment\n\
>>> variable LD_LIBRARY_PATH\n\
>>> + --glibc-hwcaps-prepend LIST\n\
>>> + search glibc-hwcaps subdirectories in LIST\n\
>>> + --glibc-hwcaps-mask LIST\n\
>>> + only search built-in subdirectories if in LIST\n\
>>> --inhibit-rpath LIST ignore RUNPATH and RPATH information in object names\n\
>>> in LIST\n\
>>> --audit LIST use objects named in LIST as auditors\n\
>>> @@ -198,6 +263,7 @@ This program interpreter self-identifies as: " RTLD "\n\
>>> ",
>>> argv0);
>>> print_search_path_for_help (state);
>>> + print_hwcaps_subdirectories (state);
>>> print_legacy_hwcap_directories ();
>>> _exit (EXIT_SUCCESS);
>>> }
>>
>> Do we really need to export the new options with the '--glibc' prefix?
>> The loader options are an implementation detail (same for the internal
>> variable names).
>
> I did that to match the glibc-hwcaps directory name. We already have
> LD_HWCAP_MASK, and --hwcaps-mask looks like it would be related to that.
Ack.
>
>>> +#ifndef MARKER
>>> +# error MARKER not defined
>>> +#endif
>>
>> Maybe also add a check for the VALUE?
>
> Compilation will fail anyway if VALUE is not defined:
Ack.
>
>>> +
>>> +int
>>> +MARKER (void)
>>> +{
>>> + return VALUE;
>>> +}
>
>>> + /* Tests for _dl_hwcaps_contains. */
>>> + TEST_VERIFY (_dl_hwcaps_contains (NULL, "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains (NULL, "", 0));
>>> + TEST_VERIFY (! _dl_hwcaps_contains ("", "first", strlen ("first")));
>>> + TEST_VERIFY (! _dl_hwcaps_contains ("firs", "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("firs", "first", strlen ("first") - 1));
>>> + for (int i = 0; i < strlen ("first"); ++i)
>>> + TEST_VERIFY (! _dl_hwcaps_contains ("first", "first", i));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first", "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first:", "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first:second",
>>> + "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains (":first:second", "first",
>>> + strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first:second", "second",
>>> + strlen ("second")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first:second:", "second",
>>> + strlen ("second")));
>>> + for (int i = 0; i < strlen ("second"); ++i)
>>> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "sec", i));
>>
>> Maybe add some tests with multiple ':'.
>
> Good idea, I came up with this:
>
> @@ -126,8 +126,17 @@ do_test (void)
> strlen ("second")));
> TEST_VERIFY (_dl_hwcaps_contains ("first:second:", "second",
> strlen ("second")));
> + TEST_VERIFY (_dl_hwcaps_contains ("first::second:", "second",
> + strlen ("second")));
> + TEST_VERIFY (_dl_hwcaps_contains ("first:second::", "second",
> + strlen ("second")));
> for (int i = 0; i < strlen ("second"); ++i)
> - TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "sec", i));
> + {
> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second", "second", i));
> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "second", i));
> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second::", "second", i));
> + TEST_VERIFY (!_dl_hwcaps_contains ("first::second", "second", i));
> + }
>
> return 0;
> }
>
> i == 0 in the loop handles empty strings.
LGTM, thanks.
>
> Thanks,
> Florian
>
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
2020-11-27 19:49 ` Florian Weimer
2020-11-30 18:59 ` Adhemerval Zanella
@ 2020-12-04 12:25 ` Matheus Castanho
2020-12-04 12:43 ` Florian Weimer
1 sibling, 1 reply; 58+ messages in thread
From: Matheus Castanho @ 2020-12-04 12:25 UTC (permalink / raw)
To: Florian Weimer, Adhemerval Zanella via Libc-alpha
Hi Florian, after merging this patch builds with --disable-tunables started failing on ppc*:
In file included from dl-hwcaps_split.c:19:
./dl-hwcaps.h:45:3: error: unknown type name ‘size_t’
size_t length; /* Number of bytes until ':' or NUL. */
^~~~~~
./dl-hwcaps.h:99:28: error: unknown type name ‘size_t’
size_t name_length) attribute_hidden;
^~~~~~
./dl-hwcaps.h:99:28: note: ‘size_t’ is defined in header ‘<stddef.h>’; did you forget to ‘#include <stddef.h>’?
./dl-hwcaps.h:25:1:
+#include <stddef.h>
./dl-hwcaps.h:99:28:
size_t name_length) attribute_hidden;
^~~~~~
In file included from dl-hwcaps-subdirs.c:19:
./dl-hwcaps.h:45:3: error: unknown type name ‘size_t’
size_t length; /* Number of bytes until ':' or NUL. */
^~~~~~
./dl-hwcaps.h:99:28: error: unknown type name ‘size_t’
size_t name_length) attribute_hidden;
^~~~~~
./dl-hwcaps.h:99:28: note: ‘size_t’ is defined in header ‘<stddef.h>’; did you forget to ‘#include <stddef.h>’?
./dl-hwcaps.h:25:1:
+#include <stddef.h>
./dl-hwcaps.h:99:28:
size_t name_length) attribute_hidden;
^~~~~~
Including stddef.h on dl-hwcaps.h fixes the issue:
diff --git a/elf/dl-hwcaps.h b/elf/dl-hwcaps.h
index e6fcac6edc..147dc3d2a8 100644
--- a/elf/dl-hwcaps.h
+++ b/elf/dl-hwcaps.h
@@ -20,6 +20,7 @@
#define _DL_HWCAPS_H
#include <stdint.h>
+#include <stddef.h>
#include <elf/dl-tunables.h>
Thanks,
Matheus Castanho
On 11/27/20 4:49 PM, Florian Weimer via Libc-alpha wrote:
> * Adhemerval Zanella via Libc-alpha:
>
>> On 09/11/2020 15:40, Florian Weimer via Libc-alpha wrote:
>>> This hacks non-power-set processing into _dl_important_hwcaps.
>>> Once the legacy hwcaps handling goes away, the subdirectory
>>> handling needs to be reworked, but it is premature to do this
>>> while both approaches are still supported.
>>
>> Could you extend the commit message with the extra options you are adding
>> on ld.so and their expected semantic? I would be good to have some
>> documentation at least on the commit message.
>
> I came up with this:
>
> ld.so supports two new arguments, --glibc-hwcaps-prepend and
> --glibc-hwcaps-mask. Each accepts a colon-separated list of
> glibc-hwcaps subdirectory names. The prepend option adds additional
> subdirectories that are searched first, in the specified order. The
> mask option restricts the automatically selected subdirectories to
> those listed in the option argument.
>
> I'll also try to come up with a way to document ld.so invocations in the
> manual, but this will take some time (and probably some command line
> interface improvements).
>
>>> @@ -1812,3 +1816,56 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
>>> '$(test-wrapper-env)' '$(run_program_env)' \
>>> '$(rpath-link)' 'test-argv0' > $@; \
>>> $(evaluate-test)
>>> +
>>> +# Most likely search subdirectories, for each supported architecture.
>>> +# Used to obtain test coverage wide test coverage.
>>
>> This comments sounds strange, specially the second line.
>
> Eh, that was quite garbled. I
>
> # A list containing the name of the most likely searched subdirectory
> # of the glibc-hwcaps directory, for each supported architecture (in
> # other words, the oldest hardware level recognized by the
> # glibc-hwcaps mechanism for this architecture). Used to obtain test
> # coverage for some glibc-hwcaps tests for the widest possible range
> # of systems.
> glibc-hwcaps-first-subdirs-for-tests =
>
>>> +/* Like _dl_hwcaps_split, but apply masking. */
>>> +_Bool _dl_hwcaps_split_masked (struct dl_hwcaps_split_masked *s)
>>> + attribute_hidden;
>>> +
>>> +/* Returns true if the colon-separated HWCAP list HWCAPS contains the
>>> + capability NAME (with length NAME_LENGTH). If HWCAPS is NULL, the
>>> + function returns true. */
>>> +_Bool _dl_hwcaps_contains (const char *hwcaps, const char *name,
>>> + size_t name_length) attribute_hidden;
>>
>> I find it confusing that an null hwcap result true while an empty
>> relies on name. Do we need that NULL to be handled special here?
>
> NULL means there is no mask. It denotes the universe in set terms.
> The caller assumes this. The empty set can be written as "".
>
>>> +/* Colon-separated string of glibc-hwcaps subdirectories, without the
>>> +/* Returns a bitmask that marks the last ACTIVE subdirectories in a
>>> + _dl_hwcaps_subdirs_active string (containing SUBDIRS directories in
>>> + total) as active. Intended for use in _dl_hwcaps_subdirs_active
>>> + implementations (if a contiguous tail of the list in
>>> + _dl_hwcaps_subdirs is selected). */
>>> +static inline uint32_t
>>> +_dl_hwcaps_subdirs_build_bitmask (int subdirs, int active)
>>> +{
>>
>> Should it any assert to check if subdirs and/or active is within the
>> expected range?
>
> We discussed this before, I think. I think GCC should warn if the
> numbers are out of range because the result is undefined. I filed GCC
> PR 97424 for this. GCC knows about the undefined nature and exploits it
> (but not as agressively as Clang), but does not warn about it because
> this issue occurs very often in code that is eventually optimized away
> completely.
>
>>> /* Write a list of hwcap subdirectories to standard output. See
>>> _dl_important_hwcaps in dl-hwcaps.c. */
>>> static void
>>> @@ -186,6 +247,10 @@ setting environment variables (which would be inherited by subprocesses).\n\
>>> --inhibit-cache Do not use " LD_SO_CACHE "\n\
>>> --library-path PATH use given PATH instead of content of the environment\n\
>>> variable LD_LIBRARY_PATH\n\
>>> + --glibc-hwcaps-prepend LIST\n\
>>> + search glibc-hwcaps subdirectories in LIST\n\
>>> + --glibc-hwcaps-mask LIST\n\
>>> + only search built-in subdirectories if in LIST\n\
>>> --inhibit-rpath LIST ignore RUNPATH and RPATH information in object names\n\
>>> in LIST\n\
>>> --audit LIST use objects named in LIST as auditors\n\
>>> @@ -198,6 +263,7 @@ This program interpreter self-identifies as: " RTLD "\n\
>>> ",
>>> argv0);
>>> print_search_path_for_help (state);
>>> + print_hwcaps_subdirectories (state);
>>> print_legacy_hwcap_directories ();
>>> _exit (EXIT_SUCCESS);
>>> }
>>
>> Do we really need to export the new options with the '--glibc' prefix?
>> The loader options are an implementation detail (same for the internal
>> variable names).
>
> I did that to match the glibc-hwcaps directory name. We already have
> LD_HWCAP_MASK, and --hwcaps-mask looks like it would be related to that.
>
>>> +#ifndef MARKER
>>> +# error MARKER not defined
>>> +#endif
>>
>> Maybe also add a check for the VALUE?
>
> Compilation will fail anyway if VALUE is not defined:
>
>>> +
>>> +int
>>> +MARKER (void)
>>> +{
>>> + return VALUE;
>>> +}
>
>>> + /* Tests for _dl_hwcaps_contains. */
>>> + TEST_VERIFY (_dl_hwcaps_contains (NULL, "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains (NULL, "", 0));
>>> + TEST_VERIFY (! _dl_hwcaps_contains ("", "first", strlen ("first")));
>>> + TEST_VERIFY (! _dl_hwcaps_contains ("firs", "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("firs", "first", strlen ("first") - 1));
>>> + for (int i = 0; i < strlen ("first"); ++i)
>>> + TEST_VERIFY (! _dl_hwcaps_contains ("first", "first", i));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first", "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first:", "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first:second",
>>> + "first", strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains (":first:second", "first",
>>> + strlen ("first")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first:second", "second",
>>> + strlen ("second")));
>>> + TEST_VERIFY (_dl_hwcaps_contains ("first:second:", "second",
>>> + strlen ("second")));
>>> + for (int i = 0; i < strlen ("second"); ++i)
>>> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "sec", i));
>>
>> Maybe add some tests with multiple ':'.
>
> Good idea, I came up with this:
>
> @@ -126,8 +126,17 @@ do_test (void)
> strlen ("second")));
> TEST_VERIFY (_dl_hwcaps_contains ("first:second:", "second",
> strlen ("second")));
> + TEST_VERIFY (_dl_hwcaps_contains ("first::second:", "second",
> + strlen ("second")));
> + TEST_VERIFY (_dl_hwcaps_contains ("first:second::", "second",
> + strlen ("second")));
> for (int i = 0; i < strlen ("second"); ++i)
> - TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "sec", i));
> + {
> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second", "second", i));
> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second:", "second", i));
> + TEST_VERIFY (!_dl_hwcaps_contains ("first:second::", "second", i));
> + TEST_VERIFY (!_dl_hwcaps_contains ("first::second", "second", i));
> + }
>
> return 0;
> }
>
> i == 0 in the loop handles empty strings.
>
> Thanks,
> Florian
>
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
2020-12-04 12:25 ` Matheus Castanho
@ 2020-12-04 12:43 ` Florian Weimer
2020-12-04 12:59 ` Matheus Castanho
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-12-04 12:43 UTC (permalink / raw)
To: Matheus Castanho; +Cc: Adhemerval Zanella via Libc-alpha
* Matheus Castanho:
> Including stddef.h on dl-hwcaps.h fixes the issue:
>
> diff --git a/elf/dl-hwcaps.h b/elf/dl-hwcaps.h
> index e6fcac6edc..147dc3d2a8 100644
> --- a/elf/dl-hwcaps.h
> +++ b/elf/dl-hwcaps.h
> @@ -20,6 +20,7 @@
> #define _DL_HWCAPS_H
>
> #include <stdint.h>
> +#include <stddef.h>
>
> #include <elf/dl-tunables.h>
Sorry for that. Fix looks good to me. Do you want me to push the fix,
or do you want to do it yourself?
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
2020-12-04 12:43 ` Florian Weimer
@ 2020-12-04 12:59 ` Matheus Castanho
0 siblings, 0 replies; 58+ messages in thread
From: Matheus Castanho @ 2020-12-04 12:59 UTC (permalink / raw)
To: Florian Weimer; +Cc: Adhemerval Zanella via Libc-alpha
On 12/4/20 9:43 AM, Florian Weimer wrote:
> * Matheus Castanho:
>
>> Including stddef.h on dl-hwcaps.h fixes the issue:
>>
>> diff --git a/elf/dl-hwcaps.h b/elf/dl-hwcaps.h
>> index e6fcac6edc..147dc3d2a8 100644
>> --- a/elf/dl-hwcaps.h
>> +++ b/elf/dl-hwcaps.h
>> @@ -20,6 +20,7 @@
>> #define _DL_HWCAPS_H
>>
>> #include <stdint.h>
>> +#include <stddef.h>
>>
>> #include <elf/dl-tunables.h>
>
> Sorry for that. Fix looks good to me. Do you want me to push the fix,
> or do you want to do it yourself?
>
> Thanks,
> Florian
>
No problem. I've pushed it already: 4f4bd9e47ba98ccfeeaa8c600c0b0c8bbabcebb3
Thanks,
Matheus Castanho
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 04/11] elf: Add endianness markup to ld.so.cache
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (2 preceding siblings ...)
2020-11-09 18:40 ` [PATCH 03/11] elf: Add glibc-hwcaps support for LD_LIBRARY_PATH Florian Weimer
@ 2020-11-09 18:40 ` Florian Weimer
2020-11-27 13:56 ` Adhemerval Zanella
2020-11-09 18:40 ` [PATCH 05/11] elf: Add extension mechanism " Florian Weimer
` (8 subsequent siblings)
12 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:40 UTC (permalink / raw)
To: libc-alpha
Use a reserved byte in the new format cache header to indicate whether
the file is in little endian or big endian format. Eventually, this
information could be used to provide a unified cache for qemu-user
and similiar scenarios.
---
elf/cache.c | 11 +++++++
elf/dl-cache.c | 20 +++++++++++-
sysdeps/generic/dl-cache.h | 62 +++++++++++++++++++++++++++++++++++++-
3 files changed, 91 insertions(+), 2 deletions(-)
diff --git a/elf/cache.c b/elf/cache.c
index c241c17ef9..ffecbe6d82 100644
--- a/elf/cache.c
+++ b/elf/cache.c
@@ -152,6 +152,14 @@ print_entry (const char *lib, int flag, unsigned int osversion,
printf (") => %s\n", key);
}
+/* Print an error and exit if the new-file cache is internally
+ inconsistent. */
+static void
+check_new_cache (struct cache_file_new *cache)
+{
+ if (! cache_file_new_matches_endian (cache))
+ error (EXIT_FAILURE, 0, _("Cache file has wrong endianness.\n"));
+}
/* Print the whole cache file, if a file contains the new cache format
hidden in the old one, print the contents of the new format. */
@@ -193,6 +201,7 @@ print_cache (const char *cache_name)
|| memcmp (cache_new->version, CACHE_VERSION,
sizeof CACHE_VERSION - 1))
error (EXIT_FAILURE, 0, _("File is not a cache file.\n"));
+ check_new_cache (cache_new);
format = 1;
/* This is where the strings start. */
cache_data = (const char *) cache_new;
@@ -222,6 +231,7 @@ print_cache (const char *cache_name)
&& memcmp (cache_new->version, CACHE_VERSION,
sizeof CACHE_VERSION - 1) == 0)
{
+ check_new_cache (cache_new);
cache_data = (const char *) cache_new;
format = 1;
}
@@ -361,6 +371,7 @@ save_cache (const char *cache_name)
file_entries_new->nlibs = cache_entry_count;
file_entries_new->len_strings = total_strlen;
+ file_entries_new->flags = cache_file_new_flags_endian_current;
}
/* Pad for alignment of cache_file_new. */
diff --git a/elf/dl-cache.c b/elf/dl-cache.c
index 45894ecd2f..02c46ffb0c 100644
--- a/elf/dl-cache.c
+++ b/elf/dl-cache.c
@@ -242,6 +242,11 @@ _dl_load_cache_lookup (const char *name)
&& ((cachesize - sizeof *cache_new) / sizeof (struct file_entry_new)
>= ((struct cache_file_new *) file)->nlibs))
{
+ if (! cache_file_new_matches_endian (file))
+ {
+ __munmap (file, cachesize);
+ file = (void *) -1;
+ }
cache_new = file;
cache = file;
}
@@ -263,7 +268,20 @@ _dl_load_cache_lookup (const char *name)
if (cachesize < (offset + sizeof (struct cache_file_new))
|| memcmp (cache_new->magic, CACHEMAGIC_VERSION_NEW,
sizeof CACHEMAGIC_VERSION_NEW - 1) != 0)
- cache_new = (void *) -1;
+ cache_new = (void *) -1;
+ else
+ {
+ if (! cache_file_new_matches_endian (cache_new))
+ {
+ /* The old-format part of the cache is bogus as well
+ if the endianness does not match. (But it is
+ unclear how the new header can be located if the
+ endianess does not match.) */
+ cache = (void *) -1;
+ cache_new = (void *) -1;
+ __munmap (file, cachesize);
+ }
+ }
}
else
{
diff --git a/sysdeps/generic/dl-cache.h b/sysdeps/generic/dl-cache.h
index 4ddd96b005..46026e0988 100644
--- a/sysdeps/generic/dl-cache.h
+++ b/sysdeps/generic/dl-cache.h
@@ -16,6 +16,11 @@
License along with the GNU C Library; if not, see
<https://www.gnu.org/licenses/>. */
+#ifndef _DL_CACHE_H
+#define _DL_CACHE_H
+
+#include <endian.h>
+#include <stdbool.h>
#include <stdint.h>
#ifndef _DL_CACHE_DEFAULT_ID
@@ -92,21 +97,76 @@ struct file_entry_new
uint64_t hwcap; /* Hwcap entry. */
};
+/* See flags member of struct cache_file_new below. */
+enum
+ {
+ /* No endianness information available. An old ldconfig version
+ without endianness support wrote the file. */
+ cache_file_new_flags_endian_unset = 0,
+
+ /* Cache is invalid and should be ignored. */
+ cache_file_new_flags_endian_invalid = 1,
+
+ /* Cache format is little endian. */
+ cache_file_new_flags_endian_little = 2,
+
+ /* Cache format is big endian. */
+ cache_file_new_flags_endian_big = 3,
+
+ /* Bit mask to extract the cache_file_new_flags_endian_*
+ values. */
+ cache_file_new_flags_endian_mask = 3,
+
+ /* Expected value of the endian bits in the flags member for the
+ current architecture. */
+ cache_file_new_flags_endian_current
+ = (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+ ? cache_file_new_flags_endian_little
+ : cache_file_new_flags_endian_big),
+ };
+
struct cache_file_new
{
char magic[sizeof CACHEMAGIC_NEW - 1];
char version[sizeof CACHE_VERSION - 1];
uint32_t nlibs; /* Number of entries. */
uint32_t len_strings; /* Size of string table. */
- uint32_t unused[5]; /* Leave space for future extensions
+
+ /* flags & cache_file_new_flags_endian_mask is one of the values
+ cache_file_new_flags_endian_unset, cache_file_new_flags_endian_invalid,
+ cache_file_new_flags_endian_little, cache_file_new_flags_endian_big.
+
+ The remaining bits are unused and should be generated as zero and
+ ignored by readers. */
+ uint8_t flags;
+
+ uint8_t padding_unsed[3]; /* Not used, for future extensions. */
+
+ uint32_t unused[4]; /* Leave space for future extensions
and align to 8 byte boundary. */
struct file_entry_new libs[0]; /* Entries describing libraries. */
/* After this the string table of size len_strings is found. */
};
+/* Returns false if *CACHE has the wrong endianness for this
+ architecture, and true if the endianness matches (or is
+ unknown). */
+static inline bool
+cache_file_new_matches_endian (const struct cache_file_new *cache)
+{
+ /* A zero value for cache->flags means that no endianness
+ information is available. */
+ return cache->flags == 0
+ || ((cache->flags & cache_file_new_flags_endian_big)
+ == cache_file_new_flags_endian_current);
+}
+
+
/* Used to align cache_file_new. */
#define ALIGN_CACHE(addr) \
(((addr) + __alignof__ (struct cache_file_new) -1) \
& (~(__alignof__ (struct cache_file_new) - 1)))
extern int _dl_cache_libcmp (const char *p1, const char *p2) attribute_hidden;
+
+#endif /* _DL_CACHE_H */
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 04/11] elf: Add endianness markup to ld.so.cache
2020-11-09 18:40 ` [PATCH 04/11] elf: Add endianness markup to ld.so.cache Florian Weimer
@ 2020-11-27 13:56 ` Adhemerval Zanella
2020-11-27 19:49 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-27 13:56 UTC (permalink / raw)
To: libc-alpha
On 09/11/2020 15:40, Florian Weimer via Libc-alpha wrote:
> Use a reserved byte in the new format cache header to indicate whether
> the file is in little endian or big endian format. Eventually, this
> information could be used to provide a unified cache for qemu-user
> and similiar scenarios.
LGTM, with some comments below.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
> elf/cache.c | 11 +++++++
> elf/dl-cache.c | 20 +++++++++++-
> sysdeps/generic/dl-cache.h | 62 +++++++++++++++++++++++++++++++++++++-
> 3 files changed, 91 insertions(+), 2 deletions(-)
>
> diff --git a/elf/cache.c b/elf/cache.c
> index c241c17ef9..ffecbe6d82 100644
> --- a/elf/cache.c
> +++ b/elf/cache.c
> @@ -152,6 +152,14 @@ print_entry (const char *lib, int flag, unsigned int osversion,
> printf (") => %s\n", key);
> }
>
> +/* Print an error and exit if the new-file cache is internally
> + inconsistent. */
> +static void
> +check_new_cache (struct cache_file_new *cache)
> +{
> + if (! cache_file_new_matches_endian (cache))
> + error (EXIT_FAILURE, 0, _("Cache file has wrong endianness.\n"));
> +}
>
> /* Print the whole cache file, if a file contains the new cache format
> hidden in the old one, print the contents of the new format. */
Ok.
> @@ -193,6 +201,7 @@ print_cache (const char *cache_name)
> || memcmp (cache_new->version, CACHE_VERSION,
> sizeof CACHE_VERSION - 1))
> error (EXIT_FAILURE, 0, _("File is not a cache file.\n"));
> + check_new_cache (cache_new);
> format = 1;
> /* This is where the strings start. */
> cache_data = (const char *) cache_new;
Ok.
> @@ -222,6 +231,7 @@ print_cache (const char *cache_name)
> && memcmp (cache_new->version, CACHE_VERSION,
> sizeof CACHE_VERSION - 1) == 0)
> {
> + check_new_cache (cache_new);
> cache_data = (const char *) cache_new;
> format = 1;
> }
Ok.
> @@ -361,6 +371,7 @@ save_cache (const char *cache_name)
>
> file_entries_new->nlibs = cache_entry_count;
> file_entries_new->len_strings = total_strlen;
> + file_entries_new->flags = cache_file_new_flags_endian_current;
> }
>
> /* Pad for alignment of cache_file_new. */
Ok.
> diff --git a/elf/dl-cache.c b/elf/dl-cache.c
> index 45894ecd2f..02c46ffb0c 100644
> --- a/elf/dl-cache.c
> +++ b/elf/dl-cache.c
> @@ -242,6 +242,11 @@ _dl_load_cache_lookup (const char *name)
> && ((cachesize - sizeof *cache_new) / sizeof (struct file_entry_new)
> >= ((struct cache_file_new *) file)->nlibs))
> {
> + if (! cache_file_new_matches_endian (file))
> + {
> + __munmap (file, cachesize);
> + file = (void *) -1;
> + }
> cache_new = file;
> cache = file;
> }
Ok.
> @@ -263,7 +268,20 @@ _dl_load_cache_lookup (const char *name)
> if (cachesize < (offset + sizeof (struct cache_file_new))
> || memcmp (cache_new->magic, CACHEMAGIC_VERSION_NEW,
> sizeof CACHEMAGIC_VERSION_NEW - 1) != 0)
> - cache_new = (void *) -1;
> + cache_new = (void *) -1;
> + else
> + {
> + if (! cache_file_new_matches_endian (cache_new))
> + {
> + /* The old-format part of the cache is bogus as well
> + if the endianness does not match. (But it is
> + unclear how the new header can be located if the
> + endianess does not match.) */
> + cache = (void *) -1;
> + cache_new = (void *) -1;
I think there is no need to set cache_new to -1 here, the function
will return anyway with the 'cache == (void *) -1' check below.
> + __munmap (file, cachesize);
> + }
> + }
> }
> else
> {
> diff --git a/sysdeps/generic/dl-cache.h b/sysdeps/generic/dl-cache.h
> index 4ddd96b005..46026e0988 100644
> --- a/sysdeps/generic/dl-cache.h
> +++ b/sysdeps/generic/dl-cache.h
> @@ -16,6 +16,11 @@
> License along with the GNU C Library; if not, see
> <https://www.gnu.org/licenses/>. */
>
> +#ifndef _DL_CACHE_H
> +#define _DL_CACHE_H
> +
> +#include <endian.h>
> +#include <stdbool.h>
> #include <stdint.h>
>
> #ifndef _DL_CACHE_DEFAULT_ID
Ok.
> @@ -92,21 +97,76 @@ struct file_entry_new
> uint64_t hwcap; /* Hwcap entry. */
> };
>
> +/* See flags member of struct cache_file_new below. */
> +enum
> + {
> + /* No endianness information available. An old ldconfig version
> + without endianness support wrote the file. */
> + cache_file_new_flags_endian_unset = 0,
> +
> + /* Cache is invalid and should be ignored. */
> + cache_file_new_flags_endian_invalid = 1,
> +
> + /* Cache format is little endian. */
> + cache_file_new_flags_endian_little = 2,
> +
> + /* Cache format is big endian. */
> + cache_file_new_flags_endian_big = 3,
> +
> + /* Bit mask to extract the cache_file_new_flags_endian_*
> + values. */
> + cache_file_new_flags_endian_mask = 3,
> +
> + /* Expected value of the endian bits in the flags member for the
> + current architecture. */
> + cache_file_new_flags_endian_current
> + = (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
> + ? cache_file_new_flags_endian_little
> + : cache_file_new_flags_endian_big),
> + };
> +
Ok.
> struct cache_file_new
> {
> char magic[sizeof CACHEMAGIC_NEW - 1];
> char version[sizeof CACHE_VERSION - 1];
> uint32_t nlibs; /* Number of entries. */
> uint32_t len_strings; /* Size of string table. */
> - uint32_t unused[5]; /* Leave space for future extensions
> +
> + /* flags & cache_file_new_flags_endian_mask is one of the values
> + cache_file_new_flags_endian_unset, cache_file_new_flags_endian_invalid,
> + cache_file_new_flags_endian_little, cache_file_new_flags_endian_big.
> +
> + The remaining bits are unused and should be generated as zero and
> + ignored by readers. */
> + uint8_t flags;
> +
> + uint8_t padding_unsed[3]; /* Not used, for future extensions. */
> +
> + uint32_t unused[4]; /* Leave space for future extensions
> and align to 8 byte boundary. */
> struct file_entry_new libs[0]; /* Entries describing libraries. */
> /* After this the string table of size len_strings is found. */
> };
>
Maybe add a _Static_assert here to check the expected
struct cache_file_new size?
> +/* Returns false if *CACHE has the wrong endianness for this
> + architecture, and true if the endianness matches (or is
> + unknown). */
> +static inline bool
> +cache_file_new_matches_endian (const struct cache_file_new *cache)
> +{
> + /* A zero value for cache->flags means that no endianness
> + information is available. */
> + return cache->flags == 0
Maybe check against cache_file_new_flags_endian_unset ?
> + || ((cache->flags & cache_file_new_flags_endian_big)
> + == cache_file_new_flags_endian_current);
> +}
> +
> +
> /* Used to align cache_file_new. */
> #define ALIGN_CACHE(addr) \
> (((addr) + __alignof__ (struct cache_file_new) -1) \
> & (~(__alignof__ (struct cache_file_new) - 1)))
>
> extern int _dl_cache_libcmp (const char *p1, const char *p2) attribute_hidden;
> +
> +#endif /* _DL_CACHE_H */
>
Ok.
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 04/11] elf: Add endianness markup to ld.so.cache
2020-11-27 13:56 ` Adhemerval Zanella
@ 2020-11-27 19:49 ` Florian Weimer
2020-11-30 19:00 ` Adhemerval Zanella
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-27 19:49 UTC (permalink / raw)
To: Adhemerval Zanella via Libc-alpha
* Adhemerval Zanella via Libc-alpha:
>> @@ -263,7 +268,20 @@ _dl_load_cache_lookup (const char *name)
>> if (cachesize < (offset + sizeof (struct cache_file_new))
>> || memcmp (cache_new->magic, CACHEMAGIC_VERSION_NEW,
>> sizeof CACHEMAGIC_VERSION_NEW - 1) != 0)
>> - cache_new = (void *) -1;
>> + cache_new = (void *) -1;
>> + else
>> + {
>> + if (! cache_file_new_matches_endian (cache_new))
>> + {
>> + /* The old-format part of the cache is bogus as well
>> + if the endianness does not match. (But it is
>> + unclear how the new header can be located if the
>> + endianess does not match.) */
>> + cache = (void *) -1;
>> + cache_new = (void *) -1;
>
> I think there is no need to set cache_new to -1 here, the function
> will return anyway with the 'cache == (void *) -1' check below.
Sure, but I want to make absolutely clear that cache_new is not usable
due to the incorrect endianness.
I'd like to leave it in.
>> struct cache_file_new
>> {
>> char magic[sizeof CACHEMAGIC_NEW - 1];
>> char version[sizeof CACHE_VERSION - 1];
>> uint32_t nlibs; /* Number of entries. */
>> uint32_t len_strings; /* Size of string table. */
>> - uint32_t unused[5]; /* Leave space for future extensions
>> +
>> + /* flags & cache_file_new_flags_endian_mask is one of the values
>> + cache_file_new_flags_endian_unset, cache_file_new_flags_endian_invalid,
>> + cache_file_new_flags_endian_little, cache_file_new_flags_endian_big.
>> +
>> + The remaining bits are unused and should be generated as zero and
>> + ignored by readers. */
>> + uint8_t flags;
>> +
>> + uint8_t padding_unsed[3]; /* Not used, for future extensions. */
>> +
>> + uint32_t unused[4]; /* Leave space for future extensions
>> and align to 8 byte boundary. */
>> struct file_entry_new libs[0]; /* Entries describing libraries. */
>> /* After this the string table of size len_strings is found. */
>> };
>>
>
> Maybe add a _Static_assert here to check the expected
> struct cache_file_new size?
I've added:
_Static_assert (sizeof (struct cache_file_new) == 48,
"size of struct cache_file_new");
>> +/* Returns false if *CACHE has the wrong endianness for this
>> + architecture, and true if the endianness matches (or is
>> + unknown). */
>> +static inline bool
>> +cache_file_new_matches_endian (const struct cache_file_new *cache)
>> +{
>> + /* A zero value for cache->flags means that no endianness
>> + information is available. */
>> + return cache->flags == 0
>
> Maybe check against cache_file_new_flags_endian_unset ?
I intended cache_file_new_flags_endian_unset to document the (reserved)
value within the masked lower two bits, not the whole flags field, so I
think 0 is correct here.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 04/11] elf: Add endianness markup to ld.so.cache
2020-11-27 19:49 ` Florian Weimer
@ 2020-11-30 19:00 ` Adhemerval Zanella
0 siblings, 0 replies; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-30 19:00 UTC (permalink / raw)
To: Florian Weimer, Adhemerval Zanella via Libc-alpha
On 27/11/2020 16:49, Florian Weimer wrote:
> * Adhemerval Zanella via Libc-alpha:
>
>>> @@ -263,7 +268,20 @@ _dl_load_cache_lookup (const char *name)
>>> if (cachesize < (offset + sizeof (struct cache_file_new))
>>> || memcmp (cache_new->magic, CACHEMAGIC_VERSION_NEW,
>>> sizeof CACHEMAGIC_VERSION_NEW - 1) != 0)
>>> - cache_new = (void *) -1;
>>> + cache_new = (void *) -1;
>>> + else
>>> + {
>>> + if (! cache_file_new_matches_endian (cache_new))
>>> + {
>>> + /* The old-format part of the cache is bogus as well
>>> + if the endianness does not match. (But it is
>>> + unclear how the new header can be located if the
>>> + endianess does not match.) */
>>> + cache = (void *) -1;
>>> + cache_new = (void *) -1;
>>
>> I think there is no need to set cache_new to -1 here, the function
>> will return anyway with the 'cache == (void *) -1' check below.
>
> Sure, but I want to make absolutely clear that cache_new is not usable
> due to the incorrect endianness.
>
> I'd like to leave it in.
Ack.
>
>>> struct cache_file_new
>>> {
>>> char magic[sizeof CACHEMAGIC_NEW - 1];
>>> char version[sizeof CACHE_VERSION - 1];
>>> uint32_t nlibs; /* Number of entries. */
>>> uint32_t len_strings; /* Size of string table. */
>>> - uint32_t unused[5]; /* Leave space for future extensions
>>> +
>>> + /* flags & cache_file_new_flags_endian_mask is one of the values
>>> + cache_file_new_flags_endian_unset, cache_file_new_flags_endian_invalid,
>>> + cache_file_new_flags_endian_little, cache_file_new_flags_endian_big.
>>> +
>>> + The remaining bits are unused and should be generated as zero and
>>> + ignored by readers. */
>>> + uint8_t flags;
>>> +
>>> + uint8_t padding_unsed[3]; /* Not used, for future extensions. */
>>> +
>>> + uint32_t unused[4]; /* Leave space for future extensions
>>> and align to 8 byte boundary. */
>>> struct file_entry_new libs[0]; /* Entries describing libraries. */
>>> /* After this the string table of size len_strings is found. */
>>> };
>>>
>>
>> Maybe add a _Static_assert here to check the expected
>> struct cache_file_new size?
>
> I've added:
>
> _Static_assert (sizeof (struct cache_file_new) == 48,
> "size of struct cache_file_new");
LGTM.
>
>>> +/* Returns false if *CACHE has the wrong endianness for this
>>> + architecture, and true if the endianness matches (or is
>>> + unknown). */
>>> +static inline bool
>>> +cache_file_new_matches_endian (const struct cache_file_new *cache)
>>> +{
>>> + /* A zero value for cache->flags means that no endianness
>>> + information is available. */
>>> + return cache->flags == 0
>>
>> Maybe check against cache_file_new_flags_endian_unset ?
>
> I intended cache_file_new_flags_endian_unset to document the (reserved)
> value within the masked lower two bits, not the whole flags field, so I
> think 0 is correct here.
Ack.
>
> Thanks,
> Florian
>
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 05/11] elf: Add extension mechanism to ld.so.cache
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (3 preceding siblings ...)
2020-11-09 18:40 ` [PATCH 04/11] elf: Add endianness markup to ld.so.cache Florian Weimer
@ 2020-11-09 18:40 ` Florian Weimer
2020-11-27 18:01 ` Adhemerval Zanella
2020-11-09 18:40 ` [PATCH 06/11] elf: Implement a string table for ldconfig, with tail merging Florian Weimer
` (7 subsequent siblings)
12 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:40 UTC (permalink / raw)
To: libc-alpha
A previously unused new-format header field is used to record
the address of an extension directory.
This change adds a demo extension which records the version of
ldconfig which builds a file.
---
elf/cache.c | 89 +++++++++++++++++++++++++++
sysdeps/generic/dl-cache.h | 123 ++++++++++++++++++++++++++++++++++++-
2 files changed, 211 insertions(+), 1 deletion(-)
diff --git a/elf/cache.c b/elf/cache.c
index ffecbe6d82..549e04ce21 100644
--- a/elf/cache.c
+++ b/elf/cache.c
@@ -15,6 +15,7 @@
You should have received a copy of the GNU General Public License
along with this program; if not, see <https://www.gnu.org/licenses/>. */
+#include <assert.h>
#include <errno.h>
#include <error.h>
#include <dirent.h>
@@ -33,6 +34,7 @@
#include <ldconfig.h>
#include <dl-cache.h>
+#include <version.h>
struct cache_entry
{
@@ -161,6 +163,21 @@ check_new_cache (struct cache_file_new *cache)
error (EXIT_FAILURE, 0, _("Cache file has wrong endianness.\n"));
}
+/* Print the extension information at the cache at start address
+ FILE_BASE, of length FILE_SIZE bytes. The new-format cache header
+ is at CACHE, and the file name for diagnostics is CACHE_NAME. */
+static void
+print_extensions (struct cache_extension_all_loaded *ext)
+{
+ if (ext->sections[cache_extension_tag_generator].base != NULL)
+ {
+ fputs (_("Cache generated by: "), stdout);
+ fwrite (ext->sections[cache_extension_tag_generator].base, 1,
+ ext->sections[cache_extension_tag_generator].size, stdout);
+ putchar ('\n');
+ }
+}
+
/* Print the whole cache file, if a file contains the new cache format
hidden in the old one, print the contents of the new format. */
void
@@ -250,6 +267,11 @@ print_cache (const char *cache_name)
}
else if (format == 1)
{
+ struct cache_extension_all_loaded ext;
+ if (!cache_extension_load (cache_new, cache, cache_size, &ext))
+ error (EXIT_FAILURE, 0,
+ _("Malformed extension data in cache file %s\n"), cache_name);
+
printf (_("%d libs found in cache `%s'\n"),
cache_new->nlibs, cache_name);
@@ -260,6 +282,7 @@ print_cache (const char *cache_name)
cache_new->libs[i].osversion,
cache_new->libs[i].hwcap,
cache_data + cache_new->libs[i].value);
+ print_extensions (&ext);
}
/* Cleanup. */
munmap (cache, cache_size);
@@ -301,6 +324,45 @@ compare (const struct cache_entry *e1, const struct cache_entry *e2)
return res;
}
+/* Size of the cache extension directory. All tags are assumed to be
+ present. */
+enum
+ {
+ cache_extension_size = (offsetof (struct cache_extension, sections)
+ + (cache_extension_count
+ * sizeof (struct cache_extension_section)))
+ };
+
+/* Write the cache extensions to FD. The extension directory is
+ assumed to be located at CACHE_EXTENSION_OFFSET. */
+static void
+write_extensions (int fd, uint32_t cache_extension_offset)
+{
+ assert ((cache_extension_offset % 4) == 0);
+
+ struct cache_extension *ext = xmalloc (cache_extension_size);
+ ext->magic = cache_extension_magic;
+ ext->count = cache_extension_count;
+
+ for (int i = 0; i < cache_extension_count; ++i)
+ {
+ ext->sections[i].tag = i;
+ ext->sections[i].flags = 0;
+ }
+
+ const char *generator
+ = "ldconfig " PKGVERSION RELEASE " release version " VERSION;
+ ext->sections[cache_extension_tag_generator].offset
+ = cache_extension_offset + cache_extension_size;
+ ext->sections[cache_extension_tag_generator].size = strlen (generator);
+
+ if (write (fd, ext, cache_extension_size) != cache_extension_size
+ || write (fd, generator, strlen (generator)) != strlen (generator))
+ error (EXIT_FAILURE, errno, _("Writing of cache extension data failed"));
+
+ free (ext);
+}
+
/* Save the contents of the cache. */
void
save_cache (const char *cache_name)
@@ -435,6 +497,25 @@ save_cache (const char *cache_name)
&& idx_old < cache_entry_old_count)
file_entries->libs[idx_old] = file_entries->libs[idx_old - 1];
+ /* Compute the location of the extension directory. This
+ implementation puts the directory after the string table. The
+ size computation matches the write calls below. The extension
+ directory does not exist with format 0, so the value does not
+ matter. */
+ uint32_t extension_offset = 0;
+ if (opt_format != opt_format_new)
+ extension_offset += file_entries_size;
+ if (opt_format != opt_format_old)
+ {
+ if (opt_format != opt_format_new)
+ extension_offset += pad;
+ extension_offset += file_entries_new_size;
+ }
+ extension_offset += total_strlen;
+ extension_offset = roundup (extension_offset, 4); /* Provide alignment. */
+ if (opt_format != opt_format_old)
+ file_entries_new->extension_offset = extension_offset;
+
/* Write out the cache. */
/* Write cache first to a temporary file and rename it later. */
@@ -473,6 +554,14 @@ save_cache (const char *cache_name)
if (write (fd, strings, total_strlen) != (ssize_t) total_strlen)
error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
+ if (opt_format != opt_format_old)
+ {
+ /* Align file position to 4. */
+ off64_t old_offset = lseek64 (fd, extension_offset, SEEK_SET);
+ assert ((unsigned long long int) (extension_offset - old_offset) < 4);
+ write_extensions (fd, extension_offset);
+ }
+
/* Make sure user can always read cache file */
if (chmod (temp_name, S_IROTH|S_IRGRP|S_IRUSR|S_IWUSR))
error (EXIT_FAILURE, errno,
diff --git a/sysdeps/generic/dl-cache.h b/sysdeps/generic/dl-cache.h
index 46026e0988..cdc24c8009 100644
--- a/sysdeps/generic/dl-cache.h
+++ b/sysdeps/generic/dl-cache.h
@@ -21,7 +21,9 @@
#include <endian.h>
#include <stdbool.h>
+#include <stddef.h>
#include <stdint.h>
+#include <string.h>
#ifndef _DL_CACHE_DEFAULT_ID
# define _DL_CACHE_DEFAULT_ID 3
@@ -142,7 +144,11 @@ struct cache_file_new
uint8_t padding_unsed[3]; /* Not used, for future extensions. */
- uint32_t unused[4]; /* Leave space for future extensions
+ /* File offset of the extension directory. See struct
+ cache_extension below. Must be a multiple of four. */
+ uint32_t extension_offset;
+
+ uint32_t unused[3]; /* Leave space for future extensions
and align to 8 byte boundary. */
struct file_entry_new libs[0]; /* Entries describing libraries. */
/* After this the string table of size len_strings is found. */
@@ -162,6 +168,121 @@ cache_file_new_matches_endian (const struct cache_file_new *cache)
}
+/* Randomly chosen magic value, which allows for additional
+ consistency verification. */
+enum { cache_extension_magic = (uint32_t) -358342284 };
+
+/* Tag values for different kinds of extension sections. Similar to
+ SHT_* constants. */
+enum cache_extension_tag
+ {
+ /* Array of bytes containing the glibc version that generated this
+ cache file. */
+ cache_extension_tag_generator,
+
+ /* Total number of known cache extension tags. */
+ cache_extension_count
+ };
+
+/* Element in the array following struct cache_extension. Similar to
+ an ELF section header. */
+struct cache_extension_section
+{
+ /* Type of the extension section. A enum cache_extension_tag value. */
+ uint32_t tag;
+
+ /* Extension-specific flags. Currently generated as zero. */
+ uint32_t flags;
+
+ /* Offset from the start of the file for the data in this extension
+ section. Specific extensions can have alignment constraints. */
+ uint32_t offset;
+
+ /* Length in bytes of the extension data. Specific extensions may
+ have size requirements. */
+ uint32_t size;
+};
+
+/* The extension directory in the cache. An array of struct
+ cache_extension_section entries. */
+struct cache_extension
+{
+ uint32_t magic; /* Always cache_extension_magic. */
+ uint32_t count; /* Number of following entries. */
+
+ /* count section descriptors of type struct cache_extension_section
+ follow. */
+ struct cache_extension_section sections[];
+};
+
+/* A relocated version of struct cache_extension_section. */
+struct cache_extension_loaded
+{
+ /* Address and size of this extension section. base is NULL if the
+ section is missing from the file. */
+ const void *base;
+ size_t size;
+
+ /* Flags from struct cache_extension_section. */
+ uint32_t flags;
+};
+
+/* All supported extension sections, relocated. Filled in by
+ cache_extension_load below. */
+struct cache_extension_all_loaded
+{
+ struct cache_extension_loaded sections[cache_extension_count];
+};
+
+static bool __attribute__ ((unused))
+cache_extension_load (const struct cache_file_new *cache,
+ const void *file_base, size_t file_size,
+ struct cache_extension_all_loaded *loaded)
+{
+ memset (loaded, 0, sizeof (*loaded));
+ if (cache->extension_offset == 0)
+ /* No extensions present. This is not a format error. */
+ return true;
+ if ((cache->extension_offset % 4) != 0)
+ /* Extension offset is misaligned. */
+ return false;
+ size_t size_tmp;
+ if (__builtin_add_overflow (cache->extension_offset,
+ sizeof (struct cache_extension), &size_tmp)
+ || size_tmp > file_size)
+ /* Extension extends beyond the end of the file. */
+ return false;
+ const struct cache_extension *ext = file_base + cache->extension_offset;
+ if (ext->magic != cache_extension_magic)
+ return false;
+ if (__builtin_mul_overflow (ext->count,
+ sizeof (struct cache_extension_section),
+ &size_tmp)
+ || __builtin_add_overflow (cache->extension_offset
+ + sizeof (struct cache_extension), size_tmp,
+ &size_tmp)
+ || size_tmp > file_size)
+ /* Extension array extends beyond the end of the file. */
+ return false;
+ for (uint32_t i = 0; i < ext->count; ++i)
+ {
+ if (__builtin_add_overflow (ext->sections[i].offset,
+ ext->sections[i].size, &size_tmp)
+ || size_tmp > file_size)
+ /* Extension data extends beyond the end of the file. */
+ return false;
+
+ uint32_t tag = ext->sections[i].tag;
+ if (tag >= cache_extension_count)
+ /* Tag is out of range and unrecognized. */
+ continue;
+ loaded->sections[tag].base = file_base + ext->sections[i].offset;
+ loaded->sections[tag].size = ext->sections[i].size;
+ loaded->sections[tag].flags = ext->sections[i].flags;
+ }
+ return true;
+}
+
/* Used to align cache_file_new. */
#define ALIGN_CACHE(addr) \
(((addr) + __alignof__ (struct cache_file_new) -1) \
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 05/11] elf: Add extension mechanism to ld.so.cache
2020-11-09 18:40 ` [PATCH 05/11] elf: Add extension mechanism " Florian Weimer
@ 2020-11-27 18:01 ` Adhemerval Zanella
2020-11-27 18:55 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-27 18:01 UTC (permalink / raw)
To: Florian Weimer, libc-alpha
On 09/11/2020 15:40, Florian Weimer via Libc-alpha wrote:
> A previously unused new-format header field is used to record
> the address of an extension directory.
>
> This change adds a demo extension which records the version of
> ldconfig which builds a file.
LGTM, thanks.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
> elf/cache.c | 89 +++++++++++++++++++++++++++
> sysdeps/generic/dl-cache.h | 123 ++++++++++++++++++++++++++++++++++++-
> 2 files changed, 211 insertions(+), 1 deletion(-)
>
> diff --git a/elf/cache.c b/elf/cache.c
> index ffecbe6d82..549e04ce21 100644
> --- a/elf/cache.c
> +++ b/elf/cache.c
> @@ -15,6 +15,7 @@
> You should have received a copy of the GNU General Public License
> along with this program; if not, see <https://www.gnu.org/licenses/>. */
>
> +#include <assert.h>
> #include <errno.h>
> #include <error.h>
> #include <dirent.h>
> @@ -33,6 +34,7 @@
>
> #include <ldconfig.h>
> #include <dl-cache.h>
> +#include <version.h>
>
> struct cache_entry
> {
> @@ -161,6 +163,21 @@ check_new_cache (struct cache_file_new *cache)
> error (EXIT_FAILURE, 0, _("Cache file has wrong endianness.\n"));
> }
>
> +/* Print the extension information at the cache at start address
> + FILE_BASE, of length FILE_SIZE bytes. The new-format cache header
> + is at CACHE, and the file name for diagnostics is CACHE_NAME. */
> +static void
> +print_extensions (struct cache_extension_all_loaded *ext)
> +{
> + if (ext->sections[cache_extension_tag_generator].base != NULL)
> + {
> + fputs (_("Cache generated by: "), stdout);
> + fwrite (ext->sections[cache_extension_tag_generator].base, 1,
> + ext->sections[cache_extension_tag_generator].size, stdout);
> + putchar ('\n');
> + }
> +}
> +
> /* Print the whole cache file, if a file contains the new cache format
> hidden in the old one, print the contents of the new format. */
> void
Ok.
> @@ -250,6 +267,11 @@ print_cache (const char *cache_name)
> }
> else if (format == 1)
> {
> + struct cache_extension_all_loaded ext;
> + if (!cache_extension_load (cache_new, cache, cache_size, &ext))
> + error (EXIT_FAILURE, 0,
> + _("Malformed extension data in cache file %s\n"), cache_name);
> +
> printf (_("%d libs found in cache `%s'\n"),
> cache_new->nlibs, cache_name);
>
Ok.
> @@ -260,6 +282,7 @@ print_cache (const char *cache_name)
> cache_new->libs[i].osversion,
> cache_new->libs[i].hwcap,
> cache_data + cache_new->libs[i].value);
> + print_extensions (&ext);
> }
> /* Cleanup. */
> munmap (cache, cache_size);
Ok.
> @@ -301,6 +324,45 @@ compare (const struct cache_entry *e1, const struct cache_entry *e2)
> return res;
> }
>
> +/* Size of the cache extension directory. All tags are assumed to be
> + present. */
> +enum
> + {
> + cache_extension_size = (offsetof (struct cache_extension, sections)
> + + (cache_extension_count
> + * sizeof (struct cache_extension_section)))
> + };
> +
> +/* Write the cache extensions to FD. The extension directory is
> + assumed to be located at CACHE_EXTENSION_OFFSET. */
> +static void
> +write_extensions (int fd, uint32_t cache_extension_offset)
> +{
> + assert ((cache_extension_offset % 4) == 0);
> +
> + struct cache_extension *ext = xmalloc (cache_extension_size);
> + ext->magic = cache_extension_magic;
> + ext->count = cache_extension_count;
> +
> + for (int i = 0; i < cache_extension_count; ++i)
> + {
> + ext->sections[i].tag = i;
> + ext->sections[i].flags = 0;
> + }
> +
> + const char *generator
> + = "ldconfig " PKGVERSION RELEASE " release version " VERSION;
> + ext->sections[cache_extension_tag_generator].offset
> + = cache_extension_offset + cache_extension_size;
> + ext->sections[cache_extension_tag_generator].size = strlen (generator);
> +
> + if (write (fd, ext, cache_extension_size) != cache_extension_size
> + || write (fd, generator, strlen (generator)) != strlen (generator))
> + error (EXIT_FAILURE, errno, _("Writing of cache extension data failed"));
> +
> + free (ext);
> +}
> +
> /* Save the contents of the cache. */
> void
> save_cache (const char *cache_name)
Ok.
> @@ -435,6 +497,25 @@ save_cache (const char *cache_name)
> && idx_old < cache_entry_old_count)
> file_entries->libs[idx_old] = file_entries->libs[idx_old - 1];
>
> + /* Compute the location of the extension directory. This
> + implementation puts the directory after the string table. The
> + size computation matches the write calls below. The extension
> + directory does not exist with format 0, so the value does not
> + matter. */
> + uint32_t extension_offset = 0;
> + if (opt_format != opt_format_new)
> + extension_offset += file_entries_size;
> + if (opt_format != opt_format_old)
> + {
> + if (opt_format != opt_format_new)
> + extension_offset += pad;
> + extension_offset += file_entries_new_size;
> + }
> + extension_offset += total_strlen;
> + extension_offset = roundup (extension_offset, 4); /* Provide alignment. */
> + if (opt_format != opt_format_old)
> + file_entries_new->extension_offset = extension_offset;
> +
> /* Write out the cache. */
>
> /* Write cache first to a temporary file and rename it later. */
Ok.
> @@ -473,6 +554,14 @@ save_cache (const char *cache_name)
> if (write (fd, strings, total_strlen) != (ssize_t) total_strlen)
> error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
>
> + if (opt_format != opt_format_old)
> + {
> + /* Align file position to 4. */
> + off64_t old_offset = lseek64 (fd, extension_offset, SEEK_SET);
> + assert ((unsigned long long int) (extension_offset - old_offset) < 4);
Maybe off64_t cast here?
> + write_extensions (fd, extension_offset);
> + }
> +
> /* Make sure user can always read cache file */
> if (chmod (temp_name, S_IROTH|S_IRGRP|S_IRUSR|S_IWUSR))
> error (EXIT_FAILURE, errno,
Ok.
> diff --git a/sysdeps/generic/dl-cache.h b/sysdeps/generic/dl-cache.h
> index 46026e0988..cdc24c8009 100644
> --- a/sysdeps/generic/dl-cache.h
> +++ b/sysdeps/generic/dl-cache.h
> @@ -21,7 +21,9 @@
>
> #include <endian.h>
> #include <stdbool.h>
> +#include <stddef.h>
> #include <stdint.h>
> +#include <string.h>
>
> #ifndef _DL_CACHE_DEFAULT_ID
> # define _DL_CACHE_DEFAULT_ID 3
> @@ -142,7 +144,11 @@ struct cache_file_new
>
> uint8_t padding_unsed[3]; /* Not used, for future extensions. */
>
> - uint32_t unused[4]; /* Leave space for future extensions
> + /* File offset of the extension directory. See struct
> + cache_extension below. Must be a multiple of four. */
> + uint32_t extension_offset;
> +
> + uint32_t unused[3]; /* Leave space for future extensions
> and align to 8 byte boundary. */
> struct file_entry_new libs[0]; /* Entries describing libraries. */
> /* After this the string table of size len_strings is found. */
Ok (32-bit should be more than enough).
> @@ -162,6 +168,121 @@ cache_file_new_matches_endian (const struct cache_file_new *cache)
> }
>
>
> +/* Randomly chosen magic value, which allows for additional
> + consistency verification. */
> +enum { cache_extension_magic = (uint32_t) -358342284 };
> +
> +/* Tag values for different kinds of extension sections. Similar to
> + SHT_* constants. */
> +enum cache_extension_tag
> + {
> + /* Array of bytes containing the glibc version that generated this
> + cache file. */
> + cache_extension_tag_generator,
> +
> + /* Total number of known cache extension tags. */
> + cache_extension_count
> + };
> +
> +/* Element in the array following struct cache_extension. Similar to
> + an ELF section header. */
> +struct cache_extension_section
> +{
> + /* Type of the extension section. A enum cache_extension_tag value. */
> + uint32_t tag;
> +
> + /* Extension-specific flags. Currently generated as zero. */
> + uint32_t flags;
> +
> + /* Offset from the start of the file for the data in this extension
> + section. Specific extensions can have alignment constraints. */
> + uint32_t offset;
> +
> + /* Length in bytes of the extension data. Specific extensions may
> + have size requirements. */
> + uint32_t size;
> +};
> +
Ok.
> +/* The extension directory in the cache. An array of struct
> + cache_extension_section entries. */
> +struct cache_extension
> +{
> + uint32_t magic; /* Always cache_extension_magic. */
> + uint32_t count; /* Number of following entries. */
> +
> + /* count section descriptors of type struct cache_extension_section
> + follow. */
> + struct cache_extension_section sections[];
> +};
> +
> +/* A relocated version of struct cache_extension_section. */
> +struct cache_extension_loaded
> +{
> + /* Address and size of this extension section. base is NULL if the
> + section is missing from the file. */
> + const void *base;
> + size_t size;
> +
> + /* Flags from struct cache_extension_section. */
> + uint32_t flags;
> +};
> +
> +/* All supported extension sections, relocated. Filled in by
> + cache_extension_load below. */
> +struct cache_extension_all_loaded
> +{
> + struct cache_extension_loaded sections[cache_extension_count];
> +};
> +
Ok.
> +static bool __attribute__ ((unused))
> +cache_extension_load (const struct cache_file_new *cache,
> + const void *file_base, size_t file_size,
> + struct cache_extension_all_loaded *loaded)
> +{
> + memset (loaded, 0, sizeof (*loaded));
> + if (cache->extension_offset == 0)
> + /* No extensions present. This is not a format error. */
> + return true;
> + if ((cache->extension_offset % 4) != 0)
> + /* Extension offset is misaligned. */
> + return false;
> + size_t size_tmp;
> + if (__builtin_add_overflow (cache->extension_offset,
> + sizeof (struct cache_extension), &size_tmp)
> + || size_tmp > file_size)
> + /* Extension extends beyond the end of the file. */
> + return false;
> + const struct cache_extension *ext = file_base + cache->extension_offset;
> + if (ext->magic != cache_extension_magic)
> + return false;
> + if (__builtin_mul_overflow (ext->count,
> + sizeof (struct cache_extension_section),
> + &size_tmp)
> + || __builtin_add_overflow (cache->extension_offset
> + + sizeof (struct cache_extension), size_tmp,
> + &size_tmp)
> + || size_tmp > file_size)
> + /* Extension array extends beyond the end of the file. */
> + return false;
Ok.
> + for (uint32_t i = 0; i < ext->count; ++i)
> + {
> + if (__builtin_add_overflow (ext->sections[i].offset,
> + ext->sections[i].size, &size_tmp)
> + || size_tmp > file_size)
> + /* Extension data extends beyond the end of the file. */
> + return false;
> +
> + uint32_t tag = ext->sections[i].tag;
> + if (tag >= cache_extension_count)
> + /* Tag is out of range and unrecognized. */
> + continue;
> + loaded->sections[tag].base = file_base + ext->sections[i].offset;
> + loaded->sections[tag].size = ext->sections[i].size;
> + loaded->sections[tag].flags = ext->sections[i].flags;
> + }
> + return true;
> +}
> +
> /* Used to align cache_file_new. */
> #define ALIGN_CACHE(addr) \
> (((addr) + __alignof__ (struct cache_file_new) -1) \
>
Ok.
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 05/11] elf: Add extension mechanism to ld.so.cache
2020-11-27 18:01 ` Adhemerval Zanella
@ 2020-11-27 18:55 ` Florian Weimer
2020-11-27 18:56 ` Adhemerval Zanella
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-27 18:55 UTC (permalink / raw)
To: Adhemerval Zanella via Libc-alpha
* Adhemerval Zanella via Libc-alpha:
>> @@ -473,6 +554,14 @@ save_cache (const char *cache_name)
>> if (write (fd, strings, total_strlen) != (ssize_t) total_strlen)
>> error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
>>
>> + if (opt_format != opt_format_old)
>> + {
>> + /* Align file position to 4. */
>> + off64_t old_offset = lseek64 (fd, extension_offset, SEEK_SET);
>> + assert ((unsigned long long int) (extension_offset - old_offset) < 4);
>
> Maybe off64_t cast here?
That would lose the implied check for a non-negative value because
off64_t is signed.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 05/11] elf: Add extension mechanism to ld.so.cache
2020-11-27 18:55 ` Florian Weimer
@ 2020-11-27 18:56 ` Adhemerval Zanella
0 siblings, 0 replies; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-27 18:56 UTC (permalink / raw)
To: Florian Weimer, Adhemerval Zanella via Libc-alpha
On 27/11/2020 15:55, Florian Weimer wrote:
> * Adhemerval Zanella via Libc-alpha:
>
>>> @@ -473,6 +554,14 @@ save_cache (const char *cache_name)
>>> if (write (fd, strings, total_strlen) != (ssize_t) total_strlen)
>>> error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
>>>
>>> + if (opt_format != opt_format_old)
>>> + {
>>> + /* Align file position to 4. */
>>> + off64_t old_offset = lseek64 (fd, extension_offset, SEEK_SET);
>>> + assert ((unsigned long long int) (extension_offset - old_offset) < 4);
>>
>> Maybe off64_t cast here?
>
> That would lose the implied check for a non-negative value because
> off64_t is signed.
Indeed, unsigned long long int is ok then.
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 06/11] elf: Implement a string table for ldconfig, with tail merging
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (4 preceding siblings ...)
2020-11-09 18:40 ` [PATCH 05/11] elf: Add extension mechanism " Florian Weimer
@ 2020-11-09 18:40 ` Florian Weimer
2020-11-27 19:29 ` Adhemerval Zanella
2020-11-09 18:41 ` [PATCH 07/11] elf: Implement tail merging of strings in ldconfig Florian Weimer
` (6 subsequent siblings)
12 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:40 UTC (permalink / raw)
To: libc-alpha
This will be used in ldconfig to reduce the ld.so.cache size slightly.
Tail merging is an optimization where a pointer points into another
string if the first string is a suffix of the second string.
The hash function FNV-1a was chosen because it is simple and achieves
good dispersion even for short strings (so that the hash table bucket
count can be a power of two). It is clearly superior to the hsearch
hash and the ELF hash in this regard.
The hash table uses chaining for collision resolution.
---
elf/Makefile | 2 +-
elf/stringtable.c | 209 +++++++++++++++++++++++++++++++++++++++++
elf/stringtable.h | 64 +++++++++++++
elf/stringtable_free.c | 33 +++++++
elf/tst-stringtable.c | 141 +++++++++++++++++++++++++++
5 files changed, 448 insertions(+), 1 deletion(-)
create mode 100644 elf/stringtable.c
create mode 100644 elf/stringtable.h
create mode 100644 elf/stringtable_free.c
create mode 100644 elf/tst-stringtable.c
diff --git a/elf/Makefile b/elf/Makefile
index d2f7e99863..5a8f116a67 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -172,7 +172,7 @@ tests-container := \
tests := tst-tls9 tst-leaks1 \
tst-array1 tst-array2 tst-array3 tst-array4 tst-array5 \
- tst-auxv
+ tst-auxv tst-stringtable
tests-internal := tst-tls1 tst-tls2 $(tests-static-internal)
tests-static := $(tests-static-normal) $(tests-static-internal)
diff --git a/elf/stringtable.c b/elf/stringtable.c
new file mode 100644
index 0000000000..e5b38148a6
--- /dev/null
+++ b/elf/stringtable.c
@@ -0,0 +1,209 @@
+/* String tables for ld.so.cache construction. Implementation.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published
+ by the Free Software Foundation; version 2 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, see <https://www.gnu.org/licenses/>. */
+
+#include <assert.h>
+#include <error.h>
+#include <ldconfig.h>
+#include <libintl.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stringtable.h>
+
+static void
+stringtable_init (struct stringtable *table)
+{
+ table->count = 0;
+
+ /* This needs to be a power of two. 128 is sufficient to keep track
+ of 42 DSOs without resizing (assuming two strings per DSOs).
+ glibc itself comes with more than 20 DSOs, so 64 would likely to
+ be too small. */
+ table->allocated = 64;
+
+ table->entries = xcalloc (table->allocated, sizeof (table->entries[0]));
+}
+
+/* 32-bit FNV-1a hash function. */
+static uint32_t
+fnv1a (const char *string, size_t length)
+{
+ const unsigned char *p = (const unsigned char *) string;
+ uint32_t hash = 2166136261U;
+ for (size_t i = 0; i < length; ++i)
+ {
+ hash ^= p[i];
+ hash *= 16777619U;
+ }
+ return hash;
+}
+
+/* Double the capacity of the hash table. */
+static void
+stringtable_rehash (struct stringtable *table)
+{
+ /* Cannot overflow because the old allocation size (in bytes) is
+ larger. */
+ uint32_t new_allocated = table->allocated * 2;
+ struct stringtable_entry **new_entries
+ = xcalloc (new_allocated, sizeof (table->entries[0]));
+
+ uint32_t mask = new_allocated - 1;
+ for (uint32_t i = 0; i < table->allocated; ++i)
+ for (struct stringtable_entry *e = table->entries[i]; e != NULL; )
+ {
+ struct stringtable_entry *next = e->next;
+ uint32_t hash = fnv1a (e->string, e->length);
+ uint32_t new_index = hash & mask;
+ e->next = new_entries[new_index];
+ new_entries[new_index] = e;
+ e = next;
+ }
+
+ free (table->entries);
+ table->entries = new_entries;
+ table->allocated = new_allocated;
+}
+
+struct stringtable_entry *
+stringtable_add (struct stringtable *table, const char *string)
+{
+ /* Check for a zero-initialized table. */
+ if (table->allocated == 0)
+ stringtable_init (table);
+
+ size_t length = strlen (string);
+ if (length > (1U << 30))
+ error (EXIT_FAILURE, 0, _("String table string is too long"));
+ uint32_t hash = fnv1a (string, length);
+
+ /* Return a previously-existing entry. */
+ for (struct stringtable_entry *e
+ = table->entries[hash & (table->allocated - 1)];
+ e != NULL; e = e->next)
+ if (e->length == length && memcmp (e->string, string, length) == 0)
+ return e;
+
+ /* Increase the size of the table if necessary. Keep utilization
+ below two thirds. */
+ if (table->count >= (1U << 30))
+ error (EXIT_FAILURE, 0, _("String table has too many entries"));
+ if (table->count * 3 > table->allocated * 2)
+ stringtable_rehash (table);
+
+ /* Add the new table entry. */
+ ++table->count;
+ struct stringtable_entry *e
+ = xmalloc (offsetof (struct stringtable_entry, string) + length + 1);
+ uint32_t index = hash & (table->allocated - 1);
+ e->next = table->entries[index];
+ table->entries[index] = e;
+ e->length = length;
+ e->offset = 0;
+ memcpy (e->string, string, length + 1);
+ return e;
+}
+
+/* Sort reversed strings in lexicographic order. This is used for tail
+ merging. */
+static int
+finalize_compare (const void *l, const void *r)
+{
+ struct stringtable_entry *left = *(struct stringtable_entry **) l;
+ struct stringtable_entry *right = *(struct stringtable_entry **) r;
+ size_t to_compare;
+ if (left->length < right->length)
+ to_compare = left->length;
+ else
+ to_compare = right->length;
+ for (ssize_t i = to_compare - 1; i >= 0; --i)
+ {
+ unsigned char lch = left->string[i];
+ unsigned char rch = right->string[i];
+ if (lch != rch)
+ return lch - rch;
+ }
+ if (left->length == right->length)
+ return 0;
+ else if (left->length < right->length)
+ /* Longer strings should come first. */
+ return 1;
+ else
+ return -1;
+}
+
+void
+stringtable_finalize (struct stringtable *table,
+ struct stringtable_finalized *result)
+{
+ if (table->count == 0)
+ {
+ result->strings = xstrdup ("");
+ result->size = 0;
+ return;
+ }
+
+ /* Optimize the order of the strings. */
+ struct stringtable_entry **array = xcalloc (table->count, sizeof (*array));
+ {
+ size_t j = 0;
+ for (uint32_t i = 0; i < table->allocated; ++i)
+ for (struct stringtable_entry *e = table->entries[i]; e != NULL;
+ e = e->next)
+ {
+ array[j] = e;
+ ++j;
+ }
+ assert (j == table->count);
+ }
+ qsort (array, table->count, sizeof (*array), finalize_compare);
+
+ /* Assign offsets, using tail merging (sharing suffixes) if possible. */
+ array[0]->offset = 0;
+ for (uint32_t j = 1; j < table->count; ++j)
+ {
+ struct stringtable_entry *previous = array[j - 1];
+ struct stringtable_entry *current = array[j];
+ if (previous->length >= current->length
+ && memcmp (&previous->string[previous->length - current->length],
+ current->string, current->length) == 0)
+ current->offset = (previous->offset + previous->length
+ - current->length);
+ else if (__builtin_add_overflow (previous->offset,
+ previous->length + 1,
+ ¤t->offset))
+ error (EXIT_FAILURE, 0, _("String table is too large"));
+ }
+
+ /* Allocate the result string. */
+ {
+ struct stringtable_entry *last = array[table->count - 1];
+ if (__builtin_add_overflow (last->offset, last->length + 1,
+ &result->size))
+ error (EXIT_FAILURE, 0, _("String table is too large"));
+ }
+ /* The strings are copied from the hash table, so the array is no
+ longer needed. */
+ free (array);
+ result->strings = xcalloc (result->size, 1);
+
+ /* Copy the strings. */
+ for (uint32_t i = 0; i < table->allocated; ++i)
+ for (struct stringtable_entry *e = table->entries[i]; e != NULL;
+ e = e->next)
+ if (result->strings[e->offset] == '\0')
+ memcpy (&result->strings[e->offset], e->string, e->length + 1);
+}
diff --git a/elf/stringtable.h b/elf/stringtable.h
new file mode 100644
index 0000000000..7d57d1bda9
--- /dev/null
+++ b/elf/stringtable.h
@@ -0,0 +1,64 @@
+/* String tables for ld.so.cache construction.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published
+ by the Free Software Foundation; version 2 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, see <https://www.gnu.org/licenses/>. */
+
+#ifndef _STRINGTABLE_H
+#define _STRINGTABLE_H
+
+#include <stddef.h>
+#include <stdint.h>
+
+/* An entry in the string table. Only the length and string fields are
+ expected to be used outside the string table code. */
+struct stringtable_entry
+{
+ struct stringtable_entry *next; /* For collision resolution. */
+ uint32_t length; /* Length of then string. */
+ uint32_t offset; /* From start of finalized table. */
+ char string[]; /* Null-terminated string. */
+};
+
+/* A string table. Zero-initialization produces a valid atable. */
+struct stringtable
+{
+ struct stringtable_entry **entries; /* Array of hash table buckets. */
+ uint32_t count; /* Number of elements in the table. */
+ uint32_t allocated; /* Length of the entries array. */
+};
+
+/* Adds STRING to TABLE. May return the address of an existing entry. */
+struct stringtable_entry *stringtable_add (struct stringtable *table,
+ const char *string);
+
+/* Result of stringtable_finalize. SIZE bytes at STRINGS should be
+ written to the file. */
+struct stringtable_finalized
+{
+ char *strings;
+ size_t size;
+};
+
+/* Assigns offsets to string table entries and computes the serialized
+ form of the string table. */
+void stringtable_finalize (struct stringtable *table,
+ struct stringtable_finalized *result);
+
+/* Deallocate the string table (but not the TABLE pointer itself).
+ (The table can be re-used for adding more strings without
+ initialization.) */
+void stringtable_free (struct stringtable *table);
+
+#endif /* _STRINGTABLE_H */
diff --git a/elf/stringtable_free.c b/elf/stringtable_free.c
new file mode 100644
index 0000000000..8588a25470
--- /dev/null
+++ b/elf/stringtable_free.c
@@ -0,0 +1,33 @@
+/* String tables for ld.so.cache construction. Deallocation (for tests only).
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published
+ by the Free Software Foundation; version 2 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, see <https://www.gnu.org/licenses/>. */
+
+#include <stdlib.h>
+#include <stringtable.h>
+
+void
+stringtable_free (struct stringtable *table)
+{
+ for (uint32_t i = 0; i < table->allocated; ++i)
+ for (struct stringtable_entry *e = table->entries[i]; e != NULL; )
+ {
+ struct stringtable_entry *next = e->next;
+ free (e);
+ e = next;
+ }
+ free (table->entries);
+ *table = (struct stringtable) { 0, };
+}
diff --git a/elf/tst-stringtable.c b/elf/tst-stringtable.c
new file mode 100644
index 0000000000..2d044b7151
--- /dev/null
+++ b/elf/tst-stringtable.c
@@ -0,0 +1,141 @@
+/* Unit test for ldconfig string tables.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published
+ by the Free Software Foundation; version 2 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, see <https://www.gnu.org/licenses/>. */
+
+#include <stdlib.h>
+#include <string.h>
+#include <stringtable.h>
+#include <support/check.h>
+#include <support/support.h>
+
+static int
+do_test (void)
+{
+ /* Empty string table. */
+ {
+ struct stringtable s = { 0, };
+ struct stringtable_finalized f;
+ stringtable_finalize (&s, &f);
+ TEST_COMPARE_STRING (f.strings, "");
+ TEST_COMPARE (f.size, 0);
+ free (f.strings);
+ stringtable_free (&s);
+ }
+
+ /* String table with one empty string. */
+ {
+ struct stringtable s = { 0, };
+ struct stringtable_entry *e = stringtable_add (&s, "");
+ TEST_COMPARE_STRING (e->string, "");
+ TEST_COMPARE (e->length, 0);
+ TEST_COMPARE (s.count, 1);
+
+ struct stringtable_finalized f;
+ stringtable_finalize (&s, &f);
+ TEST_COMPARE (e->offset, 0);
+ TEST_COMPARE_STRING (f.strings, "");
+ TEST_COMPARE (f.size, 1);
+ free (f.strings);
+ stringtable_free (&s);
+ }
+
+ /* String table with one non-empty string. */
+ {
+ struct stringtable s = { 0, };
+ struct stringtable_entry *e = stringtable_add (&s, "name");
+ TEST_COMPARE_STRING (e->string, "name");
+ TEST_COMPARE (e->length, 4);
+ TEST_COMPARE (s.count, 1);
+
+ struct stringtable_finalized f;
+ stringtable_finalize (&s, &f);
+ TEST_COMPARE (e->offset, 0);
+ TEST_COMPARE_STRING (f.strings, "name");
+ TEST_COMPARE (f.size, 5);
+ free (f.strings);
+ stringtable_free (&s);
+ }
+
+ /* Two strings, one is a prefix of the other. Tail-merging can only
+ happen in one way in this case. */
+ {
+ struct stringtable s = { 0, };
+ struct stringtable_entry *suffix = stringtable_add (&s, "suffix");
+ TEST_COMPARE_STRING (suffix->string, "suffix");
+ TEST_COMPARE (suffix->length, 6);
+ TEST_COMPARE (s.count, 1);
+
+ struct stringtable_entry *prefix
+ = stringtable_add (&s, "prefix-suffix");
+ TEST_COMPARE_STRING (prefix->string, "prefix-suffix");
+ TEST_COMPARE (prefix->length, strlen ("prefix-suffix"));
+ TEST_COMPARE (s.count, 2);
+
+ struct stringtable_finalized f;
+ stringtable_finalize (&s, &f);
+ TEST_COMPARE (prefix->offset, 0);
+ TEST_COMPARE (suffix->offset, strlen ("prefix-"));
+ TEST_COMPARE_STRING (f.strings, "prefix-suffix");
+ TEST_COMPARE (f.size, sizeof ("prefix-suffix"));
+ free (f.strings);
+ stringtable_free (&s);
+ }
+
+ /* String table with various shared prefixes. Triggers hash
+ resizing. */
+ {
+ enum { count = 1500 };
+ char *strings[2 * count];
+ struct stringtable_entry *entries[2 * count];
+ struct stringtable s = { 0, };
+ for (int i = 0; i < count; ++i)
+ {
+ strings[i] = xasprintf ("%d", i);
+ entries[i] = stringtable_add (&s, strings[i]);
+ TEST_COMPARE (entries[i]->length, strlen (strings[i]));
+ TEST_COMPARE_STRING (entries[i]->string, strings[i]);
+ strings[i + count] = xasprintf ("prefix/%d", i);
+ entries[i + count] = stringtable_add (&s, strings[i + count]);
+ TEST_COMPARE (entries[i + count]->length, strlen (strings[i + count]));
+ TEST_COMPARE_STRING (entries[i + count]->string, strings[i + count]);
+ }
+
+ struct stringtable_finalized f;
+ stringtable_finalize (&s, &f);
+
+ for (int i = 0; i < 2 * count; ++i)
+ {
+ TEST_COMPARE (entries[i]->length, strlen (strings[i]));
+ TEST_COMPARE_STRING (entries[i]->string, strings[i]);
+ TEST_COMPARE_STRING (f.strings + entries[i]->offset, strings[i]);
+ free (strings[i]);
+ }
+
+ free (f.strings);
+ stringtable_free (&s);
+ }
+
+ return 0;
+}
+
+#include <support/test-driver.c>
+
+/* Re-compile the string table implementation here. It is not
+ possible to link against the actual build because it was built for
+ use in ldconfig. */
+#define _(arg) arg
+#include "stringtable.c"
+#include "stringtable_free.c"
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 06/11] elf: Implement a string table for ldconfig, with tail merging
2020-11-09 18:40 ` [PATCH 06/11] elf: Implement a string table for ldconfig, with tail merging Florian Weimer
@ 2020-11-27 19:29 ` Adhemerval Zanella
2020-11-27 19:49 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-27 19:29 UTC (permalink / raw)
To: libc-alpha, Florian Weimer
On 09/11/2020 15:40, Florian Weimer via Libc-alpha wrote:
> This will be used in ldconfig to reduce the ld.so.cache size slightly.
>
> Tail merging is an optimization where a pointer points into another
> string if the first string is a suffix of the second string.
>
> The hash function FNV-1a was chosen because it is simple and achieves
> good dispersion even for short strings (so that the hash table bucket
> count can be a power of two). It is clearly superior to the hsearch
> hash and the ELF hash in this regard.
>
> The hash table uses chaining for collision resolution.
LGTM, with just a couple of comments below.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
> elf/Makefile | 2 +-
> elf/stringtable.c | 209 +++++++++++++++++++++++++++++++++++++++++
> elf/stringtable.h | 64 +++++++++++++
> elf/stringtable_free.c | 33 +++++++
> elf/tst-stringtable.c | 141 +++++++++++++++++++++++++++
> 5 files changed, 448 insertions(+), 1 deletion(-)
> create mode 100644 elf/stringtable.c
> create mode 100644 elf/stringtable.h
> create mode 100644 elf/stringtable_free.c
> create mode 100644 elf/tst-stringtable.c
>
> diff --git a/elf/Makefile b/elf/Makefile
> index d2f7e99863..5a8f116a67 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -172,7 +172,7 @@ tests-container := \
>
> tests := tst-tls9 tst-leaks1 \
> tst-array1 tst-array2 tst-array3 tst-array4 tst-array5 \
> - tst-auxv
> + tst-auxv tst-stringtable
> tests-internal := tst-tls1 tst-tls2 $(tests-static-internal)
> tests-static := $(tests-static-normal) $(tests-static-internal)
>
Ok.
> diff --git a/elf/stringtable.c b/elf/stringtable.c
> new file mode 100644
> index 0000000000..e5b38148a6
> --- /dev/null
> +++ b/elf/stringtable.c
> @@ -0,0 +1,209 @@
> +/* String tables for ld.so.cache construction. Implementation.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + This program is free software; you can redistribute it and/or modify
> + it under the terms of the GNU General Public License as published
> + by the Free Software Foundation; version 2 of the License, or
> + (at your option) any later version.
> +
> + This program is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + GNU General Public License for more details.
> +
> + You should have received a copy of the GNU General Public License
> + along with this program; if not, see <https://www.gnu.org/licenses/>. */
> +
> +#include <assert.h>
> +#include <error.h>
> +#include <ldconfig.h>
> +#include <libintl.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <stringtable.h>
> +
> +static void
> +stringtable_init (struct stringtable *table)
> +{
> + table->count = 0;
> +
> + /* This needs to be a power of two. 128 is sufficient to keep track
> + of 42 DSOs without resizing (assuming two strings per DSOs).
> + glibc itself comes with more than 20 DSOs, so 64 would likely to
> + be too small. */
Should we use a large value then? Asking because the comment indicates that
the chosen default value is not best suitable one.
> + table->allocated = 64;
> +
> + table->entries = xcalloc (table->allocated, sizeof (table->entries[0]));
> +}
> +
> +/* 32-bit FNV-1a hash function. */
> +static uint32_t
> +fnv1a (const char *string, size_t length)
> +{
> + const unsigned char *p = (const unsigned char *) string;
> + uint32_t hash = 2166136261U;
> + for (size_t i = 0; i < length; ++i)
> + {
> + hash ^= p[i];
> + hash *= 16777619U;
> + }
> + return hash;
> +}
> +
Ok.
> +/* Double the capacity of the hash table. */
> +static void
> +stringtable_rehash (struct stringtable *table)
> +{
> + /* Cannot overflow because the old allocation size (in bytes) is
> + larger. */
This comment is a bit confusing, I think it should be safe to assume
it won't overflow because stringtable_rehash won't be called for
large values (stringtable_add will bail early).
> + uint32_t new_allocated = table->allocated * 2;
> + struct stringtable_entry **new_entries
> + = xcalloc (new_allocated, sizeof (table->entries[0]));
> +
> + uint32_t mask = new_allocated - 1;
> + for (uint32_t i = 0; i < table->allocated; ++i)
> + for (struct stringtable_entry *e = table->entries[i]; e != NULL; )
> + {
> + struct stringtable_entry *next = e->next;
> + uint32_t hash = fnv1a (e->string, e->length);
> + uint32_t new_index = hash & mask;
> + e->next = new_entries[new_index];
> + new_entries[new_index] = e;
> + e = next;
> + }
> +
> + free (table->entries);
> + table->entries = new_entries;
> + table->allocated = new_allocated;
> +}
> +
Ok.
> +struct stringtable_entry *
> +stringtable_add (struct stringtable *table, const char *string)
> +{
> + /* Check for a zero-initialized table. */
> + if (table->allocated == 0)
> + stringtable_init (table);
> +
> + size_t length = strlen (string);
> + if (length > (1U << 30))
> + error (EXIT_FAILURE, 0, _("String table string is too long"));
> + uint32_t hash = fnv1a (string, length);
> +
> + /* Return a previously-existing entry. */
> + for (struct stringtable_entry *e
> + = table->entries[hash & (table->allocated - 1)];
> + e != NULL; e = e->next)
> + if (e->length == length && memcmp (e->string, string, length) == 0)
> + return e;
> +
> + /* Increase the size of the table if necessary. Keep utilization
> + below two thirds. */
> + if (table->count >= (1U << 30))
> + error (EXIT_FAILURE, 0, _("String table has too many entries"));
> + if (table->count * 3 > table->allocated * 2)
> + stringtable_rehash (table);
> +
> + /* Add the new table entry. */
> + ++table->count;
> + struct stringtable_entry *e
> + = xmalloc (offsetof (struct stringtable_entry, string) + length + 1);
> + uint32_t index = hash & (table->allocated - 1);
> + e->next = table->entries[index];
> + table->entries[index] = e;
> + e->length = length;
> + e->offset = 0;
> + memcpy (e->string, string, length + 1);
> + return e;
> +}
Ok.
> +
> +/* Sort reversed strings in lexicographic order. This is used for tail
> + merging. */
> +static int
> +finalize_compare (const void *l, const void *r)
> +{
> + struct stringtable_entry *left = *(struct stringtable_entry **) l;
> + struct stringtable_entry *right = *(struct stringtable_entry **) r;
> + size_t to_compare;
> + if (left->length < right->length)
> + to_compare = left->length;
> + else
> + to_compare = right->length;
> + for (ssize_t i = to_compare - 1; i >= 0; --i)
> + {
> + unsigned char lch = left->string[i];
> + unsigned char rch = right->string[i];
> + if (lch != rch)
> + return lch - rch;
> + }
> + if (left->length == right->length)
> + return 0;
> + else if (left->length < right->length)
> + /* Longer strings should come first. */
> + return 1;
> + else
> + return -1;
> +}
> +
Ok.
> +void
> +stringtable_finalize (struct stringtable *table,
> + struct stringtable_finalized *result)
> +{
> + if (table->count == 0)
> + {
> + result->strings = xstrdup ("");
> + result->size = 0;
> + return;
> + }
This is confusing because the folliwing.
struct stringtable s = { 0, };
struct stringtable_finalized f;
stringtable_finalize (&s, &f);
will result in f being { "", 0 }, while the following
struct stringtable s = { 0, };
stringtable_add (&s, "");
struct stringtable_finalized f;
stringtable_finalize (&s, &f);
will result in f being { "", 1 }.
I would expect that if strings is being "", the 'size' should be the
same (either 0 or 1 for both usages).
> +
> + /* Optimize the order of the strings. */
> + struct stringtable_entry **array = xcalloc (table->count, sizeof (*array));
> + {
> + size_t j = 0;
> + for (uint32_t i = 0; i < table->allocated; ++i)
> + for (struct stringtable_entry *e = table->entries[i]; e != NULL;
> + e = e->next)
> + {
> + array[j] = e;
> + ++j;
> + }
> + assert (j == table->count);
> + }
> + qsort (array, table->count, sizeof (*array), finalize_compare);
> +
> + /* Assign offsets, using tail merging (sharing suffixes) if possible. */
> + array[0]->offset = 0;
> + for (uint32_t j = 1; j < table->count; ++j)
> + {
> + struct stringtable_entry *previous = array[j - 1];
> + struct stringtable_entry *current = array[j];
> + if (previous->length >= current->length
> + && memcmp (&previous->string[previous->length - current->length],
> + current->string, current->length) == 0)
> + current->offset = (previous->offset + previous->length
> + - current->length);
> + else if (__builtin_add_overflow (previous->offset,
> + previous->length + 1,
> + ¤t->offset))
> + error (EXIT_FAILURE, 0, _("String table is too large"));
> + }
> +
> + /* Allocate the result string. */
> + {
> + struct stringtable_entry *last = array[table->count - 1];
> + if (__builtin_add_overflow (last->offset, last->length + 1,
> + &result->size))
> + error (EXIT_FAILURE, 0, _("String table is too large"));
> + }
> + /* The strings are copied from the hash table, so the array is no
> + longer needed. */
> + free (array);
> + result->strings = xcalloc (result->size, 1);
> +
> + /* Copy the strings. */
> + for (uint32_t i = 0; i < table->allocated; ++i)
> + for (struct stringtable_entry *e = table->entries[i]; e != NULL;
> + e = e->next)
> + if (result->strings[e->offset] == '\0')
> + memcpy (&result->strings[e->offset], e->string, e->length + 1);
> +}
Ok.
> diff --git a/elf/stringtable.h b/elf/stringtable.h
> new file mode 100644
> index 0000000000..7d57d1bda9
> --- /dev/null
> +++ b/elf/stringtable.h
> @@ -0,0 +1,64 @@
> +/* String tables for ld.so.cache construction.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + This program is free software; you can redistribute it and/or modify
> + it under the terms of the GNU General Public License as published
> + by the Free Software Foundation; version 2 of the License, or
> + (at your option) any later version.
> +
> + This program is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + GNU General Public License for more details.
> +
> + You should have received a copy of the GNU General Public License
> + along with this program; if not, see <https://www.gnu.org/licenses/>. */
> +
> +#ifndef _STRINGTABLE_H
> +#define _STRINGTABLE_H
> +
> +#include <stddef.h>
> +#include <stdint.h>
> +
> +/* An entry in the string table. Only the length and string fields are
> + expected to be used outside the string table code. */
> +struct stringtable_entry
> +{
> + struct stringtable_entry *next; /* For collision resolution. */
> + uint32_t length; /* Length of then string. */
> + uint32_t offset; /* From start of finalized table. */
> + char string[]; /* Null-terminated string. */
> +};
> +
> +/* A string table. Zero-initialization produces a valid atable. */
> +struct stringtable
> +{
> + struct stringtable_entry **entries; /* Array of hash table buckets. */
> + uint32_t count; /* Number of elements in the table. */
> + uint32_t allocated; /* Length of the entries array. */
> +};
> +
> +/* Adds STRING to TABLE. May return the address of an existing entry. */
> +struct stringtable_entry *stringtable_add (struct stringtable *table,
> + const char *string);
> +
> +/* Result of stringtable_finalize. SIZE bytes at STRINGS should be
> + written to the file. */
> +struct stringtable_finalized
> +{
> + char *strings;
> + size_t size;
> +};
> +
> +/* Assigns offsets to string table entries and computes the serialized
> + form of the string table. */
> +void stringtable_finalize (struct stringtable *table,
> + struct stringtable_finalized *result);
> +
> +/* Deallocate the string table (but not the TABLE pointer itself).
> + (The table can be re-used for adding more strings without
> + initialization.) */
> +void stringtable_free (struct stringtable *table);
> +
> +#endif /* _STRINGTABLE_H */
Ok.
> diff --git a/elf/stringtable_free.c b/elf/stringtable_free.c
> new file mode 100644
> index 0000000000..8588a25470
> --- /dev/null
> +++ b/elf/stringtable_free.c
> @@ -0,0 +1,33 @@
> +/* String tables for ld.so.cache construction. Deallocation (for tests only).
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + This program is free software; you can redistribute it and/or modify
> + it under the terms of the GNU General Public License as published
> + by the Free Software Foundation; version 2 of the License, or
> + (at your option) any later version.
> +
> + This program is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + GNU General Public License for more details.
> +
> + You should have received a copy of the GNU General Public License
> + along with this program; if not, see <https://www.gnu.org/licenses/>. */
> +
> +#include <stdlib.h>
> +#include <stringtable.h>
> +
> +void
> +stringtable_free (struct stringtable *table)
> +{
> + for (uint32_t i = 0; i < table->allocated; ++i)
> + for (struct stringtable_entry *e = table->entries[i]; e != NULL; )
> + {
> + struct stringtable_entry *next = e->next;
> + free (e);
> + e = next;
> + }
> + free (table->entries);
> + *table = (struct stringtable) { 0, };
> +}
Ok.
> diff --git a/elf/tst-stringtable.c b/elf/tst-stringtable.c
> new file mode 100644
> index 0000000000..2d044b7151
> --- /dev/null
> +++ b/elf/tst-stringtable.c
> @@ -0,0 +1,141 @@
> +/* Unit test for ldconfig string tables.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + This program is free software; you can redistribute it and/or modify
> + it under the terms of the GNU General Public License as published
> + by the Free Software Foundation; version 2 of the License, or
> + (at your option) any later version.
> +
> + This program is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + GNU General Public License for more details.
> +
> + You should have received a copy of the GNU General Public License
> + along with this program; if not, see <https://www.gnu.org/licenses/>. */
> +
> +#include <stdlib.h>
> +#include <string.h>
> +#include <stringtable.h>
> +#include <support/check.h>
> +#include <support/support.h>
> +
> +static int
> +do_test (void)
> +{
> + /* Empty string table. */
> + {
> + struct stringtable s = { 0, };
> + struct stringtable_finalized f;
> + stringtable_finalize (&s, &f);
> + TEST_COMPARE_STRING (f.strings, "");
> + TEST_COMPARE (f.size, 0);
> + free (f.strings);
> + stringtable_free (&s);
> + }
> +
> + /* String table with one empty string. */
> + {
> + struct stringtable s = { 0, };
> + struct stringtable_entry *e = stringtable_add (&s, "");
> + TEST_COMPARE_STRING (e->string, "");
> + TEST_COMPARE (e->length, 0);
> + TEST_COMPARE (s.count, 1);
> +
> + struct stringtable_finalized f;
> + stringtable_finalize (&s, &f);
> + TEST_COMPARE (e->offset, 0);
> + TEST_COMPARE_STRING (f.strings, "");
> + TEST_COMPARE (f.size, 1);
> + free (f.strings);
> + stringtable_free (&s);
> + }
> +
> + /* String table with one non-empty string. */
> + {
> + struct stringtable s = { 0, };
> + struct stringtable_entry *e = stringtable_add (&s, "name");
> + TEST_COMPARE_STRING (e->string, "name");
> + TEST_COMPARE (e->length, 4);
> + TEST_COMPARE (s.count, 1);
Ok.
> +
> + struct stringtable_finalized f;
> + stringtable_finalize (&s, &f);
> + TEST_COMPARE (e->offset, 0);
> + TEST_COMPARE_STRING (f.strings, "name");
> + TEST_COMPARE (f.size, 5);
> + free (f.strings);
> + stringtable_free (&s);
> + }
> +
Ok.
> + /* Two strings, one is a prefix of the other. Tail-merging can only
> + happen in one way in this case. */
> + {
> + struct stringtable s = { 0, };
> + struct stringtable_entry *suffix = stringtable_add (&s, "suffix");
> + TEST_COMPARE_STRING (suffix->string, "suffix");
> + TEST_COMPARE (suffix->length, 6);
> + TEST_COMPARE (s.count, 1);
> +
> + struct stringtable_entry *prefix
> + = stringtable_add (&s, "prefix-suffix");
> + TEST_COMPARE_STRING (prefix->string, "prefix-suffix");
> + TEST_COMPARE (prefix->length, strlen ("prefix-suffix"));
> + TEST_COMPARE (s.count, 2);
> +
> + struct stringtable_finalized f;
> + stringtable_finalize (&s, &f);
> + TEST_COMPARE (prefix->offset, 0);
> + TEST_COMPARE (suffix->offset, strlen ("prefix-"));
> + TEST_COMPARE_STRING (f.strings, "prefix-suffix");
> + TEST_COMPARE (f.size, sizeof ("prefix-suffix"));
> + free (f.strings);
> + stringtable_free (&s);
> + }
Ok.
> +
> + /* String table with various shared prefixes. Triggers hash
> + resizing. */
> + {
> + enum { count = 1500 };
> + char *strings[2 * count];
> + struct stringtable_entry *entries[2 * count];
> + struct stringtable s = { 0, };
> + for (int i = 0; i < count; ++i)
> + {
> + strings[i] = xasprintf ("%d", i);
> + entries[i] = stringtable_add (&s, strings[i]);
> + TEST_COMPARE (entries[i]->length, strlen (strings[i]));
> + TEST_COMPARE_STRING (entries[i]->string, strings[i]);
> + strings[i + count] = xasprintf ("prefix/%d", i);
> + entries[i + count] = stringtable_add (&s, strings[i + count]);
> + TEST_COMPARE (entries[i + count]->length, strlen (strings[i + count]));
> + TEST_COMPARE_STRING (entries[i + count]->string, strings[i + count]);
> + }
> +
> + struct stringtable_finalized f;
> + stringtable_finalize (&s, &f);
> +
> + for (int i = 0; i < 2 * count; ++i)
> + {
> + TEST_COMPARE (entries[i]->length, strlen (strings[i]));
> + TEST_COMPARE_STRING (entries[i]->string, strings[i]);
> + TEST_COMPARE_STRING (f.strings + entries[i]->offset, strings[i]);
> + free (strings[i]);
> + }
> +
> + free (f.strings);
> + stringtable_free (&s);
> + }
> +
> + return 0;
> +}
> +
> +#include <support/test-driver.c>
> +
> +/* Re-compile the string table implementation here. It is not
> + possible to link against the actual build because it was built for
> + use in ldconfig. */
> +#define _(arg) arg
> +#include "stringtable.c"
> +#include "stringtable_free.c"
>
Ok.
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 06/11] elf: Implement a string table for ldconfig, with tail merging
2020-11-27 19:29 ` Adhemerval Zanella
@ 2020-11-27 19:49 ` Florian Weimer
2020-11-30 19:01 ` Adhemerval Zanella
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-27 19:49 UTC (permalink / raw)
To: Adhemerval Zanella; +Cc: libc-alpha
* Adhemerval Zanella:
>> +static void
>> +stringtable_init (struct stringtable *table)
>> +{
>> + table->count = 0;
>> +
>> + /* This needs to be a power of two. 128 is sufficient to keep track
>> + of 42 DSOs without resizing (assuming two strings per DSOs).
>> + glibc itself comes with more than 20 DSOs, so 64 would likely to
>> + be too small. */
>> + table->allocated = 64;
>
> Should we use a large value then? Asking because the comment indicates that
> the chosen default value is not best suitable one.
You mean 128? Eh, yes.
>> +/* Double the capacity of the hash table. */
>> +static void
>> +stringtable_rehash (struct stringtable *table)
>> +{
>> + /* Cannot overflow because the old allocation size (in bytes) is
>> + larger. */
>
> This comment is a bit confusing, I think it should be safe to assume
> it won't overflow because stringtable_rehash won't be called for
> large values (stringtable_add will bail early).
What I meant is this:
/* This computation cannot overflow because the old total in-memory
size of the hash table is larger than the computed value. */
>> +void
>> +stringtable_finalize (struct stringtable *table,
>> + struct stringtable_finalized *result)
>> +{
>> + if (table->count == 0)
>> + {
>> + result->strings = xstrdup ("");
>> + result->size = 0;
>> + return;
>> + }
>
> This is confusing because the folliwing.
>
> struct stringtable s = { 0, };
> struct stringtable_finalized f;
> stringtable_finalize (&s, &f);
>
> will result in f being { "", 0 }, while the following
>
> struct stringtable s = { 0, };
> stringtable_add (&s, "");
> struct stringtable_finalized f;
> stringtable_finalize (&s, &f);
>
> will result in f being { "", 1 }.
>
> I would expect that if strings is being "", the 'size' should be the
> same (either 0 or 1 for both usages).
In the second space, storage space is needed for the null byte because
the table contains the empty string. In the first case, no storage
space is needed. We could use NULL, but then glibc has a bit of a
strange relationship with arrays of length 0 that start at a null
pointer. (They can't be used in memcpy etc.)
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 06/11] elf: Implement a string table for ldconfig, with tail merging
2020-11-27 19:49 ` Florian Weimer
@ 2020-11-30 19:01 ` Adhemerval Zanella
0 siblings, 0 replies; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-30 19:01 UTC (permalink / raw)
To: Florian Weimer; +Cc: libc-alpha
On 27/11/2020 16:49, Florian Weimer wrote:
> * Adhemerval Zanella:
>
>>> +static void
>>> +stringtable_init (struct stringtable *table)
>>> +{
>>> + table->count = 0;
>>> +
>>> + /* This needs to be a power of two. 128 is sufficient to keep track
>>> + of 42 DSOs without resizing (assuming two strings per DSOs).
>>> + glibc itself comes with more than 20 DSOs, so 64 would likely to
>>> + be too small. */
>>> + table->allocated = 64;
>>
>> Should we use a large value then? Asking because the comment indicates that
>> the chosen default value is not best suitable one.
>
> You mean 128? Eh, yes.
>
>>> +/* Double the capacity of the hash table. */
>>> +static void
>>> +stringtable_rehash (struct stringtable *table)
>>> +{
>>> + /* Cannot overflow because the old allocation size (in bytes) is
>>> + larger. */
>>
>> This comment is a bit confusing, I think it should be safe to assume
>> it won't overflow because stringtable_rehash won't be called for
>> large values (stringtable_add will bail early).
>
> What I meant is this:
>
> /* This computation cannot overflow because the old total in-memory
> size of the hash table is larger than the computed value. */
Ack.
>
>>> +void
>>> +stringtable_finalize (struct stringtable *table,
>>> + struct stringtable_finalized *result)
>>> +{
>>> + if (table->count == 0)
>>> + {
>>> + result->strings = xstrdup ("");
>>> + result->size = 0;
>>> + return;
>>> + }
>>
>> This is confusing because the folliwing.
>>
>> struct stringtable s = { 0, };
>> struct stringtable_finalized f;
>> stringtable_finalize (&s, &f);
>>
>> will result in f being { "", 0 }, while the following
>>
>> struct stringtable s = { 0, };
>> stringtable_add (&s, "");
>> struct stringtable_finalized f;
>> stringtable_finalize (&s, &f);
>>
>> will result in f being { "", 1 }.
>>
>> I would expect that if strings is being "", the 'size' should be the
>> same (either 0 or 1 for both usages).
>
> In the second space, storage space is needed for the null byte because
> the table contains the empty string. In the first case, no storage
> space is needed. We could use NULL, but then glibc has a bit of a
> strange relationship with arrays of length 0 that start at a null
> pointer. (They can't be used in memcpy etc.)
Fair enough.
>
> Thanks,
> Florian
>
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 07/11] elf: Implement tail merging of strings in ldconfig
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (5 preceding siblings ...)
2020-11-09 18:40 ` [PATCH 06/11] elf: Implement a string table for ldconfig, with tail merging Florian Weimer
@ 2020-11-09 18:41 ` Florian Weimer
2020-11-30 18:41 ` Adhemerval Zanella
2020-11-09 18:41 ` [PATCH 08/11] elf: Process glibc-hwcaps subdirectories " Florian Weimer
` (5 subsequent siblings)
12 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:41 UTC (permalink / raw)
To: libc-alpha
This simplifies the string table construction in elf/cache.c
because there is no more need to keep track of offsets explicitly;
the string table implementation does this internally.
This change slightly reduces the size of the cache on disk. The
file format does not change as a result. The strings are
null-terminated, without explicit length, so tail merging is
transparent to readers.
---
elf/Makefile | 3 ++-
elf/cache.c | 76 ++++++++++++++++++++++++++--------------------------
2 files changed, 40 insertions(+), 39 deletions(-)
diff --git a/elf/Makefile b/elf/Makefile
index 5a8f116a67..e26ac16b44 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -118,7 +118,8 @@ others-static += ldconfig
others += ldconfig
install-rootsbin += ldconfig
-ldconfig-modules := cache readlib xmalloc xstrdup chroot_canon static-stubs
+ldconfig-modules := cache readlib xmalloc xstrdup chroot_canon static-stubs \
+ stringtable
extra-objs += $(ldconfig-modules:=.o)
others-extras = $(ldconfig-modules)
endif
diff --git a/elf/cache.c b/elf/cache.c
index 549e04ce21..5a6ee20e86 100644
--- a/elf/cache.c
+++ b/elf/cache.c
@@ -35,11 +35,15 @@
#include <ldconfig.h>
#include <dl-cache.h>
#include <version.h>
+#include <stringtable.h>
+
+/* Used to store library names, paths, and other strings. */
+static struct stringtable strings;
struct cache_entry
{
- char *lib; /* Library name. */
- char *path; /* Path to find library. */
+ struct stringtable_entry *lib; /* Library name. */
+ struct stringtable_entry *path; /* Path to find library. */
int flags; /* Flags to indicate kind of library. */
unsigned int osversion; /* Required OS version. */
uint64_t hwcap; /* Important hardware capabilities. */
@@ -300,7 +304,7 @@ static int
compare (const struct cache_entry *e1, const struct cache_entry *e2)
{
/* We need to swap entries here to get the correct sort order. */
- int res = _dl_cache_libcmp (e2->lib, e1->lib);
+ int res = _dl_cache_libcmp (e2->lib->string, e1->lib->string);
if (res == 0)
{
if (e1->flags < e2->flags)
@@ -369,26 +373,24 @@ save_cache (const char *cache_name)
{
/* The cache entries are sorted already, save them in this order. */
- /* Count the length of all strings. */
- /* The old format doesn't contain hwcap entries and doesn't contain
- libraries in subdirectories with hwcaps entries. Count therefore
- also all entries with hwcap == 0. */
- size_t total_strlen = 0;
struct cache_entry *entry;
/* Number of cache entries. */
int cache_entry_count = 0;
- /* Number of normal cache entries. */
+ /* The old format doesn't contain hwcap entries and doesn't contain
+ libraries in subdirectories with hwcaps entries. Count therefore
+ also all entries with hwcap == 0. */
int cache_entry_old_count = 0;
for (entry = entries; entry != NULL; entry = entry->next)
{
- /* Account the final NULs. */
- total_strlen += strlen (entry->lib) + strlen (entry->path) + 2;
++cache_entry_count;
if (entry->hwcap == 0)
++cache_entry_old_count;
}
+ struct stringtable_finalized strings_finalized;
+ stringtable_finalize (&strings, &strings_finalized);
+
/* Create the on disk cache structure. */
struct cache_file *file_entries = NULL;
size_t file_entries_size = 0;
@@ -432,7 +434,7 @@ save_cache (const char *cache_name)
sizeof CACHE_VERSION - 1);
file_entries_new->nlibs = cache_entry_count;
- file_entries_new->len_strings = total_strlen;
+ file_entries_new->len_strings = strings_finalized.size;
file_entries_new->flags = cache_file_new_flags_endian_current;
}
@@ -449,20 +451,20 @@ save_cache (const char *cache_name)
str_offset = 0;
/* An array for all strings. */
- char *strings = xmalloc (total_strlen);
- char *str = strings;
int idx_old;
int idx_new;
for (idx_old = 0, idx_new = 0, entry = entries; entry != NULL;
entry = entry->next, ++idx_new)
{
- /* First the library. */
if (opt_format != opt_format_new && entry->hwcap == 0)
{
file_entries->libs[idx_old].flags = entry->flags;
/* XXX: Actually we can optimize here and remove duplicates. */
file_entries->libs[idx_old].key = str_offset + pad;
+ file_entries->libs[idx_new].key = str_offset + entry->lib->offset;
+ file_entries->libs[idx_new].value
+ = str_offset + entry->path->offset;
}
if (opt_format != opt_format_old)
{
@@ -473,20 +475,12 @@ save_cache (const char *cache_name)
file_entries_new->libs[idx_new].flags = entry->flags;
file_entries_new->libs[idx_new].osversion = entry->osversion;
file_entries_new->libs[idx_new].hwcap = entry->hwcap;
- file_entries_new->libs[idx_new].key = str_offset;
+ file_entries_new->libs[idx_new].key
+ = str_offset + entry->lib->offset;
+ file_entries_new->libs[idx_new].value
+ = str_offset + entry->path->offset;
}
- size_t len = strlen (entry->lib) + 1;
- str = mempcpy (str, entry->lib, len);
- str_offset += len;
- /* Then the path. */
- if (opt_format != opt_format_new && entry->hwcap == 0)
- file_entries->libs[idx_old].value = str_offset + pad;
- if (opt_format != opt_format_old)
- file_entries_new->libs[idx_new].value = str_offset;
- len = strlen (entry->path) + 1;
- str = mempcpy (str, entry->path, len);
- str_offset += len;
/* Ignore entries with hwcap for old format. */
if (entry->hwcap == 0)
++idx_old;
@@ -511,7 +505,7 @@ save_cache (const char *cache_name)
extension_offset += pad;
extension_offset += file_entries_new_size;
}
- extension_offset += total_strlen;
+ extension_offset += strings_finalized.size;
extension_offset = roundup (extension_offset, 4); /* Provide alignment. */
if (opt_format != opt_format_old)
file_entries_new->extension_offset = extension_offset;
@@ -551,7 +545,8 @@ save_cache (const char *cache_name)
error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
}
- if (write (fd, strings, total_strlen) != (ssize_t) total_strlen)
+ if (write (fd, strings_finalized.strings, strings_finalized.size)
+ != (ssize_t) strings_finalized.size)
error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
if (opt_format != opt_format_old)
@@ -580,7 +575,7 @@ save_cache (const char *cache_name)
/* Free all allocated memory. */
free (file_entries_new);
free (file_entries);
- free (strings);
+ free (strings_finalized.strings);
while (entries)
{
@@ -596,14 +591,19 @@ void
add_to_cache (const char *path, const char *lib, int flags,
unsigned int osversion, uint64_t hwcap)
{
- size_t liblen = strlen (lib) + 1;
- size_t len = liblen + strlen (path) + 1;
- struct cache_entry *new_entry
- = xmalloc (sizeof (struct cache_entry) + liblen + len);
-
- new_entry->lib = memcpy ((char *) (new_entry + 1), lib, liblen);
- new_entry->path = new_entry->lib + liblen;
- snprintf (new_entry->path, len, "%s/%s", path, lib);
+ struct cache_entry *new_entry = xmalloc (sizeof (*new_entry));
+
+ struct stringtable_entry *path_interned;
+ {
+ char *p;
+ if (asprintf (&p, "%s/%s", path, lib) < 0)
+ error (EXIT_FAILURE, errno, _("Could not create library path"));
+ path_interned = stringtable_add (&strings, p);
+ free (p);
+ }
+
+ new_entry->lib = stringtable_add (&strings, lib);
+ new_entry->path = path_interned;
new_entry->flags = flags;
new_entry->osversion = osversion;
new_entry->hwcap = hwcap;
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 07/11] elf: Implement tail merging of strings in ldconfig
2020-11-09 18:41 ` [PATCH 07/11] elf: Implement tail merging of strings in ldconfig Florian Weimer
@ 2020-11-30 18:41 ` Adhemerval Zanella
2020-12-01 10:28 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-30 18:41 UTC (permalink / raw)
To: libc-alpha, Florian Weimer
On 09/11/2020 15:41, Florian Weimer via Libc-alpha wrote:
> This simplifies the string table construction in elf/cache.c
> because there is no more need to keep track of offsets explicitly;
> the string table implementation does this internally.
>
> This change slightly reduces the size of the cache on disk. The
> file format does not change as a result. The strings are
> null-terminated, without explicit length, so tail merging is
> transparent to readers.
Any idea of the size improvement here?
LGTM, thanks.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
> elf/Makefile | 3 ++-
> elf/cache.c | 76 ++++++++++++++++++++++++++--------------------------
> 2 files changed, 40 insertions(+), 39 deletions(-)
>
> diff --git a/elf/Makefile b/elf/Makefile
> index 5a8f116a67..e26ac16b44 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -118,7 +118,8 @@ others-static += ldconfig
> others += ldconfig
> install-rootsbin += ldconfig
>
> -ldconfig-modules := cache readlib xmalloc xstrdup chroot_canon static-stubs
> +ldconfig-modules := cache readlib xmalloc xstrdup chroot_canon static-stubs \
> + stringtable
> extra-objs += $(ldconfig-modules:=.o)
> others-extras = $(ldconfig-modules)
> endif
Ok.
> diff --git a/elf/cache.c b/elf/cache.c
> index 549e04ce21..5a6ee20e86 100644
> --- a/elf/cache.c
> +++ b/elf/cache.c
> @@ -35,11 +35,15 @@
> #include <ldconfig.h>
> #include <dl-cache.h>
> #include <version.h>
> +#include <stringtable.h>
> +
> +/* Used to store library names, paths, and other strings. */
> +static struct stringtable strings;
>
> struct cache_entry
> {
> - char *lib; /* Library name. */
> - char *path; /* Path to find library. */
> + struct stringtable_entry *lib; /* Library name. */
> + struct stringtable_entry *path; /* Path to find library. */
> int flags; /* Flags to indicate kind of library. */
> unsigned int osversion; /* Required OS version. */
> uint64_t hwcap; /* Important hardware capabilities. */
> @@ -300,7 +304,7 @@ static int
Ok.
> compare (const struct cache_entry *e1, const struct cache_entry *e2)
> {
> /* We need to swap entries here to get the correct sort order. */
> - int res = _dl_cache_libcmp (e2->lib, e1->lib);
> + int res = _dl_cache_libcmp (e2->lib->string, e1->lib->string);
> if (res == 0)
> {
> if (e1->flags < e2->flags)
Ok.
> @@ -369,26 +373,24 @@ save_cache (const char *cache_name)
> {
> /* The cache entries are sorted already, save them in this order. */
>
> - /* Count the length of all strings. */
> - /* The old format doesn't contain hwcap entries and doesn't contain
> - libraries in subdirectories with hwcaps entries. Count therefore
> - also all entries with hwcap == 0. */
> - size_t total_strlen = 0;
> struct cache_entry *entry;
> /* Number of cache entries. */
> int cache_entry_count = 0;
> - /* Number of normal cache entries. */
> + /* The old format doesn't contain hwcap entries and doesn't contain
> + libraries in subdirectories with hwcaps entries. Count therefore
> + also all entries with hwcap == 0. */
> int cache_entry_old_count = 0;
>
> for (entry = entries; entry != NULL; entry = entry->next)
> {
> - /* Account the final NULs. */
> - total_strlen += strlen (entry->lib) + strlen (entry->path) + 2;
> ++cache_entry_count;
> if (entry->hwcap == 0)
> ++cache_entry_old_count;
> }
>
> + struct stringtable_finalized strings_finalized;
> + stringtable_finalize (&strings, &strings_finalized);
> +
> /* Create the on disk cache structure. */
> struct cache_file *file_entries = NULL;
> size_t file_entries_size = 0;
> @@ -432,7 +434,7 @@ save_cache (const char *cache_name)
> sizeof CACHE_VERSION - 1);
>
> file_entries_new->nlibs = cache_entry_count;
> - file_entries_new->len_strings = total_strlen;
> + file_entries_new->len_strings = strings_finalized.size;
> file_entries_new->flags = cache_file_new_flags_endian_current;
> }
>
Ok.
> @@ -449,20 +451,20 @@ save_cache (const char *cache_name)
> str_offset = 0;
>
> /* An array for all strings. */
> - char *strings = xmalloc (total_strlen);
> - char *str = strings;
> int idx_old;
> int idx_new;
>
> for (idx_old = 0, idx_new = 0, entry = entries; entry != NULL;
> entry = entry->next, ++idx_new)
> {
> - /* First the library. */
> if (opt_format != opt_format_new && entry->hwcap == 0)
> {
> file_entries->libs[idx_old].flags = entry->flags;
> /* XXX: Actually we can optimize here and remove duplicates. */
> file_entries->libs[idx_old].key = str_offset + pad;
> + file_entries->libs[idx_new].key = str_offset + entry->lib->offset;
> + file_entries->libs[idx_new].value
> + = str_offset + entry->path->offset;
> }
> if (opt_format != opt_format_old)
> {
Ok.
> @@ -473,20 +475,12 @@ save_cache (const char *cache_name)
> file_entries_new->libs[idx_new].flags = entry->flags;
> file_entries_new->libs[idx_new].osversion = entry->osversion;
> file_entries_new->libs[idx_new].hwcap = entry->hwcap;
> - file_entries_new->libs[idx_new].key = str_offset;
> + file_entries_new->libs[idx_new].key
> + = str_offset + entry->lib->offset;
> + file_entries_new->libs[idx_new].value
> + = str_offset + entry->path->offset;
> }
>
> - size_t len = strlen (entry->lib) + 1;
> - str = mempcpy (str, entry->lib, len);
> - str_offset += len;
> - /* Then the path. */
> - if (opt_format != opt_format_new && entry->hwcap == 0)
> - file_entries->libs[idx_old].value = str_offset + pad;
> - if (opt_format != opt_format_old)
> - file_entries_new->libs[idx_new].value = str_offset;
> - len = strlen (entry->path) + 1;
> - str = mempcpy (str, entry->path, len);
> - str_offset += len;
> /* Ignore entries with hwcap for old format. */
> if (entry->hwcap == 0)
> ++idx_old;
Ok.
> @@ -511,7 +505,7 @@ save_cache (const char *cache_name)
> extension_offset += pad;
> extension_offset += file_entries_new_size;
> }
> - extension_offset += total_strlen;
> + extension_offset += strings_finalized.size;
> extension_offset = roundup (extension_offset, 4); /* Provide alignment. */
> if (opt_format != opt_format_old)
> file_entries_new->extension_offset = extension_offset;
Ok.
> @@ -551,7 +545,8 @@ save_cache (const char *cache_name)
> error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
> }
>
> - if (write (fd, strings, total_strlen) != (ssize_t) total_strlen)
> + if (write (fd, strings_finalized.strings, strings_finalized.size)
> + != (ssize_t) strings_finalized.size)
> error (EXIT_FAILURE, errno, _("Writing of cache data failed"));
>
> if (opt_format != opt_format_old)
Ok.
> @@ -580,7 +575,7 @@ save_cache (const char *cache_name)
> /* Free all allocated memory. */
> free (file_entries_new);
> free (file_entries);
> - free (strings);
> + free (strings_finalized.strings);
>
> while (entries)
> {
Ok.
> @@ -596,14 +591,19 @@ void
> add_to_cache (const char *path, const char *lib, int flags,
> unsigned int osversion, uint64_t hwcap)
> {
> - size_t liblen = strlen (lib) + 1;
> - size_t len = liblen + strlen (path) + 1;
> - struct cache_entry *new_entry
> - = xmalloc (sizeof (struct cache_entry) + liblen + len);
> -
> - new_entry->lib = memcpy ((char *) (new_entry + 1), lib, liblen);
> - new_entry->path = new_entry->lib + liblen;
> - snprintf (new_entry->path, len, "%s/%s", path, lib);
> + struct cache_entry *new_entry = xmalloc (sizeof (*new_entry));
> +
> + struct stringtable_entry *path_interned;
> + {
> + char *p;
> + if (asprintf (&p, "%s/%s", path, lib) < 0)
> + error (EXIT_FAILURE, errno, _("Could not create library path"));
> + path_interned = stringtable_add (&strings, p);
> + free (p);
> + }
> +
> + new_entry->lib = stringtable_add (&strings, lib);
> + new_entry->path = path_interned;
> new_entry->flags = flags;
> new_entry->osversion = osversion;
> new_entry->hwcap = hwcap;
>
Ok.
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 07/11] elf: Implement tail merging of strings in ldconfig
2020-11-30 18:41 ` Adhemerval Zanella
@ 2020-12-01 10:28 ` Florian Weimer
0 siblings, 0 replies; 58+ messages in thread
From: Florian Weimer @ 2020-12-01 10:28 UTC (permalink / raw)
To: Adhemerval Zanella via Libc-alpha
* Adhemerval Zanella via Libc-alpha:
> On 09/11/2020 15:41, Florian Weimer via Libc-alpha wrote:
>> This simplifies the string table construction in elf/cache.c
>> because there is no more need to keep track of offsets explicitly;
>> the string table implementation does this internally.
>>
>> This change slightly reduces the size of the cache on disk. The
>> file format does not change as a result. The strings are
>> null-terminated, without explicit length, so tail merging is
>> transparent to readers.
>
> Any idea of the size improvement here?
It's smaller than expected. Looks the tails aren't actually merged.
I'll figure out what is going on.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 08/11] elf: Process glibc-hwcaps subdirectories in ldconfig
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (6 preceding siblings ...)
2020-11-09 18:41 ` [PATCH 07/11] elf: Implement tail merging of strings in ldconfig Florian Weimer
@ 2020-11-09 18:41 ` Florian Weimer
2020-11-30 19:46 ` Adhemerval Zanella
2020-11-09 18:41 ` [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing Florian Weimer
` (4 subsequent siblings)
12 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:41 UTC (permalink / raw)
To: libc-alpha
Libraries from these subdirectories are added to the cache
with a special hwcap bit DL_CACHE_HWCAP_EXTENSION, so that
they are ignored by older dynamic loaders.
---
elf/cache.c | 257 ++++++++++++++++++++++++++++++++-----
elf/ldconfig.c | 155 +++++++++++++++++++---
sysdeps/generic/dl-cache.h | 50 ++++++++
sysdeps/generic/ldconfig.h | 18 ++-
4 files changed, 427 insertions(+), 53 deletions(-)
diff --git a/elf/cache.c b/elf/cache.c
index 5a6ee20e86..b03c5319f8 100644
--- a/elf/cache.c
+++ b/elf/cache.c
@@ -40,6 +40,105 @@
/* Used to store library names, paths, and other strings. */
static struct stringtable strings;
+/* Keeping track of "glibc-hwcaps" subdirectories. During cache
+ construction, a linear search by name is performed to deduplicate
+ entries. */
+struct glibc_hwcaps_subdirectory
+{
+ struct glibc_hwcaps_subdirectory *next;
+
+ /* Interned string with the subdirectory name. */
+ struct stringtable_entry *name;
+
+ /* Array index in the cache_extension_tag_glibc_hwcaps section in
+ the stored cached file. This is computed after all the
+ subdirectories have been processed, so that subdirectory names in
+ the extension section can be sorted. */
+ uint32_t section_index;
+
+ /* True if the subdirectory is actually used for anything. */
+ bool used;
+};
+
+const char *
+glibc_hwcaps_subdirectory_name (const struct glibc_hwcaps_subdirectory *dir)
+{
+ return dir->name->string;
+}
+
+/* Linked list of known hwcaps subdirecty names. */
+static struct glibc_hwcaps_subdirectory *hwcaps;
+
+struct glibc_hwcaps_subdirectory *
+new_glibc_hwcaps_subdirectory (const char *name)
+{
+ struct stringtable_entry *name_interned = stringtable_add (&strings, name);
+ for (struct glibc_hwcaps_subdirectory *p = hwcaps; p != NULL; p = p->next)
+ if (p->name == name_interned)
+ return p;
+ struct glibc_hwcaps_subdirectory *p = xmalloc (sizeof (*p));
+ p->next = hwcaps;
+ p->name = name_interned;
+ p->section_index = 0;
+ p->used = false;
+ hwcaps = p;
+ return p;
+}
+
+/* Helper for sorting struct glibc_hwcaps_subdirectory elements by
+ name. */
+static int
+assign_glibc_hwcaps_indices_compare (const void *l, const void *r)
+{
+ const struct glibc_hwcaps_subdirectory *left
+ = *(struct glibc_hwcaps_subdirectory **)l;
+ const struct glibc_hwcaps_subdirectory *right
+ = *(struct glibc_hwcaps_subdirectory **)r;
+ return strcmp (glibc_hwcaps_subdirectory_name (left),
+ glibc_hwcaps_subdirectory_name (right));
+}
+
+/* Count the number of hwcaps subdirectories which are actually
+ used. */
+static size_t
+glibc_hwcaps_count (void)
+{
+ size_t count = 0;
+ for (struct glibc_hwcaps_subdirectory *p = hwcaps; p != NULL; p = p->next)
+ if (p->used)
+ ++count;
+ return count;
+}
+
+/* Compute the section_index fields for all */
+static void
+assign_glibc_hwcaps_indices (void)
+{
+ /* Convert the linked list into an array, so that we can use qsort.
+ Only copy the subdirectories which are actually used. */
+ size_t count = glibc_hwcaps_count ();
+ struct glibc_hwcaps_subdirectory **array
+ = xmalloc (sizeof (*array) * count);
+ {
+ size_t i = 0;
+ for (struct glibc_hwcaps_subdirectory *p = hwcaps; p != NULL; p = p->next)
+ if (p->used)
+ {
+ array[i] = p;
+ ++i;
+ }
+ assert (i == count);
+ }
+
+ qsort (array, count, sizeof (*array), assign_glibc_hwcaps_indices_compare);
+
+ /* Assign the array indices. */
+ for (size_t i = 0; i < count; ++i)
+ array[i]->section_index = i;
+
+ free (array);
+}
+
struct cache_entry
{
struct stringtable_entry *lib; /* Library name. */
@@ -48,6 +147,10 @@ struct cache_entry
unsigned int osversion; /* Required OS version. */
uint64_t hwcap; /* Important hardware capabilities. */
int bits_hwcap; /* Number of bits set in hwcap. */
+
+ /* glibc-hwcaps subdirectory. If not NULL, hwcap must be zero. */
+ struct glibc_hwcaps_subdirectory *hwcaps;
+
struct cache_entry *next; /* Next entry in list. */
};
@@ -60,7 +163,7 @@ static const char *flag_descr[] =
/* Print a single entry. */
static void
print_entry (const char *lib, int flag, unsigned int osversion,
- uint64_t hwcap, const char *key)
+ uint64_t hwcap, const char *hwcap_string, const char *key)
{
printf ("\t%s (", lib);
switch (flag & FLAG_TYPE_MASK)
@@ -132,7 +235,9 @@ print_entry (const char *lib, int flag, unsigned int osversion,
printf (",%d", flag & FLAG_REQUIRED_MASK);
break;
}
- if (hwcap != 0)
+ if (hwcap_string != NULL)
+ printf (", hwcap: \"%s\"", hwcap_string);
+ else if (hwcap != 0)
printf (", hwcap: %#.16" PRIx64, hwcap);
if (osversion != 0)
{
@@ -158,6 +263,29 @@ print_entry (const char *lib, int flag, unsigned int osversion,
printf (") => %s\n", key);
}
+/* Returns the string with the name of the glibcs-hwcaps subdirectory
+ associated with ENTRY->hwcap. file_base must be the base address
+ for string table indices. */
+static const char *
+glibc_hwcaps_string (struct cache_extension_all_loaded *ext,
+ const void *file_base, size_t file_size,
+ struct file_entry_new *entry)
+{
+ const uint32_t *hwcaps_array
+ = ext->sections[cache_extension_tag_glibc_hwcaps].base;
+ if (dl_cache_hwcap_extension (entry) && hwcaps_array != NULL)
+ {
+ uint32_t index = (uint32_t) entry->hwcap;
+ if (index < ext->sections[cache_extension_tag_glibc_hwcaps].size / 4)
+ {
+ uint32_t string_table_index = hwcaps_array[index];
+ if (string_table_index < file_size)
+ return file_base + string_table_index;
+ }
+ }
+ return NULL;
+}
+
/* Print an error and exit if the new-file cache is internally
inconsistent. */
static void
@@ -167,9 +295,7 @@ check_new_cache (struct cache_file_new *cache)
error (EXIT_FAILURE, 0, _("Cache file has wrong endianness.\n"));
}
-/* Print the extension information at the cache at start address
- FILE_BASE, of length FILE_SIZE bytes. The new-format cache header
- is at CACHE, and the file name for diagnostics is CACHE_NAME. */
+/* Print the extension information in *EXT. */
static void
print_extensions (struct cache_extension_all_loaded *ext)
{
@@ -266,7 +392,7 @@ print_cache (const char *cache_name)
/* Print everything. */
for (unsigned int i = 0; i < cache->nlibs; i++)
print_entry (cache_data + cache->libs[i].key,
- cache->libs[i].flags, 0, 0,
+ cache->libs[i].flags, 0, 0, NULL,
cache_data + cache->libs[i].value);
}
else if (format == 1)
@@ -281,11 +407,16 @@ print_cache (const char *cache_name)
/* Print everything. */
for (unsigned int i = 0; i < cache_new->nlibs; i++)
- print_entry (cache_data + cache_new->libs[i].key,
- cache_new->libs[i].flags,
- cache_new->libs[i].osversion,
- cache_new->libs[i].hwcap,
- cache_data + cache_new->libs[i].value);
+ {
+ const char *hwcaps_string
+ = glibc_hwcaps_string (&ext, cache, cache_size,
+ &cache_new->libs[i]);
+ print_entry (cache_data + cache_new->libs[i].key,
+ cache_new->libs[i].flags,
+ cache_new->libs[i].osversion,
+ cache_new->libs[i].hwcap, hwcaps_string,
+ cache_data + cache_new->libs[i].value);
+ }
print_extensions (&ext);
}
/* Cleanup. */
@@ -311,8 +442,23 @@ compare (const struct cache_entry *e1, const struct cache_entry *e2)
return 1;
else if (e1->flags > e2->flags)
return -1;
+ /* Keep the glibc-hwcaps extension entries before the regular
+ entries, and sort them by their names. search_cache in
+ dl-cache.c stops searching once the first non-extension entry
+ is found, so the extension entries need to come first. */
+ else if (e1->hwcaps != NULL && e2->hwcaps == NULL)
+ return -1;
+ else if (e1->hwcaps == NULL && e2->hwcaps != NULL)
+ return 1;
+ else if (e1->hwcaps != NULL && e2->hwcaps != NULL)
+ {
+ res = strcmp (glibc_hwcaps_subdirectory_name (e1->hwcaps),
+ glibc_hwcaps_subdirectory_name (e2->hwcaps));
+ if (res != 0)
+ return res;
+ }
/* Sort by most specific hwcap. */
- else if (e2->bits_hwcap > e1->bits_hwcap)
+ if (e2->bits_hwcap > e1->bits_hwcap)
return 1;
else if (e2->bits_hwcap < e1->bits_hwcap)
return -1;
@@ -337,30 +483,65 @@ enum
* sizeof (struct cache_extension_section)))
};
-/* Write the cache extensions to FD. The extension directory is
- assumed to be located at CACHE_EXTENSION_OFFSET. */
+/* Write the cache extensions to FD. The string table is shifted by
+ STRING_TABLE_OFFSET. The extension directory is assumed to be
+ located at CACHE_EXTENSION_OFFSET. assign_glibc_hwcaps_indices
+ must have been called. */
static void
-write_extensions (int fd, uint32_t cache_extension_offset)
+write_extensions (int fd, uint32_t str_offset,
+ uint32_t cache_extension_offset)
{
assert ((cache_extension_offset % 4) == 0);
+ /* The length and contents of the glibc-hwcaps section. */
+ uint32_t hwcaps_count = glibc_hwcaps_count ();
+ uint32_t hwcaps_offset = cache_extension_offset + cache_extension_size;
+ uint32_t hwcaps_size = hwcaps_count * sizeof (uint32_t);
+ uint32_t *hwcaps_array = xmalloc (hwcaps_size);
+ for (struct glibc_hwcaps_subdirectory *p = hwcaps; p != NULL; p = p->next)
+ if (p->used)
+ hwcaps_array[p->section_index] = str_offset + p->name->offset;
+
+ /* This is the offset of the generator string. */
+ uint32_t generator_offset = hwcaps_offset;
+ if (hwcaps_count == 0)
+ /* There is no section for the hwcaps subdirectories. */
+ generator_offset -= sizeof (struct cache_extension_section);
+ else
+ /* The string table indices for the hwcaps subdirectories shift
+ the generator string backwards. */
+ generator_offset += hwcaps_size;
+
struct cache_extension *ext = xmalloc (cache_extension_size);
ext->magic = cache_extension_magic;
- ext->count = cache_extension_count;
- for (int i = 0; i < cache_extension_count; ++i)
- {
- ext->sections[i].tag = i;
- ext->sections[i].flags = 0;
- }
+ /* Extension index current being filled. */
+ size_t xid = 0;
const char *generator
= "ldconfig " PKGVERSION RELEASE " release version " VERSION;
- ext->sections[cache_extension_tag_generator].offset
- = cache_extension_offset + cache_extension_size;
- ext->sections[cache_extension_tag_generator].size = strlen (generator);
+ ext->sections[xid].tag = cache_extension_tag_generator;
+ ext->sections[xid].flags = 0;
+ ext->sections[xid].offset = generator_offset;
+ ext->sections[xid].size = strlen (generator);
+
+ if (hwcaps_count > 0)
+ {
+ ++xid;
+ ext->sections[xid].tag = cache_extension_tag_glibc_hwcaps;
+ ext->sections[xid].flags = 0;
+ ext->sections[xid].offset = hwcaps_offset;
+ ext->sections[xid].size = hwcaps_size;
+ }
+
+ ++xid;
+ ext->count = xid;
+ assert (xid <= cache_extension_count);
- if (write (fd, ext, cache_extension_size) != cache_extension_size
+ size_t ext_size = (offsetof (struct cache_extension, sections)
+ + xid * sizeof (struct cache_extension_section));
+ if (write (fd, ext, ext_size) != ext_size
+ || write (fd, hwcaps_array, hwcaps_size) != hwcaps_size
|| write (fd, generator, strlen (generator)) != strlen (generator))
error (EXIT_FAILURE, errno, _("Writing of cache extension data failed"));
@@ -373,6 +554,8 @@ save_cache (const char *cache_name)
{
/* The cache entries are sorted already, save them in this order. */
+ assign_glibc_hwcaps_indices ();
+
struct cache_entry *entry;
/* Number of cache entries. */
int cache_entry_count = 0;
@@ -474,7 +657,11 @@ save_cache (const char *cache_name)
struct. */
file_entries_new->libs[idx_new].flags = entry->flags;
file_entries_new->libs[idx_new].osversion = entry->osversion;
- file_entries_new->libs[idx_new].hwcap = entry->hwcap;
+ if (entry->hwcaps == NULL)
+ file_entries_new->libs[idx_new].hwcap = entry->hwcap;
+ else
+ file_entries_new->libs[idx_new].hwcap
+ = DL_CACHE_HWCAP_EXTENSION | entry->hwcaps->section_index;
file_entries_new->libs[idx_new].key
= str_offset + entry->lib->offset;
file_entries_new->libs[idx_new].value
@@ -554,7 +741,7 @@ save_cache (const char *cache_name)
/* Align file position to 4. */
off64_t old_offset = lseek64 (fd, extension_offset, SEEK_SET);
assert ((unsigned long long int) (extension_offset - old_offset) < 4);
- write_extensions (fd, extension_offset);
+ write_extensions (fd, str_offset, extension_offset);
}
/* Make sure user can always read cache file */
@@ -588,27 +775,35 @@ save_cache (const char *cache_name)
/* Add one library to the cache. */
void
-add_to_cache (const char *path, const char *lib, int flags,
- unsigned int osversion, uint64_t hwcap)
+add_to_cache (const char *path, const char *filename, const char *soname,
+ int flags, unsigned int osversion, uint64_t hwcap,
+ struct glibc_hwcaps_subdirectory *hwcaps)
{
struct cache_entry *new_entry = xmalloc (sizeof (*new_entry));
struct stringtable_entry *path_interned;
{
char *p;
- if (asprintf (&p, "%s/%s", path, lib) < 0)
+ if (asprintf (&p, "%s/%s", path, filename) < 0)
error (EXIT_FAILURE, errno, _("Could not create library path"));
path_interned = stringtable_add (&strings, p);
free (p);
}
- new_entry->lib = stringtable_add (&strings, lib);
+ new_entry->lib = stringtable_add (&strings, soname);
new_entry->path = path_interned;
new_entry->flags = flags;
new_entry->osversion = osversion;
new_entry->hwcap = hwcap;
+ new_entry->hwcaps = hwcaps;
new_entry->bits_hwcap = 0;
+ if (hwcaps != NULL)
+ {
+ assert (hwcap == 0);
+ hwcaps->used = true;
+ }
+
/* Count the number of bits set in the masked value. */
for (size_t i = 0;
(~((1ULL << i) - 1) & hwcap) != 0 && i < 8 * sizeof (hwcap); ++i)
diff --git a/elf/ldconfig.c b/elf/ldconfig.c
index 006198fe59..10927a8c7f 100644
--- a/elf/ldconfig.c
+++ b/elf/ldconfig.c
@@ -16,6 +16,7 @@
along with this program; if not, see <https://www.gnu.org/licenses/>. */
#define PROCINFO_CLASS static
+#include <assert.h>
#include <alloca.h>
#include <argp.h>
#include <dirent.h>
@@ -41,6 +42,7 @@
#include <ldconfig.h>
#include <dl-cache.h>
+#include <dl-hwcaps.h>
#include <dl-procinfo.h>
@@ -85,6 +87,10 @@ struct dir_entry
dev_t dev;
const char *from_file;
int from_line;
+
+ /* Non-NULL for subdirectories under a glibc-hwcaps subdirectory. */
+ struct glibc_hwcaps_subdirectory *hwcaps;
+
struct dir_entry *next;
};
@@ -338,17 +344,20 @@ new_sub_entry (const struct dir_entry *entry, const char *path,
new_entry->from_line = entry->from_line;
new_entry->path = xstrdup (path);
new_entry->flag = entry->flag;
+ new_entry->hwcaps = NULL;
new_entry->next = NULL;
new_entry->ino = st->st_ino;
new_entry->dev = st->st_dev;
return new_entry;
}
-/* Add a single directory entry. */
-static void
+/* Add a single directory entry. Return true if the directory is
+ actually added (because it is not a duplicate). */
+static bool
add_single_dir (struct dir_entry *entry, int verbose)
{
struct dir_entry *ptr, *prev;
+ bool added = true;
ptr = dir_entries;
prev = ptr;
@@ -368,6 +377,7 @@ add_single_dir (struct dir_entry *entry, int verbose)
ptr->flag = entry->flag;
free (entry->path);
free (entry);
+ added = false;
break;
}
prev = ptr;
@@ -378,6 +388,73 @@ add_single_dir (struct dir_entry *entry, int verbose)
dir_entries = entry;
else if (ptr == NULL)
prev->next = entry;
+ return added;
+}
+
+/* Check if PATH contains a "glibc-hwcaps" subdirectory. If so, queue
+ its subdirectories for glibc-hwcaps processing. */
+static void
+add_glibc_hwcaps_subdirectories (struct dir_entry *entry, const char *path)
+{
+ /* glibc-hwcaps subdirectories do not nest. */
+ assert (entry->hwcaps == NULL);
+
+ char *glibc_hwcaps;
+ if (asprintf (&glibc_hwcaps, "%s/" GLIBC_HWCAPS_SUBDIRECTORY, path) < 0)
+ error (EXIT_FAILURE, errno, _("Could not form glibc-hwcaps path"));
+
+ DIR *dir = opendir (glibc_hwcaps);
+ if (dir != NULL)
+ {
+ while (true)
+ {
+ errno = 0;
+ struct dirent64 *e = readdir64 (dir);
+ if (e == NULL)
+ {
+ if (errno == 0)
+ break;
+ else
+ error (EXIT_FAILURE, errno, _("Listing directory %s"), path);
+ }
+
+ /* Ignore hidden subdirectories, including "." and "..", and
+ regular files. File names containing a ':' cannot be
+ looked up by the dynamic loader, so skip those as
+ well. */
+ if (e->d_name[0] == '.' || e->d_type == DT_REG
+ || strchr (e->d_name, ':') != NULL)
+ continue;
+
+ /* See if this entry eventually resolves to a directory. */
+ struct stat64 st;
+ if (fstatat64 (dirfd (dir), e->d_name, &st, 0) < 0)
+ /* Ignore unreadable entries. */
+ continue;
+
+ if (S_ISDIR (st.st_mode))
+ {
+ /* This is a directory, so it needs to be scanned for
+ libraries, associated with the hwcaps implied by the
+ subdirectory name. */
+ char *new_path;
+ if (asprintf (&new_path, "%s/" GLIBC_HWCAPS_SUBDIRECTORY "/%s",
+ /* Use non-canonicalized path here. */
+ entry->path, e->d_name) < 0)
+ error (EXIT_FAILURE, errno,
+ _("Could not form glibc-hwcaps path"));
+ struct dir_entry *new_entry = new_sub_entry (entry, new_path,
+ &st);
+ free (new_path);
+ new_entry->hwcaps = new_glibc_hwcaps_subdirectory (e->d_name);
+ add_single_dir (new_entry, 0);
+ }
+ }
+
+ closedir (dir);
+ }
+
+ free (glibc_hwcaps);
}
/* Add one directory to the list of directories to process. */
@@ -386,6 +463,7 @@ add_dir_1 (const char *line, const char *from_file, int from_line)
{
unsigned int i;
struct dir_entry *entry = xmalloc (sizeof (struct dir_entry));
+ entry->hwcaps = NULL;
entry->next = NULL;
entry->from_file = strdup (from_file);
@@ -443,7 +521,9 @@ add_dir_1 (const char *line, const char *from_file, int from_line)
entry->ino = stat_buf.st_ino;
entry->dev = stat_buf.st_dev;
- add_single_dir (entry, 1);
+ if (add_single_dir (entry, 1))
+ /* Add glibc-hwcaps subdirectories if present. */
+ add_glibc_hwcaps_subdirectories (entry, path);
}
if (opt_chroot)
@@ -695,15 +775,27 @@ struct dlib_entry
static void
search_dir (const struct dir_entry *entry)
{
- uint64_t hwcap = path_hwcap (entry->path);
- if (opt_verbose)
+ uint64_t hwcap;
+ if (entry->hwcaps == NULL)
{
- if (hwcap != 0)
- printf ("%s: (hwcap: %#.16" PRIx64 ")", entry->path, hwcap);
- else
- printf ("%s:", entry->path);
- printf (_(" (from %s:%d)\n"), entry->from_file, entry->from_line);
+ hwcap = path_hwcap (entry->path);
+ if (opt_verbose)
+ {
+ if (hwcap != 0)
+ printf ("%s: (hwcap: %#.16" PRIx64 ")", entry->path, hwcap);
+ else
+ printf ("%s:", entry->path);
+ }
}
+ else
+ {
+ hwcap = 0;
+ if (opt_verbose)
+ printf ("%s: (hwcap: \"%s\")", entry->path,
+ glibc_hwcaps_subdirectory_name (entry->hwcaps));
+ }
+ if (opt_verbose)
+ printf (_(" (from %s:%d)\n"), entry->from_file, entry->from_line);
char *dir_name;
char *real_file_name;
@@ -745,13 +837,15 @@ search_dir (const struct dir_entry *entry)
&& direntry->d_type != DT_DIR)
continue;
/* Does this file look like a shared library or is it a hwcap
- subdirectory? The dynamic linker is also considered as
+ subdirectory (if not already processing a glibc-hwcaps
+ subdirectory)? The dynamic linker is also considered as
shared library. */
if (((strncmp (direntry->d_name, "lib", 3) != 0
&& strncmp (direntry->d_name, "ld-", 3) != 0)
|| strstr (direntry->d_name, ".so") == NULL)
&& (direntry->d_type == DT_REG
- || !is_hwcap_platform (direntry->d_name)))
+ || (entry->hwcaps == NULL
+ && !is_hwcap_platform (direntry->d_name))))
continue;
size_t len = strlen (direntry->d_name);
@@ -799,7 +893,7 @@ search_dir (const struct dir_entry *entry)
}
struct stat64 stat_buf;
- int is_dir;
+ bool is_dir;
int is_link = S_ISLNK (lstat_buf.st_mode);
if (is_link)
{
@@ -837,7 +931,10 @@ search_dir (const struct dir_entry *entry)
else
is_dir = S_ISDIR (lstat_buf.st_mode);
- if (is_dir && is_hwcap_platform (direntry->d_name))
+ /* No descending into subdirectories if this directory is a
+ glibc-hwcaps subdirectory (which are not recursive). */
+ if (entry->hwcaps == NULL
+ && is_dir && is_hwcap_platform (direntry->d_name))
{
if (!is_link
&& direntry->d_type != DT_UNKNOWN
@@ -1028,13 +1125,31 @@ search_dir (const struct dir_entry *entry)
struct dlib_entry *dlib_ptr;
for (dlib_ptr = dlibs; dlib_ptr != NULL; dlib_ptr = dlib_ptr->next)
{
- /* Don't create links to links. */
- if (dlib_ptr->is_link == 0)
- create_links (dir_name, entry->path, dlib_ptr->name,
- dlib_ptr->soname);
+ /* The cached file name is the soname for non-glibc-hwcaps
+ subdirectories (relying on symbolic links; this helps with
+ library updates that change the file name), and the actual
+ file for glibc-hwcaps subdirectories. */
+ const char *filename;
+ if (entry->hwcaps == NULL)
+ {
+ /* Don't create links to links. */
+ if (dlib_ptr->is_link == 0)
+ create_links (dir_name, entry->path, dlib_ptr->name,
+ dlib_ptr->soname);
+ filename = dlib_ptr->soname;
+ }
+ else
+ {
+ /* Do not create links in glibc-hwcaps subdirectories, but
+ still log the cache addition. */
+ if (opt_verbose)
+ printf ("\t%s -> %s\n", dlib_ptr->soname, dlib_ptr->name);
+ filename = dlib_ptr->name;
+ }
if (opt_build_cache)
- add_to_cache (entry->path, dlib_ptr->soname, dlib_ptr->flag,
- dlib_ptr->osversion, hwcap);
+ add_to_cache (entry->path, filename, dlib_ptr->soname,
+ dlib_ptr->flag, dlib_ptr->osversion,
+ hwcap, entry->hwcaps);
}
/* Free all resources. */
diff --git a/sysdeps/generic/dl-cache.h b/sysdeps/generic/dl-cache.h
index cdc24c8009..85a6bdd93c 100644
--- a/sysdeps/generic/dl-cache.h
+++ b/sysdeps/generic/dl-cache.h
@@ -99,6 +99,23 @@ struct file_entry_new
uint64_t hwcap; /* Hwcap entry. */
};
+/* This bit in the hwcap field of struct file_entry_new indicates that
+ the lower 32 bits contain an index into the
+ cache_extension_tag_glibc_hwcaps section. Older glibc versions do
+ not know about this HWCAP bit, so they will ignore these
+ entries. */
+#define DL_CACHE_HWCAP_EXTENSION (1ULL << 62)
+
+/* Return true if the ENTRY->hwcap value indicates that
+ DL_CACHE_HWCAP_EXTENSION is used. */
+static inline bool
+dl_cache_hwcap_extension (struct file_entry_new *entry)
+{
+ /* If DL_CACHE_HWCAP_EXTENSION is set, but other bits as well, this
+ is a different kind of extension. */
+ return (entry->hwcap >> 32) == (DL_CACHE_HWCAP_EXTENSION >> 32);
+}
+
/* See flags member of struct cache_file_new below. */
enum
{
@@ -180,6 +197,17 @@ enum cache_extension_tag
cache file. */
cache_extension_tag_generator,
+ /* glibc-hwcaps subdirectory information. An array of uint32_t
+ values, which are indices into the string table. The strings
+ are sorted lexicographically (according to strcmp). The extra
+ level of indirection (instead of using string table indices
+ directly) allows the dynamic loader to compute the preference
+ order of the hwcaps names more efficiently.
+
+ For this section, 4-byte alignment is required, and the section
+ size must be a multiple of 4. */
+ cache_extension_tag_glibc_hwcaps,
+
/* Total number of known cache extension tags. */
cache_extension_count
};
@@ -234,6 +262,27 @@ struct cache_extension_all_loaded
struct cache_extension_loaded sections[cache_extension_count];
};
+/* Performs basic data validation based on section tag, and removes
+ the sections which are invalid. */
+static void
+cache_extension_verify (struct cache_extension_all_loaded *loaded)
+{
+ {
+ /* Section must not be empty, it must be aligned at 4 bytes, and
+ the size must be a multiple of 4. */
+ struct cache_extension_loaded *hwcaps
+ = &loaded->sections[cache_extension_tag_glibc_hwcaps];
+ if (hwcaps->size == 0
+ || ((uintptr_t) hwcaps->base % 4) != 0
+ || (hwcaps->size % 4) != 0)
+ {
+ hwcaps->base = NULL;
+ hwcaps->size = 0;
+ hwcaps->flags = 0;
+ }
+ }
+}
+
static bool __attribute__ ((unused))
cache_extension_load (const struct cache_file_new *cache,
const void *file_base, size_t file_size,
@@ -280,6 +329,7 @@ cache_extension_load (const struct cache_file_new *cache,
loaded->sections[tag].size = ext->sections[i].size;
loaded->sections[tag].flags = ext->sections[i].flags;
}
+ cache_extension_verify (loaded);
return true;
}
diff --git a/sysdeps/generic/ldconfig.h b/sysdeps/generic/ldconfig.h
index cfec9d4668..1ad1528890 100644
--- a/sysdeps/generic/ldconfig.h
+++ b/sysdeps/generic/ldconfig.h
@@ -57,8 +57,22 @@ extern void init_cache (void);
extern void save_cache (const char *cache_name);
-extern void add_to_cache (const char *path, const char *lib, int flags,
- unsigned int osversion, uint64_t hwcap);
+struct glibc_hwcaps_subdirectory;
+
+/* Return a struct describing the subdirectory for NAME. Reuse an
+ existing struct if it exists. */
+struct glibc_hwcaps_subdirectory *new_glibc_hwcaps_subdirectory
+ (const char *name);
+
+/* Returns the name that was specified when
+ add_glibc_hwcaps_subdirectory was called. */
+const char *glibc_hwcaps_subdirectory_name
+ (const struct glibc_hwcaps_subdirectory *);
+
+extern void add_to_cache (const char *path, const char *filename,
+ const char *soname,
+ int flags, unsigned int osversion, uint64_t hwcap,
+ struct glibc_hwcaps_subdirectory *);
extern void init_aux_cache (void);
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 08/11] elf: Process glibc-hwcaps subdirectories in ldconfig
2020-11-09 18:41 ` [PATCH 08/11] elf: Process glibc-hwcaps subdirectories " Florian Weimer
@ 2020-11-30 19:46 ` Adhemerval Zanella
0 siblings, 0 replies; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-30 19:46 UTC (permalink / raw)
To: libc-alpha, Florian Weimer
On 09/11/2020 15:41, Florian Weimer via Libc-alpha wrote:
> Libraries from these subdirectories are added to the cache
> with a special hwcap bit DL_CACHE_HWCAP_EXTENSION, so that
> they are ignored by older dynamic loaders.
LGTM, thanks.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
> elf/cache.c | 257 ++++++++++++++++++++++++++++++++-----
> elf/ldconfig.c | 155 +++++++++++++++++++---
> sysdeps/generic/dl-cache.h | 50 ++++++++
> sysdeps/generic/ldconfig.h | 18 ++-
> 4 files changed, 427 insertions(+), 53 deletions(-)
>
> diff --git a/elf/cache.c b/elf/cache.c
> index 5a6ee20e86..b03c5319f8 100644
> --- a/elf/cache.c
> +++ b/elf/cache.c
> @@ -40,6 +40,105 @@
> /* Used to store library names, paths, and other strings. */
> static struct stringtable strings;
>
> +/* Keeping track of "glibc-hwcaps" subdirectories. During cache
> + construction, a linear search by name is performed to deduplicate
> + entries. */
> +struct glibc_hwcaps_subdirectory
> +{
> + struct glibc_hwcaps_subdirectory *next;
> +
> + /* Interned string with the subdirectory name. */
> + struct stringtable_entry *name;
> +
> + /* Array index in the cache_extension_tag_glibc_hwcaps section in
> + the stored cached file. This is computed after all the
> + subdirectories have been processed, so that subdirectory names in
> + the extension section can be sorted. */
> + uint32_t section_index;
> +
> + /* True if the subdirectory is actually used for anything. */
> + bool used;
> +};
> +
> +const char *
> +glibc_hwcaps_subdirectory_name (const struct glibc_hwcaps_subdirectory *dir)
> +{
> + return dir->name->string;
> +}
> +
> +/* Linked list of known hwcaps subdirecty names. */
> +static struct glibc_hwcaps_subdirectory *hwcaps;
> +
> +struct glibc_hwcaps_subdirectory *
> +new_glibc_hwcaps_subdirectory (const char *name)
> +{
> + struct stringtable_entry *name_interned = stringtable_add (&strings, name);
> + for (struct glibc_hwcaps_subdirectory *p = hwcaps; p != NULL; p = p->next)
> + if (p->name == name_interned)
> + return p;
> + struct glibc_hwcaps_subdirectory *p = xmalloc (sizeof (*p));
> + p->next = hwcaps;
> + p->name = name_interned;
> + p->section_index = 0;
> + p->used = false;
> + hwcaps = p;
> + return p;
> +}
> +
> +/* Helper for sorting struct glibc_hwcaps_subdirectory elements by
> + name. */
> +static int
> +assign_glibc_hwcaps_indices_compare (const void *l, const void *r)
> +{
> + const struct glibc_hwcaps_subdirectory *left
> + = *(struct glibc_hwcaps_subdirectory **)l;
> + const struct glibc_hwcaps_subdirectory *right
> + = *(struct glibc_hwcaps_subdirectory **)r;
> + return strcmp (glibc_hwcaps_subdirectory_name (left),
> + glibc_hwcaps_subdirectory_name (right));
> +}
> +
Ok.
> +/* Count the number of hwcaps subdirectories which are actually
> + used. */
> +static size_t
> +glibc_hwcaps_count (void)
> +{
> + size_t count = 0;
> + for (struct glibc_hwcaps_subdirectory *p = hwcaps; p != NULL; p = p->next)
> + if (p->used)
> + ++count;
> + return count;
> +}
> +
> +/* Compute the section_index fields for all */
> +static void
> +assign_glibc_hwcaps_indices (void)
> +{
> + /* Convert the linked list into an array, so that we can use qsort.
> + Only copy the subdirectories which are actually used. */
> + size_t count = glibc_hwcaps_count ();
> + struct glibc_hwcaps_subdirectory **array
> + = xmalloc (sizeof (*array) * count);
> + {
> + size_t i = 0;
> + for (struct glibc_hwcaps_subdirectory *p = hwcaps; p != NULL; p = p->next)
> + if (p->used)
> + {
> + array[i] = p;
> + ++i;
> + }
> + assert (i == count);
> + }
> +
> + qsort (array, count, sizeof (*array), assign_glibc_hwcaps_indices_compare);
> +
> + /* Assign the array indices. */
> + for (size_t i = 0; i < count; ++i)
> + array[i]->section_index = i;
> +
> + free (array);
> +}
> +
> struct cache_entry
> {
> struct stringtable_entry *lib; /* Library name. */
> @@ -48,6 +147,10 @@ struct cache_entry
> unsigned int osversion; /* Required OS version. */
> uint64_t hwcap; /* Important hardware capabilities. */
> int bits_hwcap; /* Number of bits set in hwcap. */
> +
> + /* glibc-hwcaps subdirectory. If not NULL, hwcap must be zero. */
> + struct glibc_hwcaps_subdirectory *hwcaps;
> +
> struct cache_entry *next; /* Next entry in list. */
> };
>
> @@ -60,7 +163,7 @@ static const char *flag_descr[] =
> /* Print a single entry. */
> static void
> print_entry (const char *lib, int flag, unsigned int osversion,
> - uint64_t hwcap, const char *key)
> + uint64_t hwcap, const char *hwcap_string, const char *key)
> {
> printf ("\t%s (", lib);
> switch (flag & FLAG_TYPE_MASK)
> @@ -132,7 +235,9 @@ print_entry (const char *lib, int flag, unsigned int osversion,
> printf (",%d", flag & FLAG_REQUIRED_MASK);
> break;
> }
> - if (hwcap != 0)
> + if (hwcap_string != NULL)
> + printf (", hwcap: \"%s\"", hwcap_string);
> + else if (hwcap != 0)
> printf (", hwcap: %#.16" PRIx64, hwcap);
> if (osversion != 0)
> {
> @@ -158,6 +263,29 @@ print_entry (const char *lib, int flag, unsigned int osversion,
> printf (") => %s\n", key);
> }
>
> +/* Returns the string with the name of the glibcs-hwcaps subdirectory
> + associated with ENTRY->hwcap. file_base must be the base address
> + for string table indices. */
> +static const char *
> +glibc_hwcaps_string (struct cache_extension_all_loaded *ext,
> + const void *file_base, size_t file_size,
> + struct file_entry_new *entry)
> +{
> + const uint32_t *hwcaps_array
> + = ext->sections[cache_extension_tag_glibc_hwcaps].base;
> + if (dl_cache_hwcap_extension (entry) && hwcaps_array != NULL)
> + {
> + uint32_t index = (uint32_t) entry->hwcap;
> + if (index < ext->sections[cache_extension_tag_glibc_hwcaps].size / 4)
> + {
> + uint32_t string_table_index = hwcaps_array[index];
> + if (string_table_index < file_size)
> + return file_base + string_table_index;
> + }
> + }
> + return NULL;
> +}
> +
> /* Print an error and exit if the new-file cache is internally
> inconsistent. */
> static void
Ok.
> @@ -167,9 +295,7 @@ check_new_cache (struct cache_file_new *cache)
> error (EXIT_FAILURE, 0, _("Cache file has wrong endianness.\n"));
> }
>
> -/* Print the extension information at the cache at start address
> - FILE_BASE, of length FILE_SIZE bytes. The new-format cache header
> - is at CACHE, and the file name for diagnostics is CACHE_NAME. */
> +/* Print the extension information in *EXT. */
> static void
> print_extensions (struct cache_extension_all_loaded *ext)
> {
> @@ -266,7 +392,7 @@ print_cache (const char *cache_name)
> /* Print everything. */
> for (unsigned int i = 0; i < cache->nlibs; i++)
> print_entry (cache_data + cache->libs[i].key,
> - cache->libs[i].flags, 0, 0,
> + cache->libs[i].flags, 0, 0, NULL,
> cache_data + cache->libs[i].value);
> }
> else if (format == 1)
Ok.
> @@ -281,11 +407,16 @@ print_cache (const char *cache_name)
>
> /* Print everything. */
> for (unsigned int i = 0; i < cache_new->nlibs; i++)
> - print_entry (cache_data + cache_new->libs[i].key,
> - cache_new->libs[i].flags,
> - cache_new->libs[i].osversion,
> - cache_new->libs[i].hwcap,
> - cache_data + cache_new->libs[i].value);
> + {
> + const char *hwcaps_string
> + = glibc_hwcaps_string (&ext, cache, cache_size,
> + &cache_new->libs[i]);
> + print_entry (cache_data + cache_new->libs[i].key,
> + cache_new->libs[i].flags,
> + cache_new->libs[i].osversion,
> + cache_new->libs[i].hwcap, hwcaps_string,
> + cache_data + cache_new->libs[i].value);
> + }
> print_extensions (&ext);
> }
> /* Cleanup. */
Ok.
> @@ -311,8 +442,23 @@ compare (const struct cache_entry *e1, const struct cache_entry *e2)
> return 1;
> else if (e1->flags > e2->flags)
> return -1;
> + /* Keep the glibc-hwcaps extension entries before the regular
> + entries, and sort them by their names. search_cache in
> + dl-cache.c stops searching once the first non-extension entry
> + is found, so the extension entries need to come first. */
> + else if (e1->hwcaps != NULL && e2->hwcaps == NULL)
> + return -1;
> + else if (e1->hwcaps == NULL && e2->hwcaps != NULL)
> + return 1;
> + else if (e1->hwcaps != NULL && e2->hwcaps != NULL)
> + {
> + res = strcmp (glibc_hwcaps_subdirectory_name (e1->hwcaps),
> + glibc_hwcaps_subdirectory_name (e2->hwcaps));
> + if (res != 0)
> + return res;
> + }
> /* Sort by most specific hwcap. */
> - else if (e2->bits_hwcap > e1->bits_hwcap)
> + if (e2->bits_hwcap > e1->bits_hwcap)
> return 1;
> else if (e2->bits_hwcap < e1->bits_hwcap)
> return -1;
Ok.
> @@ -337,30 +483,65 @@ enum
> * sizeof (struct cache_extension_section)))
> };
>
> -/* Write the cache extensions to FD. The extension directory is
> - assumed to be located at CACHE_EXTENSION_OFFSET. */
> +/* Write the cache extensions to FD. The string table is shifted by
> + STRING_TABLE_OFFSET. The extension directory is assumed to be
> + located at CACHE_EXTENSION_OFFSET. assign_glibc_hwcaps_indices
> + must have been called. */
> static void
> -write_extensions (int fd, uint32_t cache_extension_offset)
> +write_extensions (int fd, uint32_t str_offset,
> + uint32_t cache_extension_offset)
> {
> assert ((cache_extension_offset % 4) == 0);
>
> + /* The length and contents of the glibc-hwcaps section. */
> + uint32_t hwcaps_count = glibc_hwcaps_count ();
> + uint32_t hwcaps_offset = cache_extension_offset + cache_extension_size;
> + uint32_t hwcaps_size = hwcaps_count * sizeof (uint32_t);
> + uint32_t *hwcaps_array = xmalloc (hwcaps_size);
> + for (struct glibc_hwcaps_subdirectory *p = hwcaps; p != NULL; p = p->next)
> + if (p->used)
> + hwcaps_array[p->section_index] = str_offset + p->name->offset;
> +
> + /* This is the offset of the generator string. */
> + uint32_t generator_offset = hwcaps_offset;
> + if (hwcaps_count == 0)
> + /* There is no section for the hwcaps subdirectories. */
> + generator_offset -= sizeof (struct cache_extension_section);
> + else
> + /* The string table indices for the hwcaps subdirectories shift
> + the generator string backwards. */
> + generator_offset += hwcaps_size;
> +
> struct cache_extension *ext = xmalloc (cache_extension_size);
> ext->magic = cache_extension_magic;
> - ext->count = cache_extension_count;
>
> - for (int i = 0; i < cache_extension_count; ++i)
> - {
> - ext->sections[i].tag = i;
> - ext->sections[i].flags = 0;
> - }
> + /* Extension index current being filled. */
> + size_t xid = 0;
>
> const char *generator
> = "ldconfig " PKGVERSION RELEASE " release version " VERSION;
> - ext->sections[cache_extension_tag_generator].offset
> - = cache_extension_offset + cache_extension_size;
> - ext->sections[cache_extension_tag_generator].size = strlen (generator);
> + ext->sections[xid].tag = cache_extension_tag_generator;
> + ext->sections[xid].flags = 0;
> + ext->sections[xid].offset = generator_offset;
> + ext->sections[xid].size = strlen (generator);
> +
> + if (hwcaps_count > 0)
> + {
> + ++xid;
> + ext->sections[xid].tag = cache_extension_tag_glibc_hwcaps;
> + ext->sections[xid].flags = 0;
> + ext->sections[xid].offset = hwcaps_offset;
> + ext->sections[xid].size = hwcaps_size;
> + }
> +
> + ++xid;
> + ext->count = xid;
> + assert (xid <= cache_extension_count);
>
> - if (write (fd, ext, cache_extension_size) != cache_extension_size
> + size_t ext_size = (offsetof (struct cache_extension, sections)
> + + xid * sizeof (struct cache_extension_section));
> + if (write (fd, ext, ext_size) != ext_size
> + || write (fd, hwcaps_array, hwcaps_size) != hwcaps_size
> || write (fd, generator, strlen (generator)) != strlen (generator))
> error (EXIT_FAILURE, errno, _("Writing of cache extension data failed"));
>
Ok.
> @@ -373,6 +554,8 @@ save_cache (const char *cache_name)
> {
> /* The cache entries are sorted already, save them in this order. */
>
> + assign_glibc_hwcaps_indices ();
> +
> struct cache_entry *entry;
> /* Number of cache entries. */
> int cache_entry_count = 0;
> @@ -474,7 +657,11 @@ save_cache (const char *cache_name)
> struct. */
> file_entries_new->libs[idx_new].flags = entry->flags;
> file_entries_new->libs[idx_new].osversion = entry->osversion;
> - file_entries_new->libs[idx_new].hwcap = entry->hwcap;
> + if (entry->hwcaps == NULL)
> + file_entries_new->libs[idx_new].hwcap = entry->hwcap;
> + else
> + file_entries_new->libs[idx_new].hwcap
> + = DL_CACHE_HWCAP_EXTENSION | entry->hwcaps->section_index;
> file_entries_new->libs[idx_new].key
> = str_offset + entry->lib->offset;
> file_entries_new->libs[idx_new].value
Ok.
> @@ -554,7 +741,7 @@ save_cache (const char *cache_name)
> /* Align file position to 4. */
> off64_t old_offset = lseek64 (fd, extension_offset, SEEK_SET);
> assert ((unsigned long long int) (extension_offset - old_offset) < 4);
> - write_extensions (fd, extension_offset);
> + write_extensions (fd, str_offset, extension_offset);
> }
>
> /* Make sure user can always read cache file */
> @@ -588,27 +775,35 @@ save_cache (const char *cache_name)
>
> /* Add one library to the cache. */
> void
> -add_to_cache (const char *path, const char *lib, int flags,
> - unsigned int osversion, uint64_t hwcap)
> +add_to_cache (const char *path, const char *filename, const char *soname,
> + int flags, unsigned int osversion, uint64_t hwcap,
> + struct glibc_hwcaps_subdirectory *hwcaps)
> {
> struct cache_entry *new_entry = xmalloc (sizeof (*new_entry));
>
> struct stringtable_entry *path_interned;
> {
> char *p;
> - if (asprintf (&p, "%s/%s", path, lib) < 0)
> + if (asprintf (&p, "%s/%s", path, filename) < 0)
> error (EXIT_FAILURE, errno, _("Could not create library path"));
> path_interned = stringtable_add (&strings, p);
> free (p);
> }
>
> - new_entry->lib = stringtable_add (&strings, lib);
> + new_entry->lib = stringtable_add (&strings, soname);
> new_entry->path = path_interned;
> new_entry->flags = flags;
> new_entry->osversion = osversion;
> new_entry->hwcap = hwcap;
> + new_entry->hwcaps = hwcaps;
> new_entry->bits_hwcap = 0;
>
> + if (hwcaps != NULL)
> + {
> + assert (hwcap == 0);
> + hwcaps->used = true;
> + }
> +
> /* Count the number of bits set in the masked value. */
> for (size_t i = 0;
> (~((1ULL << i) - 1) & hwcap) != 0 && i < 8 * sizeof (hwcap); ++i)
Ok.
> diff --git a/elf/ldconfig.c b/elf/ldconfig.c
> index 006198fe59..10927a8c7f 100644
> --- a/elf/ldconfig.c
> +++ b/elf/ldconfig.c
> @@ -16,6 +16,7 @@
> along with this program; if not, see <https://www.gnu.org/licenses/>. */
>
> #define PROCINFO_CLASS static
> +#include <assert.h>
> #include <alloca.h>
> #include <argp.h>
> #include <dirent.h>
> @@ -41,6 +42,7 @@
>
> #include <ldconfig.h>
> #include <dl-cache.h>
> +#include <dl-hwcaps.h>
>
> #include <dl-procinfo.h>
>
> @@ -85,6 +87,10 @@ struct dir_entry
> dev_t dev;
> const char *from_file;
> int from_line;
> +
> + /* Non-NULL for subdirectories under a glibc-hwcaps subdirectory. */
> + struct glibc_hwcaps_subdirectory *hwcaps;
> +
> struct dir_entry *next;
> };
>
Ok.
> @@ -338,17 +344,20 @@ new_sub_entry (const struct dir_entry *entry, const char *path,
> new_entry->from_line = entry->from_line;
> new_entry->path = xstrdup (path);
> new_entry->flag = entry->flag;
> + new_entry->hwcaps = NULL;
> new_entry->next = NULL;
> new_entry->ino = st->st_ino;
> new_entry->dev = st->st_dev;
> return new_entry;
> }
>
> -/* Add a single directory entry. */
> -static void
> +/* Add a single directory entry. Return true if the directory is
> + actually added (because it is not a duplicate). */
> +static bool
> add_single_dir (struct dir_entry *entry, int verbose)
> {
> struct dir_entry *ptr, *prev;
> + bool added = true;
>
> ptr = dir_entries;
> prev = ptr;
> @@ -368,6 +377,7 @@ add_single_dir (struct dir_entry *entry, int verbose)
> ptr->flag = entry->flag;
> free (entry->path);
> free (entry);
> + added = false;
> break;
> }
> prev = ptr;
> @@ -378,6 +388,73 @@ add_single_dir (struct dir_entry *entry, int verbose)
> dir_entries = entry;
> else if (ptr == NULL)
> prev->next = entry;
> + return added;
> +}
> +
Ok.
> +/* Check if PATH contains a "glibc-hwcaps" subdirectory. If so, queue
> + its subdirectories for glibc-hwcaps processing. */
> +static void
> +add_glibc_hwcaps_subdirectories (struct dir_entry *entry, const char *path)
> +{
> + /* glibc-hwcaps subdirectories do not nest. */
> + assert (entry->hwcaps == NULL);
> +
> + char *glibc_hwcaps;
> + if (asprintf (&glibc_hwcaps, "%s/" GLIBC_HWCAPS_SUBDIRECTORY, path) < 0)
> + error (EXIT_FAILURE, errno, _("Could not form glibc-hwcaps path"));
> +
> + DIR *dir = opendir (glibc_hwcaps);
> + if (dir != NULL)
> + {
> + while (true)
> + {
> + errno = 0;
> + struct dirent64 *e = readdir64 (dir);
> + if (e == NULL)
> + {
> + if (errno == 0)
> + break;
> + else
> + error (EXIT_FAILURE, errno, _("Listing directory %s"), path);
> + }
> +
> + /* Ignore hidden subdirectories, including "." and "..", and
> + regular files. File names containing a ':' cannot be
> + looked up by the dynamic loader, so skip those as
> + well. */
> + if (e->d_name[0] == '.' || e->d_type == DT_REG
> + || strchr (e->d_name, ':') != NULL)
> + continue;
> +
> + /* See if this entry eventually resolves to a directory. */
> + struct stat64 st;
> + if (fstatat64 (dirfd (dir), e->d_name, &st, 0) < 0)
> + /* Ignore unreadable entries. */
> + continue;
> +
> + if (S_ISDIR (st.st_mode))
> + {
> + /* This is a directory, so it needs to be scanned for
> + libraries, associated with the hwcaps implied by the
> + subdirectory name. */
> + char *new_path;
> + if (asprintf (&new_path, "%s/" GLIBC_HWCAPS_SUBDIRECTORY "/%s",
> + /* Use non-canonicalized path here. */
> + entry->path, e->d_name) < 0)
> + error (EXIT_FAILURE, errno,
> + _("Could not form glibc-hwcaps path"));
> + struct dir_entry *new_entry = new_sub_entry (entry, new_path,
> + &st);
> + free (new_path);
> + new_entry->hwcaps = new_glibc_hwcaps_subdirectory (e->d_name);
> + add_single_dir (new_entry, 0);
> + }
> + }
> +
> + closedir (dir);
> + }
> +
> + free (glibc_hwcaps);
> }
>
> /* Add one directory to the list of directories to process. */
Ok.
> @@ -386,6 +463,7 @@ add_dir_1 (const char *line, const char *from_file, int from_line)
> {
> unsigned int i;
> struct dir_entry *entry = xmalloc (sizeof (struct dir_entry));
> + entry->hwcaps = NULL;
> entry->next = NULL;
>
> entry->from_file = strdup (from_file);
> @@ -443,7 +521,9 @@ add_dir_1 (const char *line, const char *from_file, int from_line)
> entry->ino = stat_buf.st_ino;
> entry->dev = stat_buf.st_dev;
>
> - add_single_dir (entry, 1);
> + if (add_single_dir (entry, 1))
> + /* Add glibc-hwcaps subdirectories if present. */
> + add_glibc_hwcaps_subdirectories (entry, path);
> }
>
> if (opt_chroot)
OK.
> @@ -695,15 +775,27 @@ struct dlib_entry
> static void
> search_dir (const struct dir_entry *entry)
> {
> - uint64_t hwcap = path_hwcap (entry->path);
> - if (opt_verbose)
> + uint64_t hwcap;
> + if (entry->hwcaps == NULL)
> {
> - if (hwcap != 0)
> - printf ("%s: (hwcap: %#.16" PRIx64 ")", entry->path, hwcap);
> - else
> - printf ("%s:", entry->path);
> - printf (_(" (from %s:%d)\n"), entry->from_file, entry->from_line);
> + hwcap = path_hwcap (entry->path);
> + if (opt_verbose)
> + {
> + if (hwcap != 0)
> + printf ("%s: (hwcap: %#.16" PRIx64 ")", entry->path, hwcap);
> + else
> + printf ("%s:", entry->path);
> + }
> }
Ok.
> + else
> + {
> + hwcap = 0;
> + if (opt_verbose)
> + printf ("%s: (hwcap: \"%s\")", entry->path,
> + glibc_hwcaps_subdirectory_name (entry->hwcaps));
> + }
> + if (opt_verbose)
> + printf (_(" (from %s:%d)\n"), entry->from_file, entry->from_line);
>
> char *dir_name;
> char *real_file_name;
Ok.
> @@ -745,13 +837,15 @@ search_dir (const struct dir_entry *entry)
> && direntry->d_type != DT_DIR)
> continue;
> /* Does this file look like a shared library or is it a hwcap
> - subdirectory? The dynamic linker is also considered as
> + subdirectory (if not already processing a glibc-hwcaps
> + subdirectory)? The dynamic linker is also considered as
> shared library. */
> if (((strncmp (direntry->d_name, "lib", 3) != 0
> && strncmp (direntry->d_name, "ld-", 3) != 0)
> || strstr (direntry->d_name, ".so") == NULL)
> && (direntry->d_type == DT_REG
> - || !is_hwcap_platform (direntry->d_name)))
> + || (entry->hwcaps == NULL
> + && !is_hwcap_platform (direntry->d_name))))
> continue;
>
> size_t len = strlen (direntry->d_name);
Ok.
> @@ -799,7 +893,7 @@ search_dir (const struct dir_entry *entry)
> }
>
> struct stat64 stat_buf;
> - int is_dir;
> + bool is_dir;
> int is_link = S_ISLNK (lstat_buf.st_mode);
> if (is_link)
> {
> @@ -837,7 +931,10 @@ search_dir (const struct dir_entry *entry)
> else
> is_dir = S_ISDIR (lstat_buf.st_mode);
>
> - if (is_dir && is_hwcap_platform (direntry->d_name))
> + /* No descending into subdirectories if this directory is a
> + glibc-hwcaps subdirectory (which are not recursive). */
> + if (entry->hwcaps == NULL
> + && is_dir && is_hwcap_platform (direntry->d_name))
> {
> if (!is_link
> && direntry->d_type != DT_UNKNOWN
Ok.
> @@ -1028,13 +1125,31 @@ search_dir (const struct dir_entry *entry)
> struct dlib_entry *dlib_ptr;
> for (dlib_ptr = dlibs; dlib_ptr != NULL; dlib_ptr = dlib_ptr->next)
> {
> - /* Don't create links to links. */
> - if (dlib_ptr->is_link == 0)
> - create_links (dir_name, entry->path, dlib_ptr->name,
> - dlib_ptr->soname);
> + /* The cached file name is the soname for non-glibc-hwcaps
> + subdirectories (relying on symbolic links; this helps with
> + library updates that change the file name), and the actual
> + file for glibc-hwcaps subdirectories. */
> + const char *filename;
> + if (entry->hwcaps == NULL)
> + {
> + /* Don't create links to links. */
> + if (dlib_ptr->is_link == 0)
> + create_links (dir_name, entry->path, dlib_ptr->name,
> + dlib_ptr->soname);
> + filename = dlib_ptr->soname;
> + }
> + else
> + {
> + /* Do not create links in glibc-hwcaps subdirectories, but
> + still log the cache addition. */
> + if (opt_verbose)
> + printf ("\t%s -> %s\n", dlib_ptr->soname, dlib_ptr->name);
> + filename = dlib_ptr->name;
> + }
> if (opt_build_cache)
> - add_to_cache (entry->path, dlib_ptr->soname, dlib_ptr->flag,
> - dlib_ptr->osversion, hwcap);
> + add_to_cache (entry->path, filename, dlib_ptr->soname,
> + dlib_ptr->flag, dlib_ptr->osversion,
> + hwcap, entry->hwcaps);
> }
>
> /* Free all resources. */
Ok.
> diff --git a/sysdeps/generic/dl-cache.h b/sysdeps/generic/dl-cache.h
> index cdc24c8009..85a6bdd93c 100644
> --- a/sysdeps/generic/dl-cache.h
> +++ b/sysdeps/generic/dl-cache.h
> @@ -99,6 +99,23 @@ struct file_entry_new
> uint64_t hwcap; /* Hwcap entry. */
> };
>
> +/* This bit in the hwcap field of struct file_entry_new indicates that
> + the lower 32 bits contain an index into the
> + cache_extension_tag_glibc_hwcaps section. Older glibc versions do
> + not know about this HWCAP bit, so they will ignore these
> + entries. */
> +#define DL_CACHE_HWCAP_EXTENSION (1ULL << 62)
> +
> +/* Return true if the ENTRY->hwcap value indicates that
> + DL_CACHE_HWCAP_EXTENSION is used. */
> +static inline bool
> +dl_cache_hwcap_extension (struct file_entry_new *entry)
> +{
> + /* If DL_CACHE_HWCAP_EXTENSION is set, but other bits as well, this
> + is a different kind of extension. */
> + return (entry->hwcap >> 32) == (DL_CACHE_HWCAP_EXTENSION >> 32);
> +}
> +
> /* See flags member of struct cache_file_new below. */
> enum
> {
Ok.
> @@ -180,6 +197,17 @@ enum cache_extension_tag
> cache file. */
> cache_extension_tag_generator,
>
> + /* glibc-hwcaps subdirectory information. An array of uint32_t
> + values, which are indices into the string table. The strings
> + are sorted lexicographically (according to strcmp). The extra
> + level of indirection (instead of using string table indices
> + directly) allows the dynamic loader to compute the preference
> + order of the hwcaps names more efficiently.
> +
> + For this section, 4-byte alignment is required, and the section
> + size must be a multiple of 4. */
> + cache_extension_tag_glibc_hwcaps,
> +
> /* Total number of known cache extension tags. */
> cache_extension_count
> };
Ok.
> @@ -234,6 +262,27 @@ struct cache_extension_all_loaded
> struct cache_extension_loaded sections[cache_extension_count];
> };
>
> +/* Performs basic data validation based on section tag, and removes
> + the sections which are invalid. */
> +static void
> +cache_extension_verify (struct cache_extension_all_loaded *loaded)
> +{
> + {
> + /* Section must not be empty, it must be aligned at 4 bytes, and
> + the size must be a multiple of 4. */
> + struct cache_extension_loaded *hwcaps
> + = &loaded->sections[cache_extension_tag_glibc_hwcaps];
> + if (hwcaps->size == 0
> + || ((uintptr_t) hwcaps->base % 4) != 0
> + || (hwcaps->size % 4) != 0)
> + {
> + hwcaps->base = NULL;
> + hwcaps->size = 0;
> + hwcaps->flags = 0;
> + }
> + }
> +}
> +
> static bool __attribute__ ((unused))
> cache_extension_load (const struct cache_file_new *cache,
> const void *file_base, size_t file_size,
> @@ -280,6 +329,7 @@ cache_extension_load (const struct cache_file_new *cache,
> loaded->sections[tag].size = ext->sections[i].size;
> loaded->sections[tag].flags = ext->sections[i].flags;
> }
> + cache_extension_verify (loaded);
> return true;
> }
>
Ok.
> diff --git a/sysdeps/generic/ldconfig.h b/sysdeps/generic/ldconfig.h
> index cfec9d4668..1ad1528890 100644
> --- a/sysdeps/generic/ldconfig.h
> +++ b/sysdeps/generic/ldconfig.h
> @@ -57,8 +57,22 @@ extern void init_cache (void);
>
> extern void save_cache (const char *cache_name);
>
> -extern void add_to_cache (const char *path, const char *lib, int flags,
> - unsigned int osversion, uint64_t hwcap);
> +struct glibc_hwcaps_subdirectory;
> +
> +/* Return a struct describing the subdirectory for NAME. Reuse an
> + existing struct if it exists. */
> +struct glibc_hwcaps_subdirectory *new_glibc_hwcaps_subdirectory
> + (const char *name);
> +
> +/* Returns the name that was specified when
> + add_glibc_hwcaps_subdirectory was called. */
> +const char *glibc_hwcaps_subdirectory_name
> + (const struct glibc_hwcaps_subdirectory *);
> +
> +extern void add_to_cache (const char *path, const char *filename,
> + const char *soname,
> + int flags, unsigned int osversion, uint64_t hwcap,
> + struct glibc_hwcaps_subdirectory *);
>
> extern void init_aux_cache (void);
>
>
Ok.
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (7 preceding siblings ...)
2020-11-09 18:41 ` [PATCH 08/11] elf: Process glibc-hwcaps subdirectories " Florian Weimer
@ 2020-11-09 18:41 ` Florian Weimer
2020-11-26 17:11 ` Florian Weimer
2020-12-01 17:45 ` Adhemerval Zanella
2020-11-09 18:41 ` [PATCH 10/11] x86_64: Add glibc-hwcaps support Florian Weimer
` (3 subsequent siblings)
12 siblings, 2 replies; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:41 UTC (permalink / raw)
To: libc-alpha
This recognizes the DL_CACHE_HWCAP_EXTENSION flag and picks up
the supported cache entry with the highest priority.
---
elf/Makefile | 18 ++
elf/dl-cache.c | 182 +++++++++++++++++-
elf/dl-hwcaps.c | 78 ++++++++
elf/dl-hwcaps.h | 19 ++
elf/tst-glibc-hwcaps-cache.c | 45 +++++
.../etc/ld.so.conf | 2 +
elf/tst-glibc-hwcaps-cache.root/postclean.req | 0
elf/tst-glibc-hwcaps-cache.script | 2 +
elf/tst-glibc-hwcaps-prepend-cache.c | 133 +++++++++++++
.../postclean.req | 0
10 files changed, 476 insertions(+), 3 deletions(-)
create mode 100644 elf/tst-glibc-hwcaps-cache.c
create mode 100644 elf/tst-glibc-hwcaps-cache.root/etc/ld.so.conf
create mode 100644 elf/tst-glibc-hwcaps-cache.root/postclean.req
create mode 100644 elf/tst-glibc-hwcaps-cache.script
create mode 100644 elf/tst-glibc-hwcaps-prepend-cache.c
create mode 100644 elf/tst-glibc-hwcaps-prepend-cache.root/postclean.req
diff --git a/elf/Makefile b/elf/Makefile
index e26ac16b44..80e94b9ee2 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -171,6 +171,12 @@ tests-container := \
tst-ldconfig-bad-aux-cache \
tst-ldconfig-ld_so_conf-update
+ifeq (no,$(build-hardcoded-path-in-tests))
+# This is an ld.so.cache test, and RPATH/RUNPATH in the executable
+# interferes with its test objectives.
+tests-container += tst-glibc-hwcaps-prepend-cache
+endif
+
tests := tst-tls9 tst-leaks1 \
tst-array1 tst-array2 tst-array3 tst-array4 tst-array5 \
tst-auxv tst-stringtable
@@ -1859,6 +1865,14 @@ $(objpfx)tst-glibc-hwcaps-prepend.out: \
$< > $@; \
$(evaluate-test)
+# Like tst-glibc-hwcaps-prepend, but uses a container and loads the
+# library via ld.so.cache. Test setup is contained in the test
+# itself.
+$(objpfx)tst-glibc-hwcaps-prepend-cache: $(libdl)
+$(objpfx)tst-glibc-hwcaps-prepend-cache.out: \
+ $(objpfx)tst-glibc-hwcaps-prepend-cache $(objpfx)libmarkermod1-1.so \
+ $(objpfx)libmarkermod1-2.so $(objpfx)libmarkermod1-3.so
+
# tst-glibc-hwcaps-mask checks that --glibc-hwcaps-mask can be used to
# suppress all auto-detected subdirectories.
$(objpfx)tst-glibc-hwcaps-mask: $(objpfx)libmarkermod1-1.so
@@ -1870,3 +1884,7 @@ $(objpfx)tst-glibc-hwcaps-mask.out: \
--glibc-hwcaps-mask does-not-exist \
$< > $@; \
$(evaluate-test)
+
+# Generic dependency for sysdeps implementation of
+# tst-glibc-hwcaps-cache.
+$(objpfx)tst-glibc-hwcaps-cache.out: $(objpfx)tst-glibc-hwcaps
diff --git a/elf/dl-cache.c b/elf/dl-cache.c
index 02c46ffb0c..13efc6f95f 100644
--- a/elf/dl-cache.c
+++ b/elf/dl-cache.c
@@ -35,6 +35,132 @@ static struct cache_file *cache;
static struct cache_file_new *cache_new;
static size_t cachesize;
+#ifdef SHARED
+/* This is used to cache the priorities of glibc-hwcaps
+ subdirectories. The elements of _dl_cache_priorities correspond to
+ the strings in the cache_extension_tag_glibc_hwcaps section. */
+static uint32_t *glibc_hwcaps_priorities;
+static uint32_t glibc_hwcaps_priorities_length;
+static uint32_t glibc_hwcaps_priorities_allocated;
+
+/* True if the full malloc was used to allocated the array. */
+static bool glibc_hwcaps_priorities_malloced;
+
+/* Deallocate the glibc_hwcaps_priorities array. */
+static void
+glibc_hwcaps_priorities_free (void)
+{
+ /* When the minimal malloc is in use, free does not do anything,
+ so it does not make sense to call it. */
+ if (glibc_hwcaps_priorities_malloced)
+ free (glibc_hwcaps_priorities);
+ glibc_hwcaps_priorities = NULL;
+ glibc_hwcaps_priorities_allocated = 0;
+}
+
+/* Return the priority of the cache_extension_tag_glibc_hwcaps section
+ entry at INDEX. Zero means do not use. Otherwise, lower values
+ indicate greater preference. */
+static uint32_t __attribute__ ((noinline, noclone))
+glibc_hwcaps_priority (uint32_t index)
+{
+ /* Using a zero-length array as an indicator that nothing has been
+ loaded is not a problem: It does not lead to repeated
+ initialization attempts because caches without an extension
+ section are processed without calling this function (unless the
+ file is corrupted). */
+ if (glibc_hwcaps_priorities_length == 0)
+ {
+ struct cache_extension_all_loaded ext;
+ if (!cache_extension_load (cache_new, cache, cachesize, &ext))
+ return 0;
+
+ uint32_t length
+ = (ext.sections[cache_extension_tag_glibc_hwcaps].size
+ / sizeof (uint32_t));
+ if (length > glibc_hwcaps_priorities_allocated)
+ {
+ glibc_hwcaps_priorities_free ();
+
+ glibc_hwcaps_priorities = malloc (length * sizeof (uint32_t));
+ if (glibc_hwcaps_priorities == NULL)
+ /* Disable hwcaps on memory allocation error. */
+ return 0;
+
+ glibc_hwcaps_priorities_allocated = length;
+ glibc_hwcaps_priorities_malloced = __rtld_malloc_is_complete ();
+ }
+
+ /* Compute the priorities for the subdirectories by merging the
+ array in the cache with the dl_hwcaps_priorities array. */
+ const uint32_t *left
+ = ext.sections[cache_extension_tag_glibc_hwcaps].base;
+ const uint32_t *left_end = left + length;
+ struct dl_hwcaps_priority *right = _dl_hwcaps_priorities;
+ struct dl_hwcaps_priority *right_end
+ = right + _dl_hwcaps_priorities_length;
+ uint32_t *result = glibc_hwcaps_priorities;
+
+ while (left < left_end && right < right_end)
+ {
+ uint32_t string_table_index = *left;
+ if (string_table_index < cachesize)
+ {
+ const char *left_name
+ = (const char *) cache + string_table_index;
+ uint32_t left_name_length = strlen (left_name);
+ uint32_t to_compare;
+ if (left_name_length < right->name_length)
+ to_compare = left_name_length;
+ else
+ to_compare = right->name_length;
+ int cmp = memcmp (left_name, right->name, to_compare);
+ if (cmp == 0)
+ {
+ if (left_name_length < right->name_length)
+ cmp = -1;
+ else if (left_name_length > right->name_length)
+ cmp = 1;
+ }
+ if (cmp == 0)
+ {
+ *result = right->priority;
+ ++result;
+ ++left;
+ ++right;
+ }
+ else if (cmp < 0)
+ {
+ *result = 0;
+ ++result;
+ ++left;
+ }
+ else
+ ++right;
+ }
+ else
+ {
+ *result = 0;
+ ++result;
+ }
+ }
+ while (left < left_end)
+ {
+ *result = 0;
+ ++result;
+ ++left;
+ }
+
+ glibc_hwcaps_priorities_length = length;
+ }
+
+ if (index < glibc_hwcaps_priorities_length)
+ return glibc_hwcaps_priorities[index];
+ else
+ return 0;
+}
+#endif /* SHARED */
+
/* True if PTR is a valid string table index. */
static inline bool
_dl_cache_verify_ptr (uint32_t ptr, size_t string_table_size)
@@ -74,6 +200,9 @@ search_cache (const char *string_table, uint32_t string_table_size,
int left = 0;
int right = nlibs - 1;
const char *best = NULL;
+#ifdef SHARED
+ uint32_t best_priority = 0;
+#endif
while (left <= right)
{
@@ -129,6 +258,11 @@ search_cache (const char *string_table, uint32_t string_table_size,
{
if (best == NULL || flags == GLRO (dl_correct_cache_id))
{
+ /* Named/extension hwcaps get slightly different
+ treatment: We keep searching for a better
+ match. */
+ bool named_hwcap = false;
+
if (entry_size >= sizeof (struct file_entry_new))
{
/* The entry is large enough to include
@@ -136,7 +270,18 @@ search_cache (const char *string_table, uint32_t string_table_size,
struct file_entry_new *libnew
= (struct file_entry_new *) lib;
- if (libnew->hwcap & hwcap_exclude)
+#ifdef SHARED
+ named_hwcap = dl_cache_hwcap_extension (libnew);
+#endif
+
+ /* The entries with named/extension hwcaps
+ have been exhausted. Return the best
+ match encountered so far if there is
+ one. */
+ if (!named_hwcap && best != NULL)
+ break;
+
+ if ((libnew->hwcap & hwcap_exclude) && !named_hwcap)
continue;
if (GLRO (dl_osversion)
&& libnew->osversion > GLRO (dl_osversion))
@@ -146,14 +291,41 @@ search_cache (const char *string_table, uint32_t string_table_size,
&& ((libnew->hwcap & _DL_HWCAP_PLATFORM)
!= platform))
continue;
+
+#ifdef SHARED
+ /* For named hwcaps, determine the priority
+ and see if beats what has been found so
+ far. */
+ if (named_hwcap)
+ {
+ uint32_t entry_priority
+ = glibc_hwcaps_priority (libnew->hwcap);
+ if (entry_priority == 0)
+ /* Not usable at all. Skip. */
+ continue;
+ else if (best == NULL
+ || entry_priority < best_priority)
+ /* This entry is of higher priority
+ than the previous one, or it is the
+ first entry. */
+ best_priority = entry_priority;
+ else
+ /* An entry has already been found,
+ but it is a better match. */
+ continue;
+ }
+#endif /* SHARED */
}
best = string_table + lib->value;
- if (flags == GLRO (dl_correct_cache_id))
+ if (flags == GLRO (dl_correct_cache_id)
+ && !named_hwcap)
/* We've found an exact match for the shared
object and no general `ELF' release. Stop
- searching. */
+ searching, but not if a named (extension)
+ hwcap is used. In this case, an entry with
+ a higher priority may come up later. */
break;
}
}
@@ -346,5 +518,9 @@ _dl_unload_cache (void)
__munmap (cache, cachesize);
cache = NULL;
}
+#ifdef SHARED
+ /* This marks the glibc_hwcaps_priorities array as out-of-date. */
+ glibc_hwcaps_priorities_length = 0;
+#endif
}
#endif
diff --git a/elf/dl-hwcaps.c b/elf/dl-hwcaps.c
index f611f3a1a6..51cc787b54 100644
--- a/elf/dl-hwcaps.c
+++ b/elf/dl-hwcaps.c
@@ -89,6 +89,81 @@ copy_hwcaps (struct copy_hwcaps *target, const char *hwcaps,
}
}
+struct dl_hwcaps_priority *_dl_hwcaps_priorities;
+uint32_t _dl_hwcaps_priorities_length;
+
+/* Allocate _dl_hwcaps_priorities and fill it with data. */
+static void
+compute_priorities (size_t total_count, const char *prepend,
+ int32_t bitmask, const char *mask)
+{
+ _dl_hwcaps_priorities = malloc (total_count
+ * sizeof (*_dl_hwcaps_priorities));
+ if (_dl_hwcaps_priorities == NULL)
+ _dl_signal_error (ENOMEM, NULL, NULL,
+ N_("cannot create HWCAP priorities"));
+ _dl_hwcaps_priorities_length = total_count;
+
+ /* First the prepended subdirectories. */
+ size_t i = 0;
+ {
+ struct dl_hwcaps_split sp;
+ _dl_hwcaps_split_init (&sp, prepend);
+ while (_dl_hwcaps_split (&sp))
+ {
+ _dl_hwcaps_priorities[i].name = sp.segment;
+ _dl_hwcaps_priorities[i].name_length = sp.length;
+ _dl_hwcaps_priorities[i].priority = i + 1;
+ ++i;
+ }
+ }
+
+ /* Then the built-in subdirectories that are actually active. */
+ {
+ struct dl_hwcaps_split_masked sp;
+ _dl_hwcaps_split_masked_init (&sp, _dl_hwcaps_subdirs, bitmask, mask);
+ while (_dl_hwcaps_split_masked (&sp))
+ {
+ _dl_hwcaps_priorities[i].name = sp.split.segment;
+ _dl_hwcaps_priorities[i].name_length = sp.split.length;
+ _dl_hwcaps_priorities[i].priority = i + 1;
+ ++i;
+ }
+ }
+ assert (i == total_count);
+}
+
+/* Sort the _dl_hwcaps_priorities array by name. */
+static void
+sort_priorities_by_name (void)
+{
+ /* Insertion sort. There is no need to link qsort into the dynamic
+ loader for such a short array. */
+ for (size_t i = 1; i < _dl_hwcaps_priorities_length; ++i)
+ for (size_t j = i; j > 0; --j)
+ {
+ struct dl_hwcaps_priority *previous = _dl_hwcaps_priorities + j - 1;
+ struct dl_hwcaps_priority *current = _dl_hwcaps_priorities + j;
+
+ /* Bail out if current is greater or equal to the previous
+ value. */
+ uint32_t to_compare;
+ if (current->name_length < previous->name_length)
+ to_compare = current->name_length;
+ else
+ to_compare = previous->name_length;
+ int cmp = memcmp (current->name, previous->name, to_compare);
+ if (cmp >= 0
+ || (cmp == 0 && current->name_length >= previous->name_length))
+ break;
+
+ /* Swap *previous and *current. */
+ struct dl_hwcaps_priority tmp = *previous;
+ *previous = *current;
+ *current = tmp;
+ }
+}
+
/* Return an array of useful/necessary hardware capability names. */
const struct r_strlenpair *
_dl_important_hwcaps (const char *glibc_hwcaps_prepend,
@@ -111,6 +186,9 @@ _dl_important_hwcaps (const char *glibc_hwcaps_prepend,
update_hwcaps_counts (&hwcaps_counts, glibc_hwcaps_prepend, -1, NULL);
update_hwcaps_counts (&hwcaps_counts, _dl_hwcaps_subdirs,
hwcaps_subdirs_active, glibc_hwcaps_mask);
+ compute_priorities (hwcaps_counts.count, glibc_hwcaps_prepend,
+ hwcaps_subdirs_active, glibc_hwcaps_mask);
+ sort_priorities_by_name ();
/* Each hwcaps subdirectory has a GLIBC_HWCAPS_PREFIX string prefix
and a "/" suffix once stored in the result. */
diff --git a/elf/dl-hwcaps.h b/elf/dl-hwcaps.h
index ab39d8a46d..ddfdde278e 100644
--- a/elf/dl-hwcaps.h
+++ b/elf/dl-hwcaps.h
@@ -132,4 +132,23 @@ _dl_hwcaps_subdirs_build_bitmask (int subdirs, int active)
return mask ^ ((1U << inactive) - 1);
}
+/* Pre-computed glibc-hwcaps subdirectory priorities. Used in
+ dl-cache.c to quickly find the proprities for the stored HWCAP
+ names. */
+struct dl_hwcaps_priority
+{
+ /* The name consists of name_length bytes at name (not necessarily
+ null-terminated). */
+ const char *name;
+ uint32_t name_length;
+
+ /* Priority of this name. A positive number. */
+ uint32_t priority;
+};
+
+/* Pre-computed hwcaps priorities. Set up by
+ _dl_important_hwcaps. */
+extern struct dl_hwcaps_priority *_dl_hwcaps_priorities attribute_hidden;
+extern uint32_t _dl_hwcaps_priorities_length attribute_hidden;
+
#endif /* _DL_HWCAPS_H */
diff --git a/elf/tst-glibc-hwcaps-cache.c b/elf/tst-glibc-hwcaps-cache.c
new file mode 100644
index 0000000000..4bad56afc0
--- /dev/null
+++ b/elf/tst-glibc-hwcaps-cache.c
@@ -0,0 +1,45 @@
+/* Wrapper to invoke tst-glibc-hwcaps in a container, to test ld.so.cache.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+/* This program is just a wrapper that runs ldconfig followed by
+ tst-glibc-hwcaps. The actual test is provided via an
+ implementation in a sysdeps subdirectory. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <support/support.h>
+#include <unistd.h>
+
+int
+main (int argc, char **argv)
+{
+ /* Run ldconfig to populate the cache. */
+ {
+ char *command = xasprintf ("%s/ldconfig", support_install_rootsbindir);
+ if (system (command) != 0)
+ return 1;
+ free (command);
+ }
+
+ /* Reuse tst-glibc-hwcaps. Since this code is running in a
+ container, we can launch it directly. */
+ char *path = xasprintf ("%s/elf/tst-glibc-hwcaps", support_objdir_root);
+ execv (path, argv);
+ printf ("error: execv of %s failed: %m\n", path);
+ return 1;
+}
diff --git a/elf/tst-glibc-hwcaps-cache.root/etc/ld.so.conf b/elf/tst-glibc-hwcaps-cache.root/etc/ld.so.conf
new file mode 100644
index 0000000000..e1e74dbda2
--- /dev/null
+++ b/elf/tst-glibc-hwcaps-cache.root/etc/ld.so.conf
@@ -0,0 +1,2 @@
+# This file was created to suppress a warning from ldconfig:
+# /sbin/ldconfig: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf: No such file or directory
diff --git a/elf/tst-glibc-hwcaps-cache.root/postclean.req b/elf/tst-glibc-hwcaps-cache.root/postclean.req
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
new file mode 100644
index 0000000000..46cb5fd553
--- /dev/null
+++ b/elf/tst-glibc-hwcaps-cache.script
@@ -0,0 +1,2 @@
+# test-container does not support scripts in sysdeps directories, so
+# collect everything in one file.
diff --git a/elf/tst-glibc-hwcaps-prepend-cache.c b/elf/tst-glibc-hwcaps-prepend-cache.c
new file mode 100644
index 0000000000..eedbf5f6df
--- /dev/null
+++ b/elf/tst-glibc-hwcaps-prepend-cache.c
@@ -0,0 +1,133 @@
+/* Test that --glibc-hwcaps-prepend works, using dlopen and /etc/ld.so.cache.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <dlfcn.h>
+#include <stddef.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <support/check.h>
+#include <support/support.h>
+#include <support/xdlfcn.h>
+#include <support/xunistd.h>
+
+/* Invoke /sbin/ldconfig with some error checking. */
+static void
+run_ldconfig (void)
+{
+ char *command = xasprintf ("%s/ldconfig", support_install_rootsbindir);
+ TEST_COMPARE (system (command), 0);
+ free (command);
+}
+
+/* The library under test. */
+#define SONAME "libmarkermod1.so"
+
+static int
+do_test (void)
+{
+ if (dlopen (SONAME, RTLD_NOW) != NULL)
+ FAIL_EXIT1 (SONAME " is already on the search path");
+
+ /* Install the default implementation of libmarkermod1.so. */
+ xmkdirp ("/etc", 0777);
+ support_write_file_string ("/etc/ld.so.conf", "/glibc-test/lib\n");
+ xmkdirp ("/glibc-test/lib/glibc-hwcaps/prepend2", 0777);
+ xmkdirp ("/glibc-test/lib/glibc-hwcaps/prepend3", 0777);
+ {
+ char *src = xasprintf ("%s/elf/libmarkermod1-1.so", support_objdir_root);
+ support_copy_file (src, "/glibc-test/lib/" SONAME);
+ free (src);
+ }
+ run_ldconfig ();
+ {
+ /* The default implementation can now be loaded. */
+ void *handle = xdlopen (SONAME, RTLD_NOW);
+ int (*marker1) (void) = xdlsym (handle, "marker1");
+ TEST_COMPARE (marker1 (), 1);
+ xdlclose (handle);
+ }
+
+ /* Add the first override to the directory that is searched last. */
+ {
+ char *src = xasprintf ("%s/elf/libmarkermod1-2.so", support_objdir_root);
+ support_copy_file (src, "/glibc-test/lib/glibc-hwcaps/prepend2/"
+ SONAME);
+ free (src);
+ }
+ {
+ /* This is still the first implementation. The cache has not been
+ updated. */
+ void *handle = xdlopen (SONAME, RTLD_NOW);
+ int (*marker1) (void) = xdlsym (handle, "marker1");
+ TEST_COMPARE (marker1 (), 1);
+ xdlclose (handle);
+ }
+ run_ldconfig ();
+ {
+ /* After running ldconfig, it is the second implementation. */
+ void *handle = xdlopen (SONAME, RTLD_NOW);
+ int (*marker1) (void) = xdlsym (handle, "marker1");
+ TEST_COMPARE (marker1 (), 2);
+ xdlclose (handle);
+ }
+
+ /* Add the second override to the directory that is searched first. */
+ {
+ char *src = xasprintf ("%s/elf/libmarkermod1-3.so", support_objdir_root);
+ support_copy_file (src, "/glibc-test/lib/glibc-hwcaps/prepend3/"
+ SONAME);
+ free (src);
+ }
+ {
+ /* This is still the second implementation. */
+ void *handle = xdlopen (SONAME, RTLD_NOW);
+ int (*marker1) (void) = xdlsym (handle, "marker1");
+ TEST_COMPARE (marker1 (), 2);
+ xdlclose (handle);
+ }
+ run_ldconfig ();
+ {
+ /* After running ldconfig, it is the third implementation. */
+ void *handle = xdlopen (SONAME, RTLD_NOW);
+ int (*marker1) (void) = xdlsym (handle, "marker1");
+ TEST_COMPARE (marker1 (), 3);
+ xdlclose (handle);
+ }
+
+ return 0;
+}
+
+static void
+prepare (int argc, char **argv)
+{
+ const char *no_restart = "no-restart";
+ if (argc == 2 && strcmp (argv[1], no_restart) == 0)
+ return;
+ /* Re-execute the test with an explicit loader invocation. */
+ execl (support_objdir_elf_ldso,
+ support_objdir_elf_ldso,
+ "--glibc-hwcaps-prepend", "prepend3:prepend2",
+ argv[0], no_restart,
+ NULL);
+ printf ("error: execv of %s failed: %m\n", argv[0]);
+ _exit (1);
+}
+
+#define PREPARE prepare
+#include <support/test-driver.c>
diff --git a/elf/tst-glibc-hwcaps-prepend-cache.root/postclean.req b/elf/tst-glibc-hwcaps-prepend-cache.root/postclean.req
new file mode 100644
index 0000000000..e69de29bb2
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
2020-11-09 18:41 ` [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing Florian Weimer
@ 2020-11-26 17:11 ` Florian Weimer
2020-12-01 17:45 ` Adhemerval Zanella
1 sibling, 0 replies; 58+ messages in thread
From: Florian Weimer @ 2020-11-26 17:11 UTC (permalink / raw)
To: Florian Weimer via Libc-alpha
* Florian Weimer via Libc-alpha:
> diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
> new file mode 100644
> index 0000000000..46cb5fd553
> --- /dev/null
> +++ b/elf/tst-glibc-hwcaps-cache.script
> @@ -0,0 +1,2 @@
> +# test-container does not support scripts in sysdeps directories, so
> +# collect everything in one file.
This file is missing installation of the default libraries, like this:
cp $B/elf/libmarkermod2-1.so $L/libmarkermod2.so
cp $B/elf/libmarkermod3-1.so $L/libmarkermod3.so
cp $B/elf/libmarkermod4-1.so $L/libmarkermod4.so
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
2020-11-09 18:41 ` [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing Florian Weimer
2020-11-26 17:11 ` Florian Weimer
@ 2020-12-01 17:45 ` Adhemerval Zanella
2020-12-01 20:45 ` Florian Weimer
1 sibling, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-12-01 17:45 UTC (permalink / raw)
To: Florian Weimer, libc-alpha
On 09/11/2020 15:41, Florian Weimer via Libc-alpha wrote:
> This recognizes the DL_CACHE_HWCAP_EXTENSION flag and picks up
> the supported cache entry with the highest priority.
Maybe add a brief description of what DL_CACHE_HWCAP_EXTENSION aims to do?
Patch looks good in general, some comments below.
> ---
> elf/Makefile | 18 ++
> elf/dl-cache.c | 182 +++++++++++++++++-
> elf/dl-hwcaps.c | 78 ++++++++
> elf/dl-hwcaps.h | 19 ++
> elf/tst-glibc-hwcaps-cache.c | 45 +++++
> .../etc/ld.so.conf | 2 +
> elf/tst-glibc-hwcaps-cache.root/postclean.req | 0
> elf/tst-glibc-hwcaps-cache.script | 2 +
> elf/tst-glibc-hwcaps-prepend-cache.c | 133 +++++++++++++
> .../postclean.req | 0
> 10 files changed, 476 insertions(+), 3 deletions(-)
> create mode 100644 elf/tst-glibc-hwcaps-cache.c
> create mode 100644 elf/tst-glibc-hwcaps-cache.root/etc/ld.so.conf
> create mode 100644 elf/tst-glibc-hwcaps-cache.root/postclean.req
> create mode 100644 elf/tst-glibc-hwcaps-cache.script
> create mode 100644 elf/tst-glibc-hwcaps-prepend-cache.c
> create mode 100644 elf/tst-glibc-hwcaps-prepend-cache.root/postclean.req
>
> diff --git a/elf/Makefile b/elf/Makefile
> index e26ac16b44..80e94b9ee2 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -171,6 +171,12 @@ tests-container := \
> tst-ldconfig-bad-aux-cache \
> tst-ldconfig-ld_so_conf-update
>
> +ifeq (no,$(build-hardcoded-path-in-tests))
> +# This is an ld.so.cache test, and RPATH/RUNPATH in the executable
> +# interferes with its test objectives.
> +tests-container += tst-glibc-hwcaps-prepend-cache
> +endif
> +
> tests := tst-tls9 tst-leaks1 \
> tst-array1 tst-array2 tst-array3 tst-array4 tst-array5 \
> tst-auxv tst-stringtable
Ok.
> @@ -1859,6 +1865,14 @@ $(objpfx)tst-glibc-hwcaps-prepend.out: \
> $< > $@; \
> $(evaluate-test)
>
> +# Like tst-glibc-hwcaps-prepend, but uses a container and loads the
> +# library via ld.so.cache. Test setup is contained in the test
> +# itself.
> +$(objpfx)tst-glibc-hwcaps-prepend-cache: $(libdl)
> +$(objpfx)tst-glibc-hwcaps-prepend-cache.out: \
> + $(objpfx)tst-glibc-hwcaps-prepend-cache $(objpfx)libmarkermod1-1.so \
> + $(objpfx)libmarkermod1-2.so $(objpfx)libmarkermod1-3.so
> +
> # tst-glibc-hwcaps-mask checks that --glibc-hwcaps-mask can be used to
> # suppress all auto-detected subdirectories.
> $(objpfx)tst-glibc-hwcaps-mask: $(objpfx)libmarkermod1-1.so
> @@ -1870,3 +1884,7 @@ $(objpfx)tst-glibc-hwcaps-mask.out: \
> --glibc-hwcaps-mask does-not-exist \
> $< > $@; \
> $(evaluate-test)
> +
> +# Generic dependency for sysdeps implementation of
> +# tst-glibc-hwcaps-cache.
> +$(objpfx)tst-glibc-hwcaps-cache.out: $(objpfx)tst-glibc-hwcaps
Ok.
> diff --git a/elf/dl-cache.c b/elf/dl-cache.c
> index 02c46ffb0c..13efc6f95f 100644
> --- a/elf/dl-cache.c
> +++ b/elf/dl-cache.c
> @@ -35,6 +35,132 @@ static struct cache_file *cache;
> static struct cache_file_new *cache_new;
> static size_t cachesize;
>
> +#ifdef SHARED
> +/* This is used to cache the priorities of glibc-hwcaps
> + subdirectories. The elements of _dl_cache_priorities correspond to
> + the strings in the cache_extension_tag_glibc_hwcaps section. */
> +static uint32_t *glibc_hwcaps_priorities;
> +static uint32_t glibc_hwcaps_priorities_length;
> +static uint32_t glibc_hwcaps_priorities_allocated;
> +
> +/* True if the full malloc was used to allocated the array. */
> +static bool glibc_hwcaps_priorities_malloced;
> +
> +/* Deallocate the glibc_hwcaps_priorities array. */
> +static void
> +glibc_hwcaps_priorities_free (void)
> +{
> + /* When the minimal malloc is in use, free does not do anything,
> + so it does not make sense to call it. */
> + if (glibc_hwcaps_priorities_malloced)
> + free (glibc_hwcaps_priorities);
> + glibc_hwcaps_priorities = NULL;
> + glibc_hwcaps_priorities_allocated = 0;
> +}
> +
> +/* Return the priority of the cache_extension_tag_glibc_hwcaps section
> + entry at INDEX. Zero means do not use. Otherwise, lower values
> + indicate greater preference. */
> +static uint32_t __attribute__ ((noinline, noclone))
I have a feeling I have asked it before, but why does it need noclone/noinline
here?
> +glibc_hwcaps_priority (uint32_t index)
> +{
> + /* Using a zero-length array as an indicator that nothing has been
> + loaded is not a problem: It does not lead to repeated
> + initialization attempts because caches without an extension
> + section are processed without calling this function (unless the
> + file is corrupted). */
> + if (glibc_hwcaps_priorities_length == 0)
Maybe an early exit here? It allow move the code one identation left
and makes it more readable.
> + {
> + struct cache_extension_all_loaded ext;
> + if (!cache_extension_load (cache_new, cache, cachesize, &ext))
> + return 0;
> +
> + uint32_t length
> + = (ext.sections[cache_extension_tag_glibc_hwcaps].size
> + / sizeof (uint32_t));
> + if (length > glibc_hwcaps_priorities_allocated)
> + {
> + glibc_hwcaps_priorities_free ();
> +
> + glibc_hwcaps_priorities = malloc (length * sizeof (uint32_t));
> + if (glibc_hwcaps_priorities == NULL)
> + /* Disable hwcaps on memory allocation error. */
> + return 0;
> +
> + glibc_hwcaps_priorities_allocated = length;
> + glibc_hwcaps_priorities_malloced = __rtld_malloc_is_complete ();
> + }
> +
> + /* Compute the priorities for the subdirectories by merging the
> + array in the cache with the dl_hwcaps_priorities array. */
> + const uint32_t *left
> + = ext.sections[cache_extension_tag_glibc_hwcaps].base;
> + const uint32_t *left_end = left + length;
> + struct dl_hwcaps_priority *right = _dl_hwcaps_priorities;
> + struct dl_hwcaps_priority *right_end
> + = right + _dl_hwcaps_priorities_length;
> + uint32_t *result = glibc_hwcaps_priorities;
> +
> + while (left < left_end && right < right_end)
> + {
> + uint32_t string_table_index = *left;
> + if (string_table_index < cachesize)
> + {
> + const char *left_name
> + = (const char *) cache + string_table_index;
> + uint32_t left_name_length = strlen (left_name);
> + uint32_t to_compare;
> + if (left_name_length < right->name_length)
> + to_compare = left_name_length;
> + else
> + to_compare = right->name_length;
> + int cmp = memcmp (left_name, right->name, to_compare);
> + if (cmp == 0)
> + {
> + if (left_name_length < right->name_length)
> + cmp = -1;
> + else if (left_name_length > right->name_length)
> + cmp = 1;
> + }
> + if (cmp == 0)
> + {
> + *result = right->priority;
> + ++result;
> + ++left;
> + ++right;
> + }
Maybe add as the 'else' within the 'cmp == 0' below?
> + else if (cmp < 0)
> + {
> + *result = 0;
> + ++result;
> + ++left;
> + }
> + else
> + ++right;
> + }
> + else
> + {
> + *result = 0;
> + ++result;
> + }
> + }
> + while (left < left_end)
> + {
> + *result = 0;
> + ++result;
> + ++left;
> + }
> +
> + glibc_hwcaps_priorities_length = length;
> + }
> +
> + if (index < glibc_hwcaps_priorities_length)
> + return glibc_hwcaps_priorities[index];
> + else
> + return 0;
> +}
> +#endif /* SHARED */
> +
Ok.
> /* True if PTR is a valid string table index. */
> static inline bool
> _dl_cache_verify_ptr (uint32_t ptr, size_t string_table_size)
> @@ -74,6 +200,9 @@ search_cache (const char *string_table, uint32_t string_table_size,
> int left = 0;
> int right = nlibs - 1;
> const char *best = NULL;
> +#ifdef SHARED
> + uint32_t best_priority = 0;
> +#endif
>
> while (left <= right)
> {
> @@ -129,6 +258,11 @@ search_cache (const char *string_table, uint32_t string_table_size,
> {
> if (best == NULL || flags == GLRO (dl_correct_cache_id))
> {
> + /* Named/extension hwcaps get slightly different
> + treatment: We keep searching for a better
> + match. */
> + bool named_hwcap = false;
> +
> if (entry_size >= sizeof (struct file_entry_new))
> {
> /* The entry is large enough to include
Ok.
> @@ -136,7 +270,18 @@ search_cache (const char *string_table, uint32_t string_table_size,
> struct file_entry_new *libnew
> = (struct file_entry_new *) lib;
>
> - if (libnew->hwcap & hwcap_exclude)
> +#ifdef SHARED
> + named_hwcap = dl_cache_hwcap_extension (libnew);
> +#endif
> +
> + /* The entries with named/extension hwcaps
> + have been exhausted. Return the best
> + match encountered so far if there is
> + one. */
> + if (!named_hwcap && best != NULL)
> + break;
> +
> + if ((libnew->hwcap & hwcap_exclude) && !named_hwcap)
> continue;
> if (GLRO (dl_osversion)
> && libnew->osversion > GLRO (dl_osversion))
Ok.
> @@ -146,14 +291,41 @@ search_cache (const char *string_table, uint32_t string_table_size,
> && ((libnew->hwcap & _DL_HWCAP_PLATFORM)
> != platform))
> continue;
> +
> +#ifdef SHARED
> + /* For named hwcaps, determine the priority
> + and see if beats what has been found so
> + far. */
> + if (named_hwcap)
> + {
> + uint32_t entry_priority
> + = glibc_hwcaps_priority (libnew->hwcap);
> + if (entry_priority == 0)
> + /* Not usable at all. Skip. */
> + continue;
> + else if (best == NULL
> + || entry_priority < best_priority)
> + /* This entry is of higher priority
> + than the previous one, or it is the
> + first entry. */
> + best_priority = entry_priority;
> + else
> + /* An entry has already been found,
> + but it is a better match. */
> + continue;
> + }
> +#endif /* SHARED */
> }
Ok.
>
> best = string_table + lib->value;
>
> - if (flags == GLRO (dl_correct_cache_id))
> + if (flags == GLRO (dl_correct_cache_id)
> + && !named_hwcap)
> /* We've found an exact match for the shared
> object and no general `ELF' release. Stop
> - searching. */
> + searching, but not if a named (extension)
> + hwcap is used. In this case, an entry with
> + a higher priority may come up later. */
> break;
> }
> }
Ok.
> @@ -346,5 +518,9 @@ _dl_unload_cache (void)
> __munmap (cache, cachesize);
> cache = NULL;
> }
> +#ifdef SHARED
> + /* This marks the glibc_hwcaps_priorities array as out-of-date. */
> + glibc_hwcaps_priorities_length = 0;
> +#endif
> }
> #endif
Ok.
> diff --git a/elf/dl-hwcaps.c b/elf/dl-hwcaps.c
> index f611f3a1a6..51cc787b54 100644
> --- a/elf/dl-hwcaps.c
> +++ b/elf/dl-hwcaps.c
> @@ -89,6 +89,81 @@ copy_hwcaps (struct copy_hwcaps *target, const char *hwcaps,
> }
> }
>
> +struct dl_hwcaps_priority *_dl_hwcaps_priorities;
> +uint32_t _dl_hwcaps_priorities_length;
> +
> +/* Allocate _dl_hwcaps_priorities and fill it with data. */
> +static void
> +compute_priorities (size_t total_count, const char *prepend,
> + int32_t bitmask, const char *mask)
I think bitmask should be a 'uint32' here.
> +{
> + _dl_hwcaps_priorities = malloc (total_count
> + * sizeof (*_dl_hwcaps_priorities));
> + if (_dl_hwcaps_priorities == NULL)
> + _dl_signal_error (ENOMEM, NULL, NULL,
> + N_("cannot create HWCAP priorities"));
> + _dl_hwcaps_priorities_length = total_count;
> +
> + /* First the prepended subdirectories. */
> + size_t i = 0;
> + {
> + struct dl_hwcaps_split sp;
> + _dl_hwcaps_split_init (&sp, prepend);
> + while (_dl_hwcaps_split (&sp))
> + {
> + _dl_hwcaps_priorities[i].name = sp.segment;
> + _dl_hwcaps_priorities[i].name_length = sp.length;
> + _dl_hwcaps_priorities[i].priority = i + 1;
> + ++i;
> + }
> + }
> +
Ok.
> + /* Then the built-in subdirectories that are actually active. */
> + {
> + struct dl_hwcaps_split_masked sp;
> + _dl_hwcaps_split_masked_init (&sp, _dl_hwcaps_subdirs, bitmask, mask);
> + while (_dl_hwcaps_split_masked (&sp))
> + {
> + _dl_hwcaps_priorities[i].name = sp.split.segment;
> + _dl_hwcaps_priorities[i].name_length = sp.split.length;
> + _dl_hwcaps_priorities[i].priority = i + 1;
> + ++i;
> + }
> + }
> + assert (i == total_count);
> +}
> +
Ok.
> +/* Sort the _dl_hwcaps_priorities array by name. */
> +static void
> +sort_priorities_by_name (void)
> +{
> + /* Insertion sort. There is no need to link qsort into the dynamic
> + loader for such a short array. */
> + for (size_t i = 1; i < _dl_hwcaps_priorities_length; ++i)
> + for (size_t j = i; j > 0; --j)
> + {
> + struct dl_hwcaps_priority *previous = _dl_hwcaps_priorities + j - 1;
> + struct dl_hwcaps_priority *current = _dl_hwcaps_priorities + j;
> +
> + /* Bail out if current is greater or equal to the previous
> + value. */
> + uint32_t to_compare;
> + if (current->name_length < previous->name_length)
> + to_compare = current->name_length;
> + else
> + to_compare = previous->name_length;
> + int cmp = memcmp (current->name, previous->name, to_compare);
> + if (cmp >= 0
> + || (cmp == 0 && current->name_length >= previous->name_length))
> + break;
> +
> + /* Swap *previous and *current. */
> + struct dl_hwcaps_priority tmp = *previous;
> + *previous = *current;
> + *current = tmp;
> + }
> +}
> +
Ok.
> /* Return an array of useful/necessary hardware capability names. */
> const struct r_strlenpair *
> _dl_important_hwcaps (const char *glibc_hwcaps_prepend,
> @@ -111,6 +186,9 @@ _dl_important_hwcaps (const char *glibc_hwcaps_prepend,
> update_hwcaps_counts (&hwcaps_counts, glibc_hwcaps_prepend, -1, NULL);
> update_hwcaps_counts (&hwcaps_counts, _dl_hwcaps_subdirs,
> hwcaps_subdirs_active, glibc_hwcaps_mask);
> + compute_priorities (hwcaps_counts.count, glibc_hwcaps_prepend,
> + hwcaps_subdirs_active, glibc_hwcaps_mask);
> + sort_priorities_by_name ();
>
> /* Each hwcaps subdirectory has a GLIBC_HWCAPS_PREFIX string prefix
> and a "/" suffix once stored in the result. */
Ok.
> diff --git a/elf/dl-hwcaps.h b/elf/dl-hwcaps.h
> index ab39d8a46d..ddfdde278e 100644
> --- a/elf/dl-hwcaps.h
> +++ b/elf/dl-hwcaps.h
> @@ -132,4 +132,23 @@ _dl_hwcaps_subdirs_build_bitmask (int subdirs, int active)
> return mask ^ ((1U << inactive) - 1);
> }
>
> +/* Pre-computed glibc-hwcaps subdirectory priorities. Used in
> + dl-cache.c to quickly find the proprities for the stored HWCAP
> + names. */
> +struct dl_hwcaps_priority
> +{
> + /* The name consists of name_length bytes at name (not necessarily
> + null-terminated). */
> + const char *name;
> + uint32_t name_length;
> +
> + /* Priority of this name. A positive number. */
> + uint32_t priority;
> +};
> +
> +/* Pre-computed hwcaps priorities. Set up by
> + _dl_important_hwcaps. */
> +extern struct dl_hwcaps_priority *_dl_hwcaps_priorities attribute_hidden;
> +extern uint32_t _dl_hwcaps_priorities_length attribute_hidden;
> +
> #endif /* _DL_HWCAPS_H */
Ok.
> diff --git a/elf/tst-glibc-hwcaps-cache.c b/elf/tst-glibc-hwcaps-cache.c
> new file mode 100644
> index 0000000000..4bad56afc0
> --- /dev/null
> +++ b/elf/tst-glibc-hwcaps-cache.c
> @@ -0,0 +1,45 @@
> +/* Wrapper to invoke tst-glibc-hwcaps in a container, to test ld.so.cache.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +/* This program is just a wrapper that runs ldconfig followed by
> + tst-glibc-hwcaps. The actual test is provided via an
> + implementation in a sysdeps subdirectory. */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <support/support.h>
> +#include <unistd.h>
> +
> +int
> +main (int argc, char **argv)
> +{
> + /* Run ldconfig to populate the cache. */
> + {
> + char *command = xasprintf ("%s/ldconfig", support_install_rootsbindir);
> + if (system (command) != 0)
> + return 1;
> + free (command);
> + }
> +
> + /* Reuse tst-glibc-hwcaps. Since this code is running in a
> + container, we can launch it directly. */
> + char *path = xasprintf ("%s/elf/tst-glibc-hwcaps", support_objdir_root);
> + execv (path, argv);
> + printf ("error: execv of %s failed: %m\n", path);
> + return 1;
> +}
Ok, it should be simple enough to require use libsupport fork/timeout check.
> diff --git a/elf/tst-glibc-hwcaps-cache.root/etc/ld.so.conf b/elf/tst-glibc-hwcaps-cache.root/etc/ld.so.conf
> new file mode 100644
> index 0000000000..e1e74dbda2
> --- /dev/null
> +++ b/elf/tst-glibc-hwcaps-cache.root/etc/ld.so.conf
> @@ -0,0 +1,2 @@
> +# This file was created to suppress a warning from ldconfig:
> +# /sbin/ldconfig: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf: No such file or directory
> diff --git a/elf/tst-glibc-hwcaps-cache.root/postclean.req b/elf/tst-glibc-hwcaps-cache.root/postclean.req
> new file mode 100644
> index 0000000000..e69de29bb2
Ok.
> diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
> new file mode 100644
> index 0000000000..46cb5fd553
> --- /dev/null
> +++ b/elf/tst-glibc-hwcaps-cache.script
> @@ -0,0 +1,2 @@
> +# test-container does not support scripts in sysdeps directories, so
> +# collect everything in one file.
Ok.
> diff --git a/elf/tst-glibc-hwcaps-prepend-cache.c b/elf/tst-glibc-hwcaps-prepend-cache.c
> new file mode 100644
> index 0000000000..eedbf5f6df
> --- /dev/null
> +++ b/elf/tst-glibc-hwcaps-prepend-cache.c
> @@ -0,0 +1,133 @@
> +/* Test that --glibc-hwcaps-prepend works, using dlopen and /etc/ld.so.cache.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <dlfcn.h>
> +#include <stddef.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <support/check.h>
> +#include <support/support.h>
> +#include <support/xdlfcn.h>
> +#include <support/xunistd.h>
> +
> +/* Invoke /sbin/ldconfig with some error checking. */
> +static void
> +run_ldconfig (void)
> +{
> + char *command = xasprintf ("%s/ldconfig", support_install_rootsbindir);
> + TEST_COMPARE (system (command), 0);
> + free (command);
> +}
> +
> +/* The library under test. */
> +#define SONAME "libmarkermod1.so"
> +
> +static int
> +do_test (void)
> +{
> + if (dlopen (SONAME, RTLD_NOW) != NULL)
> + FAIL_EXIT1 (SONAME " is already on the search path");
> +
> + /* Install the default implementation of libmarkermod1.so. */
> + xmkdirp ("/etc", 0777);
> + support_write_file_string ("/etc/ld.so.conf", "/glibc-test/lib\n");
> + xmkdirp ("/glibc-test/lib/glibc-hwcaps/prepend2", 0777);
> + xmkdirp ("/glibc-test/lib/glibc-hwcaps/prepend3", 0777);
> + {
> + char *src = xasprintf ("%s/elf/libmarkermod1-1.so", support_objdir_root);
> + support_copy_file (src, "/glibc-test/lib/" SONAME);
> + free (src);
> + }
Ok.
> + run_ldconfig ();
> + {
> + /* The default implementation can now be loaded. */
> + void *handle = xdlopen (SONAME, RTLD_NOW);
> + int (*marker1) (void) = xdlsym (handle, "marker1");
> + TEST_COMPARE (marker1 (), 1);
> + xdlclose (handle);
> + }
Ok.
> +
> + /* Add the first override to the directory that is searched last. */
> + {
> + char *src = xasprintf ("%s/elf/libmarkermod1-2.so", support_objdir_root);
> + support_copy_file (src, "/glibc-test/lib/glibc-hwcaps/prepend2/"
> + SONAME);
> + free (src);
> + }
> + {
> + /* This is still the first implementation. The cache has not been
> + updated. */
> + void *handle = xdlopen (SONAME, RTLD_NOW);
> + int (*marker1) (void) = xdlsym (handle, "marker1");
> + TEST_COMPARE (marker1 (), 1);
> + xdlclose (handle);
> + }
Ok.
> + run_ldconfig ();
> + {
> + /* After running ldconfig, it is the second implementation. */
> + void *handle = xdlopen (SONAME, RTLD_NOW);
> + int (*marker1) (void) = xdlsym (handle, "marker1");
> + TEST_COMPARE (marker1 (), 2);
> + xdlclose (handle);
> + }
> +
Ok.
> + /* Add the second override to the directory that is searched first. */
> + {
> + char *src = xasprintf ("%s/elf/libmarkermod1-3.so", support_objdir_root);
> + support_copy_file (src, "/glibc-test/lib/glibc-hwcaps/prepend3/"
> + SONAME);
> + free (src);
> + }
> + {
> + /* This is still the second implementation. */
> + void *handle = xdlopen (SONAME, RTLD_NOW);
> + int (*marker1) (void) = xdlsym (handle, "marker1");
> + TEST_COMPARE (marker1 (), 2);
> + xdlclose (handle);
> + }
Ok.
> + run_ldconfig ();
> + {
> + /* After running ldconfig, it is the third implementation. */
> + void *handle = xdlopen (SONAME, RTLD_NOW);
> + int (*marker1) (void) = xdlsym (handle, "marker1");
> + TEST_COMPARE (marker1 (), 3);
> + xdlclose (handle);
> + }
> +
Ok.
Should we test the case of shared memory removal, for instance:
1. Remove second override, without ldconfig
2. Remove third override, run ldconfig
3. Add both second and third back, run ldconfig, remove third
4. Keep second, run ldconfig
> + return 0;
> +}
> +
> +static void
> +prepare (int argc, char **argv)
> +{
> + const char *no_restart = "no-restart";
> + if (argc == 2 && strcmp (argv[1], no_restart) == 0)
> + return;
> + /* Re-execute the test with an explicit loader invocation. */
> + execl (support_objdir_elf_ldso,
> + support_objdir_elf_ldso,
> + "--glibc-hwcaps-prepend", "prepend3:prepend2",
> + argv[0], no_restart,
> + NULL);
> + printf ("error: execv of %s failed: %m\n", argv[0]);
> + _exit (1);
> +}
> +
> +#define PREPARE prepare
> +#include <support/test-driver.c>
> diff --git a/elf/tst-glibc-hwcaps-prepend-cache.root/postclean.req b/elf/tst-glibc-hwcaps-prepend-cache.root/postclean.req
> new file mode 100644
> index 0000000000..e69de29bb2
>
Ok.
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
2020-12-01 17:45 ` Adhemerval Zanella
@ 2020-12-01 20:45 ` Florian Weimer
2020-12-02 12:08 ` Adhemerval Zanella
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-12-01 20:45 UTC (permalink / raw)
To: Adhemerval Zanella via Libc-alpha
* Adhemerval Zanella via Libc-alpha:
> On 09/11/2020 15:41, Florian Weimer via Libc-alpha wrote:
>> This recognizes the DL_CACHE_HWCAP_EXTENSION flag and picks up
>> the supported cache entry with the highest priority.
>
> Maybe add a brief description of what DL_CACHE_HWCAP_EXTENSION aims to
> do?
It's documented in sysdeps/generic/dl-cache.h, added in the previous
commit:
/* This bit in the hwcap field of struct file_entry_new indicates that
the lower 32 bits contain an index into the
cache_extension_tag_glibc_hwcaps section. Older glibc versions do
not know about this HWCAP bit, so they will ignore these
entries. */
#define DL_CACHE_HWCAP_EXTENSION (1ULL << 62)
>> +/* Return the priority of the cache_extension_tag_glibc_hwcaps section
>> + entry at INDEX. Zero means do not use. Otherwise, lower values
>> + indicate greater preference. */
>> +static uint32_t __attribute__ ((noinline, noclone))
>> +glibc_hwcaps_priority (uint32_t index)
>
> I have a feeling I have asked it before, but why does it need noclone/noinline
> here?
I don't recall such a discussion. It's a leftover from previous
debugging efforts.
>> +{
>> + /* Using a zero-length array as an indicator that nothing has been
>> + loaded is not a problem: It does not lead to repeated
>> + initialization attempts because caches without an extension
>> + section are processed without calling this function (unless the
>> + file is corrupted). */
>> + if (glibc_hwcaps_priorities_length == 0)
>
> Maybe an early exit here? It allow move the code one identation left
> and makes it more readable.
I've moved this into a separate initialization function.
>> + while (left < left_end && right < right_end)
>> + {
>> + uint32_t string_table_index = *left;
>> + if (string_table_index < cachesize)
>> + {
>> + const char *left_name
>> + = (const char *) cache + string_table_index;
>> + uint32_t left_name_length = strlen (left_name);
>> + uint32_t to_compare;
>> + if (left_name_length < right->name_length)
>> + to_compare = left_name_length;
>> + else
>> + to_compare = right->name_length;
>> + int cmp = memcmp (left_name, right->name, to_compare);
>> + if (cmp == 0)
>> + {
>> + if (left_name_length < right->name_length)
>> + cmp = -1;
>> + else if (left_name_length > right->name_length)
>> + cmp = 1;
>> + }
>> + if (cmp == 0)
>> + {
>> + *result = right->priority;
>> + ++result;
>> + ++left;
>> + ++right;
>> + }
>
> Maybe add as the 'else' within the 'cmp == 0' below?
>
>> + else if (cmp < 0)
>> + {
>> + *result = 0;
>> + ++result;
>> + ++left;
>> + }
>> + else
>> + ++right;
You mean like this?
if (cmp == 0)
{
*result = right->priority;
++result;
++left;
}
if (cmp < 0)
{
*result = 0;
++result;
++left;
}
if (cmp >= 0)
++right;
I don't think that's an improvement.
>> +struct dl_hwcaps_priority *_dl_hwcaps_priorities;
>> +uint32_t _dl_hwcaps_priorities_length;
>> +
>> +/* Allocate _dl_hwcaps_priorities and fill it with data. */
>> +static void
>> +compute_priorities (size_t total_count, const char *prepend,
>> + int32_t bitmask, const char *mask)
>
> I think bitmask should be a 'uint32' here.
Right, fixed.
>> +int
>> +main (int argc, char **argv)
>> +{
>> + /* Run ldconfig to populate the cache. */
>> + {
>> + char *command = xasprintf ("%s/ldconfig", support_install_rootsbindir);
>> + if (system (command) != 0)
>> + return 1;
>> + free (command);
>> + }
>> +
>> + /* Reuse tst-glibc-hwcaps. Since this code is running in a
>> + container, we can launch it directly. */
>> + char *path = xasprintf ("%s/elf/tst-glibc-hwcaps", support_objdir_root);
>> + execv (path, argv);
>> + printf ("error: execv of %s failed: %m\n", path);
>> + return 1;
>> +}
>
> Ok, it should be simple enough to require use libsupport fork/timeout
> check.
Yes, we run ldconfig without timeout elsewhere, and tst-glibc-hwcaps has
its own timeout check.
> Should we test the case of shared memory removal, for instance:
>
> 1. Remove second override, without ldconfig
> 2. Remove third override, run ldconfig
> 3. Add both second and third back, run ldconfig, remove third
> 4. Keep second, run ldconfig
I added this:
+ /* Remove the second override again, without running ldconfig.
+ Ideally, this would revert to implementation 2. However, in the
+ current implementation, the cache returns exactly one file name
+ which does not exist after unlinking, so the dlopen fails. */
+ xunlink ("/glibc-test/lib/glibc-hwcaps/prepend3/" SONAME);
+ TEST_VERIFY (dlopen (SONAME, RTLD_NOW) == NULL);
+ run_ldconfig ();
+ system("/sbin/ldconfig -p");
+ {
+ /* After running ldconfig, the second implementation is available
+ once more. */
+ void *handle = xdlopen (SONAME, RTLD_NOW);
+ int (*marker1) (void) = xdlsym (handle, "marker1");
+ TEST_COMPARE (marker1 (), 2);
+ xdlclose (handle);
+ }
I think I know what I have to do to fix this in elf/dl-load.c.
I feel this is a pre-existing issue. It also applies to the legacy
hwcaps subdirectories. This only happens for ld.so.cache bringing in
libraries from a non-searched directory, otherwise the LD_LIBRARY_PATH
processing will find an implementation if it exists. I'd like to fix it
in a follow-up patch.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
2020-12-01 20:45 ` Florian Weimer
@ 2020-12-02 12:08 ` Adhemerval Zanella
2020-12-02 12:16 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-12-02 12:08 UTC (permalink / raw)
To: Florian Weimer, Adhemerval Zanella via Libc-alpha
On 01/12/2020 17:45, Florian Weimer wrote:
> * Adhemerval Zanella via Libc-alpha:
>
>> On 09/11/2020 15:41, Florian Weimer via Libc-alpha wrote:
>>> This recognizes the DL_CACHE_HWCAP_EXTENSION flag and picks up
>>> the supported cache entry with the highest priority.
>>
>> Maybe add a brief description of what DL_CACHE_HWCAP_EXTENSION aims to
>> do?
>
> It's documented in sysdeps/generic/dl-cache.h, added in the previous
> commit:
>
> /* This bit in the hwcap field of struct file_entry_new indicates that
> the lower 32 bits contain an index into the
> cache_extension_tag_glibc_hwcaps section. Older glibc versions do
> not know about this HWCAP bit, so they will ignore these
> entries. */
> #define DL_CACHE_HWCAP_EXTENSION (1ULL << 62)
I meant to commit message, but I don't have a strong opinion about it.
>
>>> +/* Return the priority of the cache_extension_tag_glibc_hwcaps section
>>> + entry at INDEX. Zero means do not use. Otherwise, lower values
>>> + indicate greater preference. */
>>> +static uint32_t __attribute__ ((noinline, noclone))
>>> +glibc_hwcaps_priority (uint32_t index)
>>
>> I have a feeling I have asked it before, but why does it need noclone/noinline
>> here?
>
> I don't recall such a discussion. It's a leftover from previous
> debugging efforts.
Ack.
>
>>> +{
>>> + /* Using a zero-length array as an indicator that nothing has been
>>> + loaded is not a problem: It does not lead to repeated
>>> + initialization attempts because caches without an extension
>>> + section are processed without calling this function (unless the
>>> + file is corrupted). */
>>> + if (glibc_hwcaps_priorities_length == 0)
>>
>> Maybe an early exit here? It allow move the code one identation left
>> and makes it more readable.
>
> I've moved this into a separate initialization function.
Ack.
>
>>> + while (left < left_end && right < right_end)
>>> + {
>>> + uint32_t string_table_index = *left;
>>> + if (string_table_index < cachesize)
>>> + {
>>> + const char *left_name
>>> + = (const char *) cache + string_table_index;
>>> + uint32_t left_name_length = strlen (left_name);
>>> + uint32_t to_compare;
>>> + if (left_name_length < right->name_length)
>>> + to_compare = left_name_length;
>>> + else
>>> + to_compare = right->name_length;
>>> + int cmp = memcmp (left_name, right->name, to_compare);
>>> + if (cmp == 0)
>>> + {
>>> + if (left_name_length < right->name_length)
>>> + cmp = -1;
>>> + else if (left_name_length > right->name_length)
>>> + cmp = 1;
>>> + }
>>> + if (cmp == 0)
>>> + {
>>> + *result = right->priority;
>>> + ++result;
>>> + ++left;
>>> + ++right;
>>> + }
>>
>> Maybe add as the 'else' within the 'cmp == 0' below?
>>
>>> + else if (cmp < 0)
>>> + {
>>> + *result = 0;
>>> + ++result;
>>> + ++left;
>>> + }
>>> + else
>>> + ++right;
>
> You mean like this?
>
> if (cmp == 0)
> {
> *result = right->priority;
> ++result;
> ++left;
> }
> if (cmp < 0)
> {
> *result = 0;
> ++result;
> ++left;
> }
> if (cmp >= 0)
> ++right;
The v5 seems more readable in fact.
>
> I don't think that's an improvement.
>
>>> +struct dl_hwcaps_priority *_dl_hwcaps_priorities;
>>> +uint32_t _dl_hwcaps_priorities_length;
>>> +
>>> +/* Allocate _dl_hwcaps_priorities and fill it with data. */
>>> +static void
>>> +compute_priorities (size_t total_count, const char *prepend,
>>> + int32_t bitmask, const char *mask)
>>
>> I think bitmask should be a 'uint32' here.
>
> Right, fixed.
>
>>> +int
>>> +main (int argc, char **argv)
>>> +{
>>> + /* Run ldconfig to populate the cache. */
>>> + {
>>> + char *command = xasprintf ("%s/ldconfig", support_install_rootsbindir);
>>> + if (system (command) != 0)
>>> + return 1;
>>> + free (command);
>>> + }
>>> +
>>> + /* Reuse tst-glibc-hwcaps. Since this code is running in a
>>> + container, we can launch it directly. */
>>> + char *path = xasprintf ("%s/elf/tst-glibc-hwcaps", support_objdir_root);
>>> + execv (path, argv);
>>> + printf ("error: execv of %s failed: %m\n", path);
>>> + return 1;
>>> +}
>>
>> Ok, it should be simple enough to require use libsupport fork/timeout
>> check.
>
> Yes, we run ldconfig without timeout elsewhere, and tst-glibc-hwcaps has
> its own timeout check.
>
>> Should we test the case of shared memory removal, for instance:
>>
>> 1. Remove second override, without ldconfig
>> 2. Remove third override, run ldconfig
>> 3. Add both second and third back, run ldconfig, remove third
>> 4. Keep second, run ldconfig
>
> I added this:
>
> + /* Remove the second override again, without running ldconfig.
> + Ideally, this would revert to implementation 2. However, in the
> + current implementation, the cache returns exactly one file name
> + which does not exist after unlinking, so the dlopen fails. */
> + xunlink ("/glibc-test/lib/glibc-hwcaps/prepend3/" SONAME);
> + TEST_VERIFY (dlopen (SONAME, RTLD_NOW) == NULL);
> + run_ldconfig ();
> + system("/sbin/ldconfig -p");
I think this might a leftover of debugging.
> + {
> + /* After running ldconfig, the second implementation is available
> + once more. */
> + void *handle = xdlopen (SONAME, RTLD_NOW);
> + int (*marker1) (void) = xdlsym (handle, "marker1");
> + TEST_COMPARE (marker1 (), 2);
> + xdlclose (handle);
> + }
>
> I think I know what I have to do to fix this in elf/dl-load.c.
>
> I feel this is a pre-existing issue. It also applies to the legacy
> hwcaps subdirectories. This only happens for ld.so.cache bringing in
> libraries from a non-searched directory, otherwise the LD_LIBRARY_PATH
> processing will find an implementation if it exists. I'd like to fix it
> in a follow-up patch.
Fair enough.
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
2020-12-02 12:08 ` Adhemerval Zanella
@ 2020-12-02 12:16 ` Florian Weimer
0 siblings, 0 replies; 58+ messages in thread
From: Florian Weimer @ 2020-12-02 12:16 UTC (permalink / raw)
To: Adhemerval Zanella via Libc-alpha
* Adhemerval Zanella via Libc-alpha:
>> + /* Remove the second override again, without running ldconfig.
>> + Ideally, this would revert to implementation 2. However, in the
>> + current implementation, the cache returns exactly one file name
>> + which does not exist after unlinking, so the dlopen fails. */
>> + xunlink ("/glibc-test/lib/glibc-hwcaps/prepend3/" SONAME);
>> + TEST_VERIFY (dlopen (SONAME, RTLD_NOW) == NULL);
>> + run_ldconfig ();
>> + system("/sbin/ldconfig -p");
>
> I think this might a leftover of debugging.
Thanks, I've removed the ldconfig -p invocation.
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 10/11] x86_64: Add glibc-hwcaps support
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (8 preceding siblings ...)
2020-11-09 18:41 ` [PATCH 09/11] elf: Add glibc-hwcaps subdirectory support to ld.so cache processing Florian Weimer
@ 2020-11-09 18:41 ` Florian Weimer
2020-12-02 12:49 ` Adhemerval Zanella
2020-11-09 18:41 ` [PATCH 11/11] powerpc64le: " Florian Weimer
` (2 subsequent siblings)
12 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:41 UTC (permalink / raw)
To: libc-alpha
The subdirectories match those in the x86-64 psABI:
https://gitlab.com/x86-psABIs/x86-64-ABI/-/commit/77566eb03bc6a326811cb7e9a6b9396884b67c7c
---
elf/Makefile | 2 +-
elf/tst-glibc-hwcaps-cache.script | 10 ++++
sysdeps/x86_64/Makefile | 39 +++++++++++++++
sysdeps/x86_64/dl-hwcaps-subdirs.c | 66 ++++++++++++++++++++++++++
sysdeps/x86_64/tst-glibc-hwcaps.c | 76 ++++++++++++++++++++++++++++++
5 files changed, 192 insertions(+), 1 deletion(-)
create mode 100644 sysdeps/x86_64/dl-hwcaps-subdirs.c
create mode 100644 sysdeps/x86_64/tst-glibc-hwcaps.c
diff --git a/elf/Makefile b/elf/Makefile
index 80e94b9ee2..0a44b19a0e 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -1826,7 +1826,7 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
# Most likely search subdirectories, for each supported architecture.
# Used to obtain test coverage wide test coverage.
-glibc-hwcaps-first-subdirs-for-tests =
+glibc-hwcaps-first-subdirs-for-tests = x86-64-v2
# The test modules are parameterized by preprocessor macros.
LDFLAGS-libmarkermod1-1.so += -Wl,-soname,libmarkermod1.so
diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
index 46cb5fd553..2aef9fb177 100644
--- a/elf/tst-glibc-hwcaps-cache.script
+++ b/elf/tst-glibc-hwcaps-cache.script
@@ -1,2 +1,12 @@
# test-container does not support scripts in sysdeps directories, so
# collect everything in one file.
+
+mkdirp 0770 $L/glibc-hwcaps/x86-64-v2
+cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod2.so
+mkdirp 0770 $L/glibc-hwcaps/x86-64-v3
+cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod3.so
+cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/x86-64-v3/libmarkermod3.so
+mkdirp 0770 $L/glibc-hwcaps/x86-64-v4
+cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod4.so
+cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/x86-64-v3/libmarkermod4.so
+cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/x86-64-v4/libmarkermod4.so
diff --git a/sysdeps/x86_64/Makefile b/sysdeps/x86_64/Makefile
index 42b97c5cc7..d1d7cb9d2e 100644
--- a/sysdeps/x86_64/Makefile
+++ b/sysdeps/x86_64/Makefile
@@ -144,8 +144,47 @@ CFLAGS-tst-auditmod10b.c += $(AVX512-CFLAGS)
CFLAGS-tst-avx512-aux.c += $(AVX512-CFLAGS)
CFLAGS-tst-avx512mod.c += $(AVX512-CFLAGS)
endif
+
+$(objpfx)tst-glibc-hwcaps: $(objpfx)libmarkermod2-1.so \
+ $(objpfx)libmarkermod3-1.so $(objpfx)libmarkermod4-1.so
+$(objpfx)tst-glibc-hwcaps.out: \
+ $(objpfx)libmarkermod2.so \
+ $(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod2.so \
+ $(objpfx)libmarkermod3.so \
+ $(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod3.so \
+ $(objpfx)glibc-hwcaps/x86-64-v3/libmarkermod3.so \
+ $(objpfx)libmarkermod4.so \
+ $(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod4.so \
+ $(objpfx)glibc-hwcaps/x86-64-v3/libmarkermod4.so \
+ $(objpfx)glibc-hwcaps/x86-64-v4/libmarkermod4.so \
+
+$(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod2.so: $(objpfx)libmarkermod2-2.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod3.so: $(objpfx)libmarkermod3-2.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/x86-64-v3/libmarkermod3.so: $(objpfx)libmarkermod3-3.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod4.so: $(objpfx)libmarkermod4-2.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/x86-64-v3/libmarkermod4.so: $(objpfx)libmarkermod4-3.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/x86-64-v4/libmarkermod4.so: $(objpfx)libmarkermod4-4.so
+ $(make-target-directory)
+ cp $< $@
+
+ifeq (no,$(build-hardcoded-path-in-tests))
+# This is an ld.so.cache test, and RPATH/RUNPATH in the executable
+# interferes with its test objectives.
+tests-container += tst-glibc-hwcaps-cache
endif
+endif # $(subdir) == elf
+
ifeq ($(subdir),csu)
gen-as-const-headers += tlsdesc.sym rtld-offsets.sym
endif
diff --git a/sysdeps/x86_64/dl-hwcaps-subdirs.c b/sysdeps/x86_64/dl-hwcaps-subdirs.c
new file mode 100644
index 0000000000..8810a822ef
--- /dev/null
+++ b/sysdeps/x86_64/dl-hwcaps-subdirs.c
@@ -0,0 +1,66 @@
+/* Architecture-specific glibc-hwcaps subdirectories. x86 version.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <dl-hwcaps.h>
+#include <cpu-features.h>
+
+const char _dl_hwcaps_subdirs[] = "x86-64-v4:x86-64-v3:x86-64-v2";
+enum { subdirs_count = 3 }; /* Number of components in _dl_hwcaps_subdirs. */
+
+uint32_t
+_dl_hwcaps_subdirs_active (void)
+{
+ int active = 0;
+
+ /* Test in reverse preference order. */
+
+ /* x86-64-v2. */
+ if (!(CPU_FEATURE_USABLE (CMPXCHG16B)
+ && CPU_FEATURE_USABLE (LAHF64_SAHF64)
+ && CPU_FEATURE_USABLE (POPCNT)
+ && CPU_FEATURE_USABLE (SSE3)
+ && CPU_FEATURE_USABLE (SSE4_1)
+ && CPU_FEATURE_USABLE (SSE4_2)
+ && CPU_FEATURE_USABLE (SSSE3)))
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+ ++active;
+
+ /* x86-64-v3. */
+ if (!(CPU_FEATURE_USABLE (AVX)
+ && CPU_FEATURE_USABLE (AVX2)
+ && CPU_FEATURE_USABLE (BMI1)
+ && CPU_FEATURE_USABLE (BMI2)
+ && CPU_FEATURE_USABLE (F16C)
+ && CPU_FEATURE_USABLE (FMA)
+ && CPU_FEATURE_USABLE (LZCNT)
+ && CPU_FEATURE_USABLE (MOVBE)
+ && CPU_FEATURE_USABLE (OSXSAVE)))
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+ ++active;
+
+ /* x86-64-v4. */
+ if (!(CPU_FEATURE_USABLE (AVX512F)
+ && CPU_FEATURE_USABLE (AVX512BW)
+ && CPU_FEATURE_USABLE (AVX512CD)
+ && CPU_FEATURE_USABLE (AVX512DQ)
+ && CPU_FEATURE_USABLE (AVX512VL)))
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+ ++active;
+
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+}
diff --git a/sysdeps/x86_64/tst-glibc-hwcaps.c b/sysdeps/x86_64/tst-glibc-hwcaps.c
new file mode 100644
index 0000000000..3075a8286d
--- /dev/null
+++ b/sysdeps/x86_64/tst-glibc-hwcaps.c
@@ -0,0 +1,76 @@
+/* glibc-hwcaps subdirectory test. x86_64 version.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <stdio.h>
+#include <support/check.h>
+#include <sys/param.h>
+#include <sys/platform/x86.h>
+
+extern int marker2 (void);
+extern int marker3 (void);
+extern int marker4 (void);
+
+/* Return the x86-64-vN level, 1 for the baseline. */
+static int
+compute_level (void)
+{
+ const struct cpu_features *cpu_features
+ = __x86_get_cpu_features (COMMON_CPUID_INDEX_MAX);
+
+ if (!(CPU_FEATURE_USABLE_P (cpu_features, CMPXCHG16B)
+ && CPU_FEATURE_USABLE_P (cpu_features, LAHF64_SAHF64)
+ && CPU_FEATURE_USABLE_P (cpu_features, POPCNT)
+ && CPU_FEATURE_USABLE_P (cpu_features, MMX)
+ && CPU_FEATURE_USABLE_P (cpu_features, SSE)
+ && CPU_FEATURE_USABLE_P (cpu_features, SSE2)
+ && CPU_FEATURE_USABLE_P (cpu_features, SSE3)
+ && CPU_FEATURE_USABLE_P (cpu_features, SSSE3)
+ && CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)
+ && CPU_FEATURE_USABLE_P (cpu_features, SSE4_2)))
+ return 1;
+ if (!(CPU_FEATURE_USABLE_P (cpu_features, AVX)
+ && CPU_FEATURE_USABLE_P (cpu_features, AVX2)
+ && CPU_FEATURE_USABLE_P (cpu_features, BMI1)
+ && CPU_FEATURE_USABLE_P (cpu_features, BMI2)
+ && CPU_FEATURE_USABLE_P (cpu_features, F16C)
+ && CPU_FEATURE_USABLE_P (cpu_features, FMA)
+ && CPU_FEATURE_USABLE_P (cpu_features, LZCNT)
+ && CPU_FEATURE_USABLE_P (cpu_features, MOVBE)
+ && CPU_FEATURE_USABLE_P (cpu_features, OSXSAVE)))
+ return 2;
+ if (!(CPU_FEATURE_USABLE_P (cpu_features, AVX512F)
+ && CPU_FEATURE_USABLE_P (cpu_features, AVX512BW)
+ && CPU_FEATURE_USABLE_P (cpu_features, AVX512CD)
+ && CPU_FEATURE_USABLE_P (cpu_features, AVX512DQ)
+ && CPU_FEATURE_USABLE_P (cpu_features, AVX512VL)))
+ return 3;
+ return 4;
+}
+
+static int
+do_test (void)
+{
+ int level = compute_level ();
+ printf ("info: detected x86-64 micro-architecture level: %d\n", level);
+ TEST_COMPARE (marker2 (), MIN (level, 2));
+ TEST_COMPARE (marker3 (), MIN (level, 3));
+ TEST_COMPARE (marker4 (), MIN (level, 4));
+ return 0;
+}
+
+#include <support/test-driver.c>
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 10/11] x86_64: Add glibc-hwcaps support
2020-11-09 18:41 ` [PATCH 10/11] x86_64: Add glibc-hwcaps support Florian Weimer
@ 2020-12-02 12:49 ` Adhemerval Zanella
0 siblings, 0 replies; 58+ messages in thread
From: Adhemerval Zanella @ 2020-12-02 12:49 UTC (permalink / raw)
To: libc-alpha, Florian Weimer
On 09/11/2020 15:41, Florian Weimer via Libc-alpha wrote:
> The subdirectories match those in the x86-64 psABI:
>
> https://gitlab.com/x86-psABIs/x86-64-ABI/-/commit/77566eb03bc6a326811cb7e9a6b9396884b67c7c
LGTM, thanks.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
> elf/Makefile | 2 +-
> elf/tst-glibc-hwcaps-cache.script | 10 ++++
> sysdeps/x86_64/Makefile | 39 +++++++++++++++
> sysdeps/x86_64/dl-hwcaps-subdirs.c | 66 ++++++++++++++++++++++++++
> sysdeps/x86_64/tst-glibc-hwcaps.c | 76 ++++++++++++++++++++++++++++++
> 5 files changed, 192 insertions(+), 1 deletion(-)
> create mode 100644 sysdeps/x86_64/dl-hwcaps-subdirs.c
> create mode 100644 sysdeps/x86_64/tst-glibc-hwcaps.c
>
> diff --git a/elf/Makefile b/elf/Makefile
> index 80e94b9ee2..0a44b19a0e 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -1826,7 +1826,7 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
>
> # Most likely search subdirectories, for each supported architecture.
> # Used to obtain test coverage wide test coverage.
> -glibc-hwcaps-first-subdirs-for-tests =
> +glibc-hwcaps-first-subdirs-for-tests = x86-64-v2
>
> # The test modules are parameterized by preprocessor macros.
> LDFLAGS-libmarkermod1-1.so += -Wl,-soname,libmarkermod1.so
Ok.
> diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
> index 46cb5fd553..2aef9fb177 100644
> --- a/elf/tst-glibc-hwcaps-cache.script
> +++ b/elf/tst-glibc-hwcaps-cache.script
> @@ -1,2 +1,12 @@
> # test-container does not support scripts in sysdeps directories, so
> # collect everything in one file.
> +
> +mkdirp 0770 $L/glibc-hwcaps/x86-64-v2
> +cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod2.so
> +mkdirp 0770 $L/glibc-hwcaps/x86-64-v3
> +cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod3.so
> +cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/x86-64-v3/libmarkermod3.so
> +mkdirp 0770 $L/glibc-hwcaps/x86-64-v4
> +cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod4.so
> +cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/x86-64-v3/libmarkermod4.so
> +cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/x86-64-v4/libmarkermod4.so
Ok.
> diff --git a/sysdeps/x86_64/Makefile b/sysdeps/x86_64/Makefile
> index 42b97c5cc7..d1d7cb9d2e 100644
> --- a/sysdeps/x86_64/Makefile
> +++ b/sysdeps/x86_64/Makefile
> @@ -144,8 +144,47 @@ CFLAGS-tst-auditmod10b.c += $(AVX512-CFLAGS)
> CFLAGS-tst-avx512-aux.c += $(AVX512-CFLAGS)
> CFLAGS-tst-avx512mod.c += $(AVX512-CFLAGS)
> endif
> +
> +$(objpfx)tst-glibc-hwcaps: $(objpfx)libmarkermod2-1.so \
> + $(objpfx)libmarkermod3-1.so $(objpfx)libmarkermod4-1.so
> +$(objpfx)tst-glibc-hwcaps.out: \
> + $(objpfx)libmarkermod2.so \
> + $(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod2.so \
> + $(objpfx)libmarkermod3.so \
> + $(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod3.so \
> + $(objpfx)glibc-hwcaps/x86-64-v3/libmarkermod3.so \
> + $(objpfx)libmarkermod4.so \
> + $(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod4.so \
> + $(objpfx)glibc-hwcaps/x86-64-v3/libmarkermod4.so \
> + $(objpfx)glibc-hwcaps/x86-64-v4/libmarkermod4.so \
> +
> +$(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod2.so: $(objpfx)libmarkermod2-2.so
> + $(make-target-directory)
> + cp $< $@
> +$(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod3.so: $(objpfx)libmarkermod3-2.so
> + $(make-target-directory)
> + cp $< $@
> +$(objpfx)glibc-hwcaps/x86-64-v3/libmarkermod3.so: $(objpfx)libmarkermod3-3.so
> + $(make-target-directory)
> + cp $< $@
> +$(objpfx)glibc-hwcaps/x86-64-v2/libmarkermod4.so: $(objpfx)libmarkermod4-2.so
> + $(make-target-directory)
> + cp $< $@
> +$(objpfx)glibc-hwcaps/x86-64-v3/libmarkermod4.so: $(objpfx)libmarkermod4-3.so
> + $(make-target-directory)
> + cp $< $@
> +$(objpfx)glibc-hwcaps/x86-64-v4/libmarkermod4.so: $(objpfx)libmarkermod4-4.so
> + $(make-target-directory)
> + cp $< $@
> +
> +ifeq (no,$(build-hardcoded-path-in-tests))
> +# This is an ld.so.cache test, and RPATH/RUNPATH in the executable
> +# interferes with its test objectives.
> +tests-container += tst-glibc-hwcaps-cache
> endif
>
> +endif # $(subdir) == elf
> +
> ifeq ($(subdir),csu)
> gen-as-const-headers += tlsdesc.sym rtld-offsets.sym
> endif
Ok.
> diff --git a/sysdeps/x86_64/dl-hwcaps-subdirs.c b/sysdeps/x86_64/dl-hwcaps-subdirs.c
> new file mode 100644
> index 0000000000..8810a822ef
> --- /dev/null
> +++ b/sysdeps/x86_64/dl-hwcaps-subdirs.c
> @@ -0,0 +1,66 @@
> +/* Architecture-specific glibc-hwcaps subdirectories. x86 version.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <dl-hwcaps.h>
> +#include <cpu-features.h>
> +
> +const char _dl_hwcaps_subdirs[] = "x86-64-v4:x86-64-v3:x86-64-v2";
> +enum { subdirs_count = 3 }; /* Number of components in _dl_hwcaps_subdirs. */
> +
> +uint32_t
> +_dl_hwcaps_subdirs_active (void)
> +{
> + int active = 0;
> +
> + /* Test in reverse preference order. */
> +
> + /* x86-64-v2. */
> + if (!(CPU_FEATURE_USABLE (CMPXCHG16B)
> + && CPU_FEATURE_USABLE (LAHF64_SAHF64)
> + && CPU_FEATURE_USABLE (POPCNT)
> + && CPU_FEATURE_USABLE (SSE3)
> + && CPU_FEATURE_USABLE (SSE4_1)
> + && CPU_FEATURE_USABLE (SSE4_2)
> + && CPU_FEATURE_USABLE (SSSE3)))
> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> + ++active;
> +
Ok, it seems to match the x86-64-ABI/low-level-sys-info.tex description.
> + /* x86-64-v3. */
> + if (!(CPU_FEATURE_USABLE (AVX)
> + && CPU_FEATURE_USABLE (AVX2)
> + && CPU_FEATURE_USABLE (BMI1)
> + && CPU_FEATURE_USABLE (BMI2)
> + && CPU_FEATURE_USABLE (F16C)
> + && CPU_FEATURE_USABLE (FMA)
> + && CPU_FEATURE_USABLE (LZCNT)
> + && CPU_FEATURE_USABLE (MOVBE)
> + && CPU_FEATURE_USABLE (OSXSAVE)))
> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> + ++active;
Ok.
> +
> + /* x86-64-v4. */
> + if (!(CPU_FEATURE_USABLE (AVX512F)
> + && CPU_FEATURE_USABLE (AVX512BW)
> + && CPU_FEATURE_USABLE (AVX512CD)
> + && CPU_FEATURE_USABLE (AVX512DQ)
> + && CPU_FEATURE_USABLE (AVX512VL)))
> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> + ++active;
> +
Ok.
> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> +}
> diff --git a/sysdeps/x86_64/tst-glibc-hwcaps.c b/sysdeps/x86_64/tst-glibc-hwcaps.c
> new file mode 100644
> index 0000000000..3075a8286d
> --- /dev/null
> +++ b/sysdeps/x86_64/tst-glibc-hwcaps.c
> @@ -0,0 +1,76 @@
> +/* glibc-hwcaps subdirectory test. x86_64 version.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <stdio.h>
> +#include <support/check.h>
> +#include <sys/param.h>
> +#include <sys/platform/x86.h>
> +
> +extern int marker2 (void);
> +extern int marker3 (void);
> +extern int marker4 (void);
> +
> +/* Return the x86-64-vN level, 1 for the baseline. */
> +static int
> +compute_level (void)
> +{
> + const struct cpu_features *cpu_features
> + = __x86_get_cpu_features (COMMON_CPUID_INDEX_MAX);
> +
> + if (!(CPU_FEATURE_USABLE_P (cpu_features, CMPXCHG16B)
> + && CPU_FEATURE_USABLE_P (cpu_features, LAHF64_SAHF64)
> + && CPU_FEATURE_USABLE_P (cpu_features, POPCNT)
> + && CPU_FEATURE_USABLE_P (cpu_features, MMX)
> + && CPU_FEATURE_USABLE_P (cpu_features, SSE)
> + && CPU_FEATURE_USABLE_P (cpu_features, SSE2)
> + && CPU_FEATURE_USABLE_P (cpu_features, SSE3)
> + && CPU_FEATURE_USABLE_P (cpu_features, SSSE3)
> + && CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)
> + && CPU_FEATURE_USABLE_P (cpu_features, SSE4_2)))
> + return 1;
This slight deviates from _dl_hwcaps_subdirs_active, since it is considering
MMX and SSE2.
> + if (!(CPU_FEATURE_USABLE_P (cpu_features, AVX)
> + && CPU_FEATURE_USABLE_P (cpu_features, AVX2)
> + && CPU_FEATURE_USABLE_P (cpu_features, BMI1)
> + && CPU_FEATURE_USABLE_P (cpu_features, BMI2)
> + && CPU_FEATURE_USABLE_P (cpu_features, F16C)
> + && CPU_FEATURE_USABLE_P (cpu_features, FMA)
> + && CPU_FEATURE_USABLE_P (cpu_features, LZCNT)
> + && CPU_FEATURE_USABLE_P (cpu_features, MOVBE)
> + && CPU_FEATURE_USABLE_P (cpu_features, OSXSAVE)))
> + return 2;
> + if (!(CPU_FEATURE_USABLE_P (cpu_features, AVX512F)
> + && CPU_FEATURE_USABLE_P (cpu_features, AVX512BW)
> + && CPU_FEATURE_USABLE_P (cpu_features, AVX512CD)
> + && CPU_FEATURE_USABLE_P (cpu_features, AVX512DQ)
> + && CPU_FEATURE_USABLE_P (cpu_features, AVX512VL)))
> + return 3;
> + return 4;
> +}
> +
> +static int
> +do_test (void)
> +{
> + int level = compute_level ();
> + printf ("info: detected x86-64 micro-architecture level: %d\n", level);
> + TEST_COMPARE (marker2 (), MIN (level, 2));
> + TEST_COMPARE (marker3 (), MIN (level, 3));
> + TEST_COMPARE (marker4 (), MIN (level, 4));
> + return 0;
> +}
> +
> +#include <support/test-driver.c>
>
Ok.
I am wondering if it would be possible to add an internal tests where
it tests the hwcap selection without relying on underlying capabilities
so we could decouple the test from the runtime where it is ran.
It might be possible to simulate it through qemu. Without it, for x86_64
it would require to change the cpu_features internal state to simulate
different x86_64 abi-set. Not sure how feasible it would be.
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH 11/11] powerpc64le: Add glibc-hwcaps support
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (9 preceding siblings ...)
2020-11-09 18:41 ` [PATCH 10/11] x86_64: Add glibc-hwcaps support Florian Weimer
@ 2020-11-09 18:41 ` Florian Weimer
2020-12-02 13:46 ` Adhemerval Zanella
2020-11-09 21:56 ` [PATCH v4 00/11] " Dan Horák
2020-11-25 15:46 ` Florian Weimer
12 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-11-09 18:41 UTC (permalink / raw)
To: libc-alpha
The "power10" and "power9" subdirectories are selected in a way
that matches the -mcpu=power10 and -mcpu=power9 options of GCC.
---
elf/Makefile | 2 +-
elf/tst-glibc-hwcaps-cache.script | 6 +++
sysdeps/powerpc/powerpc64/le/Makefile | 28 ++++++++++
.../powerpc/powerpc64/le/dl-hwcaps-subdirs.c | 46 ++++++++++++++++
.../powerpc/powerpc64/le/tst-glibc-hwcaps.c | 54 +++++++++++++++++++
5 files changed, 135 insertions(+), 1 deletion(-)
create mode 100644 sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c
create mode 100644 sysdeps/powerpc/powerpc64/le/tst-glibc-hwcaps.c
diff --git a/elf/Makefile b/elf/Makefile
index 0a44b19a0e..60e8662e5d 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -1826,7 +1826,7 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
# Most likely search subdirectories, for each supported architecture.
# Used to obtain test coverage wide test coverage.
-glibc-hwcaps-first-subdirs-for-tests = x86-64-v2
+glibc-hwcaps-first-subdirs-for-tests = power9 x86-64-v2
# The test modules are parameterized by preprocessor macros.
LDFLAGS-libmarkermod1-1.so += -Wl,-soname,libmarkermod1.so
diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
index 2aef9fb177..a3e3317bbf 100644
--- a/elf/tst-glibc-hwcaps-cache.script
+++ b/elf/tst-glibc-hwcaps-cache.script
@@ -10,3 +10,9 @@ mkdirp 0770 $L/glibc-hwcaps/x86-64-v4
cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod4.so
cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/x86-64-v3/libmarkermod4.so
cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/x86-64-v4/libmarkermod4.so
+
+mkdirp 0770 $L/glibc-hwcaps/power9
+cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/power9/libmarkermod2.so
+mkdirp 0770 $L/glibc-hwcaps/power10
+cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/power9/libmarkermod3.so
+cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/power10/libmarkermod3.so
diff --git a/sysdeps/powerpc/powerpc64/le/Makefile b/sysdeps/powerpc/powerpc64/le/Makefile
index 033dc77b01..7c036b45fc 100644
--- a/sysdeps/powerpc/powerpc64/le/Makefile
+++ b/sysdeps/powerpc/powerpc64/le/Makefile
@@ -188,3 +188,31 @@ ifeq ($(subdir),nptl)
CFLAGS-tst-thread_local1.cc += -mno-float128
CFLAGS-tst-minstack-throw.cc += -mno-float128
endif
+
+ifeq ($(subdir),elf)
+$(objpfx)tst-glibc-hwcaps: \
+ $(objpfx)libmarkermod2-1.so $(objpfx)libmarkermod3-1.so
+$(objpfx)tst-glibc-hwcaps.out: \
+ $(objpfx)libmarkermod2.so \
+ $(objpfx)glibc-hwcaps/power9/libmarkermod2.so \
+ $(objpfx)libmarkermod3.so \
+ $(objpfx)glibc-hwcaps/power9/libmarkermod3.so \
+ $(objpfx)glibc-hwcaps/power10/libmarkermod3.so \
+
+$(objpfx)glibc-hwcaps/power9/libmarkermod2.so: $(objpfx)libmarkermod2-2.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/power9/libmarkermod3.so: $(objpfx)libmarkermod3-2.so
+ $(make-target-directory)
+ cp $< $@
+$(objpfx)glibc-hwcaps/power10/libmarkermod3.so: $(objpfx)libmarkermod3-3.so
+ $(make-target-directory)
+ cp $< $@
+
+ifeq (no,$(build-hardcoded-path-in-tests))
+# This is an ld.so.cache test, and RPATH/RUNPATH in the executable
+# interferes with its test objectives.
+tests-container += tst-glibc-hwcaps-cache
+endif
+
+endif # $(subdir) == elf
diff --git a/sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c b/sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c
new file mode 100644
index 0000000000..0ce76c2fe5
--- /dev/null
+++ b/sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c
@@ -0,0 +1,46 @@
+/* Architecture-specific glibc-hwcaps subdirectories. powerpc64le version.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <dl-hwcaps.h>
+#include <ldsodefs.h>
+
+const char _dl_hwcaps_subdirs[] = "power10:power9";
+enum { subdirs_count = 2 }; /* Number of components in _dl_hwcaps_subdirs. */
+
+uint32_t
+_dl_hwcaps_subdirs_active (void)
+{
+ int active = 0;
+
+ /* Test in reverse preference order. Altivec and VSX are implied by
+ the powerpc64le ABI definition. */
+
+ /* POWER9. GCC enables float128 hardware support for -mcpu=power9. */
+ if ((GLRO (dl_hwcap2) & PPC_FEATURE2_ARCH_3_00) == 0
+ || (GLRO (dl_hwcap2) & PPC_FEATURE2_HAS_IEEE128) == 0)
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+ ++active;
+
+ /* POWER10. GCC defines __MMA__ for -mcpu=power10. */
+ if ((GLRO (dl_hwcap2) & PPC_FEATURE2_ARCH_3_1) == 0
+ || (GLRO (dl_hwcap2) & PPC_FEATURE2_MMA) == 0)
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+ ++active;
+
+ return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+}
diff --git a/sysdeps/powerpc/powerpc64/le/tst-glibc-hwcaps.c b/sysdeps/powerpc/powerpc64/le/tst-glibc-hwcaps.c
new file mode 100644
index 0000000000..e510fca80a
--- /dev/null
+++ b/sysdeps/powerpc/powerpc64/le/tst-glibc-hwcaps.c
@@ -0,0 +1,54 @@
+/* glibc-hwcaps subdirectory test. powerpc64le version.
+ Copyright (C) 2020 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <stdio.h>
+#include <string.h>
+#include <support/check.h>
+#include <sys/auxv.h>
+#include <sys/param.h>
+
+extern int marker2 (void);
+extern int marker3 (void);
+
+/* Return the POWER level, 8 for the baseline. */
+static int
+compute_level (void)
+{
+ const char *platform = (const char *) getauxval (AT_PLATFORM);
+ if (strcmp (platform, "power8") == 0)
+ return 8;
+ if (strcmp (platform, "power9") == 0)
+ return 9;
+ if (strcmp (platform, "power10") == 0)
+ return 10;
+ printf ("warning: unrecognized AT_PLATFORM value: %s\n", platform);
+ /* Assume that the new platform supports POWER10. */
+ return 10;
+}
+
+static int
+do_test (void)
+{
+ int level = compute_level ();
+ printf ("info: detected POWER level: %d\n", level);
+ TEST_COMPARE (marker2 (), MIN (level - 7, 2));
+ TEST_COMPARE (marker3 (), MIN (level - 7, 3));
+ return 0;
+}
+
+#include <support/test-driver.c>
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 11/11] powerpc64le: Add glibc-hwcaps support
2020-11-09 18:41 ` [PATCH 11/11] powerpc64le: " Florian Weimer
@ 2020-12-02 13:46 ` Adhemerval Zanella
2020-12-02 13:51 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-12-02 13:46 UTC (permalink / raw)
To: libc-alpha, Florian Weimer
On 09/11/2020 15:41, Florian Weimer via Libc-alpha wrote:
> The "power10" and "power9" subdirectories are selected in a way
> that matches the -mcpu=power10 and -mcpu=power9 options of GCC.
LGTM, a comment about testing below.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
> elf/Makefile | 2 +-
> elf/tst-glibc-hwcaps-cache.script | 6 +++
> sysdeps/powerpc/powerpc64/le/Makefile | 28 ++++++++++
> .../powerpc/powerpc64/le/dl-hwcaps-subdirs.c | 46 ++++++++++++++++
> .../powerpc/powerpc64/le/tst-glibc-hwcaps.c | 54 +++++++++++++++++++
> 5 files changed, 135 insertions(+), 1 deletion(-)
> create mode 100644 sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c
> create mode 100644 sysdeps/powerpc/powerpc64/le/tst-glibc-hwcaps.c
>
> diff --git a/elf/Makefile b/elf/Makefile
> index 0a44b19a0e..60e8662e5d 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -1826,7 +1826,7 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
>
> # Most likely search subdirectories, for each supported architecture.
> # Used to obtain test coverage wide test coverage.
> -glibc-hwcaps-first-subdirs-for-tests = x86-64-v2
> +glibc-hwcaps-first-subdirs-for-tests = power9 x86-64-v2
>
> # The test modules are parameterized by preprocessor macros.
> LDFLAGS-libmarkermod1-1.so += -Wl,-soname,libmarkermod1.so
Ok.
> diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
> index 2aef9fb177..a3e3317bbf 100644
> --- a/elf/tst-glibc-hwcaps-cache.script
> +++ b/elf/tst-glibc-hwcaps-cache.script
> @@ -10,3 +10,9 @@ mkdirp 0770 $L/glibc-hwcaps/x86-64-v4
> cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod4.so
> cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/x86-64-v3/libmarkermod4.so
> cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/x86-64-v4/libmarkermod4.so
> +
> +mkdirp 0770 $L/glibc-hwcaps/power9
> +cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/power9/libmarkermod2.so
> +mkdirp 0770 $L/glibc-hwcaps/power10
> +cp $B/elf/libmarkermod3-2.so $L/glibc-hwcaps/power9/libmarkermod3.so
> +cp $B/elf/libmarkermod3-3.so $L/glibc-hwcaps/power10/libmarkermod3.so
Ok.
> diff --git a/sysdeps/powerpc/powerpc64/le/Makefile b/sysdeps/powerpc/powerpc64/le/Makefile
> index 033dc77b01..7c036b45fc 100644
> --- a/sysdeps/powerpc/powerpc64/le/Makefile
> +++ b/sysdeps/powerpc/powerpc64/le/Makefile
> @@ -188,3 +188,31 @@ ifeq ($(subdir),nptl)
> CFLAGS-tst-thread_local1.cc += -mno-float128
> CFLAGS-tst-minstack-throw.cc += -mno-float128
> endif
> +
> +ifeq ($(subdir),elf)
> +$(objpfx)tst-glibc-hwcaps: \
> + $(objpfx)libmarkermod2-1.so $(objpfx)libmarkermod3-1.so
> +$(objpfx)tst-glibc-hwcaps.out: \
> + $(objpfx)libmarkermod2.so \
> + $(objpfx)glibc-hwcaps/power9/libmarkermod2.so \
> + $(objpfx)libmarkermod3.so \
> + $(objpfx)glibc-hwcaps/power9/libmarkermod3.so \
> + $(objpfx)glibc-hwcaps/power10/libmarkermod3.so \
> +
> +$(objpfx)glibc-hwcaps/power9/libmarkermod2.so: $(objpfx)libmarkermod2-2.so
> + $(make-target-directory)
> + cp $< $@
> +$(objpfx)glibc-hwcaps/power9/libmarkermod3.so: $(objpfx)libmarkermod3-2.so
> + $(make-target-directory)
> + cp $< $@
> +$(objpfx)glibc-hwcaps/power10/libmarkermod3.so: $(objpfx)libmarkermod3-3.so
> + $(make-target-directory)
> + cp $< $@
> +
> +ifeq (no,$(build-hardcoded-path-in-tests))
> +# This is an ld.so.cache test, and RPATH/RUNPATH in the executable
> +# interferes with its test objectives.
> +tests-container += tst-glibc-hwcaps-cache
> +endif
> +
> +endif # $(subdir) == elf
Ok.
> diff --git a/sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c b/sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c
> new file mode 100644
> index 0000000000..0ce76c2fe5
> --- /dev/null
> +++ b/sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c
> @@ -0,0 +1,46 @@
> +/* Architecture-specific glibc-hwcaps subdirectories. powerpc64le version.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <dl-hwcaps.h>
> +#include <ldsodefs.h>
> +
> +const char _dl_hwcaps_subdirs[] = "power10:power9";
> +enum { subdirs_count = 2 }; /* Number of components in _dl_hwcaps_subdirs. */
> +
> +uint32_t
> +_dl_hwcaps_subdirs_active (void)
> +{
> + int active = 0;
> +
> + /* Test in reverse preference order. Altivec and VSX are implied by
> + the powerpc64le ABI definition. */
> +
> + /* POWER9. GCC enables float128 hardware support for -mcpu=power9. */
> + if ((GLRO (dl_hwcap2) & PPC_FEATURE2_ARCH_3_00) == 0
> + || (GLRO (dl_hwcap2) & PPC_FEATURE2_HAS_IEEE128) == 0)
> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> + ++active;
Should we test PPC_FEATURE2_DARN as well? I think cryptographic and related
libraries might use it as source of entropy.
> +
> + /* POWER10. GCC defines __MMA__ for -mcpu=power10. */
Double space after period.
> + if ((GLRO (dl_hwcap2) & PPC_FEATURE2_ARCH_3_1) == 0
> + || (GLRO (dl_hwcap2) & PPC_FEATURE2_MMA) == 0)
> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> + ++active;
> +
> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> +}
Ok.
> diff --git a/sysdeps/powerpc/powerpc64/le/tst-glibc-hwcaps.c b/sysdeps/powerpc/powerpc64/le/tst-glibc-hwcaps.c
> new file mode 100644
> index 0000000000..e510fca80a
> --- /dev/null
> +++ b/sysdeps/powerpc/powerpc64/le/tst-glibc-hwcaps.c
> @@ -0,0 +1,54 @@
> +/* glibc-hwcaps subdirectory test. powerpc64le version.
> + Copyright (C) 2020 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <https://www.gnu.org/licenses/>. */
> +
> +#include <stdio.h>
> +#include <string.h>
> +#include <support/check.h>
> +#include <sys/auxv.h>
> +#include <sys/param.h>
> +
> +extern int marker2 (void);
> +extern int marker3 (void);
> +
> +/* Return the POWER level, 8 for the baseline. */
> +static int
> +compute_level (void)
> +{
> + const char *platform = (const char *) getauxval (AT_PLATFORM);
> + if (strcmp (platform, "power8") == 0)
> + return 8;
> + if (strcmp (platform, "power9") == 0)
> + return 9;
> + if (strcmp (platform, "power10") == 0)
> + return 10;
Why not use HWCAP? Is it to decouple from the loader code that uses it?
I am asking because by using hwcap we can check if this tests is working
on different chips by using qemu-ppc64le with -mcpu.
> + printf ("warning: unrecognized AT_PLATFORM value: %s\n", platform);
> + /* Assume that the new platform supports POWER10. */
> + return 10;
> +}
> +
> +static int
> +do_test (void)
> +{
> + int level = compute_level ();
> + printf ("info: detected POWER level: %d\n", level);
> + TEST_COMPARE (marker2 (), MIN (level - 7, 2));
> + TEST_COMPARE (marker3 (), MIN (level - 7, 3));
> + return 0;
> +}
> +
> +#include <support/test-driver.c>
>
Ok.
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 11/11] powerpc64le: Add glibc-hwcaps support
2020-12-02 13:46 ` Adhemerval Zanella
@ 2020-12-02 13:51 ` Florian Weimer
2020-12-02 14:27 ` Adhemerval Zanella
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-12-02 13:51 UTC (permalink / raw)
To: Adhemerval Zanella; +Cc: libc-alpha
* Adhemerval Zanella:
>> +uint32_t
>> +_dl_hwcaps_subdirs_active (void)
>> +{
>> + int active = 0;
>> +
>> + /* Test in reverse preference order. Altivec and VSX are implied by
>> + the powerpc64le ABI definition. */
>> +
>> + /* POWER9. GCC enables float128 hardware support for -mcpu=power9. */
>> + if ((GLRO (dl_hwcap2) & PPC_FEATURE2_ARCH_3_00) == 0
>> + || (GLRO (dl_hwcap2) & PPC_FEATURE2_HAS_IEEE128) == 0)
>> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
>> + ++active;
>
> Should we test PPC_FEATURE2_DARN as well? I think cryptographic and related
> libraries might use it as source of entropy.
It does not show up in the compiler preprocessor macros. There also
have been cases where randomness-generating instructions have been
disabled in firmware (but probably not on POWER). I think it's safer
not to include cryptographic stuff.
>> +
>> + /* POWER10. GCC defines __MMA__ for -mcpu=power10. */
>
> Double space after period.
Thanks, fixed.
>> +/* Return the POWER level, 8 for the baseline. */
>> +static int
>> +compute_level (void)
>> +{
>> + const char *platform = (const char *) getauxval (AT_PLATFORM);
>> + if (strcmp (platform, "power8") == 0)
>> + return 8;
>> + if (strcmp (platform, "power9") == 0)
>> + return 9;
>> + if (strcmp (platform, "power10") == 0)
>> + return 10;
>
> Why not use HWCAP? Is it to decouple from the loader code that uses it?
> I am asking because by using hwcap we can check if this tests is working
> on different chips by using qemu-ppc64le with -mcpu.
And that doesn't affect AT_PLATFORM? That would be a QEMU bug.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 11/11] powerpc64le: Add glibc-hwcaps support
2020-12-02 13:51 ` Florian Weimer
@ 2020-12-02 14:27 ` Adhemerval Zanella
2020-12-02 14:31 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Adhemerval Zanella @ 2020-12-02 14:27 UTC (permalink / raw)
To: Florian Weimer; +Cc: libc-alpha
On 02/12/2020 10:51, Florian Weimer wrote:
> * Adhemerval Zanella:
>
>>> +uint32_t
>>> +_dl_hwcaps_subdirs_active (void)
>>> +{
>>> + int active = 0;
>>> +
>>> + /* Test in reverse preference order. Altivec and VSX are implied by
>>> + the powerpc64le ABI definition. */
>>> +
>>> + /* POWER9. GCC enables float128 hardware support for -mcpu=power9. */
>>> + if ((GLRO (dl_hwcap2) & PPC_FEATURE2_ARCH_3_00) == 0
>>> + || (GLRO (dl_hwcap2) & PPC_FEATURE2_HAS_IEEE128) == 0)
>>> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
>>> + ++active;
>>
>> Should we test PPC_FEATURE2_DARN as well? I think cryptographic and related
>> libraries might use it as source of entropy.
>
> It does not show up in the compiler preprocessor macros. There also
> have been cases where randomness-generating instructions have been
> disabled in firmware (but probably not on POWER). I think it's safer
> not to include cryptographic stuff.
GCC does provides it through a compiler builtin, only enabled for
power9 and newer. And I think hardware entropy instruction are not
used on cryptographic, but also on different fields as simulations.
>
>>> +
>>> + /* POWER10. GCC defines __MMA__ for -mcpu=power10. */
>>
>> Double space after period.
>
> Thanks, fixed.
>
>>> +/* Return the POWER level, 8 for the baseline. */
>>> +static int
>>> +compute_level (void)
>>> +{
>>> + const char *platform = (const char *) getauxval (AT_PLATFORM);
>>> + if (strcmp (platform, "power8") == 0)
>>> + return 8;
>>> + if (strcmp (platform, "power9") == 0)
>>> + return 9;
>>> + if (strcmp (platform, "power10") == 0)
>>> + return 10;
>>
>> Why not use HWCAP? Is it to decouple from the loader code that uses it?
>> I am asking because by using hwcap we can check if this tests is working
>> on different chips by using qemu-ppc64le with -mcpu.
>
> And that doesn't affect AT_PLATFORM? That would be a QEMU bug.
At least not with v5.2.0-rc0-99-g3493c36f03, not sure if is has been
fixed/change upstream.
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 11/11] powerpc64le: Add glibc-hwcaps support
2020-12-02 14:27 ` Adhemerval Zanella
@ 2020-12-02 14:31 ` Florian Weimer
2020-12-02 15:34 ` Carlos Seo
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-12-02 14:31 UTC (permalink / raw)
To: Adhemerval Zanella; +Cc: Tulio Magno Quites Machado Filho, libc-alpha
* Adhemerval Zanella:
> On 02/12/2020 10:51, Florian Weimer wrote:
>> * Adhemerval Zanella:
>>
>>>> +uint32_t
>>>> +_dl_hwcaps_subdirs_active (void)
>>>> +{
>>>> + int active = 0;
>>>> +
>>>> + /* Test in reverse preference order. Altivec and VSX are implied by
>>>> + the powerpc64le ABI definition. */
>>>> +
>>>> + /* POWER9. GCC enables float128 hardware support for -mcpu=power9. */
>>>> + if ((GLRO (dl_hwcap2) & PPC_FEATURE2_ARCH_3_00) == 0
>>>> + || (GLRO (dl_hwcap2) & PPC_FEATURE2_HAS_IEEE128) == 0)
>>>> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
>>>> + ++active;
>>>
>>> Should we test PPC_FEATURE2_DARN as well? I think cryptographic and related
>>> libraries might use it as source of entropy.
>>
>> It does not show up in the compiler preprocessor macros. There also
>> have been cases where randomness-generating instructions have been
>> disabled in firmware (but probably not on POWER). I think it's safer
>> not to include cryptographic stuff.
>
> GCC does provides it through a compiler builtin, only enabled for
> power9 and newer. And I think hardware entropy instruction are not
> used on cryptographic, but also on different fields as simulations.
Maybe Tulio can comment. I do not have a strong opinion (although the
x86 precedent is kind of unsettling).
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 11/11] powerpc64le: Add glibc-hwcaps support
2020-12-02 14:31 ` Florian Weimer
@ 2020-12-02 15:34 ` Carlos Seo
2020-12-02 20:14 ` Tulio Magno Quites Machado Filho
0 siblings, 1 reply; 58+ messages in thread
From: Carlos Seo @ 2020-12-02 15:34 UTC (permalink / raw)
To: Florian Weimer
Cc: Adhemerval Zanella, libc-alpha, Tulio Magno Quites Machado Filho
'darn' requires firmware support, so that's why it has a hwcap2 bit
separate from ISA 3.0 (see commit a4700a26 in the kernel tree). Although I
personally have not seen any firmware that disables that on POWER9, it is
safer to assume that ISA 3.0 does not imply that 'darn' is available.
On Wed, 2 Dec 2020 at 11:31, Florian Weimer via Libc-alpha <
libc-alpha@sourceware.org> wrote:
> * Adhemerval Zanella:
>
> > On 02/12/2020 10:51, Florian Weimer wrote:
> >> * Adhemerval Zanella:
> >>
> >>>> +uint32_t
> >>>> +_dl_hwcaps_subdirs_active (void)
> >>>> +{
> >>>> + int active = 0;
> >>>> +
> >>>> + /* Test in reverse preference order. Altivec and VSX are implied
> by
> >>>> + the powerpc64le ABI definition. */
> >>>> +
> >>>> + /* POWER9. GCC enables float128 hardware support for
> -mcpu=power9. */
> >>>> + if ((GLRO (dl_hwcap2) & PPC_FEATURE2_ARCH_3_00) == 0
> >>>> + || (GLRO (dl_hwcap2) & PPC_FEATURE2_HAS_IEEE128) == 0)
> >>>> + return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> >>>> + ++active;
> >>>
> >>> Should we test PPC_FEATURE2_DARN as well? I think cryptographic and
> related
> >>> libraries might use it as source of entropy.
> >>
> >> It does not show up in the compiler preprocessor macros. There also
> >> have been cases where randomness-generating instructions have been
> >> disabled in firmware (but probably not on POWER). I think it's safer
> >> not to include cryptographic stuff.
> >
> > GCC does provides it through a compiler builtin, only enabled for
> > power9 and newer. And I think hardware entropy instruction are not
> > used on cryptographic, but also on different fields as simulations.
>
> Maybe Tulio can comment. I do not have a strong opinion (although the
> x86 precedent is kind of unsettling).
>
> Thanks,
> Florian
> --
> Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
> O'Neill
>
>
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH 11/11] powerpc64le: Add glibc-hwcaps support
2020-12-02 15:34 ` Carlos Seo
@ 2020-12-02 20:14 ` Tulio Magno Quites Machado Filho
2020-12-04 8:56 ` Florian Weimer
0 siblings, 1 reply; 58+ messages in thread
From: Tulio Magno Quites Machado Filho @ 2020-12-02 20:14 UTC (permalink / raw)
To: Carlos Seo, Florian Weimer, Adhemerval Zanella
Cc: libc-alpha, Matheus Salgueiro Castanho, Paul Clarke
Carlos Seo via Libc-alpha <libc-alpha@sourceware.org> writes:
> 'darn' requires firmware support, so that's why it has a hwcap2 bit
> separate from ISA 3.0 (see commit a4700a26 in the kernel tree). Although I
> personally have not seen any firmware that disables that on POWER9, it is
> safer to assume that ISA 3.0 does not imply that 'darn' is available.
I think we have 2 different questions to be answered here:
1. Is it safe to use darn on a processor compatible with the POWER ISA 3.0b?
2. Should glibc-hwcaps test for PPC_FEATURE2_DARN when loading POWER9-specific
libraries?
I agree with Carlos regarding question #1: it's unsafe to use darn without
testing for PPC_FEATURE2_DARN.
Regarding question #2, I see darn much like HTM even if darn doesn't have its
own -m parameter. What I mean is: I don't think the entire power9 directory
should be disabled if a kernel (or firmware) decides to disable
PPC_FEATURE2_DARN.
With that said, I don't have a strong opinion on question #2.
By the way, I have the same opinion for scv: it's unsafe to use it without
testing for PPC_FEATURE2_SCV and I think glibc-hwcaps should not require it.
--
Tulio Magno
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH 11/11] powerpc64le: Add glibc-hwcaps support
2020-12-02 20:14 ` Tulio Magno Quites Machado Filho
@ 2020-12-04 8:56 ` Florian Weimer
2020-12-04 12:35 ` Adhemerval Zanella
0 siblings, 1 reply; 58+ messages in thread
From: Florian Weimer @ 2020-12-04 8:56 UTC (permalink / raw)
To: Tulio Magno Quites Machado Filho via Libc-alpha
Cc: Carlos Seo, Adhemerval Zanella, Tulio Magno Quites Machado Filho,
Paul Clarke
* Tulio Magno Quites Machado Filho via Libc-alpha:
> Carlos Seo via Libc-alpha <libc-alpha@sourceware.org> writes:
>
>> 'darn' requires firmware support, so that's why it has a hwcap2 bit
>> separate from ISA 3.0 (see commit a4700a26 in the kernel tree). Although I
>> personally have not seen any firmware that disables that on POWER9, it is
>> safer to assume that ISA 3.0 does not imply that 'darn' is available.
>
> I think we have 2 different questions to be answered here:
>
> 1. Is it safe to use darn on a processor compatible with the POWER ISA 3.0b?
>
> 2. Should glibc-hwcaps test for PPC_FEATURE2_DARN when loading POWER9-specific
> libraries?
>
> I agree with Carlos regarding question #1: it's unsafe to use darn without
> testing for PPC_FEATURE2_DARN.
>
> Regarding question #2, I see darn much like HTM even if darn doesn't have its
> own -m parameter. What I mean is: I don't think the entire power9 directory
> should be disabled if a kernel (or firmware) decides to disable
> PPC_FEATURE2_DARN.
>
> With that said, I don't have a strong opinion on question #2.
>
> By the way, I have the same opinion for scv: it's unsafe to use it without
> testing for PPC_FEATURE2_SCV and I think glibc-hwcaps should not require it.
Okay, then the posted patch is okay as is? May I consider it reviewed?
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH 11/11] powerpc64le: Add glibc-hwcaps support
2020-12-04 8:56 ` Florian Weimer
@ 2020-12-04 12:35 ` Adhemerval Zanella
0 siblings, 0 replies; 58+ messages in thread
From: Adhemerval Zanella @ 2020-12-04 12:35 UTC (permalink / raw)
To: Florian Weimer, Tulio Magno Quites Machado Filho via Libc-alpha
Cc: Carlos Seo, Tulio Magno Quites Machado Filho, Paul Clarke
On 04/12/2020 05:56, Florian Weimer wrote:
> * Tulio Magno Quites Machado Filho via Libc-alpha:
>
>> Carlos Seo via Libc-alpha <libc-alpha@sourceware.org> writes:
>>
>>> 'darn' requires firmware support, so that's why it has a hwcap2 bit
>>> separate from ISA 3.0 (see commit a4700a26 in the kernel tree). Although I
>>> personally have not seen any firmware that disables that on POWER9, it is
>>> safer to assume that ISA 3.0 does not imply that 'darn' is available.
>>
>> I think we have 2 different questions to be answered here:
>>
>> 1. Is it safe to use darn on a processor compatible with the POWER ISA 3.0b?
>>
>> 2. Should glibc-hwcaps test for PPC_FEATURE2_DARN when loading POWER9-specific
>> libraries?
>>
>> I agree with Carlos regarding question #1: it's unsafe to use darn without
>> testing for PPC_FEATURE2_DARN.
>>
>> Regarding question #2, I see darn much like HTM even if darn doesn't have its
>> own -m parameter. What I mean is: I don't think the entire power9 directory
>> should be disabled if a kernel (or firmware) decides to disable
>> PPC_FEATURE2_DARN.
>>
>> With that said, I don't have a strong opinion on question #2.
>>
>> By the way, I have the same opinion for scv: it's unsafe to use it without
>> testing for PPC_FEATURE2_SCV and I think glibc-hwcaps should not require it.
>
> Okay, then the posted patch is okay as is? May I consider it reviewed?
LGTM as is, I think we can add PPC_FEATURE2_DARN as an extra hwcap folder
is required.
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 00/11] glibc-hwcaps support
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (10 preceding siblings ...)
2020-11-09 18:41 ` [PATCH 11/11] powerpc64le: " Florian Weimer
@ 2020-11-09 21:56 ` Dan Horák
2020-11-10 10:41 ` Florian Weimer
2020-11-25 15:46 ` Florian Weimer
12 siblings, 1 reply; 58+ messages in thread
From: Dan Horák @ 2020-11-09 21:56 UTC (permalink / raw)
To: libc-alpha; +Cc: Florian Weimer
On Mon, 09 Nov 2020 19:40:10 +0100
Florian Weimer via Libc-alpha <libc-alpha@sourceware.org> wrote:
> This patch series incorporates Adhemerval's feedback. It picks up the
> suggestion for introducing enum opt_format in ldconfig.
>
> It also fixes a a ldconfig bug (wrong size in xmalloc call) uncovered by
> testing on i686-linux-gnu.
>
> I have dropped the s390x patch until I have located a suitable machine
> on which to test it. We also need to discuss how to achieve GCC
> alignment there.
Florian, what kind of machine do you need?
Dan
> Patch order is different than before because both the x86_64 and
> powerpc64le patches contain tests now, including a test that runs the
> ld.so-based test without special options in the container, to check for
> ld.so.cache support.
>
> The powerpc64le patch aligns the subdirectory selection with GCC.
>
> I feel like this series has a reasonable level of test coverage. It
> does not test all possible permutations (ld.so option combinations and
> ld.so.cache contents), but I hope that's okay.
>
> (Not sure if this actually v4, I sort of lost count.)
>
> Thanks,
> Florian
>
> Florian Weimer (11):
> support: Add support_copy_file
> elf: Introduce enum opt_format in the ldconfig implementation
> elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
> elf: Add endianness markup to ld.so.cache
> elf: Add extension mechanism to ld.so.cache
> elf: Implement a string table for ldconfig, with tail merging
> elf: Implement tail merging of strings in ldconfig
> elf: Process glibc-hwcaps subdirectories in ldconfig
> elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
> x86_64: Add glibc-hwcaps support
> powerpc64le: Add glibc-hwcaps support
>
> elf/Makefile | 90 +++-
> elf/cache.c | 413 +++++++++++++++---
> elf/dl-cache.c | 202 ++++++++-
> elf/dl-hwcaps-subdirs.c | 29 ++
> elf/dl-hwcaps.c | 216 ++++++++-
> elf/dl-hwcaps.h | 124 ++++++
> elf/dl-hwcaps_split.c | 77 ++++
> elf/dl-load.c | 7 +-
> elf/dl-main.h | 11 +-
> elf/dl-support.c | 5 +-
> elf/dl-usage.c | 68 ++-
> elf/ldconfig.c | 164 +++++--
> elf/markermodMARKER-VALUE.c | 29 ++
> elf/rtld.c | 18 +
> elf/stringtable.c | 209 +++++++++
> elf/stringtable.h | 64 +++
> elf/stringtable_free.c | 33 ++
> elf/tst-dl-hwcaps_split.c | 139 ++++++
> elf/tst-glibc-hwcaps-cache.c | 45 ++
> .../etc/ld.so.conf | 2 +
> elf/tst-glibc-hwcaps-cache.root/postclean.req | 0
> elf/tst-glibc-hwcaps-cache.script | 18 +
> elf/tst-glibc-hwcaps-mask.c | 31 ++
> elf/tst-glibc-hwcaps-prepend-cache.c | 133 ++++++
> .../postclean.req | 0
> elf/tst-glibc-hwcaps-prepend.c | 32 ++
> elf/tst-glibc-hwcaps.c | 28 ++
> elf/tst-stringtable.c | 141 ++++++
> support/Makefile | 1 +
> support/support.h | 5 +
> support/support_copy_file.c | 43 ++
> sysdeps/generic/dl-cache.h | 233 +++++++++-
> sysdeps/generic/ldconfig.h | 27 +-
> sysdeps/generic/ldsodefs.h | 20 +-
> sysdeps/powerpc/powerpc64/le/Makefile | 28 ++
> .../powerpc/powerpc64/le/dl-hwcaps-subdirs.c | 46 ++
> .../powerpc/powerpc64/le/tst-glibc-hwcaps.c | 54 +++
> sysdeps/x86_64/Makefile | 39 ++
> sysdeps/x86_64/dl-hwcaps-subdirs.c | 66 +++
> sysdeps/x86_64/tst-glibc-hwcaps.c | 76 ++++
> 40 files changed, 2839 insertions(+), 127 deletions(-)
> create mode 100644 elf/dl-hwcaps-subdirs.c
> create mode 100644 elf/dl-hwcaps_split.c
> create mode 100644 elf/markermodMARKER-VALUE.c
> create mode 100644 elf/stringtable.c
> create mode 100644 elf/stringtable.h
> create mode 100644 elf/stringtable_free.c
> create mode 100644 elf/tst-dl-hwcaps_split.c
> create mode 100644 elf/tst-glibc-hwcaps-cache.c
> create mode 100644 elf/tst-glibc-hwcaps-cache.root/etc/ld.so.conf
> create mode 100644 elf/tst-glibc-hwcaps-cache.root/postclean.req
> create mode 100644 elf/tst-glibc-hwcaps-cache.script
> create mode 100644 elf/tst-glibc-hwcaps-mask.c
> create mode 100644 elf/tst-glibc-hwcaps-prepend-cache.c
> create mode 100644 elf/tst-glibc-hwcaps-prepend-cache.root/postclean.req
> create mode 100644 elf/tst-glibc-hwcaps-prepend.c
> create mode 100644 elf/tst-glibc-hwcaps.c
> create mode 100644 elf/tst-stringtable.c
> create mode 100644 support/support_copy_file.c
> create mode 100644 sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c
> create mode 100644 sysdeps/powerpc/powerpc64/le/tst-glibc-hwcaps.c
> create mode 100644 sysdeps/x86_64/dl-hwcaps-subdirs.c
> create mode 100644 sysdeps/x86_64/tst-glibc-hwcaps.c
>
> --
> Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
>
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH v4 00/11] glibc-hwcaps support
2020-11-09 21:56 ` [PATCH v4 00/11] " Dan Horák
@ 2020-11-10 10:41 ` Florian Weimer
0 siblings, 0 replies; 58+ messages in thread
From: Florian Weimer @ 2020-11-10 10:41 UTC (permalink / raw)
To: Dan Horák; +Cc: libc-alpha
* Dan Horák:
>> I have dropped the s390x patch until I have located a suitable machine
>> on which to test it. We also need to discuss how to achieve GCC
>> alignment there.
>
> Florian, what kind of machine do you need?
z15 with unrestricted user namespaces. z/VM vs KVM should not matter
(but it would be nice to test with a typical z/VM setup too).
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 00/11] glibc-hwcaps support
2020-11-09 18:40 [PATCH v4 00/11] glibc-hwcaps support Florian Weimer
` (11 preceding siblings ...)
2020-11-09 21:56 ` [PATCH v4 00/11] " Dan Horák
@ 2020-11-25 15:46 ` Florian Weimer
2020-11-25 15:58 ` H.J. Lu
2020-11-25 17:20 ` Adhemerval Zanella
12 siblings, 2 replies; 58+ messages in thread
From: Florian Weimer @ 2020-11-25 15:46 UTC (permalink / raw)
To: Florian Weimer via Libc-alpha
* Florian Weimer via Libc-alpha:
> Florian Weimer (11):
> support: Add support_copy_file
> elf: Introduce enum opt_format in the ldconfig implementation
> elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
> elf: Add endianness markup to ld.so.cache
> elf: Add extension mechanism to ld.so.cache
> elf: Implement a string table for ldconfig, with tail merging
> elf: Implement tail merging of strings in ldconfig
> elf: Process glibc-hwcaps subdirectories in ldconfig
> elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
> x86_64: Add glibc-hwcaps support
> powerpc64le: Add glibc-hwcaps support
Ping. I'd like to move this forward.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH v4 00/11] glibc-hwcaps support
2020-11-25 15:46 ` Florian Weimer
@ 2020-11-25 15:58 ` H.J. Lu
2020-11-25 17:20 ` Adhemerval Zanella
1 sibling, 0 replies; 58+ messages in thread
From: H.J. Lu @ 2020-11-25 15:58 UTC (permalink / raw)
To: Florian Weimer; +Cc: Florian Weimer via Libc-alpha
On Wed, Nov 25, 2020 at 7:47 AM Florian Weimer via Libc-alpha
<libc-alpha@sourceware.org> wrote:
>
> * Florian Weimer via Libc-alpha:
>
> > Florian Weimer (11):
> > support: Add support_copy_file
> > elf: Introduce enum opt_format in the ldconfig implementation
> > elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
> > elf: Add endianness markup to ld.so.cache
> > elf: Add extension mechanism to ld.so.cache
> > elf: Implement a string table for ldconfig, with tail merging
> > elf: Implement tail merging of strings in ldconfig
> > elf: Process glibc-hwcaps subdirectories in ldconfig
> > elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
> > x86_64: Add glibc-hwcaps support
> > powerpc64le: Add glibc-hwcaps support
>
> Ping. I'd like to move this forward.
>
x86_64 bits look good to me.
Thanks.
--
H.J.
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: [PATCH v4 00/11] glibc-hwcaps support
2020-11-25 15:46 ` Florian Weimer
2020-11-25 15:58 ` H.J. Lu
@ 2020-11-25 17:20 ` Adhemerval Zanella
1 sibling, 0 replies; 58+ messages in thread
From: Adhemerval Zanella @ 2020-11-25 17:20 UTC (permalink / raw)
To: libc-alpha
On 25/11/2020 12:46, Florian Weimer via Libc-alpha wrote:
> * Florian Weimer via Libc-alpha:
>
>> Florian Weimer (11):
>> support: Add support_copy_file
>> elf: Introduce enum opt_format in the ldconfig implementation
>> elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
>> elf: Add endianness markup to ld.so.cache
>> elf: Add extension mechanism to ld.so.cache
>> elf: Implement a string table for ldconfig, with tail merging
>> elf: Implement tail merging of strings in ldconfig
>> elf: Process glibc-hwcaps subdirectories in ldconfig
>> elf: Add glibc-hwcaps subdirectory support to ld.so cache processing
>> x86_64: Add glibc-hwcaps support
>> powerpc64le: Add glibc-hwcaps support
>
> Ping. I'd like to move this forward.
I will try to review this fully until the end this week.
^ permalink raw reply [flat|nested] 58+ messages in thread