* [PATCH 0/2] Update Makefile fragments for DSO sorting tests
@ 2021-11-07 15:07 H.J. Lu
2021-11-07 15:07 ` [PATCH 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu
2021-11-07 15:07 ` [PATCH 2/2] elf: Pre-generate Makefile fragments for DSO sorting tests " H.J. Lu
0 siblings, 2 replies; 5+ messages in thread
From: H.J. Lu @ 2021-11-07 15:07 UTC (permalink / raw)
To: libc-alpha
One problem of using
$(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c)
compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags)
compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS)
to build DSO sorting tests is that $< may not be the source file with
3 "make -j 28" parallel builds on a machine with 112 cores. When
generating Makefile fragments at build time, the same file may be
generated more than once at the same time with parallel builds.
1. Update dso-ordering-test.py to put all sources for DSO sorting tests
in a single directory, dso-sort-tests-src, and compile tests with
"$(compile.c) $(OUTPUT_OPTION)".
2. Generate Makefile fragments for DSO sorting tests at configure time
to avoid generate them in the elf directory during build.
H.J. Lu (2):
dso-ordering-test.py: Put all sources in one directory [BZ #28550]
elf: Pre-generate Makefile fragments for DSO sorting tests [BZ #28550]
configure | 11 +++++++++
configure.ac | 12 +++++++++
elf/Makefile | 22 ++++-------------
scripts/dso-ordering-test.py | 47 ++++++++++++++++++++++++++----------
4 files changed, 62 insertions(+), 30 deletions(-)
--
2.33.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550]
2021-11-07 15:07 [PATCH 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu
@ 2021-11-07 15:07 ` H.J. Lu
2021-11-08 13:46 ` Florian Weimer
2021-11-07 15:07 ` [PATCH 2/2] elf: Pre-generate Makefile fragments for DSO sorting tests " H.J. Lu
1 sibling, 1 reply; 5+ messages in thread
From: H.J. Lu @ 2021-11-07 15:07 UTC (permalink / raw)
To: libc-alpha
Put all sources for DSO sorting tests in the dso-sort-tests-src directory
and compile tests with "$(compile.c) $(OUTPUT_OPTION)" to avoid random
$< values from $(before-compile) when compiling tests with
$(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c)
compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags)
compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS)
for 3 "make -j 28" parallel builds on a machine with 112 cores at the
same time.
Also use a temporary file to generate Makefile fragments for DSO sorting
tests.
This fixes BZ #28550.
---
scripts/dso-ordering-test.py | 47 ++++++++++++++++++++++++++----------
1 file changed, 34 insertions(+), 13 deletions(-)
diff --git a/scripts/dso-ordering-test.py b/scripts/dso-ordering-test.py
index 944ee74052..26497159a0 100644
--- a/scripts/dso-ordering-test.py
+++ b/scripts/dso-ordering-test.py
@@ -526,9 +526,13 @@ def process_testcase(t):
base_test_name = t.test_name
test_subdir = base_test_name + "-dir"
testpfx = objpfx + test_subdir + "/"
+ test_srcdir = "dso-sort-tests-src/"
+ testpfx_src = objpfx + test_srcdir
if not os.path.exists(testpfx):
os.mkdir(testpfx)
+ if not os.path.exists(testpfx_src):
+ os.mkdir(testpfx_src)
def find_objs_not_depended_on(t):
objs_not_depended_on = []
@@ -595,6 +599,11 @@ def process_testcase(t):
# Print out needed Makefile fragments for use in glibc/elf/Makefile.
module_names = ""
for o in test_descr.objs:
+ rule = ("$(objpfx)" + test_subdir + "/" + test_name
+ + "-" + o + ".os: $(objpfx)" + test_srcdir
+ + test_name + "-" + o + ".c\n"
+ "\t$(compile.c) $(OUTPUT_OPTION)\n")
+ makefile.write (rule);
module_names += " " + test_subdir + "/" + test_name + "-" + o
makefile.write("modules-names +=%s\n" % (module_names))
@@ -637,7 +646,7 @@ def process_testcase(t):
# object. This only needs to be done at most once for
# an object name.
if not dep in fake_created:
- f = open(testpfx + test_name + "-" + dep
+ f = open(testpfx_src + test_name + "-" + dep
+ ".FAKE.c", "w")
f.write(" \n")
f.close()
@@ -648,6 +657,12 @@ def process_testcase(t):
% (test_name + "-" + dep + ".FAKE.so",
("$(objpfx)" + test_subdir + "/"
+ test_name + "-" + dep + ".so")))
+ rule = ("$(objpfx)" + test_subdir + "/"
+ + test_name + "-" + dep + ".FAKE.os: "
+ "$(objpfx)" + test_srcdir
+ + test_name + "-" + dep + ".FAKE.c\n"
+ "\t$(compile.c) $(OUTPUT_OPTION)\n")
+ makefile.write (rule);
makefile.write \
("modules-names += %s\n"
% (test_subdir + "/"
@@ -687,6 +702,10 @@ def process_testcase(t):
+ test_descr.soname_map['#'] + ".so")
ldflags += (" -Wl,-soname=" + soname)
makefile.write("LDFLAGS-%s = %s\n" % (test_name, ldflags))
+ rule = ("$(objpfx)" + test_subdir + "/" + test_name + ".o: "
+ "$(objpfx)" + test_srcdir + test_name + ".c\n"
+ "\t$(compile.c) $(OUTPUT_OPTION)\n")
+ makefile.write (rule);
not_depended_objs = find_objs_not_depended_on(test_descr)
if not_depended_objs:
@@ -745,7 +764,7 @@ def process_testcase(t):
" something_failed=true\n"
"else\n"
" diff -wu ${common_objpfx}elf/%s/%s%s.output \\\n"
- " ${common_objpfx}elf/%s/%s%s.exp\n"
+ " ${common_objpfx}elf/%s%s%s.exp\n"
" if [ $? -ne 0 ]; then\n"
" echo '%sFAIL: %s%s expected output comparison'\n"
" something_failed=true\n"
@@ -753,14 +772,14 @@ def process_testcase(t):
"fi\n"
% (("X" if xfail else ""), test_name, tunable_descr,
test_subdir, test_name, tunable_sfx,
- test_subdir, base_test_name, exp_tunable_sfx,
+ test_srcdir, base_test_name, exp_tunable_sfx,
("X" if xfail else ""), test_name, tunable_descr))
# Generate C files according to dependency and calling relations from
# description string.
for obj in test_descr.objs:
src_name = test_name + "-" + obj + ".c"
- f = open(testpfx + src_name, "w")
+ f = open(testpfx_src + src_name, "w")
if obj in test_descr.callrefs:
called_objs = test_descr.callrefs[obj]
for callee in called_objs:
@@ -804,7 +823,7 @@ def process_testcase(t):
f.close()
# Open C file for writing main program
- f = open(testpfx + test_name + ".c", "w")
+ f = open(testpfx_src + test_name + ".c", "w")
# if there are some operations in main(), it means we need -ldl
f.write("#include <stdio.h>\n")
@@ -885,7 +904,7 @@ def process_testcase(t):
for obj in test_descr.objs:
src_name = test_name + "-" + obj + ".c"
obj_name = test_name + "-" + obj + ".os"
- run_cmd([build_gcc, "-c", "-fPIC", testpfx + src_name,
+ run_cmd([build_gcc, "-c", "-fPIC", testpfx_src + src_name,
"-o", testpfx + obj_name])
obj_processed = {}
@@ -903,10 +922,12 @@ def process_testcase(t):
deps.append(dep + ".FAKE")
if not dep in fake_created:
base_name = testpfx + test_name + "-" + dep
+ src_base_name = (testpfx_src + test_name
+ + "-" + dep)
cmd = [build_gcc, "-Wl,--no-as-needed",
("-Wl,-soname=" + base_name + ".so"),
"-shared", base_name + ".FAKE.c",
- "-o", base_name + ".FAKE.so"]
+ "-o", src_base_name + ".FAKE.so"]
run_cmd(cmd)
fake_created[dep] = True
dso_deps = map(lambda d: testpfx + test_name + "-" + d + ".so",
@@ -932,7 +953,7 @@ def process_testcase(t):
main_deps = map(lambda d: testpfx + test_name + "-" + d + ".so",
deps)
cmd = [build_gcc, "-Wl,--no-as-needed", "-o", testpfx + test_name,
- testpfx + test_name + ".c", "-L%s" % (os.getcwd()),
+ testpfx_src + test_name + ".c", "-L%s" % (os.getcwd()),
"-Wl,-rpath-link=%s" % (os.getcwd())]
if '#' in test_descr.soname_map:
soname = ("-Wl,-soname=" + testpfx + test_name + "-"
@@ -987,14 +1008,14 @@ def process_testcase(t):
sfx = ""
if r[0] != "":
sfx = "-" + r[0].replace("=","_")
- f = open(testpfx + t.test_name + sfx + ".exp", "w")
+ f = open(testpfx_src + t.test_name + sfx + ".exp", "w")
(output, xfail) = r[1]
f.write('%s' % output)
f.close()
# Create header part of top-level testcase shell script, to wrap execution
# and output comparison together.
- t.sh = open(testpfx + t.test_name + ".sh", "w")
+ t.sh = open(testpfx_src + t.test_name + ".sh", "w")
t.sh.write("#!/bin/sh\n")
t.sh.write("# Test driver for %s, generated by "
"dso-ordering-test.py\n" % (t.test_name))
@@ -1022,12 +1043,12 @@ def process_testcase(t):
sfx = ""
if r[0] != "":
sfx = "-" + r[0].replace("=","_")
- expected_output_files += " $(objpfx)%s/%s%s.exp" % (test_subdir,
+ expected_output_files += " $(objpfx)%s%s%s.exp" % (test_srcdir,
t.test_name, sfx)
makefile.write \
- ("$(objpfx)%s.out: $(objpfx)%s/%s.sh%s "
+ ("$(objpfx)%s.out: $(objpfx)%s%s.sh%s "
"$(common-objpfx)support/test-run-command\n"
- % (t.test_name, test_subdir, t.test_name,
+ % (t.test_name, test_srcdir, t.test_name,
expected_output_files))
makefile.write("\t$(SHELL) $< $(common-objpfx) '$(test-wrapper-env)' "
"'$(run-program-env)' > $@; $(evaluate-test)\n")
--
2.33.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 2/2] elf: Pre-generate Makefile fragments for DSO sorting tests [BZ #28550]
2021-11-07 15:07 [PATCH 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu
2021-11-07 15:07 ` [PATCH 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu
@ 2021-11-07 15:07 ` H.J. Lu
1 sibling, 0 replies; 5+ messages in thread
From: H.J. Lu @ 2021-11-07 15:07 UTC (permalink / raw)
To: libc-alpha
Generate Makefile fragments for DSO sorting tests at configure time to
avoid generate them in the elf directory during build.
This fixes BZ #28550.
---
configure | 11 +++++++++++
configure.ac | 12 ++++++++++++
elf/Makefile | 22 +++++-----------------
3 files changed, 28 insertions(+), 17 deletions(-)
diff --git a/configure b/configure
index 2f9adca064..91993da3b9 100755
--- a/configure
+++ b/configure
@@ -8484,3 +8484,14 @@ if test -n "$ac_unrecognized_opts" && test "$enable_option_checking" != no; then
$as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2;}
fi
+
+if test "$have_tunables" = yes; then
+ test -d elf || mkdir elf
+ objdir=`pwd`
+ rm -f elf/dso-sort-tests.mk
+ (for d in dso-sort-tests-1.def dso-sort-tests-2.def
+ do
+ $PYTHON $srcdir/scripts/dso-ordering-test.py \
+ --description-file $srcdir/elf/$d --objpfx $objdir/elf/
+ done) > elf/dso-sort-tests.mk
+fi
diff --git a/configure.ac b/configure.ac
index 7eb4239359..faf70b930c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1901,3 +1901,15 @@ echo "$config_vars" >> config.make;;
esac
test -d bits || mkdir bits]],[[config_vars='$config_vars']])
AC_OUTPUT
+
+dnl Generate Makefile fragments for DSO sorting tests.
+if test "$have_tunables" = yes; then
+ test -d elf || mkdir elf
+ objdir=`pwd`
+ rm -f elf/dso-sort-tests.mk
+ (for d in dso-sort-tests-1.def dso-sort-tests-2.def
+ do
+ $PYTHON $srcdir/scripts/dso-ordering-test.py \
+ --description-file $srcdir/elf/$d --objpfx $objdir/elf/
+ done) > elf/dso-sort-tests.mk
+fi
diff --git a/elf/Makefile b/elf/Makefile
index 41c19668c3..6995ccaa0f 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -366,6 +366,11 @@ modules-names = testobj1 testobj2 testobj3 testobj4 testobj5 testobj6 \
tst-auxvalmod \
tst-dlmopen-gethostbyname-mod tst-ro-dynamic-mod \
+ifeq (yes,$(have-tunables))
+# Include Makefile fragments for DSO sorting tests.
+include $(objpfx)dso-sort-tests.mk
+endif
+
# Most modules build with _ISOMAC defined, but those filtered out
# depend on internal headers.
modules-names-tests = $(filter-out ifuncmod% tst-tlsmod%,\
@@ -486,23 +491,6 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \
$(objpfx)tst-unused-dep-cmp.out
endif
-# DSO sorting tests:
-# The dso-ordering-test.py script generates testcase source files in $(objpfx),
-# creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a
-# Makefile fragment to be included.
-define include_dsosort_tests
-$(objpfx)$(1).generated-makefile: $(1)
- $(PYTHON) $(..)scripts/dso-ordering-test.py \
- --description-file $$< --objpfx $(objpfx) --output-makefile $$@
-include $(objpfx)$(1).generated-makefile
-endef
-
-# Generate from each testcase description file
-ifeq (yes,$(have-tunables))
-$(eval $(call include_dsosort_tests,dso-sort-tests-1.def))
-$(eval $(call include_dsosort_tests,dso-sort-tests-2.def))
-endif
-
check-abi: $(objpfx)check-abi-ld.out
tests-special += $(objpfx)check-abi-ld.out
update-abi: update-abi-ld
--
2.33.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550]
2021-11-07 15:07 ` [PATCH 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu
@ 2021-11-08 13:46 ` Florian Weimer
2021-11-08 14:07 ` H.J. Lu
0 siblings, 1 reply; 5+ messages in thread
From: Florian Weimer @ 2021-11-08 13:46 UTC (permalink / raw)
To: H.J. Lu via Libc-alpha
* H. J. Lu via Libc-alpha:
> Also use a temporary file to generate Makefile fragments for DSO sorting
> tests.
I don't see this part in the patch?
> diff --git a/scripts/dso-ordering-test.py b/scripts/dso-ordering-test.py
> index 944ee74052..26497159a0 100644
> --- a/scripts/dso-ordering-test.py
> +++ b/scripts/dso-ordering-test.py
> @@ -526,9 +526,13 @@ def process_testcase(t):
> base_test_name = t.test_name
> test_subdir = base_test_name + "-dir"
> testpfx = objpfx + test_subdir + "/"
> + test_srcdir = "dso-sort-tests-src/"
> + testpfx_src = objpfx + test_srcdir
>
> if not os.path.exists(testpfx):
> os.mkdir(testpfx)
> + if not os.path.exists(testpfx_src):
> + os.mkdir(testpfx_src)
>
> def find_objs_not_depended_on(t):
> objs_not_depended_on = []
> @@ -595,6 +599,11 @@ def process_testcase(t):
> # Print out needed Makefile fragments for use in glibc/elf/Makefile.
> module_names = ""
> for o in test_descr.objs:
> + rule = ("$(objpfx)" + test_subdir + "/" + test_name
> + + "-" + o + ".os: $(objpfx)" + test_srcdir
> + + test_name + "-" + o + ".c\n"
> + "\t$(compile.c) $(OUTPUT_OPTION)\n")
> + makefile.write (rule);
Spurious semicolon at end of line (there are multiple such cases).
Why do you put the generated test sources into a separate subdirectory?
Thanks,
Florian
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550]
2021-11-08 13:46 ` Florian Weimer
@ 2021-11-08 14:07 ` H.J. Lu
0 siblings, 0 replies; 5+ messages in thread
From: H.J. Lu @ 2021-11-08 14:07 UTC (permalink / raw)
To: Florian Weimer; +Cc: H.J. Lu via Libc-alpha
On Mon, Nov 8, 2021 at 5:46 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * H. J. Lu via Libc-alpha:
>
> > Also use a temporary file to generate Makefile fragments for DSO sorting
> > tests.
>
> I don't see this part in the patch?
This is the leftover from the old changes.
> > diff --git a/scripts/dso-ordering-test.py b/scripts/dso-ordering-test.py
> > index 944ee74052..26497159a0 100644
> > --- a/scripts/dso-ordering-test.py
> > +++ b/scripts/dso-ordering-test.py
> > @@ -526,9 +526,13 @@ def process_testcase(t):
> > base_test_name = t.test_name
> > test_subdir = base_test_name + "-dir"
> > testpfx = objpfx + test_subdir + "/"
> > + test_srcdir = "dso-sort-tests-src/"
> > + testpfx_src = objpfx + test_srcdir
> >
> > if not os.path.exists(testpfx):
> > os.mkdir(testpfx)
> > + if not os.path.exists(testpfx_src):
> > + os.mkdir(testpfx_src)
> >
> > def find_objs_not_depended_on(t):
> > objs_not_depended_on = []
> > @@ -595,6 +599,11 @@ def process_testcase(t):
> > # Print out needed Makefile fragments for use in glibc/elf/Makefile.
> > module_names = ""
> > for o in test_descr.objs:
> > + rule = ("$(objpfx)" + test_subdir + "/" + test_name
> > + + "-" + o + ".os: $(objpfx)" + test_srcdir
> > + + test_name + "-" + o + ".c\n"
> > + "\t$(compile.c) $(OUTPUT_OPTION)\n")
> > + makefile.write (rule);
>
> Spurious semicolon at end of line (there are multiple such cases).
I will remove them.
> Why do you put the generated test sources into a separate subdirectory?
>
To avoid the default make rule:
$(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c)
which causes the problem for parallel build.
--
H.J.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-11-08 14:08 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-07 15:07 [PATCH 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu
2021-11-07 15:07 ` [PATCH 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu
2021-11-08 13:46 ` Florian Weimer
2021-11-08 14:07 ` H.J. Lu
2021-11-07 15:07 ` [PATCH 2/2] elf: Pre-generate Makefile fragments for DSO sorting tests " H.J. Lu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).