* [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests @ 2021-11-08 15:42 H.J. Lu 2021-11-08 15:42 ` [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu ` (2 more replies) 0 siblings, 3 replies; 15+ messages in thread From: H.J. Lu @ 2021-11-08 15:42 UTC (permalink / raw) To: libc-alpha; +Cc: Florian Weimer One problem of using $(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c) compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags) compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS) to build DSO sorting relocatable objects is that $< may not be the source file with 3 "make -j 28" parallel builds on a machine with 112 cores. When generating Makefile fragments at build time, the same file may be generated more than once at the same time with parallel builds. 1. Update dso-ordering-test.py to put all sources for DSO sorting tests in a single directory, dso-sort-tests-src, and compile relocatable objects with $(objpfx)tst-dso-ordering1-dir/tst-dso-ordering1-a.os: $(objpfx)dso-sort-tests-src/tst-dso-ordering1-a.c $(compile.c) $(OUTPUT_OPTION) 2. Generate Makefile fragments for DSO sorting tests at configure time to avoid generate them in the elf directory at build time. H.J. Lu (2): dso-ordering-test.py: Put all sources in one directory [BZ #28550] elf: Pre-generate Makefile fragment for DSO sorting tests [BZ #28550] configure | 11 +++++++++ configure.ac | 12 +++++++++ elf/Makefile | 22 ++++------------- scripts/dso-ordering-test.py | 47 ++++++++++++++++++++++++++---------- 4 files changed, 62 insertions(+), 30 deletions(-) -- 2.33.1 ^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] 2021-11-08 15:42 [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu @ 2021-11-08 15:42 ` H.J. Lu 2021-11-11 17:52 ` Chung-Lin Tang 2021-11-15 18:50 ` Adhemerval Zanella 2021-11-08 15:42 ` [PATCH v2 2/2] elf: Pre-generate Makefile fragment for DSO sorting tests " H.J. Lu 2021-11-11 13:46 ` [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu 2 siblings, 2 replies; 15+ messages in thread From: H.J. Lu @ 2021-11-08 15:42 UTC (permalink / raw) To: libc-alpha; +Cc: Florian Weimer Put all sources for DSO sorting tests in the dso-sort-tests-src directory and compile test relocatable objects with $(objpfx)tst-dso-ordering1-dir/tst-dso-ordering1-a.os: $(objpfx)dso-sort-tests-src/tst-dso-ordering1-a.c $(compile.c) $(OUTPUT_OPTION) to avoid random $< values from $(before-compile) when compiling test relocatable objects with $(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c) compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags) compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS) for 3 "make -j 28" parallel builds on a machine with 112 cores at the same time. This partially fixes BZ #28550. --- scripts/dso-ordering-test.py | 47 ++++++++++++++++++++++++++---------- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/scripts/dso-ordering-test.py b/scripts/dso-ordering-test.py index 944ee74052..bde0406be9 100644 --- a/scripts/dso-ordering-test.py +++ b/scripts/dso-ordering-test.py @@ -526,9 +526,13 @@ def process_testcase(t): base_test_name = t.test_name test_subdir = base_test_name + "-dir" testpfx = objpfx + test_subdir + "/" + test_srcdir = "dso-sort-tests-src/" + testpfx_src = objpfx + test_srcdir if not os.path.exists(testpfx): os.mkdir(testpfx) + if not os.path.exists(testpfx_src): + os.mkdir(testpfx_src) def find_objs_not_depended_on(t): objs_not_depended_on = [] @@ -595,6 +599,11 @@ def process_testcase(t): # Print out needed Makefile fragments for use in glibc/elf/Makefile. module_names = "" for o in test_descr.objs: + rule = ("$(objpfx)" + test_subdir + "/" + test_name + + "-" + o + ".os: $(objpfx)" + test_srcdir + + test_name + "-" + o + ".c\n" + "\t$(compile.c) $(OUTPUT_OPTION)\n") + makefile.write (rule) module_names += " " + test_subdir + "/" + test_name + "-" + o makefile.write("modules-names +=%s\n" % (module_names)) @@ -637,7 +646,7 @@ def process_testcase(t): # object. This only needs to be done at most once for # an object name. if not dep in fake_created: - f = open(testpfx + test_name + "-" + dep + f = open(testpfx_src + test_name + "-" + dep + ".FAKE.c", "w") f.write(" \n") f.close() @@ -648,6 +657,12 @@ def process_testcase(t): % (test_name + "-" + dep + ".FAKE.so", ("$(objpfx)" + test_subdir + "/" + test_name + "-" + dep + ".so"))) + rule = ("$(objpfx)" + test_subdir + "/" + + test_name + "-" + dep + ".FAKE.os: " + "$(objpfx)" + test_srcdir + + test_name + "-" + dep + ".FAKE.c\n" + "\t$(compile.c) $(OUTPUT_OPTION)\n") + makefile.write (rule) makefile.write \ ("modules-names += %s\n" % (test_subdir + "/" @@ -687,6 +702,10 @@ def process_testcase(t): + test_descr.soname_map['#'] + ".so") ldflags += (" -Wl,-soname=" + soname) makefile.write("LDFLAGS-%s = %s\n" % (test_name, ldflags)) + rule = ("$(objpfx)" + test_subdir + "/" + test_name + ".o: " + "$(objpfx)" + test_srcdir + test_name + ".c\n" + "\t$(compile.c) $(OUTPUT_OPTION)\n") + makefile.write (rule) not_depended_objs = find_objs_not_depended_on(test_descr) if not_depended_objs: @@ -745,7 +764,7 @@ def process_testcase(t): " something_failed=true\n" "else\n" " diff -wu ${common_objpfx}elf/%s/%s%s.output \\\n" - " ${common_objpfx}elf/%s/%s%s.exp\n" + " ${common_objpfx}elf/%s%s%s.exp\n" " if [ $? -ne 0 ]; then\n" " echo '%sFAIL: %s%s expected output comparison'\n" " something_failed=true\n" @@ -753,14 +772,14 @@ def process_testcase(t): "fi\n" % (("X" if xfail else ""), test_name, tunable_descr, test_subdir, test_name, tunable_sfx, - test_subdir, base_test_name, exp_tunable_sfx, + test_srcdir, base_test_name, exp_tunable_sfx, ("X" if xfail else ""), test_name, tunable_descr)) # Generate C files according to dependency and calling relations from # description string. for obj in test_descr.objs: src_name = test_name + "-" + obj + ".c" - f = open(testpfx + src_name, "w") + f = open(testpfx_src + src_name, "w") if obj in test_descr.callrefs: called_objs = test_descr.callrefs[obj] for callee in called_objs: @@ -804,7 +823,7 @@ def process_testcase(t): f.close() # Open C file for writing main program - f = open(testpfx + test_name + ".c", "w") + f = open(testpfx_src + test_name + ".c", "w") # if there are some operations in main(), it means we need -ldl f.write("#include <stdio.h>\n") @@ -885,7 +904,7 @@ def process_testcase(t): for obj in test_descr.objs: src_name = test_name + "-" + obj + ".c" obj_name = test_name + "-" + obj + ".os" - run_cmd([build_gcc, "-c", "-fPIC", testpfx + src_name, + run_cmd([build_gcc, "-c", "-fPIC", testpfx_src + src_name, "-o", testpfx + obj_name]) obj_processed = {} @@ -903,10 +922,12 @@ def process_testcase(t): deps.append(dep + ".FAKE") if not dep in fake_created: base_name = testpfx + test_name + "-" + dep + src_base_name = (testpfx_src + test_name + + "-" + dep) cmd = [build_gcc, "-Wl,--no-as-needed", ("-Wl,-soname=" + base_name + ".so"), "-shared", base_name + ".FAKE.c", - "-o", base_name + ".FAKE.so"] + "-o", src_base_name + ".FAKE.so"] run_cmd(cmd) fake_created[dep] = True dso_deps = map(lambda d: testpfx + test_name + "-" + d + ".so", @@ -932,7 +953,7 @@ def process_testcase(t): main_deps = map(lambda d: testpfx + test_name + "-" + d + ".so", deps) cmd = [build_gcc, "-Wl,--no-as-needed", "-o", testpfx + test_name, - testpfx + test_name + ".c", "-L%s" % (os.getcwd()), + testpfx_src + test_name + ".c", "-L%s" % (os.getcwd()), "-Wl,-rpath-link=%s" % (os.getcwd())] if '#' in test_descr.soname_map: soname = ("-Wl,-soname=" + testpfx + test_name + "-" @@ -987,14 +1008,14 @@ def process_testcase(t): sfx = "" if r[0] != "": sfx = "-" + r[0].replace("=","_") - f = open(testpfx + t.test_name + sfx + ".exp", "w") + f = open(testpfx_src + t.test_name + sfx + ".exp", "w") (output, xfail) = r[1] f.write('%s' % output) f.close() # Create header part of top-level testcase shell script, to wrap execution # and output comparison together. - t.sh = open(testpfx + t.test_name + ".sh", "w") + t.sh = open(testpfx_src + t.test_name + ".sh", "w") t.sh.write("#!/bin/sh\n") t.sh.write("# Test driver for %s, generated by " "dso-ordering-test.py\n" % (t.test_name)) @@ -1022,12 +1043,12 @@ def process_testcase(t): sfx = "" if r[0] != "": sfx = "-" + r[0].replace("=","_") - expected_output_files += " $(objpfx)%s/%s%s.exp" % (test_subdir, + expected_output_files += " $(objpfx)%s%s%s.exp" % (test_srcdir, t.test_name, sfx) makefile.write \ - ("$(objpfx)%s.out: $(objpfx)%s/%s.sh%s " + ("$(objpfx)%s.out: $(objpfx)%s%s.sh%s " "$(common-objpfx)support/test-run-command\n" - % (t.test_name, test_subdir, t.test_name, + % (t.test_name, test_srcdir, t.test_name, expected_output_files)) makefile.write("\t$(SHELL) $< $(common-objpfx) '$(test-wrapper-env)' " "'$(run-program-env)' > $@; $(evaluate-test)\n") -- 2.33.1 ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] 2021-11-08 15:42 ` [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu @ 2021-11-11 17:52 ` Chung-Lin Tang 2021-11-11 18:21 ` H.J. Lu 2021-11-15 18:50 ` Adhemerval Zanella 1 sibling, 1 reply; 15+ messages in thread From: Chung-Lin Tang @ 2021-11-11 17:52 UTC (permalink / raw) To: H.J. Lu, libc-alpha; +Cc: Florian Weimer On 2021/11/8 11:42 PM, H.J. Lu via Libc-alpha wrote: > Put all sources for DSO sorting tests in the dso-sort-tests-src directory > and compile test relocatable objects with > > $(objpfx)tst-dso-ordering1-dir/tst-dso-ordering1-a.os: $(objpfx)dso-sort-tests-src/tst-dso-ordering1-a.c > $(compile.c) $(OUTPUT_OPTION) > > to avoid random $< values from $(before-compile) when compiling test > relocatable objects with > > $(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c) > compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags) > compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS) I kind of see what you mean, but it appears that the explicit compile rule is the main required addition? If so, why the directory change to "dso-sort-tests-src"? (it kind of adds more complexity to the already somewhat complex placement of files, at least to someone new to the setup) Chung-Lin > for 3 "make -j 28" parallel builds on a machine with 112 cores at the > same time. > > This partially fixes BZ #28550. > --- > scripts/dso-ordering-test.py | 47 ++++++++++++++++++++++++++---------- > 1 file changed, 34 insertions(+), 13 deletions(-) > > diff --git a/scripts/dso-ordering-test.py b/scripts/dso-ordering-test.py > index 944ee74052..bde0406be9 100644 > --- a/scripts/dso-ordering-test.py > +++ b/scripts/dso-ordering-test.py > @@ -526,9 +526,13 @@ def process_testcase(t): > base_test_name = t.test_name > test_subdir = base_test_name + "-dir" > testpfx = objpfx + test_subdir + "/" > + test_srcdir = "dso-sort-tests-src/" > + testpfx_src = objpfx + test_srcdir > > if not os.path.exists(testpfx): > os.mkdir(testpfx) > + if not os.path.exists(testpfx_src): > + os.mkdir(testpfx_src) > > def find_objs_not_depended_on(t): > objs_not_depended_on = [] > @@ -595,6 +599,11 @@ def process_testcase(t): > # Print out needed Makefile fragments for use in glibc/elf/Makefile. > module_names = "" > for o in test_descr.objs: > + rule = ("$(objpfx)" + test_subdir + "/" + test_name > + + "-" + o + ".os: $(objpfx)" + test_srcdir > + + test_name + "-" + o + ".c\n" > + "\t$(compile.c) $(OUTPUT_OPTION)\n") > + makefile.write (rule) > module_names += " " + test_subdir + "/" + test_name + "-" + o > makefile.write("modules-names +=%s\n" % (module_names)) > > @@ -637,7 +646,7 @@ def process_testcase(t): > # object. This only needs to be done at most once for > # an object name. > if not dep in fake_created: > - f = open(testpfx + test_name + "-" + dep > + f = open(testpfx_src + test_name + "-" + dep > + ".FAKE.c", "w") > f.write(" \n") > f.close() > @@ -648,6 +657,12 @@ def process_testcase(t): > % (test_name + "-" + dep + ".FAKE.so", > ("$(objpfx)" + test_subdir + "/" > + test_name + "-" + dep + ".so"))) > + rule = ("$(objpfx)" + test_subdir + "/" > + + test_name + "-" + dep + ".FAKE.os: " > + "$(objpfx)" + test_srcdir > + + test_name + "-" + dep + ".FAKE.c\n" > + "\t$(compile.c) $(OUTPUT_OPTION)\n") > + makefile.write (rule) > makefile.write \ > ("modules-names += %s\n" > % (test_subdir + "/" > @@ -687,6 +702,10 @@ def process_testcase(t): > + test_descr.soname_map['#'] + ".so") > ldflags += (" -Wl,-soname=" + soname) > makefile.write("LDFLAGS-%s = %s\n" % (test_name, ldflags)) > + rule = ("$(objpfx)" + test_subdir + "/" + test_name + ".o: " > + "$(objpfx)" + test_srcdir + test_name + ".c\n" > + "\t$(compile.c) $(OUTPUT_OPTION)\n") > + makefile.write (rule) > > not_depended_objs = find_objs_not_depended_on(test_descr) > if not_depended_objs: > @@ -745,7 +764,7 @@ def process_testcase(t): > " something_failed=true\n" > "else\n" > " diff -wu ${common_objpfx}elf/%s/%s%s.output \\\n" > - " ${common_objpfx}elf/%s/%s%s.exp\n" > + " ${common_objpfx}elf/%s%s%s.exp\n" > " if [ $? -ne 0 ]; then\n" > " echo '%sFAIL: %s%s expected output comparison'\n" > " something_failed=true\n" > @@ -753,14 +772,14 @@ def process_testcase(t): > "fi\n" > % (("X" if xfail else ""), test_name, tunable_descr, > test_subdir, test_name, tunable_sfx, > - test_subdir, base_test_name, exp_tunable_sfx, > + test_srcdir, base_test_name, exp_tunable_sfx, > ("X" if xfail else ""), test_name, tunable_descr)) > > # Generate C files according to dependency and calling relations from > # description string. > for obj in test_descr.objs: > src_name = test_name + "-" + obj + ".c" > - f = open(testpfx + src_name, "w") > + f = open(testpfx_src + src_name, "w") > if obj in test_descr.callrefs: > called_objs = test_descr.callrefs[obj] > for callee in called_objs: > @@ -804,7 +823,7 @@ def process_testcase(t): > f.close() > > # Open C file for writing main program > - f = open(testpfx + test_name + ".c", "w") > + f = open(testpfx_src + test_name + ".c", "w") > > # if there are some operations in main(), it means we need -ldl > f.write("#include <stdio.h>\n") > @@ -885,7 +904,7 @@ def process_testcase(t): > for obj in test_descr.objs: > src_name = test_name + "-" + obj + ".c" > obj_name = test_name + "-" + obj + ".os" > - run_cmd([build_gcc, "-c", "-fPIC", testpfx + src_name, > + run_cmd([build_gcc, "-c", "-fPIC", testpfx_src + src_name, > "-o", testpfx + obj_name]) > > obj_processed = {} > @@ -903,10 +922,12 @@ def process_testcase(t): > deps.append(dep + ".FAKE") > if not dep in fake_created: > base_name = testpfx + test_name + "-" + dep > + src_base_name = (testpfx_src + test_name > + + "-" + dep) > cmd = [build_gcc, "-Wl,--no-as-needed", > ("-Wl,-soname=" + base_name + ".so"), > "-shared", base_name + ".FAKE.c", > - "-o", base_name + ".FAKE.so"] > + "-o", src_base_name + ".FAKE.so"] > run_cmd(cmd) > fake_created[dep] = True > dso_deps = map(lambda d: testpfx + test_name + "-" + d + ".so", > @@ -932,7 +953,7 @@ def process_testcase(t): > main_deps = map(lambda d: testpfx + test_name + "-" + d + ".so", > deps) > cmd = [build_gcc, "-Wl,--no-as-needed", "-o", testpfx + test_name, > - testpfx + test_name + ".c", "-L%s" % (os.getcwd()), > + testpfx_src + test_name + ".c", "-L%s" % (os.getcwd()), > "-Wl,-rpath-link=%s" % (os.getcwd())] > if '#' in test_descr.soname_map: > soname = ("-Wl,-soname=" + testpfx + test_name + "-" > @@ -987,14 +1008,14 @@ def process_testcase(t): > sfx = "" > if r[0] != "": > sfx = "-" + r[0].replace("=","_") > - f = open(testpfx + t.test_name + sfx + ".exp", "w") > + f = open(testpfx_src + t.test_name + sfx + ".exp", "w") > (output, xfail) = r[1] > f.write('%s' % output) > f.close() > > # Create header part of top-level testcase shell script, to wrap execution > # and output comparison together. > - t.sh = open(testpfx + t.test_name + ".sh", "w") > + t.sh = open(testpfx_src + t.test_name + ".sh", "w") > t.sh.write("#!/bin/sh\n") > t.sh.write("# Test driver for %s, generated by " > "dso-ordering-test.py\n" % (t.test_name)) > @@ -1022,12 +1043,12 @@ def process_testcase(t): > sfx = "" > if r[0] != "": > sfx = "-" + r[0].replace("=","_") > - expected_output_files += " $(objpfx)%s/%s%s.exp" % (test_subdir, > + expected_output_files += " $(objpfx)%s%s%s.exp" % (test_srcdir, > t.test_name, sfx) > makefile.write \ > - ("$(objpfx)%s.out: $(objpfx)%s/%s.sh%s " > + ("$(objpfx)%s.out: $(objpfx)%s%s.sh%s " > "$(common-objpfx)support/test-run-command\n" > - % (t.test_name, test_subdir, t.test_name, > + % (t.test_name, test_srcdir, t.test_name, > expected_output_files)) > makefile.write("\t$(SHELL) $< $(common-objpfx) '$(test-wrapper-env)' " > "'$(run-program-env)' > $@; $(evaluate-test)\n") > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] 2021-11-11 17:52 ` Chung-Lin Tang @ 2021-11-11 18:21 ` H.J. Lu 0 siblings, 0 replies; 15+ messages in thread From: H.J. Lu @ 2021-11-11 18:21 UTC (permalink / raw) To: Chung-Lin Tang; +Cc: GNU C Library, Florian Weimer On Thu, Nov 11, 2021 at 9:52 AM Chung-Lin Tang <cltang@codesourcery.com> wrote: > > > > On 2021/11/8 11:42 PM, H.J. Lu via Libc-alpha wrote: > > Put all sources for DSO sorting tests in the dso-sort-tests-src directory > > and compile test relocatable objects with > > > > $(objpfx)tst-dso-ordering1-dir/tst-dso-ordering1-a.os: $(objpfx)dso-sort-tests-src/tst-dso-ordering1-a.c > > $(compile.c) $(OUTPUT_OPTION) > > > > to avoid random $< values from $(before-compile) when compiling test > > relocatable objects with > > > > $(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c) > > compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags) > > compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS) > > I kind of see what you mean, but it appears that the explicit compile rule is the main required addition? > If so, why the directory change to "dso-sort-tests-src"? > (it kind of adds more complexity to the already somewhat complex placement of files, at least to someone new to the setup) > If the generated files are placed in the same directory where the object files are, they will match the default rule: $(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c) compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags) compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS) which is the root cause of the parallel build problem. -- H.J. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] 2021-11-08 15:42 ` [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu 2021-11-11 17:52 ` Chung-Lin Tang @ 2021-11-15 18:50 ` Adhemerval Zanella 1 sibling, 0 replies; 15+ messages in thread From: Adhemerval Zanella @ 2021-11-15 18:50 UTC (permalink / raw) To: libc-alpha, H.J. Lu On 08/11/2021 12:42, H.J. Lu via Libc-alpha wrote: > Put all sources for DSO sorting tests in the dso-sort-tests-src directory > and compile test relocatable objects with > > $(objpfx)tst-dso-ordering1-dir/tst-dso-ordering1-a.os: $(objpfx)dso-sort-tests-src/tst-dso-ordering1-a.c > $(compile.c) $(OUTPUT_OPTION) > > to avoid random $< values from $(before-compile) when compiling test > relocatable objects with > > $(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c) > compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags) > compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS) > > for 3 "make -j 28" parallel builds on a machine with 112 cores at the > same time. > > This partially fixes BZ #28550. LGTM, thanks. Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org> > --- > scripts/dso-ordering-test.py | 47 ++++++++++++++++++++++++++---------- > 1 file changed, 34 insertions(+), 13 deletions(-) > > diff --git a/scripts/dso-ordering-test.py b/scripts/dso-ordering-test.py > index 944ee74052..bde0406be9 100644 > --- a/scripts/dso-ordering-test.py > +++ b/scripts/dso-ordering-test.py > @@ -526,9 +526,13 @@ def process_testcase(t): > base_test_name = t.test_name > test_subdir = base_test_name + "-dir" > testpfx = objpfx + test_subdir + "/" > + test_srcdir = "dso-sort-tests-src/" > + testpfx_src = objpfx + test_srcdir > > if not os.path.exists(testpfx): > os.mkdir(testpfx) > + if not os.path.exists(testpfx_src): > + os.mkdir(testpfx_src) > > def find_objs_not_depended_on(t): > objs_not_depended_on = [] > @@ -595,6 +599,11 @@ def process_testcase(t): > # Print out needed Makefile fragments for use in glibc/elf/Makefile. > module_names = "" > for o in test_descr.objs: > + rule = ("$(objpfx)" + test_subdir + "/" + test_name > + + "-" + o + ".os: $(objpfx)" + test_srcdir > + + test_name + "-" + o + ".c\n" > + "\t$(compile.c) $(OUTPUT_OPTION)\n") > + makefile.write (rule) > module_names += " " + test_subdir + "/" + test_name + "-" + o > makefile.write("modules-names +=%s\n" % (module_names)) > > @@ -637,7 +646,7 @@ def process_testcase(t): > # object. This only needs to be done at most once for > # an object name. > if not dep in fake_created: > - f = open(testpfx + test_name + "-" + dep > + f = open(testpfx_src + test_name + "-" + dep > + ".FAKE.c", "w") > f.write(" \n") > f.close() > @@ -648,6 +657,12 @@ def process_testcase(t): > % (test_name + "-" + dep + ".FAKE.so", > ("$(objpfx)" + test_subdir + "/" > + test_name + "-" + dep + ".so"))) > + rule = ("$(objpfx)" + test_subdir + "/" > + + test_name + "-" + dep + ".FAKE.os: " > + "$(objpfx)" + test_srcdir > + + test_name + "-" + dep + ".FAKE.c\n" > + "\t$(compile.c) $(OUTPUT_OPTION)\n") > + makefile.write (rule) > makefile.write \ > ("modules-names += %s\n" > % (test_subdir + "/" > @@ -687,6 +702,10 @@ def process_testcase(t): > + test_descr.soname_map['#'] + ".so") > ldflags += (" -Wl,-soname=" + soname) > makefile.write("LDFLAGS-%s = %s\n" % (test_name, ldflags)) > + rule = ("$(objpfx)" + test_subdir + "/" + test_name + ".o: " > + "$(objpfx)" + test_srcdir + test_name + ".c\n" > + "\t$(compile.c) $(OUTPUT_OPTION)\n") > + makefile.write (rule) > > not_depended_objs = find_objs_not_depended_on(test_descr) > if not_depended_objs: > @@ -745,7 +764,7 @@ def process_testcase(t): > " something_failed=true\n" > "else\n" > " diff -wu ${common_objpfx}elf/%s/%s%s.output \\\n" > - " ${common_objpfx}elf/%s/%s%s.exp\n" > + " ${common_objpfx}elf/%s%s%s.exp\n" > " if [ $? -ne 0 ]; then\n" > " echo '%sFAIL: %s%s expected output comparison'\n" > " something_failed=true\n" > @@ -753,14 +772,14 @@ def process_testcase(t): > "fi\n" > % (("X" if xfail else ""), test_name, tunable_descr, > test_subdir, test_name, tunable_sfx, > - test_subdir, base_test_name, exp_tunable_sfx, > + test_srcdir, base_test_name, exp_tunable_sfx, > ("X" if xfail else ""), test_name, tunable_descr)) > > # Generate C files according to dependency and calling relations from > # description string. > for obj in test_descr.objs: > src_name = test_name + "-" + obj + ".c" > - f = open(testpfx + src_name, "w") > + f = open(testpfx_src + src_name, "w") > if obj in test_descr.callrefs: > called_objs = test_descr.callrefs[obj] > for callee in called_objs: > @@ -804,7 +823,7 @@ def process_testcase(t): > f.close() > > # Open C file for writing main program > - f = open(testpfx + test_name + ".c", "w") > + f = open(testpfx_src + test_name + ".c", "w") > > # if there are some operations in main(), it means we need -ldl > f.write("#include <stdio.h>\n") > @@ -885,7 +904,7 @@ def process_testcase(t): > for obj in test_descr.objs: > src_name = test_name + "-" + obj + ".c" > obj_name = test_name + "-" + obj + ".os" > - run_cmd([build_gcc, "-c", "-fPIC", testpfx + src_name, > + run_cmd([build_gcc, "-c", "-fPIC", testpfx_src + src_name, > "-o", testpfx + obj_name]) > > obj_processed = {} > @@ -903,10 +922,12 @@ def process_testcase(t): > deps.append(dep + ".FAKE") > if not dep in fake_created: > base_name = testpfx + test_name + "-" + dep > + src_base_name = (testpfx_src + test_name > + + "-" + dep) > cmd = [build_gcc, "-Wl,--no-as-needed", > ("-Wl,-soname=" + base_name + ".so"), > "-shared", base_name + ".FAKE.c", > - "-o", base_name + ".FAKE.so"] > + "-o", src_base_name + ".FAKE.so"] > run_cmd(cmd) > fake_created[dep] = True > dso_deps = map(lambda d: testpfx + test_name + "-" + d + ".so", > @@ -932,7 +953,7 @@ def process_testcase(t): > main_deps = map(lambda d: testpfx + test_name + "-" + d + ".so", > deps) > cmd = [build_gcc, "-Wl,--no-as-needed", "-o", testpfx + test_name, > - testpfx + test_name + ".c", "-L%s" % (os.getcwd()), > + testpfx_src + test_name + ".c", "-L%s" % (os.getcwd()), > "-Wl,-rpath-link=%s" % (os.getcwd())] > if '#' in test_descr.soname_map: > soname = ("-Wl,-soname=" + testpfx + test_name + "-" > @@ -987,14 +1008,14 @@ def process_testcase(t): > sfx = "" > if r[0] != "": > sfx = "-" + r[0].replace("=","_") > - f = open(testpfx + t.test_name + sfx + ".exp", "w") > + f = open(testpfx_src + t.test_name + sfx + ".exp", "w") > (output, xfail) = r[1] > f.write('%s' % output) > f.close() > > # Create header part of top-level testcase shell script, to wrap execution > # and output comparison together. > - t.sh = open(testpfx + t.test_name + ".sh", "w") > + t.sh = open(testpfx_src + t.test_name + ".sh", "w") > t.sh.write("#!/bin/sh\n") > t.sh.write("# Test driver for %s, generated by " > "dso-ordering-test.py\n" % (t.test_name)) > @@ -1022,12 +1043,12 @@ def process_testcase(t): > sfx = "" > if r[0] != "": > sfx = "-" + r[0].replace("=","_") > - expected_output_files += " $(objpfx)%s/%s%s.exp" % (test_subdir, > + expected_output_files += " $(objpfx)%s%s%s.exp" % (test_srcdir, > t.test_name, sfx) > makefile.write \ > - ("$(objpfx)%s.out: $(objpfx)%s/%s.sh%s " > + ("$(objpfx)%s.out: $(objpfx)%s%s.sh%s " > "$(common-objpfx)support/test-run-command\n" > - % (t.test_name, test_subdir, t.test_name, > + % (t.test_name, test_srcdir, t.test_name, > expected_output_files)) > makefile.write("\t$(SHELL) $< $(common-objpfx) '$(test-wrapper-env)' " > "'$(run-program-env)' > $@; $(evaluate-test)\n") > Ok. ^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v2 2/2] elf: Pre-generate Makefile fragment for DSO sorting tests [BZ #28550] 2021-11-08 15:42 [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu 2021-11-08 15:42 ` [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu @ 2021-11-08 15:42 ` H.J. Lu 2021-11-15 18:59 ` Adhemerval Zanella 2021-11-11 13:46 ` [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu 2 siblings, 1 reply; 15+ messages in thread From: H.J. Lu @ 2021-11-08 15:42 UTC (permalink / raw) To: libc-alpha; +Cc: Florian Weimer Generate Makefile fragment for DSO sorting tests at configure time. This partially fixes BZ #28550. --- configure | 11 +++++++++++ configure.ac | 12 ++++++++++++ elf/Makefile | 22 +++++----------------- 3 files changed, 28 insertions(+), 17 deletions(-) diff --git a/configure b/configure index 2f9adca064..91993da3b9 100755 --- a/configure +++ b/configure @@ -8484,3 +8484,14 @@ if test -n "$ac_unrecognized_opts" && test "$enable_option_checking" != no; then $as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2;} fi + +if test "$have_tunables" = yes; then + test -d elf || mkdir elf + objdir=`pwd` + rm -f elf/dso-sort-tests.mk + (for d in dso-sort-tests-1.def dso-sort-tests-2.def + do + $PYTHON $srcdir/scripts/dso-ordering-test.py \ + --description-file $srcdir/elf/$d --objpfx $objdir/elf/ + done) > elf/dso-sort-tests.mk +fi diff --git a/configure.ac b/configure.ac index 7eb4239359..faf70b930c 100644 --- a/configure.ac +++ b/configure.ac @@ -1901,3 +1901,15 @@ echo "$config_vars" >> config.make;; esac test -d bits || mkdir bits]],[[config_vars='$config_vars']]) AC_OUTPUT + +dnl Generate Makefile fragments for DSO sorting tests. +if test "$have_tunables" = yes; then + test -d elf || mkdir elf + objdir=`pwd` + rm -f elf/dso-sort-tests.mk + (for d in dso-sort-tests-1.def dso-sort-tests-2.def + do + $PYTHON $srcdir/scripts/dso-ordering-test.py \ + --description-file $srcdir/elf/$d --objpfx $objdir/elf/ + done) > elf/dso-sort-tests.mk +fi diff --git a/elf/Makefile b/elf/Makefile index 41c19668c3..6995ccaa0f 100644 --- a/elf/Makefile +++ b/elf/Makefile @@ -366,6 +366,11 @@ modules-names = testobj1 testobj2 testobj3 testobj4 testobj5 testobj6 \ tst-auxvalmod \ tst-dlmopen-gethostbyname-mod tst-ro-dynamic-mod \ +ifeq (yes,$(have-tunables)) +# Include Makefile fragments for DSO sorting tests. +include $(objpfx)dso-sort-tests.mk +endif + # Most modules build with _ISOMAC defined, but those filtered out # depend on internal headers. modules-names-tests = $(filter-out ifuncmod% tst-tlsmod%,\ @@ -486,23 +491,6 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ $(objpfx)tst-unused-dep-cmp.out endif -# DSO sorting tests: -# The dso-ordering-test.py script generates testcase source files in $(objpfx), -# creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a -# Makefile fragment to be included. -define include_dsosort_tests -$(objpfx)$(1).generated-makefile: $(1) - $(PYTHON) $(..)scripts/dso-ordering-test.py \ - --description-file $$< --objpfx $(objpfx) --output-makefile $$@ -include $(objpfx)$(1).generated-makefile -endef - -# Generate from each testcase description file -ifeq (yes,$(have-tunables)) -$(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) -$(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) -endif - check-abi: $(objpfx)check-abi-ld.out tests-special += $(objpfx)check-abi-ld.out update-abi: update-abi-ld -- 2.33.1 ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 2/2] elf: Pre-generate Makefile fragment for DSO sorting tests [BZ #28550] 2021-11-08 15:42 ` [PATCH v2 2/2] elf: Pre-generate Makefile fragment for DSO sorting tests " H.J. Lu @ 2021-11-15 18:59 ` Adhemerval Zanella 2021-11-15 19:51 ` H.J. Lu 0 siblings, 1 reply; 15+ messages in thread From: Adhemerval Zanella @ 2021-11-15 18:59 UTC (permalink / raw) To: H.J. Lu, libc-alpha; +Cc: Florian Weimer On 08/11/2021 12:42, H.J. Lu via Libc-alpha wrote: > Generate Makefile fragment for DSO sorting tests at configure time. > > This partially fixes BZ #28550. I am not sure if configure.ac is the right place to generate the Makefile fragment required for tests. Can we use 'avoid-generated' instead? diff --git a/elf/Makefile b/elf/Makefile index a311c3e23c..8556105a20 100644 --- a/elf/Makefile +++ b/elf/Makefile @@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ $(objpfx)tst-unused-dep-cmp.out endif +ifndef avoid-generated # DSO sorting tests: # The dso-ordering-test.py script generates testcase source files in $(objpfx), # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a @@ -507,6 +508,7 @@ ifeq (yes,$(have-tunables)) $(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) $(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) endif +endif check-abi: $(objpfx)check-abi-ld.out tests-special += $(objpfx)check-abi-ld.out > --- > configure | 11 +++++++++++ > configure.ac | 12 ++++++++++++ > elf/Makefile | 22 +++++----------------- > 3 files changed, 28 insertions(+), 17 deletions(-) > > diff --git a/configure b/configure > index 2f9adca064..91993da3b9 100755 > --- a/configure > +++ b/configure > @@ -8484,3 +8484,14 @@ if test -n "$ac_unrecognized_opts" && test "$enable_option_checking" != no; then > $as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2;} > fi > > + > +if test "$have_tunables" = yes; then > + test -d elf || mkdir elf > + objdir=`pwd` > + rm -f elf/dso-sort-tests.mk > + (for d in dso-sort-tests-1.def dso-sort-tests-2.def > + do > + $PYTHON $srcdir/scripts/dso-ordering-test.py \ > + --description-file $srcdir/elf/$d --objpfx $objdir/elf/ > + done) > elf/dso-sort-tests.mk > +fi > diff --git a/configure.ac b/configure.ac > index 7eb4239359..faf70b930c 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -1901,3 +1901,15 @@ echo "$config_vars" >> config.make;; > esac > test -d bits || mkdir bits]],[[config_vars='$config_vars']]) > AC_OUTPUT > + > +dnl Generate Makefile fragments for DSO sorting tests. > +if test "$have_tunables" = yes; then > + test -d elf || mkdir elf > + objdir=`pwd` > + rm -f elf/dso-sort-tests.mk > + (for d in dso-sort-tests-1.def dso-sort-tests-2.def > + do > + $PYTHON $srcdir/scripts/dso-ordering-test.py \ > + --description-file $srcdir/elf/$d --objpfx $objdir/elf/ > + done) > elf/dso-sort-tests.mk > +fi > diff --git a/elf/Makefile b/elf/Makefile > index 41c19668c3..6995ccaa0f 100644 > --- a/elf/Makefile > +++ b/elf/Makefile > @@ -366,6 +366,11 @@ modules-names = testobj1 testobj2 testobj3 testobj4 testobj5 testobj6 \ > tst-auxvalmod \ > tst-dlmopen-gethostbyname-mod tst-ro-dynamic-mod \ > > +ifeq (yes,$(have-tunables)) > +# Include Makefile fragments for DSO sorting tests. > +include $(objpfx)dso-sort-tests.mk > +endif > + > # Most modules build with _ISOMAC defined, but those filtered out > # depend on internal headers. > modules-names-tests = $(filter-out ifuncmod% tst-tlsmod%,\ > @@ -486,23 +491,6 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ > $(objpfx)tst-unused-dep-cmp.out > endif > > -# DSO sorting tests: > -# The dso-ordering-test.py script generates testcase source files in $(objpfx), > -# creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a > -# Makefile fragment to be included. > -define include_dsosort_tests > -$(objpfx)$(1).generated-makefile: $(1) > - $(PYTHON) $(..)scripts/dso-ordering-test.py \ > - --description-file $$< --objpfx $(objpfx) --output-makefile $$@ > -include $(objpfx)$(1).generated-makefile > -endef > - > -# Generate from each testcase description file > -ifeq (yes,$(have-tunables)) > -$(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) > -$(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) > -endif > - > check-abi: $(objpfx)check-abi-ld.out > tests-special += $(objpfx)check-abi-ld.out > update-abi: update-abi-ld > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 2/2] elf: Pre-generate Makefile fragment for DSO sorting tests [BZ #28550] 2021-11-15 18:59 ` Adhemerval Zanella @ 2021-11-15 19:51 ` H.J. Lu 2021-11-15 20:18 ` Adhemerval Zanella 0 siblings, 1 reply; 15+ messages in thread From: H.J. Lu @ 2021-11-15 19:51 UTC (permalink / raw) To: Adhemerval Zanella; +Cc: GNU C Library, Florian Weimer On Mon, Nov 15, 2021 at 10:59 AM Adhemerval Zanella <adhemerval.zanella@linaro.org> wrote: > > > > On 08/11/2021 12:42, H.J. Lu via Libc-alpha wrote: > > Generate Makefile fragment for DSO sorting tests at configure time. > > > > This partially fixes BZ #28550. > > I am not sure if configure.ac is the right place to generate the Makefile > fragment required for tests. Can we use 'avoid-generated' instead? > > > diff --git a/elf/Makefile b/elf/Makefile > index a311c3e23c..8556105a20 100644 > --- a/elf/Makefile > +++ b/elf/Makefile > @@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ > $(objpfx)tst-unused-dep-cmp.out > endif > > +ifndef avoid-generated > # DSO sorting tests: > # The dso-ordering-test.py script generates testcase source files in $(objpfx), > # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a > @@ -507,6 +508,7 @@ ifeq (yes,$(have-tunables)) > $(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) > $(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) > endif > +endif It doesn't work. "make check -j28" on a machine with 112 cores failed: /export/users/hjl/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/elf/dso-sort-tests-1.def.generated-makefile:938: warning: ignoring old recipe for target '/export/users/hjl/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/elf/tst-dso-ordering9-dir/tst-dso-ordering9_25-bacde-b.so' ... make[3]: *** [Makefile:483: elf/tests] Error 2 I couldn't find the exact error message in a very long make log file. > check-abi: $(objpfx)check-abi-ld.out > tests-special += $(objpfx)check-abi-ld.out > > > > --- > > configure | 11 +++++++++++ > > configure.ac | 12 ++++++++++++ > > elf/Makefile | 22 +++++----------------- > > 3 files changed, 28 insertions(+), 17 deletions(-) > > > > diff --git a/configure b/configure > > index 2f9adca064..91993da3b9 100755 > > --- a/configure > > +++ b/configure > > @@ -8484,3 +8484,14 @@ if test -n "$ac_unrecognized_opts" && test "$enable_option_checking" != no; then > > $as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2;} > > fi > > > > + > > +if test "$have_tunables" = yes; then > > + test -d elf || mkdir elf > > + objdir=`pwd` > > + rm -f elf/dso-sort-tests.mk > > + (for d in dso-sort-tests-1.def dso-sort-tests-2.def > > + do > > + $PYTHON $srcdir/scripts/dso-ordering-test.py \ > > + --description-file $srcdir/elf/$d --objpfx $objdir/elf/ > > + done) > elf/dso-sort-tests.mk > > +fi > > diff --git a/configure.ac b/configure.ac > > index 7eb4239359..faf70b930c 100644 > > --- a/configure.ac > > +++ b/configure.ac > > @@ -1901,3 +1901,15 @@ echo "$config_vars" >> config.make;; > > esac > > test -d bits || mkdir bits]],[[config_vars='$config_vars']]) > > AC_OUTPUT > > + > > +dnl Generate Makefile fragments for DSO sorting tests. > > +if test "$have_tunables" = yes; then > > + test -d elf || mkdir elf > > + objdir=`pwd` > > + rm -f elf/dso-sort-tests.mk > > + (for d in dso-sort-tests-1.def dso-sort-tests-2.def > > + do > > + $PYTHON $srcdir/scripts/dso-ordering-test.py \ > > + --description-file $srcdir/elf/$d --objpfx $objdir/elf/ > > + done) > elf/dso-sort-tests.mk > > +fi > > diff --git a/elf/Makefile b/elf/Makefile > > index 41c19668c3..6995ccaa0f 100644 > > --- a/elf/Makefile > > +++ b/elf/Makefile > > @@ -366,6 +366,11 @@ modules-names = testobj1 testobj2 testobj3 testobj4 testobj5 testobj6 \ > > tst-auxvalmod \ > > tst-dlmopen-gethostbyname-mod tst-ro-dynamic-mod \ > > > > +ifeq (yes,$(have-tunables)) > > +# Include Makefile fragments for DSO sorting tests. > > +include $(objpfx)dso-sort-tests.mk > > +endif > > + > > # Most modules build with _ISOMAC defined, but those filtered out > > # depend on internal headers. > > modules-names-tests = $(filter-out ifuncmod% tst-tlsmod%,\ > > @@ -486,23 +491,6 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ > > $(objpfx)tst-unused-dep-cmp.out > > endif > > > > -# DSO sorting tests: > > -# The dso-ordering-test.py script generates testcase source files in $(objpfx), > > -# creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a > > -# Makefile fragment to be included. > > -define include_dsosort_tests > > -$(objpfx)$(1).generated-makefile: $(1) > > - $(PYTHON) $(..)scripts/dso-ordering-test.py \ > > - --description-file $$< --objpfx $(objpfx) --output-makefile $$@ > > -include $(objpfx)$(1).generated-makefile > > -endef > > - > > -# Generate from each testcase description file > > -ifeq (yes,$(have-tunables)) > > -$(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) > > -$(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) > > -endif > > - > > check-abi: $(objpfx)check-abi-ld.out > > tests-special += $(objpfx)check-abi-ld.out > > update-abi: update-abi-ld > > -- H.J. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 2/2] elf: Pre-generate Makefile fragment for DSO sorting tests [BZ #28550] 2021-11-15 19:51 ` H.J. Lu @ 2021-11-15 20:18 ` Adhemerval Zanella 2021-11-16 4:48 ` [PATCH] elf: Use a temporary file to generate Makefile fragments " H.J. Lu 0 siblings, 1 reply; 15+ messages in thread From: Adhemerval Zanella @ 2021-11-15 20:18 UTC (permalink / raw) To: H.J. Lu; +Cc: GNU C Library, Florian Weimer On 15/11/2021 16:51, H.J. Lu wrote: > On Mon, Nov 15, 2021 at 10:59 AM Adhemerval Zanella > <adhemerval.zanella@linaro.org> wrote: >> >> >> >> On 08/11/2021 12:42, H.J. Lu via Libc-alpha wrote: >>> Generate Makefile fragment for DSO sorting tests at configure time. >>> >>> This partially fixes BZ #28550. >> >> I am not sure if configure.ac is the right place to generate the Makefile >> fragment required for tests. Can we use 'avoid-generated' instead? >> >> >> diff --git a/elf/Makefile b/elf/Makefile >> index a311c3e23c..8556105a20 100644 >> --- a/elf/Makefile >> +++ b/elf/Makefile >> @@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ >> $(objpfx)tst-unused-dep-cmp.out >> endif >> >> +ifndef avoid-generated >> # DSO sorting tests: >> # The dso-ordering-test.py script generates testcase source files in $(objpfx), >> # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a >> @@ -507,6 +508,7 @@ ifeq (yes,$(have-tunables)) >> $(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) >> $(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) >> endif >> +endif > > It doesn't work. "make check -j28" on a machine with 112 cores failed: > > /export/users/hjl/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/elf/dso-sort-tests-1.def.generated-makefile:938: > warning: ignoring old recipe for target > '/export/users/hjl/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/elf/tst-dso-ordering9-dir/tst-dso-ordering9_25-bacde-b.so' > ... > make[3]: *** [Makefile:483: elf/tests] Error 2 > > I couldn't find the exact error message in a very long make log file. This is usual way to create other auto-generated Makefile fragments, such as sysd-sycalls and time64-compat.mk. Maybe the below helps (I forgot to use '-include' and add the fragments on postclean-generated). diff --git a/elf/Makefile b/elf/Makefile index a311c3e23c..0e6d7ffb51 100644 --- a/elf/Makefile +++ b/elf/Makefile @@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ $(objpfx)tst-unused-dep-cmp.out endif +ifndef avoid-generated # DSO sorting tests: # The dso-ordering-test.py script generates testcase source files in $(objpfx), # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a @@ -499,7 +500,7 @@ define include_dsosort_tests $(objpfx)$(1).generated-makefile: $(1) $(PYTHON) $(..)scripts/dso-ordering-test.py \ --description-file $$< --objpfx $(objpfx) --output-makefile $$@ -include $(objpfx)$(1).generated-makefile +-include $(objpfx)$(1).generated-makefile endef # Generate from each testcase description file @@ -507,6 +508,10 @@ ifeq (yes,$(have-tunables)) $(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) $(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) endif +endif + +postclean-generated += $(objpfx)/dso-sort-tests-2.generated-makefile \ + $(objpfx)/dso-sort-tests-2.generated-makefile check-abi: $(objpfx)check-abi-ld.out tests-special += $(objpfx)check-abi-ld.out ^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH] elf: Use a temporary file to generate Makefile fragments [BZ #28550] 2021-11-15 20:18 ` Adhemerval Zanella @ 2021-11-16 4:48 ` H.J. Lu 2021-11-16 12:52 ` Adhemerval Zanella 0 siblings, 1 reply; 15+ messages in thread From: H.J. Lu @ 2021-11-16 4:48 UTC (permalink / raw) To: Adhemerval Zanella; +Cc: GNU C Library, Florian Weimer [-- Attachment #1: Type: text/plain, Size: 3405 bytes --] On Mon, Nov 15, 2021 at 12:18 PM Adhemerval Zanella <adhemerval.zanella@linaro.org> wrote: > > > > On 15/11/2021 16:51, H.J. Lu wrote: > > On Mon, Nov 15, 2021 at 10:59 AM Adhemerval Zanella > > <adhemerval.zanella@linaro.org> wrote: > >> > >> > >> > >> On 08/11/2021 12:42, H.J. Lu via Libc-alpha wrote: > >>> Generate Makefile fragment for DSO sorting tests at configure time. > >>> > >>> This partially fixes BZ #28550. > >> > >> I am not sure if configure.ac is the right place to generate the Makefile > >> fragment required for tests. Can we use 'avoid-generated' instead? > >> > >> > >> diff --git a/elf/Makefile b/elf/Makefile > >> index a311c3e23c..8556105a20 100644 > >> --- a/elf/Makefile > >> +++ b/elf/Makefile > >> @@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ > >> $(objpfx)tst-unused-dep-cmp.out > >> endif > >> > >> +ifndef avoid-generated > >> # DSO sorting tests: > >> # The dso-ordering-test.py script generates testcase source files in $(objpfx), > >> # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a > >> @@ -507,6 +508,7 @@ ifeq (yes,$(have-tunables)) > >> $(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) > >> $(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) > >> endif > >> +endif > > > > It doesn't work. "make check -j28" on a machine with 112 cores failed: > > > > /export/users/hjl/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/elf/dso-sort-tests-1.def.generated-makefile:938: > > warning: ignoring old recipe for target > > '/export/users/hjl/build/gnu/tools-build/glibc-cet-gitlab/build-x86_64-linux/elf/tst-dso-ordering9-dir/tst-dso-ordering9_25-bacde-b.so' > > ... > > make[3]: *** [Makefile:483: elf/tests] Error 2 > > > > I couldn't find the exact error message in a very long make log file. > This is usual way to create other auto-generated Makefile fragments, such as > sysd-sycalls and time64-compat.mk. Maybe the below helps (I forgot to use > '-include' and add the fragments on postclean-generated). > > > diff --git a/elf/Makefile b/elf/Makefile > index a311c3e23c..0e6d7ffb51 100644 > --- a/elf/Makefile > +++ b/elf/Makefile > @@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ > $(objpfx)tst-unused-dep-cmp.out > endif > > +ifndef avoid-generated > # DSO sorting tests: > # The dso-ordering-test.py script generates testcase source files in $(objpfx), > # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a > @@ -499,7 +500,7 @@ define include_dsosort_tests > $(objpfx)$(1).generated-makefile: $(1) > $(PYTHON) $(..)scripts/dso-ordering-test.py \ > --description-file $$< --objpfx $(objpfx) --output-makefile $$@ > -include $(objpfx)$(1).generated-makefile > +-include $(objpfx)$(1).generated-makefile > endef > > # Generate from each testcase description file > @@ -507,6 +508,10 @@ ifeq (yes,$(have-tunables)) > $(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) > $(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) > endif > +endif > + > +postclean-generated += $(objpfx)/dso-sort-tests-2.generated-makefile \ > + $(objpfx)/dso-sort-tests-2.generated-makefile > > check-abi: $(objpfx)check-abi-ld.out > tests-special += $(objpfx)check-abi-ld.out This works for me. Thanks. -- H.J. [-- Attachment #2: 0001-elf-Use-a-temporary-file-to-generate-Makefile-fragme.patch --] [-- Type: text/x-patch, Size: 1006 bytes --] From a928b52c036bb888797b3e81014f441f0d876b65 Mon Sep 17 00:00:00 2001 From: "H.J. Lu" <hjl.tools@gmail.com> Date: Mon, 15 Nov 2021 16:28:39 -0800 Subject: [PATCH] elf: Use a temporary file to generate Makefile fragments [BZ #28550] Use a temporary file to generate Makefile fragments for DSO sorting tests and use -include on them. This partially fixes BZ #28550. --- elf/Makefile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/elf/Makefile b/elf/Makefile index a311c3e23c..57374fc690 100644 --- a/elf/Makefile +++ b/elf/Makefile @@ -498,8 +498,9 @@ endif define include_dsosort_tests $(objpfx)$(1).generated-makefile: $(1) $(PYTHON) $(..)scripts/dso-ordering-test.py \ - --description-file $$< --objpfx $(objpfx) --output-makefile $$@ -include $(objpfx)$(1).generated-makefile + --description-file $$< --objpfx $(objpfx) --output-makefile $$@T + mv $$@T $$@ +-include $(objpfx)$(1).generated-makefile endef # Generate from each testcase description file -- 2.33.1 ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] elf: Use a temporary file to generate Makefile fragments [BZ #28550] 2021-11-16 4:48 ` [PATCH] elf: Use a temporary file to generate Makefile fragments " H.J. Lu @ 2021-11-16 12:52 ` Adhemerval Zanella 2021-11-16 13:15 ` H.J. Lu 0 siblings, 1 reply; 15+ messages in thread From: Adhemerval Zanella @ 2021-11-16 12:52 UTC (permalink / raw) To: H.J. Lu; +Cc: GNU C Library, Florian Weimer On 16/11/2021 01:48, H.J. Lu wrote: > > This works for me. > > Thanks. > > From a928b52c036bb888797b3e81014f441f0d876b65 Mon Sep 17 00:00:00 2001 > From: "H.J. Lu" <hjl.tools@gmail.com> > Date: Mon, 15 Nov 2021 16:28:39 -0800 > Subject: [PATCH] elf: Use a temporary file to generate Makefile fragments [BZ > #28550] > > Use a temporary file to generate Makefile fragments for DSO sorting tests > and use -include on them. > > This partially fixes BZ #28550. > --- > elf/Makefile | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/elf/Makefile b/elf/Makefile > index a311c3e23c..57374fc690 100644 > --- a/elf/Makefile > +++ b/elf/Makefile > @@ -498,8 +498,9 @@ endif > define include_dsosort_tests > $(objpfx)$(1).generated-makefile: $(1) > $(PYTHON) $(..)scripts/dso-ordering-test.py \ > - --description-file $$< --objpfx $(objpfx) --output-makefile $$@ > -include $(objpfx)$(1).generated-makefile > + --description-file $$< --objpfx $(objpfx) --output-makefile $$@T > + mv $$@T $$@ > +-include $(objpfx)$(1).generated-makefile > endef > > # Generate from each testcase description file > -- > 2.33.1 I think you need the below on top of your patch. The 'avoid-generated' and 'postclean-generated' are required so a 'make clean' removes the autogenerated fragments so a subsequent 'make' regenerates them. diff --git a/elf/Makefile b/elf/Makefile index 57374fc690..31759279d4 100644 --- a/elf/Makefile +++ b/elf/Makefile @@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ $(objpfx)tst-unused-dep-cmp.out endif +ifndef avoid-generated # DSO sorting tests: # The dso-ordering-test.py script generates testcase source files in $(objpfx), # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a @@ -508,6 +509,9 @@ ifeq (yes,$(have-tunables)) $(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) $(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) endif +endif +postclean-generated += $(objpfx)/dso-sort-tests-2.generated-makefile \ + $(objpfx)/dso-sort-tests-2.generated-makefile check-abi: $(objpfx)check-abi-ld.out tests-special += $(objpfx)check-abi-ld.out ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] elf: Use a temporary file to generate Makefile fragments [BZ #28550] 2021-11-16 12:52 ` Adhemerval Zanella @ 2021-11-16 13:15 ` H.J. Lu 0 siblings, 0 replies; 15+ messages in thread From: H.J. Lu @ 2021-11-16 13:15 UTC (permalink / raw) To: Adhemerval Zanella; +Cc: GNU C Library, Florian Weimer On Tue, Nov 16, 2021 at 4:52 AM Adhemerval Zanella <adhemerval.zanella@linaro.org> wrote: > > > > On 16/11/2021 01:48, H.J. Lu wrote: > > > > This works for me. > > > > Thanks. > > > > > From a928b52c036bb888797b3e81014f441f0d876b65 Mon Sep 17 00:00:00 2001 > > From: "H.J. Lu" <hjl.tools@gmail.com> > > Date: Mon, 15 Nov 2021 16:28:39 -0800 > > Subject: [PATCH] elf: Use a temporary file to generate Makefile fragments [BZ > > #28550] > > > > Use a temporary file to generate Makefile fragments for DSO sorting tests > > and use -include on them. > > > > This partially fixes BZ #28550. > > --- > > elf/Makefile | 5 +++-- > > 1 file changed, 3 insertions(+), 2 deletions(-) > > > > diff --git a/elf/Makefile b/elf/Makefile > > index a311c3e23c..57374fc690 100644 > > --- a/elf/Makefile > > +++ b/elf/Makefile > > @@ -498,8 +498,9 @@ endif > > define include_dsosort_tests > > $(objpfx)$(1).generated-makefile: $(1) > > $(PYTHON) $(..)scripts/dso-ordering-test.py \ > > - --description-file $$< --objpfx $(objpfx) --output-makefile $$@ > > -include $(objpfx)$(1).generated-makefile > > + --description-file $$< --objpfx $(objpfx) --output-makefile $$@T > > + mv $$@T $$@ > > +-include $(objpfx)$(1).generated-makefile > > endef > > > > # Generate from each testcase description file > > -- > > 2.33.1 > > I think you need the below on top of your patch. The 'avoid-generated' and > 'postclean-generated' are required so a 'make clean' removes the autogenerated > fragments so a subsequent 'make' regenerates them. Fixed in the v2 patch. Thanks. > diff --git a/elf/Makefile b/elf/Makefile > index 57374fc690..31759279d4 100644 > --- a/elf/Makefile > +++ b/elf/Makefile > @@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \ > $(objpfx)tst-unused-dep-cmp.out > endif > > +ifndef avoid-generated > # DSO sorting tests: > # The dso-ordering-test.py script generates testcase source files in $(objpfx), > # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a > @@ -508,6 +509,9 @@ ifeq (yes,$(have-tunables)) > $(eval $(call include_dsosort_tests,dso-sort-tests-1.def)) > $(eval $(call include_dsosort_tests,dso-sort-tests-2.def)) > endif > +endif > +postclean-generated += $(objpfx)/dso-sort-tests-2.generated-makefile \ > + $(objpfx)/dso-sort-tests-2.generated-makefile > > check-abi: $(objpfx)check-abi-ld.out > tests-special += $(objpfx)check-abi-ld.out -- H.J. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests 2021-11-08 15:42 [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu 2021-11-08 15:42 ` [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu 2021-11-08 15:42 ` [PATCH v2 2/2] elf: Pre-generate Makefile fragment for DSO sorting tests " H.J. Lu @ 2021-11-11 13:46 ` H.J. Lu 2021-11-11 14:05 ` Adhemerval Zanella 2 siblings, 1 reply; 15+ messages in thread From: H.J. Lu @ 2021-11-11 13:46 UTC (permalink / raw) To: GNU C Library; +Cc: Florian Weimer On Mon, Nov 8, 2021 at 7:42 AM H.J. Lu <hjl.tools@gmail.com> wrote: > > One problem of using > > $(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c) > compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags) > compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS) > > to build DSO sorting relocatable objects is that $< may not be the source > file with 3 "make -j 28" parallel builds on a machine with 112 cores. > When generating Makefile fragments at build time, the same file may be > generated more than once at the same time with parallel builds. > > 1. Update dso-ordering-test.py to put all sources for DSO sorting tests > in a single directory, dso-sort-tests-src, and compile relocatable objects > with > > $(objpfx)tst-dso-ordering1-dir/tst-dso-ordering1-a.os: $(objpfx)dso-sort-tests-src/tst-dso-ordering1-a.c > $(compile.c) $(OUTPUT_OPTION) > > 2. Generate Makefile fragments for DSO sorting tests at configure time > to avoid generate them in the elf directory at build time. > > H.J. Lu (2): > dso-ordering-test.py: Put all sources in one directory [BZ #28550] > elf: Pre-generate Makefile fragment for DSO sorting tests [BZ #28550] > > configure | 11 +++++++++ > configure.ac | 12 +++++++++ > elf/Makefile | 22 ++++------------- > scripts/dso-ordering-test.py | 47 ++++++++++++++++++++++++++---------- > 4 files changed, 62 insertions(+), 30 deletions(-) > > -- > 2.33.1 > Any comments on the v2 patches? "make check -j28" has been failing for me on machines with many cores. I'd like to resolve it soon. -- H.J. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests 2021-11-11 13:46 ` [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu @ 2021-11-11 14:05 ` Adhemerval Zanella 2021-11-15 14:43 ` H.J. Lu 0 siblings, 1 reply; 15+ messages in thread From: Adhemerval Zanella @ 2021-11-11 14:05 UTC (permalink / raw) To: H.J. Lu, GNU C Library; +Cc: Florian Weimer On 11/11/2021 10:46, H.J. Lu via Libc-alpha wrote: > On Mon, Nov 8, 2021 at 7:42 AM H.J. Lu <hjl.tools@gmail.com> wrote: >> >> One problem of using >> >> $(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c) >> compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags) >> compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS) >> >> to build DSO sorting relocatable objects is that $< may not be the source >> file with 3 "make -j 28" parallel builds on a machine with 112 cores. >> When generating Makefile fragments at build time, the same file may be >> generated more than once at the same time with parallel builds. >> >> 1. Update dso-ordering-test.py to put all sources for DSO sorting tests >> in a single directory, dso-sort-tests-src, and compile relocatable objects >> with >> >> $(objpfx)tst-dso-ordering1-dir/tst-dso-ordering1-a.os: $(objpfx)dso-sort-tests-src/tst-dso-ordering1-a.c >> $(compile.c) $(OUTPUT_OPTION) >> >> 2. Generate Makefile fragments for DSO sorting tests at configure time >> to avoid generate them in the elf directory at build time. >> >> H.J. Lu (2): >> dso-ordering-test.py: Put all sources in one directory [BZ #28550] >> elf: Pre-generate Makefile fragment for DSO sorting tests [BZ #28550] >> >> configure | 11 +++++++++ >> configure.ac | 12 +++++++++ >> elf/Makefile | 22 ++++------------- >> scripts/dso-ordering-test.py | 47 ++++++++++++++++++++++++++---------- >> 4 files changed, 62 insertions(+), 30 deletions(-) >> >> -- >> 2.33.1 >> > > Any comments on the v2 patches? "make check -j28" has been failing for me > on machines with many cores. I'd like to resolve it soon. > It is on my list, just sorting out Florian comments to rtld-audit patchset. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests 2021-11-11 14:05 ` Adhemerval Zanella @ 2021-11-15 14:43 ` H.J. Lu 0 siblings, 0 replies; 15+ messages in thread From: H.J. Lu @ 2021-11-15 14:43 UTC (permalink / raw) To: Adhemerval Zanella; +Cc: GNU C Library, Florian Weimer On Thu, Nov 11, 2021 at 6:05 AM Adhemerval Zanella <adhemerval.zanella@linaro.org> wrote: > > > > On 11/11/2021 10:46, H.J. Lu via Libc-alpha wrote: > > On Mon, Nov 8, 2021 at 7:42 AM H.J. Lu <hjl.tools@gmail.com> wrote: > >> > >> One problem of using > >> > >> $(objpfx)%$o: $(objpfx)%.c $(before-compile); $$(compile-command.c) > >> compile-command.c = $(compile.c) $(OUTPUT_OPTION) $(compile-mkdep-flags) > >> compile.c = $(CC) $< -c $(CFLAGS) $(CPPFLAGS) > >> > >> to build DSO sorting relocatable objects is that $< may not be the source > >> file with 3 "make -j 28" parallel builds on a machine with 112 cores. > >> When generating Makefile fragments at build time, the same file may be > >> generated more than once at the same time with parallel builds. > >> > >> 1. Update dso-ordering-test.py to put all sources for DSO sorting tests > >> in a single directory, dso-sort-tests-src, and compile relocatable objects > >> with > >> > >> $(objpfx)tst-dso-ordering1-dir/tst-dso-ordering1-a.os: $(objpfx)dso-sort-tests-src/tst-dso-ordering1-a.c > >> $(compile.c) $(OUTPUT_OPTION) > >> > >> 2. Generate Makefile fragments for DSO sorting tests at configure time > >> to avoid generate them in the elf directory at build time. > >> > >> H.J. Lu (2): > >> dso-ordering-test.py: Put all sources in one directory [BZ #28550] > >> elf: Pre-generate Makefile fragment for DSO sorting tests [BZ #28550] > >> > >> configure | 11 +++++++++ > >> configure.ac | 12 +++++++++ > >> elf/Makefile | 22 ++++------------- > >> scripts/dso-ordering-test.py | 47 ++++++++++++++++++++++++++---------- > >> 4 files changed, 62 insertions(+), 30 deletions(-) > >> > >> -- > >> 2.33.1 > >> > > > > Any comments on the v2 patches? "make check -j28" has been failing for me > > on machines with many cores. I'd like to resolve it soon. > > > > It is on my list, just sorting out Florian comments to rtld-audit patchset. This bug has been causing false negative results with "makc check -jN" on machines with many cores. I will check it in on Tuesday if there are no objections. Thanks. -- H.J. ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2021-11-16 13:15 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-11-08 15:42 [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu 2021-11-08 15:42 ` [PATCH v2 1/2] dso-ordering-test.py: Put all sources in one directory [BZ #28550] H.J. Lu 2021-11-11 17:52 ` Chung-Lin Tang 2021-11-11 18:21 ` H.J. Lu 2021-11-15 18:50 ` Adhemerval Zanella 2021-11-08 15:42 ` [PATCH v2 2/2] elf: Pre-generate Makefile fragment for DSO sorting tests " H.J. Lu 2021-11-15 18:59 ` Adhemerval Zanella 2021-11-15 19:51 ` H.J. Lu 2021-11-15 20:18 ` Adhemerval Zanella 2021-11-16 4:48 ` [PATCH] elf: Use a temporary file to generate Makefile fragments " H.J. Lu 2021-11-16 12:52 ` Adhemerval Zanella 2021-11-16 13:15 ` H.J. Lu 2021-11-11 13:46 ` [PATCH v2 0/2] Update Makefile fragments for DSO sorting tests H.J. Lu 2021-11-11 14:05 ` Adhemerval Zanella 2021-11-15 14:43 ` H.J. Lu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).