[PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[H.J. Lu]

1. Use a temporary file to generate Makefile fragments for DSO sorting
tests and use -include on them.
2. Add Makefile fragments to postclean-generated so that a "make clean"
removes the autogenerated fragments and a subsequent "make" regenerates
them.

This partially fixes BZ #28550.
---
 elf/Makefile | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/elf/Makefile b/elf/Makefile
index a311c3e23c..525f302d1a 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \
 		 $(objpfx)tst-unused-dep-cmp.out
 endif
 
+ifndef avoid-generated
 # DSO sorting tests:
 # The dso-ordering-test.py script generates testcase source files in $(objpfx),
 # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a
@@ -498,9 +499,14 @@ endif
 define include_dsosort_tests
 $(objpfx)$(1).generated-makefile: $(1)
 	$(PYTHON) $(..)scripts/dso-ordering-test.py \
-	--description-file $$< --objpfx $(objpfx) --output-makefile $$@
-include $(objpfx)$(1).generated-makefile
+	--description-file $$< --objpfx $(objpfx) --output-makefile $$@T
+	mv $$@T $$@
+-include $(objpfx)$(1).generated-makefile
 endef
+endif
+
+postclean-generated += $(objpfx)/dso-sort-tests-2.generated-makefile \
+		       $(objpfx)/dso-sort-tests-2.generated-makefile
 
 # Generate from each testcase description file
 ifeq (yes,$(have-tunables))
-- 
2.33.1

Re: [PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[Adhemerval Zanella]

On 16/11/2021 10:14, H.J. Lu wrote:
> 1. Use a temporary file to generate Makefile fragments for DSO sorting
> tests and use -include on them.
> 2. Add Makefile fragments to postclean-generated so that a "make clean"
> removes the autogenerated fragments and a subsequent "make" regenerates
> them.
> 
> This partially fixes BZ #28550.

LGTM, thanks.

Reviewed-by: Adhemerval Zanella  <adhemerval.zanella@linaro.org>

> ---
>  elf/Makefile | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/elf/Makefile b/elf/Makefile
> index a311c3e23c..525f302d1a 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -491,6 +491,7 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \
>  		 $(objpfx)tst-unused-dep-cmp.out
>  endif
>  
> +ifndef avoid-generated
>  # DSO sorting tests:
>  # The dso-ordering-test.py script generates testcase source files in $(objpfx),
>  # creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a
> @@ -498,9 +499,14 @@ endif
>  define include_dsosort_tests
>  $(objpfx)$(1).generated-makefile: $(1)
>  	$(PYTHON) $(..)scripts/dso-ordering-test.py \
> -	--description-file $$< --objpfx $(objpfx) --output-makefile $$@
> -include $(objpfx)$(1).generated-makefile
> +	--description-file $$< --objpfx $(objpfx) --output-makefile $$@T
> +	mv $$@T $$@
> +-include $(objpfx)$(1).generated-makefile
>  endef
> +endif
> +
> +postclean-generated += $(objpfx)/dso-sort-tests-2.generated-makefile \
> +		       $(objpfx)/dso-sort-tests-2.generated-makefile
>  
>  # Generate from each testcase description file
>  ifeq (yes,$(have-tunables))
> 

Re: [PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[Florian Weimer]

I still see

  /builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux/elf/dso-sort-tests-2.def.generated-makefile:1067: warning: overriding recipe for target '/builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux/elf/tst-redhat-1162810-dir/tst-redhat-1162810-M22X17.os'

and eventually a build failure:

  make[2]: Leaving directory '/builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/elf'
  make[1]: *** [Makefile:483: elf/tests] Error 2
  make: *** [Makefile:9: check] Error 2

There's no error before that, or other message indicating which make
target is failing.

The full build log is here:

  <https://koji.fedoraproject.org/koji/taskinfo?taskID=78987342>
  <https://kojipkgs.fedoraproject.org//work/tasks/7342/78987342/build.log>

The make error messages are not very helpful to me though.

Thanks,
Florian

Re: [PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[H.J. Lu]

On Wed, Nov 17, 2021 at 5:49 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> I still see
>
>   /builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux/elf/dso-sort-tests-2.def.generated-makefile:1067: warning: overriding recipe for target '/builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux/elf/tst-redhat-1162810-dir/tst-redhat-1162810-M22X17.os'
>
> and eventually a build failure:
>
>   make[2]: Leaving directory '/builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/elf'
>   make[1]: *** [Makefile:483: elf/tests] Error 2
>   make: *** [Makefile:9: check] Error 2
>
> There's no error before that, or other message indicating which make
> target is failing.
>
> The full build log is here:
>
>   <https://koji.fedoraproject.org/koji/taskinfo?taskID=78987342>
>   <https://kojipkgs.fedoraproject.org//work/tasks/7342/78987342/build.log>
>
> The make error messages are not very helpful to me though.
>

It also happened to me in a build directory whose path contains a
symbolic linked directory.  Make was confused to include the generated
makefile twice, which could be reached with 2 different pathnames.
After removing the symbolic linked directory, the generated makefile
can be reached with a single pathname, the make error went away.

-- 
H.J.

Re: [PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[Florian Weimer]

* H. J. Lu:

> On Wed, Nov 17, 2021 at 5:49 AM Florian Weimer <fweimer@redhat.com> wrote:
>>
>> I still see
>>
>>   /builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux/elf/dso-sort-tests-2.def.generated-makefile:1067: warning: overriding recipe for target '/builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux/elf/tst-redhat-1162810-dir/tst-redhat-1162810-M22X17.os'
>>
>> and eventually a build failure:
>>
>>   make[2]: Leaving directory '/builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/elf'
>>   make[1]: *** [Makefile:483: elf/tests] Error 2
>>   make: *** [Makefile:9: check] Error 2
>>
>> There's no error before that, or other message indicating which make
>> target is failing.
>>
>> The full build log is here:
>>
>>   <https://koji.fedoraproject.org/koji/taskinfo?taskID=78987342>
>>   <https://kojipkgs.fedoraproject.org//work/tasks/7342/78987342/build.log>
>>
>> The make error messages are not very helpful to me though.
>>
>
> It also happened to me in a build directory whose path contains a
> symbolic linked directory.  Make was confused to include the generated
> makefile twice, which could be reached with 2 different pathnames.
> After removing the symbolic linked directory, the generated makefile
> can be reached with a single pathname, the make error went away.

This is a standard Koji build environment.  The build directory is not a
symbolic link:

+ readlink -f .
/builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux

And we reach that via /builddir/build/BUILD.

So I still don't know what's going on there.

Thanks,
Florian

Re: [PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[H.J. Lu]

On Wed, Nov 17, 2021 at 7:29 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * H. J. Lu:
>
> > On Wed, Nov 17, 2021 at 5:49 AM Florian Weimer <fweimer@redhat.com> wrote:
> >>
> >> I still see
> >>
> >>   /builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux/elf/dso-sort-tests-2.def.generated-makefile:1067: warning: overriding recipe for target '/builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux/elf/tst-redhat-1162810-dir/tst-redhat-1162810-M22X17.os'
> >>
> >> and eventually a build failure:
> >>
> >>   make[2]: Leaving directory '/builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/elf'
> >>   make[1]: *** [Makefile:483: elf/tests] Error 2
> >>   make: *** [Makefile:9: check] Error 2
> >>
> >> There's no error before that, or other message indicating which make
> >> target is failing.
> >>
> >> The full build log is here:
> >>
> >>   <https://koji.fedoraproject.org/koji/taskinfo?taskID=78987342>
> >>   <https://kojipkgs.fedoraproject.org//work/tasks/7342/78987342/build.log>
> >>
> >> The make error messages are not very helpful to me though.
> >>
> >
> > It also happened to me in a build directory whose path contains a
> > symbolic linked directory.  Make was confused to include the generated
> > makefile twice, which could be reached with 2 different pathnames.
> > After removing the symbolic linked directory, the generated makefile
> > can be reached with a single pathname, the make error went away.
>
> This is a standard Koji build environment.  The build directory is not a
> symbolic link:
>
> + readlink -f .
> /builddir/build/BUILD/glibc-2.34.9000-294-ga43c0b5483/build-x86_64-redhat-linux
>
> And we reach that via /builddir/build/BUILD.
>
> So I still don't know what's going on there.
>

In my case, the same makefile fragment was included twice.   Can you try

https://sourceware.org/pipermail/libc-alpha/2021-November/132795.html

-- 
H.J.

Re: [PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[Florian Weimer]

* H. J. Lu:

> In my case, the same makefile fragment was included twice.   Can you try
>
> https://sourceware.org/pipermail/libc-alpha/2021-November/132795.html

Do you have an updated version?  It doesn't apply anymore.

Thanks,
Florian

Re: [PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[H.J. Lu]

[-- Attachment #1: Type: text/plain, Size: 336 bytes --]

On Wed, Nov 17, 2021 at 7:52 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * H. J. Lu:
>
> > In my case, the same makefile fragment was included twice.   Can you try
> >
> > https://sourceware.org/pipermail/libc-alpha/2021-November/132795.html
>
> Do you have an updated version?  It doesn't apply anymore.
>

Here it is.

-- 
H.J.

[-- Attachment #2: 0001-elf-Pre-generate-Makefile-fragment-for-DSO-sorting-t.patch --]
[-- Type: text/x-patch, Size: 3102 bytes --]

From 66d865d6059c07e4650759e786178d7383cd7e65 Mon Sep 17 00:00:00 2001
From: "H.J. Lu" <hjl.tools@gmail.com>
Date: Fri, 5 Nov 2021 12:20:01 -0700
Subject: [PATCH] elf: Pre-generate Makefile fragment for DSO sorting tests [BZ
 #28550]

Generate Makefile fragment for DSO sorting tests at configure time.

This partially fixes BZ #28550.
---
 configure    | 11 +++++++++++
 configure.ac | 12 ++++++++++++
 elf/Makefile | 24 ++++--------------------
 3 files changed, 27 insertions(+), 20 deletions(-)

diff --git a/configure b/configure
index 2f9adca064..91993da3b9 100755
--- a/configure
+++ b/configure
@@ -8484,3 +8484,14 @@ if test -n "$ac_unrecognized_opts" && test "$enable_option_checking" != no; then
 $as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2;}
 fi
 
+
+if test "$have_tunables" = yes; then
+  test -d elf || mkdir elf
+  objdir=`pwd`
+  rm -f elf/dso-sort-tests.mk
+  (for d in dso-sort-tests-1.def dso-sort-tests-2.def
+   do
+    $PYTHON $srcdir/scripts/dso-ordering-test.py \
+	--description-file $srcdir/elf/$d --objpfx $objdir/elf/
+   done) > elf/dso-sort-tests.mk
+fi
diff --git a/configure.ac b/configure.ac
index 7eb4239359..faf70b930c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1901,3 +1901,15 @@ echo "$config_vars" >> config.make;;
 esac
 test -d bits || mkdir bits]],[[config_vars='$config_vars']])
 AC_OUTPUT
+
+dnl Generate Makefile fragments for DSO sorting tests.
+if test "$have_tunables" = yes; then
+  test -d elf || mkdir elf
+  objdir=`pwd`
+  rm -f elf/dso-sort-tests.mk
+  (for d in dso-sort-tests-1.def dso-sort-tests-2.def
+   do
+    $PYTHON $srcdir/scripts/dso-ordering-test.py \
+	--description-file $srcdir/elf/$d --objpfx $objdir/elf/
+   done) > elf/dso-sort-tests.mk
+fi
diff --git a/elf/Makefile b/elf/Makefile
index 2365c0c863..17acd4a5ab 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -491,29 +491,13 @@ tests-special += $(objpfx)order-cmp.out $(objpfx)tst-array1-cmp.out \
 		 $(objpfx)tst-unused-dep-cmp.out
 endif
 
-ifndef avoid-generated
-# DSO sorting tests:
-# The dso-ordering-test.py script generates testcase source files in $(objpfx),
-# creating a $(objpfx)<testcase-name>-dir for each testcase, and creates a
-# Makefile fragment to be included.
-define include_dsosort_tests
-$(objpfx)$(1).generated-makefile: $(1)
-	$(PYTHON) $(..)scripts/dso-ordering-test.py \
-	--description-file $$< --objpfx $(objpfx) --output-makefile $$@T
-	mv $$@T $$@
--include $(objpfx)$(1).generated-makefile
-endef
-endif
-
-postclean-generated += $(objpfx)/dso-sort-tests-2.generated-makefile \
-		       $(objpfx)/dso-sort-tests-2.generated-makefile
-
-# Generate from each testcase description file
 ifeq (yes,$(have-tunables))
-$(eval $(call include_dsosort_tests,dso-sort-tests-1.def))
-$(eval $(call include_dsosort_tests,dso-sort-tests-2.def))
+# Include Makefile fragments for DSO sorting tests.
+include $(objpfx)dso-sort-tests.mk
 endif
 
+postclean-generated += $(objpfx)/dso-sort-tests.mk
+
 check-abi: $(objpfx)check-abi-ld.out
 tests-special += $(objpfx)check-abi-ld.out
 update-abi: update-abi-ld
-- 
2.33.1

Re: [PATCH v2] elf: Use a temporary file to generate Makefile fragments [BZ #28550]

[Florian Weimer]

* H. J. Lu:

> On Wed, Nov 17, 2021 at 7:52 AM Florian Weimer <fweimer@redhat.com> wrote:
>>
>> * H. J. Lu:
>>
>> > In my case, the same makefile fragment was included twice.   Can you try
>> >
>> > https://sourceware.org/pipermail/libc-alpha/2021-November/132795.html
>>
>> Do you have an updated version?  It doesn't apply anymore.
>>
>
> Here it is.

The build finished with the patch:

  <https://koji.fedoraproject.org/koji/taskinfo?taskID=78992550>

I haven't looked at the logs yet.

Thanks,
Florian