[PATCH] test-in-container: Install locales into the test container

[PATCH] test-in-container: Install locales into the test container

[Florian Weimer]

The testing time impact of this change is significant: It increases
total test time from ~13 to ~15 minutes for me.  However, I do not
see any other way to run locale-dependent tests in a container.

2019-07-09  Florian Weimer  <fweimer@redhat.com>

	* Makefile (testroot.pristine/install.stamp): Install locales.

diff --git a/Makefile b/Makefile
index dc5de7aa6b..23c4326980 100644
--- a/Makefile
+++ b/Makefile
@@ -402,6 +402,7 @@ ifeq ($(run-built-tests),yes)
 	  done
 endif
 	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine
+	$(MAKE) localedata/install-locales DESTDIR=$(objpfx)testroot.pristine
 	touch $(objpfx)testroot.pristine/install.stamp
 
 tests-special-notdir = $(patsubst $(objpfx)%, %, $(tests-special))

Re: [PATCH] test-in-container: Install locales into the test container

[DJ Delorie]

Florian Weimer <fweimer@redhat.com> writes:
> The testing time impact of this change is significant: It increases
> total test time from ~13 to ~15 minutes for me.  However, I do not
> see any other way to run locale-dependent tests in a container.

This is also the only way to auto-test "make install-locales" too :-)

>  	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine
> +	$(MAKE) localedata/install-locales DESTDIR=$(objpfx)testroot.pristine
>  	touch $(objpfx)testroot.pristine/install.stamp

This matches what's documented in INSTALL so LGTM.

Reviewed-by: DJ Delorie <dj@redhat.com>

Re: [PATCH] test-in-container: Install locales into the test container

[Florian Weimer]

* DJ Delorie:

> Florian Weimer <fweimer@redhat.com> writes:
>> The testing time impact of this change is significant: It increases
>> total test time from ~13 to ~15 minutes for me.  However, I do not
>> see any other way to run locale-dependent tests in a container.
>
> This is also the only way to auto-test "make install-locales" too :-)
>
>>  	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine
>> +	$(MAKE) localedata/install-locales DESTDIR=$(objpfx)testroot.pristine
>>  	touch $(objpfx)testroot.pristine/install.stamp
>
> This matches what's documented in INSTALL so LGTM.
>
> Reviewed-by: DJ Delorie <dj@redhat.com>

The problem is that the delay also applies to partial runs of the test
suite:

$ time make -j8 subdirs=resolv check

goes from:

real    0m8.792s
user    0m15.486s
sys     0m6.017s

to:

real    2m5.588s
user    13m20.667s
sys     0m44.631s

Compare this to the time of a clean glibc build on this machine:

real    1m42.489s
user    8m20.556s
sys     2m6.281s

This is a bit embarrassing because the pristine container constructed in
this way is not even correct (bug 24794).

I think applying this patch as-is would make glibc development
substantially more difficult for me.

Thanks,
Florian

Re: [PATCH] test-in-container: Install locales into the test container

[DJ Delorie]

Florian Weimer <fweimer@redhat.com> writes:
> The problem is that the delay also applies to partial runs of the test
> suite:

The first time, or each other time?

> $ time make -j8 subdirs=resolv check

This does (should :) create the pristine testroot, but only once.

Re: [PATCH] test-in-container: Install locales into the test container

[Florian Weimer]

* DJ Delorie:

> Florian Weimer <fweimer@redhat.com> writes:
>> The problem is that the delay also applies to partial runs of the test
>> suite:
>
> The first time, or each other time?

The first time.  It's still bad in some cases (depending on what I'm
working).

>> $ time make -j8 subdirs=resolv check
>
> This does (should :) create the pristine testroot, but only once.

Yes, and its contents stays incomplete.

Thanks,
Florian

Re: [PATCH] test-in-container: Install locales into the test container

[Carlos O'Donell]

On 7/9/19 4:52 PM, Florian Weimer wrote:
> I think applying this patch as-is would make glibc development
> substantially more difficult for me.

Where do you draw the line? Eventually the testing is going to
slow down your edit/build/test cycle. If this is a problem for
developers then we need some kind of objective "line" that we
try not to cross.

Should we have a staggered approach?

make check-sanity
  -- Run as many tests as fit into 10s.
  -- Run all tests which are not support_become_root,
     test-in-container, or xcheck tests, since these
     can be expensive.

make check
  -- No time limit for testing.
  -- Run all tests which are not xcheck tests.

make xhceck
  -- No time limit for testing.
  -- Run all tests including xcheck tests.

-- 
Cheers,
Carlos.

Partial test suite run builds corrupt test-in-container testroot

[Carlos O'Donell]

On 7/9/19 5:02 PM, Florian Weimer wrote:
> * DJ Delorie:
>>> $ time make -j8 subdirs=resolv check
>>
>> This does (should :) create the pristine testroot, but only once.
> 
> Yes, and its contents stays incomplete.

I'm changing the subject because this is a distinct htread.

Thanks for filling this:
https://sourceware.org/bugzilla/show_bug.cgi?id=24794

We should work to solve this, since it's a use case that many
developers expect to work.

-- 
Cheers,
Carlos.

Re: [PATCH] test-in-container: Install locales into the test container

[DJ Delorie]

"Carlos O'Donell" <carlos@redhat.com> writes:
> Where do you draw the line?

Originally I wanted the "make install" to trigger just before the first
test that actually needed it, but I found it very difficult to do that.
Eventually more and more tests will run inside the container, and that
optimization would be useless anyway.

Perhaps we could at least add $(PARALLELMFLAGS) to the "make install" ?
If it's not already propogated by some other means?  Or otherwise
optimize the install process, to limit the overhead during testing?

Re: Partial test suite run builds corrupt test-in-container testroot

[DJ Delorie]

I did a "make" followed by "make libio/tests" and compared with a "make;
make check" and they produced identical testroots (including iconvconfig).

I tried a "./configure; make libio/tests" and it failed pretty quickly
(shell-container not found) since support/ hadn't been built.

Please provide more detailed reproduction instructions, as I don't see a
"corrupt testroot" yet.

Re: Partial test suite run builds corrupt test-in-container testroot

[Florian Weimer]

* DJ Delorie:

> I did a "make" followed by "make libio/tests" and compared with a "make;
> make check" and they produced identical testroots (including iconvconfig).
>
> I tried a "./configure; make libio/tests" and it failed pretty quickly
> (shell-container not found) since support/ hadn't been built.
>
> Please provide more detailed reproduction instructions, as I don't see a
> "corrupt testroot" yet.

You need to build the tree first before running the test suite subset.
For example:

  make -j8
  make -j8 check subdirs=libio

Thanks,
Florian

Re: Partial test suite run builds corrupt test-in-container testroot

[DJ Delorie]

Florian Weimer <fweimer@redhat.com> writes:
> You need to build the tree first before running the test suite subset.
> For example:
>
>   make -j8
>   make -j8 check subdirs=libio

Yup, as I said:

>> I did a "make" followed by "make libio/tests" and compared with a "make;
>> make check" and they produced identical testroots (including iconvconfig).

Re: Partial test suite run builds corrupt test-in-container testroot

[Florian Weimer]

* DJ Delorie:

> Florian Weimer <fweimer@redhat.com> writes:
>> You need to build the tree first before running the test suite subset.
>> For example:
>>
>>   make -j8
>>   make -j8 check subdirs=libio
>
> Yup, as I said:
>
>>> I did a "make" followed by "make libio/tests" and compared with a "make;
>>> make check" and they produced identical testroots (including iconvconfig).

Huh.  What's your make version?

Thanks,
Florian

Re: Partial test suite run builds corrupt test-in-container testroot

[DJ Delorie]

Florian Weimer <fweimer@redhat.com> writes:
> Huh.  What's your make version?

4.2.1 - Fedora 30

Re: [PATCH] test-in-container: Install locales into the test container

[Carlos O'Donell]

On 7/9/19 7:54 PM, DJ Delorie wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
>> Where do you draw the line?
> 
> Originally I wanted the "make install" to trigger just before the first
> test that actually needed it, but I found it very difficult to do that.
> Eventually more and more tests will run inside the container, and that
> optimization would be useless anyway.
> 
> Perhaps we could at least add $(PARALLELMFLAGS) to the "make install" ?
> If it's not already propogated by some other means?  Or otherwise
> optimize the install process, to limit the overhead during testing?

My point is that no matter how good you optimize, eventually there will
be enough tests that the time will creep up again.

I want Florian to be productive, and I want a build system that supports
a straight forward edit/build/test workflow.

My question is: How long is too long to wait for the results of testing
when you are doing a edit/build/test/debug cycle?

Should 'make check' be running enough tests to fill X seconds?

-- 
Cheers,
Carlos.

Re: [PATCH] test-in-container: Install locales into the test container

[Florian Weimer]

* DJ Delorie:

> Perhaps we could at least add $(PARALLELMFLAGS) to the "make install" ?
> If it's not already propogated by some other means?  Or otherwise
> optimize the install process, to limit the overhead during testing?

$(MAKE) already implies $(PARALLELMFLAGS).

Thanks,
Florian

Re: Partial test suite run builds corrupt test-in-container testroot

[Florian Weimer]

* DJ Delorie:

> Florian Weimer <fweimer@redhat.com> writes:
>> Huh.  What's your make version?
>
> 4.2.1 - Fedora 30

Here's what I did.  I got a box from Beaker with Fedora 30.

# dnf install screen
# dnf download --source glibc
# dnf builddep glibc-*.src.rpm
# screen
# dnf install git
# git clone --depth 1 git://sourceware.org/git/glibc.git git
# mkdir build
# cd build
# ../git/configure --prefix=/usr
# time nohup make -j24
# time nohup make -j24 subdirs=libio check

The output ends with:

FAIL: libio/tst-wfile-ascii
Summary of test results:
      1 FAIL
     87 PASS
make[1]: *** [Makefile:411: tests] Error 1
make[1]: Leaving directory '/root/git'
make: *** [Makefile:9: check] Error 2

The test log contains:

tst-wfile-ascii.c:37: numeric comparison failure
   left: 256 (0x100); from: system (iconvconfig)
  right: 0 (0x0); from: 0
error: 1 test failures

Locale settings (inherited via SSH):

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=en_US.UTF-8
LC_TIME=en_US.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=en_US.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=en_US.UTF-8
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT=en_US.UTF-8
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Root vs non-root doesn't make a difference.

Thanks,
Florian

[PATCH] test-container: Install with $(all-subdirs) [BZ #24794]

[Tulio Magno Quites Machado Filho]

Whenever a sub-make is created, it inherits the variable subdirs from its
parent.  This is also true when make check is called with a restricted
list of subdirs.  In this scenario, make install is executed "partially"
and testroot.pristine ends up with an incomplete installation.

2019-07-22  Tulio Magno Quites Machado Filho  <tuliom@linux.ibm.com>

	[BZ #24794]
	* Makefile (testroot.pristine/install.stamp): Pass
	subdirs='$(all-subdirs)' to make install.
---
 Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index a4ed747cef..9fbf705200 100644
--- a/Makefile
+++ b/Makefile
@@ -401,7 +401,8 @@ ifeq ($(run-built-tests),yes)
 	    $(test-wrapper) cp $$dso $(objpfx)testroot.pristine$$dso ;\
 	  done
 endif
-	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine
+	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine \
+	  subdirs='$(all-subdirs)'
 	touch $(objpfx)testroot.pristine/install.stamp
 
 tests-special-notdir = $(patsubst $(objpfx)%, %, $(tests-special))
-- 
2.14.5

Re: [PATCH] test-container: Install with $(all-subdirs) [BZ #24794]

[DJ Delorie]

Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com> writes:
> Whenever a sub-make is created, it inherits the variable subdirs from its
> parent.  This is also true when make check is called with a restricted
> list of subdirs.  In this scenario, make install is executed "partially"
> and testroot.pristine ends up with an incomplete installation.

"obvious in retrospect"  Sigh.

> 2019-07-22  Tulio Magno Quites Machado Filho  <tuliom@linux.ibm.com>
>
> 	[BZ #24794]
> 	* Makefile (testroot.pristine/install.stamp): Pass
> 	subdirs='$(all-subdirs)' to make install.

LGTM.  Thanks!

Reviewed-by: DJ Delorie <dj@redhat.com>

Re: [PATCH] test-container: Install with $(all-subdirs) [BZ #24794]

[Tulio Magno Quites Machado Filho]

DJ Delorie <dj@redhat.com> writes:

> Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com> writes:
>> Whenever a sub-make is created, it inherits the variable subdirs from its
>> parent.  This is also true when make check is called with a restricted
>> list of subdirs.  In this scenario, make install is executed "partially"
>> and testroot.pristine ends up with an incomplete installation.
>
> "obvious in retrospect"  Sigh.
>
>> 2019-07-22  Tulio Magno Quites Machado Filho  <tuliom@linux.ibm.com>
>>
>> 	[BZ #24794]
>> 	* Makefile (testroot.pristine/install.stamp): Pass
>> 	subdirs='$(all-subdirs)' to make install.
>
> LGTM.  Thanks!
>
> Reviewed-by: DJ Delorie <dj@redhat.com>

Pushed as 35e038c1d2ccb.

Thanks!

-- 
Tulio Magno

Re: [PATCH] test-container: Install with $(all-subdirs) [BZ #24794]

[Stefan Liebler]

Hi,

starting with this commit, I see the following fail:
cat nss/tst-nss-files-hosts-long.out
error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
error: 1 test failures

Does anybody else also see this fail?

While running "make check" I've recognized the output of 
support/shell-container.c:
sh: execing getent failed: No such file or directory File
sh: execing /usr/bin/makedb failed: No such file or directory File

And indeed getent does not exist in <build-dir>/testroot.root or 
<build-dir>/testroot.pristine. Without this commit, it was available.

In older logs, I've found these commands:
/usr/bin/install -c build/nss/getent 
build/testroot.pristine/usr/bin/getent.new
/usr/bin/install -c build/nss/makedb 
build/testroot.pristine/usr/bin/makedb.new
mv -f build/testroot.pristine/usr/bin/getent.new 
build/testroot.pristine/usr/bin/getent
mv -f build/testroot.pristine/usr/bin/makedb.new 
build/testroot.pristine/usr/bin/makedb

Can anybody help?

Bye
Stefan

On 7/23/19 6:29 PM, Tulio Magno Quites Machado Filho wrote:
> DJ Delorie <dj@redhat.com> writes:
> 
>> Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com> writes:
>>> Whenever a sub-make is created, it inherits the variable subdirs from its
>>> parent.  This is also true when make check is called with a restricted
>>> list of subdirs.  In this scenario, make install is executed "partially"
>>> and testroot.pristine ends up with an incomplete installation.
>>
>> "obvious in retrospect"  Sigh.
>>
>>> 2019-07-22  Tulio Magno Quites Machado Filho  <tuliom@linux.ibm.com>
>>>
>>> 	[BZ #24794]
>>> 	* Makefile (testroot.pristine/install.stamp): Pass
>>> 	subdirs='$(all-subdirs)' to make install.
>>
>> LGTM.  Thanks!
>>
>> Reviewed-by: DJ Delorie <dj@redhat.com>
> 
> Pushed as 35e038c1d2ccb.
> 
> Thanks!
> 

Re: [PATCH] test-container: Install with $(all-subdirs) [BZ #24794]

[Tulio Magno Quites Machado Filho]

Stefan Liebler <stli@linux.ibm.com> writes:

> starting with this commit, I see the following fail:
> cat nss/tst-nss-files-hosts-long.out
> error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
> error: 1 test failures
>
> Does anybody else also see this fail?

I reproduced it here.
I'm sorry, I thought it was a bug hidden by the bug fixed in my previous
patch.

> While running "make check" I've recognized the output of 
> support/shell-container.c:
> sh: execing getent failed: No such file or directory File
> sh: execing /usr/bin/makedb failed: No such file or directory File
>
> And indeed getent does not exist in <build-dir>/testroot.root or 
> <build-dir>/testroot.pristine. Without this commit, it was available.
>
> In older logs, I've found these commands:
> /usr/bin/install -c build/nss/getent 
> build/testroot.pristine/usr/bin/getent.new
> /usr/bin/install -c build/nss/makedb 
> build/testroot.pristine/usr/bin/makedb.new
> mv -f build/testroot.pristine/usr/bin/getent.new 
> build/testroot.pristine/usr/bin/getent
> mv -f build/testroot.pristine/usr/bin/makedb.new 
> build/testroot.pristine/usr/bin/makedb
>
> Can anybody help?

I'll take a look and see what's missing.

-- 
Tulio Magno

Re: [PATCH] test-container: Install with $(all-subdirs) [BZ #24794]

[Carlos O'Donell]

On 7/24/19 11:32 AM, Tulio Magno Quites Machado Filho wrote:
> Stefan Liebler <stli@linux.ibm.com> writes:
> 
>> starting with this commit, I see the following fail:
>> cat nss/tst-nss-files-hosts-long.out
>> error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
>> error: 1 test failures
>>
>> Does anybody else also see this fail?
> 
> I reproduced it here.
> I'm sorry, I thought it was a bug hidden by the bug fixed in my previous
> patch.
> 
>> While running "make check" I've recognized the output of
>> support/shell-container.c:
>> sh: execing getent failed: No such file or directory File
>> sh: execing /usr/bin/makedb failed: No such file or directory File
>>
>> And indeed getent does not exist in <build-dir>/testroot.root or
>> <build-dir>/testroot.pristine. Without this commit, it was available.
>>
>> In older logs, I've found these commands:
>> /usr/bin/install -c build/nss/getent
>> build/testroot.pristine/usr/bin/getent.new
>> /usr/bin/install -c build/nss/makedb
>> build/testroot.pristine/usr/bin/makedb.new
>> mv -f build/testroot.pristine/usr/bin/getent.new
>> build/testroot.pristine/usr/bin/getent
>> mv -f build/testroot.pristine/usr/bin/makedb.new
>> build/testroot.pristine/usr/bin/makedb
>>
>> Can anybody help?
> 
> I'll take a look and see what's missing.

Regression tests catching regressions! ;-)

-- 
Cheers,
Carlos.

Re: [PATCH] test-container: Install with $(all-subdirs) [BZ #24794]

[DJ Delorie]

Tulio Magno Quites Machado Filho <tuliom@ascii.art.br> writes:
> I'll take a look and see what's missing.

How about this?  The initial setting of subdirs is from Makeconfig:

subdirs = $(sorted-subdirs)

With this change, tst-nss-files-hosts-long passes, where
it failed before with a clean testroot install.

diff --git a/Makefile b/Makefile
index 9fbf705200..ac1125853b 100644
--- a/Makefile
+++ b/Makefile
@@ -402,7 +402,7 @@ ifeq ($(run-built-tests),yes)
 	  done
 endif
 	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine \
-	  subdirs='$(all-subdirs)'
+	  subdirs='$(sorted-subdirs)'
 	touch $(objpfx)testroot.pristine/install.stamp
 
 tests-special-notdir = $(patsubst $(objpfx)%, %, $(tests-special))

[PATCH] test-container: Install with $(sorted-subdirs) [BZ #24794]

[Tulio Magno Quites Machado Filho]

Commit 35e038c1d2ccb3a75395662f9c4f28d85a61444f started to use an
incomplete list of subdirs based on $(all-subdirs) causing
testroot.pristine to miss files from nss.

Tested if the list of files in testroot.pristine remains the same.

2019-07-24  Tulio Magno Quites Machado Filho  <tuliom@linux.ibm.com>

	[BZ #24794]
	* Makeconfig (all-subdirs): Improved source comments.
	* Makefile (testroot.pristine/install.stamp): Pass
	subdirs='$(sorted-subdirs)' to make install.
---
 Makeconfig | 6 +++---
 Makefile   | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/Makeconfig b/Makeconfig
index 0e386fbc19..fd36c58c04 100644
--- a/Makeconfig
+++ b/Makeconfig
@@ -1267,9 +1267,9 @@ else
 libsupport = $(common-objpfx)support/libsupport.a
 endif
 
-# These are the subdirectories containing the library source.  The order
-# is more or less arbitrary.  The sorting step will take care of the
-# dependencies.
+# This is a partial list of subdirectories containing the library source.
+# The order is more or less arbitrary.  The sorting step will take care of the
+# dependencies and generate sorted-subdirs dynamically.
 all-subdirs = csu assert ctype locale intl catgets math setjmp signal	    \
 	      stdlib stdio-common libio malloc string wcsmbs time dirent    \
 	      grp pwd posix io termios resource misc socket sysvipc gmon    \
diff --git a/Makefile b/Makefile
index 9fbf705200..ac1125853b 100644
--- a/Makefile
+++ b/Makefile
@@ -402,7 +402,7 @@ ifeq ($(run-built-tests),yes)
 	  done
 endif
 	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine \
-	  subdirs='$(all-subdirs)'
+	  subdirs='$(sorted-subdirs)'
 	touch $(objpfx)testroot.pristine/install.stamp
 
 tests-special-notdir = $(patsubst $(objpfx)%, %, $(tests-special))
-- 
2.14.5

Re: [PATCH] test-container: Install with $(sorted-subdirs) [BZ #24794]

[DJ Delorie]

Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com> writes:
> -	  subdirs='$(all-subdirs)'
> +	  subdirs='$(sorted-subdirs)'

Considering I posted this as a suggested fix an hour before you did, I'm
going to say LGTM.  Of course, I said that last time too, so maybe we
need an independent confirmation?  ;-)

Re: [PATCH] test-container: Install with $(sorted-subdirs) [BZ #24794]

[Szabolcs Nagy]

On 25/07/2019 01:03, Tulio Magno Quites Machado Filho wrote:
> Commit 35e038c1d2ccb3a75395662f9c4f28d85a61444f started to use an
> incomplete list of subdirs based on $(all-subdirs) causing
> testroot.pristine to miss files from nss.
> 
> Tested if the list of files in testroot.pristine remains the same.

fwiw with this patch aarch64 buildbot would go back to green.

> 
> 2019-07-24  Tulio Magno Quites Machado Filho  <tuliom@linux.ibm.com>
> 
> 	[BZ #24794]
> 	* Makeconfig (all-subdirs): Improved source comments.
> 	* Makefile (testroot.pristine/install.stamp): Pass
> 	subdirs='$(sorted-subdirs)' to make install.
> ---
>  Makeconfig | 6 +++---
>  Makefile   | 2 +-
>  2 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/Makeconfig b/Makeconfig
> index 0e386fbc19..fd36c58c04 100644
> --- a/Makeconfig
> +++ b/Makeconfig
> @@ -1267,9 +1267,9 @@ else
>  libsupport = $(common-objpfx)support/libsupport.a
>  endif
>  
> -# These are the subdirectories containing the library source.  The order
> -# is more or less arbitrary.  The sorting step will take care of the
> -# dependencies.
> +# This is a partial list of subdirectories containing the library source.
> +# The order is more or less arbitrary.  The sorting step will take care of the
> +# dependencies and generate sorted-subdirs dynamically.
>  all-subdirs = csu assert ctype locale intl catgets math setjmp signal	    \
>  	      stdlib stdio-common libio malloc string wcsmbs time dirent    \
>  	      grp pwd posix io termios resource misc socket sysvipc gmon    \
> diff --git a/Makefile b/Makefile
> index 9fbf705200..ac1125853b 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -402,7 +402,7 @@ ifeq ($(run-built-tests),yes)
>  	  done
>  endif
>  	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine \
> -	  subdirs='$(all-subdirs)'
> +	  subdirs='$(sorted-subdirs)'
>  	touch $(objpfx)testroot.pristine/install.stamp
>  
>  tests-special-notdir = $(patsubst $(objpfx)%, %, $(tests-special))
> 

Re: [PATCH] test-container: Install with $(sorted-subdirs) [BZ #24794]

[Carlos O'Donell]

On 7/25/19 8:46 AM, Szabolcs Nagy wrote:
> On 25/07/2019 01:03, Tulio Magno Quites Machado Filho wrote:
>> Commit 35e038c1d2ccb3a75395662f9c4f28d85a61444f started to use an
>> incomplete list of subdirs based on $(all-subdirs) causing
>> testroot.pristine to miss files from nss.
>>
>> Tested if the list of files in testroot.pristine remains the same.
> 
> fwiw with this patch aarch64 buildbot would go back to green.

This is good enough confirmation for me.

Please install the fix so we can get clean builds for the release.

>>
>> 2019-07-24  Tulio Magno Quites Machado Filho  <tuliom@linux.ibm.com>
>>
>> 	[BZ #24794]
>> 	* Makeconfig (all-subdirs): Improved source comments.
>> 	* Makefile (testroot.pristine/install.stamp): Pass
>> 	subdirs='$(sorted-subdirs)' to make install.
>> ---
>>   Makeconfig | 6 +++---
>>   Makefile   | 2 +-
>>   2 files changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/Makeconfig b/Makeconfig
>> index 0e386fbc19..fd36c58c04 100644
>> --- a/Makeconfig
>> +++ b/Makeconfig
>> @@ -1267,9 +1267,9 @@ else
>>   libsupport = $(common-objpfx)support/libsupport.a
>>   endif
>>   
>> -# These are the subdirectories containing the library source.  The order
>> -# is more or less arbitrary.  The sorting step will take care of the
>> -# dependencies.
>> +# This is a partial list of subdirectories containing the library source.
>> +# The order is more or less arbitrary.  The sorting step will take care of the
>> +# dependencies and generate sorted-subdirs dynamically.
>>   all-subdirs = csu assert ctype locale intl catgets math setjmp signal	    \
>>   	      stdlib stdio-common libio malloc string wcsmbs time dirent    \
>>   	      grp pwd posix io termios resource misc socket sysvipc gmon    \
>> diff --git a/Makefile b/Makefile
>> index 9fbf705200..ac1125853b 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -402,7 +402,7 @@ ifeq ($(run-built-tests),yes)
>>   	  done
>>   endif
>>   	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine \
>> -	  subdirs='$(all-subdirs)'
>> +	  subdirs='$(sorted-subdirs)'
>>   	touch $(objpfx)testroot.pristine/install.stamp
>>   
>>   tests-special-notdir = $(patsubst $(objpfx)%, %, $(tests-special))
>>
> 


-- 
Cheers,
Carlos.

Re: [PATCH] test-container: Install with $(sorted-subdirs) [BZ #24794]

[Stefan Liebler]

Hi Tulio,

with this patch, testroot.pristine contains the missing getent / makedb 
executables and libnss_* libraries.
Now nss/tst-nss-files-hosts-long and all other container tests are passing.

Thanks,
Stefan

On 7/25/19 2:03 AM, Tulio Magno Quites Machado Filho wrote:
> Commit 35e038c1d2ccb3a75395662f9c4f28d85a61444f started to use an
> incomplete list of subdirs based on $(all-subdirs) causing
> testroot.pristine to miss files from nss.
> 
> Tested if the list of files in testroot.pristine remains the same.
> 
> 2019-07-24  Tulio Magno Quites Machado Filho  <tuliom@linux.ibm.com>
> 
> 	[BZ #24794]
> 	* Makeconfig (all-subdirs): Improved source comments.
> 	* Makefile (testroot.pristine/install.stamp): Pass
> 	subdirs='$(sorted-subdirs)' to make install.
> ---
>   Makeconfig | 6 +++---
>   Makefile   | 2 +-
>   2 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/Makeconfig b/Makeconfig
> index 0e386fbc19..fd36c58c04 100644
> --- a/Makeconfig
> +++ b/Makeconfig
> @@ -1267,9 +1267,9 @@ else
>   libsupport = $(common-objpfx)support/libsupport.a
>   endif
>   
> -# These are the subdirectories containing the library source.  The order
> -# is more or less arbitrary.  The sorting step will take care of the
> -# dependencies.
> +# This is a partial list of subdirectories containing the library source.
> +# The order is more or less arbitrary.  The sorting step will take care of the
> +# dependencies and generate sorted-subdirs dynamically.
>   all-subdirs = csu assert ctype locale intl catgets math setjmp signal	    \
>   	      stdlib stdio-common libio malloc string wcsmbs time dirent    \
>   	      grp pwd posix io termios resource misc socket sysvipc gmon    \
> diff --git a/Makefile b/Makefile
> index 9fbf705200..ac1125853b 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -402,7 +402,7 @@ ifeq ($(run-built-tests),yes)
>   	  done
>   endif
>   	$(MAKE) install DESTDIR=$(objpfx)testroot.pristine \
> -	  subdirs='$(all-subdirs)'
> +	  subdirs='$(sorted-subdirs)'
>   	touch $(objpfx)testroot.pristine/install.stamp
>   
>   tests-special-notdir = $(patsubst $(objpfx)%, %, $(tests-special))
> 

Re: [PATCH] test-container: Install with $(sorted-subdirs) [BZ #24794]

[Tulio Magno Quites Machado Filho]

Carlos O'Donell <carlos@redhat.com> writes:

> On 7/25/19 8:46 AM, Szabolcs Nagy wrote:
>> On 25/07/2019 01:03, Tulio Magno Quites Machado Filho wrote:
>>> Commit 35e038c1d2ccb3a75395662f9c4f28d85a61444f started to use an
>>> incomplete list of subdirs based on $(all-subdirs) causing
>>> testroot.pristine to miss files from nss.
>>>
>>> Tested if the list of files in testroot.pristine remains the same.
>> 
>> fwiw with this patch aarch64 buildbot would go back to green.
>
> This is good enough confirmation for me.
>
> Please install the fix so we can get clean builds for the release.

I've just pushed it to master.

DJ,
I took the freedom to add your name to the ChangeLog file too as you proposed
the same fix.  ;-)

Pushed as 354e4c1adddb1.

-- 
Tulio Magno

Re: [PATCH] test-container: Install with $(sorted-subdirs) [BZ #24794]

[H.J. Lu]

On Mon, Jul 29, 2019 at 6:37 AM Tulio Magno Quites Machado Filho
<tuliom@linux.ibm.com> wrote:
>
> Carlos O'Donell <carlos@redhat.com> writes:
>
> > On 7/25/19 8:46 AM, Szabolcs Nagy wrote:
> >> On 25/07/2019 01:03, Tulio Magno Quites Machado Filho wrote:
> >>> Commit 35e038c1d2ccb3a75395662f9c4f28d85a61444f started to use an
> >>> incomplete list of subdirs based on $(all-subdirs) causing
> >>> testroot.pristine to miss files from nss.
> >>>
> >>> Tested if the list of files in testroot.pristine remains the same.
> >>
> >> fwiw with this patch aarch64 buildbot would go back to green.
> >
> > This is good enough confirmation for me.
> >
> > Please install the fix so we can get clean builds for the release.
>
> I've just pushed it to master.
>
> DJ,
> I took the freedom to add your name to the ChangeLog file too as you proposed
> the same fix.  ;-)
>
> Pushed as 354e4c1adddb1.
>
> --
> Tulio Magno

I am backporting this to older release branches.

-- 
H.J.