[PATCH 0/1] Add linting checks to 'make check'

[PATCH 0/1] Add linting checks to 'make check'

[Carlos O'Donell]

We have some existing linting checks in scripts/ around the specifics of
the installed headers, but nothing specifically about the sources for
glibc. Since this is the first time I see myself plumbing $(srcdir) down
into a test, and then using it in that test,  I think we should have a
discussion around this patch to see if we as developers consider it
acceptable to lint the Makefile format in 'make check' itself. The check
needs to run the python interpreter about 200 times, but it's fairly
fast and on a slow system takes ~3s. My gut feeling is that basic
linting checks could go into 'make check', but anything more complicated
would have to live in pre-commit CI. Deciding what is "basic" is going
to have to be evaluated on an ongoing basis and with input from what
developers consider too slow or onerous for 'make check'.

I want to make the case here for basic linting because it provides
immediate and positive feedback to new contributors of the project that
they have made a mistake in writing their Makefile. This is even better
than pre-commit CI linting; which you often want to run locally anyway.

I don't think we should stop here. I think we should take Noah's
.clang-format and hammer it into a lint for 'make check' that covers all
source files that we know we want formatted a particularly way and keep
expanding that. This would likely have to run in pre-commit CI, but we
would keep all the test infrastructure in scripts/ or the project tree
itself with a special make target.

On that note, we could add a special target for this e.g. make lint,
and run that in pre-commit CI. For now I'm suggesting we adopt a small
Makefile linting pass in 'make check', but the options to change this in
the future are open.

Carlos O'Donell (1):
  Add lint-makefiles Makefile linting test.

 Makefile                  |  6 +++++
 scripts/lint-makefiles.sh | 47 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 scripts/lint-makefiles.sh

-- 
2.40.0

[PATCH 1/1] Add lint-makefiles Makefile linting test.

[Carlos O'Donell]

We add a `make check` test that lints all Makefiles in the source
directory of the glibc build. This linting test ensures that the
lines in all Makefiles will be sorted correctly as developers
creates patches.

The test adds ~3s to a test run.

No regressions on x86_64 and i686.
---
 Makefile                  |  6 +++++
 scripts/lint-makefiles.sh | 47 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 scripts/lint-makefiles.sh

diff --git a/Makefile b/Makefile
index 224c792185..93e5a60af9 100644
--- a/Makefile
+++ b/Makefile
@@ -564,6 +564,12 @@ $(objpfx)check-wrapper-headers.out: scripts/check-wrapper-headers.py $(headers)
 	  --generated $(common-generated) > $@; $(evaluate-test)
 endif # $(headers)
 
+# Lint all Makefiles including this one to check for correctly sorted lines.
+tests-special += $(objpfx)lint-makefiles.out
+$(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh
+	$(SHELL) $< "$(PYTHON)" "$(srcdir)" > $@ ; \
+	$(evaluate-test)
+
 define summarize-tests
 @grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true
 @echo "Summary of test results$2:"
diff --git a/scripts/lint-makefiles.sh b/scripts/lint-makefiles.sh
new file mode 100644
index 0000000000..cf63a4ab9f
--- /dev/null
+++ b/scripts/lint-makefiles.sh
@@ -0,0 +1,47 @@
+#!/bin/bash
+# Copyright (C) 2023 Free Software Foundation, Inc.
+# This file is part of the GNU C Library.
+
+# The GNU C Library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+
+# The GNU C Library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+
+# You should have received a copy of the GNU Lesser General Public
+# License along with the GNU C Library; if not, see
+# <https://www.gnu.org/licenses/>.
+
+# This script checks to see that all Makefiles in the source tree
+# conform to the sorted variable rules as defined by:
+# scripts/sort-makefile-lines.py.
+# Any difference is an error and should be corrected e.g. the lines
+# reordered to sort correctly.
+# The intent with this check is to ensure that changes made by
+# developers match the expected format for the project.
+
+set -e
+export LC_ALL=C
+
+tmpfile="$(mktemp)"
+
+cleanup () {
+  rm -f -- "$tmpfile"
+}
+
+trap cleanup 0
+
+PYTHON=$1
+srcdir=$2
+
+echo "Linting Makefiles:"
+echo "Check: Are all lines sorted correctly?"
+for mfile in `find "$srcdir" -name Makefile`; do
+    echo "$mfile"
+    $PYTHON "${srcdir}scripts/sort-makefile-lines.py" < "$mfile" > "$tmpfile"
+    diff -q "$mfile" "$tmpfile"
+done
-- 
2.40.0

Re: [PATCH 1/1] Add lint-makefiles Makefile linting test.

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 4105 bytes --]

Hi Carlos,

On 5/19/23 14:13, Carlos O'Donell via Libc-alpha wrote:
> We add a `make check` test that lints all Makefiles in the source
> directory of the glibc build. This linting test ensures that the
> lines in all Makefiles will be sorted correctly as developers
> creates patches.

make check is usually to test the result of a build, according the
GNU guidelines in:

<https://www.gnu.org/software/make/manual/html_node/Standard-Targets.html>

I believe linting the source code is a different thing, and so I
used a different target in the Linux man-pages Makefile: 'lint'.

So, in the man-pages project, it goes like this:

lint -> build -> check -> install

Maybe you could follow a similar scheme?  Otherwise, it's a bit
weird to run the linters _after_ building (since `make check`
requires previously building).

If you want to have a look at the Linux man-pages' makefiles,
you should start by running `make help` and `make help-variables`,
and also read the README, CONTRIBUTING (see 'Lint & check' section),
and INSTALL files.

Cheers,
Alex

> 
> The test adds ~3s to a test run.
> 
> No regressions on x86_64 and i686.
> ---
>   Makefile                  |  6 +++++
>   scripts/lint-makefiles.sh | 47 +++++++++++++++++++++++++++++++++++++++
>   2 files changed, 53 insertions(+)
>   create mode 100644 scripts/lint-makefiles.sh
> 
> diff --git a/Makefile b/Makefile
> index 224c792185..93e5a60af9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -564,6 +564,12 @@ $(objpfx)check-wrapper-headers.out: scripts/check-wrapper-headers.py $(headers)
>   	  --generated $(common-generated) > $@; $(evaluate-test)
>   endif # $(headers)
>   
> +# Lint all Makefiles including this one to check for correctly sorted lines.
> +tests-special += $(objpfx)lint-makefiles.out
> +$(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh
> +	$(SHELL) $< "$(PYTHON)" "$(srcdir)" > $@ ; \
> +	$(evaluate-test)
> +
>   define summarize-tests
>   @grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true
>   @echo "Summary of test results$2:"
> diff --git a/scripts/lint-makefiles.sh b/scripts/lint-makefiles.sh
> new file mode 100644
> index 0000000000..cf63a4ab9f
> --- /dev/null
> +++ b/scripts/lint-makefiles.sh
> @@ -0,0 +1,47 @@
> +#!/bin/bash
> +# Copyright (C) 2023 Free Software Foundation, Inc.
> +# This file is part of the GNU C Library.
> +
> +# The GNU C Library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2.1 of the License, or (at your option) any later version.
> +
> +# The GNU C Library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +# Lesser General Public License for more details.
> +
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with the GNU C Library; if not, see
> +# <https://www.gnu.org/licenses/>.
> +
> +# This script checks to see that all Makefiles in the source tree
> +# conform to the sorted variable rules as defined by:
> +# scripts/sort-makefile-lines.py.
> +# Any difference is an error and should be corrected e.g. the lines
> +# reordered to sort correctly.
> +# The intent with this check is to ensure that changes made by
> +# developers match the expected format for the project.
> +
> +set -e
> +export LC_ALL=C
> +
> +tmpfile="$(mktemp)"
> +
> +cleanup () {
> +  rm -f -- "$tmpfile"
> +}
> +
> +trap cleanup 0
> +
> +PYTHON=$1
> +srcdir=$2
> +
> +echo "Linting Makefiles:"
> +echo "Check: Are all lines sorted correctly?"
> +for mfile in `find "$srcdir" -name Makefile`; do
> +    echo "$mfile"
> +    $PYTHON "${srcdir}scripts/sort-makefile-lines.py" < "$mfile" > "$tmpfile"
> +    diff -q "$mfile" "$tmpfile"
> +done

-- 
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/1] Add lint-makefiles Makefile linting test.

[Florian Weimer]

* Alejandro Colomar:

> Hi Carlos,
>
> On 5/19/23 14:13, Carlos O'Donell via Libc-alpha wrote:
>> We add a `make check` test that lints all Makefiles in the source
>> directory of the glibc build. This linting test ensures that the
>> lines in all Makefiles will be sorted correctly as developers
>> creates patches.
>
> make check is usually to test the result of a build, according the
> GNU guidelines in:
>
> <https://www.gnu.org/software/make/manual/html_node/Standard-Targets.html>
>
> I believe linting the source code is a different thing, and so I
> used a different target in the Linux man-pages Makefile: 'lint'.
>
> So, in the man-pages project, it goes like this:
>
> lint -> build -> check -> install
>
> Maybe you could follow a similar scheme?  Otherwise, it's a bit
> weird to run the linters _after_ building (since `make check`
> requires previously building).

I think the consistency with the existing tests is more important.  The
way Carlos implemented it, it becomes part of developer workflows
automatically.

Thanks,
Florian

Re: [PATCH 1/1] Add lint-makefiles Makefile linting test.

[Carlos O'Donell]

On 5/19/23 08:45, Florian Weimer wrote:
> * Alejandro Colomar:
> 
>> Hi Carlos,
>>
>> On 5/19/23 14:13, Carlos O'Donell via Libc-alpha wrote:
>>> We add a `make check` test that lints all Makefiles in the source
>>> directory of the glibc build. This linting test ensures that the
>>> lines in all Makefiles will be sorted correctly as developers
>>> creates patches.
>>
>> make check is usually to test the result of a build, according the
>> GNU guidelines in:



>> <https://www.gnu.org/software/make/manual/html_node/Standard-Targets.html>
>>
>> I believe linting the source code is a different thing, and so I
>> used a different target in the Linux man-pages Makefile: 'lint'.
>>
>> So, in the man-pages project, it goes like this:
>>
>> lint -> build -> check -> install
>>
>> Maybe you could follow a similar scheme?  Otherwise, it's a bit
>> weird to run the linters _after_ building (since `make check`
>> requires previously building).
> 
> I think the consistency with the existing tests is more important.  The
> way Carlos implemented it, it becomes part of developer workflows
> automatically.

Correct.

The goal here is to improve the developer workflow.

The current workflow is:
* Edit sources.
* make
* make check

And adding to `make check` means we are part of that workflow and improve the
new developer experience.

Let me digress a little bit here. glibc has an 'xcheck' target which is
conceptually flawed. I say it is flawed because having two ways to test
glibc is not simple and requires using a non-standard target to add these
extra tests. We should work to merge every 'xcheck' test into 'check'
and make a single 'check' that must be run every time you are ready to
check glibc, either after developing, or before pushing to a CI/CD system.

If a build system cannot run 'check' (hardened) then the binary artifacts
should be re-testable outside of the build system in a CI/CD system that
can run those tests. We don't easily support this today, but it is IMO the
right direction to go in e.g. test existing binary artifacts + sources.

In summary:
- I do not want to complicate the developer workflow by adding another target.
- I want to reduce the number of targets and eventually get rid of xcheck.
- Expensive checks can be delegated to scripts like scripts/build-many-glibcs.py
  which can be run in CI/CD systems.

-- 
Cheers,
Carlos.

Re: [PATCH 1/1] Add lint-makefiles Makefile linting test.

[Carlos O'Donell]

On 5/19/23 08:45, Florian Weimer wrote:
> * Alejandro Colomar:
> 
>> Hi Carlos,
>>
>> On 5/19/23 14:13, Carlos O'Donell via Libc-alpha wrote:
>>> We add a `make check` test that lints all Makefiles in the source
>>> directory of the glibc build. This linting test ensures that the
>>> lines in all Makefiles will be sorted correctly as developers
>>> creates patches.
>>
>> make check is usually to test the result of a build, according the
>> GNU guidelines in:
>>
>> <https://www.gnu.org/software/make/manual/html_node/Standard-Targets.html>
>>
>> I believe linting the source code is a different thing, and so I
>> used a different target in the Linux man-pages Makefile: 'lint'.
>>
>> So, in the man-pages project, it goes like this:
>>
>> lint -> build -> check -> install
>>
>> Maybe you could follow a similar scheme?  Otherwise, it's a bit
>> weird to run the linters _after_ building (since `make check`
>> requires previously building).
> 
> I think the consistency with the existing tests is more important.  The
> way Carlos implemented it, it becomes part of developer workflows
> automatically.

In the intervening time we've already regressed sorting in elf/Makefile, which to me
indicates that this needs to be a light-weight linting check during 'make check.'

I've reposted the light-weight linting with addtional comments:
https://sourceware.org/pipermail/libc-alpha/2023-May/148634.html
https://sourceware.org/pipermail/libc-alpha/2023-May/148635.html

-- 
Cheers,
Carlos.

Re: [PATCH 0/1] Add linting checks to 'make check'

[Joseph Myers]

On Fri, 19 May 2023, Carlos O'Donell via Libc-alpha wrote:

> acceptable to lint the Makefile format in 'make check' itself. The check
> needs to run the python interpreter about 200 times, but it's fairly
> fast and on a slow system takes ~3s. My gut feeling is that basic

If something is significantly slower, I'd suggest parallelizing at the 
makefile level (not relevant for a 3s test, but certainly helps for e.g. 
the conform/ tests which take quite a lot of time if run in serial but are 
extremely parallel).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH 0/1] Add linting checks to 'make check'

[Carlos O'Donell]

On 5/19/23 12:55, Joseph Myers wrote:
> On Fri, 19 May 2023, Carlos O'Donell via Libc-alpha wrote:
> 
>> acceptable to lint the Makefile format in 'make check' itself. The check
>> needs to run the python interpreter about 200 times, but it's fairly
>> fast and on a slow system takes ~3s. My gut feeling is that basic
> 
> If something is significantly slower, I'd suggest parallelizing at the 
> makefile level (not relevant for a 3s test, but certainly helps for e.g. 
> the conform/ tests which take quite a lot of time if run in serial but are 
> extremely parallel).
 
Agreed.

Source linting is in that category.

-- 
Cheers,
Carlos.