[PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Daniel Santos]

It took me a while to figure out how to do this so I figured that it should be
in the docs.  OK for trunk?

	* doc/install.texi: Add more details on selecting multiple tests.

Thanks,
Daniel

Signed-off-by: Daniel Santos <daniel.santos@pobox.com>
---
 gcc/doc/install.texi | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/gcc/doc/install.texi b/gcc/doc/install.texi
index 7c9e2f25d44..6aefd213901 100644
--- a/gcc/doc/install.texi
+++ b/gcc/doc/install.texi
@@ -2737,6 +2737,16 @@ the testsuite with filenames matching @samp{9805*}, you would use
 make check-g++ RUNTESTFLAGS="old-deja.exp=9805* @var{other-options}"
 @end smallexample
 
+The file-matching expression following @var{filename}@command{.exp=} is treated
+as a series of whitespace-delimited glob expressions so that multiple patterns
+may be passed, although any whitespace must either be escaped or surrounded by
+tick marks if multiple expressions are desired. For example,
+
+@smallexample
+make check-g++ RUNTESTFLAGS="old-deja.exp=9805*\ virtual2.c @var{other-options}"
+make check-g++ RUNTESTFLAGS="'old-deja.exp=9805* virtual2.c' @var{other-options}"
+@end smallexample
+
 The @file{*.exp} files are located in the testsuite directories of the GCC
 source, the most important ones being @file{compile.exp},
 @file{execute.exp}, @file{dg.exp} and @file{old-deja.exp}.
-- 
2.13.3

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Martin Sebor]

On 08/21/2017 07:41 PM, Daniel Santos wrote:
> It took me a while to figure out how to do this so I figured that it should be
> in the docs.  OK for trunk?
>
> 	* doc/install.texi: Add more details on selecting multiple tests.

Thank you!  It had taken me some time to figure this out.

> +The file-matching expression following @var{filename}@command{.exp=} is treated
> +as a series of whitespace-delimited glob expressions so that multiple patterns
> +may be passed, although any whitespace must either be escaped or surrounded by
> +tick marks if multiple expressions are desired. For example,

Do you mean single quotes?  I would suggest "escaped or quoted."
The whole argument to RUNTESTFLAGS can be quoted in either single
or double quotes and, AFAICT, so can the space-separated test
names within it.

Martin

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Mike Stump]

> On Aug 22, 2017, at 8:58 AM, Martin Sebor <msebor@gmail.com> wrote:
> 
> On 08/21/2017 07:41 PM, Daniel Santos wrote:
>> It took me a while to figure out how to do this so I figured that it should be
>> in the docs.  OK for trunk?
>> 
>> 	* doc/install.texi: Add more details on selecting multiple tests.
> 
> Thank you!  It had taken me some time to figure this out.
> 
>> +The file-matching expression following @var{filename}@command{.exp=} is treated
>> +as a series of whitespace-delimited glob expressions so that multiple patterns
>> +may be passed, although any whitespace must either be escaped or surrounded by
>> +tick marks if multiple expressions are desired. For example,

> Do you mean single quotes?  I would suggest "escaped or quoted."


Yes, I agree.  Please, no tick, slightly too informal.

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Mike Stump]

On Aug 22, 2017, at 8:58 AM, Martin Sebor <msebor@gmail.com> wrote:
> 
> On 08/21/2017 07:41 PM, Daniel Santos wrote:
>> It took me a while to figure out how to do this so I figured that it should be
>> in the docs.  OK for trunk?

Oh, yeah, with the correction mentioned, Ok.

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Daniel Santos]

On 08/22/2017 10:58 AM, Martin Sebor wrote:
> On 08/21/2017 07:41 PM, Daniel Santos wrote:
>> It took me a while to figure out how to do this so I figured that it
>> should be
>> in the docs.  OK for trunk?
>>
>>     * doc/install.texi: Add more details on selecting multiple tests.
>
> Thank you!  It had taken me some time to figure this out.
>
>> +The file-matching expression following @var{filename}@command{.exp=}
>> is treated
>> +as a series of whitespace-delimited glob expressions so that
>> multiple patterns
>> +may be passed, although any whitespace must either be escaped or
>> surrounded by
>> +tick marks if multiple expressions are desired. For example,
>
> Do you mean single quotes?

Yes.  I guess I've heard the terms "tick marks" and "single quotes" used
before.  Perhaps using 'single quotes' would be a good way to express it
(with the quotes).

>   I would suggest "escaped or quoted."
> The whole argument to RUNTESTFLAGS can be quoted in either single
> or double quotes and, AFAICT, so can the space-separated test
> names within it.

Well, mysteriously, double quotes do not work.  So if I pass
RUNTESTFLAGS='"i386.exp=pr80969-[12]*.c pr80969-4.c"' then the second
pattern isn't used.  I have NO idea what happens to it because it I pass
RUNTESTFLAGS='i386.exp=pr80969-[12]*.c pr80969-4.c' then runtest
properly demands that I tell it what in the hell pr80969-4.c is supposed
to mean.  As an experiment, I created a symlink named \"pr80969-4.c and
using RUNTESTFLAGS='"i386.exp=pr80969-[12]*.c "pr80969-4.c' but it
didn't pick it up.  This is probably JAB (just another bug) in DejaGNU.

Among the variations I've tried are enclosing the expressions in
{braces},  \{escaped braces\} and comma-delimited \{escaped,braces\},
but none of these worked.

Daniel

> Martin
>

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Mike Stump]

On Aug 22, 2017, at 10:32 AM, Daniel Santos <daniel.santos@pobox.com> wrote:
> 
>>  I would suggest "escaped or quoted."
>> The whole argument to RUNTESTFLAGS can be quoted in either single
>> or double quotes and, AFAICT, so can the space-separated test
>> names within it.
> 
> Well, mysteriously, double quotes do not work.

Did you try the obvious:

"\"pdf pdf\" pdf"

?  I think it should work fine.

[PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Daniel Santos]

OK, how's this one?

	* doc/install.texi: Modify to add more details on running
	selected tests.

Thanks,
Daniel

Signed-off-by: Daniel Santos <daniel.santos@pobox.com>
---
 gcc/doc/install.texi | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/gcc/doc/install.texi b/gcc/doc/install.texi
index 7c9e2f25d44..da360da1c50 100644
--- a/gcc/doc/install.texi
+++ b/gcc/doc/install.texi
@@ -2737,6 +2737,16 @@ the testsuite with filenames matching @samp{9805*}, you would use
 make check-g++ RUNTESTFLAGS="old-deja.exp=9805* @var{other-options}"
 @end smallexample
 
+The file-matching expression following @var{filename}@command{.exp=} is treated
+as a series of whitespace-delimited glob expressions so that multiple patterns
+may be passed, although any whitespace must either be escaped or surrounded by
+single quotes if multiple expressions are desired. For example,
+
+@smallexample
+make check-g++ RUNTESTFLAGS="old-deja.exp=9805*\ virtual2.c @var{other-options}"
+make check-g++ RUNTESTFLAGS="'old-deja.exp=9805* virtual2.c' @var{other-options}"
+@end smallexample
+
 The @file{*.exp} files are located in the testsuite directories of the GCC
 source, the most important ones being @file{compile.exp},
 @file{execute.exp}, @file{dg.exp} and @file{old-deja.exp}.
-- 
2.13.3

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Mike Stump]

On Aug 22, 2017, at 10:44 AM, Daniel Santos <daniel.santos@pobox.com> wrote:
> 
> OK, how's this one?

Ok.

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Daniel Santos]

On 08/22/2017 12:32 PM, Mike Stump wrote:
> On Aug 22, 2017, at 10:32 AM, Daniel Santos <daniel.santos@pobox.com> wrote:
>>>  I would suggest "escaped or quoted."
>>> The whole argument to RUNTESTFLAGS can be quoted in either single
>>> or double quotes and, AFAICT, so can the space-separated test
>>> names within it.
>> Well, mysteriously, double quotes do not work.
> Did you try the obvious:
>
> "\"pdf pdf\" pdf"
>
> ?  I think it should work fine.

Yes.  As I explained in the rest of my email I tried a great many
variations.  I can debug runtest some more and try to better understand
how this is getting parsed.

Daniel

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Daniel Santos]

On 08/22/2017 12:32 PM, Mike Stump wrote:
> On Aug 22, 2017, at 10:32 AM, Daniel Santos <daniel.santos@pobox.com> wrote:
>>>  I would suggest "escaped or quoted."
>>> The whole argument to RUNTESTFLAGS can be quoted in either single
>>> or double quotes and, AFAICT, so can the space-separated test
>>> names within it.
>> Well, mysteriously, double quotes do not work.
> Did you try the obvious:
>
> "\"pdf pdf\" pdf"
>
> ?  I think it should work fine.

I have found one additional working mechanism:

RUNTESTFLAGS='i386.exp=\"pr80969-[12]*.c pr80969-4.c\"'

But using double quotes for both does NOT work:

RUNTESTFLAGS="i386.exp=\"pr80969-[12]*.c pr80969-4.c\""

So the three working options appears to be:
1. Escaping whitespace
2. Using double quotes for the whole value and single quotes for the
file.exp=patterns expression
3. Using single quotes for the whole value and double quotes for the
file.exp=patterns expression

Daniel

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Daniel Santos]

OK, the problem is at line 4014 of gcc/Makefile.in:

          $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)"
RUNTESTFLAGS="$(RUNTESTFLAGS)" \
            check-parallel-$* \
 
Even worse, one can inject arbitrary shell commands here, not that I can
think of a scenario where it would be an actual security problem:

RUNTESTFLAGS="i386.exp=a b\"; beep\"" check-c

I presume that the solution would be to re-escape the contents of
RUNTESTFLAGS.

Daniel

Re: [PATCH] [docs] Explain how to use multiple file-name patterns in RUNTESTFLAGS

[Mike Stump]

On Aug 22, 2017, at 11:53 AM, Daniel Santos <daniel.santos@pobox.com> wrote:
> 
> OK, the problem is at line 4014 of gcc/Makefile.in:
> 
>          $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)"
> RUNTESTFLAGS="$(RUNTESTFLAGS)" \
>            check-parallel-$* \

So, this is typical of sh scripting.  Most kids don't quote and know how to quote in other than trivial cases.  It is one of the reasons why scripting is both better and worse.  sh from day 1 should have had a quote function that would quote the operand, it doesn't.  If it did, we'd put $(quote ...) in there instead of "..." and it would just work.

> I presume that the solution would be to re-escape the contents of
> RUNTESTFLAGS.

Yes.  The annoyance factor is so high, that no one ever does.  Feel free rot ignore the problem.