[PATCH] gdb/testsuite: Improve testing of GDB's completion functions

[PATCH] gdb/testsuite: Improve testing of GDB's completion functions

[Bruno Larsen]

When looking at some failures of gdb.linespec/cp-completion-aliases.exp,
I noticed that when a completion test will fail, it always fails with a
timeout.  This is because most completion tests use gdb_test_multiple
and only add a check for the correct output.  This commit adds new
options for both, tab and command completion.

For command completion, the new option will check if the prompt was
printed, and fail in this case. This is enough to know that the test has
failed because the check comes after the PASS path. For tab completion,
we have to check if GDB outputted more than just the input line, because
sometimes GDB would have printed a partial line before finishing with
the correct completion.
---
 gdb/testsuite/lib/completion-support.exp | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/gdb/testsuite/lib/completion-support.exp b/gdb/testsuite/lib/completion-support.exp
index bf9c5ad352c..275f8874f15 100644
--- a/gdb/testsuite/lib/completion-support.exp
+++ b/gdb/testsuite/lib/completion-support.exp
@@ -94,6 +94,9 @@ proc test_gdb_complete_tab_none { line } {
 	-re "^$line_re$completion::bell_re$" {
 	    pass "$test"
 	}
+	-re "$line_re\[^ \]+ $" {
+	    fail "$test"
+	}
     }
 
     clear_input_line $test
@@ -108,11 +111,15 @@ proc test_gdb_complete_tab_unique { input_line complete_line_re append_char_re }
 
     set test "tab complete \"$input_line\""
     send_gdb "$input_line\t"
+    set partial_complete [string_to_regexp $input_line]
     set res 1
     gdb_test_multiple "" "$test" {
 	-re "^$complete_line_re$append_char_re$" {
 	    pass "$test"
 	}
+	-re "$partial_complete\[^ \]+ $" {
+	    fail "$test"
+	}
 	timeout {
 	    fail "$test (timeout)"
 	    set res -1
@@ -164,6 +171,9 @@ proc test_gdb_complete_tab_multiple { input_line add_completed_line \
 			}
 		    }
 		}
+		-re "${maybe_bell}\r\n.+\r\n$gdb_prompt $" {
+		    fail "$test"
+		}
 	    }
 	}
     }
@@ -191,6 +201,9 @@ proc test_gdb_complete_cmd_unique { input_line complete_line_re } {
 	-re "^$cmd_re\r\n$complete_line_re\r\n$gdb_prompt $" {
 	    pass $test
 	}
+	-re "$gdb_prompt $" {
+	    fail "$test"
+	}
     }
 }
 
@@ -217,6 +230,9 @@ proc test_gdb_complete_cmd_multiple { cmd_prefix completion_word completion_list
 	-re "^$cmd_re\r\n$expected_re$gdb_prompt $" {
 	    pass $test
 	}
+	-re "$gdb_prompt $" {
+	    fail "$test"
+	}
     }
 }
 
-- 
2.39.1

Re: [PATCH] gdb/testsuite: Improve testing of GDB's completion functions

[Tom Tromey]

>>>>> "Bruno" == Bruno Larsen via Gdb-patches <gdb-patches@sourceware.org> writes:

Bruno> When looking at some failures of gdb.linespec/cp-completion-aliases.exp,
Bruno> I noticed that when a completion test will fail, it always fails with a
Bruno> timeout.  This is because most completion tests use gdb_test_multiple
Bruno> and only add a check for the correct output.  This commit adds new
Bruno> options for both, tab and command completion.

Looks good to me.
Approved-By: Tom Tromey <tom@tromey.com>

Tom

Re: [PATCH] gdb/testsuite: Improve testing of GDB's completion functions

[Bruno Larsen]

On 24/02/2023 20:06, Tom Tromey wrote:
>>>>>> "Bruno" == Bruno Larsen via Gdb-patches <gdb-patches@sourceware.org> writes:
> Bruno> When looking at some failures of gdb.linespec/cp-completion-aliases.exp,
> Bruno> I noticed that when a completion test will fail, it always fails with a
> Bruno> timeout.  This is because most completion tests use gdb_test_multiple
> Bruno> and only add a check for the correct output.  This commit adds new
> Bruno> options for both, tab and command completion.
>
> Looks good to me.
> Approved-By: Tom Tromey <tom@tromey.com>
>
> Tom
>
Thanks, pushed!

-- 
Cheers,
Bruno

Re: [PATCH] gdb/testsuite: Improve testing of GDB's completion functions

[Tom de Vries]

On 2/22/23 10:11, Bruno Larsen via Gdb-patches wrote:
> When looking at some failures of gdb.linespec/cp-completion-aliases.exp,
> I noticed that when a completion test will fail, it always fails with a
> timeout.  This is because most completion tests use gdb_test_multiple
> and only add a check for the correct output.  This commit adds new
> options for both, tab and command completion.
> 
> For command completion, the new option will check if the prompt was
> printed, and fail in this case. This is enough to know that the test has
> failed because the check comes after the PASS path. For tab completion,
> we have to check if GDB outputted more than just the input line, because
> sometimes GDB would have printed a partial line before finishing with
> the correct completion.

This causes quite a few regressions with check-read1.

For instance:
...
(gdb) break baz(int, FAIL: gdb.cp/cpcompletion.exp: tab complete "break 
baz(int"
double) Quit^M
(gdb)
...

Thanks,
- Tom

> ---
>   gdb/testsuite/lib/completion-support.exp | 16 ++++++++++++++++
>   1 file changed, 16 insertions(+)
> 
> diff --git a/gdb/testsuite/lib/completion-support.exp b/gdb/testsuite/lib/completion-support.exp
> index bf9c5ad352c..275f8874f15 100644
> --- a/gdb/testsuite/lib/completion-support.exp
> +++ b/gdb/testsuite/lib/completion-support.exp
> @@ -94,6 +94,9 @@ proc test_gdb_complete_tab_none { line } {
>   	-re "^$line_re$completion::bell_re$" {
>   	    pass "$test"
>   	}
> +	-re "$line_re\[^ \]+ $" {
> +	    fail "$test"
> +	}
>       }
>   
>       clear_input_line $test
> @@ -108,11 +111,15 @@ proc test_gdb_complete_tab_unique { input_line complete_line_re append_char_re }
>   
>       set test "tab complete \"$input_line\""
>       send_gdb "$input_line\t"
> +    set partial_complete [string_to_regexp $input_line]
>       set res 1
>       gdb_test_multiple "" "$test" {
>   	-re "^$complete_line_re$append_char_re$" {
>   	    pass "$test"
>   	}
> +	-re "$partial_complete\[^ \]+ $" {
> +	    fail "$test"
> +	}
>   	timeout {
>   	    fail "$test (timeout)"
>   	    set res -1
> @@ -164,6 +171,9 @@ proc test_gdb_complete_tab_multiple { input_line add_completed_line \
>   			}
>   		    }
>   		}
> +		-re "${maybe_bell}\r\n.+\r\n$gdb_prompt $" {
> +		    fail "$test"
> +		}
>   	    }
>   	}
>       }
> @@ -191,6 +201,9 @@ proc test_gdb_complete_cmd_unique { input_line complete_line_re } {
>   	-re "^$cmd_re\r\n$complete_line_re\r\n$gdb_prompt $" {
>   	    pass $test
>   	}
> +	-re "$gdb_prompt $" {
> +	    fail "$test"
> +	}
>       }
>   }
>   
> @@ -217,6 +230,9 @@ proc test_gdb_complete_cmd_multiple { cmd_prefix completion_word completion_list
>   	-re "^$cmd_re\r\n$expected_re$gdb_prompt $" {
>   	    pass $test
>   	}
> +	-re "$gdb_prompt $" {
> +	    fail "$test"
> +	}
>       }
>   }
>   

Re: [PATCH] gdb/testsuite: Improve testing of GDB's completion functions

[Bruno Larsen]

On 15/07/2023 14:13, Tom de Vries wrote:
> On 2/22/23 10:11, Bruno Larsen via Gdb-patches wrote:
>> When looking at some failures of gdb.linespec/cp-completion-aliases.exp,
>> I noticed that when a completion test will fail, it always fails with a
>> timeout.  This is because most completion tests use gdb_test_multiple
>> and only add a check for the correct output.  This commit adds new
>> options for both, tab and command completion.
>>
>> For command completion, the new option will check if the prompt was
>> printed, and fail in this case. This is enough to know that the test has
>> failed because the check comes after the PASS path. For tab completion,
>> we have to check if GDB outputted more than just the input line, because
>> sometimes GDB would have printed a partial line before finishing with
>> the correct completion.
>
> This causes quite a few regressions with check-read1.
>
> For instance:
> ...
> (gdb) break baz(int, FAIL: gdb.cp/cpcompletion.exp: tab complete 
> "break baz(int"
> double) Quit^M
> (gdb)
> ...
Hi! Sorry for taking so long to respond. I'd appreciate some help in 
solving, if you have the time.
>
> Thanks,
> - Tom
>
>> ---
>>   gdb/testsuite/lib/completion-support.exp | 16 ++++++++++++++++
>>   1 file changed, 16 insertions(+)
>>
>> diff --git a/gdb/testsuite/lib/completion-support.exp 
>> b/gdb/testsuite/lib/completion-support.exp
>> index bf9c5ad352c..275f8874f15 100644
>> --- a/gdb/testsuite/lib/completion-support.exp
>> +++ b/gdb/testsuite/lib/completion-support.exp
>> @@ -94,6 +94,9 @@ proc test_gdb_complete_tab_none { line } {
>>       -re "^$line_re$completion::bell_re$" {
>>           pass "$test"
>>       }
>> +    -re "$line_re\[^ \]+ $" {
>> +        fail "$test"
>> +    }
>>       }
>>         clear_input_line $test
>> @@ -108,11 +111,15 @@ proc test_gdb_complete_tab_unique { input_line 
>> complete_line_re append_char_re }
>>         set test "tab complete \"$input_line\""
>>       send_gdb "$input_line\t"
>> +    set partial_complete [string_to_regexp $input_line]
>>       set res 1
>>       gdb_test_multiple "" "$test" {
>>       -re "^$complete_line_re$append_char_re$" {
>>           pass "$test"
>>       }
>> +    -re "$partial_complete\[^ \]+ $" {
>> +        fail "$test"
>> +    }

This is the specific change that causes the failures. The thinking 
behind it was that if we receive more characters, but not the whole 
complete_line, we got a failure. Something like this could detect if we 
have a unique - but wrong - suggestion or multiple options. This way it 
doesn't have to go to timeout every time, because it was making clang 
testing take too long.

Is there any other way to detect if GDB is done with the suggestion? Or 
can we detect that read1 is being used, so this gets special cased?

-- 
Cheers,
Bruno

>>       timeout {
>>           fail "$test (timeout)"
>>           set res -1
>> @@ -164,6 +171,9 @@ proc test_gdb_complete_tab_multiple { input_line 
>> add_completed_line \
>>               }
>>               }
>>           }
>> +        -re "${maybe_bell}\r\n.+\r\n$gdb_prompt $" {
>> +            fail "$test"
>> +        }
>>           }
>>       }
>>       }
>> @@ -191,6 +201,9 @@ proc test_gdb_complete_cmd_unique { input_line 
>> complete_line_re } {
>>       -re "^$cmd_re\r\n$complete_line_re\r\n$gdb_prompt $" {
>>           pass $test
>>       }
>> +    -re "$gdb_prompt $" {
>> +        fail "$test"
>> +    }
>>       }
>>   }
>>   @@ -217,6 +230,9 @@ proc test_gdb_complete_cmd_multiple { 
>> cmd_prefix completion_word completion_list
>>       -re "^$cmd_re\r\n$expected_re$gdb_prompt $" {
>>           pass $test
>>       }
>> +    -re "$gdb_prompt $" {
>> +        fail "$test"
>> +    }
>>       }
>>   }
>

Re: [PATCH] gdb/testsuite: Improve testing of GDB's completion functions

[Tom de Vries]

On 7/25/23 17:40, Bruno Larsen wrote:
> On 15/07/2023 14:13, Tom de Vries wrote:
>> On 2/22/23 10:11, Bruno Larsen via Gdb-patches wrote:
>>> When looking at some failures of gdb.linespec/cp-completion-aliases.exp,
>>> I noticed that when a completion test will fail, it always fails with a
>>> timeout.  This is because most completion tests use gdb_test_multiple
>>> and only add a check for the correct output.  This commit adds new
>>> options for both, tab and command completion.
>>>
>>> For command completion, the new option will check if the prompt was
>>> printed, and fail in this case. This is enough to know that the test has
>>> failed because the check comes after the PASS path. For tab completion,
>>> we have to check if GDB outputted more than just the input line, because
>>> sometimes GDB would have printed a partial line before finishing with
>>> the correct completion.
>>
>> This causes quite a few regressions with check-read1.
>>
>> For instance:
>> ...
>> (gdb) break baz(int, FAIL: gdb.cp/cpcompletion.exp: tab complete 
>> "break baz(int"
>> double) Quit^M
>> (gdb)
>> ...
> Hi! Sorry for taking so long to respond. I'd appreciate some help in 
> solving, if you have the time.
>>
>> Thanks,
>> - Tom
>>
>>> ---
>>>   gdb/testsuite/lib/completion-support.exp | 16 ++++++++++++++++
>>>   1 file changed, 16 insertions(+)
>>>
>>> diff --git a/gdb/testsuite/lib/completion-support.exp 
>>> b/gdb/testsuite/lib/completion-support.exp
>>> index bf9c5ad352c..275f8874f15 100644
>>> --- a/gdb/testsuite/lib/completion-support.exp
>>> +++ b/gdb/testsuite/lib/completion-support.exp
>>> @@ -94,6 +94,9 @@ proc test_gdb_complete_tab_none { line } {
>>>       -re "^$line_re$completion::bell_re$" {
>>>           pass "$test"
>>>       }
>>> +    -re "$line_re\[^ \]+ $" {
>>> +        fail "$test"
>>> +    }
>>>       }
>>>         clear_input_line $test
>>> @@ -108,11 +111,15 @@ proc test_gdb_complete_tab_unique { input_line 
>>> complete_line_re append_char_re }
>>>         set test "tab complete \"$input_line\""
>>>       send_gdb "$input_line\t"
>>> +    set partial_complete [string_to_regexp $input_line]
>>>       set res 1
>>>       gdb_test_multiple "" "$test" {
>>>       -re "^$complete_line_re$append_char_re$" {
>>>           pass "$test"
>>>       }
>>> +    -re "$partial_complete\[^ \]+ $" {
>>> +        fail "$test"
>>> +    }
> 
> This is the specific change that causes the failures. The thinking 
> behind it was that if we receive more characters, but not the whole 
> complete_line, we got a failure. Something like this could detect if we 
> have a unique - but wrong - suggestion or multiple options. This way it 
> doesn't have to go to timeout every time, because it was making clang 
> testing take too long.
> 
> Is there any other way to detect if GDB is done with the suggestion? Or 
> can we detect that read1 is being used, so this gets special cased?
> 

The purpose of read1 is to reliably exercise FAILs in the test-suite, 
that are otherwise only occasionally occurring (see also "Race 
detection" in gdb/testsuite/README).

It's typically a test-case problem where it passes or fails depending on 
how fast the input arrives.

When read1 finds such a FAIL, we want to fix it because we want 
deterministic results.

So, I'd say the relevant question is: did the change make the related 
test-cases racy, and does special casing try to hide the race?

Thanks,
- Tom

Re: [PATCH] gdb/testsuite: Improve testing of GDB's completion functions

[Guinevere Larsen]

On 15/08/2023 07:45, Tom de Vries wrote:
> On 7/25/23 17:40, Bruno Larsen wrote:
>> On 15/07/2023 14:13, Tom de Vries wrote:
>>> On 2/22/23 10:11, Bruno Larsen via Gdb-patches wrote:
>>>> When looking at some failures of 
>>>> gdb.linespec/cp-completion-aliases.exp,
>>>> I noticed that when a completion test will fail, it always fails 
>>>> with a
>>>> timeout.  This is because most completion tests use gdb_test_multiple
>>>> and only add a check for the correct output.  This commit adds new
>>>> options for both, tab and command completion.
>>>>
>>>> For command completion, the new option will check if the prompt was
>>>> printed, and fail in this case. This is enough to know that the 
>>>> test has
>>>> failed because the check comes after the PASS path. For tab 
>>>> completion,
>>>> we have to check if GDB outputted more than just the input line, 
>>>> because
>>>> sometimes GDB would have printed a partial line before finishing with
>>>> the correct completion.
>>>
>>> This causes quite a few regressions with check-read1.
>>>
>>> For instance:
>>> ...
>>> (gdb) break baz(int, FAIL: gdb.cp/cpcompletion.exp: tab complete 
>>> "break baz(int"
>>> double) Quit^M
>>> (gdb)
>>> ...
>> Hi! Sorry for taking so long to respond. I'd appreciate some help in 
>> solving, if you have the time.
>>>
>>> Thanks,
>>> - Tom
>>>
>>>> ---
>>>>   gdb/testsuite/lib/completion-support.exp | 16 ++++++++++++++++
>>>>   1 file changed, 16 insertions(+)
>>>>
>>>> diff --git a/gdb/testsuite/lib/completion-support.exp 
>>>> b/gdb/testsuite/lib/completion-support.exp
>>>> index bf9c5ad352c..275f8874f15 100644
>>>> --- a/gdb/testsuite/lib/completion-support.exp
>>>> +++ b/gdb/testsuite/lib/completion-support.exp
>>>> @@ -94,6 +94,9 @@ proc test_gdb_complete_tab_none { line } {
>>>>       -re "^$line_re$completion::bell_re$" {
>>>>           pass "$test"
>>>>       }
>>>> +    -re "$line_re\[^ \]+ $" {
>>>> +        fail "$test"
>>>> +    }
>>>>       }
>>>>         clear_input_line $test
>>>> @@ -108,11 +111,15 @@ proc test_gdb_complete_tab_unique { 
>>>> input_line complete_line_re append_char_re }
>>>>         set test "tab complete \"$input_line\""
>>>>       send_gdb "$input_line\t"
>>>> +    set partial_complete [string_to_regexp $input_line]
>>>>       set res 1
>>>>       gdb_test_multiple "" "$test" {
>>>>       -re "^$complete_line_re$append_char_re$" {
>>>>           pass "$test"
>>>>       }
>>>> +    -re "$partial_complete\[^ \]+ $" {
>>>> +        fail "$test"
>>>> +    }
>>
>> This is the specific change that causes the failures. The thinking 
>> behind it was that if we receive more characters, but not the whole 
>> complete_line, we got a failure. Something like this could detect if 
>> we have a unique - but wrong - suggestion or multiple options. This 
>> way it doesn't have to go to timeout every time, because it was 
>> making clang testing take too long.
>>
>> Is there any other way to detect if GDB is done with the suggestion? 
>> Or can we detect that read1 is being used, so this gets special cased?
>>
>
> The purpose of read1 is to reliably exercise FAILs in the test-suite, 
> that are otherwise only occasionally occurring (see also "Race 
> detection" in gdb/testsuite/README).
>
> It's typically a test-case problem where it passes or fails depending 
> on how fast the input arrives.
>
> When read1 finds such a FAIL, we want to fix it because we want 
> deterministic results.
>
> So, I'd say the relevant question is: did the change make the related 
> test-cases racy, and does special casing try to hide the race?
>
Yeah, I spoke to Andrew off-list and he explained this to me. The test 
itself wasn't racy on a light machine, but could be if it was under 
heavy load or if the expected output was too big. I have sent a v2 that 
fixes this without special casing: 
https://sourceware.org/pipermail/gdb-patches/2023-August/201361.html

-- 
Cheers,
Guinevere Larsen
She/Her/Hers