Re: [PATCH] compare_tests: distinguish c-c++-common results by tool

[Jeff Law <jeffreyalaw@gmail.com>]

On 12/19/23 23:31, Alexandre Oliva wrote:
> On Dec 20, 2023, Jeff Law <jeffreyalaw@gmail.com> wrote:
> 
>> So the strub tests in c-c++-common are problematical.  They get run
>> twice, once for C, once for C++.  Yet the name of the test is the same
>> in both runs. (by the name, I mean the name emitted into the dejagnu
>> summary and log files).
> 
>> Thus if you have a test in there which passes in one context, but
>> fails in the other, comparison tools like contrib/compare_tests may
>> erroneously report the tests as both a test which now fails, but
>> passed before and a test which now passes but failed before.
> 
>> It looks like some of the strub tests are currently known to fail with
>> C++ and are triggering this problem
> 
> Yeah, type warnings/errors are different between C and C++, and this is
> noticeable with permissible conversions between strub types.
> 
> 
>> A third option would be to change the compare_tests tool to somehow
>> distinguish between the C and C++ tests.  Not sure how feasible that
>> is.
> 
> Most feasible among the possibilities ;-)
> 
> I've tested the following by comparing my obj-x86_64-linux-gnu test tree
> with itself.  Ok to install?
> 
> 
> When compare_tests compares both C and C++ tests in c-c++-common, they
> get the same identifier, so expected differences in results across
> languages become undesirably noisy.
> 
> This patch adds tool identifiers to tests, so that runs by different
> tools are not confused by the compare logic.
> 
> It also fixes a bug in reporting differences, that would attempt to
> print an undefined fname (the definitions are in subshell loops), and
> adjusts the target insertion to match tabs in addition to blanks after
> colons.
> 
> 
> for  contrib/ChangeLog
> 
> 	* compare_tests: Add tool to test lines.  Match tabs besides
> 	blanks to insert tool and target.  Don't print undefined fname
Go for it.  It's hard to know if there'll be fallout in the consumers, 
but I think if there is pain, it should be short term.

Jeff