Re: [PATCH v2] Add regression tests for ctf reading

[Dodji Seketeli <dodji@seketeli.org>]

Guillermo Martinez via Libabigail <libabigail@sourceware.org> a écrit:

[...]

> On Tuesday, November 23, 2021 9:48:35 AM CST Jose E. Marchesi wrote:
>
> Hello Jose/everybody
>
> Thanks for your comments!, answers below:
>
> The following items summarise steps currently implemented in
> the testsuite for DWARF and CTF readers:
>
> 1) Create corpus using the ELF input file, e.g:
>       *src*/libabigail/tests/data/test-read-ctf/test0
>      - In this step the SUTs are: *the front-end and readers*.
>
> 2) Serialize the corpus object to XML ABI description in output
>     directory:
>      *build*/libabigail_x86_64/tests/output/test-read-ctf/test0.abi
>      - SUT: *writter* using write_corpus function

Right.

> (it doesn't use libxml (IMHO something to change if we want to use in
> the future properties, name space, etc).

I am not sure about this, but hey, the future will tell :-)

>
> 3) Spawn *abidw* tool with ELF input file using --abidiff argument, e.g:
>      abidw --abidiff  --ctf *src*/libabigail/tests/data/test-read-ctf/test0
>
>     Internally abiw works as follow:
>      
>    3.1) Create corpus from ELF input file, e.g:
>             *src*/libabigail/tests/data/test-read-ctf/test0
>     - In this step SUTs: *DWARF/CTF frond-end readers*
>
>    3.2) Serialize the (*first*) corpus object to XML ABI description in 
>           *temp* directory:
>              /tmp/libbigail-xyz
>     - SUTs: ABI writter,
>
>    3.3) Build a (*second*) corpus from this temporary file (*same file!*):
>         - SUTs: *XML reader*.
>
>    3.4) Compute the corpus differences
>               compute_diff(corp, corp2 ..)
>      - SUTs: *comparison algorithm* (diff_context/corpus diff)
>
>        if there are differences return 1 otherwise return 0.
>

Right.

> 4) The return value is read by test-read-ctf and if it's 1 test
>     is marked as *FAILED*
>
> 5) Use a external *diff command* to compare the XML abi input file with
>     the file outputted in the step 2. The return code of the diff command
>     is used to mark the test as SUCCESS or FAILED
>
> So, the abidw doesn't use the expected XML file used in the testsuite, so
> if there are changes on the input ELF or in the libabigail subsystems (readers,
> writer, corpus, etc) we could get false negatives, because it is working with
> itself result as an incoming file, instead of use an expected file.

Actually, this won't be a false negative.  It's a true negative. because
what we want to see with abidw --abidiff is that the abixml file that is
emitted has the same *ABI* as the incoming ELF file.  In other words, we
want to see if the DWARF/CTF, the in-memory IR, and the abixml are all
coherent, so to speak.  The exact output details of the XML itself
(things like spaces or order of type definitions etc) don't matter to us
at that point.  We want those insignificant details of the abixml format
to be able to fluctuate without having the fundamentals of ABI
compatibility in general to be impacted.

However, regardless of abidw --abidiff (which cares about stability of
the ABI representation across DWARF/CTF, in-memory IR and abixml), the
test-read-dwarf.cc harness also wants to detect minutes changes to the
abixml that is emitted by abidw.  This is so that developers are forced
to inspect abixml changes that are due to source code modification of
libabigail before submitting their changes.

This is why we use "diff" to compare the emitted abixml against the
expected/reference one If the emitted one is different because of a new
source code change, but the ABI is stable, we'll just accept that new
change and promote the new abidw output as being the new
reference/expected output.

> I think that we should use the ABI XML reader from the expected file (e.g:
> tests/data/test-read-ctf/test0.abi) to build the corpus to be compared
> with the corpus built with ELF input file (e.g: tests/data/test-read-ctf/test0)
> and in this way replace the external abidw and diff command calls,

Heh.  This is exactly what abidw --abidiff does.  And we want to use
abidw too to, so that the *tool* is tested too.
this also

But as I have explained above, we also want to use 'diff' for a
different level of testing.

[...]

> helps to avoid false positives when XML ABIs files has the same nodes but not
> in the same order.


[...]


On Tuesday, November 23, 2021 9:48:35 AM CST Jose E. Marchesi wrote

>> 2) I am surprised by the fact the CTF reader seems to be working as good
>>    as the DWARF reader re. the testsuite.

Well, we don't know that ;-)

To know that, it's needed to *inspect* the result of abidw --ctf on the
binaries and compare it to what we see in the source code of the binary.

We can also introduce new ctf binaries for abidiff-based tests and
compare their output to their DWARF counter part.

There are going to be differences, I am quite sure of that.  But at
least we'll see what they are.  I am not afraid, though.  I think the
CTF support is super cleanly done.  If there are issues, we'll tackle
them.

This testing work is a critical part of that work.  Many thanks to
Guillermo for handling it!

[...]

Cheers,

-- 
		Dodji