From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by sourceware.org (Postfix) with ESMTPS id 35C1A3858D28 for ; Thu, 25 Nov 2021 10:40:33 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 35C1A3858D28 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=seketeli.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=seketeli.org Received: (Authenticated sender: dodji@seketeli.org) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id A92CBC0016; Thu, 25 Nov 2021 10:40:30 +0000 (UTC) Received: by localhost (Postfix, from userid 1000) id C8DDE581C2F; Thu, 25 Nov 2021 11:40:29 +0100 (CET) From: Dodji Seketeli To: Guillermo Martinez via Libabigail Subject: Re: [PATCH v2] Add regression tests for ctf reading Organization: Me, myself and I References: <20211118041625.622972-1-guillermo.e.martinez@oracle.com> <20211122213353.2456208-1-guillermo.e.martinez@oracle.com> <87fsrmhnzg.fsf@oracle.com> <1748826.RoRjW6BIRr@sali> X-Operating-System: Fedora 36 X-URL: http://www.seketeli.net/~dodji Date: Thu, 25 Nov 2021 11:40:29 +0100 In-Reply-To: <1748826.RoRjW6BIRr@sali> (Guillermo Martinez via Libabigail's message of "Tue, 23 Nov 2021 18:54:28 +0000") Message-ID: <87a6hstt5u.fsf@seketeli.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00, JMQ_SPF_NEUTRAL, KAM_DMARC_STATUS, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Nov 2021 10:40:36 -0000 Guillermo Martinez via Libabigail a =C3=A9crit: [...] > On Tuesday, November 23, 2021 9:48:35 AM CST Jose E. Marchesi wrote: > > Hello Jose/everybody > > Thanks for your comments!, answers below: > > The following items summarise steps currently implemented in > the testsuite for DWARF and CTF readers: > > 1) Create corpus using the ELF input file, e.g: > *src*/libabigail/tests/data/test-read-ctf/test0 > - In this step the SUTs are: *the front-end and readers*. > > 2) Serialize the corpus object to XML ABI description in output > directory: > *build*/libabigail_x86_64/tests/output/test-read-ctf/test0.abi > - SUT: *writter* using write_corpus function Right. > (it doesn't use libxml (IMHO something to change if we want to use in > the future properties, name space, etc). I am not sure about this, but hey, the future will tell :-) > > 3) Spawn *abidw* tool with ELF input file using --abidiff argument, e.g: > abidw --abidiff --ctf *src*/libabigail/tests/data/test-read-ctf/tes= t0 > > Internally abiw works as follow: >=20=20=20=20=20=20 > 3.1) Create corpus from ELF input file, e.g: > *src*/libabigail/tests/data/test-read-ctf/test0 > - In this step SUTs: *DWARF/CTF frond-end readers* > > 3.2) Serialize the (*first*) corpus object to XML ABI description in=20 > *temp* directory: > /tmp/libbigail-xyz > - SUTs: ABI writter, > > 3.3) Build a (*second*) corpus from this temporary file (*same file!*): > - SUTs: *XML reader*. > > 3.4) Compute the corpus differences > compute_diff(corp, corp2 ..) > - SUTs: *comparison algorithm* (diff_context/corpus diff) > > if there are differences return 1 otherwise return 0. > Right. > 4) The return value is read by test-read-ctf and if it's 1 test > is marked as *FAILED* > > 5) Use a external *diff command* to compare the XML abi input file with > the file outputted in the step 2. The return code of the diff command > is used to mark the test as SUCCESS or FAILED > > So, the abidw doesn't use the expected XML file used in the testsuite, so > if there are changes on the input ELF or in the libabigail subsystems (re= aders, > writer, corpus, etc) we could get false negatives, because it is working = with > itself result as an incoming file, instead of use an expected file. Actually, this won't be a false negative. It's a true negative. because what we want to see with abidw --abidiff is that the abixml file that is emitted has the same *ABI* as the incoming ELF file. In other words, we want to see if the DWARF/CTF, the in-memory IR, and the abixml are all coherent, so to speak. The exact output details of the XML itself (things like spaces or order of type definitions etc) don't matter to us at that point. We want those insignificant details of the abixml format to be able to fluctuate without having the fundamentals of ABI compatibility in general to be impacted. However, regardless of abidw --abidiff (which cares about stability of the ABI representation across DWARF/CTF, in-memory IR and abixml), the test-read-dwarf.cc harness also wants to detect minutes changes to the abixml that is emitted by abidw. This is so that developers are forced to inspect abixml changes that are due to source code modification of libabigail before submitting their changes. This is why we use "diff" to compare the emitted abixml against the expected/reference one If the emitted one is different because of a new source code change, but the ABI is stable, we'll just accept that new change and promote the new abidw output as being the new reference/expected output. > I think that we should use the ABI XML reader from the expected file (e.g: > tests/data/test-read-ctf/test0.abi) to build the corpus to be compared > with the corpus built with ELF input file (e.g: tests/data/test-read-ctf/= test0) > and in this way replace the external abidw and diff command calls, Heh. This is exactly what abidw --abidiff does. And we want to use abidw too to, so that the *tool* is tested too. this also But as I have explained above, we also want to use 'diff' for a different level of testing. [...] > helps to avoid false positives when XML ABIs files has the same nodes but= not > in the same order. [...] On Tuesday, November 23, 2021 9:48:35 AM CST Jose E. Marchesi wrote >> 2) I am surprised by the fact the CTF reader seems to be working as good >> as the DWARF reader re. the testsuite. Well, we don't know that ;-) To know that, it's needed to *inspect* the result of abidw --ctf on the binaries and compare it to what we see in the source code of the binary. We can also introduce new ctf binaries for abidiff-based tests and compare their output to their DWARF counter part. There are going to be differences, I am quite sure of that. But at least we'll see what they are. I am not afraid, though. I think the CTF support is super cleanly done. If there are issues, we'll tackle them. This testing work is a critical part of that work. Many thanks to Guillermo for handling it! [...] Cheers, --=20 Dodji