From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by sourceware.org (Postfix) with ESMTPS id 3090C3858D28 for ; Tue, 23 Nov 2021 18:54:33 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3090C3858D28 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1ANHuHU7010833 for ; Tue, 23 Nov 2021 18:54:32 GMT Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3cg55g4a3w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 23 Nov 2021 18:54:31 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 1ANIknD2076746 for ; Tue, 23 Nov 2021 18:54:29 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2177.outbound.protection.outlook.com [104.47.58.177]) by aserp3030.oracle.com with ESMTP id 3ceq2er3gy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 23 Nov 2021 18:54:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MyiYwFaxH2yOBxAYcMySRs7lMzIagUM6NG9UMwZCjJYCnt5zwzBAJufRU+yFaJgppNNnJ959auBfqf4oyNI7NBcnS4jzj5llzrBq8G8vBMLeibJiuLwKm133hjJRiFWyweJ/n5yIIeNx1y5MI2biPNqguFGOvP0sLfcx//FOQIU51fbPoeZ3XzBza/fZ/24w7/RPYabu7IdHtHXWP4Tq2VjQbh4CqzGCIGHgfqd1lITuXqASqZFtkcvnYAL1upJ+7D5PDXtZjS4m2L8q+asPTugnS0QuWZK8zotNgT0jt3xvRSKIopjePTZ0UCy3URaNRaLuDkjNQ+vOXtjYv5LZfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fkFkVXh9O0hhJTMkm4d3RqNDPX5e89Sj6BHN+Xp4Dqk=; b=in3xp3ZZQ/J2NmowhUZTMxL0S6xIOdNpL5kA8sill8bDeG5INPI4+P9p2mKqEKs5+uVjel8gO9Vol/zU/yyFFLAu2tfNia9A723t/duVs4LULdErlWzU0lOo8x7VF4KehvUEVE7+8syLuxJR0wuOEbSDHfiu9vQGyTbBaZZyOTnOTj6nFvChBw/91TkFF6maQ17YQtXHOL4lBHFyfCH7aVIQ8OPdT/w46lA1QF1PXclw/hVVlau/z2SHZRJTtA2Bu54zBtEs+I3OZUurQFrudn9t+AWZOJ7UFH28tt6OLnuEhBe5n4H0nx0GP4YvxMA3uZdzb9ea0SlZfllP+XUfHg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none Received: from BL0PR10MB2852.namprd10.prod.outlook.com (2603:10b6:208:76::22) by MN2PR10MB3855.namprd10.prod.outlook.com (2603:10b6:208:1b6::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4713.22; Tue, 23 Nov 2021 18:54:28 +0000 Received: from BL0PR10MB2852.namprd10.prod.outlook.com ([fe80::7126:e98e:9cff:8765]) by BL0PR10MB2852.namprd10.prod.outlook.com ([fe80::7126:e98e:9cff:8765%7]) with mapi id 15.20.4713.025; Tue, 23 Nov 2021 18:54:28 +0000 From: Guillermo Martinez To: "Guillermo E. Martinez via Libabigail" , Jose Marchesi Subject: Re: [PATCH v2] Add regression tests for ctf reading Thread-Topic: [PATCH v2] Add regression tests for ctf reading Thread-Index: AQHX3+i0sxPxY2A1xE+yYqgUFCa9sqwRQ2O7gAAz3YA= Date: Tue, 23 Nov 2021 18:54:28 +0000 Message-ID: <1748826.RoRjW6BIRr@sali> References: <20211118041625.622972-1-guillermo.e.martinez@oracle.com> <20211122213353.2456208-1-guillermo.e.martinez@oracle.com> <87fsrmhnzg.fsf@oracle.com> In-Reply-To: <87fsrmhnzg.fsf@oracle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 275eb730-78f9-488e-ec53-08d9aeb2ad28 x-ms-traffictypediagnostic: MN2PR10MB3855: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: d5oCCAnCibUyvXnH4wXO1FhjQHONZA+5KcG1f0MMLgZe/jJdz74BrTFo9t0Nv5piyevglBMZ3hrG+7U8NqpWT+Hr7wjOZcLpzsDwdACZgNehyHyXTAM3H51hnxqX6oK8vKWXdY4dTwhQZm1y8H+bqDKBkFgTqEELkp7W/ZJCt5yU1vUVsrRkSbvAhbS8hNnCPQWZ2eZ0kyU1abhIUPk6SWZJrfjKfsZLbG1CIM6EyGeMf3jDRTdS7TlRZkX+IREeiwBHk0JdUUsU6kOXDXtZRt6hHq8S5uafzgyaeA8d4MmZam0Qx4CKoJdaqZYlaWLmLY9zI910Of8hx4ZtaMmSKrGnS8gNxdh1JHaDvPcKjWSEXtRy+vqw2ymV8sml5bDFXeIE8jqUDAWkZSiYsZQ9KKWjYx8BR/1xQsWuJ9+MJc8RtLLXpE6da2ByPrf+4WDVts88ocHHiVJewFZkHEF+5HmtmiOStm1XduRlK4DxaswEqxv/k/OKA959y/LXzYCdtp+S3awLOrAc6sbeak3eKvPiCSBU4kEB+fHwA71Ru2ap6NVMGpIoWI/PQgr6MO9f2nkp6yF+R+bxQ2qKU8Om2dFo6SFXBtfh4Yf0YVoPXx6HZtfqoAMx/1kgmP1yd1Wj+Trvd0VHkLvCme6X1LBjnKuCDVKyqz3h0zt8c8sanWumDrEMRs+TiI6p2IFz5Wn+thTt/yV/luPA86BAUo4CWHNWsJouFJfsqPg7vldGt42RDYhTLgc0QwDoNYrQi7X5zZ+BL0mLhsm2tS5zB3hjiI4VBZPxkOBlQ7g+/4gthgzd0lxC5sm7EVXELm8PYIEA7Sq2i3A+8v/brjX9Vm+dQVhOwFqDJH7arDvvcV5zcVo= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR10MB2852.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(7916004)(366004)(5660300002)(66556008)(8676002)(91956017)(110136005)(66946007)(6512007)(53546011)(76116006)(508600001)(66476007)(86362001)(122000001)(6506007)(8936002)(38070700005)(83380400001)(33716001)(6486002)(9686003)(71200400001)(64756008)(186003)(66446008)(2906002)(966005)(316002)(38100700002)(6636002)(39026012); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?+nj7btCmQpmXUFaVJrlWTu11NNPCp4E37M0Zv01EWbaj41tqgGpw+pO3rFXb?= =?us-ascii?Q?KGu/5vZ5+pBHxk7NYFBWkEhYq01FdFGFYdlZEM5yiL83nvh8owY7tGEh49Ls?= =?us-ascii?Q?7gJndgVFH/EmOshAQ79KTf/Iojh4YOMPYbbsgIhStXrF2oviG5wgupqqVnU+?= =?us-ascii?Q?eJu3lqrb9gwvId4kKH/XN8sLO7d6W4EJtjp6v8pP+7xQX1d+h+26MBjiDule?= =?us-ascii?Q?KFs3FoHfQhYkU43SWphl01lEJGE8GzROKVtcyK2I8GE507xLK9a2IrSlz+9k?= =?us-ascii?Q?tu4tcnIMGBpJpiCe83Tr/ZfO2q0RjZ9uWYE96hJLpiV7xVbskEjh7KwqABUE?= =?us-ascii?Q?n6//hpVhLjzKNcJglT3yhdlWTZ3BDhJJDEBjeqjx+0Yi8en6yKNmFI+vLvhb?= =?us-ascii?Q?+ew11nTsIWCNBKQmeoZJSX5a3qAx4QJ1hmmJEsWh3XtX7jb6lHX0hEynA0uz?= =?us-ascii?Q?EYLuMqQb8GgSLvi/WICUCY4f3xbXkT5APA4lI877ryi4lYf44k9vhyqFD6bi?= =?us-ascii?Q?JbyKpb/MqNrw6D+/zSiBJXgRedlkJjDeWd7UiY5mjpAhJ31aH1OmHv7cnHro?= =?us-ascii?Q?giDQRMe5NGKtfU00nBTH+n/fBbiPMUQvpsqgM5cvohqMv+mTr13CsW7gCzOA?= =?us-ascii?Q?mRfyVQJYV6B7lcNsfjbPLZ186obOnF+NspEElQf+phhMKpUka3lngArmvCct?= =?us-ascii?Q?ysSmPqqT8p7JPBPfCckf8gMA7hpm2XC/sddQoEv/ehZ38aRcSV3hAhtkI3q+?= =?us-ascii?Q?pbIvi5P697xQNDRODnYf+jtVK0iScm4NWqcUc8YTv5SSGg++f5AjsiHMBqnt?= =?us-ascii?Q?v/2oV9haUQvfrIWc2bGYA6RaQgBBchHvU1L0G8j97w1TxNLGdKxvM3pEKeAL?= =?us-ascii?Q?HKlwdOXt1J+rNjNqIDSYjfzdovjUSVhOift+pqJ1MEk+j/7TLLq3AKHr6o4H?= =?us-ascii?Q?nlsHB/4LwZ7uqjtGkqyqklxPCanFGjB9ITlr+WR/cHX5iFOhfW7UTf9u+abU?= =?us-ascii?Q?sPb6au7i71mANHzoqPQc5S/pseFds0IxcKj9cE2nFSRO8/EUrvEEOpdzFwa4?= =?us-ascii?Q?BiHenG7K5EnZxoqZ3eIgiRYasiszWSkLdtIW5bvVnmKeZ++Y82qpR81jnY8t?= =?us-ascii?Q?2OJIrpjUr3y25FoVVDvs8pDNXfLFcRDuVroLIZOxzULSTFZ8lt9rkZCnxVxf?= =?us-ascii?Q?BsUy9SPXcUVBUWBCjkWNCaZ5/E53mhQkLVEOOpKm1x0gn1xbpXt7oWAZ2hJw?= =?us-ascii?Q?Tj1qB/YIvil5mC563oZJJ0PxAStPVTAhdOQZPYx3o4NFmhLK6221t34z0zAy?= =?us-ascii?Q?xN8z1lDbm2TuPvumfJc61V19Jjd8XkhfzyDSgouo8E7w5bRdidZj87oC3Nj4?= =?us-ascii?Q?XoANTtnP74xaRKfLxZF+VMC4AIsYBJ7W/gLHz9NRXqQk2NiUZoKplsotvx/R?= =?us-ascii?Q?v29f8ynve+2L1TCiRkZPreayWso4kpuw2k5PnQOnFBjV6wd9gTaWwnGTsV8O?= =?us-ascii?Q?CH4jnHT8AWfcQH2Jez6Gr4dzUp342T2KNY3b6trUB0fKUHDfeMXqYbQWk0Xx?= =?us-ascii?Q?3cvOgAUsgnKnTWk9JReDsracV1WOLeiL3VSTCjNzXoQUA+PdO/RR+oPVoJsH?= =?us-ascii?Q?KRdnZNtNnII/0I5Z+lQu+QWzZcqsrUCRNufhfsnX4XWOH6TQk2SU7T7zm+cY?= =?us-ascii?Q?acMM5afuB0xxAmtLFWwxL63Jd89YV41JVf1uL6Z6I+Ec9xvD?= Content-Type: text/plain; charset="us-ascii" Content-ID: <821657EF47362C4B9FEACEBF317EF8DC@namprd10.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BL0PR10MB2852.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 275eb730-78f9-488e-ec53-08d9aeb2ad28 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Nov 2021 18:54:28.1688 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 9ETCdLwE89TBPfRhLD93O3/mmItRlpt1JYIrTYF7P/pDCsfa3/RbsW0vNs+wggDO1bLoCTr4TmWzIiuweRtZ5+ar0BzL1CwTTr5vGDk5IQs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB3855 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10177 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 spamscore=0 bulkscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2111230092 X-Proofpoint-ORIG-GUID: pIZpfcdTnq6E9WztE7ZwF9WOWpqLNplc X-Proofpoint-GUID: pIZpfcdTnq6E9WztE7ZwF9WOWpqLNplc X-Spam-Status: No, score=-6.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Nov 2021 18:54:36 -0000 On Tuesday, November 23, 2021 9:48:35 AM CST Jose E. Marchesi wrote:=0A= =0A= Hello Jose/everybody=0A= =0A= Thanks for your comments!, answers below:=0A= =0A= The following items summarise steps currently implemented in=0A= the testsuite for DWARF and CTF readers:=0A= =0A= 1) Create corpus using the ELF input file, e.g:=0A= *src*/libabigail/tests/data/test-read-ctf/test0=0A= - In this step the SUTs are: *the front-end and readers*.=0A= =0A= 2) Serialize the corpus object to XML ABI description in output=0A= directory:=0A= *build*/libabigail_x86_64/tests/output/test-read-ctf/test0.abi=0A= - SUT: *writter* using write_corpus function (it doesn't=0A= use libxml (IMHO something to change if we want to use in the=0A= future properties, name space, etc).=0A= =0A= 3) Spawn *abidw* tool with ELF input file using --abidiff argument, e.g:=0A= abidw --abidiff --ctf *src*/libabigail/tests/data/test-read-ctf/test0= =0A= =0A= Internally abiw works as follow:=0A= =0A= 3.1) Create corpus from ELF input file, e.g:=0A= *src*/libabigail/tests/data/test-read-ctf/test0=0A= - In this step SUTs: *DWARF/CTF frond-end readers*=0A= =0A= 3.2) Serialize the (*first*) corpus object to XML ABI description in =0A= *temp* directory:=0A= /tmp/libbigail-xyz=0A= - SUTs: ABI writter,=0A= =0A= 3.3) Build a (*second*) corpus from this temporary file (*same file!*):= =0A= - SUTs: *XML reader*.=0A= =0A= 3.4) Compute the corpus differences=0A= compute_diff(corp, corp2 ..)=0A= - SUTs: *comparison algorithm* (diff_context/corpus diff)=0A= =0A= if there are differences return 1 otherwise return 0.=0A= =0A= 4) The return value is read by test-read-ctf and if it's 1 test=0A= is marked as *FAILED*=0A= =0A= 5) Use a external *diff command* to compare the XML abi input file with=0A= the file outputted in the step 2. The return code of the diff command= =0A= is used to mark the test as SUCCESS or FAILED=0A= =0A= So, the abidw doesn't use the expected XML file used in the testsuite, so= =0A= if there are changes on the input ELF or in the libabigail subsystems (read= ers,=0A= writer, corpus, etc) we could get false negatives, because it is working wi= th=0A= itself result as an incoming file, instead of use an expected file.=0A= =0A= I think that we should use the ABI XML reader from the expected file (e.g:= =0A= tests/data/test-read-ctf/test0.abi) to build the corpus to be compared=0A= with the corpus built with ELF input file (e.g: tests/data/test-read-ctf/te= st0)=0A= and in this way replace the external abidw and diff command calls, this als= o=0A= helps to avoid false positives when XML ABIs files has the same nodes but n= ot=0A= in the same order.=0A= =0A= What do you think?=0A= =0A= So, my answer below: =0A= =0A= > I have a couple of general comments/questions regarding libabigail=0A= > testing and DWARF/CTF.=0A= > =0A= > 1) The usage of diff in the testsuite puzzles me. I would have expected= =0A= > to find at least two kinds of tests in the testsuite:=0A= > =0A= > a) Tests testing the frontend/readers, which can assume the XML=0A= > reader and comparison algorithm works properly.=0A= Done in steps 1, 3.1.=0A= =0A= > b) Tests testing the comparison algorithm, which can assume the=0A= > frontend/readers work properly.=0A= Done by abidw in step: 3.4 =0A= > =0A= > This way no external comparison (like diffing) would be necessary.=0A= > Is this not the case?=0A= Yes, agree.=0A= > =0A= > 2) I am surprised by the fact the CTF reader seems to be working as good= =0A= > as the DWARF reader re. the testsuite. Not buying it. I think that= =0A= > the testsuite skips .cc tests when testing CTF support.=0A= Correct, cc files are skipped, just I collected the C test used in d= warf test reader,=0A= and the test used by libctf, validating the expected result, so the = functionality=0A= tested in libctf is also tested in libabigail. (except aliases symbo= ls).=0A= =0A= > So I wonder... to what extent are C features (not C++) tested using=0A= > .cc files? If we want good coverage for both DWARF and CTF (and=0A= > eventually BTF or other formats not supporting C++) then we would=0A= > need to turn these .cc files into .c files...=0A= sure, I can "translate" C++ test's data and add it to CTF testsuite.= =0A= > =0A= > > Hello libabigail team,=0A= > >=0A= > > This patch includes regression tests v2 for CTF.=0A= > >=0A= > > The change from v1 is disable CTF test alias, because=0A= > > currently is is not supported.=0A= > >=0A= > > Dependencies/limitations:=0A= > >=0A= > > * It was worked on the top of the following patches:=0A= > > https://sourceware.org/pipermail/libabigail/2021q4/003853.html=0A= > >=0A= > > * Some CTF tests were *disabled* because it generates the XML ABI=0A= > > corpus with *same information* but the XML nodes *are not* always=0A= > > in the *same order*, so using diff command fails. Details here:=0A= > > https://sourceware.org/pipermail/libabigail/2021q4/003824.html=0A= > >=0A= [...]=0A= =0A= Sincerely,=0A= Guillermo =0A=