From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by sourceware.org (Postfix) with ESMTPS id A9D053858403 for ; Thu, 25 Nov 2021 06:50:16 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org A9D053858403 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1AP6E48L016279; Thu, 25 Nov 2021 06:50:13 GMT Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3chk005fj4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Nov 2021 06:50:12 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 1AP6lBcp072141; Thu, 25 Nov 2021 06:50:11 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2175.outbound.protection.outlook.com [104.47.57.175]) by aserp3030.oracle.com with ESMTP id 3ceq2h7m7y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Nov 2021 06:50:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ec6FOqhks4ud38Yyeiuh1Gh3QIazs3fBcDSBBOJpe0dKWMAk2hj8mVjUby0wnhYGPBREUA6p0RkhkgTVrWsyXYlRezZa/Jli3s+8xxEB8gTswm2pVZagavHkqKpWV2fTAi1Rl5xPr0MCvuRaR8EK304fkApOk84aakIXBVAtmz0zjL44moAeV5W+Esxxd2+O0vCctKnLzQEZQzFUTJyOjugEGpkOx33IB9dHFN8Gdko+/Cwn8PGdHMrgydWpzKvca/yCqKUa/5PNMhUi+8fK/pf5zOlBIrQRUA1dSIkKK06QIynZase709dZ25V2AaKxxX+qiLANhCt663QfeKIOqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MsbEYvoGY8dpiN7fe7oA5JUrDkoSFlGOnR1OJsL98uA=; b=FjmvoQ2tFfVk/NIYsa/frzseDmFBQ/t53W5Duar3RzticGt25eFtsES4kys6aDtlxjiSWV4iLI9Mk+KqMgj1s8Ul1TI9ZXPwNRWPTAAsGwIi10t23gJWPPGTqsCerxCgWCbvLBSXOme+cRFy1Kg1wxbYVxDIzQol2LxSfjqb3laIH7o24D6+qRv7/ZLDTaLyqPS+kRBtAluRs+0gSnSyo23nR3i9GtxNjlgNyPewjYJsPMyx+rF+pqCNvEZRNiLU/MVVJx09fORsw6/kgXEzFfFJMBim1IlV7XRul1qmsclBzZbuJ4Arrhx1fAgeJdA++6u/HQt9EHb+XQM8Udvotg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none Received: from DM6PR10MB2890.namprd10.prod.outlook.com (2603:10b6:5:71::31) by DM6PR10MB4107.namprd10.prod.outlook.com (2603:10b6:5:17a::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.20; Thu, 25 Nov 2021 06:50:09 +0000 Received: from DM6PR10MB2890.namprd10.prod.outlook.com ([fe80::78ba:e811:54e5:b32e]) by DM6PR10MB2890.namprd10.prod.outlook.com ([fe80::78ba:e811:54e5:b32e%3]) with mapi id 15.20.4713.025; Thu, 25 Nov 2021 06:50:09 +0000 From: "Jose E. Marchesi" To: Ben Woodard via Libabigail Subject: Re: [PATCH v2] Add regression tests for ctf reading References: <20211118041625.622972-1-guillermo.e.martinez@oracle.com> <20211122213353.2456208-1-guillermo.e.martinez@oracle.com> <87ee75tsst.fsf@seketeli.org> Date: Thu, 25 Nov 2021 07:50:00 +0100 In-Reply-To: (Ben Woodard via Libabigail's message of "Wed, 24 Nov 2021 11:09:59 -0800") Message-ID: <87bl28sp9j.fsf@oracle.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: AM0PR03CA0047.eurprd03.prod.outlook.com (2603:10a6:208::24) To DM6PR10MB2890.namprd10.prod.outlook.com (2603:10b6:5:71::31) MIME-Version: 1.0 Received: from termi.oracle.com (141.143.193.74) by AM0PR03CA0047.eurprd03.prod.outlook.com (2603:10a6:208::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.20 via Frontend Transport; Thu, 25 Nov 2021 06:50:08 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2b086f3e-fc6d-44d5-1050-08d9afdfd26d X-MS-TrafficTypeDiagnostic: DM6PR10MB4107: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2hyvPThIl42I/iVGV/coYbKKXAt+pqixY4EypOe/P476NxF/qMFuZu3yO7amuIkvE3ij2It+PDwPUFrHlKUIHcMxjULjR/InH8Q+n1jYWIRuoUmUVBxQre904XwWv31q0uA8y+orhteqv6puSZzXEdSckahRAWURQe6DI3mtDfA52RShwaYPutLDQ49uilN5CqDDDWvcF1muAnrb0KCpXE33UryjKp0qOipEB2xss+XI4SZqbO7XjZMEO8uDbRhSJavYJ5v/BS/SPctsZBh1o6TIuq1pYoAQHoHIaEXH9+LK03wqbF3BykPFm1M3gj5MPEWhB8iIzVxFNnHe3ldTyQx1Dc/TsQtsryf3Q68a3sFq4Zz+qnU03PMwsytq5EKSssLyVc4NcLIwmvUlRiCYzl67e+9jspsK5HfIqMVDKUci3e6l/btnk6Gi426cTa4cdQBWQ3RMjxQt9adI2og02YV9E7WOybNayHg3xQnkDrthaAn8enQokpzH4Z7GxQl2AFfXDUT9M0mBqaPFPI8WlHuJ7RSzrabFvqrhayDBR4bfXA+3fsDl/KnWkfArsBSvoeteGsFU+CHiVFfhSiW98Ml+vt98kawU6GRZEpEiMj8mo+2FjDLc07ZLvxMS063f/BWTZnMNpdIyUUwcSXA81ffz3Ft9qmECk/DrXTnrxzXANmxBjd/C9l65lR8QFmmiNRcclgtx/z5H1p4zjnktK7RJ9VqMNo6Z3fqwOqZPLYljlnX31MOaYn5qViWf9EoPu0j/0j6tMBJfWEKXrpwDvu+PpZuV7M/I/FvIjUAO+QjIOGGwVlMW+27ONZt5jDHCS/IARI/mDnuZvC5mX/xXtA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR10MB2890.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(316002)(54906003)(4326008)(2616005)(956004)(2906002)(186003)(26005)(38100700002)(38350700002)(6916009)(966005)(66556008)(7696005)(53546011)(66946007)(83380400001)(36756003)(6486002)(66476007)(86362001)(52116002)(6666004)(8936002)(8676002)(5660300002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TGtyczZGL29KaXZtUFM5UmlBdTVXaHlDQ2xEeVJFaWV0TThESCtoanE3ZnFE?= =?utf-8?B?RTNkaWZsalk5R01DVlVFZzhlRXk5VE9HQ0MxMkhsNVEwK2dzMXNjbWg1cy9q?= =?utf-8?B?VXVMMTFUdXpLbjBTK0VhTjZDaW5uczB2cnovMTBuajZ3cSt4T3BiREY0UUNG?= =?utf-8?B?MTdDc0NiaTRmV3o4RGtDMWliWjFpY2xiQVFKQ3pKekJUcllCZklSaDhucjk2?= =?utf-8?B?RW41Yy9VWGJacFdFaVVjYmZrMUxFZzhCU25yN3FPbWJkV1BzdGI5TDRQRzd4?= =?utf-8?B?SklucUx0NEFVN29YVWRsTWdHMjFLdGprcjAvWFZmaG9HbGVNWnVNU0J2SmJq?= =?utf-8?B?Q0VzSUFTRmtwNHNNaEFLd01lbG9ZWmdLZmowYm16Y01Gam00cHBpNmRZRUpW?= =?utf-8?B?UDc5QXkwb1d5bGtIQ0t0QlZTeTZkL3FlZEFRa1d5OHROWWJ0OHRuNzlpY0VW?= =?utf-8?B?Q0c1QU80NWhaWlVxZUs2UzVLR256VERjL1hkS3FWbU9IZDB5K252bUV0c2VL?= =?utf-8?B?UkRQV3ZxcnVORGtUdUhqcnBCdzRic3U1OHFSYXUvcVFYMUlwdDE2NWZtYURT?= =?utf-8?B?d1pOSVl5dTBVYnFiOVdLTStjSjEreEZOa3BLSGZEZTJDYW5RRmw0WldGREFI?= =?utf-8?B?NmNkTDdGZ2RUbndEVE1hNnVkWGZmOVF6Wm0rakc4Wm1ZMzR4WFRwbTZaUmxx?= =?utf-8?B?R1JRc3FtTHY2QlNIWUU4VzVmZGlTVDNGMWtLTWNzaFZVejd5MUVMa2Q2bEM1?= =?utf-8?B?Q0tneWdMSVBjdVpTRi92NCtOSGU1a0E5U2xJUFJxa0VYeExCSFF4SGlIb3Ju?= =?utf-8?B?MnNKTStwbUYzN2U3RUwvN1ZYOHNTVURma2puWnk3SjJmcWZGU1VvOEdTdlRv?= =?utf-8?B?SWQ2YXJkN1NWTng2cTNydW9iRi9pa3F2bWczRnluRjdkK3pjSUp2NVZuNC9k?= =?utf-8?B?N3dYajZHTTliS2RnZGlZUW5EUWR0RDRDL080TlpiNkJoZ0Y0OGpUQlRPVCtv?= =?utf-8?B?NWRLNW40U0pwb3h5NkNoQmZTTUlyaWozSUt0THRhd2FtRmJYM0hsbjVIelM5?= =?utf-8?B?d25xd2tJOGNTMFgrb3BqeE0xQjFzTVc0WHloaTJIaGwrQkJLZkRud2tUaFJi?= =?utf-8?B?U3F5Ti9wZEtnb01iSkl0RWNKQU9oSWF0YXJnOTVlUnowWU1LOGZ1dEJyd3dB?= =?utf-8?B?L3FEM1hQK25uYTRiZ0JDWHFMS2ZQRkZna01DR1pKa3JLeHdYNlNJdW9iRTRU?= =?utf-8?B?cGxJc0M1OUQ5dFZ1Tjg5WjhoODBmcE55allVM1JCN3RFZUptVHpYWjU3YStr?= =?utf-8?B?WDd2RVZoNnRLa2Jwa2swREM0WjNMZjYvYlJERzQ0ZUU1RG52S1NVVzdBYnla?= =?utf-8?B?bk81allGaVowOHNWQXdDREdLZ1V2UzBNU0ZRMHRoRnF1R2p2WFNHaVc2b3hB?= =?utf-8?B?R1V3Y1F3YlpHeXVoQmN2SHZlMlhMU0piS2NNM3dWRDRQS1hYTFR0eWIrS3R4?= =?utf-8?B?M1F0VXRsdkEvUkN4YlNielFIdVQxR1RQWjhuZzFHOUs0UWFhOHlJT2pONFdj?= =?utf-8?B?Z2pET3ZPUUN1WDFTcmV5UmFWclRxcWVaN2ovVUtHZ2NoaXVNbEx6ZTUvREhh?= =?utf-8?B?em1RZnRwbi9QK2J3Zmk5akhMTlF0cUJ4NXAzUW5QeE50emlaMmdycHFIMkVL?= =?utf-8?B?aUh2dkFCM0x1OVBYM3BPS3BvVndZb3Vqa29nc1ZubjRMUzFpMVpNZkordS9K?= =?utf-8?B?c1lqTFZVbHQvVnB3M2Q1ODB3emZLQVVwSFNJRDN6eWRJdldSTzhlWFVYRE01?= =?utf-8?B?WHVUL253QUNQbENacHhMckcrWGdYQ0RueGdiV1pDUzIySUNhUzk4RkRNN2FZ?= =?utf-8?B?MVNHZkl6dW8rUklNUlVEV3NhcEZUcVdvRGpZdmZPZGY0TjJGQldvRHZUekFt?= =?utf-8?B?SlhzQjdySnJQaDJrU1JBZG5YZDkwNDhONVl3VlBtd29VOWRWRTNoSktFaU5G?= =?utf-8?B?Qk1tdzBrVXQ4OVN1MTc2a3l4WFpLd3c2ZG9BTGJIRis3WFFOcCtTc3IwWldU?= =?utf-8?B?UW9FbVpBaDh6VDY5ZTYvWUhQc1E5NVRGbUxhUW5XazZBY0hvNmZmSXB5bEow?= =?utf-8?B?citBenN5Wld0Zk53SUJLQm9WWnFNaGZDU1g5NkVVeVBtUVI0WXFsOG9reW9l?= =?utf-8?B?OVZjTVpyZ2xXT0hHMVFKUjBSd1U5L3BtanI3R0thbEIxTlc1MkdhbzFMS2Zv?= =?utf-8?Q?Sso+WC/0O7TiI/KmfbKoMel139QUJs3I5TJm50Iek4=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2b086f3e-fc6d-44d5-1050-08d9afdfd26d X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB2890.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Nov 2021 06:50:09.6624 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3YN2YdK81XOBGIGG0WBHRzI6UMEFGa560BnniEopA/iBiHNlDujAFwtYzg4ZA6t58Ku6Sq243Ur5Vskb8OhxCNMTwls0J+/bHwAPsSYGk6E= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB4107 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10178 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 spamscore=0 bulkscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2111250037 X-Proofpoint-ORIG-GUID: lrq8Jr-JauloYwE3whsjWnahhGQ7FLD0 X-Proofpoint-GUID: lrq8Jr-JauloYwE3whsjWnahhGQ7FLD0 X-Spam-Status: No, score=-6.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, MSGID_FROM_MTA_HEADER, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Nov 2021 06:50:19 -0000 >> On Nov 24, 2021, at 8:36 AM, Dodji Seketeli wrote: >>=20 >> Hello, >>=20 >> [...] >>=20 >> Thanks for working on this. Nice patch, by the way! I like its >> direction. >>=20 >> I have a few comments and I believe that when they are addressed, we'll >> be able to apply the patch. >>=20 >> [...] >>=20 >>> Dependencies/limitations: >>>=20 >>> * It was worked on the top of the following patches: >>> https://sourceware.org/pipermail/libabigail/2021q4/003853.html >>>=20 >>> * Some CTF tests were *disabled* because it generates the XML ABI >>> corpus with *same information* but the XML nodes *are not* always >>> in the *same order*, so using diff command fails. Details here: >>> https://sourceware.org/pipermail/libabigail/2021q4/003824.html >>=20 >> In those cases where the abixml file generated from CTF is different >> from the one generated from DWARF, I think we should have two >> different reference abixml files to diff against. No CTF test should >> be disabled, I think. > > Here is a somewhat deeper question that I think needs to be > considered. I=E2=80=99ve generally referred to it as =E2=80=9CDWARF Idiom= s=E2=80=9D but this > email makes me think that it is even larger than that. > > For my work, I need libabigail to generate an abstract notion of the > ABI corpus. How it constructs that abstract notion of the ABI needs to > be independent of the producer. Think of it this way, say we take the > same compiler and have it compile the same library producing both CTF > and DWARF, the ABI of the library doesn=E2=80=99t change. Since it is > literally the same object, the program text is same. Therefore the ABI > is unquestionably the the same. Any difference reported by libabigail > therefore is a problem with libabigail. It is not taking the source > material and abstracting it enough into the ABI artifacts to separate > the artifacts from the implementation. > > So I kind of believe that we need to look more deeply into WHY the CTF > and DWARF are not comparing as equivalent and begin the process of > filing the compiler bugs when we need to, and doing what is necessary > to abstract the ABI from the source material that libabigail used to > construct its IR of the ABI corpus from. > > So, I must say that I disagree with both dodji=E2=80=99s approach here an= d to > a lesser extent Guillermo=E2=80=99s approach of disabling the tests. I th= ink > that the tests where the CTF doesn=E2=80=99t match the DWARF should be > investigated and when necessary marked =E2=80=9Cxfail=E2=80=9D with a not= e citing > their individual cause. > > I think that what we need to work towards is: > abidw produces the same output (or more precisely libabigail produces > the same IR) whether you compile with -gdwarf-3 -gdwarf-4 -gdwarf-5 > -gsplit-dwarf -gctf > also for the most part the ABI should not change between compiler > versions. There may be a few cases that we need to look into where the > compiler actually breaks ABI and of course libabigail should flag > those but I would assert that libabigail needs to abstract its IR of > the ABI enough that compiler version changes that don=E2=80=99t actually > change the ABI of the ELF object are not reported as ABI breaks. It > currently does pretty well at this at the moment. > Then once that foundation is built, being able to abstract the ABI IR > enough that differences in toolchains e.g. LLVM vs GCC are not flagged > as changes in the object=E2=80=99s ABI. This is important to provide peop= le > with a tool that will allow them to mix toolchains within a project to > achieve optimal code. I'm so glad you are raising these concerns. I agree with you that ideally abidw should produce "same output" (read equivalent output ABI-wise) regardless of the origin of the ABI corpus... but then: Is the relative order of the <*-decl > nodes in the ABI XML part of the ABI? Is the location information (present in DWARF, not present in CTF nor BTF) part of the ABI? Are the names assigned to internally-generated types (such as the underlying type of an enumerator) part of the ABI? Are the newlines and blank spaces after XML marks part of the ABI? I am no expert in ABI analysis, but I would say the answers in all these cases is `no'. The internal libabigail ABI comparison algorithms would not be impacted by these things, right? It seems to me that the main problems here are that ABI XML, as it is, conveys way too much information (useful, no doubt, but superfluous ABI-wise) and that `diff' is used to perform a textual comparison of these files. Textual comparison would work if we defined a "canonicalized" version of ABI XML. Translating from ABI XML to canonical ABI XML would involve: 1) Strip _everything_ that is not strictly part of the ABI (such as location information, anonymous type names, etc). 2) Sort the XML elements in a predefined way. Whatever sort criteria is used, it would need to be defined only by ABI-constituent elements, such as the names of non-anonymous types. And of course the criteria itself must not change. The only reason you could operate with ABI XML files heavily annotated with non-ABI determining information (such as the loc info and superfluous and arbitrary names) is having an unique source for the ABI info. That's no longer the case: we are using CTF, the Google chaps are using BTF. The above considered, I think we have three options now: a) To define that canonical ABI XML and write an abistrip tool that does the translation, maybe add --canonical options to the tools that generate ABI XML. Then use `diff' in the testsuite. or b) To use an external ABI comparison tool in the testsuite, non-textual, that consumes ABI XML. or c) To use the "internal" approach I already suggested in another email, having two different sets of tests: one set to test the front-ends that assume the libabigail comparison algorithm works properly, another set to test the comparison algorithm that uses either hand-written input files or assumes the front-ends work properly. I personally think (c) would be best.