From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70044.outbound.protection.outlook.com [40.107.7.44]) by sourceware.org (Postfix) with ESMTPS id 2E40B382B5E6; Fri, 10 Jun 2022 10:58:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 2E40B382B5E6 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=WgkZogUOA1yHttfRqOUW8mQl1SygOu/ZStV6/6cTEyGwxz/5fXb9RKQeEB0EKIMsuiXqDwBf/49NBahu1Hbl8tA+/EwMHSUzH8BoWzGovTWRZs7TCinh/roZTvTsPVSEWne1b7ac2CEpVeEGW6IspwSVHczUYZhqVRK/9wECeIndXDMxd1rKINt8oDfipLhTCG9wjJgIQ4o/GlioK/scu65EPOVa6QscJ26H3SB6RWnvglV+TsV8PBmIYhy8dHFt6Doy+9mPU23yzZtVhYCsDVC6TEd+JEhYXyPr6FV2Dw/oiYS5H3QdZ7LruS4POaFou5+yXLH3GPZN2hptkVeHMw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IkMtY+ZRgHfqZ7jg1TxshwnJ05dILMXtxQTSuhNQOJU=; b=UnQEl97bJiUSFqo7oNBQmXy7b4x1hdg8vDuqVqXP2B46J7OIrohtADZoZnxpb33oboIARXvj0Xvy1/aqn6Qh4edC3opKHbtLCnACZCwzJb5sKVvhBDfju3l7mPKBdAOEPxUp3e4X0dtYFaBueS2k/CCf8R+ygYK2qdeMSwaGJEBX+ddxYEm793Cl0jxonev8m2sjAyQHgsiQdqUl4x3IbEqUnBa5RNKbEKpat1XEb+PcOwFwt6cxaGgt29yTLp7PqFvlfsLLIAK867tjnw85wW/oDJM2kOSMDomM9G8MRv2tdj6BtqAZlERsPpiaanXrfF1A0DeKQpdufSkgEETPzw== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=sourceware.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dkim=[1,1,header.d=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IkMtY+ZRgHfqZ7jg1TxshwnJ05dILMXtxQTSuhNQOJU=; b=HQyUTAqyTyDMFBA78b6oDRX3Pw7IFNqBs97z9b2p76I+V3JehC96O2g5NywffL2cBvG3i/YNVorAbJxehGxS9DPFguBnZqYFqE/Azh11Vqsz2BHUTYARim65yVCNGXxyUo7lL7HO6qbQZ1mSYWVNTFQwMO6pn79YmVapnDtKUP4= Received: from DU2PR04CA0217.eurprd04.prod.outlook.com (2603:10a6:10:2b1::12) by GV1PR08MB7332.eurprd08.prod.outlook.com (2603:10a6:150:1f::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.12; Fri, 10 Jun 2022 10:58:44 +0000 Received: from DBAEUR03FT052.eop-EUR03.prod.protection.outlook.com (2603:10a6:10:2b1:cafe::c1) by DU2PR04CA0217.outlook.office365.com (2603:10a6:10:2b1::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.13 via Frontend Transport; Fri, 10 Jun 2022 10:58:44 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DBAEUR03FT052.mail.protection.outlook.com (100.127.142.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.12 via Frontend Transport; Fri, 10 Jun 2022 10:58:44 +0000 Received: ("Tessian outbound 1766a3bff204:v120"); Fri, 10 Jun 2022 10:58:44 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 3b679ed3ae593502 X-CR-MTA-TID: 64aa7808 Received: from 60180e6565c3.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 64B25E9B-5521-4A9B-85C3-97DC3651F1BF.1; Fri, 10 Jun 2022 10:58:37 +0000 Received: from EUR03-VE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 60180e6565c3.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 10 Jun 2022 10:58:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WDLee8DfWnJy2W4q7sJRBdeHWpc/fc4lCgViO7tSSBKGwcxagQ8QOrmCb3B59N5NduX9s4XsLiCiiAo+LW9p32hLs7Si/ongLCB3gEPcNPcW9o8IQLkZYJRB6u8WTZU74Y8HqMK/6HNWB5O/vPHOeIMHUjXWF197j4m3RaIUqoaKw1MiRrsW3Rug6xRzcqplRp6sQvXAk3VlRHhyq/HV8379xPQATsjlJwHRcOB10MCohKwXb44KZ1g81jjBvn+tDyn6pDSNI+S/4BrhVL2Wi///S6qmo7WYJk7aNO16cd0r5MkQBAl6g11w1IBRlqVpvU2SOjIYqASCMdbbMvbnzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IkMtY+ZRgHfqZ7jg1TxshwnJ05dILMXtxQTSuhNQOJU=; b=G47jwq84gPXigL4TZdCgV3RgySc8H9Xxef5R8443rHYkpPs7bCvwgrzZdzgX+00EIiOjvGJYvyYb347lGZWuOGzOTLHyNeW7GamK92kfRCrInR/glb6+YsCq2g3D7XbUagLs4wEErcnLfd39eSCtZNAYc1pYjs79Ac1Rdi88TGwZ886M+Z2cmIGyGTL9Gl6S8xQdxc07tmfx9Nv3vIUGtEQ+/kl/Zk79DH7yYyPp2NO1fOXzf1EcFAeYDATeteo0jEpOiYMF19TY+kjuUw6lGzQCrQcZ0lneUVw4nJNT3MNbCyCXSCE44jiEZ3MBFcizTdKvJXoCWrTURgxBFTJjsg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IkMtY+ZRgHfqZ7jg1TxshwnJ05dILMXtxQTSuhNQOJU=; b=HQyUTAqyTyDMFBA78b6oDRX3Pw7IFNqBs97z9b2p76I+V3JehC96O2g5NywffL2cBvG3i/YNVorAbJxehGxS9DPFguBnZqYFqE/Azh11Vqsz2BHUTYARim65yVCNGXxyUo7lL7HO6qbQZ1mSYWVNTFQwMO6pn79YmVapnDtKUP4= Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; Received: from VI1PR08MB3919.eurprd08.prod.outlook.com (2603:10a6:803:c4::31) by AM6PR08MB3208.eurprd08.prod.outlook.com (2603:10a6:209:4b::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.17; Fri, 10 Jun 2022 10:58:35 +0000 Received: from VI1PR08MB3919.eurprd08.prod.outlook.com ([fe80::d9c0:539c:a641:5735]) by VI1PR08MB3919.eurprd08.prod.outlook.com ([fe80::d9c0:539c:a641:5735%2]) with mapi id 15.20.5332.014; Fri, 10 Jun 2022 10:58:35 +0000 Message-ID: <85fa6e2d-caf5-8afe-a7ec-40cc62ff347a@arm.com> Date: Fri, 10 Jun 2022 11:58:33 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Subject: Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp Content-Language: en-US To: Mark Wielaard , Carl Love , Will Schmidt Cc: gdb@sourceware.org, buildbot@sourceware.org References: <20220608185138.675964-1-mark@klomp.org> <2281be8e-8920-7ba5-9e96-ffd713462b9c@arm.com> <8bfa7ec63758afc45bde1f10b0e4ab91e21e9d06.camel@klomp.org> From: Luis Machado In-Reply-To: <8bfa7ec63758afc45bde1f10b0e4ab91e21e9d06.camel@klomp.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO2P265CA0177.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a::21) To VI1PR08MB3919.eurprd08.prod.outlook.com (2603:10a6:803:c4::31) MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: cef3bf1d-6f8a-4ec5-6d90-08da4ad0302c X-MS-TrafficTypeDiagnostic: AM6PR08MB3208:EE_|DBAEUR03FT052:EE_|GV1PR08MB7332:EE_ X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: V9guTozYxTJqHNQ7V0yi5mLoHYmRJRqJptm0GCHvS0p2ZBpfKLrd77kOC3TjYN40DGMoeseqcgRNvklw8nOL4kzNnC14+uARx8KixOBMJvPeHJh92o7RuGG8ii1TiZWJ8JUd34b7X6ACQKzCHcG8NpfIfD3LG05sOh3OsY/lJeybyizOchrAi7Ijlkst7SbnYKAtpVA06VzAuwkOycBYJsyZT/+rFn35U471IHMcoxdE1YNM9ZNWvXYjIfVZ5XO+CbQED6HfMrLdmjIOVVlzPupm58qklDBHhkka077YTCsYmdckBPrOGwXGr9HuUBvK5attVJCsIgWVapPpykqHSFmAyUyGj5zD+8xLTnLodtKQv98bhZTOSW03K8xHYozr72HeBI2mcgwYWOKLE6T1QDyb6G19NtVmikZ5+JO8dBDWiWWaEvdzVVJTwh28L33YpGaX0tKDBoX0MURQSHJmiFI5+WNoeDF3pUuwfrQQKcbzIjlheAxPbP0LhIe42K/tASjeTUeB/xunHx6E3FzPzZDWgJqQViikSIG8k7rCTjV92Snlu+2YjiYOOGuAoqhsS/RyL+LQk4VuMDHY0Sbhj4caNkYL8wjCU07pOEp4/cET/fZXG/3rwbyG/ZH8r2fAEgA7B069d6nEwyn0g8u4JRpkzqflXtFWofGkMagcdV9JsZ767+xz0GbcRDgFmSCOlhRsz8jpc3sayc+qi/z8sPwsXPN62wdDbGeYfTTDnKVCN91ZEFR4aZ5JOaxhDByap+kG9hcIR+HGXcLAeFrVGNBpmUSxOlAwYhs7asq170zi4WJ7VHgtnnba1r1Y7raSmfmYfdTR7aB4Bb5yfD97fg== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VI1PR08MB3919.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(110136005)(316002)(36756003)(8936002)(5660300002)(38100700002)(31696002)(86362001)(66476007)(66556008)(8676002)(4326008)(66946007)(44832011)(966005)(31686004)(6486002)(508600001)(2906002)(186003)(2616005)(6506007)(84970400001)(53546011)(26005)(6512007)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3208 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DBAEUR03FT052.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: dff0e4c7-0d0b-4f4d-6f5a-08da4ad02a35 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LaktehdeiPGaN7kiW6vsfsra1CNyksa1iCkTy24Vd096xBD7KFQCx9+5ZTQoR80AtTxpOIzKFAw1d7m9Fee8ygdLKYUP6S7U/X9dXdEFtnmgD3A3FsfyHTbTrKFKvaHJfAGr97uY1HUQYOZqtMa1cFGRql7hDXLV4Ok3o55f8qVL3qpXGeD576Y+Cae3F8H/HwVrdQyQ+fOJCzXN5jHKkyBe2HVymwF3MNT+kOP26z/rm7ZjbCXeryxKTvjyjYArQo56hIStj33Pk+mlPLJUvVE3omDs3y3QNKcumFhi9osdoqjd5UXAJvJbPQECJqlVF48LO13hNlsKsznGPjI2fbYyoMuXMfEKLBSvHzCXGhLuL/sIMBkm2bk2w55RltEgu/Z1HDPnqXYQuwsfR6oOf4uv1W14h4P9EIdwt3DFw1cfzukwt5+k8Bt/YauJ4mirHevAx7iJLeF3ZLZ3lhxUNqCTUQxv8Y21Ye9EEHLqHJ2OppOfO+KRhBnzPVaAXdEZsNaBux0AYA5eRiHPa4oHQAvWpmAoAy9yG43wdGPtAWwbn4qzceW2gzDaS8k9EdN2wmiHIFKHSApQ9B7Cg9rxPj/TkL0Gmyx/ZsC3Icl22Oy/gOovkrMhn0R+gp0+U7iR1wV9aVUOmmgsdG4SrYrsnlq5mka2QhCvKYCRZ6lB7EL+XT6Vj0zqIgTlcnJg80fmXOKpyPxTE/WP5tsfXd7dVNG0QUZ/eirWq0saIooG/R8TuPyPIWYzMS6F7Y+paEqQwMgxK9TLSOiF1OJWnN4QJAWMp/QfgNRTAMiM+PBvTiw= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(40470700004)(46966006)(53546011)(31686004)(36756003)(44832011)(36860700001)(110136005)(4326008)(6506007)(508600001)(5660300002)(8936002)(186003)(8676002)(2906002)(47076005)(81166007)(316002)(450100002)(40460700003)(2616005)(70586007)(70206006)(336012)(86362001)(6486002)(6512007)(356005)(966005)(82310400005)(84970400001)(31696002)(26005)(43740500002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jun 2022 10:58:44.8061 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cef3bf1d-6f8a-4ec5-6d90-08da4ad0302c X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DBAEUR03FT052.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR08MB7332 X-Spam-Status: No, score=-7.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FORGED_SPF_HELO, KAM_DMARC_NONE, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: buildbot@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "The https://builder.sourceware.org/ buildbot" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jun 2022 10:58:51 -0000 On 6/10/22 11:50, Mark Wielaard wrote: > Hi, > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote: >> On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote: >>> On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote: >>>> I always use gdb.base/break.exp as a good smoke test. If that one >>>> fails, then things >>>> are really broken. >>>> >>>> I think gdb.base/break*.exp should make a good smoke test list. >>>> We just need to exclude >>>> gdb.base/break-interp.exp, which is problematic on some targets. >>> >>> It never is just easy is it? :) You are right, I saw break- >>> interp.exp >>> fail... I tried to come up with a regexp but gave up given that it >>> has to go throug python first and then we don't know whether the >>> worker uses bash as /bin/sh so I just added them all (exclusing >>> break-interp.exp) as a list. >> >> Sigh, sorry, looks like gdb.base/break-unload-file.exp also sometimes >> fails. >> I have removed from the list. Hopefully the remaining list does >> actually pass. > > And it didn't :{ > Yeah. As expected, the GDB testsuite is a bit delicate when you start dealing with multiple architectures and modes. But I think this is good progress already. > Both debian-ppc64 and fedora-ppc64le failed (UNRESOLVED) > gdb.base/break-idempotent.exp under both native-gdbserver and native- > extended-gdbserver > https://builder.sourceware.org/buildbot/#builders/76/builds/446 > https://builder.sourceware.org/buildbot/#builders/85/builds/294 Those might be genuine issues. I'm cc-ing Carl and Will so they can chime in. > > So I have removed that one too from the list. > The CI test list now looks like: > > # Only a small subset of tests that are fast and known to PASS. > gdb_test_exp = ("TESTS= " > "gdb.base/break-always.exp " > "gdb.base/break-caller-line.exp " > "gdb.base/break-entry.exp " > "gdb.base/break.exp " > "gdb.base/break-fun-addr.exp " > "gdb.base/break-include.exp " > "gdb.base/break-inline.exp " > "gdb.base/break-main-file-remove-fail.exp " > "gdb.base/break-on-linker-gcd-function.exp " > "gdb.base/breakpoint-in-ro-region.exp " > "gdb.base/breakpoint-shadow.exp " > "gdb.base/break-probes.exp " > "gdb.gdb/unittest.exp " > "gdb.server/unittest.exp ") > > Which will be run three times with make gdb-check, once without a > target_board, once with native-gdbserver and once with native-extended- > gdbserver on centos-x86_64, fedora-x86_64, debian-armhf, debian-arm64, > fedora-s390x, debian-ppc64, fedora-ppc64le, opensusetw-x86_64, > opensuseleap-x86_64 (debian-armhf only does a build, no make gdb-check > because of https://sourceware.org/bugzilla/show_bug.cgi?id=28561) > I'll add a debian-i386 builder so there is more 32bit coverage. > > All are green now (with the latest change to remove break-idempotent) > https://builder.sourceware.org/buildbot/#/builders?tags=gdb Looks nice! :-) > > Question is if this is a good list, does it need more tests? And should > it maybe be maintained in the binutils-gdb repo instead of in the > builder repo? > > For example we could have a make check-gdb-ci target which does what > the buildbot would do (and then the buildbot could just call that). Having a new check-gdb target that only does minimal smoke tests should be easy to do. Once we determine a subset of critical tests, we can put something together if folks think it is a good idea. I like it, as it make it easier to deal with stability issues of GDB's testsuite across different targets. > > Cheers, > > Mark