From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2051.outbound.protection.outlook.com [40.107.244.51]) by sourceware.org (Postfix) with ESMTPS id 64FE53858C2D for ; Fri, 12 May 2023 16:55:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 64FE53858C2D Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=roku.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=roku.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HGhc3uDyCm9R8Pjm7kA45/yOqfmJGiprrhio0B1zUzD+T3Z0JwD9Afg5lxTJTsf0ZleKfgFWT7FY8GUf2XPFy5F0tjgp+joS9+n9RL7vjmQDpZhnHvlLzYrP0SAypT3gj5tbvJRjPrOSFghQl7MLBJjKIFhDuZNz1DBzLn1mFD8d355VAG1Z0vGgS80QUradmuNe2iyb0Ecz8XmoLGzqzZHDTSqMNzZvcq9yrXJdXb4j1x42ZZtq1Pc8LaNoJHiNuiJZDTxbWrzxiu2/jGZHB7nNN/yu/5xhzR5s5kpgrhgVyQ1gZbiugWgmdLxVAJpsvpOmo7VgJ27pPl9KReJ4Ug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qmhzGX4X/Uce5mfsyyLNv6szChKDHn18KgQMYDhI3So=; b=hlC+cy04b9jB+btbnx/eGuYisS4LIDxXUvUpIOq6cdi8Niel2+GWAK+SZKKc/43TMNQvICWh+OFovfdBUQEOf6vGxO9yhOP0TvTHD9oUZyclkq0/pFDFxYN9ivy46CQZtsL+o+5z0em7Wl/O+yQPSs+k5N2CLPDqXBXXy1h2oTRsTtRXuh9CoQ9RrcIxEPTnBEsNmvfWtOt5XHKQpUwho08AAXMexx7FL8EEO2UEIMRJxN/QkaFqF+atO3N88b6yKV4IUgiXX7huvj2uuqfxP33dtX98qWpkdYLuNJUfgiz1lxzjlvwX+1smLTUj+iF1BUBFKEXEWPmFF2C5Ra9q1Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=roku.com; dmarc=pass action=none header.from=roku.com; dkim=pass header.d=roku.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=roku.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qmhzGX4X/Uce5mfsyyLNv6szChKDHn18KgQMYDhI3So=; b=H4UeKzY27c+TpLzkt8iVecPfS/tDCqQMGMpryLoIzljZ8uAuNKdsNNRK7bBdvvr5J5M1SuMDBoly21jltNfMLLImANx6x5iKGqOcYirRXsm4yw0na1Z5cz1cOfVRQdsbvE9aJ9wfhy7Kn82cM8Yloc/EYZc5F1TCRVYYsfBKy4zOXnh7mL1ddIL32tvxUhiuWLzwGrNqBxKYdQXS63nJ3n4l4bN3iRLXwvtc2LMzOzxXMnCIv8piNY+pYykKL3z3346IBuQwmMAZUxxND98Y5DHO9i4Js1y64PnjCJV9vo9VzT2Gh0QiaXKxYu6vwXJlH/uGsLeP+aqWgsNfLSvcnQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=roku.com; Received: from SN6PR01MB4285.prod.exchangelabs.com (2603:10b6:805:ac::33) by DM5PR0101MB2937.prod.exchangelabs.com (2603:10b6:4:34::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6387.20; Fri, 12 May 2023 16:55:52 +0000 Received: from SN6PR01MB4285.prod.exchangelabs.com ([fe80::30ab:946a:55cd:522d]) by SN6PR01MB4285.prod.exchangelabs.com ([fe80::30ab:946a:55cd:522d%6]) with mapi id 15.20.6363.029; Fri, 12 May 2023 16:55:52 +0000 Message-ID: <3a148590-1dbe-22a5-e98e-797b173c2a34@roku.com> Date: Fri, 12 May 2023 17:55:46 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.5.0 Subject: Re: [EXTERNAL] Re: Hitting g dwfl->lookup_elts limit in report_r_debug, so not all modules show up and backtracing fails Content-Language: en-US To: Mark Wielaard , Florian Weimer , Luke Diamand via Elfutils-devel References: <87y1m7tbaw.fsf@oldenburg.str.redhat.com> <20756d5ff06a0541e51e2f862312f107a60d525d.camel@klomp.org> From: Luke Diamand In-Reply-To: <20756d5ff06a0541e51e2f862312f107a60d525d.camel@klomp.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: FR0P281CA0220.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:ac::15) To SN6PR01MB4285.prod.exchangelabs.com (2603:10b6:805:ac::33) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN6PR01MB4285:EE_|DM5PR0101MB2937:EE_ X-MS-Office365-Filtering-Correlation-Id: 52968668-439c-4da3-66bc-08db5309bed1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: M3rDTyzSWcjXq+oJd2ypmYMJLK7/4WfYt8wDCvPCpSKUvb4wJJMFRLGy6Nz+xqckdLPdKdcA8ZO1Pr9fHqa7esLvlLqMWq531Z37D8HV9vy5hjh/tGp/tD42DPikXCNWSx4uVEsh1bMJYKjdauiHBdIkndeRzYkBy9R+ByzEnv1GnUKVxuv2UDqfjXgtoBqBKcKnlWmMjniMjjkX7C9tgJecm23Hz6HhnpxDW112uT1VrMeVyDh3a1U2Dmu7FkEgzsCUkfSeg1HBk6i+zRQtZ4NDoU1FYnUEDxmVHSCtDPG4gTnSr6Bui/HGcdOO2sWxICWqPvytZXGR7ulgLYLPbcxDd9A9pdCQBqBBJ9UDPXoWOyFPplbjdZ2FenIninhFCJgLGWHC+PYOA+SGUcid8zoqBUZCnbeP+L8rCd6vPpjcsnIbqmCKvBIE31g2qZGsk38iCkgMxcz9M9AGi28YZNUkgPRXlVN5ij/QpmW42SEJoKAuX+Vy6ipwkSTjezAnrH+3xEcVpqQs396lBhqaCRNmWRIweUkagHWPHAB70wQFUXUwM4ZtKKiQ/2SEFuEoHoAuqx5qphlH4tO9Apv6E8+adtqZ2Etx7ebZh4dq78dIHOSgM8yyyU7TyWKOKSW4 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR01MB4285.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230028)(39860400002)(136003)(396003)(376002)(366004)(346002)(451199021)(186003)(2616005)(2906002)(83380400001)(38100700002)(86362001)(36756003)(31696002)(6486002)(8936002)(41300700001)(316002)(8676002)(6666004)(31686004)(5660300002)(478600001)(110136005)(66556008)(66476007)(66946007)(26005)(6512007)(6506007)(53546011)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UEVET3lQUVBQc2htY3RkSGxmc2ZOZVRZOXR0ZEhEb2QwTGw1T2wrZXd6b21H?= =?utf-8?B?VlQ3UnMyVE9VZjZhaFZLdS9wc005Ui80WkhQV3hTYTFvRFlOVlcvbFQ2bkhM?= =?utf-8?B?azlzT2M0aytMbUg3ZjZFZTZrUTFoTlpNZEJkS2Fva20rRVUyRlRsK0xEZzBo?= =?utf-8?B?V21ONCt6QWhUVUVlV2x3STQzeVovdm5ybXFpTnJhdElueUJHSjFEOVUzeW1C?= =?utf-8?B?TnNsZ2JudFZXbzVxOFgyN2VGMmNwaVg3V0V4MVBBaXcrYlk5Wit4aHpTRXJm?= =?utf-8?B?b1pzWUsrckZwYnUweGJIWmtxQzYxZDkrYVQ4dnc3WTlFODV2R21Vd1BrM3li?= =?utf-8?B?ZXQ3UGx2OWQ3dmlMeUVPYVY0TWtVNnRNNVJxaEVhZzgzZEc1TXdPZXdFakdJ?= =?utf-8?B?amRyRE5wRDdaVklhdUpibG94S3ZSSm5VaXp3VzJZRnRtb05YQmtoaSsvSUV6?= =?utf-8?B?aUtQcVZqc2thZzFBQ2tZN1dORlFad1paUFluUm9neUE4aWtwUm5LUEpiQVo2?= =?utf-8?B?ZGpwZWhJKzZ6dmV1dHA5enZwRGRsKzU3ckhoSERRK3Vvd2xGbUloTFFSOWow?= =?utf-8?B?SU12M1VrUm9PdEkvNzI3RitpTHgzc0NSM2kwZFlMMnZvRXBOZXh0YW1GeW53?= =?utf-8?B?bHJYOSs2UkhTQWg3SmQyb0pVSXFNclZhYVBjRG1iNC9KMTExQjZ4K24zcEo2?= =?utf-8?B?elJlYjZoNkJLY3RlRjMyRkgwcnRCVncyYWNmVVBrUmJ3c3ZkelllTHNKSHk5?= =?utf-8?B?NkthaWxLYU95OEhJV0NRcklxRVoyZ1dQMjdiMGVEbmNKOEI3RnBpQkxTV040?= =?utf-8?B?Ym8yWi9DTnVxemtDYStvTk1tOTd1akZaKzU4SlYxWmxncmd4a1dyOXZkNDJ0?= =?utf-8?B?RlFnMzJ2REc1WHpMdlVWQ1lhYWczdkM1K1hJdmM0MXpqUGJLaDBiUGJkdFUv?= =?utf-8?B?NjZzNkFkVjlSRDF1K3NsSUZvL1JCa2FpaS8wWlg3VDN3SGo0alY4Yi9JN2pv?= =?utf-8?B?eGhTUStZbWt4Wk1YQThsWklrQWIxU2VlZDF6cnl3eGVoRkg4NWQ2dFZncUFP?= =?utf-8?B?WDdGcXI2R0lOQ25QSW1iVXhxMm9nYWVyTnl2cTNCQUdnSU9pc3Z4Q2NwNXJW?= =?utf-8?B?bVhHSFhQaDlNK2ovREkyeDVtZVpYSGtNbjNYckl0TC9vU1hQeVp3UUhpa3JK?= =?utf-8?B?dnhRQU9aaTBkakNxYXlNWTBDWFl4Tm5Ea2M4MDJwZFR4MGE2SUloMk96ZXh1?= =?utf-8?B?aHdCOFppN1l3Znp6RXp5USt0YWxOc3BiM0dnOVdXNTJnSk9XVjR4MXdFcFp2?= =?utf-8?B?Y0Jmb05YU3NwdFFpUENNTEpYR3Zub0hydy9qbHNJckpRa1ZFMGNtclBxQll0?= =?utf-8?B?QWwrNDNONW56MUdlVDcxUXVMREtNanVLVWN1dUtCQVBMRjdtU25wdXp1ZXlV?= =?utf-8?B?Rzl5RTZNY2x4TGlpV3pRNTNIM3R2cmMrYUtld1dzVXhzeitnTjlwQ0FyNW44?= =?utf-8?B?Tmp6R3J4MDRJdG55NjgrMXljMXRXUnZhbVBnUjFaNGlyclBNRFZzd3hmdW4z?= =?utf-8?B?TzdJVW5sdFFxRElydm53b2I3UU5Kb0ZKYkhKazIzeVgya01EOHNGSFVvY1Mx?= =?utf-8?B?NFZxdVU2TnJYOEUzYi93M3J5VjZvOUZSeDhZRFFnV2lrM0pRV2RYcHRFNTRo?= =?utf-8?B?ZjBUWGxoZGtMMWh2S1VnRXJ3MWNNaHJ6dVdlczBaYmtrS0xNK0Y1aCs5cmlP?= =?utf-8?B?SGx6MzdJWjVLc2x3VUFVUGFxc3V3TjE4QkZHOExRMzkvVzZOOTVWdkNiN1A5?= =?utf-8?B?U3d3alA2a3M1eTBmMkxGL0lpVXRicVJETkRUSEZGUXJTM1ZpK2VteWJnd0dZ?= =?utf-8?B?clNUNG1hSUJaTER3UXRnV3FJbDFDdm5yZTdIN2E3TW4rK0c5ZjVqR0V3RjA5?= =?utf-8?B?TGRFMjBpblh3Y0hkQmFBdUx5SUZNYWRybEdldHlzSUxMVXhVeEF0RUVzUUtl?= =?utf-8?B?Q1g3NUlBMEhBMDByZDBTTHFTZ1ZWVzZ6MC9KN2ppSVVaa01pYVV6RmtvVmRU?= =?utf-8?B?blpuRE1kaFdlbUQ4WDE4cFpFazRaZGFzYUtUQmxxY29qdGpBaER0R0hrUGND?= =?utf-8?Q?wUrfnPr4uc8TIEkP2gjsFiRmy?= X-OriginatorOrg: roku.com X-MS-Exchange-CrossTenant-Network-Message-Id: 52968668-439c-4da3-66bc-08db5309bed1 X-MS-Exchange-CrossTenant-AuthSource: SN6PR01MB4285.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 May 2023 16:55:52.5633 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 00fc7e42-ad65-4c4c-ab54-848ba124a5b7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: iiy5n8vJYpULosmfrZvmw6rcr9E4D9yjraVuzacGNw9eBsgSW2eX/ZqlbQl/potVCloTKez/8M97qevFHqTrQA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR0101MB2937 X-Spam-Status: No, score=-11.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE,URIBL_BLACK autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 08/05/2023 17:35, Mark Wielaard wrote: > Hi Florian, Hi Luke, > > On Tue, 2023-05-02 at 09:57 +0200, Florian Weimer via Elfutils-devel > wrote: >> * Luke Diamand via Elfutils-devel: >> >>> I've got a few cores where report_r_debug() in link_map.c fails to >>> find all of the modules - for example I had libc.so missing. This >>> obviously meant that elfutils could not backtrace my core. >>> >>> It seems to be related to this code: >>> >>> /* There can't be more elements in the link_map list than there are >>> segments. DWFL->lookup_elts is probably twice that number, so it >>> is certainly above the upper bound. If we iterate too many times, >>> there must be a loop in the pointers due to link_map clobberation. */ >>> size_t iterations = 0; >>> >>> while (next != 0 && ++iterations < dwfl->lookup_elts) >>> >>> I've changed this to just keep going until it reaches >>> dwfl->lookup_elts*5, which seems to "fix" it, but I feel there must be >>> a better fix! >>> >>> The most recent core I saw with this had lookup_elts=36, and hit 109 >>> iterations of the loop and then backtraced just fine. >> >> It's probably another fallout from -z separate-code, which tends to >> create four LOAD segments. The magic number 5 sounds about right, as >> gold also has -z text-unlikely-segment, which might result in creating >> that number of load segments (but I haven't tried). > > Wow, that had never occurred to me. Thanks. > > Luke does the binary/libraries from which your core file was generated > contain multiple PT_LOAD segments? > > We could add something like: > > diff --git a/libdwfl/link_map.c b/libdwfl/link_map.c > index 06d85eb6..76f23354 100644 > --- a/libdwfl/link_map.c > +++ b/libdwfl/link_map.c > @@ -331,11 +331,17 @@ report_r_debug (uint_fast8_t elfclass, uint_fast8_t elfdata, > int result = 0; > > /* There can't be more elements in the link_map list than there are > - segments. DWFL->lookup_elts is probably twice that number, so it > - is certainly above the upper bound. If we iterate too many times, > - there must be a loop in the pointers due to link_map clobberation. */ > + segments. A segment is created for each PT_LOAD and there can be > + up to 5 per module (-z separate-code, tends to create four LOAD > + segments, gold has -z text-unlikely-segment, which might result > + in creating that number of load segments) DWFL->lookup_elts is > + probably twice the number of modules, so that multiplied by max > + PT_LOADs is certainly above the upper bound. If we iterate too > + many times, there must be a loop in the pointers due to link_map > + clobberation. */ > +#define MAX_PT_LOAD 5 > size_t iterations = 0; > - while (next != 0 && ++iterations < dwfl->lookup_elts) > + while (next != 0 && ++iterations < dwfl->lookup_elts * MAX_PT_LOAD) > { > if (read_addrs (&memory_closure, elfclass, elfdata, > &buffer, &buffer_available, next, &read_vaddr, > > Does that sound reasonable? Sorry - I did not see this until just after sending in my patch! Let me try it with this change and I will re-roll it. Luke