From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10047.outbound.protection.outlook.com [40.107.1.47]) by sourceware.org (Postfix) with ESMTPS id 4105E385780A for ; Wed, 30 Sep 2020 10:34:10 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 4105E385780A Received: from DB6PR0501CA0043.eurprd05.prod.outlook.com (2603:10a6:4:67::29) by AM0PR08MB3332.eurprd08.prod.outlook.com (2603:10a6:208:5f::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3433.32; Wed, 30 Sep 2020 10:34:03 +0000 Received: from DB5EUR03FT034.eop-EUR03.prod.protection.outlook.com (2603:10a6:4:67:cafe::f5) by DB6PR0501CA0043.outlook.office365.com (2603:10a6:4:67::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3433.34 via Frontend Transport; Wed, 30 Sep 2020 10:34:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; sourceware.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;sourceware.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT034.mail.protection.outlook.com (10.152.20.87) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3433.34 via Frontend Transport; Wed, 30 Sep 2020 10:34:03 +0000 Received: ("Tessian outbound 7fc8f57bdedc:v64"); Wed, 30 Sep 2020 10:34:03 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 5e4b91f248f85dd7 X-CR-MTA-TID: 64aa7808 Received: from ebe543c5441a.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 8497C069-B51A-4AC3-B0BB-BA4D52CB3447.1; Wed, 30 Sep 2020 10:33:57 +0000 Received: from EUR01-VE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id ebe543c5441a.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 30 Sep 2020 10:33:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=md0MKkiuPRxsq2Hb2FTqTKqkpSvg/C41j9HPfzwf+cbQwmnp5gsuKzkxO4ZWlukYHvX8UyW/kiGtmR7C3bt13OqJJVOuAKQt8NlRO0TvNI++O4cRu8yCtsYhvp4u1o5RGK7kGejj7GSGEtt5NtU2j0oTtNPxqqICxGloz52vhCs87QSY8sIGyJq8ZFytA++SdTn/wEpFeGQP0nptkrfoUKVeqhpTHhP9+xNBx0Tu6GnqVbMAvbAbMMPskfg20E09MFmhSiiT3epcMBekiMz8+G+7pyCDM0akTx1nGSP62pjD/a8Dg7GnFZQX1UTGDeqnbkUxohPF7HCkeJjvWO0LDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6qG2pJLlif21JilocYtAj91SPDjALL7bSWVxERGy6wc=; b=XO5oszXYVGOPeQmJZsdHYx8L6+Fsp92o1lVBFk82cxdWeZfetHXZiG+VnXCpOpPxsHQEy0acobxJXyIjDhIKMfcauXlFQyVhQGbQ2jwyPch1awyVzVnwLvY6QkdSbuxD4r2HlI4cAOvRHGRdwP6/MgQhHRSBN2F2Ea4DBz8Be9RDUMC4Oe7ck+2vPeZGD8P6cGvR2ETwXCz5ttEP5nTMpJpDjBSZ+TKe1aCejJnYFURljGciupWfaH1frnDjqpnU+tCvxyfBK87FX90TnWnhEUMTOo4UAWNepAEzu/CU5xHiEzsFFbuheHJvzytmN+0OAsDIdv1zkZNrQ7/LI172dg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none Received: from VI1PR08MB3198.eurprd08.prod.outlook.com (2603:10a6:803:49::20) by VI1PR08MB3198.eurprd08.prod.outlook.com (2603:10a6:803:49::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3433.35; Wed, 30 Sep 2020 10:33:54 +0000 Received: from VI1PR08MB3198.eurprd08.prod.outlook.com ([fe80::2d10:9409:39d0:dd1e]) by VI1PR08MB3198.eurprd08.prod.outlook.com ([fe80::2d10:9409:39d0:dd1e%6]) with mapi id 15.20.3433.035; Wed, 30 Sep 2020 10:33:54 +0000 From: Peter Smith To: Fangrui Song CC: "binutils@sourceware.org" Subject: Re: [PATCH] dependency list for static libraries Thread-Topic: [PATCH] dependency list for static libraries Thread-Index: AQHWlw4S9+VyiH/LgU+Ws/W+4ClM+A== Date: Wed, 30 Sep 2020 10:33:54 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: Authentication-Results-Original: maskray.me; dkim=none (message not signed) header.d=none;maskray.me; dmarc=none action=none header.from=arm.com; x-originating-ip: [82.30.113.194] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 8caff346-d5f3-4675-e18b-08d8652c59d2 x-ms-traffictypediagnostic: VI1PR08MB3198:|AM0PR08MB3332: X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: xiArM9S6p7+JgIRd0qBI8C29xVifoWzMnggGo2skLlJ6omNDyGqRpmJBmWmm0nreELKdBhwBMmL86H6hOZoIH1VW2H87H7Q9TzYxff+Z4p0Y9Bz2SPMlSu3MP4HE2QUjzcqOtW22m4vmkABwTJRfuguda8+k5hCuVl/erLwDb7yIXt2VgHet5zITBuDPoSW78f5N+U5Mczr5m5NINgTr9gcINEgH8dMU6TRnyqTefNMxsNM4vNE+HPHx8/SlJvrZX5CZke4jggTQZq9m8ZiB39+GPHaESVAEW30jMVxUXTx9jwfroXrYeD4yQpkXwSXT38KNfFSMDfUfzlo7m4jytAV+36uYFPzfWPSKWeITaL0Z1i1ZZrzU9WQc48a5UaQAieJ0I/zWPApuAGMiAclB2iJql+wRGixYp6gtKqpKsn/zE+TwF34cT7xNqccppokw X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VI1PR08MB3198.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(39860400002)(346002)(396003)(376002)(366004)(4326008)(9686003)(6506007)(55236004)(71200400001)(5660300002)(8676002)(966005)(86362001)(8936002)(83080400001)(66446008)(66946007)(76116006)(64756008)(26005)(478600001)(55016002)(83380400001)(52536014)(186003)(7696005)(6916009)(316002)(33656002)(66476007)(66556008)(2906002)(781001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: 6eq2HbHSfeZif5CgOFG9XEdeejJu+0+hh/2YxelaVEX+Y/yAcVjN/j4XB/NDax6NOKh8gUUi3Jtj/fBhiJtbztvRp/7r6UdQy9pE4u0RXvYiqEMnFCE48rv/R6OOtVUpeglRLgeTmcwtcdtD/ivDRqAOpSCo/oVGN+OpGQJrdNr4j6XywXM22xL73FbOqljABJLO6dFf//97gmIg/WYdEfZQidOMErIIaLXrLfsl3BdMlhj3CDBlFI0BO9fYoxF5PbINM+hj25QoHwuB30kv3MkeLT12GNby2pDQDjh3PKLqvBtrg36a3C94e1kF1uRNe6KipUz59uFCD5xdnwSqTyjhg2csBBPB/ocgWDv0gth5Am9YuYtN9sqET1xHhMF+74LuJRYnnVXTId/05P76ltF+gh7+WpYglYm3LdEP3yCIDO1yU8oDn6CNGrcfYWYGC7KOMuVASDWdQTeIomyVdOzak00UsLUmkYJ6mdz8Z2lm/YCjHs3rY3UNwQvJ249pBf6w7AVprWIEDycixDkk5zAAhEcUdU5OchpxH0a13LuazMaYJzZVvqPhMYB3/YdxJvO7eUDKAmh/nCtl2r4txpEshI9fhvqlFBs+vdLlzAoQgF2AP8jxBojLWHSMvvr3pXC+Z5GCe0PGjPq1R7uiSw== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB3198 Original-Authentication-Results: maskray.me; dkim=none (message not signed) header.d=none;maskray.me; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT034.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 6cd5d710-2932-4807-e465-08d8652c5495 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iYqGy+3RgCDZ/+a0seaU55jUJB6ZgvSPAD/Xijz9xm8uv1SNInhI5BIn7oBgtiMND41NCOkoCOFSu0GcbbyZYlPkdJq3SVB/p7Wy4Dh/V2M7Ogj683Zp+kfaNC+KLkOFpNmWJmWKECthnSOVI4gkJeRL6NCJpnQsYZKITfRLNvuOQbqI/Fp7gZi06aIfmwjKN96sizjKL1p91tLHYyrW5A7T3kKN9zoqSBCO7zHB3wPvS5yC8/ajkTy3oYkMPX1aMsdXI7N5nU7YZ1fzYLTvrl34+12my5sf5FqiIaEGyfku7Pq9EH8hP4Bzt5tTEaxfONo73n4KuJcUhbseJTPRSievA3Jm2UpJ/r4wcfHlTx9Jbfk5l09bMZiylbjgj+JnAH0zL/d5ku+t8DYY6mTgRIqtSXxRUcjm8QtowGdcgTYirx02HSqZ3gQ5xPyOt+vPgMyHurvHaOfe8xUE3klgVmTDCX+wKUVeMkKlNUCPAe8= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(376002)(136003)(39860400002)(346002)(396003)(46966005)(2906002)(336012)(5660300002)(55016002)(26005)(86362001)(7696005)(33656002)(186003)(52536014)(55236004)(83380400001)(6506007)(8936002)(70586007)(6862004)(70206006)(4326008)(316002)(9686003)(81166007)(83080400001)(8676002)(356005)(82740400003)(82310400003)(966005)(478600001)(47076004)(781001); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2020 10:34:03.2814 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8caff346-d5f3-4675-e18b-08d8652c59d2 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DB5EUR03FT034.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB3332 X-Spam-Status: No, score=1.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP, UNPARSEABLE_RELAY, WEBMAIL_BODY autolearn=no autolearn_force=no version=3.4.2 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: binutils@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Binutils mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Sep 2020 10:34:12 -0000 > Thanks to Nick for mentioning me :)=0A= >=0A= >On 2020-09-23, Howard Chu wrote:=0A= >>Nick Clifton wrote:=0A= >>> Hi Howard,=0A= >>>=0A= >>>> Sorry for the noise, this attached is the same as previous but with fi= xes=0A= >>>> to whitespace / indentation.=0A= >>>=0A= >>> Thanks - the patch looks good now, although there are still a couple of= minor=0A= >>> issues - and one major issue:=0A= >>>=0A= >>>> * The new test fails for the alpha-vms target. This is not serious = however=0A= >>> as almost all of the ar tests fail for this target. One day I will= track=0A= >>> down what is going wrong and either fix it, or arrange to skip thes= e tests=0A= >>> for alpha-vms.=0A= >>>=0A= >>> * There are a couple of minor formatting issues. Specifically: comme= nts should=0A= >>> end in a period followed by two spaces before the closing marker. /= * Like this. */=0A= >>> Plus function calls should have a space between the function name a= nd the=0A= >>> opening parenthesis of the argument list. like (this)=0A= >>=0A= >>I can send an updated patch after the major issue is addressed.=0A= >>=0A= >>> The major issue is that I would really like for this extension to be su= pported=0A= >>> by the LLVM community as well. It would be a shame to add it to the bi= nutils=0A= >>> only to have a different solution implemented there. I might have misr= ead the=0A= >>> emails but I believe that Fangrui still has some misgivings about this = approach.=0A= >>> Is that correct ? If so, then I would very much like to see them resol= ved=0A= >>> before we commit the patch.=0A= >>=0A= >>It still seems pretty obvious that handling dependencies once at the libr= ary=0A= >>level is more efficient than sticking a bunch of free-form notes in every= single=0A= >>object file. And it should also be obvious that choice of library is a bu= ildtime=0A= >>decision, not a decision made solely at the time the source code is writt= en.=0A= >>(And for shared libraries, it's a runtime decision - even further removed= from=0A= >>the time of code being written.)=0A= >=0A= > From my experience (I have investigated hundreds of link issues),=0A= >archive link errors are usually caused by the following two problems:=0A= >=0A= >* There is a backward reference between two archives. This is related to a= .out/ELF style linker >behaviors=0A= >(https://sourceware.org/pipermail/binutils/2020-September/113194.html ).= =0A= > + A dependency exists but is incorrectly omitted. The linker actually c= aptures a layering problem.=0A= > + A dependency is intentionally omitted. If the linker allows backward = references, this will not be >a problem.=0A= >=0A= > I have recently thought a lot on the topic and written http://lld.llvm.= org/ELF/warn_backrefs.html=0A= >=0A= >* Some archives are incorrectly omitted: the executable uses A, but A's=0A= > dependency B is not on the command line. Now I hope Howard can clarify= on this=0A= > topic:) What do you want to achieve with the dependency recorded in the= archive?=0A= > I assume that you want the linker to smartly link B. This is indeed the= #pragma=0A= > comment(lib, ...) and .deplibs feature clang/LLD support.=0A= >=0A= > The proposal does not mention the ld part, which seems important. How= =0A= > does ld handle __.LIBDEP ?=0A= >=0A= > + The executable also references B and you don't want to write its depe= ndency on B.=0A= > This is the traditional --copy-dt-needed-entries behavior (that the= =0A= > designers of gold explicitly do not want to support). The linker trav= erses all=0A= > dependencies of shared objects recursively. This is against the "inclu= de what=0A= > you use" philosophy. https://github.com/include-what-you-use/include= -what-you->use/blob/master/docs/WhyIWYU.md=0A= > It imposes a lot of extra work for the linker. It makes the build bri= ttle - if=0A= > A removes its dependency on B, you need to find and fix all the trans= itive users=0A= > of A. I hope this proposal is not about the bullet point.=0A= > + The executable does not references B. I am still unsure why you want = to move the dependency information from the=0A= > object file to the archive.=0A= > - First, I want to mention that in some cases archives are not needed= . For=0A= > some use cases people propose thin archives. For some thin archive= use cases,=0A= > --start-lib --end-lib surrounded object files are probably more use= ful. gold=0A= > pioneered the idea but GNU ld still does not support the feature=0A= > https://sourceware.org/bugzilla/show_bug.cgi?id=3D24600=0A= > - Moving the dependency to the archive can actually increases the wor= k for=0A= > the linker. Say the two members of A are a0.o and a1.o. If a0.o de= pends on B=0A= > but a0.o is not selected, then B does not need to be linked. Having= the=0A= > dependency in the archive will cause B to be linked.=0A= > - How do you ensure the dependency information does not become stale?= i.e.=0A= > How do you maintain the dependency information when members are add= ed to/removed from=0A= > the archive? The information is more difficult to maintain than the= index, which only contains=0A= > the definitions. You may argue that the archive is finalized after = it is created - then=0A= > we go back to square one, --start-lib may be more suitable.=0A= >=0A= >---=0A= >=0A= >Last,=0A= >=0A= >> fprintf (s, _(" [L LIBDEPS] - specify dependencies of this library\n"= ));=0A= >=0A= >In llvm-ar, 'L' is an extension used with 'q' to add all members of an arc= hive to the current archive.=0A= >It is similar to ADDLIB (https://sourceware.org/binutils/docs/binutils/ar-= scripts.html )=0A= >Hope the two tools don't have conflicting operations.=0A= =0A= Apologies in advance if the webmail client mangles the response or threadin= g.=0A= =0A= Just wanted to add to MaskRay's concerns about what the semantics are for a= linker.=0A= * Where are the static library dependencies placed in the order of input fi= les? Immediately after the static library seems the most logical choice alt= hough this will make it harder for users to interpose objects/libraries to = override choices.=0A= * What happens when there are cyclic dependencies between libraries? Would = the user be expected to know to wrap an individual library in a start-group= end-group? Should the linker implicitly add start-group end-group around t= he dependencies?=0A= * This does seem like it will work for libraries designed to be incorporate= d as a whole (like a dynamic library), but it seems like it could have awkw= ard corner cases.=0A= =0A= Although not fundamentally opposed to this; personally I'm not a fan of enc= oding dependencies for static libraries. I see these more of a collection o= f objects and not a single entity like a dynamic library. If static librari= es are being used whole, one way of implementing this would be to add a mem= ber to the library that is just a linker script with INPUT and GROUP comman= ds, and link using --whole-archive.=0A= =0A= Peter =0A= =0A= (infrequent LLD contributor)=0A= =0A= =0A= =0A=