From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by sourceware.org (Postfix) with ESMTPS id 8B2F03858D37 for ; Wed, 17 Aug 2022 18:06:28 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 8B2F03858D37 Received: by mail-wm1-x331.google.com with SMTP id d5so4738470wms.5 for ; Wed, 17 Aug 2022 11:06:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc; bh=41zyRbFf3i27pE1T29Le5mdIN0NiqAKkuP+pCGRgIxo=; b=POeL+K1d24gd9mrHBFxn1thD45YUwrBjsq1niUihsxXRMMy7ALoAsL9LGumSKLhTT4 n03KAJ9s+sLfuFqmkWUiAvXZ9KTKwJBSAwZs+jEFf91EtE1wuihLEVVqq3IVZwEADRkL ltO986E3F9NJYegVASZF9ucAxwHiamWSVoRFagBEsbqQz8pzv88FZSgN5HbeHPTDtiHd p6scUzNWNDvOg91dHkWTbZGJANxzBXxjLFk8P+XeyDf1mxXqqWAu4jEkYYVLxanm3nxM nstOtI/Pr0EpffUvnziBR9GEEPdQk+PrnzCXjfh4OMWpl1dIXiy+vvNKucrbOa6FL0od EsxA== X-Gm-Message-State: ACgBeo3uLA9eDaXQ1sdDdLoUHcJr/Yk4J1t8FI6AQ7EC5IWZGYh6XoxP omkpgHN5W6yOJV7LeCYrPNd9Aw== X-Google-Smtp-Source: AA6agR5L5EsdY8feULIhuYHEQItoSUKqkuj5g2m/LLWi9quZ9I0x6SUlnPkZ1qeMygCuMRmam9fhlg== X-Received: by 2002:a05:600c:4f07:b0:3a3:2277:7064 with SMTP id l7-20020a05600c4f0700b003a322777064mr2830047wmq.181.1660759586605; Wed, 17 Aug 2022 11:06:26 -0700 (PDT) Received: from ?IPv6:2001:8b0:aba:5f3c:3694:2ef4:2932:37d5? ([2001:8b0:aba:5f3c:3694:2ef4:2932:37d5]) by smtp.gmail.com with ESMTPSA id j18-20020a05600c191200b003a5f54e3bbbsm3516845wmq.38.2022.08.17.11.06.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 11:06:25 -0700 (PDT) Message-ID: <35eec45bafcbb75c6254125b937516318f8fbec0.camel@linuxfoundation.org> Subject: Re: Reproducible builds - supporting relative paths in *-prefix-map From: Richard Purdie To: Mark Wielaard Cc: gcc@gcc.gnu.org, Khem Raj , Sergio Durigan Junior Date: Wed, 17 Aug 2022 19:06:25 +0100 In-Reply-To: References: <1c0ab23fd0cc1bc34a229287529963526141327a.camel@linuxfoundation.org> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.44.1-0ubuntu1 MIME-Version: 1.0 X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2022 18:06:30 -0000 Hi Mark, Thanks for the reply! On Wed, 2022-08-17 at 13:23 +0200, Mark Wielaard wrote: > Hi Richard, >=20 > On Mon, Aug 15, 2022 at 09:29:03PM +0100, Richard Purdie wrote: > > On Mon, 2022-08-15 at 21:55 +0200, Mark Wielaard wrote: > > > I might be misinterpreting the issue you are seeing. > > >=20 > > > But one problem with debuginfo/DWARF is that relative source paths > > > aren't clearly defined. If you move or install the executable or > > > (split) debug file out of the build directory a DWARF reader has no > > > way to know what the paths are relative to. > > >=20 > > > So for DWARF the paths always have to be absolute (they can still be > > > relative to the compilation dir (DW_AT_comp_dir), but at least that > > > has to be absolute (and the compiler should turn any relative path > > > into an absolute one or make sure they are relative to an absolute > > > compilation directory path). > >=20 > > It gets slightly more complicated as we build in a directory separate > > to the source where we can. Some source files are generated source > > files and placed in the build directory whilst many are in the source > > directory. DW_AT_comp_dir can be set to one or the other but it is the > > relative path between build and source which is problematic. >=20 > Could you give an example directory structure when that creates a > problem? Any debug paths generated should be absolute or relative to > an absolute path. In particular the DW_AT_comp_dir is normally an > absolute path (getpwd). So normally just remapping some prefix of the > pwd (either srcdir and/or builddir) should work to keep the relative > paths correct when moving the (generated) sources under that new > absolute path. I am probably missing something about the directory > setup that makes that reasoning invalid. An example from my build machine would be the gcc source here: /media/build1/poky/build/tmp/work-shared/gcc-12.1.0-r0/gcc-12.1.0/ and libgcc being compiled here: /media/build1/poky/build/tmp/work/core2-64-poky-linux/libgcc/12.1.0-r0/gcc-= 12.1.0/build.x86_64-poky-linux.x86_64-poky-linux so relatively that is: ../../../work/core2-64-poky-linux/libgcc/12.1.0-r0/gcc-12.1.0/build.x86_64-= poky-linux.x86_64-poky-linux On target we put the gcc source at /usr/src/debug/gcc/12.1.0/. To make this work we'd have to put the actual gcc source three levels under that, and replicate the "work/core2-64-poky-linux/libgcc/12.1.0- r0/gcc-12.1.0/build.x86_64-poky-linux.x86_64-poky-linux" structure, neither of which are things we really want to do. It gets more complicated as we support the idea of "external source" which is from an arbitrary location on the system but built within tmp/work/xxx. In the general case we can't remap all possibilities. > > e split the debuginfo into a separate package. We also look at the > > sources it references and those go into a different separate package > > too. We support populating a remote debuginfod server with these or > > installing them onto the target. >=20 > Nice. How do you lookup the referenced sources and where are they > then installed? We use dwarfsrcfiles which it looks like you may have written! :) https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/dwarfsrcfiles/= files/dwarfsrcfiles.c Installation is configurable but usually in /usr/src/debug/// > > > Using known absolute paths generated with debugedit or > > > -fdebug-prefix-map makes sure the paths used in the debuginfo/DWARF > > > are always the same independent from the current srcdir or builddir t= o > > > make them reproducible. And the user/tools don't have to guess what > > > the relative paths are relative to. > >=20 > > We have that working and set debug-prefix-map today. What is > > problematic is trying to recreate the relative paths on target between > > our source and build directories. Currently, most generated files in > > the build directory just don't get handled correctly on target. We'd > > like to fix that. There is currently no way to remap a relative path > > though, at least as far as I could determine. >=20 > So is the problem that when collecting the generated source files you > cannot map the debug-prefix-map back anymore? Correct, if we build with relative paths to autoconf (which we effectively have to so we can avoid lots of full path hardcoding), we can't collect up and remap the generated files. > The relative paths are relative to the original (absolute) prefix, but= =C2=A0 > you only have the remapped prefix? We can't recreate the relative paths on target (in the general external source case) and we can't remap them with the code as it stands today. Cheers, Richard