* [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) [not found] <87egf1mxfl.fsf@alice.fifthhorseman.net> @ 2015-12-10 17:36 ` Daniel Kahn Gillmor 2015-12-10 22:05 ` Daniel Kahn Gillmor ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Daniel Kahn Gillmor @ 2015-12-10 17:36 UTC (permalink / raw) To: gcc-patches; +Cc: rb-general, anthraxx, niels, Daniel Kahn Gillmor Work on the reproducible-builds project [0] has identified that build paths are one cause of output variation between builds. This changeset allows users to avoid this variation when building C objects with debug symbols, while leaving the default behavior unchanged. Background ---------- gcc includes the build path in any generated DWARF debugging symbols, specifically in DW_AT_comp_dir, but allows the embedded path to be changed via -fdebug-prefix-map. When -fdebug-prefix-map is used with the current build path, it removes the build path from DW_AT_comp_dir but places it instead in DW_AT_producer, so the reproducibility problem isn't resolved. When building software for binary redistribution, the actual build path on the build machine is irrelevant, and doesn't need to be exposed in the debug symbols. Resolution ---------- This patch extends the first argument to -fdebug-prefix-map ("old") to be able to read from the environment, which allows a packager to avoid embedded build paths in the debugging symbols with something like: export SOURCE_BUILD_DIR="$(pwd)" gcc -fdebug-prefix-map=\$SOURCE_BUILD_DIR=/usr/src Details ------- Specifically, if the first character of the "old" argument is a literal $, then gcc will treat it as an environment variable name, and use the value of the env var for prefix mapping. As a result, DW_AT_producer contains the literal envvar name, DW_AT_comp_dir contains the transformed build path, and the actual build path is not at all present in the generated object file. This has been tested successfully on amd64 machines, and i see no reason why it would be platform-specific. More discussion of alternate approaches considered and discarded in the development of this change can be found at [1] for those interested. Feedback welcome! [0] https://reproducible-builds.org [1] https://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20151130/004051.html --- gcc/doc/invoke.texi | 4 +++- gcc/final.c | 27 +++++++++++++++++++++++++-- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index 5256031..234432f 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -6440,7 +6440,9 @@ link processing time. Merging is enabled by default. @item -fdebug-prefix-map=@var{old}=@var{new} @opindex fdebug-prefix-map When compiling files in directory @file{@var{old}}, record debugging -information describing them as in @file{@var{new}} instead. +information describing them as in @file{@var{new}} instead. If +@file{@var{old}} starts with a @samp{$}, the corresponding environment +variable will be dereferenced, and its value will be used instead. @item -fno-dwarf2-cfi-asm @opindex fdwarf2-cfi-asm diff --git a/gcc/final.c b/gcc/final.c index 8cb5533..bc43b61 100644 --- a/gcc/final.c +++ b/gcc/final.c @@ -1525,6 +1525,9 @@ add_debug_prefix_map (const char *arg) { debug_prefix_map *map; const char *p; + char *env; + const char *old; + size_t oldlen; p = strchr (arg, '='); if (!p) @@ -1532,9 +1535,29 @@ add_debug_prefix_map (const char *arg) error ("invalid argument %qs to -fdebug-prefix-map", arg); return; } + if (*arg == '$') + { + env = xstrndup (arg+1, p - (arg+1)); + old = getenv(env); + if (!old) + { + warning (0, "environment variable %qs not set in argument to " + "-fdebug-prefix-map", env); + free(env); + return; + } + oldlen = strlen(old); + free(env); + } + else + { + old = xstrndup (arg, p - arg); + oldlen = p - arg; + } + map = XNEW (debug_prefix_map); - map->old_prefix = xstrndup (arg, p - arg); - map->old_len = p - arg; + map->old_prefix = old; + map->old_len = oldlen; p++; map->new_prefix = xstrdup (p); map->new_len = strlen (p); -- 2.6.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-10 17:36 ` [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) Daniel Kahn Gillmor @ 2015-12-10 22:05 ` Daniel Kahn Gillmor 2015-12-10 23:59 ` Joseph Myers 2015-12-22 12:16 ` [rb-general] " Thomas Klausner 2 siblings, 0 replies; 12+ messages in thread From: Daniel Kahn Gillmor @ 2015-12-10 22:05 UTC (permalink / raw) To: gcc-patches On Thu 2015-12-10 12:36:18 -0500, Daniel Kahn Gillmor wrote: > Work on the reproducible-builds project [0] has identified that build > paths are one cause of output variation between builds. This > changeset allows users to avoid this variation when building C objects > with debug symbols, while leaving the default behavior unchanged. I've now opened a bugzilla issue about this as well, with patch attached: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68848 --dkg ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-10 17:36 ` [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) Daniel Kahn Gillmor 2015-12-10 22:05 ` Daniel Kahn Gillmor @ 2015-12-10 23:59 ` Joseph Myers 2015-12-11 0:13 ` Daniel Kahn Gillmor 2015-12-22 12:16 ` [rb-general] " Thomas Klausner 2 siblings, 1 reply; 12+ messages in thread From: Joseph Myers @ 2015-12-10 23:59 UTC (permalink / raw) To: Daniel Kahn Gillmor; +Cc: gcc-patches, rb-general, anthraxx, niels On Thu, 10 Dec 2015, Daniel Kahn Gillmor wrote: > Specifically, if the first character of the "old" argument is a > literal $, then gcc will treat it as an environment variable name, and > use the value of the env var for prefix mapping. I don't think a literal $ in option arguments is a good idea; it's far too hard to pass through a sequence of shells and makefiles that you typically get in recursive make. You end up with things like '-Wl,-rpath,'\''\\\$$\$$\\\$$\$$ORIGIN'\''/../' (part of a process for using $ORIGIN when linking GDB) if you try. -- Joseph S. Myers joseph@codesourcery.com ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-10 23:59 ` Joseph Myers @ 2015-12-11 0:13 ` Daniel Kahn Gillmor 2015-12-11 16:49 ` Daniel Kahn Gillmor 0 siblings, 1 reply; 12+ messages in thread From: Daniel Kahn Gillmor @ 2015-12-11 0:13 UTC (permalink / raw) To: Joseph Myers; +Cc: gcc-patches, rb-general, anthraxx, niels On Thu 2015-12-10 18:59:33 -0500, Joseph Myers wrote: > On Thu, 10 Dec 2015, Daniel Kahn Gillmor wrote: > >> Specifically, if the first character of the "old" argument is a >> literal $, then gcc will treat it as an environment variable name, and >> use the value of the env var for prefix mapping. > > I don't think a literal $ in option arguments is a good idea; it's far too > hard to pass through a sequence of shells and makefiles that you typically > get in recursive make. You end up with things like > '-Wl,-rpath,'\''\\\$$\$$\\\$$\$$ORIGIN'\''/../' (part of a process for > using $ORIGIN when linking GDB) if you try. yow, that's truly monstrous! Is there a different symbol or string you'd be OK using instead for the same approach? What about looking for an "ENV:" prefix? so something like: -fdebug-prefix-map=ENV:SOURCE_BUILD_DIR=/usr/src wdyt? I could rework the patch pretty easily if that seems acceptable. --dkg ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-11 0:13 ` Daniel Kahn Gillmor @ 2015-12-11 16:49 ` Daniel Kahn Gillmor 2015-12-11 17:03 ` Bernd Schmidt 0 siblings, 1 reply; 12+ messages in thread From: Daniel Kahn Gillmor @ 2015-12-11 16:49 UTC (permalink / raw) To: Joseph Myers; +Cc: gcc-patches, rb-general, anthraxx, niels [-- Attachment #1: Type: text/plain, Size: 1281 bytes --] On Thu 2015-12-10 19:12:57 -0500, Daniel Kahn Gillmor wrote: > On Thu 2015-12-10 18:59:33 -0500, Joseph Myers wrote: >> On Thu, 10 Dec 2015, Daniel Kahn Gillmor wrote: >> >>> Specifically, if the first character of the "old" argument is a >>> literal $, then gcc will treat it as an environment variable name, and >>> use the value of the env var for prefix mapping. >> >> I don't think a literal $ in option arguments is a good idea; it's far too >> hard to pass through a sequence of shells and makefiles that you typically >> get in recursive make. You end up with things like >> '-Wl,-rpath,'\''\\\$$\$$\\\$$\$$ORIGIN'\''/../' (part of a process for >> using $ORIGIN when linking GDB) if you try. > > yow, that's truly monstrous! > > Is there a different symbol or string you'd be OK using instead for the > same approach? What about looking for an "ENV:" prefix? > > so something like: > > -fdebug-prefix-map=ENV:SOURCE_BUILD_DIR=/usr/src > > wdyt? I could rework the patch pretty easily if that seems acceptable. I've re-rolled the patch (attached below, here) to use the ENV: prefix instead of the $. I've also updated the patch on https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68848 to match this approach. Thanks for the input, Joseph. Regards, --dkg [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: v2-0001-gcc-read-fdebug-prefix-map-OLD-from-environment-i.patch --] [-- Type: text/x-diff, Size: 4435 bytes --] From a1cc9f9ec219c06b06d3de0ce9d055a2e02422c3 Mon Sep 17 00:00:00 2001 From: Daniel Kahn Gillmor <dkg@fifthhorseman.net> Date: Thu, 10 Dec 2015 12:09:45 -0500 Subject: [PATCH v2] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) Work on the reproducible-builds project [0] has identified that build paths are one cause of output variation between builds. This changeset allows users to avoid this variation when building C objects with debug symbols, while leaving the default behavior unchanged. Background ---------- gcc includes the build path in any generated DWARF debugging symbols, specifically in DW_AT_comp_dir, but allows the embedded path to be changed via -fdebug-prefix-map. When -fdebug-prefix-map is used with the current build path, it removes the build path from DW_AT_comp_dir but places it instead in DW_AT_producer, so the reproducibility problem isn't resolved. When building software for binary redistribution, the actual build path on the build machine is irrelevant, and doesn't need to be exposed in the debug symbols. Resolution ---------- This patch extends the first argument to -fdebug-prefix-map ("old") to be able to read from the environment, which allows a packager to avoid embedded build paths in the debugging symbols with something like: export SOURCE_BUILD_DIR="$(pwd)" gcc -fdebug-prefix-map=ENV:SOURCE_BUILD_DIR=/usr/src Details ------- Specifically, if the "old" argument starts with a literal "ENV:", then gcc will treat it as an environment variable name, and use the value of the env var for prefix mapping. As a result, DW_AT_producer contains the literal envvar name, DW_AT_comp_dir contains the transformed build path, and the actual build path is not at all present in the generated object file. This has been tested successfully on amd64 machines, and i see no reason why it would be platform-specific. More discussion of alternate approaches considered and discarded in the development of this change can be found at [1] for those interested. Feedback welcome! [0] https://reproducible-builds.org [1] https://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20151130/004051.html --- gcc/doc/invoke.texi | 4 +++- gcc/final.c | 30 ++++++++++++++++++++++++++++-- 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index 5256031..3f76e03 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -6440,7 +6440,9 @@ link processing time. Merging is enabled by default. @item -fdebug-prefix-map=@var{old}=@var{new} @opindex fdebug-prefix-map When compiling files in directory @file{@var{old}}, record debugging -information describing them as in @file{@var{new}} instead. +information describing them as in @file{@var{new}} instead. If +@file{@var{old}} starts with a @samp{ENV:}, the corresponding environment +variable will be dereferenced, and its value will be used instead. @item -fno-dwarf2-cfi-asm @opindex fdwarf2-cfi-asm diff --git a/gcc/final.c b/gcc/final.c index 8cb5533..1800184 100644 --- a/gcc/final.c +++ b/gcc/final.c @@ -1520,11 +1520,17 @@ static debug_prefix_map *debug_prefix_maps; /* Record a debug file prefix mapping. ARG is the argument to -fdebug-prefix-map and must be of the form OLD=NEW. */ +#define ENV_PREFIX "ENV:" +#define ENV_PREFIX_OFFSET (sizeof(ENV_PREFIX) - 1) + void add_debug_prefix_map (const char *arg) { debug_prefix_map *map; const char *p; + char *env; + const char *old; + size_t oldlen; p = strchr (arg, '='); if (!p) @@ -1532,9 +1538,29 @@ add_debug_prefix_map (const char *arg) error ("invalid argument %qs to -fdebug-prefix-map", arg); return; } + if (0 == strncmp(ENV_PREFIX, arg, ENV_PREFIX_OFFSET)) + { + env = xstrndup (arg+ENV_PREFIX_OFFSET, p - (arg+ENV_PREFIX_OFFSET)); + old = getenv(env); + if (!old) + { + warning (0, "environment variable %qs not set in argument to " + "-fdebug-prefix-map", env); + free(env); + return; + } + oldlen = strlen(old); + free(env); + } + else + { + old = xstrndup (arg, p - arg); + oldlen = p - arg; + } + map = XNEW (debug_prefix_map); - map->old_prefix = xstrndup (arg, p - arg); - map->old_len = p - arg; + map->old_prefix = old; + map->old_len = oldlen; p++; map->new_prefix = xstrdup (p); map->new_len = strlen (p); -- 2.6.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-11 16:49 ` Daniel Kahn Gillmor @ 2015-12-11 17:03 ` Bernd Schmidt 2015-12-11 19:14 ` Daniel Kahn Gillmor 0 siblings, 1 reply; 12+ messages in thread From: Bernd Schmidt @ 2015-12-11 17:03 UTC (permalink / raw) To: Daniel Kahn Gillmor, Joseph Myers Cc: gcc-patches, rb-general, anthraxx, niels On 12/11/2015 05:49 PM, Daniel Kahn Gillmor wrote: > I've re-rolled the patch (attached below, here) to use the ENV: prefix > instead of the $. It might be irrelevant at this point, but the "ENV:" prefix is used in AmigaOS and could be part of a filename. > + if (0 == strncmp(ENV_PREFIX, arg, ENV_PREFIX_OFFSET)) > + { > + env = xstrndup (arg+ENV_PREFIX_OFFSET, p - (arg+ENV_PREFIX_OFFSET)); Spaces before ( and around operators like +. Please review our coding guidelines and have a look at the surrounding code. Wouldn't it be simpler just to special-case -fdebug-prefix-map in gen_producer_string? The environment variable thing strikes me as unnecessary. Bernd ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-11 17:03 ` Bernd Schmidt @ 2015-12-11 19:14 ` Daniel Kahn Gillmor 2015-12-15 12:19 ` Bernd Schmidt 0 siblings, 1 reply; 12+ messages in thread From: Daniel Kahn Gillmor @ 2015-12-11 19:14 UTC (permalink / raw) To: Bernd Schmidt, Joseph Myers; +Cc: gcc-patches, rb-general, anthraxx, niels [-- Attachment #1: Type: text/plain, Size: 1603 bytes --] On Fri 2015-12-11 12:03:28 -0500, Bernd Schmidt wrote: > On 12/11/2015 05:49 PM, Daniel Kahn Gillmor wrote: >> I've re-rolled the patch (attached below, here) to use the ENV: prefix >> instead of the $. > > It might be irrelevant at this point, but the "ENV:" prefix is used in > AmigaOS and could be part of a filename. As a full-path prefix? >> + if (0 == strncmp(ENV_PREFIX, arg, ENV_PREFIX_OFFSET)) >> + { >> + env = xstrndup (arg+ENV_PREFIX_OFFSET, p - (arg+ENV_PREFIX_OFFSET)); > > Spaces before ( and around operators like +. Please review our coding > guidelines and have a look at the surrounding code. right, sorry about that. Attached at the bottom of this mail is a patch that i think should clean this up. (i've also updated bugzilla correctly after a couple tries -- sorry for the noise there) > Wouldn't it be simpler just to special-case -fdebug-prefix-map in > gen_producer_string? The environment variable thing strikes me as > unnecessary. I think you mean so that we would just ignore -fdebug-prefix-map entirely when writing DW_AT_producer, so that you could build reproducibly with (for example): gcc -fdebug-prefix-map=$(pwd)=/usr/src We'd considered and discarded this approach in the past out of concern for possible build systems that can easily vary the environment, but work with a static list of CFLAGS to pass to the compiler. On further inspection, it's not clear that anyone has a concrete example of a build system with this constraint. Here's a one-liner patch for this approach (also at https://gcc.gnu.org/bugzilla/attachment.cgi?id=37007): [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-gcc-ignore-fdebug-prefix-map-in-DW_AT_producer-impro.patch --] [-- Type: text/x-diff, Size: 1835 bytes --] From 13ca37337660f5385295052b3c1aebfc4e29092c Mon Sep 17 00:00:00 2001 From: Daniel Kahn Gillmor <dkg@fifthhorseman.net> Date: Fri, 11 Dec 2015 13:47:45 -0500 Subject: [PATCH] gcc: ignore -fdebug-prefix-map in DW_AT_producer (improved reproducibility) Work on the reproducible-builds project [0] has identified that build paths are one cause of output variation between builds. This changeset allows users to avoid this variation when building C objects with debug symbols, while leaving the default behavior unchanged. Background ---------- gcc includes the build path in any generated DWARF debugging symbols, specifically in DW_AT_comp_dir, but allows the embedded path to be changed via -fdebug-prefix-map. When -fdebug-prefix-map is used with the current build path, it removes the build path from DW_AT_comp_dir but places it instead in DW_AT_producer, so the reproducibility problem isn't resolved. When building software for binary redistribution, the actual build path on the build machine is irrelevant, and doesn't need to be exposed in the debug symbols. Resolution ---------- This patch resolves the problem by never including -fdebug-prefix-map in the DW_AT_producer attribute. People interested in building software reproducibly irrespective of build path can do so by passing the build path itself as the first argument to -fdebug-prefix-map. [0] https://reproducible-builds.org --- gcc/dwarf2out.c | 1 + 1 file changed, 1 insertion(+) diff --git a/gcc/dwarf2out.c b/gcc/dwarf2out.c index 6af57b5..1a75ed7 100644 --- a/gcc/dwarf2out.c +++ b/gcc/dwarf2out.c @@ -20151,6 +20151,7 @@ gen_producer_string (void) case OPT_fpreprocessed: case OPT_fltrans_output_list_: case OPT_fresolution_: + case OPT_fdebug_prefix_map_: /* Ignore these. */ continue; default: -- 2.6.2 [-- Attachment #3: Type: text/plain, Size: 148 bytes --] And here's the revised ENV:-prefixed approach with corrected coding conventions (also at https://gcc.gnu.org/bugzilla/attachment.cgi?id=37005): [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #4: v3-0001-gcc-read-fdebug-prefix-map-OLD-from-environment-i.patch --] [-- Type: text/x-diff, Size: 4441 bytes --] From 41d3bcf94ecd9b1ce2c4bccc0039bdff0b464315 Mon Sep 17 00:00:00 2001 From: Daniel Kahn Gillmor <dkg@fifthhorseman.net> Date: Thu, 10 Dec 2015 12:09:45 -0500 Subject: [PATCH v3] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) Work on the reproducible-builds project [0] has identified that build paths are one cause of output variation between builds. This changeset allows users to avoid this variation when building C objects with debug symbols, while leaving the default behavior unchanged. Background ---------- gcc includes the build path in any generated DWARF debugging symbols, specifically in DW_AT_comp_dir, but allows the embedded path to be changed via -fdebug-prefix-map. When -fdebug-prefix-map is used with the current build path, it removes the build path from DW_AT_comp_dir but places it instead in DW_AT_producer, so the reproducibility problem isn't resolved. When building software for binary redistribution, the actual build path on the build machine is irrelevant, and doesn't need to be exposed in the debug symbols. Resolution ---------- This patch extends the first argument to -fdebug-prefix-map ("old") to be able to read from the environment, which allows a packager to avoid embedded build paths in the debugging symbols with something like: export SOURCE_BUILD_DIR="$(pwd)" gcc -fdebug-prefix-map=ENV:SOURCE_BUILD_DIR=/usr/src Details ------- Specifically, if the "old" argument starts with a literal "ENV:", then gcc will treat it as an environment variable name, and use the value of the env var for prefix mapping. As a result, DW_AT_producer contains the literal envvar name, DW_AT_comp_dir contains the transformed build path, and the actual build path is not at all present in the generated object file. This has been tested successfully on amd64 machines, and i see no reason why it would be platform-specific. More discussion of alternate approaches considered and discarded in the development of this change can be found at [1] for those interested. Feedback welcome! [0] https://reproducible-builds.org [1] https://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20151130/004051.html --- gcc/doc/invoke.texi | 4 +++- gcc/final.c | 30 ++++++++++++++++++++++++++++-- 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index 5256031..3f76e03 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -6440,7 +6440,9 @@ link processing time. Merging is enabled by default. @item -fdebug-prefix-map=@var{old}=@var{new} @opindex fdebug-prefix-map When compiling files in directory @file{@var{old}}, record debugging -information describing them as in @file{@var{new}} instead. +information describing them as in @file{@var{new}} instead. If +@file{@var{old}} starts with a @samp{ENV:}, the corresponding environment +variable will be dereferenced, and its value will be used instead. @item -fno-dwarf2-cfi-asm @opindex fdwarf2-cfi-asm diff --git a/gcc/final.c b/gcc/final.c index 8cb5533..670941d 100644 --- a/gcc/final.c +++ b/gcc/final.c @@ -1520,11 +1520,17 @@ static debug_prefix_map *debug_prefix_maps; /* Record a debug file prefix mapping. ARG is the argument to -fdebug-prefix-map and must be of the form OLD=NEW. */ +#define ENV_PREFIX "ENV:" +#define ENV_PREFIX_OFFSET (sizeof (ENV_PREFIX) - 1) + void add_debug_prefix_map (const char *arg) { debug_prefix_map *map; const char *p; + char *env; + const char *old; + size_t oldlen; p = strchr (arg, '='); if (!p) @@ -1532,9 +1538,29 @@ add_debug_prefix_map (const char *arg) error ("invalid argument %qs to -fdebug-prefix-map", arg); return; } + if (0 == strncmp (ENV_PREFIX, arg, ENV_PREFIX_OFFSET)) + { + env = xstrndup (arg + ENV_PREFIX_OFFSET, p - (arg + ENV_PREFIX_OFFSET)); + old = getenv(env); + if (!old) + { + warning (0, "environment variable %qs not set in argument to " + "-fdebug-prefix-map", env); + free(env); + return; + } + oldlen = strlen(old); + free(env); + } + else + { + old = xstrndup (arg, p - arg); + oldlen = p - arg; + } + map = XNEW (debug_prefix_map); - map->old_prefix = xstrndup (arg, p - arg); - map->old_len = p - arg; + map->old_prefix = old; + map->old_len = oldlen; p++; map->new_prefix = xstrdup (p); map->new_len = strlen (p); -- 2.6.2 [-- Attachment #5: Type: text/plain, Size: 109 bytes --] I've tested both patches and they work for me, independently. Thanks for your consideration, --dkg ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-11 19:14 ` Daniel Kahn Gillmor @ 2015-12-15 12:19 ` Bernd Schmidt 2015-12-15 14:15 ` Daniel Kahn Gillmor 0 siblings, 1 reply; 12+ messages in thread From: Bernd Schmidt @ 2015-12-15 12:19 UTC (permalink / raw) To: Daniel Kahn Gillmor, Joseph Myers Cc: gcc-patches, rb-general, anthraxx, niels On 12/11/2015 08:14 PM, Daniel Kahn Gillmor wrote: > I think you mean so that we would just ignore -fdebug-prefix-map > entirely when writing DW_AT_producer, so that you could build > reproducibly with (for example): > > gcc -fdebug-prefix-map=$(pwd)=/usr/src > > We'd considered and discarded this approach in the past out of concern > for possible build systems that can easily vary the environment, but > work with a static list of CFLAGS to pass to the compiler. On further > inspection, it's not clear that anyone has a concrete example of a build > system with this constraint. > > Here's a one-liner patch for this approach (also at > https://gcc.gnu.org/bugzilla/attachment.cgi?id=37007): I think that one-liner is fine, even for now. Bernd ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-15 12:19 ` Bernd Schmidt @ 2015-12-15 14:15 ` Daniel Kahn Gillmor 2015-12-15 16:08 ` Bernd Schmidt 0 siblings, 1 reply; 12+ messages in thread From: Daniel Kahn Gillmor @ 2015-12-15 14:15 UTC (permalink / raw) To: Bernd Schmidt, Joseph Myers; +Cc: gcc-patches, rb-general, anthraxx, niels On Tue 2015-12-15 07:19:30 -0500, Bernd Schmidt wrote: > On 12/11/2015 08:14 PM, Daniel Kahn Gillmor wrote: >> Here's a one-liner patch for this approach (also at >> https://gcc.gnu.org/bugzilla/attachment.cgi?id=37007): > > I think that one-liner is fine, even for now. great! what would be the next steps for getting this applied upstream? Thanks for your review, --dkg ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-15 14:15 ` Daniel Kahn Gillmor @ 2015-12-15 16:08 ` Bernd Schmidt 2015-12-15 16:32 ` Daniel Kahn Gillmor 0 siblings, 1 reply; 12+ messages in thread From: Bernd Schmidt @ 2015-12-15 16:08 UTC (permalink / raw) To: Daniel Kahn Gillmor, Joseph Myers Cc: gcc-patches, rb-general, anthraxx, niels On 12/15/2015 03:14 PM, Daniel Kahn Gillmor wrote: > On Tue 2015-12-15 07:19:30 -0500, Bernd Schmidt wrote: >> On 12/11/2015 08:14 PM, Daniel Kahn Gillmor wrote: >>> Here's a one-liner patch for this approach (also at >>> https://gcc.gnu.org/bugzilla/attachment.cgi?id=37007): >> >> I think that one-liner is fine, even for now. > > great! what would be the next steps for getting this applied upstream? I'm guessing you don't have an account so I'll bootstrap and test it and then commit. (with an extra testcase, as below - adapted from another testcase in the debug/dwarf2 directory). Bernd * gcc.dg/debug/dwarf2/prod-options.c: New file. /* Verify that the DW_AT_producer does not contain certain compiler options such as -fdebug-prefix-map=; this is undesirable since path names make the build not reproducible. Other skipped options could be tested here as well. */ /* { dg-do compile } */ /* { dg-options "-O2 -gdwarf -dA -fdebug-prefix-map=a=b" } */ /* { dg-final { scan-assembler "DW_AT_producer: \"GNU C" } } */ /* { dg-final { scan-assembler-not "debug-prefix-map" } } */ void func (void) { } ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-15 16:08 ` Bernd Schmidt @ 2015-12-15 16:32 ` Daniel Kahn Gillmor 0 siblings, 0 replies; 12+ messages in thread From: Daniel Kahn Gillmor @ 2015-12-15 16:32 UTC (permalink / raw) To: Bernd Schmidt, Joseph Myers; +Cc: gcc-patches, rb-general, anthraxx, niels On Tue 2015-12-15 11:08:23 -0500, Bernd Schmidt wrote: > I'm guessing you don't have an account I don't have an account (though i'd be happy to set one up if you point me toward the process). > so I'll bootstrap and test it and then commit. fwiw, I've tested it myself, and can confirm that it does work. > (with an extra testcase, as below - adapted from another testcase in > the debug/dwarf2 directory). Thanks also for this example, it points me in the right direction for future work. Regards, --dkg ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [rb-general] [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) 2015-12-10 17:36 ` [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) Daniel Kahn Gillmor 2015-12-10 22:05 ` Daniel Kahn Gillmor 2015-12-10 23:59 ` Joseph Myers @ 2015-12-22 12:16 ` Thomas Klausner 2 siblings, 0 replies; 12+ messages in thread From: Thomas Klausner @ 2015-12-22 12:16 UTC (permalink / raw) To: General discussions about reproducible builds Cc: gcc-patches, niels, Daniel Kahn Gillmor, anthraxx, Joerg Sonnenberger I just found a related patch again that we have in NetBSD's gcc that allows fixing __FILE__ references. See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47047 No progress since 2010 in getting this included though :( Thomas On Thu, Dec 10, 2015 at 12:36:18PM -0500, Daniel Kahn Gillmor wrote: > Work on the reproducible-builds project [0] has identified that build > paths are one cause of output variation between builds. This > changeset allows users to avoid this variation when building C objects > with debug symbols, while leaving the default behavior unchanged. > > Background > ---------- > > gcc includes the build path in any generated DWARF debugging symbols, > specifically in DW_AT_comp_dir, but allows the embedded path to be > changed via -fdebug-prefix-map. > > When -fdebug-prefix-map is used with the current build path, it > removes the build path from DW_AT_comp_dir but places it instead in > DW_AT_producer, so the reproducibility problem isn't resolved. > > When building software for binary redistribution, the actual build > path on the build machine is irrelevant, and doesn't need to be > exposed in the debug symbols. > > Resolution > ---------- > > This patch extends the first argument to -fdebug-prefix-map ("old") to > be able to read from the environment, which allows a packager to avoid > embedded build paths in the debugging symbols with something like: > > export SOURCE_BUILD_DIR="$(pwd)" > gcc -fdebug-prefix-map=\$SOURCE_BUILD_DIR=/usr/src > > Details > ------- > > Specifically, if the first character of the "old" argument is a > literal $, then gcc will treat it as an environment variable name, and > use the value of the env var for prefix mapping. > > As a result, DW_AT_producer contains the literal envvar name, > DW_AT_comp_dir contains the transformed build path, and the actual > build path is not at all present in the generated object file. > > This has been tested successfully on amd64 machines, and i see no > reason why it would be platform-specific. > > More discussion of alternate approaches considered and discarded in > the development of this change can be found at [1] for those > interested. > > Feedback welcome! > > [0] https://reproducible-builds.org > [1] https://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20151130/004051.html > --- > gcc/doc/invoke.texi | 4 +++- > gcc/final.c | 27 +++++++++++++++++++++++++-- > 2 files changed, 28 insertions(+), 3 deletions(-) > > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi > index 5256031..234432f 100644 > --- a/gcc/doc/invoke.texi > +++ b/gcc/doc/invoke.texi > @@ -6440,7 +6440,9 @@ link processing time. Merging is enabled by default. > @item -fdebug-prefix-map=@var{old}=@var{new} > @opindex fdebug-prefix-map > When compiling files in directory @file{@var{old}}, record debugging > -information describing them as in @file{@var{new}} instead. > +information describing them as in @file{@var{new}} instead. If > +@file{@var{old}} starts with a @samp{$}, the corresponding environment > +variable will be dereferenced, and its value will be used instead. > > @item -fno-dwarf2-cfi-asm > @opindex fdwarf2-cfi-asm > diff --git a/gcc/final.c b/gcc/final.c > index 8cb5533..bc43b61 100644 > --- a/gcc/final.c > +++ b/gcc/final.c > @@ -1525,6 +1525,9 @@ add_debug_prefix_map (const char *arg) > { > debug_prefix_map *map; > const char *p; > + char *env; > + const char *old; > + size_t oldlen; > > p = strchr (arg, '='); > if (!p) > @@ -1532,9 +1535,29 @@ add_debug_prefix_map (const char *arg) > error ("invalid argument %qs to -fdebug-prefix-map", arg); > return; > } > + if (*arg == '$') > + { > + env = xstrndup (arg+1, p - (arg+1)); > + old = getenv(env); > + if (!old) > + { > + warning (0, "environment variable %qs not set in argument to " > + "-fdebug-prefix-map", env); > + free(env); > + return; > + } > + oldlen = strlen(old); > + free(env); > + } > + else > + { > + old = xstrndup (arg, p - arg); > + oldlen = p - arg; > + } > + > map = XNEW (debug_prefix_map); > - map->old_prefix = xstrndup (arg, p - arg); > - map->old_len = p - arg; > + map->old_prefix = old; > + map->old_len = oldlen; > p++; > map->new_prefix = xstrdup (p); > map->new_len = strlen (p); > -- > 2.6.2 > > _______________________________________________ > rb-general@lists.reproducible-builds.org mailing list > > To unsubscribe or change your options, please visit: > https://lists.reproducible-builds.org/listinfo/rb-general > ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2015-12-22 12:16 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <87egf1mxfl.fsf@alice.fifthhorseman.net> 2015-12-10 17:36 ` [PATCH] gcc: read -fdebug-prefix-map OLD from environment (improved reproducibility) Daniel Kahn Gillmor 2015-12-10 22:05 ` Daniel Kahn Gillmor 2015-12-10 23:59 ` Joseph Myers 2015-12-11 0:13 ` Daniel Kahn Gillmor 2015-12-11 16:49 ` Daniel Kahn Gillmor 2015-12-11 17:03 ` Bernd Schmidt 2015-12-11 19:14 ` Daniel Kahn Gillmor 2015-12-15 12:19 ` Bernd Schmidt 2015-12-15 14:15 ` Daniel Kahn Gillmor 2015-12-15 16:08 ` Bernd Schmidt 2015-12-15 16:32 ` Daniel Kahn Gillmor 2015-12-22 12:16 ` [rb-general] " Thomas Klausner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).