Re: [PING^4][PATCH v2] Generate reproducible output independently of the build-path

[Yury Gribov <tetra2005.patches@gmail.com>]

On Thu, Aug 3, 2017 at 12:45 PM, Ximin Luo <infinity0@pwned.gg> wrote:
> Yury Gribov:
>> On 03.08.2017 3:06, Ximin Luo wrote:
>>> Jeff Law:
>>>> On 07/21/2017 10:15 AM, Ximin Luo wrote:
>>>>> (Please keep me on CC, I am not subscribed)
>>>>>
>>>>>
>>>>> Proposal
>>>>> ========
>>>>>
>>>>> This patch series adds a new environment variable BUILD_PATH_PREFIX_MAP. When
>>>>> this is set, GCC will treat this as extra implicit "-fdebug-prefix-map=$value"
>>>>> command-line arguments that precede any explicit ones. This makes the final
>>>>> binary output reproducible, and also hides the unreproducible value (the source
>>>>> path prefixes) from CFLAGS et. al. which many build tools (understandably)
>>>>> embed as-is into their build output.
>>>> I'd *really* avoid doing this with magic environment variables.  Make it
>>>> a first class option to the compiler.  Yes, it means projects that want
>>>> this behavior have to arrange to pass that flag to their compiler, but
>>>> IMHO that's much preferred over environment variables.
>>>>
>>>> Jeff
>>>>
>>>
>>> Hi Jeff,
>>>
>>> If by "first class option" you meant a command-line flag, GCC *already has* that (-fdebug-prefix-map) > and it wasn't enough to achieve reproducibility in many cases we tested.
>>
>> Shouldn't -fdebug-prefix-map be updated to use the same syntax as BUILD_PATH_PREFIX_MAP?
>>
>
> -fdebug-prefix-map is a CLI option and can be given multiple times, each flag given is in the form of $from=$to where $from can't contain a '='.
>
> BUILD_PATH_PREFIX_MAP is a single envvar that encodes a list-of-pairs of the form $to=$from:$to=$from with some escaping for flexibility and to support things like windows paths. Since it's a new envvar, Ian Jackson suggested $to=$from to emphasise the reproducible ($to) part. I liked the idea so I implemented it like that. (We did a lot of bikeshedding over on the rb-general mailing list about the exact format and this is what we settled on, I'd like to avoid getting into that again but would nevertheless do it, if it's necessary to get this patch accepted.)
>
> Because -fdebug-prefix-map currently only encodes one $from=$to pair, it would be a very disruptive and highly backward-incompatible change to make it use the same syntax as B_P_P_M. A slightly less disruptive but still backward-incompatible change would be to make it encode a single $to=$from pair, but I don't really see the advantage to doing so - what were your thoughts on this?

I believe it would much easier to reason about environment variable
behavior when it boils down to "prepend some standard flag to
command-line flags".  It would also simplify maintenance of local
compiler patch as core functionality can be merged to mainline GCC
whereas debatable environment variable part stays in the distro.

> If by "first class option" you meant a command-line flag, GCC *already has* that (-fdebug-prefix-map) and it wasn't enough to achieve reproducibility in many cases we tested.
> dpkg-buildflags actually already adds these flags to CFLAGS CXXFLAGS etc on Debian. However, with this patch using the environment variable, we are able to reproduce 1800 more packages out of 26000.

Just curious, why -fdebug-prefix-map (maybe modified to support
multiple renames) was not enough for these packages (and why they
can't be fixed instead)?

-Y