Deterministic builds

Deterministic builds

[Ken Brown]

Is it possible to build an executable on Cygwin so that subsequent 
builds (with no change in source) produce identical results?  Currently, 
the timestamp embedded in executables prevents this.  (I don't know if 
that's the only obstacle.)

For example:

$ cat hello.c
#include <stdio.h>
int
main ()
{
  printf("Hello, world!\n");
  return 0;
}

$ gcc hello.c -o hello1

$ gcc hello.c -o hello2

$ objdump -p hello1.exe | grep Time/Date
Time/Date               Wed May  4 09:20:24 2016

$ objdump -p hello2.exe | grep Time/Date
Time/Date               Wed May  4 09:20:29 2016

My actual use case is that I'm building a package that produces a large 
number of executables.  If I make a change in one source file, I'd like 
to be able to know which executables change.

Ken


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Deterministic builds

[Greg Chicares]

On 2016-05-04 13:38, Ken Brown wrote:
> Is it possible to build an executable on Cygwin so that subsequent 
> builds (with no change in source) produce identical results?  Currently, 
> the timestamp embedded in executables prevents this.  (I don't know if 
> that's the only obstacle.)

'-fno-guess-branch-probability' removes one potential obstacle:
  https://lists.debian.org/debian-devel/2005/02/msg01154.html
I don't know if there are other obstacles than these.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Deterministic builds

[Achim Gratz]

Ken Brown writes:
> Is it possible to build an executable on Cygwin so that subsequent
> builds (with no change in source) produce identical results?
> Currently, the timestamp embedded in executables prevents this.  (I
> don't know if that's the only obstacle.)

I think the basic problems and solutions could be sussed from the work
of Debian folks on reproducible builds:

https://wiki.debian.org/ReproducibleBuilds

> My actual use case is that I'm building a package that produces a
> large number of executables.  If I make a change in one source file,
> I'd like to be able to know which executables change.

Just looking at executables, you will probably find that in gcc there
are a few optimizations that do not produce the same bits on each
compile (depending on available memory, CPU speed or some random seed).
For some of these there are switches to stabilize the result at least.
Timestamps in various file formats are another source of chatter,
although you might be able to either fix thetime or just ignore those
timestamps during comparison (but that's generally more costly than just
doing a plain comparison).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Deterministic builds

[Ismail Donmez]

Hi,

On Wed, May 4, 2016 at 4:38 PM, Ken Brown <kbrown@cornell.edu> wrote:
> Is it possible to build an executable on Cygwin so that subsequent builds
> (with no change in source) produce identical results?  Currently, the
> timestamp embedded in executables prevents this.  (I don't know if that's
> the only obstacle.)
>
> For example:
>
> $ cat hello.c
> #include <stdio.h>
> int
> main ()
> {
>  printf("Hello, world!\n");
>  return 0;
> }
>
> $ gcc hello.c -o hello1
>
> $ gcc hello.c -o hello2
>
> $ objdump -p hello1.exe | grep Time/Date
> Time/Date               Wed May  4 09:20:24 2016
>
> $ objdump -p hello2.exe | grep Time/Date
> Time/Date               Wed May  4 09:20:29 2016

You can easily disable this feature:

latte ~ > gcc -Wl,--no-insert-timestamp hello.c
latte ~ > objdump -p a.exe | grep Time/Date
Time/Date               Thu Jan  1 03:31:53 1970
latte ~ > gcc -Wl,--no-insert-timestamp hello.c
latte ~ > objdump -p a.exe | grep Time/Date
Time/Date               Thu Jan  1 03:31:53 1970

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Deterministic builds

[Ken Brown]

On 5/4/2016 1:21 PM, Ismail Donmez wrote:
> You can easily disable this feature:
>
> latte ~ > gcc -Wl,--no-insert-timestamp hello.c
> latte ~ > objdump -p a.exe | grep Time/Date
> Time/Date               Thu Jan  1 03:31:53 1970
> latte ~ > gcc -Wl,--no-insert-timestamp hello.c
> latte ~ > objdump -p a.exe | grep Time/Date
> Time/Date               Thu Jan  1 03:31:53 1970

Thank you!  That's exactly what I was looking for.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Deterministic builds

[Ken Brown]

On 5/4/2016 1:39 PM, Ken Brown wrote:
> On 5/4/2016 1:21 PM, Ismail Donmez wrote:
>> You can easily disable this feature:
>>
>> latte ~ > gcc -Wl,--no-insert-timestamp hello.c
>> latte ~ > objdump -p a.exe | grep Time/Date
>> Time/Date               Thu Jan  1 03:31:53 1970
>> latte ~ > gcc -Wl,--no-insert-timestamp hello.c
>> latte ~ > objdump -p a.exe | grep Time/Date
>> Time/Date               Thu Jan  1 03:31:53 1970
>
> Thank you!  That's exactly what I was looking for.

Just for the record, in case anyone else finds this useful, Ismail's 
suggestion did indeed produce deterministic builds in my setup.  I built 
a large project with about 150 executables, changed a few source files, 
removed the build directory, rebuilt, and found that only the (expected) 
few executables changed.

Ken


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Deterministic builds

[Warren Young]

On May 5, 2016, at 11:59 AM, Ken Brown <kbrown@cornell.edu> wrote:
> 
> Ismail's suggestion did indeed produce deterministic builds in my setup.  I built a large project with about 150 executables, changed a few source files, removed the build directory, rebuilt, and found that only the (expected) few executables changed.

…and does it do the same on a very different system?  e.g. Try it on both 64-bit Windows 10 and on 32-bit Windows 7.

Perhaps you don’t need it, but part of the reason for the big push recently for reproducible builds is to be able to verify that binaries from a given source (e.g. Red Hat’s RPM feed) are in fact buildable from the sources distributed from the same source (e.g. Red Hat’s SRPMs).

The usual motivation for that is security: it’s no good receiving an SRPM with a security patch if the binary that yum installs still has the bug.

Therefore, if you get “reproducible” builds on only a single machine, you may not have achieved any useful result.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Deterministic builds

[Ken Brown]

On 5/5/2016 4:26 PM, Warren Young wrote:
> On May 5, 2016, at 11:59 AM, Ken Brown <kbrown@cornell.edu> wrote:
>>
>> Ismail's suggestion did indeed produce deterministic builds in my setup.  I built a large project with about 150 executables, changed a few source files, removed the build directory, rebuilt, and found that only the (expected) few executables changed.
>
> …and does it do the same on a very different system?  e.g. Try it on both 64-bit Windows 10 and on 32-bit Windows 7.
>
> Perhaps you don’t need it, but part of the reason for the big push recently for reproducible builds is to be able to verify that binaries from a given source (e.g. Red Hat’s RPM feed) are in fact buildable from the sources distributed from the same source (e.g. Red Hat’s SRPMs).

Yes, that's a much more ambitious goal, and it's not what I was trying 
to do.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Deterministic builds

[Andrey Repin]

Greetings, Ken Brown!

> Is it possible to build an executable on Cygwin so that subsequent 
> builds (with no change in source) produce identical results?

General answer is "no". It is possible to build a consistent object binary,
but executable linked from it will be different on different systems, for
various reasons.

> Currently,
> the timestamp embedded in executables prevents this.  (I don't know if 
> that's the only obstacle.)

Timestamps are the least of your issues.
Think of things like stack allocation bases.

> My actual use case is that I'm building a package that produces a large
> number of executables.  If I make a change in one source file, I'd like 
> to be able to know which executables change.

If your interest is purely self-educational, you can limit the noise on a
particular given system. But I wouldn't try to make any universal claims if I
were you.


-- 
With best regards,
Andrey Repin
Thursday, May 5, 2016 04:32:14

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple