Reproducible binary builds [was: complibs/zlib: Add zlib support for binutils and gdb]

Reproducible binary builds [was: complibs/zlib: Add zlib support for binutils and gdb]

[Michael Hope]

Hi there.  I thought I'd pop this out of the zlib discussion as the
thread was getting a bit deep.  Take it back to the top level
requirements and such.

We're looking at making and distributing a binary build of the Linaro
toolchain for ARM.

We want:
 * A generic Linux build that runs on most distros from RHEL 5 onwards
 * A Windows build that runs on Windows XP and up
 * Support for 32 bit and 64 bit hosts

and various qualities that go with that:
 * Reproducible builds
 * A sharable build system so others can reproduce and customise the build
 * A minimum number of binaries, meaning that the binary should run on
32 and 64 bit hosts
 * To minimise testing by making the Windows and Linux builds as
similar as possible

To do this it makes sense to:
 * Use the same build scripts (i.e. crosstool-NG) for Windows and Linux
 * Use the same library and application versions
 * Build everything from scratch

We could pick something like Ubuntu 8.04 LTS and build on that to lock
down the library versions, but that makes it hard to have the same
version on Windows and harder for others to reproduce the build.

We could use the prebuilt mingw/msys versions of libraries (such as
http://sourceforge.net/projects/mingw/files/MinGW/Base/libiconv/libiconv-1.14-2/)
but then the Windows and Linux versions would be different.

There's a few options for running on a range of Linuxs':
 1. Statically link everything (lose any security/bug fixes)
 2. Statically link everything but libc and libm
 3. Use LSB
 4. Ship dynamic libraries and pick them up using RPATH

A 32 bit version of (1) ill work out-of-the-box on 32 bit and 64 bit
hosts.  None of the rest will as 64 bit hosts don't include the 32 bit
libraries by default, but they're pretty close and just a 'apt-get
install lsb' or 'yum install redhat-lsb.i386' away.

The biggest thing is to use the same versions on Windows and Linux
which really means building from scratch.  Re-using the crosstool-NG
infrastructure makes more sense then doing it outside.  I'm happy to
keep these patches in our own tree so that it doesn't pollute tip and
increase the number of packages you need to support.

-- Michael

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Reproducible binary builds [was: complibs/zlib: Add zlib support for binutils and gdb]

[Yann E. MORIN]

Michael, Zhenqiang, All,

On Thursday 24 November 2011 23:26:41 Michael Hope wrote:
> Hi there.  I thought I'd pop this out of the zlib discussion as the
> thread was getting a bit deep.  Take it back to the top level
> requirements and such.
> 
> We're looking at making and distributing a binary build of the Linaro
> toolchain for ARM.
> 
> We want:
>  * A generic Linux build that runs on most distros from RHEL 5 onwards
>  * A Windows build that runs on Windows XP and up
>  * Support for 32 bit and 64 bit hosts
> 
> and various qualities that go with that:
>  * Reproducible builds
>  * A sharable build system so others can reproduce and customise the build
>  * A minimum number of binaries, meaning that the binary should run on
> 32 and 64 bit hosts
>  * To minimise testing by making the Windows and Linux builds as
> similar as possible

OK, this is nice to see the overall goal, explained in precise terms.
Thanks!

> To do this it makes sense to:
>  * Use the same build scripts (i.e. crosstool-NG) for Windows and Linux
>  * Use the same library and application versions
>  * Build everything from scratch
> 
> We could pick something like Ubuntu 8.04 LTS and build on that to lock
> down the library versions, but that makes it hard to have the same
> version on Windows and harder for others to reproduce the build.
> 
> We could use the prebuilt mingw/msys versions of libraries (such as
> http://sourceforge.net/projects/mingw/files/MinGW/Base/libiconv/libiconv-1.14-2/)
> but then the Windows and Linux versions would be different.
> 
> There's a few options for running on a range of Linuxs':
>  1. Statically link everything (lose any security/bug fixes)
>  2. Statically link everything but libc and libm
>  3. Use LSB
>  4. Ship dynamic libraries and pick them up using RPATH
> 
> A 32 bit version of (1) ill work out-of-the-box on 32 bit and 64 bit
> hosts.  None of the rest will as 64 bit hosts don't include the 32 bit
> libraries by default, but they're pretty close and just a 'apt-get
> install lsb' or 'yum install redhat-lsb.i386' away.
> 
> The biggest thing is to use the same versions on Windows and Linux
> which really means building from scratch.  Re-using the crosstool-NG
> infrastructure makes more sense then doing it outside.  I'm happy to
> keep these patches in our own tree so that it doesn't pollute tip and
> increase the number of packages you need to support.

OK, that all makes a lot of sense! So, here is the list of the required
dependencies (that I know of) for each component:

   library   | binutils | gcc | gdb |
   ----------+----------+-----+-----+
   expat     |          |     |  Y  |
   ncurses   |          |     |  Y  |
-> zlib      |    Y     |  Y  |  Y  |
-> libiconv  |    Y     |  Y  |  Y  |
   gmp       |          |  Y  |     |
   mpfr      |          |  Y  |     |
   ppl       |          |  Y  |     |
   cloog/ppl |          |  Y  |     |
   mpc       |          |  Y  |     |
   libelf    |          |  Y  |     |

I believe this is a complete listing. If not, then please add to it.
I think that the discussion so far was enough to warrant inclusion of
those libs in the build process. The two that are missing are zlib
and libiconv. I think we can live with adding only these two libs.

I'm pretty busy these days, so I won't be able to really work on this
until Tuesday, though.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Reproducible binary builds [was: complibs/zlib: Add zlib support for binutils and gdb]

[Zhenqiang Chen]

> OK, that all makes a lot of sense! So, here is the list of the required
> dependencies (that I know of) for each component:
>
>   library   | binutils | gcc | gdb |
>   ----------+----------+-----+-----+
>   expat     |          |     |  Y  |
>   ncurses   |          |     |  Y  |
> -> zlib      |    Y     |  Y  |  Y  |
> -> libiconv  |    Y     |  Y  |  Y  |
>   gmp       |          |  Y  |     |
>   mpfr      |          |  Y  |     |
>   ppl       |          |  Y  |     |
>   cloog/ppl |          |  Y  |     |
>   mpc       |          |  Y  |     |
>   libelf    |          |  Y  |     |
>

Both gdb native and cross need expat, ncurses, zlib and libiconv.

Currently gdb will build fail due to depend on expat for canadian build.

Thanks!
-Zhenqiang

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Reproducible binary builds [was: complibs/zlib: Add zlib support for binutils and gdb]

[Michael Hope]

On Mon, Nov 28, 2011 at 2:50 PM, Zhenqiang Chen
<zhenqiang.chen@linaro.org> wrote:
>> OK, that all makes a lot of sense! So, here is the list of the required
>> dependencies (that I know of) for each component:
>>
>>   library   | binutils | gcc | gdb |
>>   ----------+----------+-----+-----+
>>   expat     |          |     |  Y  |
>>   ncurses   |          |     |  Y  |
>> -> zlib      |    Y     |  Y  |  Y  |
>> -> libiconv  |    Y     |  Y  |  Y  |
>>   gmp       |          |  Y  |     |
>>   mpfr      |          |  Y  |     |
>>   ppl       |          |  Y  |     |
>>   cloog/ppl |          |  Y  |     |
>>   mpc       |          |  Y  |     |
>>   libelf    |          |  Y  |     |
>>
>
> Both gdb native and cross need expat, ncurses, zlib and libiconv.
>
> Currently gdb will build fail due to depend on expat for canadian build.

The GDB build script scripts/build/debug/300-gdb.sh has support for
building expat and ncurses as part of the GDB build itself.  It's
currently only turned on for native though - perhaps it should be
turned on for Canadian cross as well?

I'm not fond of having it hidden like that but that's how things are.

-- Michael

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: Reproducible binary builds [was: complibs/zlib: Add zlib support for binutils and gdb]

[Yann E. MORIN]

Michael, Zhenqiang, All,

On Monday 28 November 2011 03:05:18 Michael Hope wrote:
> On Mon, Nov 28, 2011 at 2:50 PM, Zhenqiang Chen
> <zhenqiang.chen@linaro.org> wrote:
> >> OK, that all makes a lot of sense! So, here is the list of the required
> >> dependencies (that I know of) for each component:
> >>
> >>   library   | binutils | gcc | gdb |
> >>   ----------+----------+-----+-----+
> >>   expat     |          |     |  Y  |
> >>   ncurses   |          |     |  Y  |
> >> -> zlib      |    Y     |  Y  |  Y  |
> >> -> libiconv  |    Y     |  Y  |  Y  |
> >>   gmp       |          |  Y  |     |
> >>   mpfr      |          |  Y  |     |
> >>   ppl       |          |  Y  |     |
> >>   cloog/ppl |          |  Y  |     |
> >>   mpc       |          |  Y  |     |
> >>   libelf    |          |  Y  |     |
> >>
> >
> > Both gdb native and cross need expat, ncurses, zlib and libiconv.

Yes, the table above does not differentiate the type of build; it only
lists the dependencies.

> > Currently gdb will build fail due to depend on expat for canadian build.

That happens only for the cross gdb. The native gdb already builds expat
and ncurses (but not zlib).

> The GDB build script scripts/build/debug/300-gdb.sh has support for
> building expat and ncurses as part of the GDB build itself.  It's
> currently only turned on for native though - perhaps it should be
> turned on for Canadian cross as well?

If we build zlib, expat and ncurses for the host, it does not mind what
type of build this is: canadian or not.

But, if we use LSB, I understand that the 'base' dependencies will get
pulled in automaticcally, and those three libs are in LSB. I have to admit
that I am not really up to speed wrt LSB. I'll read it carefully soonish...

> I'm not fond of having it hidden like that but that's how things are.

Basically, I'm not fond about having to handle such dependencies at all;
I see these as a necessary evil.

For GMP, MPFR et al. we have the choice of version, because they are 'new'
libraries, and it can make sense to use the exact version recommended by
each version of gcc. So we need to have that choice selection.

But for more traditional libraries (zlib, ncurses and expat), it's only a
matter of making it easier to build a toolchain that is portable. So, these
libraries will be blindly built: no version choice.

Now, that the build procedure is hidden inside the gdb build is historical.
At the time they were added, there was no need to add them for the host,
only for the target. So it made sense to hide them in the gdb build. Now,
these two libs should be extracted, of course.


But globally, about companion libraries, here's what I'd like we support:
 - by default, use the libs availble on the host:
   - the libs provided by the build distro for cross builds
   - the libs already built by another buildsystem, in case of canadian
 - at the user's request, build our own libs


And speaking about canadian-cross: I'm doing a lot of rework to make it
more streamlined than it is now. It takes time, but the current state of
the MQ is visible (and has been visible for the past 11+ months) there:
    http://crosstool-ng.org/hg/crosstool-ng/ct-ng.canadian/

Use with:
    hg qclone -p http://crosstool-ng.org/hg/crosstool-ng/ct-ng.canadian \
              http://crosstool-ng.org/hg/crosstool-ng                   \
              ctng.canadian
    cd ctng.canadian
    hg up -r 986e1c75432a
    hg qseries
    hg qpush -a

(It is needed to update to #986e1c75432a because I did not rebase the MQ
ontop the latest changesets, yet).

This MQ is now rather mature, but is not yet ready to be 'qfinish'ed, and
needs much testing. Hopefully, I'll be able to finish this before the next
release (end January 2012).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq