Automated Toolchain Building and Testing

Automated Toolchain Building and Testing

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 2137 bytes --]

Hi!

My first try on a build robot (http://toolchain.lug-owl.de/buildbot/
and http://toolchain.lug-owl.de/buildbot/timeline.php) is running for
some time now, so I'd like to do a next step.

    (The current homegrown build script is designed to do a
    cross-build with a named --target and no --build/--host, building
    binutils, gold, gdb and stace1 gcc. The testsuite isn't run as of
    now.  And since this isn't a full bootstrap re-building GCC with
    itself, it's mostly a compatibility check for the toolchain
    sources to build with the detected system C compiler.)

While the current setup works quite fine, it's time to extend it to
(as a minimum) run the test suite and report on it. As people
suggested, I had at some other Continuous Integration systems, mostly
Jenkins and BuildBot. Both share a common problem: They require extra
software on a given build slave to work. (Jenkins expects a working
Java installation, BuildBot needs Python and Twisted.) Without having
checked this, I guess that other CI systems impose similar
dependencies.

At this point, I'd like to get some feedback from you!  Do you have
any CI system running that only needs eg. ssh to log-in into a slave
box?  Do you think that it's acceptable for a Toolchain Build Robot to
require Python/Twisted or Java on a slave system? I'm thinking about
some of the more obscure systems (m68k-netbsd, ...) that probably
won't have Java readily running. And even gcj may not be ported to
all systems that otherwise could run a Build Robot slave instance.

The decision thus is like this: Continue to run some self-written CI
system that fits our needs and specifically covers the rarely-used
platforms? Or run one with well-known software, though we'll probably
loose obscure systems?

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:         "really soon now":      an unspecified period of time, likly to
the second  :                                 be greater than any reasonable definition
                                              of "soon".

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Automated Toolchain Building and Testing

[Sebastian Huber]

Hello,

you can also use a cross compiler and run the tests on a simulator or remote 
target.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber@embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

Re: Automated Toolchain Building and Testing

[Samuel Mi]

Hi Jan,

Looks like you for now have been trying to find out a solution
suitable for you to automatically build GCC from source combined with
certain continuous systems like Jenkins. As a matter of fact, Jenkins
is exactly a good choice to do such thing just mentioned, due to
itself with so many plugins[1] you can pick up to fit your needs.

In addition, there is a instance of jenkins out there used to nightly
build toolchain[2] as a good starting point for you to setup such a CI
(Continuous Integration) infrastructure.


References:
[1] https://wiki.jenkins-ci.org/display/JENKINS/Plugins
[2] https://android-build.linaro.org/jenkins/view/Toolchain/


Yours truly,
Samuel

On Wed, Aug 28, 2013 at 3:54 AM, Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:
> Hi!
>
> My first try on a build robot (http://toolchain.lug-owl.de/buildbot/
> and http://toolchain.lug-owl.de/buildbot/timeline.php) is running for
> some time now, so I'd like to do a next step.
>
>     (The current homegrown build script is designed to do a
>     cross-build with a named --target and no --build/--host, building
>     binutils, gold, gdb and stace1 gcc. The testsuite isn't run as of
>     now.  And since this isn't a full bootstrap re-building GCC with
>     itself, it's mostly a compatibility check for the toolchain
>     sources to build with the detected system C compiler.)
>
> While the current setup works quite fine, it's time to extend it to
> (as a minimum) run the test suite and report on it. As people
> suggested, I had at some other Continuous Integration systems, mostly
> Jenkins and BuildBot. Both share a common problem: They require extra
> software on a given build slave to work. (Jenkins expects a working
> Java installation, BuildBot needs Python and Twisted.) Without having
> checked this, I guess that other CI systems impose similar
> dependencies.
>
> At this point, I'd like to get some feedback from you!  Do you have
> any CI system running that only needs eg. ssh to log-in into a slave
> box?  Do you think that it's acceptable for a Toolchain Build Robot to
> require Python/Twisted or Java on a slave system? I'm thinking about
> some of the more obscure systems (m68k-netbsd, ...) that probably
> won't have Java readily running. And even gcj may not be ported to
> all systems that otherwise could run a Build Robot slave instance.
>
> The decision thus is like this: Continue to run some self-written CI
> system that fits our needs and specifically covers the rarely-used
> platforms? Or run one with well-known software, though we'll probably
> loose obscure systems?
>
> MfG, JBG
>
> --
>       Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
> Signature of:         "really soon now":      an unspecified period of time, likly to
> the second  :                                 be greater than any reasonable definition
>                                               of "soon".

Re: Automated Toolchain Building and Testing

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

On Wed, 2013-08-28 23:26:29 +0800, Samuel Mi <samuel.miing@gmail.com> wrote:
> Looks like you for now have been trying to find out a solution
> suitable for you to automatically build GCC from source combined with
> certain continuous systems like Jenkins. As a matter of fact, Jenkins
> is exactly a good choice to do such thing just mentioned, due to
> itself with so many plugins[1] you can pick up to fit your needs.

I'm not too sure if Jenkins is actually a good choice, just because I
question that there's a working Java especially for old Unix-alike
systems that GCC still (in theory) supports. What about eg. older IRIX
or Ultrix systems?  ...or can you, instead of using the Java-based
client part of Jenkins, issue all commands over a SSH (or maybe even
Telnet...) session?  Is there a module for this available?

We've been using Jenkins at the company I work for, though only for
up-to-date mainstream systems with current state-of-the-art software.
But what I really like is to also support old and legacy systems, as
well as probably current ones with only a minimal userbase.

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:                     Eine Freie Meinung in einem Freien Kopf
the second  :                   für einen Freien Staat voll Freier Bürger.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Automated Toolchain Building and Testing

[Samuel Mi]

> I'm not too sure if Jenkins is actually a good choice, just because I
> question that there's a working Java especially for old Unix-alike
> systems that GCC still (in theory) supports. What about eg. older IRIX
> or Ultrix systems?
I have no such experience on running jenkins under java runtime on old
and legacy systems. In my case, it's exactly focused on such
environment which is with modern linux-like systems like Ubuntu. plus,
the minimum runtime environment of running jenkins with up-to-date
version is java 1.5 or later.

> ...or can you, instead of using the Java-based
> client part of Jenkins, issue all commands over a SSH (or maybe even
> Telnet...) session?  Is there a module for this available?
If making jenkins running on target systems you want whether old or
modern, then take a look at Jenkins-SSH
(https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+SSH) to remotely
control over ssh.

Re: Automated Toolchain Building and Testing

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 731 bytes --]

On Thu, 2013-08-29 02:43:54 +0800, Samuel Mi <samuel.miing@gmail.com> wrote:
> > ...or can you, instead of using the Java-based
> > client part of Jenkins, issue all commands over a SSH (or maybe even
> > Telnet...) session?  Is there a module for this available?
> If making jenkins running on target systems you want whether old or
> modern, then take a look at Jenkins-SSH
> (https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+SSH) to remotely
> control over ssh.

This looks like a SSH connector for the Jenkins server side, no?

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:                 Friends are relatives you make for yourself.
the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Automated Toolchain Building and Testing

[Samuel Mi]

On Thu, Aug 29, 2013 at 2:54 AM, Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:
> On Thu, 2013-08-29 02:43:54 +0800, Samuel Mi <samuel.miing@gmail.com> wrote:
>> > ...or can you, instead of using the Java-based
>> > client part of Jenkins, issue all commands over a SSH (or maybe even
>> > Telnet...) session?  Is there a module for this available?
>> If making jenkins running on target systems you want whether old or
>> modern, then take a look at Jenkins-SSH
>> (https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+SSH) to remotely
>> control over ssh.
>
> This looks like a SSH connector for the Jenkins server side, no?
No. Actually, Jenkins implements a built-in SSH server within itself.
At this point, it's consider to be a normal SSH server. So, you can
remotely access Jenkins server via SSH after setting up corresponding
configurations within it.

Re: Automated Toolchain Building and Testing

[Dan Kegel]

On Wed, Aug 28, 2013 at 5:52 PM, Samuel Mi <samuel.miing@gmail.com> wrote:
>> This looks like a SSH connector for the Jenkins server side, no?
> No. Actually, Jenkins implements a built-in SSH server within itself.

Doesn't really help platforms that can boot linux but that don't have
a sufficient
version of Java/python.  For them, one really wants a buildbot/jenkins/whatever
build node written in C/C++, since those are the languages most likely to be
universally available on such machines.
- Dan

Re: Automated Toolchain Building and Testing

[Paul_Koning]

On Aug 28, 2013, at 8:52 PM, Samuel Mi <samuel.miing@gmail.com> wrote:

> On Thu, Aug 29, 2013 at 2:54 AM, Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:
>> On Thu, 2013-08-29 02:43:54 +0800, Samuel Mi <samuel.miing@gmail.com> wrote:
>>>> ...or can you, instead of using the Java-based
>>>> client part of Jenkins, issue all commands over a SSH (or maybe even
>>>> Telnet...) session?  Is there a module for this available?
>>> If making jenkins running on target systems you want whether old or
>>> modern, then take a look at Jenkins-SSH
>>> (https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+SSH) to remotely
>>> control over ssh.
>> 
>> This looks like a SSH connector for the Jenkins server side, no?
> No. Actually, Jenkins implements a built-in SSH server within itself.
> At this point, it's consider to be a normal SSH server. So, you can
> remotely access Jenkins server via SSH after setting up corresponding
> configurations within it.

What non-antique Linux doesn't come with Python?

	paul

Re: Automated Toolchain Building and Testing

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1853 bytes --]

On Thu, 2013-08-29 01:18:32 +0000, Paul_Koning@Dell.com <Paul_Koning@Dell.com> wrote:
> On Aug 28, 2013, at 8:52 PM, Samuel Mi <samuel.miing@gmail.com> wrote:
> > On Thu, Aug 29, 2013 at 2:54 AM, Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:
> > > On Thu, 2013-08-29 02:43:54 +0800, Samuel Mi <samuel.miing@gmail.com> wrote:
> > > > > ...or can you, instead of using the Java-based
> > > > > client part of Jenkins, issue all commands over a SSH (or maybe even
> > > > > Telnet...) session?  Is there a module for this available?
> > > > If making jenkins running on target systems you want whether old or
> > > > modern, then take a look at Jenkins-SSH
> > > > (https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+SSH) to remotely
> > > > control over ssh.
> > > This looks like a SSH connector for the Jenkins server side, no?
> > No. Actually, Jenkins implements a built-in SSH server within itself.
> > At this point, it's consider to be a normal SSH server. So, you can
> > remotely access Jenkins server via SSH after setting up corresponding
> > configurations within it.
> 
> What non-antique Linux doesn't come with Python?

GCC and Binutils don't exclusively run on Linux platforms. Actually,
the Linux targets are those which are actually tested the most by
usual day-to-day usage.  I'd specifically like to run tests on
platforms that are _not_ used by large user bases, but still supported
by GCC.

Those platforms tend to be non-Linux, non-i386, non-recent, or any
combination if those. *This* is what I want to support. (It still
should also run on Linux, though.)

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:                   ...und wenn Du denkst, es geht nicht mehr,
the second  :                          kommt irgendwo ein Lichtlein her.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Automated Toolchain Building and Testing

[Rainer Orth]

Jan-Benedict Glaw <jbglaw@lug-owl.de> writes:

> On Wed, 2013-08-28 23:26:29 +0800, Samuel Mi <samuel.miing@gmail.com> wrote:
>> Looks like you for now have been trying to find out a solution
>> suitable for you to automatically build GCC from source combined with
>> certain continuous systems like Jenkins. As a matter of fact, Jenkins
>> is exactly a good choice to do such thing just mentioned, due to
>> itself with so many plugins[1] you can pick up to fit your needs.
>
> I'm not too sure if Jenkins is actually a good choice, just because I
> question that there's a working Java especially for old Unix-alike
> systems that GCC still (in theory) supports. What about eg. older IRIX
> or Ultrix systems?  ...or can you, instead of using the Java-based

I honestly wouldn't worry about such legacy systems: their respective
maintainers take care of testing them, and it would be hard nowadays to
even find both hardware and OS media to set up a new system.

FWIW, IRIX 6.5 was last supported in gcc 4.7, and that's the only IRIX
release I'm still testing.  IRIX 5.3/6.x was deprecated in gcc 4.5
already, a release no longer supported and thus irrelevant.  Same for
Tru64 UNIX: V5.1 support was deprecated in 4.7; still testing that
either.

Hope this helps.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: Automated Toolchain Building and Testing

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1811 bytes --]

On Thu, 2013-08-29 10:34:40 +0200, Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> wrote:
> Jan-Benedict Glaw <jbglaw@lug-owl.de> writes:
> > On Wed, 2013-08-28 23:26:29 +0800, Samuel Mi <samuel.miing@gmail.com> wrote:
> > > Looks like you for now have been trying to find out a solution
> > > suitable for you to automatically build GCC from source combined with
> > > certain continuous systems like Jenkins. As a matter of fact, Jenkins
> > > is exactly a good choice to do such thing just mentioned, due to
> > > itself with so many plugins[1] you can pick up to fit your needs.
> >
> > I'm not too sure if Jenkins is actually a good choice, just because I
> > question that there's a working Java especially for old Unix-alike
> > systems that GCC still (in theory) supports. What about eg. older IRIX
> > or Ultrix systems?  ...or can you, instead of using the Java-based
> 
> I honestly wouldn't worry about such legacy systems: their respective
> maintainers take care of testing them, and it would be hard nowadays to
> even find both hardware and OS media to set up a new system.

Well, I do.

Just for example, I do care about the VAX backend. I've had something
similar to my current build robot running for a while, then it broke
(due to other circumstances) and it took me quite some time to track
down some "easter eggs" introduced in the mean time.  I'd like to
catch such things early, personally for the VAX backend, but if
possible in general.

Shouldn't be too hard to have something that dispatches commands
purely through SSH.  If there's nothing available, I'll just write it.

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:               http://www.eyrie.org/~eagle/faqs/questions.html
the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Automated Toolchain Building and Testing

[Diego Novillo]

On Thu, Aug 29, 2013 at 6:02 AM, Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:
> On Thu, 2013-08-29 10:34:40 +0200, Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> wrote:
>>
>> I honestly wouldn't worry about such legacy systems: their respective
>> maintainers take care of testing them, and it would be hard nowadays to
>> even find both hardware and OS media to set up a new system.
>
> Well, I do.

That's fine, but I don't think we should not hold a good solution in
the quest for the perfect one. How about we start with this version?
Whoever is interested in extending it to other systems, can do it
incrementally.

I have not yet caught up to the whole thread, but I suppose the
possibility of running it on the Compile Farm has been discussed?


Diego.

Re: Automated Toolchain Building and Testing

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1536 bytes --]

On Thu, 2013-08-29 07:21:28 -0400, Diego Novillo <dnovillo@google.com> wrote:
> On Thu, Aug 29, 2013 at 6:02 AM, Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:
> > On Thu, 2013-08-29 10:34:40 +0200, Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> wrote:
> > > I honestly wouldn't worry about such legacy systems: their respective
> > > maintainers take care of testing them, and it would be hard nowadays to
> > > even find both hardware and OS media to set up a new system.
> >
> > Well, I do.
> 
> That's fine, but I don't think we should not hold a good solution in
> the quest for the perfect one. How about we start with this version?
> Whoever is interested in extending it to other systems, can do it
> incrementally.

It's already running :)  This thread is already about the next
version. The current variant will easily run on anything that is able
to run GIT (probably any other SCM) and SSH (and it'd be easy to adopt
it to anything else like telnet or rlogin.)

> I have not yet caught up to the whole thread, but I suppose the
> possibility of running it on the Compile Farm has been discussed?

David Edelsohn already pointed me to that and I requested an accound
some time ago, but I'm still waiting for a reply. (Though it's holiday
time, so I guess the people doing account maintenance are just out for
a trip.)

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:                 Gib Dein Bestes. Dann übertriff Dich selbst!
the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]