public inbox for buildbot@sourceware.org
 help / color / mirror / Atom feed
From: Mark Wielaard <mark@klomp.org>
To: "Martin Liška" <mliska@suse.cz>
Cc: buildbot@sourceware.org
Subject: Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it
Date: Sat, 18 Jun 2022 17:47:09 +0200	[thread overview]
Message-ID: <Yq3zfXYwSVnXO585@wildebeest.org> (raw)
In-Reply-To: <d29bad94-01ea-ff27-3cf0-c67b2bb650d8@suse.cz>

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

Hi Martin,

On Fri, Jun 17, 2022 at 11:17:28AM +0200, Martin Liška wrote:
> Doing echo "..." > bb-start.sh adds one more level of escaping
> and it is quite ugly. Use rather a proper file and discard all
> the Debian switch to bash and back.
> 
> Mark can you please test it if it works with the newly set 2 ENV variables:
> ENV IMAGE_NAME=debian-stable
> ENV CCACHE_LIBDIR=/usr/lib/ccache

I like this. But we cannot simply COPY in the bb-start.sh file.
Because it isn't available on the container host. The
DockerLatentWorker only sents the Container file. But we could fetch
it from the git repo using:

RUN wget -O /home/builder/bb-start.sh \
    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
    && chmod 755 /home/builder/bb-start.sh

We could also git clone the whole builder.git repo and then copy the
bb-start.sh file from the checkout, but this seems simpler.

If this works for you, please commit.

Thanks,

Mark

[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 8727 bytes --]

diff --git a/builder/containers/Containerfile-debian-stable b/builder/containers/Containerfile-debian-stable
index e7cd134..d69f725 100644
--- a/builder/containers/Containerfile-debian-stable
+++ b/builder/containers/Containerfile-debian-stable
@@ -22,34 +22,14 @@ RUN apt-get update && \
 # Setup user with same id as host user id.
 RUN adduser --home /home/builder --uid 1001 builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-# This script needs bash, so set sh to bash and then back...
-# Yes, this creates 3 layers... sigh
-RUN echo "dash dash/sh boolean false" | debconf-set-selections; \
-    DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
-RUN cd /home/builder; \
-    echo $'image_name=debian-stable\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
-RUN echo "dash dash/sh boolean true" | debconf-set-selections; \
-    DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash;
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=debian-stable
+ENV CCACHE_LIBDIR=/usr/lib/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+    && chmod 755 /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
index ce3b07e..057f27d 100644
--- a/builder/containers/Containerfile-fedora-latest
+++ b/builder/containers/Containerfile-fedora-latest
@@ -19,28 +19,14 @@ RUN dnf upgrade -y && \
 # Setup user with same id as host user id.
 RUN adduser -u 1001 -U -d /home/builder -m builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
-    echo $'image_name=fedora-latest\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib64/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=fedora-latest
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+    && chmod 755 /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-opensuse-leap b/builder/containers/Containerfile-opensuse-leap
index 51e74c4..6299e9b 100644
--- a/builder/containers/Containerfile-opensuse-leap
+++ b/builder/containers/Containerfile-opensuse-leap
@@ -25,28 +25,14 @@ RUN pip install buildbot-worker
 # Setup user with same id as host user id.
 RUN useradd -u 1001 -U -d /home/builder -m builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
-    echo $'image_name=opensuse-leap\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib64/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=opensuse-leap
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+    && chmod 755 /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-opensuse-tumbleweed b/builder/containers/Containerfile-opensuse-tumbleweed
index d20ad84..d574767 100644
--- a/builder/containers/Containerfile-opensuse-tumbleweed
+++ b/builder/containers/Containerfile-opensuse-tumbleweed
@@ -21,28 +21,14 @@ RUN pip install buildbot-worker
 # Setup user with same id as host user id.
 RUN useradd -u 1001 -U -d /home/builder -m builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
-    echo $'image_name=opensuse-tumbleweed\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib64/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=opensuse-tumbleweed
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+    && chmod 755 /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/bb-start.sh b/builder/containers/bb-start.sh
new file mode 100755
index 0000000..f8203ef
--- /dev/null
+++ b/builder/containers/bb-start.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+
+# The script is entrypoint for container workers.
+# The following environment variables must by set
+# by the corresponding Container files:
+#
+# - IMAGE_NAME - name of the image
+# - CCACHE_LIBDIR - path where to search for ccache (usually /usr/lib64/ccache)
+
+worker_dir=shared/$IMAGE_NAME/worker
+tac_file=$worker_dir/buildbot.tac
+if [ ! -f $tac_file ]; then
+  mkdir -p $worker_dir
+  buildbot-worker create-worker $worker_dir $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS
+  echo buildbot@sourceware.org > $worker_dir/info/admin
+  echo $IMAGE_NAME > $worker_dir/info/host
+else
+  sed -ie "s/^passwd = .*/passwd = '$WORKERPASS'/" $tac_file
+fi
+unset WORKERPASS
+export PATH=$CCACHE_LIBDIR:$PATH
+mkdir -p shared/$IMAGE_NAME/ccache
+export CCACHE_DIR=/home/builder/shared/$IMAGE_NAME/ccache
+buildbot-worker start --nodaemon $worker_dir


  reply	other threads:[~2022-06-18 15:47 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-06-17  9:17 Martin Liška
2022-06-18 15:47 ` Mark Wielaard [this message]
2022-06-18 18:35   ` Martin Liška
2022-06-18 18:38     ` Martin Liška
2022-06-18 20:11     ` Mark Wielaard
2022-06-19  7:12       ` Martin Liška

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=Yq3zfXYwSVnXO585@wildebeest.org \
    --to=mark@klomp.org \
    --cc=buildbot@sourceware.org \
    --cc=mliska@suse.cz \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).