[RFC,PATCH] containers: extract bb-start.sh to a file and reuse it

[RFC,PATCH] containers: extract bb-start.sh to a file and reuse it

[Martin Liška]

Doing echo "..." > bb-start.sh adds one more level of escaping
and it is quite ugly. Use rather a proper file and discard all
the Debian switch to bash and back.

Mark can you please test it if it works with the newly set 2 ENV variables:
ENV IMAGE_NAME=debian-stable
ENV CCACHE_LIBDIR=/usr/lib/ccache

Thanks,
Martin

---
 .../containers/Containerfile-debian-stable    | 31 ++-----------------
 .../containers/Containerfile-fedora-latest    | 25 ++-------------
 .../containers/Containerfile-opensuse-leap    | 25 ++-------------
 .../Containerfile-opensuse-tumbleweed         | 25 ++-------------
 builder/containers/bb-start.sh                | 24 ++++++++++++++
 5 files changed, 36 insertions(+), 94 deletions(-)
 create mode 100755 builder/containers/bb-start.sh

diff --git a/builder/containers/Containerfile-debian-stable b/builder/containers/Containerfile-debian-stable
index e7cd134..86ed88f 100644
--- a/builder/containers/Containerfile-debian-stable
+++ b/builder/containers/Containerfile-debian-stable
@@ -22,34 +22,9 @@ RUN apt-get update && \
 # Setup user with same id as host user id.
 RUN adduser --home /home/builder --uid 1001 builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-# This script needs bash, so set sh to bash and then back...
-# Yes, this creates 3 layers... sigh
-RUN echo "dash dash/sh boolean false" | debconf-set-selections; \
-    DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
-RUN cd /home/builder; \
-    echo $'image_name=debian-stable\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
-RUN echo "dash dash/sh boolean true" | debconf-set-selections; \
-    DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash;
+ENV IMAGE_NAME=debian-stable
+ENV CCACHE_LIBDIR=/usr/lib/ccache
+COPY bb-start.sh /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
index ce3b07e..450eea6 100644
--- a/builder/containers/Containerfile-fedora-latest
+++ b/builder/containers/Containerfile-fedora-latest
@@ -19,28 +19,9 @@ RUN dnf upgrade -y && \
 # Setup user with same id as host user id.
 RUN adduser -u 1001 -U -d /home/builder -m builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
-    echo $'image_name=fedora-latest\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib64/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
+ENV IMAGE_NAME=fedora-latest
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+COPY bb-start.sh /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-opensuse-leap b/builder/containers/Containerfile-opensuse-leap
index 51e74c4..197f1af 100644
--- a/builder/containers/Containerfile-opensuse-leap
+++ b/builder/containers/Containerfile-opensuse-leap
@@ -25,28 +25,9 @@ RUN pip install buildbot-worker
 # Setup user with same id as host user id.
 RUN useradd -u 1001 -U -d /home/builder -m builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
-    echo $'image_name=opensuse-leap\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib64/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
+ENV IMAGE_NAME=opensuse-leap
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+COPY bb-start.sh /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-opensuse-tumbleweed b/builder/containers/Containerfile-opensuse-tumbleweed
index d20ad84..d8157ed 100644
--- a/builder/containers/Containerfile-opensuse-tumbleweed
+++ b/builder/containers/Containerfile-opensuse-tumbleweed
@@ -21,28 +21,9 @@ RUN pip install buildbot-worker
 # Setup user with same id as host user id.
 RUN useradd -u 1001 -U -d /home/builder -m builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
-    echo $'image_name=opensuse-tumbleweed\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib64/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
+ENV IMAGE_NAME=opensuse-tumbleweed
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+COPY bb-start.sh /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/bb-start.sh b/builder/containers/bb-start.sh
new file mode 100755
index 0000000..f8203ef
--- /dev/null
+++ b/builder/containers/bb-start.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+
+# The script is entrypoint for container workers.
+# The following environment variables must by set
+# by the corresponding Container files:
+#
+# - IMAGE_NAME - name of the image
+# - CCACHE_LIBDIR - path where to search for ccache (usually /usr/lib64/ccache)
+
+worker_dir=shared/$IMAGE_NAME/worker
+tac_file=$worker_dir/buildbot.tac
+if [ ! -f $tac_file ]; then
+  mkdir -p $worker_dir
+  buildbot-worker create-worker $worker_dir $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS
+  echo buildbot@sourceware.org > $worker_dir/info/admin
+  echo $IMAGE_NAME > $worker_dir/info/host
+else
+  sed -ie "s/^passwd = .*/passwd = '$WORKERPASS'/" $tac_file
+fi
+unset WORKERPASS
+export PATH=$CCACHE_LIBDIR:$PATH
+mkdir -p shared/$IMAGE_NAME/ccache
+export CCACHE_DIR=/home/builder/shared/$IMAGE_NAME/ccache
+buildbot-worker start --nodaemon $worker_dir
-- 
2.36.1

Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

Hi Martin,

On Fri, Jun 17, 2022 at 11:17:28AM +0200, Martin Liška wrote:
> Doing echo "..." > bb-start.sh adds one more level of escaping
> and it is quite ugly. Use rather a proper file and discard all
> the Debian switch to bash and back.
> 
> Mark can you please test it if it works with the newly set 2 ENV variables:
> ENV IMAGE_NAME=debian-stable
> ENV CCACHE_LIBDIR=/usr/lib/ccache

I like this. But we cannot simply COPY in the bb-start.sh file.
Because it isn't available on the container host. The
DockerLatentWorker only sents the Container file. But we could fetch
it from the git repo using:

RUN wget -O /home/builder/bb-start.sh \
    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
    && chmod 755 /home/builder/bb-start.sh

We could also git clone the whole builder.git repo and then copy the
bb-start.sh file from the checkout, but this seems simpler.

If this works for you, please commit.

Thanks,

Mark

[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 8727 bytes --]

diff --git a/builder/containers/Containerfile-debian-stable b/builder/containers/Containerfile-debian-stable
index e7cd134..d69f725 100644
--- a/builder/containers/Containerfile-debian-stable
+++ b/builder/containers/Containerfile-debian-stable
@@ -22,34 +22,14 @@ RUN apt-get update && \
 # Setup user with same id as host user id.
 RUN adduser --home /home/builder --uid 1001 builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-# This script needs bash, so set sh to bash and then back...
-# Yes, this creates 3 layers... sigh
-RUN echo "dash dash/sh boolean false" | debconf-set-selections; \
-    DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
-RUN cd /home/builder; \
-    echo $'image_name=debian-stable\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
-RUN echo "dash dash/sh boolean true" | debconf-set-selections; \
-    DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash;
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=debian-stable
+ENV CCACHE_LIBDIR=/usr/lib/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+    && chmod 755 /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
index ce3b07e..057f27d 100644
--- a/builder/containers/Containerfile-fedora-latest
+++ b/builder/containers/Containerfile-fedora-latest
@@ -19,28 +19,14 @@ RUN dnf upgrade -y && \
 # Setup user with same id as host user id.
 RUN adduser -u 1001 -U -d /home/builder -m builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
-    echo $'image_name=fedora-latest\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib64/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=fedora-latest
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+    && chmod 755 /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-opensuse-leap b/builder/containers/Containerfile-opensuse-leap
index 51e74c4..6299e9b 100644
--- a/builder/containers/Containerfile-opensuse-leap
+++ b/builder/containers/Containerfile-opensuse-leap
@@ -25,28 +25,14 @@ RUN pip install buildbot-worker
 # Setup user with same id as host user id.
 RUN useradd -u 1001 -U -d /home/builder -m builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
-    echo $'image_name=opensuse-leap\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib64/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=opensuse-leap
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+    && chmod 755 /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-opensuse-tumbleweed b/builder/containers/Containerfile-opensuse-tumbleweed
index d20ad84..d574767 100644
--- a/builder/containers/Containerfile-opensuse-tumbleweed
+++ b/builder/containers/Containerfile-opensuse-tumbleweed
@@ -21,28 +21,14 @@ RUN pip install buildbot-worker
 # Setup user with same id as host user id.
 RUN useradd -u 1001 -U -d /home/builder -m builder
 
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
-    echo $'image_name=opensuse-tumbleweed\n\
-        worker_dir=shared/$image_name/worker\n\
-        tac_file=$worker_dir/buildbot.tac\n\
-        if [ ! -f $tac_file ]; then\n\
-        mkdir -p $worker_dir\n\
-        buildbot-worker create-worker $worker_dir \
-          $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
-        echo buildbot@sourceware.org > $worker_dir/info/admin\n\
-        echo $image_name > $worker_dir/info/host\n\
-      else\n\
-        sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
-      fi\n\
-      unset WORKERPASS\n\
-      export PATH=/usr/lib64/ccache:$PATH\n\
-      mkdir -p shared/$image_name/ccache\n\
-      export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
-      buildbot-worker start --nodaemon $worker_dir\n' \
-	> bb-start.sh && \
-    chmod 755 bb-start.sh
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=opensuse-tumbleweed
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+    'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+    && chmod 755 /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/bb-start.sh b/builder/containers/bb-start.sh
new file mode 100755
index 0000000..f8203ef
--- /dev/null
+++ b/builder/containers/bb-start.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+
+# The script is entrypoint for container workers.
+# The following environment variables must by set
+# by the corresponding Container files:
+#
+# - IMAGE_NAME - name of the image
+# - CCACHE_LIBDIR - path where to search for ccache (usually /usr/lib64/ccache)
+
+worker_dir=shared/$IMAGE_NAME/worker
+tac_file=$worker_dir/buildbot.tac
+if [ ! -f $tac_file ]; then
+  mkdir -p $worker_dir
+  buildbot-worker create-worker $worker_dir $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS
+  echo buildbot@sourceware.org > $worker_dir/info/admin
+  echo $IMAGE_NAME > $worker_dir/info/host
+else
+  sed -ie "s/^passwd = .*/passwd = '$WORKERPASS'/" $tac_file
+fi
+unset WORKERPASS
+export PATH=$CCACHE_LIBDIR:$PATH
+mkdir -p shared/$IMAGE_NAME/ccache
+export CCACHE_DIR=/home/builder/shared/$IMAGE_NAME/ccache
+buildbot-worker start --nodaemon $worker_dir

Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it

[Martin Liška]

On 6/18/22 17:47, Mark Wielaard wrote:
> Hi Martin,
> 
> On Fri, Jun 17, 2022 at 11:17:28AM +0200, Martin Liška wrote:
>> Doing echo "..." > bb-start.sh adds one more level of escaping
>> and it is quite ugly. Use rather a proper file and discard all
>> the Debian switch to bash and back.
>>
>> Mark can you please test it if it works with the newly set 2 ENV variables:
>> ENV IMAGE_NAME=debian-stable
>> ENV CCACHE_LIBDIR=/usr/lib/ccache
> 
> I like this. But we cannot simply COPY in the bb-start.sh file.
> Because it isn't available on the container host. The
> DockerLatentWorker only sents the Container file. But we could fetch
> it from the git repo using:
> 
> RUN wget -O /home/builder/bb-start.sh \
>     'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
>     && chmod 755 /home/builder/bb-start.sh

Good, I've done that and pushed to main.

However, I see an error now:
https://builder.sourceware.org/buildbot/#/builders/97/builds/248

Can you please check buildbot master log files in order to investigate what's wrong?

Thanks,
Martin

> 
> We could also git clone the whole builder.git repo and then copy the
> bb-start.sh file from the checkout, but this seems simpler.
> 
> If this works for you, please commit.
> 
> Thanks,
> 
> Mark

Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it

[Martin Liška]

On 6/18/22 20:35, Martin Liška wrote:
> However, I see an error now:
> https://builder.sourceware.org/buildbot/#/builders/97/builds/248

Well, the error message is:

buildbot.interfaces.LatentWorkerCannotSubstantiate: Image "bb1_139886867151000_image" not found on docker host.

and it happens when I re-run a build.

So it likely means one can't re-run such a build that uses LatentWorkers which
images are rebuilt time to time.

Cheers,
Martin

Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

Hi Martin,

On Sat, Jun 18, 2022 at 08:35:06PM +0200, Martin Liška wrote:
> > I like this. But we cannot simply COPY in the bb-start.sh file.
> > Because it isn't available on the container host. The
> > DockerLatentWorker only sents the Container file. But we could fetch
> > it from the git repo using:
> > 
> > RUN wget -O /home/builder/bb-start.sh \
> >     'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
> >     && chmod 755 /home/builder/bb-start.sh
> 
> Good, I've done that and pushed to main.
> 
> However, I see an error now:
> https://builder.sourceware.org/buildbot/#/builders/97/builds/248
> 
> Can you please check buildbot master log files in order to investigate what's wrong?

The issue was that there were now both a RUN wget && chmod and a COPY
in the Container files. Removing the COPY seems to have made it work again.

Cheers,

Mark

[-- Attachment #2: 0001-Containerfiles-Just-RUN-wget-chmod-don-t-also-COPY-b.patch --]
[-- Type: text/x-diff, Size: 3985 bytes --]

From 35bafdb6c93177ca33ef99b394b0207b14334da9 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 18 Jun 2022 21:32:25 +0200
Subject: [PATCH] Containerfiles: Just RUN wget && chmod, don't also COPY
 bb-start.sh

---
 builder/containers/Containerfile-debian-stable       | 3 ++-
 builder/containers/Containerfile-fedora-latest       | 3 ++-
 builder/containers/Containerfile-opensuse-leap       | 3 ++-
 builder/containers/Containerfile-opensuse-tumbleweed | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/builder/containers/Containerfile-debian-stable b/builder/containers/Containerfile-debian-stable
index 6776c3b..d69f725 100644
--- a/builder/containers/Containerfile-debian-stable
+++ b/builder/containers/Containerfile-debian-stable
@@ -22,13 +22,14 @@ RUN apt-get update && \
 # Setup user with same id as host user id.
 RUN adduser --home /home/builder --uid 1001 builder
 
+# Set image specific environment variables used by bb-start.sh
 ENV IMAGE_NAME=debian-stable
 ENV CCACHE_LIBDIR=/usr/lib/ccache
 
+# Put bb-start.sh script in homedir and make it executable
 RUN wget -O /home/builder/bb-start.sh \
     'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
     && chmod 755 /home/builder/bb-start.sh
-COPY bb-start.sh /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
index 556a959..057f27d 100644
--- a/builder/containers/Containerfile-fedora-latest
+++ b/builder/containers/Containerfile-fedora-latest
@@ -19,13 +19,14 @@ RUN dnf upgrade -y && \
 # Setup user with same id as host user id.
 RUN adduser -u 1001 -U -d /home/builder -m builder
 
+# Set image specific environment variables used by bb-start.sh
 ENV IMAGE_NAME=fedora-latest
 ENV CCACHE_LIBDIR=/usr/lib64/ccache
 
+# Put bb-start.sh script in homedir and make it executable
 RUN wget -O /home/builder/bb-start.sh \
     'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
     && chmod 755 /home/builder/bb-start.sh
-COPY bb-start.sh /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-opensuse-leap b/builder/containers/Containerfile-opensuse-leap
index 4e88f8c..6299e9b 100644
--- a/builder/containers/Containerfile-opensuse-leap
+++ b/builder/containers/Containerfile-opensuse-leap
@@ -25,13 +25,14 @@ RUN pip install buildbot-worker
 # Setup user with same id as host user id.
 RUN useradd -u 1001 -U -d /home/builder -m builder
 
+# Set image specific environment variables used by bb-start.sh
 ENV IMAGE_NAME=opensuse-leap
 ENV CCACHE_LIBDIR=/usr/lib64/ccache
 
+# Put bb-start.sh script in homedir and make it executable
 RUN wget -O /home/builder/bb-start.sh \
     'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
     && chmod 755 /home/builder/bb-start.sh
-COPY bb-start.sh /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
diff --git a/builder/containers/Containerfile-opensuse-tumbleweed b/builder/containers/Containerfile-opensuse-tumbleweed
index 002291a..d574767 100644
--- a/builder/containers/Containerfile-opensuse-tumbleweed
+++ b/builder/containers/Containerfile-opensuse-tumbleweed
@@ -21,13 +21,14 @@ RUN pip install buildbot-worker
 # Setup user with same id as host user id.
 RUN useradd -u 1001 -U -d /home/builder -m builder
 
+# Set image specific environment variables used by bb-start.sh
 ENV IMAGE_NAME=opensuse-tumbleweed
 ENV CCACHE_LIBDIR=/usr/lib64/ccache
 
+# Put bb-start.sh script in homedir and make it executable
 RUN wget -O /home/builder/bb-start.sh \
     'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
     && chmod 755 /home/builder/bb-start.sh
-COPY bb-start.sh /home/builder/bb-start.sh
 
 # And now run the script
 USER builder
-- 
2.30.2

Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it

[Martin Liška]

On 6/18/22 22:11, Mark Wielaard wrote:
> Hi Martin,
> 
> On Sat, Jun 18, 2022 at 08:35:06PM +0200, Martin Liška wrote:
>>> I like this. But we cannot simply COPY in the bb-start.sh file.
>>> Because it isn't available on the container host. The
>>> DockerLatentWorker only sents the Container file. But we could fetch
>>> it from the git repo using:
>>>
>>> RUN wget -O /home/builder/bb-start.sh \
>>>     'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
>>>     && chmod 755 /home/builder/bb-start.sh
>>
>> Good, I've done that and pushed to main.
>>
>> However, I see an error now:
>> https://builder.sourceware.org/buildbot/#/builders/97/builds/248
>>
>> Can you please check buildbot master log files in order to investigate what's wrong?
> 
> The issue was that there were now both a RUN wget && chmod and a COPY
> in the Container files. Removing the COPY seems to have made it work again.

Oh, what a stupid mistake I've made.

Thanks for the fix!

Martin

> 
> Cheers,
> 
> Mark