* [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it
@ 2022-06-17 9:17 Martin Liška
2022-06-18 15:47 ` Mark Wielaard
0 siblings, 1 reply; 6+ messages in thread
From: Martin Liška @ 2022-06-17 9:17 UTC (permalink / raw)
To: buildbot; +Cc: Mark Wielaard
Doing echo "..." > bb-start.sh adds one more level of escaping
and it is quite ugly. Use rather a proper file and discard all
the Debian switch to bash and back.
Mark can you please test it if it works with the newly set 2 ENV variables:
ENV IMAGE_NAME=debian-stable
ENV CCACHE_LIBDIR=/usr/lib/ccache
Thanks,
Martin
---
.../containers/Containerfile-debian-stable | 31 ++-----------------
.../containers/Containerfile-fedora-latest | 25 ++-------------
.../containers/Containerfile-opensuse-leap | 25 ++-------------
.../Containerfile-opensuse-tumbleweed | 25 ++-------------
builder/containers/bb-start.sh | 24 ++++++++++++++
5 files changed, 36 insertions(+), 94 deletions(-)
create mode 100755 builder/containers/bb-start.sh
diff --git a/builder/containers/Containerfile-debian-stable b/builder/containers/Containerfile-debian-stable
index e7cd134..86ed88f 100644
--- a/builder/containers/Containerfile-debian-stable
+++ b/builder/containers/Containerfile-debian-stable
@@ -22,34 +22,9 @@ RUN apt-get update && \
# Setup user with same id as host user id.
RUN adduser --home /home/builder --uid 1001 builder
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-# This script needs bash, so set sh to bash and then back...
-# Yes, this creates 3 layers... sigh
-RUN echo "dash dash/sh boolean false" | debconf-set-selections; \
- DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
-RUN cd /home/builder; \
- echo $'image_name=debian-stable\n\
- worker_dir=shared/$image_name/worker\n\
- tac_file=$worker_dir/buildbot.tac\n\
- if [ ! -f $tac_file ]; then\n\
- mkdir -p $worker_dir\n\
- buildbot-worker create-worker $worker_dir \
- $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
- echo buildbot@sourceware.org > $worker_dir/info/admin\n\
- echo $image_name > $worker_dir/info/host\n\
- else\n\
- sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
- fi\n\
- unset WORKERPASS\n\
- export PATH=/usr/lib/ccache:$PATH\n\
- mkdir -p shared/$image_name/ccache\n\
- export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
- buildbot-worker start --nodaemon $worker_dir\n' \
- > bb-start.sh && \
- chmod 755 bb-start.sh
-RUN echo "dash dash/sh boolean true" | debconf-set-selections; \
- DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash;
+ENV IMAGE_NAME=debian-stable
+ENV CCACHE_LIBDIR=/usr/lib/ccache
+COPY bb-start.sh /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
index ce3b07e..450eea6 100644
--- a/builder/containers/Containerfile-fedora-latest
+++ b/builder/containers/Containerfile-fedora-latest
@@ -19,28 +19,9 @@ RUN dnf upgrade -y && \
# Setup user with same id as host user id.
RUN adduser -u 1001 -U -d /home/builder -m builder
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
- echo $'image_name=fedora-latest\n\
- worker_dir=shared/$image_name/worker\n\
- tac_file=$worker_dir/buildbot.tac\n\
- if [ ! -f $tac_file ]; then\n\
- mkdir -p $worker_dir\n\
- buildbot-worker create-worker $worker_dir \
- $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
- echo buildbot@sourceware.org > $worker_dir/info/admin\n\
- echo $image_name > $worker_dir/info/host\n\
- else\n\
- sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
- fi\n\
- unset WORKERPASS\n\
- export PATH=/usr/lib64/ccache:$PATH\n\
- mkdir -p shared/$image_name/ccache\n\
- export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
- buildbot-worker start --nodaemon $worker_dir\n' \
- > bb-start.sh && \
- chmod 755 bb-start.sh
+ENV IMAGE_NAME=fedora-latest
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+COPY bb-start.sh /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/Containerfile-opensuse-leap b/builder/containers/Containerfile-opensuse-leap
index 51e74c4..197f1af 100644
--- a/builder/containers/Containerfile-opensuse-leap
+++ b/builder/containers/Containerfile-opensuse-leap
@@ -25,28 +25,9 @@ RUN pip install buildbot-worker
# Setup user with same id as host user id.
RUN useradd -u 1001 -U -d /home/builder -m builder
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
- echo $'image_name=opensuse-leap\n\
- worker_dir=shared/$image_name/worker\n\
- tac_file=$worker_dir/buildbot.tac\n\
- if [ ! -f $tac_file ]; then\n\
- mkdir -p $worker_dir\n\
- buildbot-worker create-worker $worker_dir \
- $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
- echo buildbot@sourceware.org > $worker_dir/info/admin\n\
- echo $image_name > $worker_dir/info/host\n\
- else\n\
- sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
- fi\n\
- unset WORKERPASS\n\
- export PATH=/usr/lib64/ccache:$PATH\n\
- mkdir -p shared/$image_name/ccache\n\
- export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
- buildbot-worker start --nodaemon $worker_dir\n' \
- > bb-start.sh && \
- chmod 755 bb-start.sh
+ENV IMAGE_NAME=opensuse-leap
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+COPY bb-start.sh /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/Containerfile-opensuse-tumbleweed b/builder/containers/Containerfile-opensuse-tumbleweed
index d20ad84..d8157ed 100644
--- a/builder/containers/Containerfile-opensuse-tumbleweed
+++ b/builder/containers/Containerfile-opensuse-tumbleweed
@@ -21,28 +21,9 @@ RUN pip install buildbot-worker
# Setup user with same id as host user id.
RUN useradd -u 1001 -U -d /home/builder -m builder
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
- echo $'image_name=opensuse-tumbleweed\n\
- worker_dir=shared/$image_name/worker\n\
- tac_file=$worker_dir/buildbot.tac\n\
- if [ ! -f $tac_file ]; then\n\
- mkdir -p $worker_dir\n\
- buildbot-worker create-worker $worker_dir \
- $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
- echo buildbot@sourceware.org > $worker_dir/info/admin\n\
- echo $image_name > $worker_dir/info/host\n\
- else\n\
- sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
- fi\n\
- unset WORKERPASS\n\
- export PATH=/usr/lib64/ccache:$PATH\n\
- mkdir -p shared/$image_name/ccache\n\
- export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
- buildbot-worker start --nodaemon $worker_dir\n' \
- > bb-start.sh && \
- chmod 755 bb-start.sh
+ENV IMAGE_NAME=opensuse-tumbleweed
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+COPY bb-start.sh /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/bb-start.sh b/builder/containers/bb-start.sh
new file mode 100755
index 0000000..f8203ef
--- /dev/null
+++ b/builder/containers/bb-start.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+
+# The script is entrypoint for container workers.
+# The following environment variables must by set
+# by the corresponding Container files:
+#
+# - IMAGE_NAME - name of the image
+# - CCACHE_LIBDIR - path where to search for ccache (usually /usr/lib64/ccache)
+
+worker_dir=shared/$IMAGE_NAME/worker
+tac_file=$worker_dir/buildbot.tac
+if [ ! -f $tac_file ]; then
+ mkdir -p $worker_dir
+ buildbot-worker create-worker $worker_dir $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS
+ echo buildbot@sourceware.org > $worker_dir/info/admin
+ echo $IMAGE_NAME > $worker_dir/info/host
+else
+ sed -ie "s/^passwd = .*/passwd = '$WORKERPASS'/" $tac_file
+fi
+unset WORKERPASS
+export PATH=$CCACHE_LIBDIR:$PATH
+mkdir -p shared/$IMAGE_NAME/ccache
+export CCACHE_DIR=/home/builder/shared/$IMAGE_NAME/ccache
+buildbot-worker start --nodaemon $worker_dir
--
2.36.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it
2022-06-17 9:17 [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it Martin Liška
@ 2022-06-18 15:47 ` Mark Wielaard
2022-06-18 18:35 ` Martin Liška
0 siblings, 1 reply; 6+ messages in thread
From: Mark Wielaard @ 2022-06-18 15:47 UTC (permalink / raw)
To: Martin Liška; +Cc: buildbot
[-- Attachment #1: Type: text/plain, Size: 970 bytes --]
Hi Martin,
On Fri, Jun 17, 2022 at 11:17:28AM +0200, Martin Liška wrote:
> Doing echo "..." > bb-start.sh adds one more level of escaping
> and it is quite ugly. Use rather a proper file and discard all
> the Debian switch to bash and back.
>
> Mark can you please test it if it works with the newly set 2 ENV variables:
> ENV IMAGE_NAME=debian-stable
> ENV CCACHE_LIBDIR=/usr/lib/ccache
I like this. But we cannot simply COPY in the bb-start.sh file.
Because it isn't available on the container host. The
DockerLatentWorker only sents the Container file. But we could fetch
it from the git repo using:
RUN wget -O /home/builder/bb-start.sh \
'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
&& chmod 755 /home/builder/bb-start.sh
We could also git clone the whole builder.git repo and then copy the
bb-start.sh file from the checkout, but this seems simpler.
If this works for you, please commit.
Thanks,
Mark
[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 8727 bytes --]
diff --git a/builder/containers/Containerfile-debian-stable b/builder/containers/Containerfile-debian-stable
index e7cd134..d69f725 100644
--- a/builder/containers/Containerfile-debian-stable
+++ b/builder/containers/Containerfile-debian-stable
@@ -22,34 +22,14 @@ RUN apt-get update && \
# Setup user with same id as host user id.
RUN adduser --home /home/builder --uid 1001 builder
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-# This script needs bash, so set sh to bash and then back...
-# Yes, this creates 3 layers... sigh
-RUN echo "dash dash/sh boolean false" | debconf-set-selections; \
- DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
-RUN cd /home/builder; \
- echo $'image_name=debian-stable\n\
- worker_dir=shared/$image_name/worker\n\
- tac_file=$worker_dir/buildbot.tac\n\
- if [ ! -f $tac_file ]; then\n\
- mkdir -p $worker_dir\n\
- buildbot-worker create-worker $worker_dir \
- $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
- echo buildbot@sourceware.org > $worker_dir/info/admin\n\
- echo $image_name > $worker_dir/info/host\n\
- else\n\
- sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
- fi\n\
- unset WORKERPASS\n\
- export PATH=/usr/lib/ccache:$PATH\n\
- mkdir -p shared/$image_name/ccache\n\
- export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
- buildbot-worker start --nodaemon $worker_dir\n' \
- > bb-start.sh && \
- chmod 755 bb-start.sh
-RUN echo "dash dash/sh boolean true" | debconf-set-selections; \
- DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash;
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=debian-stable
+ENV CCACHE_LIBDIR=/usr/lib/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+ 'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+ && chmod 755 /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
index ce3b07e..057f27d 100644
--- a/builder/containers/Containerfile-fedora-latest
+++ b/builder/containers/Containerfile-fedora-latest
@@ -19,28 +19,14 @@ RUN dnf upgrade -y && \
# Setup user with same id as host user id.
RUN adduser -u 1001 -U -d /home/builder -m builder
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
- echo $'image_name=fedora-latest\n\
- worker_dir=shared/$image_name/worker\n\
- tac_file=$worker_dir/buildbot.tac\n\
- if [ ! -f $tac_file ]; then\n\
- mkdir -p $worker_dir\n\
- buildbot-worker create-worker $worker_dir \
- $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
- echo buildbot@sourceware.org > $worker_dir/info/admin\n\
- echo $image_name > $worker_dir/info/host\n\
- else\n\
- sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
- fi\n\
- unset WORKERPASS\n\
- export PATH=/usr/lib64/ccache:$PATH\n\
- mkdir -p shared/$image_name/ccache\n\
- export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
- buildbot-worker start --nodaemon $worker_dir\n' \
- > bb-start.sh && \
- chmod 755 bb-start.sh
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=fedora-latest
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+ 'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+ && chmod 755 /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/Containerfile-opensuse-leap b/builder/containers/Containerfile-opensuse-leap
index 51e74c4..6299e9b 100644
--- a/builder/containers/Containerfile-opensuse-leap
+++ b/builder/containers/Containerfile-opensuse-leap
@@ -25,28 +25,14 @@ RUN pip install buildbot-worker
# Setup user with same id as host user id.
RUN useradd -u 1001 -U -d /home/builder -m builder
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
- echo $'image_name=opensuse-leap\n\
- worker_dir=shared/$image_name/worker\n\
- tac_file=$worker_dir/buildbot.tac\n\
- if [ ! -f $tac_file ]; then\n\
- mkdir -p $worker_dir\n\
- buildbot-worker create-worker $worker_dir \
- $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
- echo buildbot@sourceware.org > $worker_dir/info/admin\n\
- echo $image_name > $worker_dir/info/host\n\
- else\n\
- sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
- fi\n\
- unset WORKERPASS\n\
- export PATH=/usr/lib64/ccache:$PATH\n\
- mkdir -p shared/$image_name/ccache\n\
- export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
- buildbot-worker start --nodaemon $worker_dir\n' \
- > bb-start.sh && \
- chmod 755 bb-start.sh
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=opensuse-leap
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+ 'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+ && chmod 755 /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/Containerfile-opensuse-tumbleweed b/builder/containers/Containerfile-opensuse-tumbleweed
index d20ad84..d574767 100644
--- a/builder/containers/Containerfile-opensuse-tumbleweed
+++ b/builder/containers/Containerfile-opensuse-tumbleweed
@@ -21,28 +21,14 @@ RUN pip install buildbot-worker
# Setup user with same id as host user id.
RUN useradd -u 1001 -U -d /home/builder -m builder
-# Create buildbot-worker script which sets up shared worker and ccache
-# data with the host. Each run gets a new password, so update that too.
-RUN cd /home/builder; \
- echo $'image_name=opensuse-tumbleweed\n\
- worker_dir=shared/$image_name/worker\n\
- tac_file=$worker_dir/buildbot.tac\n\
- if [ ! -f $tac_file ]; then\n\
- mkdir -p $worker_dir\n\
- buildbot-worker create-worker $worker_dir \
- $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS\n\
- echo buildbot@sourceware.org > $worker_dir/info/admin\n\
- echo $image_name > $worker_dir/info/host\n\
- else\n\
- sed -ie \"s/^passwd = .*/passwd = \'$WORKERPASS\'/\" $tac_file\n\
- fi\n\
- unset WORKERPASS\n\
- export PATH=/usr/lib64/ccache:$PATH\n\
- mkdir -p shared/$image_name/ccache\n\
- export CCACHE_DIR=/home/builder/shared/$image_name/ccache\n\
- buildbot-worker start --nodaemon $worker_dir\n' \
- > bb-start.sh && \
- chmod 755 bb-start.sh
+# Set image specific environment variables used by bb-start.sh
+ENV IMAGE_NAME=opensuse-tumbleweed
+ENV CCACHE_LIBDIR=/usr/lib64/ccache
+
+# Put bb-start.sh script in homedir and make it executable
+RUN wget -O /home/builder/bb-start.sh \
+ 'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
+ && chmod 755 /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/bb-start.sh b/builder/containers/bb-start.sh
new file mode 100755
index 0000000..f8203ef
--- /dev/null
+++ b/builder/containers/bb-start.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+
+# The script is entrypoint for container workers.
+# The following environment variables must by set
+# by the corresponding Container files:
+#
+# - IMAGE_NAME - name of the image
+# - CCACHE_LIBDIR - path where to search for ccache (usually /usr/lib64/ccache)
+
+worker_dir=shared/$IMAGE_NAME/worker
+tac_file=$worker_dir/buildbot.tac
+if [ ! -f $tac_file ]; then
+ mkdir -p $worker_dir
+ buildbot-worker create-worker $worker_dir $BUILDMASTER:$BUILDMASTER_PORT $WORKERNAME $WORKERPASS
+ echo buildbot@sourceware.org > $worker_dir/info/admin
+ echo $IMAGE_NAME > $worker_dir/info/host
+else
+ sed -ie "s/^passwd = .*/passwd = '$WORKERPASS'/" $tac_file
+fi
+unset WORKERPASS
+export PATH=$CCACHE_LIBDIR:$PATH
+mkdir -p shared/$IMAGE_NAME/ccache
+export CCACHE_DIR=/home/builder/shared/$IMAGE_NAME/ccache
+buildbot-worker start --nodaemon $worker_dir
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it
2022-06-18 15:47 ` Mark Wielaard
@ 2022-06-18 18:35 ` Martin Liška
2022-06-18 18:38 ` Martin Liška
2022-06-18 20:11 ` Mark Wielaard
0 siblings, 2 replies; 6+ messages in thread
From: Martin Liška @ 2022-06-18 18:35 UTC (permalink / raw)
To: Mark Wielaard; +Cc: buildbot
On 6/18/22 17:47, Mark Wielaard wrote:
> Hi Martin,
>
> On Fri, Jun 17, 2022 at 11:17:28AM +0200, Martin Liška wrote:
>> Doing echo "..." > bb-start.sh adds one more level of escaping
>> and it is quite ugly. Use rather a proper file and discard all
>> the Debian switch to bash and back.
>>
>> Mark can you please test it if it works with the newly set 2 ENV variables:
>> ENV IMAGE_NAME=debian-stable
>> ENV CCACHE_LIBDIR=/usr/lib/ccache
>
> I like this. But we cannot simply COPY in the bb-start.sh file.
> Because it isn't available on the container host. The
> DockerLatentWorker only sents the Container file. But we could fetch
> it from the git repo using:
>
> RUN wget -O /home/builder/bb-start.sh \
> 'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
> && chmod 755 /home/builder/bb-start.sh
Good, I've done that and pushed to main.
However, I see an error now:
https://builder.sourceware.org/buildbot/#/builders/97/builds/248
Can you please check buildbot master log files in order to investigate what's wrong?
Thanks,
Martin
>
> We could also git clone the whole builder.git repo and then copy the
> bb-start.sh file from the checkout, but this seems simpler.
>
> If this works for you, please commit.
>
> Thanks,
>
> Mark
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it
2022-06-18 18:35 ` Martin Liška
@ 2022-06-18 18:38 ` Martin Liška
2022-06-18 20:11 ` Mark Wielaard
1 sibling, 0 replies; 6+ messages in thread
From: Martin Liška @ 2022-06-18 18:38 UTC (permalink / raw)
To: Mark Wielaard; +Cc: buildbot
On 6/18/22 20:35, Martin Liška wrote:
> However, I see an error now:
> https://builder.sourceware.org/buildbot/#/builders/97/builds/248
Well, the error message is:
buildbot.interfaces.LatentWorkerCannotSubstantiate: Image "bb1_139886867151000_image" not found on docker host.
and it happens when I re-run a build.
So it likely means one can't re-run such a build that uses LatentWorkers which
images are rebuilt time to time.
Cheers,
Martin
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it
2022-06-18 18:35 ` Martin Liška
2022-06-18 18:38 ` Martin Liška
@ 2022-06-18 20:11 ` Mark Wielaard
2022-06-19 7:12 ` Martin Liška
1 sibling, 1 reply; 6+ messages in thread
From: Mark Wielaard @ 2022-06-18 20:11 UTC (permalink / raw)
To: Martin Liška; +Cc: buildbot
[-- Attachment #1: Type: text/plain, Size: 901 bytes --]
Hi Martin,
On Sat, Jun 18, 2022 at 08:35:06PM +0200, Martin Liška wrote:
> > I like this. But we cannot simply COPY in the bb-start.sh file.
> > Because it isn't available on the container host. The
> > DockerLatentWorker only sents the Container file. But we could fetch
> > it from the git repo using:
> >
> > RUN wget -O /home/builder/bb-start.sh \
> > 'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
> > && chmod 755 /home/builder/bb-start.sh
>
> Good, I've done that and pushed to main.
>
> However, I see an error now:
> https://builder.sourceware.org/buildbot/#/builders/97/builds/248
>
> Can you please check buildbot master log files in order to investigate what's wrong?
The issue was that there were now both a RUN wget && chmod and a COPY
in the Container files. Removing the COPY seems to have made it work again.
Cheers,
Mark
[-- Attachment #2: 0001-Containerfiles-Just-RUN-wget-chmod-don-t-also-COPY-b.patch --]
[-- Type: text/x-diff, Size: 3985 bytes --]
From 35bafdb6c93177ca33ef99b394b0207b14334da9 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 18 Jun 2022 21:32:25 +0200
Subject: [PATCH] Containerfiles: Just RUN wget && chmod, don't also COPY
bb-start.sh
---
builder/containers/Containerfile-debian-stable | 3 ++-
builder/containers/Containerfile-fedora-latest | 3 ++-
builder/containers/Containerfile-opensuse-leap | 3 ++-
builder/containers/Containerfile-opensuse-tumbleweed | 3 ++-
4 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/builder/containers/Containerfile-debian-stable b/builder/containers/Containerfile-debian-stable
index 6776c3b..d69f725 100644
--- a/builder/containers/Containerfile-debian-stable
+++ b/builder/containers/Containerfile-debian-stable
@@ -22,13 +22,14 @@ RUN apt-get update && \
# Setup user with same id as host user id.
RUN adduser --home /home/builder --uid 1001 builder
+# Set image specific environment variables used by bb-start.sh
ENV IMAGE_NAME=debian-stable
ENV CCACHE_LIBDIR=/usr/lib/ccache
+# Put bb-start.sh script in homedir and make it executable
RUN wget -O /home/builder/bb-start.sh \
'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
&& chmod 755 /home/builder/bb-start.sh
-COPY bb-start.sh /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
index 556a959..057f27d 100644
--- a/builder/containers/Containerfile-fedora-latest
+++ b/builder/containers/Containerfile-fedora-latest
@@ -19,13 +19,14 @@ RUN dnf upgrade -y && \
# Setup user with same id as host user id.
RUN adduser -u 1001 -U -d /home/builder -m builder
+# Set image specific environment variables used by bb-start.sh
ENV IMAGE_NAME=fedora-latest
ENV CCACHE_LIBDIR=/usr/lib64/ccache
+# Put bb-start.sh script in homedir and make it executable
RUN wget -O /home/builder/bb-start.sh \
'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
&& chmod 755 /home/builder/bb-start.sh
-COPY bb-start.sh /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/Containerfile-opensuse-leap b/builder/containers/Containerfile-opensuse-leap
index 4e88f8c..6299e9b 100644
--- a/builder/containers/Containerfile-opensuse-leap
+++ b/builder/containers/Containerfile-opensuse-leap
@@ -25,13 +25,14 @@ RUN pip install buildbot-worker
# Setup user with same id as host user id.
RUN useradd -u 1001 -U -d /home/builder -m builder
+# Set image specific environment variables used by bb-start.sh
ENV IMAGE_NAME=opensuse-leap
ENV CCACHE_LIBDIR=/usr/lib64/ccache
+# Put bb-start.sh script in homedir and make it executable
RUN wget -O /home/builder/bb-start.sh \
'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
&& chmod 755 /home/builder/bb-start.sh
-COPY bb-start.sh /home/builder/bb-start.sh
# And now run the script
USER builder
diff --git a/builder/containers/Containerfile-opensuse-tumbleweed b/builder/containers/Containerfile-opensuse-tumbleweed
index 002291a..d574767 100644
--- a/builder/containers/Containerfile-opensuse-tumbleweed
+++ b/builder/containers/Containerfile-opensuse-tumbleweed
@@ -21,13 +21,14 @@ RUN pip install buildbot-worker
# Setup user with same id as host user id.
RUN useradd -u 1001 -U -d /home/builder -m builder
+# Set image specific environment variables used by bb-start.sh
ENV IMAGE_NAME=opensuse-tumbleweed
ENV CCACHE_LIBDIR=/usr/lib64/ccache
+# Put bb-start.sh script in homedir and make it executable
RUN wget -O /home/builder/bb-start.sh \
'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
&& chmod 755 /home/builder/bb-start.sh
-COPY bb-start.sh /home/builder/bb-start.sh
# And now run the script
USER builder
--
2.30.2
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it
2022-06-18 20:11 ` Mark Wielaard
@ 2022-06-19 7:12 ` Martin Liška
0 siblings, 0 replies; 6+ messages in thread
From: Martin Liška @ 2022-06-19 7:12 UTC (permalink / raw)
To: Mark Wielaard; +Cc: buildbot
On 6/18/22 22:11, Mark Wielaard wrote:
> Hi Martin,
>
> On Sat, Jun 18, 2022 at 08:35:06PM +0200, Martin Liška wrote:
>>> I like this. But we cannot simply COPY in the bb-start.sh file.
>>> Because it isn't available on the container host. The
>>> DockerLatentWorker only sents the Container file. But we could fetch
>>> it from the git repo using:
>>>
>>> RUN wget -O /home/builder/bb-start.sh \
>>> 'https://sourceware.org/git/?p=builder.git;a=blob_plain;f=builder/containers/bb-start.sh' \
>>> && chmod 755 /home/builder/bb-start.sh
>>
>> Good, I've done that and pushed to main.
>>
>> However, I see an error now:
>> https://builder.sourceware.org/buildbot/#/builders/97/builds/248
>>
>> Can you please check buildbot master log files in order to investigate what's wrong?
>
> The issue was that there were now both a RUN wget && chmod and a COPY
> in the Container files. Removing the COPY seems to have made it work again.
Oh, what a stupid mistake I've made.
Thanks for the fix!
Martin
>
> Cheers,
>
> Mark
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-06-19 7:12 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-17 9:17 [RFC,PATCH] containers: extract bb-start.sh to a file and reuse it Martin Liška
2022-06-18 15:47 ` Mark Wielaard
2022-06-18 18:35 ` Martin Liška
2022-06-18 18:38 ` Martin Liška
2022-06-18 20:11 ` Mark Wielaard
2022-06-19 7:12 ` Martin Liška
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).