Strange mkdirs issue on power8

Strange mkdirs issue on power8

[Mark Wielaard]

Hi Will,

There is a strange update error on the power8 worker.
See https://builder.sourceware.org/buildbot/#/workers/18

The issue is this code that tries to create a working directory before
running the git update command:

        if not os.path.exists(workdir):
            os.makedirs(workdir)

The apparently os.path.exists returns false, but then os.makedirs
raises a FileExistsError.  Which doesn't really make sense.

Might there be something odd with the filesystem?

Confused,

Mark

Re: Strange mkdirs issue on power8

[Mark Wielaard]

Hi Will,

On Tue, 2022-11-22 at 23:48 +0100, Mark Wielaard wrote:
> There is a strange update error on the power8 worker.
> See https://builder.sourceware.org/buildbot/#/workers/18

The worker disconnected and didn't return. I have canceled pending
builds. Could you take a look?

Thanks,

Mark

Re: Strange mkdirs issue on power8

[will schmidt]

On Mon, 2022-11-28 at 11:51 +0100, Mark Wielaard wrote:
> Hi Will,
> 
> On Tue, 2022-11-22 at 23:48 +0100, Mark Wielaard wrote:
> > There is a strange update error on the power8 worker.
> > See https://builder.sourceware.org/buildbot/#/workers/18
> 
> The worker disconnected and didn't return. I have canceled pending
> builds. Could you take a look?
> 


Yeah, I'm not sure what has occurred.  I have a question in to the system admin
to see whats going on.

Thanks
-Will



> Thanks,
> 
> Mark

Re: Strange mkdirs issue on power8

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

Hi Will,

On Mon, 2022-11-28 at 16:37 -0600, will schmidt wrote:
> On Mon, 2022-11-28 at 11:51 +0100, Mark Wielaard wrote:
> > On Tue, 2022-11-22 at 23:48 +0100, Mark Wielaard wrote:
> > > There is a strange update error on the power8 worker.
> > > See https://builder.sourceware.org/buildbot/#/workers/18
> > 
> > The worker disconnected and didn't return. I have canceled pending
> > builds. Could you take a look?
> > 
> Yeah, I'm not sure what has occurred.  I have a question in to the
> system admin
> to see whats going on.

Since the builder doesn't seem to reconnect anymore I have cancelled
all pending builds and disabled the builders using it for now.

Please let me know if it gets back so we can enable the builds again.

Thanks,

Mark

[-- Attachment #2: 0001-Disable-gdb-valgrind-ibm-power8-builders-for-now.patch --]
[-- Type: text/x-patch, Size: 2888 bytes --]

From 4a66ce85c9589a995cf768213781d979201fca65 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Tue, 6 Dec 2022 15:48:25 +0100
Subject: [PATCH] Disable {gdb,valgrind}-ibm-power8 builders for now

---
 builder/master.cfg | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/builder/master.cfg b/builder/master.cfg
index 7ddc981..51a0db3 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -591,7 +591,7 @@ valgrind_scheduler = schedulers.SingleBranchScheduler(
                       "valgrind-debian-testing-x86_64",
                       "valgrind-debian-armhf",
                       "valgrind-debian-i386",
-                      "valgrind-ibm-power8",
+#                      "valgrind-ibm-power8",
                       "valgrind-ibm-power9",
                       "valgrind-ibm-power10",
                       "valgrind-fedora-ppc64le",
@@ -769,7 +769,7 @@ gdb_scheduler = schedulers.SingleBranchScheduler(
                       "gdb-armhf-ubuntu20_04",
                       "gdb-armhf-ubuntu22_04",
                       "gdb-debian-i386",
-                      "gdb-ibm-power8",
+#                      "gdb-ibm-power8",
                       "gdb-ibm-power9",
                       "gdb-ibm-power10",
                       "gdb-fedora-s390x",
@@ -2723,12 +2723,12 @@ gdb_try_debian_testing_x86_64_builder = util.BuilderConfig(
         factory=gdb_factory)
 c['builders'].append(gdb_try_debian_testing_x86_64_builder)
 
-gdb_ibm_power8_builder = util.BuilderConfig(
-	name="gdb-ibm-power8",
-        workernames=["ibm_power8"],
-        tags=["gdb", "ibm", "power8"],
-        factory=gdb_factory)
-c['builders'].append(gdb_ibm_power8_builder)
+# gdb_ibm_power8_builder = util.BuilderConfig(
+# 	name="gdb-ibm-power8",
+#         workernames=["ibm_power8"],
+#         tags=["gdb", "ibm", "power8"],
+#         factory=gdb_factory)
+# c['builders'].append(gdb_ibm_power8_builder)
 
 gdb_ibm_power9_builder = util.BuilderConfig(
 	name="gdb-ibm-power9",
@@ -3644,12 +3644,12 @@ valgrind_fedora_ppc64le_builder = util.BuilderConfig(
         factory=valgrind_make_check_aux_factory)
 c['builders'].append(valgrind_fedora_ppc64le_builder)
 
-valgrind_ibm_power8_builder = util.BuilderConfig(
-        name="valgrind-ibm-power8",
-        workernames=["ibm_power8"],
-        tags=["valgrind", "ibm", "power8"],
-        factory=valgrind_make_check_aux_factory)
-c['builders'].append(valgrind_ibm_power8_builder)
+# valgrind_ibm_power8_builder = util.BuilderConfig(
+#         name="valgrind-ibm-power8",
+#         workernames=["ibm_power8"],
+#         tags=["valgrind", "ibm", "power8"],
+#         factory=valgrind_make_check_aux_factory)
+# c['builders'].append(valgrind_ibm_power8_builder)
 
 valgrind_ibm_power9_builder = util.BuilderConfig(
         name="valgrind-ibm-power9",
-- 
2.18.4

Re: Strange mkdirs issue on power8

[will schmidt]

On Tue, 2022-12-06 at 15:51 +0100, Mark Wielaard wrote:
> Hi Will,
> 
> On Mon, 2022-11-28 at 16:37 -0600, will schmidt wrote:
> > On Mon, 2022-11-28 at 11:51 +0100, Mark Wielaard wrote:
> > > On Tue, 2022-11-22 at 23:48 +0100, Mark Wielaard wrote:
> > > > There is a strange update error on the power8 worker.
> > > > See https://builder.sourceware.org/buildbot/#/workers/18
> > > 
> > > The worker disconnected and didn't return. I have canceled pending
> > > builds. Could you take a look?
> > > 
> > Yeah, I'm not sure what has occurred.  I have a question in to the
> > system admin
> > to see whats going on.
> 
> Since the builder doesn't seem to reconnect anymore I have cancelled
> all pending builds and disabled the builders using it for now.
> 
> Please let me know if it gets back so we can enable the builds again.

Yup, a drive failure on the host,..   I'll get back to you. 

> 
> Thanks,
> 
> Mark