Buildbot failure in Wildebeest Builder on whole buildset

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building dwz.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/23/builds/311

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Tom de Vries <tdevries@suse.de>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Fwd: Buildbot failure in Wildebeest Builder on whole buildset

[Tom de Vries]

Hi Mark,

I've just committed patch "Move hardlink handling out of dwz function"
and there's this buildbot failure.

Is there something easy that you can do to find out if this is a fluke
or not, f.i. retry the build for that bot?

Thanks,
- Tom


-------- Forwarded Message --------
Subject: Buildbot failure in Wildebeest Builder on whole buildset
Date: Thu, 25 Mar 2021 08:41:56 +0000
From: buildbot@builder.wildebeest.org
To: dwz@sourceware.org

The Buildbot has detected a failed build on builder whole buildset while
building dwz.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/23/builds/311

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Tom de Vries <tdevries@suse.de>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Fwd: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi Tom,

On Thu, 2021-03-25 at 09:50 +0100, Tom de Vries wrote:
> I've just committed patch "Move hardlink handling out of dwz
> function"
> and there's this buildbot failure.
> 
> Is there something easy that you can do to find out if this is a
> fluke
> or not, f.i. retry the build for that bot?

I logged into the buildbot worker and did a dwz build myself.
make check does produce 32 unexpected failures.
Removing the last commit makes everything pass.
Adding the commit again produces the failures again.
It isn't a fluke.

The problem is simply that:
cp hello 1
dwz 1

Makes dwz produce an exit code of 1 which seems to be interpreted as
child process exited abnormally in a make check run.

running under valgrind --track-origins=yes gives:

==31313== Conditional jump or move depends on uninitialised value(s)
==31313==    at 0x127974: dwz (dwz.c:15287)
==31313==    by 0x10BB7C: dwz_with_low_mem (dwz.c:16227)
==31313==    by 0x10BB7C: dwz_one_file (dwz.c:16253)
==31313==    by 0x10BB7C: main (dwz.c:16532)
==31313==  Uninitialised value was created by a stack allocation
==31313==    at 0x10B59B: main (dwz.c:16513)

I haven't tracked down what res->res precisely depends on that doesn't
have a defined value.

Cheers,

Mark

Re: Fwd: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi Tom,

On Thu, 2021-03-25 at 11:41 +0100, Mark Wielaard wrote:
> On Thu, 2021-03-25 at 09:50 +0100, Tom de Vries wrote:
> > I've just committed patch "Move hardlink handling out of dwz
> > function"
> > and there's this buildbot failure.
> > 
> > Is there something easy that you can do to find out if this is a
> > fluke
> > or not, f.i. retry the build for that bot?
> 
> I logged into the buildbot worker and did a dwz build myself.
> make check does produce 32 unexpected failures.
> Removing the last commit makes everything pass.
> Adding the commit again produces the failures again.
> It isn't a fluke.
> 
> The problem is simply that:
> cp hello 1
> dwz 1
> 
> Makes dwz produce an exit code of 1 which seems to be interpreted as
> child process exited abnormally in a make check run.
> 
> running under valgrind --track-origins=yes gives:
> 
> ==31313== Conditional jump or move depends on uninitialised value(s)
> ==31313==    at 0x127974: dwz (dwz.c:15287)
> ==31313==    by 0x10BB7C: dwz_with_low_mem (dwz.c:16227)
> ==31313==    by 0x10BB7C: dwz_one_file (dwz.c:16253)
> ==31313==    by 0x10BB7C: main (dwz.c:16532)
> ==31313==  Uninitialised value was created by a stack allocation
> ==31313==    at 0x10B59B: main (dwz.c:16513)
> 
> I haven't tracked down what res->res precisely depends on that
> doesn't
> have a defined value.

Note that dwz_one_file was inlined into main, causing valgrind to
pinpoint the wrong frame. Without inlining we get:

==2502== Conditional jump or move depends on uninitialised value(s)
==2502==    at 0x13CF4F: dwz (dwz.c:15287)
==2502==    by 0x13FE3A: dwz_with_low_mem (dwz.c:16227)
==2502==    by 0x13FEAE: dwz_one_file (dwz.c:16253)
==2502==    by 0x140B59: main (dwz.c:16532)
==2502==  Uninitialised value was created by a stack allocation
==2502==    at 0x13FE7F: dwz_one_file (dwz.c:16245)

And indeed we see:

static int
dwz_one_file (const char *file, const char *outfile)
{
  struct file_result res;

  if (stats_p)
    init_stats (file);

  res.die_count = 0;

  return dwz_with_low_mem (file, outfile, &res, NULL);
}

Defines and passes down res, but only initializes the die_count field.

Then in dwz_with_low_mem we simply pass that res to dwz:

  ret = (low_mem_die_limit == 0
         ? 2
         : dwz (file, outfile, res));

And in dwz the first thing we do is:

  if (res->res == -1)
    return 1;

But res->res was never given a value and so could be anything.

Cheers,

Mark

Re: Fwd: Buildbot failure in Wildebeest Builder on whole buildset

[Tom de Vries]

On 3/25/21 11:49 AM, Mark Wielaard wrote:
> Hi Tom,
> 
> On Thu, 2021-03-25 at 11:41 +0100, Mark Wielaard wrote:
>> On Thu, 2021-03-25 at 09:50 +0100, Tom de Vries wrote:
>>> I've just committed patch "Move hardlink handling out of dwz
>>> function"
>>> and there's this buildbot failure.
>>>
>>> Is there something easy that you can do to find out if this is a
>>> fluke
>>> or not, f.i. retry the build for that bot?
>>
>> I logged into the buildbot worker and did a dwz build myself.
>> make check does produce 32 unexpected failures.
>> Removing the last commit makes everything pass.
>> Adding the commit again produces the failures again.
>> It isn't a fluke.
>>
>> The problem is simply that:
>> cp hello 1
>> dwz 1
>>
>> Makes dwz produce an exit code of 1 which seems to be interpreted as
>> child process exited abnormally in a make check run.
>>
>> running under valgrind --track-origins=yes gives:
>>
>> ==31313== Conditional jump or move depends on uninitialised value(s)
>> ==31313==    at 0x127974: dwz (dwz.c:15287)
>> ==31313==    by 0x10BB7C: dwz_with_low_mem (dwz.c:16227)
>> ==31313==    by 0x10BB7C: dwz_one_file (dwz.c:16253)
>> ==31313==    by 0x10BB7C: main (dwz.c:16532)
>> ==31313==  Uninitialised value was created by a stack allocation
>> ==31313==    at 0x10B59B: main (dwz.c:16513)
>>
>> I haven't tracked down what res->res precisely depends on that
>> doesn't
>> have a defined value.
> 
> Note that dwz_one_file was inlined into main, causing valgrind to
> pinpoint the wrong frame. Without inlining we get:
> 
> ==2502== Conditional jump or move depends on uninitialised value(s)
> ==2502==    at 0x13CF4F: dwz (dwz.c:15287)
> ==2502==    by 0x13FE3A: dwz_with_low_mem (dwz.c:16227)
> ==2502==    by 0x13FEAE: dwz_one_file (dwz.c:16253)
> ==2502==    by 0x140B59: main (dwz.c:16532)
> ==2502==  Uninitialised value was created by a stack allocation
> ==2502==    at 0x13FE7F: dwz_one_file (dwz.c:16245)
> 
> And indeed we see:
> 
> static int
> dwz_one_file (const char *file, const char *outfile)
> {
>   struct file_result res;
> 
>   if (stats_p)
>     init_stats (file);
> 
>   res.die_count = 0;
> 
>   return dwz_with_low_mem (file, outfile, &res, NULL);
> }
> 
> Defines and passes down res, but only initializes the die_count field.
> 
> Then in dwz_with_low_mem we simply pass that res to dwz:
> 
>   ret = (low_mem_die_limit == 0
>          ? 2
>          : dwz (file, outfile, res));
> 
> And in dwz the first thing we do is:
> 
>   if (res->res == -1)
>     return 1;
> 
> But res->res was never given a value and so could be anything.
> 

Hi Mark,

thanks for helping out with this.

I've fixed this now, and verified that the buildbot is back to all-green.

After seeing your analysis, I realized I had tried -fsanitize=address
but not valgrind, and indeed using valgrind I managed to reproduce the
problem myself.

I've submitted a patch proposing a check-valgrind target in the
Makefile, to remind me to try this next time.

Thanks,
- Tom