[Bug libgomp/111024] New: libgomp: FAILs with oldish libnuma/libmemkind

[Bug libgomp/111024] New: libgomp: FAILs with oldish libnuma/libmemkind

[tschwinge at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111024

            Bug ID: 111024
           Summary: libgomp: FAILs with oldish libnuma/libmemkind
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Keywords: openmp
          Severity: normal
          Priority: P3
         Component: libgomp
          Assignee: unassigned at gcc dot gnu.org
          Reporter: tschwinge at gcc dot gnu.org
                CC: burnus at gcc dot gnu.org, jakub at gcc dot gnu.org
  Target Milestone: ---

Re commit r14-2462-g450b05ce54d3f08c583c3b5341233ce0df99725b "libgomp: Use
libnuma for OpenMP's partition=nearest allocation trait" (plus commit
r14-2514-g407d68daed00e040a7d9545b2a18aa27bf93a106 "libgomp: Fix allocator
handling for Linux when libnuma is not available"), on our amdfury2, nvidia-4a
systems (not necessarily an exhaustive list), I see:

    +PASS: libgomp.c/../libgomp.c-c++-common/alloc-11.c (test for excess
errors)
    +FAIL: libgomp.c/../libgomp.c-c++-common/alloc-11.c execution test

    alloc-11.exe: src/memkind_interleave.c:54: memkind_interleave_init_once:
Assertion `err == 0' failed.

    +PASS: libgomp.c/../libgomp.c-c++-common/alloc-12.c (test for excess
errors)
    +FAIL: libgomp.c/../libgomp.c-c++-common/alloc-12.c execution test

    alloc-12.exe: src/memkind_interleave.c:54: memkind_interleave_init_once:
Assertion `err == 0' failed.

Same for C++.

    (gdb) bt
    #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
    #1  0x00007ffff75b2921 in __GI_abort () at abort.c:79
    #2  0x00007ffff75a248a in __assert_fail_base (fmt=0x7ffff7729750
"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x7ffff7169682 "err == 0",
file=file@entry=0x7ffff716986a "src/memkind_interleave.c", line=line@entry=54,
function=function@entry=0x7ffff7169890 "memkind_interleave_init_once") at
assert.c:92
    #3  0x00007ffff75a2502 in __GI___assert_fail (assertion=0x7ffff7169682 "err
== 0", file=0x7ffff716986a "src/memkind_interleave.c", line=54,
function=0x7ffff7169890 "memkind_interleave_init_once") at assert.c:101
    #4  0x00007ffff71691c4 in memkind_interleave_init_once () from
/usr/lib/x86_64-linux-gnu/libmemkind.so.0
    #5  0x00007ffff7972907 in __pthread_once_slow (once_control=0x7ffff736d72c,
init_routine=0x7ffff7169180 <memkind_interleave_init_once>) at
pthread_once.c:116
    #6  0x00007ffff7166d95 in memkind_malloc () from
/usr/lib/x86_64-linux-gnu/libmemkind.so.0
    #7  0x00007ffff7bbed04 in omp_aligned_alloc (alignment=1, size=4,
allocator=6309568) at [...]/libgomp/config/linux/../../allocator.c:521
    #8  0x0000000000400fd9 in main () at
source-gcc/libgomp/testsuite/libgomp.c-c++-common/alloc-11.c:182

Note that the failure is in libmemkind; the commits that trigger it are for
libnuma support.

Both these system have Ubuntu bionic libnuma1 2.0.11-2.1ubuntu0.1, libmemkind0
1.1.0-0ubuntu1.  Assuming this is similar to the latter's upstream v1.1.0,
that's
<https://github.com/memkind/memkind/blob/v1.1.0/src/memkind_interleave.c#L53-L54>,
where, per assembly-level debugging, we've got 'err = MEMKIND_ERROR_MALLCTL'
from '‎memkind_arena_create_map‎';
<https://github.com/memkind/memkind/blob/v1.1.0/src/memkind_arena.c#L95-L133>.

Might be a bug in libnuma/libmemkind, fixed in later versions (we're not seeing
these FAILs on other systems), or is the bug on our side, due to calling into
libnuma/libmemkind with some unexpected parameters, for example?

[Bug libgomp/111024] libgomp: FAILs with oldish libnuma/libmemkind

[burnus at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111024

--- Comment #1 from Tobias Burnus <burnus at gcc dot gnu.org> ---
Created attachment 55742
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=55742&action=edit
libgomp/allocator.c's gomp_init_libnuma - call numa_available()

Given that (lib)memkind uses jemalloc - on Bionic in version 3.6.0, it could
also be an error on that side.

In libmemkind, the arena stuff seems to be rather unrelated to libnuma - and at
least in the linked version, the call to numa_available() comes after the
failing line. ("numa_available" must be called (and return != -1) before using
any numa function.

Thus, it seems to be unrelated to libnuma - and, hence, us calling it.

Except: Well, a call to numa_available() is missing. It is worthwhile to check
whether the attached patch fixes the issue - and even if not, it probably
should be applied nonetheless.

[Bug libgomp/111024] libgomp: FAILs with oldish libnuma/libmemkind

[burnus at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111024

--- Comment #2 from Tobias Burnus <burnus at gcc dot gnu.org> ---
My guess feeling that this fail is completely unrelated to the new commits,
except that only the new alloc-11.c / alloc-12.c test omp_atk_partition =
omp_atv_interleaved.

I now disabled libnuma calls by not defining LIBGOMP_USE_LIBNUMA (in
libgomp/config/linux/allocator.c).

Result: On a bionic system (nvidia-3a) it still fails.

Granted, other changes in the GCC commits could have caused this, but this
seems to be less likely.


On the other hand, while libmemkind v1.1.1 was released only 6 weeks after
v1.1.0, none of the fixes look related to interleave (directly or via jemalloc)
and I assume that their testsuite did cover interleave (already in v0.3 and the
next release v1.0.0).

Still: Given that v1.1.0 was released on May 23, 2016 and the current release
is v1.14.0 (13 major and 2 minor releases later), I am not sure whether
debugging helps with regards to libmemkind and jemalloc (+ their dependencies
like: Linux kernel, libnuma, ...).

* * *

Probably not surprising, but attached patch did not help (but should be still
be added for correctness reasons).

Crossref: I filed PR111044 for using additional memkind kinds and documenting
which memkind kinds libgomp actually uses.

[Bug libgomp/111024] libgomp: FAILs with oldish libnuma/libmemkind

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111024

--- Comment #3 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Tobias Burnus <burnus@gcc.gnu.org>:

https://gcc.gnu.org/g:8f3c4517b1fff965f2bdedcf376dcfd91cda422b

commit r14-3287-g8f3c4517b1fff965f2bdedcf376dcfd91cda422b
Author: Tobias Burnus <tobias@codesourcery.com>
Date:   Thu Aug 17 15:20:55 2023 +0200

    libgomp: call numa_available first when using libnuma

    The documentation requires that numa_available() is called and only
    when successful, other libnuma function may be called. Internally,
    it does a syscall to get_mempolicy with flag=0 (which would return
    the default policy if mode were not NULL). If this returns -1 (and
    not 0) and errno == ENOSYS, the Linux kernel does not have the
    get_mempolicy syscall function; if so, numa_available() returns -1
    (otherwise: 0).

    libgomp/

            PR libgomp/111024
            * allocator.c (gomp_init_libnuma): Call numa_available; if
            not available or not returning 0, disable libnuma usage.