[Bug libc/28991] New: sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[Bug libc/28991] New: sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[robert at ocallahan dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

            Bug ID: 28991
           Summary: sysconf(_SC_NPROCESSORS_CONF) should read
                    /sys/devices/system/cpu/possible
           Product: glibc
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libc
          Assignee: unassigned at sourceware dot org
          Reporter: robert at ocallahan dot org
                CC: drepper.fsp at gmail dot com
  Target Milestone: ---

Currently on Linux `__get_nprocs_conf` first tries to enumerate
`/sys/devices/system/cpu`. This enumerates the CPUs that are present in the
system (but possibly offline). However, new CPUs can be added via hotplugging
that were not previously enumerated. This is a problem for applications that
want to allocate an array and index it by CPU ID (e.g. the `cpu_id` field of
`struct rseq`). Currently on Linux the only safe way to allocate such an array
is to read `/sys/devices/system/cpu/possible` to determine the array size. Not
many applications are doing that.

Rather than require applications to do that (which is non-portable as well as
being painful), I think it would make more sense to modify `__get_nprocs_conf`
to try reading `/sys/devices/system/cpu/possible` instead of enumerating the
CPUs.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |adhemerval.zanella at linaro dot o
                   |                            |rg

--- Comment #1 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
I am not sure if we can actually /sys/devices/system/cpu/possible, since for
some architecture it reports possible hotplug cpus that might not actually be
present in the system.

For instance, on my system with a Ryzen 5900x (12c, 24t) I see:

$ sudo dmesg
[...]
[    0.065499] smpboot: Allowing 32 CPUs, 8 hotplug CPUs
[...]
$ cat /sys/devices/system/cpu/possible 
0-31
$ cat /sys/devices/system/cpu/online 
0-23
$ cat /sys/devices/system/cpu/offline 
24-31

And thus using 'possible' will wrongly report 32 cpus on a system with the
maximum of 24. 

The Linux source give us some hints why it is doing it:

arch/x86/kernel/smpboot.c
1513  * Three ways to find out the number of additional hotplug CPUs:
1514  * - If the BIOS specified disabled CPUs in ACPI/mptables use that.
1515  * - The user can overwrite it with possible_cpus=NUM
1516  * - Otherwise don't reserve additional CPUs.
1517  * We do this because additional CPUs waste a lot of memory.

So I am guessing that BIOS in my case it reporting 32 because it supports the
Ryzen 5950x, even thought he system does not actually support a large number of
cpus.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #2 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
I see a similar behaviour on a sparc64 machine:

$ cat /sys/devices/system/cpu/possible
0-255
$ cat /sys/devices/system/cpu/online 
0-63
$ cat /sys/devices/system/cpu/offline 
64-255

So I think reporting the actual presented CPU for __get_nprocs_conf is the
correct semantic here.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #3 from Andreas Schwab <schwab@linux-m68k.org> ---
But the systems are configured for 32 and 256 cpus, resp.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #4 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
Does it really make sense to report a large number of cpus even if caller won't
be able to actually activate them? And can you assure changing this interface
won't regress again?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #5 from Andreas Schwab <schwab@linux-m68k.org> ---
How do you know that they cannot be activated?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #6 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
Because it is not listed on /sys/devices/system/cpu/cpu*.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #7 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
But I am more interested whether it would trigger any regressions, as we
recently hit some in recent changes.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #8 from Andreas Schwab <schwab@linux-m68k.org> ---
The are not listed because they aren't currently present.  But they can be
hotplugged any time (in theory).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #9 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
And that is my question: does it make to report the configured cpu number on a
system where hotplug might not be possible? And are the programs that use this
interface aware of it?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #10 from Andreas Schwab <schwab@linux-m68k.org> ---
How do you know if hotplug is not possible?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #11 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
That's why I put 'might not be possible'. Do you think we should make this
change?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #12 from Andreas Schwab <schwab@linux-m68k.org> ---
The same question: how do you know that?  We can only use the information that
the system provides.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[robert at ocallahan dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #13 from Robert O'Callahan <robert at ocallahan dot org> ---
Applications that allocate an array indexed by rseq cpu_id or getcpu() or
similar need to size it to accommodate all possible CPUs. On Linux they have to
use /sys/devices/system/cpu/possible, regardless of how accurate that is. (I
don't think there's any better source for this information, and if there is,
the kernel should use it to make /sys/devices/system/cpu/possible more
accurate.)

It's not a big problem if the number of CPUs from
/sys/devices/system/cpu/possible is larger than necessary. That will only waste
a small amount of memory. The kernel internally is already sizing its per-cpu
structures based on cpu_possible_mask.

The only real question is what sysconf(_SC_NPROCESSORS_CONF) should actually
mean. If it's intended to be an upper bound for CPU IDs, then it needs to use 
/sys/devices/system/cpu/possible. If it isn't, then I don't know what it's
useful for, but at least it should have a big warning that it is not an upper
bound for CPU IDs.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[robert at ocallahan dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

--- Comment #14 from Robert O'Callahan <robert at ocallahan dot org> ---
And if you choose to not use /sys/devices/system/cpu/possible for
_SC_NPROCESSORS_CONF, then someone what _SC_NPROCESSORS_CONF is useful for,
because I have no idea.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Last reconfirmed|                            |2022-05-16
           Assignee|unassigned at sourceware dot org   |adhemerval.zanella at linaro dot o
                   |                            |rg
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |NEW
   Target Milestone|---                         |2.36

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/28991] sysconf(_SC_NPROCESSORS_CONF) should read /sys/devices/system/cpu/possible

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28991

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #15 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
Fixed on 2.36.

-- 
You are receiving this mail because:
You are on the CC list for the bug.