[Bug libc/30135] New: sysconf: _SC_NPROCESSORS*: tweak?

[Bug libc/30135] New: sysconf: _SC_NPROCESSORS*: tweak?

[steffen at sdaoden dot eu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30135

            Bug ID: 30135
           Summary: sysconf: _SC_NPROCESSORS*: tweak?
           Product: glibc
           Version: 2.36
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libc
          Assignee: unassigned at sourceware dot org
          Reporter: steffen at sdaoden dot eu
                CC: drepper.fsp at gmail dot com
  Target Milestone: ---

#?0|kent:tmp$ cat t.c
#include <stdio.h>
#include <unistd.h>
int main(void){
        long c,o;
        c = sysconf(_SC_NPROCESSORS_CONF);
        o = sysconf(_SC_NPROCESSORS_ONLN);
        printf("c=%ld o=%ld\n", c, o);
        return 0;
}


On bare metal:

#?0|kent:tmp$ tcc -run t.c
c=8 o=4
#?0|kent:tmp$ nproc
4

But in a box with cgroup-limited CPUs:

bash-5.2$ tcc -run t.c
c=4 o=4
bash-5.2$ nproc
3


Could this be a "bug" in the C library, as this checks
/sys/devices/system/cpu/online, /proc/stat, and only
then (iirc now) uses sched_getaffinity.  It documents
get_nprocs(3) as "number of processors currently
available in the system".

But POSIX standardized these[1] as _SC_NPROCESSORS_CONF
+ _SC_NPROCESSORS_ONLN for sysconf(3) as well as
NPROCESSORS_CONF and NPROCESSORS_ONLN for getconf(1),
saying

  ...
    The maximum number of execution units | _SC_NPROCESSORS_ONLN
    currently available to run threads† |
  ....
    † The nature of an execution unit and the precise conditions
    under which an execution unit is considered to be available,
    or can be made available, or how many threads it can execute
    in parallel, are implementation-defined.

Which is not the same.  Implementation-defined though.
But i mean if it is easy to give correct results for an execution
context (i would).

  [1] https://austingroupbugs.net/view.php?id=339

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/30135] sysconf: _SC_NPROCESSORS*: tweak?

[steffen at sdaoden dot eu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30135

Steffen (Daode) Nurpmeso <steffen at sdaoden dot eu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |steffen at sdaoden dot eu

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/30135] sysconf: _SC_NPROCESSORS*: tweak?

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30135

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |adhemerval.zanella at linaro dot o
                   |                            |rg

--- Comment #1 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
We had a similar RFE to use sched_getaffinity for _SC_NPROCESSOR_* [1], which
was implemented by 903bc7dcc2a.  It used the same Austin ticked as a
justification, however, on the very bug reports there are indications that it
triggered multiple regressions [2].

So it was later reverted by 342298278eabc, with the justification that using
sched_getaffinity changed the _SC_NPROCESSOR_* semantic from system-wide scope
of online CPUs to per-process one (which can be changed with kernel cpusets or
book parameters in VM).

I think we should mark this as a duplicate of BZ#27645 or BZ#28310, since we
already settle that using sched_getaffinity should be only used as a fallback.

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=27645
[2] https://sourceware.org/bugzilla/show_bug.cgi?id=28310

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/30135] sysconf: _SC_NPROCESSORS*: tweak?

[steffen at sdaoden dot eu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30135

--- Comment #2 from Steffen Nurpmeso <steffen at sdaoden dot eu> ---
. sysconf and _SC_NPROCESSORS_ONLN is the only standardized variant,
sched_getaffinity() is not a portable interface.

. I feel "currently available to run threads" pretty much matches that
CPU_COUNT() thing that nproc(1) uses.

. For example, look at zstd(1), they use sysconf(_SC_NPROCESSORS_ONLN) to
realize -T0.  Same for xz(1).
This is _all_ wrong in my cgroup restricted environment, they potentially
"over-thread".

. Quite the opposite, top(1) is to be changed (a bit strange as my version is
from https://gitlab.com/procps-ng aka /proc based)
If i run it in a cgroup-mutilated environment with three CPUs, it lists a
fourth one that is out of scope!

. I have not checked virtual machines.  It astounds me they use synthesized CPU
counts instead of the CPU count that truly is available.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/30135] sysconf: _SC_NPROCESSORS*: tweak?

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30135

--- Comment #3 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
(In reply to Steffen Nurpmeso from comment #2)
> . sysconf and _SC_NPROCESSORS_ONLN is the only standardized variant,
> sched_getaffinity() is not a portable interface.
> 
> . I feel "currently available to run threads" pretty much matches that
> CPU_COUNT() thing that nproc(1) uses.
> 
> . For example, look at zstd(1), they use sysconf(_SC_NPROCESSORS_ONLN) to
> realize -T0.  Same for xz(1).
> This is _all_ wrong in my cgroup restricted environment, they potentially
> "over-thread".
> 
> . Quite the opposite, top(1) is to be changed (a bit strange as my version
> is from https://gitlab.com/procps-ng aka /proc based)
> If i run it in a cgroup-mutilated environment with three CPUs, it lists a
> fourth one that is out of scope!
> 
> . I have not checked virtual machines.  It astounds me they use synthesized
> CPU counts instead of the CPU count that truly is available.

Fell free to rehash this discussion on libc-alpha, but as I said we already
tried it and since it turned out to generate a lot of unseen problems we
decided to revert and use sysfs instead.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/30135] sysconf: _SC_NPROCESSORS*: tweak?

[steffen at sdaoden dot eu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30135

Steffen Nurpmeso <steffen at sdaoden dot eu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WONTFIX
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #4 from Steffen Nurpmeso <steffen at sdaoden dot eu> ---
Then let us close it?!

(Really tough.  The build system of my Linux distro picks up the right thing
via nproc, but the final compression will then "over-thread".)

Ciao!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/30135] sysconf: _SC_NPROCESSORS*: tweak?

[sam at gentoo dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30135

Sam James <sam at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sam at gentoo dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/30135] sysconf: _SC_NPROCESSORS*: tweak?

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30135

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fweimer at redhat dot com

--- Comment #5 from Florian Weimer <fweimer at redhat dot com> ---
We currently do not have a proper protocol for communicating maximum
concurrency from container hosts to container workloads. The cgroup protocol is
not a stable userspace interface, so it's not usable from within glibc (it's
also very slow). Some container hosts override affinity masks, but that breaks
certain synchronization patterns, so other container hosts do not do that, and
the affinity mask does not reflect actual CPU quotas available to the workload.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/30135] sysconf: _SC_NPROCESSORS*: tweak?

[steffen at sdaoden dot eu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30135

--- Comment #6 from Steffen Nurpmeso <steffen at sdaoden dot eu> ---
Quite honestly.  That is too high for me.

All i know is nproc(1) (of busybox, coreutils, now also FreeBSD (devel)) gets
it right, but (portable) compiled applications using sysconf(3) do not.

(I was only looking at two known-threaded, xz and zstd, they both will be
trapped in doing the wrong thing.)

And "not stable" sounds a bit peculiar given that cgroups are decades old, no?
Linux documentation in-file dates are 2004 for v1 and 2015 for v2.
And cpuset.cpus is actually in both, i think

    mkdir /sys/fs/cgroup/NAME || return
    echo 1-3 > /sys/fs/cgroup/NAME/cpuset.cpus

To me this looks as if Linux userspace is forcing (somewhat, now standardized)
portable applications to do the wrong thing for many, many years.
I do not know about "cgroup protocol" or why that is slow --
sched_getaffinity(2) is what is used by nproc(1)s.
Broken synchronization patterns.  Heh.

-- 
You are receiving this mail because:
You are on the CC list for the bug.