[Bug libc/27645] New: [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[Bug libc/27645] New: [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[crrodriguez at opensuse dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

            Bug ID: 27645
           Summary: [linux] sysconf(_SC_NPROCESSOR...) breaks down on
                    containers
           Product: glibc
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libc
          Assignee: unassigned at sourceware dot org
          Reporter: crrodriguez at opensuse dot org
                CC: drepper.fsp at gmail dot com
  Target Milestone: ---

On linux sysconf(_SC_NPROCESSORS_¨*) ought to be implemented on top
sched_getaffinity and CPU_COUNT() the number of cpus available to the current
thread instead of what is available on the whole system, otherwise it breaks on
container engines with CPU limits or by using cpusets.

Musl does it this way, glibc should too, most programs use _ONLYN for choosing
how many threads to start... and the assumptions made no longer hold.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at sourceware dot org   |adhemerval.zanella at linaro dot o
                   |                            |rg
                 CC|                            |adhemerval.zanella at linaro dot o
                   |                            |rg

--- Comment #1 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
It should be feasible to use the sched_getaffinity for _SC_NPROCESSORS_ONLN,
however I think we should keep using the sysfs entries for _SC_NPROCESSORS_CONF
and only fallback to sched_getaffinity if sysfs can be read.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[crrodriguez at opensuse dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

Cristian Rodríguez <crrodriguez at opensuse dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://www.austingroupbugs
                   |                            |.net/view.php?id=339

--- Comment #2 from Cristian Rodríguez <crrodriguez at opensuse dot org> ---
Austin group bug 339 also refers to this problem.


Also the manual
https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html
says

"sysconf (_SC_NPROCESSORS_CONF)
which returns the number of processors the operating system configured"

This wording appears to allow it to be implemented with sched_getaffinity..
because those are "the number of processors the operating system configured"
(at this point in the for the current thread)

I'm fine with the suggested compromise you state in the comment, but I still
believe the real behaviour should be clarifed to mean exactly "number of
available/online or configured cpus to the current thread at the time of the
call" (because you can change the affinity mask on a running process too)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

--- Comment #3 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
The problem with sched_getaffinity is we can't really infer the high bound of
offline processor, since it only returns a mask of online CPUs.

And by 'configured' my understanding is it should map to the 'possible' sysfs
cpu field [1], while _SC_NPROCESSORS_ONLN should map to the 'present'.

The current code try both with sysfs and procfs to get the possible CPU number,
but it falls back to present value if sysfs or procfs can't accessed. I think
we can eliminate the procfs fallback, afaik on usual deployments it is rare to
provide just one of them and it does simplified the code a lot.

[1] https://www.kernel.org/doc/html/latest/admin-guide/cputopology.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |2.34
         Resolution|---                         |FIXED
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #4 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
Fixed on 2.34.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[nsaenzju at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

Nicolas Saenz Julienne <nsaenzju at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |nsaenzju at redhat dot com
         Resolution|FIXED                       |---
     Ever confirmed|0                           |1
             Status|RESOLVED                    |REOPENED
   Last reconfirmed|                            |2021-09-01

--- Comment #5 from Nicolas Saenz Julienne <nsaenzju at redhat dot com> ---
I'm seeing regressions on standard system monitoring tools[1] after this
change. Specifically on systems that make use of CPU isolation
(isolcpus=<cpumask>), as CPU isolation forces an affinity mask on the init
process and all its children. This affinity mask can be overridden during
runtime.

Available/online CPUs have changed their meaning from "The number of processors
capable of running processes" (system-wide scope and in line with what the
Linux kernel means by an 'online' CPU) to "The number of processors allowed to
run the current process". Note that manpages for get_nprocs() (Linux and
FreeBSD) and "https://www.austingroupbugs.net/view.php?id=339" imply this is
system-wide API. The libc manual is a bit more ambiguous. But IMO the
expectation up to 2.34 was to get system-wide data. Hence the breakage I
mention. Ultimately, how should applications query system-wide CPU information
now?

It would be useful for the discussion to have a little more background on how
the previous behavior broke container engines. But overall I'd suggest this
change be reverted and work on a solution that fits all users.

[1] For ex: this changes top's (the command line utility) behavior, as it no
longer lists per-CPU information correctly, but only CPUs under its affinity
mask. To reproduce run top with a specific CPU affinity, and while in program
press 1 to list per-CPU information. Not all CPUs are listed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fweimer at redhat dot com

--- Comment #6 from Florian Weimer <fweimer at redhat dot com> ---
Yes, I think we need to revert this change.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

--- Comment #7 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
Cristian Rodríguez suggested this change stating that *not* using
sched_getaffinity potentially breaks program within containters (since they
will get a higher CPU count that the one it can actually use), so the question
is why programs  know the *online* system wide cpus if they can't actually use
on sched_setaffinity or any other interface?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

--- Comment #8 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
I am not against reverting it, but we do have two different users stating that
with or without the changes containers will break.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

--- Comment #9 from Florian Weimer <fweimer at redhat dot com> ---
The change breaks the top command in common hypervisor configurations (and also
other monitoring tools). That's far more drastic than some container use case
that does not even work reliably with most container hosts (only those that lie
about affinity masks or are very unusual in that they do not oversubscribe
CPUs).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

--- Comment #10 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
If this is the only possible breakage about not using sched_getaffinity for
_SC_NPROCESSORS_ONLY it should be fine, but it would be good to hear from
Cristian if he saw something else that broke due returning system-wide
information.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[crrodriguez at opensuse dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

--- Comment #11 from Cristian Rodríguez <crrodriguez at opensuse dot org> ---
This is what I was afraid it will happen..I am fine with reverting this change
if an actually correct API is provided to do the right thing.. which is NOT how
many CPUs has the system online but how many cpus can the current thread use at
the moment the function was called..

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[nsaenzju at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

--- Comment #12 from Nicolas Saenz Julienne <nsaenzju at redhat dot com> ---
Why can't the process use sched_getaffinity() directly?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[crrodriguez at opensuse dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

--- Comment #13 from Cristian Rodríguez <crrodriguez at opensuse dot org> ---
(In reply to Nicolas Saenz Julienne from comment #12)
> Why can't the process use sched_getaffinity() directly?

Yes, it can..the idea was not having to update possible hundreds of packages to
work as expected in a container enviroment..  examples that come to min are all
the parallel de/compression tools like pigz. when ran in a container with cpu
count limit does not start up to 8 threads when the enviroment says it can use
2 CPU..and so on..  

I googled quite a bit before posting this request and didn't found anywhere
that _SC_NPROCESSORS_ONLN was explictly specified as system-wide..

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

--- Comment #14 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Cristian Rodríguez from comment #13)
> (In reply to Nicolas Saenz Julienne from comment #12)
> > Why can't the process use sched_getaffinity() directly?
> 
> Yes, it can..the idea was not having to update possible hundreds of packages
> to work as expected in a container enviroment..  examples that come to min
> are all the parallel de/compression tools like pigz. when ran in a container
> with cpu count limit does not start up to 8 threads when the enviroment says
> it can use 2 CPU..and so on..  

It only works on container hosts that disable/ignore sched_setaffinity and
return a synthetic CPU mask with sched_getaffinity. The more common approach
seems to involve the CPU cgroup controller, and that does not change the
affinity mask as far as I can tell.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://sourceware.org/bugz
                   |                            |illa/show_bug.cgi?id=28310

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27645] [linux] sysconf(_SC_NPROCESSOR...) breaks down on containers

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27645

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|REOPENED                    |RESOLVED

--- Comment #15 from Florian Weimer <fweimer at redhat dot com> ---
I'm closing this in favor of bug 28310 for the regression fix.

-- 
You are receiving this mail because:
You are on the CC list for the bug.