[Bug libc/29585] New: sched_getcpu returns invalid results

[Bug libc/29585] New: sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

            Bug ID: 29585
           Summary: sched_getcpu returns invalid results
           Product: glibc
           Version: 2.35
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libc
          Assignee: unassigned at sourceware dot org
          Reporter: mjbaars1977 at gmail dot com
                CC: drepper.fsp at gmail dot com
  Target Milestone: ---

Hello,

I noticed the following while I was trying SCHED_FIFO, because FIFO processes
are not supposed to migrate from processor to processor.

When looking at the manual page of sched_getcpu(), it states that:

The call

           cpu = sched_getcpu();

       is equivalent to the following getcpu(2) call:

           int c, s;
           s = getcpu(&c, NULL, NULL);
           cpu = (s == -1) ? s : c;

But when you compare the output of these two in a multi-threaded program (using
clone()), the two produce very different outcomes. Only the second one will
give you the right answer.

The results can be reproduced with sched_setaffinity() before calling clone()
and a call to both methods above from within the child process. My version uses
a timer_create() to make this call periodically.

        sched_get_cpu_timer_init(0);
//      core_from     = sched_get_cpu() + 1;    

        double x = 1.0;
        double y = (double) (random() + 1) * 1.00E-19f;

        for     (long i = 0; i < ((long) 1 << 32); i++) x *= 1.00f + y;

//      sleep   (1);
//      usleep  (random() % 1000000L);

//      core_upto     = sched_get_cpu() + 1;
        sched_get_cpu_timer_free();

Compilation is done with -Ofast -mfpmath=387 -fno-toplevel-reorder, for sake of
completeness.

Best regards,
Mischa Baars.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
     Ever confirmed|0                           |1
                 CC|                            |fweimer at redhat dot com
   Last reconfirmed|                            |2022-09-19

--- Comment #1 from Florian Weimer <fweimer at redhat dot com> ---
Would you please post a complete, self-contained reproducer, and state your
glibc and kernel versions? Thanks.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #2 from Mischa Baars <mjbaars1977 at gmail dot com> ---
(In reply to Mischa Baars from comment #0)
> Hello,
> 
> I noticed the following while I was trying SCHED_FIFO, because FIFO
> processes are not supposed to migrate from processor to processor.
> 
> When looking at the manual page of sched_getcpu(), it states that:
> 
> The call
> 
>            cpu = sched_getcpu();
> 
>        is equivalent to the following getcpu(2) call:
> 
>            int c, s;
>            s = getcpu(&c, NULL, NULL);
>            cpu = (s == -1) ? s : c;
> 
> But when you compare the output of these two in a multi-threaded program
> (using clone()), the two produce very different outcomes. Only the second
> one will give you the right answer.
> 
> The results can be reproduced with sched_setaffinity() before calling
> clone() and a call to both methods above from within the child process. My
> version uses a timer_create() to make this call periodically.
> 
> 	sched_get_cpu_timer_init(0);
> //	core_from     =	sched_get_cpu() + 1;	
> 
> 	double x = 1.0;
> 	double y = (double) (random() + 1) * 1.00E-19f;
> 
> 	for	(long i = 0; i < ((long) 1 << 32); i++)	x *= 1.00f + y;
> 
> //	sleep	(1);
> //	usleep	(random() % 1000000L);
> 	
> //	core_upto     =	sched_get_cpu() + 1;
> 	sched_get_cpu_timer_free();
> 
> Compilation is done with -Ofast -mfpmath=387 -fno-toplevel-reorder, for sake
> of completeness.
> 
> Best regards,
> Mischa Baars.

(In reply to Florian Weimer from comment #1)
> Would you please post a complete, self-contained reproducer, and state your
> glibc and kernel versions? Thanks.

Yes... working on it.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #3 from Mischa Baars <mjbaars1977 at gmail dot com> ---
Created attachment 14348
  --> https://sourceware.org/bugzilla/attachment.cgi?id=14348&action=edit
getcpu() (V) / sched_getcpu() (X)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #4 from Mischa Baars <mjbaars1977 at gmail dot com> ---
Created attachment 14350
  --> https://sourceware.org/bugzilla/attachment.cgi?id=14350&action=edit
reproducer

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #5 from Mischa Baars <mjbaars1977 at gmail dot com> ---
(In reply to Florian Weimer from comment #1)
> Would you please post a complete, self-contained reproducer, and state your
> glibc and kernel versions? Thanks.

Name         : glibc
Version      : 2.35
Release      : 17.fc36
Architecture : x86_64

Name         : kernel
Version      : 5.17.5
Release      : 300.fc36
Architecture : x86_64

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #6 from Mischa Baars <mjbaars1977 at gmail dot com> ---
(In reply to Florian Weimer from comment #1)
> Would you please post a complete, self-contained reproducer, and state your
> glibc and kernel versions? Thanks.

Hi Florian,

I see that the status of this bug is still set to WAITING. Do you need any
additional information or is the bug ready to be fixed by the glibc team like
this?

Best regards,
Mischa Baars.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|WAITING                     |RESOLVED

--- Comment #7 from Florian Weimer <fweimer at redhat dot com> ---
You cannot call glibc function after creating a bare thread using clone, you
need to use direct system calls. The bare thread does not have a proper thread
control block set up for glibc.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #8 from Mischa Baars <mjbaars1977 at gmail dot com> ---
(In reply to Florian Weimer from comment #7)
> You cannot call glibc function after creating a bare thread using clone, you
> need to use direct system calls. The bare thread does not have a proper
> thread control block set up for glibc.

Ok, no problem. I can use syscalls to retrieve the processor the process is
running on, but shouldn't glibc check for the existence of this thread control
block and return a -1 when it does not exist instead of a incorrect processor
number?

I think that a small note in the manual page about pthreads vs fork / clone
would be in order. For example, if the existence of both pthread_yield and
sched_yield makes you think that sched_yield is not specifically intended for
use with pthreads, then why would sched_getcpu be?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #9 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Mischa Baars from comment #8)
> (In reply to Florian Weimer from comment #7)
> > You cannot call glibc function after creating a bare thread using clone, you
> > need to use direct system calls. The bare thread does not have a proper
> > thread control block set up for glibc.
> 
> Ok, no problem. I can use syscalls to retrieve the processor the process is
> running on, but shouldn't glibc check for the existence of this thread
> control block and return a -1 when it does not exist instead of a incorrect
> processor number?

There is no efficient way to check if the TCB is safe to use from the current
thread.

> I think that a small note in the manual page about pthreads vs fork / clone
> would be in order. For example, if the existence of both pthread_yield and
> sched_yield makes you think that sched_yield is not specifically intended
> for use with pthreads, then why would sched_getcpu be?

sched_getcpu is not related to pthreads any more than the rest of glibc.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #10 from Mischa Baars <mjbaars1977 at gmail dot com> ---
(In reply to Florian Weimer from comment #9)
> (In reply to Mischa Baars from comment #8)
> > (In reply to Florian Weimer from comment #7)
> > > You cannot call glibc function after creating a bare thread using clone, you
> > > need to use direct system calls. The bare thread does not have a proper
> > > thread control block set up for glibc.
> > 
> > Ok, no problem. I can use syscalls to retrieve the processor the process is
> > running on, but shouldn't glibc check for the existence of this thread
> > control block and return a -1 when it does not exist instead of a incorrect
> > processor number?
> 
> There is no efficient way to check if the TCB is safe to use from the
> current thread.
> 
> > I think that a small note in the manual page about pthreads vs fork / clone
> > would be in order. For example, if the existence of both pthread_yield and
> > sched_yield makes you think that sched_yield is not specifically intended
> > for use with pthreads, then why would sched_getcpu be?
> 
> sched_getcpu is not related to pthreads any more than the rest of glibc.

But you just told me that the TCB is not bare thread related, so it must be
pthread related. Never mind, I'll just use direct system calls to do the
scheduling.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #11 from Mischa Baars <mjbaars1977 at gmail dot com> ---
(In reply to Florian Weimer from comment #9)
> (In reply to Mischa Baars from comment #8)
> > (In reply to Florian Weimer from comment #7)
> > > You cannot call glibc function after creating a bare thread using clone, you
> > > need to use direct system calls. The bare thread does not have a proper
> > > thread control block set up for glibc.
> > 
> > Ok, no problem. I can use syscalls to retrieve the processor the process is
> > running on, but shouldn't glibc check for the existence of this thread
> > control block and return a -1 when it does not exist instead of a incorrect
> > processor number?
> 
> There is no efficient way to check if the TCB is safe to use from the
> current thread.
> 
> > I think that a small note in the manual page about pthreads vs fork / clone
> > would be in order. For example, if the existence of both pthread_yield and
> > sched_yield makes you think that sched_yield is not specifically intended
> > for use with pthreads, then why would sched_getcpu be?
> 
> sched_getcpu is not related to pthreads any more than the rest of glibc.

Hi Florian,

A little more detail of what we're dealing with here:

I can't tell if this this is in any way related to a thread control block, but
without the CLONE_VM option passed to clone(), sched_getcpu() is working fine,
with the CLONE_VM option passed to clone(), it isn't.

Just in case you were still interested :)

Best regards,
Mischa Baars.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #12 from Andreas Schwab <schwab@linux-m68k.org> ---
clone without CLONE_VM is the same as fork (but without running the fork
handlers you still get undefined behaviour).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #13 from Mischa Baars <mjbaars1977 at gmail dot com> ---
You're more of less right, I now have function input and output through the
child function argument pointer, if configured correctly with mmap(),
instead of only an exit code. On the other hand, mmap() provides the
ability to write to parent memory with fork() as well if I'm correct. Still
fiddling around a little with all the options.

The biggest problem with CLONE_VM is that malloc() seems to be unaware of
its own doings in other kernel worker threads and on occasion returns the
same memory address more than once.

I was looking for a way to connect to the database in parallel, but
PQconnect() uses malloc() internally.


On Wed, 28 Sep 2022, 12:09 schwab@linux-m68k.org, <
sourceware-bugzilla@sourceware.org> wrote:

> https://sourceware.org/bugzilla/show_bug.cgi?id=29585
>
> --- Comment #12 from Andreas Schwab <schwab@linux-m68k.org> ---
> clone without CLONE_VM is the same as fork (but without running the fork
> handlers you still get undefined behaviour).
>
> --
> You are receiving this mail because:
> You reported the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #14 from Florian Weimer <fweimer at redhat dot com> ---
I wonder if we are approaching this from the wrong angle.

What would we have to change so that you can use pthread_create for your use
case? Then these issues with glibc interoperability would go away.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #15 from Mischa Baars <mjbaars1977 at gmail dot com> ---
I'll think about it, but something killed my keyboard in the meantime and I
had to flash the BIOS and reinstall the complete system before I realized
it was the keyboard instead of a virus. Keyboards are ROM, but I suppose
parts of the keyboard are still vulnerable to attack nonetheless. I should
have sold it to the Russians.

On Mon, 10 Oct 2022, 11:18 fweimer at redhat dot com, <
sourceware-bugzilla@sourceware.org> wrote:

> https://sourceware.org/bugzilla/show_bug.cgi?id=29585
>
> --- Comment #14 from Florian Weimer <fweimer at redhat dot com> ---
> I wonder if we are approaching this from the wrong angle.
>
> What would we have to change so that you can use pthread_create for your
> use
> case? Then these issues with glibc interoperability would go away.
>
> --
> You are receiving this mail because:
> You reported the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #16 from Mischa Baars <mjbaars1977 at gmail dot com> ---
And this all happened right after a found another bug (except from the
buffered output) in the kernel code. Full duplex transmissions display a
very serious slowdown on occasion (sometimes the very first time, sometimes
the 500th), but my thought being, who would want to destroy my computer for
stumbling upon someone else's programming mistakes? So I'm a little
preoccupied and out of the comfort zone, but I'll think about it.

Never actually used pthreads before. I used only fork() until recently
because of its simplicity and because it appeared to work sufficiently well
with PostgreSQL.

Clone() has the advance that you can pass arguments, i.e. the number of the
computation, so that in combination with an indexed output, you can now do
computations in parallel, solutions to differential equations at multiple
points in time for example.

On Mon, 19 Sep 2022, 16:21 fweimer at redhat dot com, <
sourceware-bugzilla@sourceware.org> wrote:

> https://sourceware.org/bugzilla/show_bug.cgi?id=29585
>
> Florian Weimer <fweimer at redhat dot com> changed:
>
>            What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>              Status|UNCONFIRMED                 |WAITING
>      Ever confirmed|0                           |1
>                  CC|                            |fweimer at redhat dot com
>    Last reconfirmed|                            |2022-09-19
>
> --- Comment #1 from Florian Weimer <fweimer at redhat dot com> ---
> Would you please post a complete, self-contained reproducer, and state your
> glibc and kernel versions? Thanks.
>
> --
> You are receiving this mail because:
> You reported the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #17 from Florian Weimer <fweimer at redhat dot com> ---
pthread_create supports passing arguments in exactly the same way __clone does.
So I guess there is nothing for us to do here on the glibc side.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #18 from Mischa Baars <mjbaars1977 at gmail dot com> ---
Other than the glibc sched_getcpu() function, which is returning incorrect
numbers when CLONE_VM is passed to clone(), no I don't think there's
anything to do for the moment.

On Mon, 10 Oct 2022, 18:31 fweimer at redhat dot com, <
sourceware-bugzilla@sourceware.org> wrote:

> https://sourceware.org/bugzilla/show_bug.cgi?id=29585
>
> --- Comment #17 from Florian Weimer <fweimer at redhat dot com> ---
> pthread_create supports passing arguments in exactly the same way __clone
> does.
> So I guess there is nothing for us to do here on the glibc side.
>
> --
> You are receiving this mail because:
> You reported the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #19 from Mischa Baars <mjbaars1977 at gmail dot com> ---
The fact remains that the sched_getcpu() manual page is contradicting the
sched_getcpu() function output when CLONE_VM is passed to clone().

According to the manual page, the output should have been equivalent to:

temp = getcpu(&proc, NULL); return ((temp < 0) ? temp : proc);

and it certainly isn't under the circumstances.

On Mon, 10 Oct 2022, 18:31 fweimer at redhat dot com, <
sourceware-bugzilla@sourceware.org> wrote:

> https://sourceware.org/bugzilla/show_bug.cgi?id=29585
>
> --- Comment #17 from Florian Weimer <fweimer at redhat dot com> ---
> pthread_create supports passing arguments in exactly the same way __clone
> does.
> So I guess there is nothing for us to do here on the glibc side.
>
> --
> You are receiving this mail because:
> You reported the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #20 from Andreas Schwab <schwab@linux-m68k.org> ---
The manual only describes the function in absence of undefined behaviour.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #21 from Mischa Baars <mjbaars1977 at gmail dot com> ---
Indeed :)

On Mon, 10 Oct 2022, 19:09 schwab@linux-m68k.org, <
sourceware-bugzilla@sourceware.org> wrote:

> https://sourceware.org/bugzilla/show_bug.cgi?id=29585
>
> --- Comment #20 from Andreas Schwab <schwab@linux-m68k.org> ---
> The manual only describes the function in absence of undefined behaviour.
>
> --
> You are receiving this mail because:
> You reported the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #22 from Mischa Baars <mjbaars1977 at gmail dot com> ---
(In reply to Florian Weimer from comment #14)
> I wonder if we are approaching this from the wrong angle.
> 
> What would we have to change so that you can use pthread_create for your use
> case? Then these issues with glibc interoperability would go away.

Hi Florian,

I've just checked out the master branch, and if I'm correct nothing has changed
in sched_getcpu() since our last correspondence, or actually nothing has
changed since version 2.35.

Were you or anyone else responsible for the maintenance of glibc planning to
fix the issue described in this thread or was that not even part of the plan?

Best regards,
Mischa Baars.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #23 from Florian Weimer <fweimer at redhat dot com> ---
I don't think there is nothing to fix in glibc because you use __clone
incorrectly—and unnecessarily. This issue will not happen with pthread_create.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #24 from Mischa Baars <mjbaars1977 at gmail dot com> ---
Ok, I see. I can try pthread_create, but that doesn't fix the phenomenon.
Neither does it fix missing SIGCHLD signals on fork()/clone() termination.
However unimportant that seems to you now, there are other (SIGPOLL)
signals missing as well, that are from my point of view, essential when
trying to distribute tasks across the network. In trying so, I was only
able to come up with a client (connected to four servers) take consumes 84%
processor time in it's main loop, because waiting for SIGPOLL fails each
time and again.

On Mon, 14 Nov 2022, 13:55 fweimer at redhat dot com, <
sourceware-bugzilla@sourceware.org> wrote:

> https://sourceware.org/bugzilla/show_bug.cgi?id=29585
>
> --- Comment #23 from Florian Weimer <fweimer at redhat dot com> ---
> I don't think there is nothing to fix in glibc because you use __clone
> incorrectly—and unnecessarily. This issue will not happen with
> pthread_create.
>
> --
> You are receiving this mail because:
> You reported the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/29585] sched_getcpu returns invalid results

[mjbaars1977 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29585

--- Comment #25 from Mischa Baars <mjbaars1977 at gmail dot com> ---
If you think that I'm using certain system calls incorrectly, such that
undefined behavior from using sched_getcpu() is the result, and
pthread_create() does a better job because your code is presumably better
than the system calls' code, then why are these SIGCHLD signals missing?
For now I see only a threading system that seems to be missing essential
bits and pieces.

On Mon, 14 Nov 2022, 13:55 fweimer at redhat dot com, <
sourceware-bugzilla@sourceware.org> wrote:

> https://sourceware.org/bugzilla/show_bug.cgi?id=29585
>
> --- Comment #23 from Florian Weimer <fweimer at redhat dot com> ---
> I don't think there is nothing to fix in glibc because you use __clone
> incorrectly—and unnecessarily. This issue will not happen with
> pthread_create.
>
> --
> You are receiving this mail because:
> You reported the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.