[Bug libc/27576] New: gmon.out not consistently created

[Bug libc/27576] New: gmon.out not consistently created

[wsnyder at wsnyder dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

            Bug ID: 27576
           Summary: gmon.out not consistently created
           Product: glibc
           Version: 2.31
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libc
          Assignee: unassigned at sourceware dot org
          Reporter: wsnyder at wsnyder dot org
                CC: drepper.fsp at gmail dot com
  Target Milestone: ---

Created attachment 13307
  --> https://sourceware.org/bugzilla/attachment.cgi?id=13307&action=edit
Testcase

== Description

Type "make" in this tarball's directory.

This will show (extra lines omitted):

  g++  -pg -O0  -c -o Vt_case_huge_prof__main.o Vt_case_huge_prof__main.ii
  @@@@@ Did we get a gmon?
  make: [Makefile:15: x] Error 1 (ignored)

  g++  -pg -Os  -c -o Vt_case_huge_prof__main.o Vt_case_huge_prof__main.ii
  @@@@@ Did we get a gmon?
    4554816     28 -rw-rw-r--   1 user  user     26943 Mar 13 11:33 ./gmon.out

  g++  -pg -O0  -c -o Vt_case_huge_prof__main.o Vt_case_huge_prof__main__ok.ii
  @@@@@ Did we get a gmon?
    4554816    884 -rw-rw-r--   1 user  user    897414 Mar 13 11:33 ./gmon.out


In short gcc -pg with -O0 is not reliably creating a gmon.

The problem goes away using -Os.

The problem goes away if line 60549 of ~/d/case/Vt_case_huge_prof__main.ii

    std::array<CData , 16384> __Vtablechg1;

is changed into

    CData __Vtablechg1[16384];

Others have reported the problem appears timing related and sometimes gmon.out
will get produced but on my system it is completely reliably broken.

Experiments show clang++ (with same glibc on same system) gives same
misbehavior.

While this "feels" like a memory corruption sort of bug, the program is
sanitizer and warning clean. This was originally reported as GCC bug at
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99579 but appears the compiler is
creating reasonable code.

== gcc --version

gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

== glibc verison

glibc-source/focal-updates 2.31-0ubuntu9.2 all

== cat /etc/*release

DISTRIB_DESCRIPTION="Ubuntu 20.04 LTS"

== cat /proc/cpuinfo

... AMD Ryzen 9 3950X 16-Core

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[wsnyder at wsnyder dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

--- Comment #1 from Wilson Snyder <wsnyder at wsnyder dot org> ---
Pasting additional info from https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99579

"GCC is emitting the mcount call correctly for each function.
Glibc is the library where the mcount is located and enabling of the timer
(SIGPROF) is enabled and it controls the outputting of the gmon.out.
>From what I can tell if the timer does not happen, then there will be no
gmon.out outputted.
With a slower/older machine (AMD Athlon(tm) II X4 640 Processor), I sometimes
get gmon.out and sometimes don't.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[leo at yuriev dot ru]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

Leo Yuriev <leo at yuriev dot ru> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |leo at yuriev dot ru

--- Comment #2 from Leo Yuriev <leo at yuriev dot ru> ---
This may be a manifestation of the bug described and fixed (patch attached) in
Bug 29444.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[pinskia at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pinskia at gcc dot gnu.org
           See Also|                            |https://sourceware.org/bugz
                   |                            |illa/show_bug.cgi?id=29444

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[pinskia at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://gcc.gnu.org/bugzill
                   |                            |a/show_bug.cgi?id=99579

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[pinskia at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://gcc.gnu.org/bugzill
                   |                            |a/show_bug.cgi?id=108146

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[skissane at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

Simon Kissane <skissane at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |skissane at gmail dot com

--- Comment #3 from Simon Kissane <skissane at gmail dot com> ---
I'm pretty sure this is an overflow in mcount.

Here's how I can tell:

$ gdb ./Vt_case_huge_prof 
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./Vt_case_huge_prof...
(No debugging symbols found in ./Vt_case_huge_prof)
(gdb) break _mcleanup
Function "_mcleanup" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_mcleanup) pending.
(gdb) run
Starting program: /home/skissane/bz27576/case/Vt_case_huge_prof 
- t/t_case_huge.v:206: Verilog $finish

Breakpoint 1, _mcleanup () at gmon.c:440
440     gmon.c: No such file or directory.
(gdb) print _gmonparam
$1 = {state = 2, kcount = 0x7ffff7972a98, kcountsize = 508200, froms =
0x7ffff79eebc0, fromssize = 508194, tos = 0x7ffff78c0010, tossize = 731784,
tolimit = 30491, lowpc = 93824992247264, 
  highpc = 93824993263652, textsize = 1016388, hashfraction = 2,
log_hashfraction = 4}

Note that state=2. If you look at /usr/include/sys/gmon.h you will see:

#define GMON_PROF_ERROR 2

So, what has happened, is gmon has gone into error state.

Per the source code, only two ways this can happen:

1) monstartup runs out of memory - not case here since then tos will be NULL,
but tos is non-NULL
2) overflow in mcount

So clearly this is an overflow in mcount. There is too much profiling data and
mcount runs out of space in its buffer to store it. Probably the reason why
changing compilation options around optimisation/etc makes the issue go away,
is that changes how the code is laid out (how many functions are called vs
inlined etc) which causes less consumption of space in the buffer.

Contrary to the comment by "account removed" (who I understand is someone who
had their Bugzilla access revoked due to unpleasant behaviour), I don't believe
this is related to bug 29444. That bug complains that kcountsize and fromsize
are calculated incorrectly, causing a buffer overflow. Even if that's true, the
overflow condition is based on exceeding tolimit, and the patch for that bug
isn't changing the calculation of tolimit.

One problem here, is mcount doesn't print any error message when the overflow
happens, leaving the user confused as to what went wrong, as has happened in
this case.

Another problem, is that there is no way to increase the buffer size beyond
MAXARCS entries without recompiling glibc. If MAXARCS (and maybe MINARCS too,
in case the buffer sizing heuristic is malfunctioning) were tunables, the user
could respond to the overflow error message (were it printed) by increasing
that tunable.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[skissane at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

--- Comment #4 from Simon Kissane <skissane at gmail dot com> ---
Created attachment 14676
  --> https://sourceware.org/bugzilla/attachment.cgi?id=14676&action=edit
proposed patch (also sent to mailing list)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

--- Comment #5 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by DJ Delorie <dj@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=31be941e4367c001b2009308839db5c67bf9dcbc

commit 31be941e4367c001b2009308839db5c67bf9dcbc
Author: Simon Kissane <skissane@gmail.com>
Date:   Sat Feb 11 20:12:13 2023 +1100

    gmon: improve mcount overflow handling [BZ# 27576]

    When mcount overflows, no gmon.out file is generated, but no message is
printed
    to the user, leaving the user with no idea why, and thinking maybe there is
    some bug - which is how BZ 27576 ended up being logged. Print a message to
    stderr in this case so the user knows what is going on.

    As a comment in sys/gmon.h acknowledges, the hardcoded MAXARCS value is too
    small for some large applications, including the test case in that BZ.
Rather
    than increase it, add tunables to enable MINARCS and MAXARCS to be
overridden
    at runtime (glibc.gmon.minarcs and glibc.gmon.maxarcs). So if a user gets
the
    mcount overflow error, they can try increasing maxarcs (they might need to
    increase minarcs too if the heuristic is wrong in their case.)

    Note setting minarcs/maxarcs too large can cause monstartup to fail with an
    out of memory error. If you set them large enough, it can cause an integer
    overflow in calculating the buffer size. I haven't done anything to defend
    against that - it would not generally be a security vulnerability, since
these
    tunables will be ignored in suid/sgid programs (due to the SXID_ERASE
default),
    and if you can set GLIBC_TUNABLES in the environment of a process, you can
take
    it over anyway (LD_PRELOAD, LD_LIBRARY_PATH, etc). I thought about
modifying
    the code of monstartup to defend against integer overflows, but doing so is
    complicated, and I realise the existing code is susceptible to them even
prior
    to this change (e.g. try passing a pathologically large highpc argument to
    monstartup), so I decided just to leave that possibility in-place.

    Add a test case which demonstrates mcount overflow and the tunables.

    Document the new tunables in the manual.

    Signed-off-by: Simon Kissane <skissane@gmail.com>
    Reviewed-by: DJ Delorie <dj@redhat.com>

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

--- Comment #6 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The release/2.37/master branch has been updated by Florian Weimer
<fw@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=d230623264e300ac1c827cb83ad7818f122a6a98

commit d230623264e300ac1c827cb83ad7818f122a6a98
Author: Simon Kissane <skissane@gmail.com>
Date:   Sat Feb 11 20:12:13 2023 +1100

    gmon: improve mcount overflow handling [BZ# 27576]

    When mcount overflows, no gmon.out file is generated, but no message is
printed
    to the user, leaving the user with no idea why, and thinking maybe there is
    some bug - which is how BZ 27576 ended up being logged. Print a message to
    stderr in this case so the user knows what is going on.

    As a comment in sys/gmon.h acknowledges, the hardcoded MAXARCS value is too
    small for some large applications, including the test case in that BZ.
Rather
    than increase it, add tunables to enable MINARCS and MAXARCS to be
overridden
    at runtime (glibc.gmon.minarcs and glibc.gmon.maxarcs). So if a user gets
the
    mcount overflow error, they can try increasing maxarcs (they might need to
    increase minarcs too if the heuristic is wrong in their case.)

    Note setting minarcs/maxarcs too large can cause monstartup to fail with an
    out of memory error. If you set them large enough, it can cause an integer
    overflow in calculating the buffer size. I haven't done anything to defend
    against that - it would not generally be a security vulnerability, since
these
    tunables will be ignored in suid/sgid programs (due to the SXID_ERASE
default),
    and if you can set GLIBC_TUNABLES in the environment of a process, you can
take
    it over anyway (LD_PRELOAD, LD_LIBRARY_PATH, etc). I thought about
modifying
    the code of monstartup to defend against integer overflows, but doing so is
    complicated, and I realise the existing code is susceptible to them even
prior
    to this change (e.g. try passing a pathologically large highpc argument to
    monstartup), so I decided just to leave that possibility in-place.

    Add a test case which demonstrates mcount overflow and the tunables.

    Document the new tunables in the manual.

    Signed-off-by: Simon Kissane <skissane@gmail.com>
    Reviewed-by: DJ Delorie <dj@redhat.com>
    (cherry picked from commit 31be941e4367c001b2009308839db5c67bf9dcbc)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

--- Comment #7 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The release/2.36/master branch has been updated by Florian Weimer
<fw@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=8920855c4568fa99c67cb1c7d6b29465c25f51c2

commit 8920855c4568fa99c67cb1c7d6b29465c25f51c2
Author: Simon Kissane <skissane@gmail.com>
Date:   Sat Feb 11 20:12:13 2023 +1100

    gmon: improve mcount overflow handling [BZ# 27576]

    When mcount overflows, no gmon.out file is generated, but no message is
printed
    to the user, leaving the user with no idea why, and thinking maybe there is
    some bug - which is how BZ 27576 ended up being logged. Print a message to
    stderr in this case so the user knows what is going on.

    As a comment in sys/gmon.h acknowledges, the hardcoded MAXARCS value is too
    small for some large applications, including the test case in that BZ.
Rather
    than increase it, add tunables to enable MINARCS and MAXARCS to be
overridden
    at runtime (glibc.gmon.minarcs and glibc.gmon.maxarcs). So if a user gets
the
    mcount overflow error, they can try increasing maxarcs (they might need to
    increase minarcs too if the heuristic is wrong in their case.)

    Note setting minarcs/maxarcs too large can cause monstartup to fail with an
    out of memory error. If you set them large enough, it can cause an integer
    overflow in calculating the buffer size. I haven't done anything to defend
    against that - it would not generally be a security vulnerability, since
these
    tunables will be ignored in suid/sgid programs (due to the SXID_ERASE
default),
    and if you can set GLIBC_TUNABLES in the environment of a process, you can
take
    it over anyway (LD_PRELOAD, LD_LIBRARY_PATH, etc). I thought about
modifying
    the code of monstartup to defend against integer overflows, but doing so is
    complicated, and I realise the existing code is susceptible to them even
prior
    to this change (e.g. try passing a pathologically large highpc argument to
    monstartup), so I decided just to leave that possibility in-place.

    Add a test case which demonstrates mcount overflow and the tunables.

    Document the new tunables in the manual.

    Signed-off-by: Simon Kissane <skissane@gmail.com>
    Reviewed-by: DJ Delorie <dj@redhat.com>
    (cherry picked from commit 31be941e4367c001b2009308839db5c67bf9dcbc)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

--- Comment #8 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The release/2.35/master branch has been updated by Florian Weimer
<fw@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=9f81b8fa65798233dff2794ff0d8d2d5d8062e8b

commit 9f81b8fa65798233dff2794ff0d8d2d5d8062e8b
Author: Simon Kissane <skissane@gmail.com>
Date:   Sat Feb 11 20:12:13 2023 +1100

    gmon: improve mcount overflow handling [BZ# 27576]

    When mcount overflows, no gmon.out file is generated, but no message is
printed
    to the user, leaving the user with no idea why, and thinking maybe there is
    some bug - which is how BZ 27576 ended up being logged. Print a message to
    stderr in this case so the user knows what is going on.

    As a comment in sys/gmon.h acknowledges, the hardcoded MAXARCS value is too
    small for some large applications, including the test case in that BZ.
Rather
    than increase it, add tunables to enable MINARCS and MAXARCS to be
overridden
    at runtime (glibc.gmon.minarcs and glibc.gmon.maxarcs). So if a user gets
the
    mcount overflow error, they can try increasing maxarcs (they might need to
    increase minarcs too if the heuristic is wrong in their case.)

    Note setting minarcs/maxarcs too large can cause monstartup to fail with an
    out of memory error. If you set them large enough, it can cause an integer
    overflow in calculating the buffer size. I haven't done anything to defend
    against that - it would not generally be a security vulnerability, since
these
    tunables will be ignored in suid/sgid programs (due to the SXID_ERASE
default),
    and if you can set GLIBC_TUNABLES in the environment of a process, you can
take
    it over anyway (LD_PRELOAD, LD_LIBRARY_PATH, etc). I thought about
modifying
    the code of monstartup to defend against integer overflows, but doing so is
    complicated, and I realise the existing code is susceptible to them even
prior
    to this change (e.g. try passing a pathologically large highpc argument to
    monstartup), so I decided just to leave that possibility in-place.

    Add a test case which demonstrates mcount overflow and the tunables.

    Document the new tunables in the manual.

    Signed-off-by: Simon Kissane <skissane@gmail.com>
    Reviewed-by: DJ Delorie <dj@redhat.com>
    (cherry picked from commit 31be941e4367c001b2009308839db5c67bf9dcbc)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27576] gmon.out not consistently created

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27576

--- Comment #9 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The release/2.34/master branch has been updated by Florian Weimer
<fw@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=4dd89b2a8fc91bc74ea85a442ae4c672b6dda113

commit 4dd89b2a8fc91bc74ea85a442ae4c672b6dda113
Author: Simon Kissane <skissane@gmail.com>
Date:   Sat Feb 11 20:12:13 2023 +1100

    gmon: improve mcount overflow handling [BZ# 27576]

    When mcount overflows, no gmon.out file is generated, but no message is
printed
    to the user, leaving the user with no idea why, and thinking maybe there is
    some bug - which is how BZ 27576 ended up being logged. Print a message to
    stderr in this case so the user knows what is going on.

    As a comment in sys/gmon.h acknowledges, the hardcoded MAXARCS value is too
    small for some large applications, including the test case in that BZ.
Rather
    than increase it, add tunables to enable MINARCS and MAXARCS to be
overridden
    at runtime (glibc.gmon.minarcs and glibc.gmon.maxarcs). So if a user gets
the
    mcount overflow error, they can try increasing maxarcs (they might need to
    increase minarcs too if the heuristic is wrong in their case.)

    Note setting minarcs/maxarcs too large can cause monstartup to fail with an
    out of memory error. If you set them large enough, it can cause an integer
    overflow in calculating the buffer size. I haven't done anything to defend
    against that - it would not generally be a security vulnerability, since
these
    tunables will be ignored in suid/sgid programs (due to the SXID_ERASE
default),
    and if you can set GLIBC_TUNABLES in the environment of a process, you can
take
    it over anyway (LD_PRELOAD, LD_LIBRARY_PATH, etc). I thought about
modifying
    the code of monstartup to defend against integer overflows, but doing so is
    complicated, and I realise the existing code is susceptible to them even
prior
    to this change (e.g. try passing a pathologically large highpc argument to
    monstartup), so I decided just to leave that possibility in-place.

    Add a test case which demonstrates mcount overflow and the tunables.

    Document the new tunables in the manual.

    Signed-off-by: Simon Kissane <skissane@gmail.com>
    Reviewed-by: DJ Delorie <dj@redhat.com>
    (cherry picked from commit 31be941e4367c001b2009308839db5c67bf9dcbc)

-- 
You are receiving this mail because:
You are on the CC list for the bug.