[Bug gcov-profile/105238] New: Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[Bug gcov-profile/105238] New: Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[esgergn at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

            Bug ID: 105238
           Summary: Regression: using -fprofile-dir: gcno files not ccache
                    cachable anymore / gcovr report broken
           Product: gcc
           Version: 9.4.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: gcov-profile
          Assignee: unassigned at gcc dot gnu.org
          Reporter: esgergn at hotmail dot com
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

Gist:

- gcc 9 / 10 (and later?) gcno files are not ccache cachable anymore because of
the absolute CWD path they contain
- when using -fprofile-dir with gcc 9 and 10 gcovr is not able to produce
correct reports where it can do so when not using -fprofile-dir.


Situation:

CMake project with multiple subprojects with the following structure
(condensed)

/home/esger/src/
    application/
        CMakeLists.txt
        source/
            staticlibA/
                CMakeLists.txt
                source/
                    A.cpp
            staticlibB/
                CMakeLists.txt
                source/
                    B.cpp
            main.cpp
        test/
            CMakeLists.txt
            source/
            test_main.cpp

We are building, with coverage, from different directories with often only
minor code changes between directories, therefore ccache is able to provide a
very significant speedup. When using ccache 4.5.1 (with a small bug fix, see
https://github.com/ccache/ccache/issues/1032) and gcc 7.5.0 this works
perfectly. When compiling for instance B.cpp we use flags:

-fprofile-dir=./build/source/staticlibB/CMakeFiles/staticlibB.dir/source
-fdebug-prefix-map=/home/esger/src/application=.

and

CCACHE_BASEDIR=/home/esger/src/application
(this rewrites the absolute paths CMake uses to relative paths starting at the
base dir. Ie.
/home/esger/src/application/build/source/staticlibB/CMakeFiles/staticlibB.dir/source/B.cpp
becomes ./build/source/staticlibB/CMakeFiles/staticlibB.dir/source/B.cpp)

with
/home/esger/src/application/build/source/staticlibB/CMakeFiles/staticlibB.dir/source
as cwd for the compilation of B.cpp.

This produces .o and .gcno files containing relative paths that can be cached
by ccache. When running ./build/bin/test_main from /home/esger/src/application
gcda files are created next to the gcno files. When compiling a second time
from a new directory /home/esger/src/application-2, the .o and .gcno are taken
from the cache and when running ./build/bin/test_main from
/home/esger/src/application-2 again gcda are created next to the .gcno files
below application-2.

Gcovr produces an identical coverage report for both dirs. So far so good ;)


When switching to gcc 9.3 however this does not work anymore and the coverage
reports are empty (even on builds were the cache was empty and no cached
.o/gcno's were used). I found
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97193 about incorrect mangling of
gcno files so tried gcc 9.4 as well which still does not work.

It seems the behavior of -fprofile-dir has changed according to the
documentation:

"-fprofile-dir=path

Set the directory to search for the profile data files in to path. This option
affects only the profile data generated by -fprofile-generate, -ftest-coverage,
-fprofile-arcs and used by -fprofile-use and -fbranch-probabilities and its
related options. Both absolute and relative paths can be used. By default, GCC
uses the current directory as path, thus the profile data file appears in the
same directory as the object file. In order to prevent the file name clashing,
if the object file name is not an absolute path, we mangle the absolute path of
the sourcename.gcda file and use it as the file name of a .gcda file." 


Note that the latter part where the absolute path of the source is mangled and
used as the file name is not helpful in our use case. We absolutely want a
relative path when specifying a relative path to -fprofile-dir and have no
absolute paths anywhere. I am certain in some cases there can be file name
clashing, but as far as I am aware that is not a problem at all in our use
case. (so it would be nice if this behavior at least can be turned off)

Added to that in 9.4 there seem to be more problems with -fprofile-dir? Given a
trivial main.cpp:

cwd = /home/esger/src/application/coverage_test/dir1/source

$ g++ --coverage main.cpp
$ ls
a.out  main.cpp  main.gcno
$ ./a.out
$ ls
a.out  main.cpp  main.gcda  main.gcno
$ gcovr --html -o coverage.html .
[coverage report OK]

$ g++ --coverage -fprofile-dir=. main.cpp
$ ls
a.out  main.cpp  main.gcno
$ ./a.out
$ ls
'#home#esger#src#application#coverage_test#dir1#source#main.gcda'   a.out  
main.cpp   main.gcno
$ gcovr --html -o coverage.html .
[coverage report broken, shows main.cpp but with 0% coverage]

$ g++ --coverage
-fprofile-dir=/home/esger/src/application/coverage_test/dir1/source main.cpp
$ ls
a.out  main.cpp  main.gcno
$ ./a.out
$ ls
'#home#esger#src#application#coverage_test#dir1#source#main.gcda'   a.out  
main.cpp   main.gcno
$ gcovr --html -o coverage.html .
[coverage report broken, shows main.cpp but with 0% coverage]

Either one of the last two should be the default value (when not specifying
-fprofile-dir) I'd imagine, but the behavior is different from the first case?


$ g++ --coverage -fprofile-dir=./somedir/ main.cpp
$ ls
a.out  main.cpp  main.gcno
$ ./a.out
$ ls
a.out  main.cpp  main.gcno  somedir
$ ls somedir
'#home#esger#src#application#coverage_test#dir1#source#main.gcda'
$ gcovr --html -o coverage.html .
[coverage report broken, shows main.cpp but with 0% coverage]

It seems that whenever -fprofile-dir is used the gcda name is always mangled
and gcovr (version 5.1) is not able to produce a correct report.


If I repeat the same four cases above with gcc 7.5.0 the first three all
produce gcda files named main.gcda and a correct coverage report. The last case
also produces a 0% coverage report (even though -v output seems to indicate it
is finding the source and gcno files, not sure what is happening there). Gcc
10.3 shows the same behavior as 9.4 for all four cases.


Back to our original use case, if I compare gcno files between gcc 7.5 and 9.4
it seems the 9.4 ones contain the cwd from compilation, which would bar them
from being cached (or when cached contain not-entirely-correct information).

gcc 7.5 (anonymised file from actual project):

$ gcov-dump somefile.cpp.gcno
somefile.cpp.gcno:note:magic `gcno':version `A75*'
somefile.cpp.gcno:stamp 3077156474
somefile.cpp.gcno: 01000000:  41:FUNCTION ident=57878848,
lineno_checksum=0xc0902347, cfg_checksum=0xc0bbb23e,
`_GLOBAL__sub_I__ZN6some9function12hereIN6with8somedEttE5nameE'
../../../../../source/common/source/somefile.cpp:22
somefile.cpp.gcno:  01410000:   4:BLOCKS 4 blocks
somefile.cpp.gcno:  01430000:   3:ARCS 1 arcs
somefile.cpp.gcno:  01430000:   5:ARCS 2 arcs
somefile.cpp.gcno:  01430000:   3:ARCS 1 arcs
somefile.cpp.gcno:  01450000:  21:LINES
...

Note the relative path to the source file.

gcc 9.4:

$ gcov-dump main.gcno
main.gcno:note:magic `gcno':version `A94*'
main.gcno:stamp 492126994
main.gcno:cwd: /home/esger/src/application/coverage_test/dir1/source
main.gcno:  01000000:  20:FUNCTION ident=1826217343,
lineno_checksum=0x36a9d8e5, cfg_checksum=0xc0bbb23e, `_GLOBAL__sub_I__Z5funcAi'
main.cpp:21:1-21:1, artificial
main.gcno:    01410000:   1:BLOCKS 4 blocks
main.gcno:    01430000:   3:ARCS 1 arcs
main.gcno:    01430000:   5:ARCS 2 arcs
main.gcno:    01430000:   3:ARCS 1 arcs
main.gcno:    01450000:   9:LINES
main.gcno:    01450000:   9:LINES
...

Contains undesired absolute path.


I don't think our use case (non-incremental builds from different directories
on a CI server with coverage for unittests) is at all esoteric or unusual but
as it stands with gcc 9.4 (or gcc 10.3) we either have to disable coverage or
disable ccache (and not use -fprofile-dir) and suffer much much longer build
times.

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://gcc.gnu.org/bugzill
                   |                            |a/show_bug.cgi?id=105063
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2022-04-12

--- Comment #1 from Martin Liška <marxin at gcc dot gnu.org> ---
Thanks for the bug report. I think you described very similar problem to
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063.

Can you check the suggested patch in the PR, please?

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[esgergn at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

--- Comment #2 from Esger Abbink <esgergn at hotmail dot com> ---
There certainly is overlap yes, in particular:

"(a) build directory relative paths were stored as the .gcda paths
(/a/b/c/master/path/to/file.gcda becomes path/to/file.gcda)"

from https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063#c8 is exactly what we
want (without mangling).

Where the build directory would then be /home/esger/src/application in the
situation described above.


I am happy to test the patch, but not quite sure how to proceed.
-fprofile-prefix-map is not a valid option for gcc 9 or 10 as far as I can
tell. To which gcc version/branch should I apply the patch? (the other issue
mentions a 10.3.x branch, but a clone of the git gcc source does not seem to
have such a branch?)

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

--- Comment #3 from Martin Liška <marxin at gcc dot gnu.org> ---
> I am happy to test the patch, but not quite sure how to proceed.

I would recommend building the current master (or you can wait for a couple of
weeks and apply it on the top of GCC 12.1 release that will happen soonish).
What distro do you use?

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[esgergn at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

--- Comment #4 from Esger Abbink <esgergn at hotmail dot com> ---
We use Ubuntu 20.04 currently for our jenkins images.

With gcc built from master with patch using -fprofile-prefix-map works as
expected:

with a main.cpp in /home/esger/src/application/coverage_test/dir1/source

compiling with:

cd /home/esger/src/application/coverage_test/dir1/source/ && g++-11 --coverage
-fprofile-prefix-map=/home/esger/src/application/coverage_test/dir1=dir2 -c
/home/esger/src/application/coverage_test/dir1/source/main.cpp 

cd /home/esger/src/application/coverage_test/dir1/source/ && g++-11 --coverage
-fprofile-prefix-map=/home/esger/src/application/coverage_test/dir1=dir2 main.o
-o ../bin/a.out

results in a main.gcno in the source directory, and after running
./dir1/bin/a.out from /home/esger/src/application/coverage_test there is a
main.gcda (non-mangled) in dir2/source.


Note that I was not able to produce a meaningful coverage report even for the
simplest/first case of the four above with gcc 11. I think gcovr might not yet
support newer formats used? So I can't currently verify if with
-fprofile-prefix-map the coverage report is ok, but I would assume so.

(Still, I don't quite understand why behavior that was working is now
broken/changed and needs a new option? ie. why is -fprofile-dir mangling the
name, and -fprofile-prefix-map not? and 9.4 behavior with -fprofile-dir still
seems broken when explicitly specifying the default?)


If I perform a small experiment more closely resembling what cmake + ccache do:

cd /home/esger/src/application/coverage_test/dir1/source/ &&
CCACHE_BASEDIR=/home/esger/src/application/coverage_test/dir1 ccache
/usr/bin/g++-11 --coverage
-fprofile-prefix-map=/home/esger/src/application/coverage_test/dir1=.
-fdebug-prefix-map=/home/esger/src/application/coverage_test/dir1=. -c
/home/esger/src/application/coverage_test/dir1/source/main.cpp

cd /home/esger/src/application/coverage_test/dir1/source/ &&
CCACHE_BASEDIR=/home/esger/src/application/coverage_test/dir1 ccache
/usr/bin/g++-11 --coverage
-fprofile-prefix-map=/home/esger/src/application/coverage_test/dir1=.
-fdebug-prefix-map=/home/esger/src/application/coverage_test/dir1=. main.o -o
../bin/a.out

and then from /home/esger/src/application/coverage_test/dir2 run
../dir1/bin/a.out (to simulate a run with a binary built from cached .o and
.gcno run from a different directory) again the gcda is created in the
dir2/source directory with a non-mangled name. yey :)

I can also see the source file paths in the gcno are relative:

main.gcno:note:magic `gcno':version `B20 '
main.gcno:stamp 625107654
main.gcno:checksum 0
main.gcno:cwd: /home/esger/src/application/coverage_test/dir1/source
main.gcno:  01000000:  83:FUNCTION ident=1482908279,
lineno_checksum=0x2c72f060, cfg_checksum=0xc0bbb23e, `_GLOBAL__sub_I__Z5funcAi'
./source/main.cpp:21:1-21:1, artificial
main.gcno:    01410000:   4:BLOCKS 4 blocks
main.gcno:    01430000:  12:ARCS 1 arcs
main.gcno:    01430000:  20:ARCS 2 arcs
main.gcno:    01430000:  12:ARCS 1 arcs
main.gcno:    01450000:  42:LINES
main.gcno:    01450000:  42:LINES
...

So with a patched gcc11, dropping -fprofile-dir and using -fprofile-prefix-map
instead, gcda creation seems ok. The gcno files seem to contain relative paths
to the source file too. However, the gcno file also still contains the
compilation CWD which interferes with caching it and re-using it for different
build dirs. Is there any way to get rid of that?

And of course I still need to verify such created gcda files will actually
result in a proper coverage report. Applying the patch to 9.4 or 10.3 as
vit9696 did should allow me to test that. Will try that tomorrow if I have
time.

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[esgergn at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

--- Comment #5 from Esger Abbink <esgergn at hotmail dot com> ---
Cherry picking
https://github.com/gcc-mirror/gcc/commit/cde87638bf5cf6aafffb590986b6a890da0ba06c
on top of releases-9.4.0 allowed me to apply the patch to get a patched gcc9.4
supporting -fprofile-prefix-map and that leads to the same (desired) behavior
as described in the previous comment for patched gcc11, plus gcovr is now able
to produce coverage reports.

However, all is still not well, because ccache does not know the
-fprofile-prefix-map option and therefore marks files compiled with it as
uncacheable.. :(

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[esgergn at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

--- Comment #6 from Esger Abbink <esgergn at hotmail dot com> ---
After patching ccache 4.5.1 to understand -fprofile-prefix-map it does cache .o
and .gcno:

$ ccache -z -C

$ cd /home/esger/src/application/coverage_test/dir1/source/ && CCACHE_DEBUG=
CCACHE_BASEDIR=/home/esger/src/application/coverage_test/dir1 ccache
/usr/bin/g++-9.4p --coverage
-fprofile-prefix-map=/home/esger/src/application/coverage_test/dir1=.
-fdebug-prefix-map=/home/esger/src/application/coverage_test/dir1=. -c
/home/esger/src/application/coverage_test/dir1/source/main.cpp

$ ccache -s
Summary:
  Hits:               0 /    1 (0.00 %)
    Direct:           0 /    1 (0.00 %)
    Preprocessed:     0 /    1 (0.00 %)
  Misses:             1
    Direct:           1
    Preprocessed:     1
Primary storage:
  Hits:               0 /    2 (0.00 %)
  Misses:             2
  Cache size (GB): 0.00 / 5.00 (0.00 %)
  Cleanups:           4

$ cd /home/esger/src/application/coverage_test/dir2/source/ && CCACHE_DEBUG=
CCACHE_BASEDIR=/home/esger/src/application/coverage_test/dir2 ccache
/usr/bin/g++-9.4p --coverage
-fprofile-prefix-map=/home/esger/src/application/coverage_test/dir2=.
-fdebug-prefix-map=/home/esger/src/application/coverage_test/dir2=. -c
/home/esger/src/application/coverage_test/dir2/source/main.cpp

$ ccache -s
Summary:
  Hits:               1 /    2 (50.00 %)
    Direct:           1 /    2 (50.00 %)
    Preprocessed:     0 /    1 (0.00 %)
  Misses:             1
    Direct:           1
    Preprocessed:     1
Primary storage:
  Hits:               2 /    4 (50.00 %)
  Misses:             2
  Cache size (GB): 0.00 / 5.00 (0.00 %)
  Cleanups:           4

Using the cached .o/.gcno also generates a gcda in the correct location:

$ cd /home/esger/src/application/coverage_test/dir2/source/ && CCACHE_DEBUG=
CCACHE_BASEDIR=/home/esger/src/application/coverage_test/dir2 ccache
/usr/bin/g++-9.4p --coverage
-fprofile-prefix-map=/home/esger/src/application/coverage_test/dir2=.
-fdebug-prefix-map=/home/esger/src/application/coverage_test/dir2=. main.o -o
../bin/a.out

$ cd /home/esger/src/application/coverage_test/dir2 && ./bin/a.out

$ ls source/
main.cpp  main.gcda  main.gcno  main.o

$ gcovr --html coverage.html .
[ coverage report OK ]



Though the gcno of source still contains a wrong CWD:

$ gcov-dump-9.4p source/main.gcno |more
main.gcno:note:magic `gcno':version `A94*'
main.gcno:stamp 761679940
main.gcno:cwd: /home/esger/src/application/coverage_test/dir1/source
main.gcno:  01000000:  20:FUNCTION ident=1826217343,
lineno_checksum=0x36a9d8e5, cfg_checksum=0xc0bbb23e, `_GLOBAL__sub_I__Z5funcAi'
main.cpp:21:1-21:1, artificial
main.gcno:    01410000:   1:BLOCKS 4 blocks
main.gcno:    01430000:   3:ARCS 1 arcs
main.gcno:    01430000:   5:ARCS 2 arcs
main.gcno:    01430000:   3:ARCS 1 arcs
main.gcno:    01450000:   9:LINES
main.gcno:    01450000:   9:LINES
...


Not sure though what (negative) impact that will/could have?

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[esgergn at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

--- Comment #7 from Esger Abbink <esgergn at hotmail dot com> ---
Building a real project with the patched gcc9.4 and ccache 4.5.1 (replacing
-fprofile-dir with -fprofile-prefix-map=${CMAKE_SOURCE_DIR}=.) it looks ok at
first sight:

- 100% cache hits for the second compilation in a different directory
- gcda created next to .o & .gcda for both first and second
compilations/directories

However when running the unittests binaries there are quite a number of gcov
warnings:

[==========] 2807 tests from 297 test suites ran. (272928 ms total)
[  PASSED  ] 2807 tests.

libgcov profiling error:
:overwriting an existing profile data with a different timestamp
libgcov profiling error:#:overwriting an existing profile data with a different
timestamp
libgcov profiling error:
:overwriting an existing profile data with a different timestamp
libgcov profiling error::overwriting an existing profile data with a different
timestamp
libgcov profiling error::overwriting an existing profile data with a different
timestamp
libgcov profiling error:#:overwriting an existing profile data with a different
timestamp
libgcov profiling error:&:overwriting an existing profile data with a different
timestamp
libgcov profiling error:#:overwriting an existing profile data with a different
timestamp
libgcov profiling error:#:overwriting an existing profile data with a different
timestamp
libgcov profiling error:':overwriting an existing profile data with a different
timestamp
profiling::Cannot open
libgcov profiling error:#:overwriting an existing profile data with a different
timestamp
libgcov profiling error:#:overwriting an existing profile data with a different
timestamp
libgcov profiling error::overwriting an existing profile data with a different
timestamp
...
[this continues for quite a number of lines]

Note that these warnings are produced both for the directory where compilation
really took place as for the directory where .o & .gcno were taken from the
cache.

Since the gcda files are created next to the .o * .gcno there should not be any
name clashes (for identical names the path would still be different). So I am
not sure what is happening here? Unfortunately, the warning lines do not
include any information about which file is being written or which file is
clashing.



Still, gcovr is able to produce a coverage report with information that looks
sane, but there are differences with the gcc7.5 reports. I would assume because
of the above warnings. 

The coverage report for the second compilation (using cached .o & .gcno) is for
all intents and purposes the same as the one for the first compilation. The
difference being 2 lines and 2 branches on totals of 17515 and 14727.

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |marxin at gcc dot gnu.org

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |patch

--- Comment #8 from Martin Liška <marxin at gcc dot gnu.org> ---
Patch candidate has been sent:
https://gcc.gnu.org/pipermail/gcc-patches/2023-April/615048.html

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |14.0

[Bug gcov-profile/105238] Regression: using -fprofile-dir: gcno files not ccache cachable anymore / gcovr report broken

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105238

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|14.0                        |14.2

--- Comment #9 from Richard Biener <rguenth at gcc dot gnu.org> ---
GCC 14.1 is being released, retargeting bugs to GCC 14.2.