[Bug gcov-profile/105063] New: [GCOV] Ability to map .gcda paths

[Bug gcov-profile/105063] New: [GCOV] Ability to map .gcda paths

[vit9696 at protonmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

            Bug ID: 105063
           Summary: [GCOV] Ability to map .gcda paths
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: gcov-profile
          Assignee: unassigned at gcc dot gnu.org
          Reporter: vit9696 at protonmail dot com
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

When compiling code with `-fprofile-arcs` gcc encodes .gcda file path relative
to object file. In some cases this is infeasible, as the compilation directory
may have extra long prefix, which is not practical to store on embedded
targets.

To avoid this issue we would like to replace this prefix with a custom value.
Currently there are 3 options in GCC, which allow one to do coverage file
mapping:

* -fprofile-dir
* -fprofile-prefix-map
* -fprofile-prefix-path

We assumed that at least one of them should do what we need, yet it is not
true:

* Normal gives /app/output.gcda (https://godbolt.org/z/jaj6vhnd5)
* -fprofile-dir=/test gives /test//app/output.gcda
(https://godbolt.org/z/8h7brYE84)
* -fprofile-prefix-map=/app=/test gives /app/output.gcda
(https://godbolt.org/z/c9TxeMPzM)
* -fprofile-prefix-path=/test -fprofile-generate=/app give
/app//app/output.gcda (https://godbolt.org/z/bh7esxPcK)

I see two routes to solve this issue:

1. One of the existing options is fixed to do this. E.g.
-fprofile-prefix-map=/app=/test could result in /test/output.gcda.
2. A new option is introduced to map gcda files.

The relevant code is in coverage.cc:coverage_init
(https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/coverage.cc;h=8ece5db680e614f8225d9e8407dd89bd27020b4d;hb=refs/heads/master#l1253).

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2022-03-28

--- Comment #1 from Martin Liška <marxin at gcc dot gnu.org> ---
Can you please provide a self-contained test-case that shows your problem?

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[vit9696 at protonmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #2 from vit9696 <vit9696 at protonmail dot com> ---
Sure, the example is published on godbolt, but for completeness:

int main() {}

gcc -fprofile-arcs -S t.c -o t.s
cat t.s | grep gcda

Adding the mentioned arguments (e.g. -fprofile-prefix-map=/home/user=/test)
shows that it is not possible to make the output contain the desired:

.string "/test/t.gcda

But it will remain to be something like:

.string "/home/user/t.gcda

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #3 from Martin Liška <marxin at gcc dot gnu.org> ---
> 
> But it will remain to be something like:
> 
> .string	"/home/user/t.gcda

Well, -fprofile-prefix-map is about the mapping of the source file.

So what exactly is your problem? Are the long filenames for .gcda files? What
path length limitation does have an embedded target?

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[vit9696 at protonmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #4 from vit9696 <vit9696 at protonmail dot com> ---
Path length limitation in the current case is 200 bytes, but in general the
issue is that we would like _to be able to properly set the gcda path for the
target_. Currently the specified path may not even exist when cross-compiling,
and no approach to change it without prepending to the build path is simply
inconvenient.

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #5 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to vit9696 from comment #4)
> Path length limitation in the current case is 200 bytes, but in general the
> issue is that we would like _to be able to properly set the gcda path for
> the target_.(In reply to vit9696 from comment #4)
> Path length limitation in the current case is 200 bytes, but in general the
> issue is that we would like _to be able to properly set the gcda path for
> the target_. Currently the specified path may not even exist when
> cross-compiling, and no approach to change it without prepending to the
> build path is simply inconvenient.

Well, if you do a cross-compilation, then you can use -fprofile-dir=/tmp/foobar
and then use -fprofile-use on the same system where you run the instrumentation
run.

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[vit9696 at protonmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #6 from vit9696 <vit9696 at protonmail dot com> ---
While true, this scenario is simply inconvenient in many cases.

(1) When filesystem path limitations exist, they will unavoidably lead to being
unable to save data due to extra large resulting paths. To remind you, even
Windows, a desktop os, had 260 path length limit not so long ago.
(2) When analysing gcda files on a different machine one needs to be aware of
the original paths on the machine used for compilation, and we do not want to
require this knowledge.

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #7 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to vit9696 from comment #6)
> While true, this scenario is simply inconvenient in many cases.
> 
> (1) When filesystem path limitations exist, they will unavoidably lead to
> being unable to save data due to extra large resulting paths. To remind you,
> even Windows, a desktop os, had 260 path length limit not so long ago.

Sure, well, I can imagine introducing something similar to what we have for
gcov:

$ gcov --help | grep hash
  -x, --hash-filenames            Hash long pathnames

> (2) When analysing gcda files on a different machine one needs to be aware
> of the original paths on the machine used for compilation, and we do not
> want to require this knowledge.

Well, I don't fully understand your needs. On one hand, you're unhappy about
the relative placement of the .gcda files.
And on the other, you don't like -fprofile-dir=/path. Note you can copy the
*.gcda files to an arbitrary location and use it for -fprofile-use=/some/path.
That does not require knowledge of -fprofile-generate=/original/path.

I'm sorry for defensive argumentation, but believe me the .gcda filename
handling is very complex even in the current situation.

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[vit9696 at protonmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #8 from vit9696 <vit9696 at protonmail dot com> ---
> Sure, well, I can imagine introducing something similar to what we have for gcov:
> 
> $ gcov --help | grep hash
>   -x, --hash-filenames            Hash long pathnames

Yes, that would likely solve the issue, though in an unexpected way for me.
However, while it is highly unlikely, I can expect collisions in this case, and
would rather not risk it.

> Well, I don't fully understand your needs. On one hand, you're unhappy about the relative placement of the .gcda files.
And on the other, you don't like -fprofile-dir=/path. Note you can copy the
*.gcda files to an arbitrary location and use it for -fprofile-use=/some/path.
That does not require knowledge of -fprofile-generate=/original/path.

> I'm sorry for defensive argumentation, but believe me the .gcda filename handling is very complex even in the current situation.

Sorry. I guess there are multiple issues that arose on our side, which also
made the report rather confusing.

What we have here is a CI building code on different nodes at different paths
(they are different, because multiple builds can run on the same node).
Depending on various meta information like node name, branch name, and so on,
which is present in the build prefix, the gcda files we get may be too long and
randomly exceed the 200 byte path limitation we have.

The thing is, our source paths do not exceed this limitation, and the problem
is just with the prefix, which we would like to entirely strip. It would have
worked fine if:
(a) build directory relative paths were stored as the .gcda paths
(/a/b/c/master/path/to/file.gcda becomes path/to/file.gcda) or
(b) special prefix + build directory relative paths were stored as the .gcda
paths (/a/b/c/master/path/to/file.gcda becomes /special/path/to/file.gcda).

Thus the need for the option.

Going this way also allows us to resolve a second issue. To speedup testing we
prebuild most of the source code on one machine, and then distribute the
resulting .o and .a files to other machines. Currently this is unimplemented
for coverage, but we would also like to distribute .gcno files the same way,
and avoid the need to compile on the node when doing coverage gathering.

The issue here is that different processes on the target may share some object
code (e.g. libraries). Dumping coverage for different processes means that we
have to produce multiple .gcda files on the node from the same .gcda paths in
the object code. I.e. they not only depend on the build node path, but also the
process we are dumping the information from. To be able to produce an adequate
.gcda tree on the node with the current implementation we would have to
remember all the original build node prefixes of the library code. Otherwise we
would be unable to remap build node path to a run node path for a particular
process (to get the .gcno files for instance).

However, if all userspace code is compiled with e.g. /special prefix, and the
original build prefix is stripped, only one path needs to be handled:

/special/lib1/path/to/file1.gcda
/special/lib2/path/to/file2.gcda
/special/lib3/path/to/file3.gcda
/special/path/to/source.gcda

Hope this makes the situation clearer, and helps to understand why the
particular option is desired.

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW

--- Comment #9 from Martin Liška <marxin at gcc dot gnu.org> ---
Ok, so let's use the existing -fprofile-prefix-map and use it for the
modification of where .gcda files are stored.
Can you please prepare a patch that will fulfill your needs?

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[vit9696 at protonmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #10 from vit9696 <vit9696 at protonmail dot com> ---
I believe this is possible. Since we use both clang and gcc, I filed an issue
in llvm first to make sure both compilers can be updated in a similar way
(https://github.com/llvm/llvm-project/issues/54670).

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #11 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to vit9696 from comment #10)
> I believe this is possible. Since we use both clang and gcc, I filed an
> issue in llvm first to make sure both compilers can be updated in a similar
> way (https://github.com/llvm/llvm-project/issues/54670).

Thanks. Btw. what's the name of an embedded project where you need the
functionality?

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[vit9696 at protonmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #12 from vit9696 <vit9696 at protonmail dot com> ---
It is an in-house airborne RTOS we develop in ISP RAS. There is a legacy paper
in English telling a bit more about the project
(https://www.ispras.ru/proceedings/docs/2016/28/2/isp_28_2016_2_181.pdf).

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #13 from Martin Liška <marxin at gcc dot gnu.org> ---
Created attachment 52724
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52724&action=edit
Tentative patch

Can you please experiment with the following patch?

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[vit9696 at protonmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

--- Comment #14 from vit9696 <vit9696 at protonmail dot com> ---
I have just tested this patch after rebasing it on 10.3.x branch, and can
confirm it works as intended. Thank you!

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |marxin at gcc dot gnu.org

--- Comment #15 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to vit9696 from comment #14)
> I have just tested this patch after rebasing it on 10.3.x branch, and can
> confirm it works as intended. Thank you!

Good to know, so let's see what Clang folks think about the option name.

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |patch

--- Comment #16 from Martin Liška <marxin at gcc dot gnu.org> ---
Patch candidate has been sent:
https://gcc.gnu.org/pipermail/gcc-patches/2023-April/615048.html

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |14.0

[Bug gcov-profile/105063] [GCOV] Ability to map .gcda paths

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105063

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|14.0                        |14.2

--- Comment #17 from Richard Biener <rguenth at gcc dot gnu.org> ---
GCC 14.1 is being released, retargeting bugs to GCC 14.2.