[Bug gcov-profile/94928] New: Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[Bug gcov-profile/94928] New: Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

            Bug ID: 94928
           Summary: Doc comments in gcov-io.h do not show cwd and unexec
                    blocks in the Notes file format
           Product: gcc
           Version: 11.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: gcov-profile
          Assignee: unassigned at gcc dot gnu.org
          Reporter: myron.walker at gmail dot com
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

The documentation of the notes file format below do not show the cwd or
unexec_blocks data items.

        note: unit function-graph*
        unit: header int32:checksum string:source
        function-graph: announce_function basic_blocks {arcs | lines}*
        announce_function: header int32:ident
                int32:lineno_checksum int32:cfg_checksum
                string:name string:source int32:start_lineno int32:start_column 
                int32:end_lineno
        basic_block: header int32:flags*
        arcs: header int32:block_no arc*
        arc:  int32:dest_block int32:flags
        lines: header int32:block_no line*
               int32:0 string:NULL
        line:  int32:line_no | int32:0 string:filename


When you look at the gcov.c file, you can see that at the beginning of the
notes file right after the stamp, there is an item for cwd and unexec blocks.

  bbg_stamp = gcov_read_unsigned();
  bbg_cwd = xstrdup(gcov_read_string());                 <<<<< Not documented
  bbg_supports_has_unexecuted_blocks = gcov_read_unsigned();   <<<<<< Not
documented


There also is a missing int32:artificial that should be between the function
name and function source file.

      function_name = xstrdup(gcov_read_string());
      unsigned artificial = gcov_read_unsigned();      <<<< Not documented
      unsigned src_idx = find_source(gcov_read_string());

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |11.0
     Ever confirmed|0                           |1
           Assignee|unassigned at gcc dot gnu.org      |marxin at gcc dot gnu.org
             Status|UNCONFIRMED                 |ASSIGNED
   Last reconfirmed|                            |2020-05-04

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #1 from Martin Liška <marxin at gcc dot gnu.org> ---
You are right, the documentation is not complete.
Btw. are you parsing a .gcda or .gcna format for some reason?

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #2 from Myron Walker <myron.walker at gmail dot com> ---
I am parsinv both gcno and gcda files.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #3 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Myron Walker from comment #2)
> I am parsinv both gcno and gcda files.

These files are not intended to be parsed :/
Can you please describe your use-case?

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #4 from Myron Walker <myron.walker at gmail dot com> ---
A python tool that can do distributed code coverage analysis.  Gcda files from
cluster nodes from a web interface, gcno from a web interface or file share in
a build archive, and source directly from github.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #5 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Myron Walker from comment #4)
> A python tool that can do distributed code coverage analysis.  Gcda files
> from cluster nodes from a web interface, gcno from a web interface or file
> share in a build archive, and source directly from github.

Can you please use gcov --json-format:
https://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html#Invoking-Gcov
?

What kind of information do you need to get from these files? Is it about
finding a corresponding files?

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #6 from Myron Walker <myron.walker at gmail dot com> ---
I use the gcno file to build a the graph, pull counters from the gcda files and
then solve the graph for the missing counts.  I am merging the data from
multiple gcda sources.  Multiple nodes running the same software.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #7 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Myron Walker from comment #6)
> I use the gcno file to build a the graph, pull counters from the gcda files
> and then solve the graph for the missing counts.

That's what gcov does itself.

> I am merging the data from
> multiple gcda sources.  Multiple nodes running the same software.

I would recommend writing a simple merging tool on top of the JSON files. That
will save you a lot of time.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #8 from Martin Liška <marxin at gcc dot gnu.org> ---
Or even better: you can merge various .gcda files with:
gcov-tool merge ...
https://gcc.gnu.org/onlinedocs/gcc/Gcov-tool-Intro.html

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #9 from Myron Walker <myron.walker at gmail dot com> ---
How you I process data files from multiple sources and multiple runs with gcov.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #10 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Myron Walker from comment #9)
> How you I process data files from multiple sources and multiple runs with
> gcov.

$ man gcov-tool

$ gcov-tool merge [merge-options] directory1 directory2

So you basically take 2 folders of 2 runs and merge them into a destination
one.
The folders are traversed for .gcda files and corresponding files are merged.
Having N runs, you need to run log2(N) merge operations.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #11 from Myron Walker <myron.walker at gmail dot com> ---
Ok.  I'll look into it

On Wed, May 6, 2020, 7:25 AM marxin at gcc dot gnu.org <
gcc-bugzilla@gcc.gnu.org> wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928
>
> --- Comment #10 from Martin Liška <marxin at gcc dot gnu.org> ---
> (In reply to Myron Walker from comment #9)
> > How you I process data files from multiple sources and multiple runs with
> > gcov.
>
> $ man gcov-tool
>
> $ gcov-tool merge [merge-options] directory1 directory2
>
> So you basically take 2 folders of 2 runs and merge them into a destination
> one.
> The folders are traversed for .gcda files and corresponding files are
> merged.
> Having N runs, you need to run log2(N) merge operations.
>
> --
> You are receiving this mail because:
> You reported the bug.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #12 from Myron Walker <myron.walker at gmail dot com> ---
What would be helpful then is if gcno, gcda and source files could all have
separate root file system prefixes.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #13 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Myron Walker from comment #12)
> What would be helpful then is if gcno, gcda and source files could all have
> separate root file system prefixes.

Can you please describe more the situation?
Is it something you can handle with
https://gcc.gnu.org/onlinedocs/gcc/Cross-profiling.html#Cross-profiling
?
Do you build your object files with relative paths or absolute?

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #14 from Myron Walker <myron.walker at gmail dot com> ---
There are three types of files used to create a code coverage report.  Notes,
Data, and Source.  It is likely that each type of file might have its own
prefix GCOV_SRC_PREFIX, GCOV_NOTES_PREFIX and GCOC_DATA_PREFIX

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #15 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Myron Walker from comment #14)
> There are three types of files used to create a code coverage report. 
> Notes, Data, and Source.  It is likely that each type of file might have its
> own prefix GCOV_SRC_PREFIX, GCOV_NOTES_PREFIX and GCOC_DATA_PREFIX

How difficult would it be to arrange all 3 types together? I mean x.gcda y.gcno
files should be in a same folder. With source files, it's more difficult as
they can be compiled in a prefixed directory, but e.g. system header files are
not prepended with a prefix, right?

The easiest approach seem to me copying .gcda and .gcno files into
corresponding location in source file directory.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #16 from Myron Walker <myron.walker at gmail dot com> ---
Just seems easier if the tool offers a choice.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #17 from Martin Liška <marxin at gcc dot gnu.org> ---
All right, I'll prepare a patch for that.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #18 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Martin Liska <marxin@gcc.gnu.org>:

https://gcc.gnu.org/g:10a9bf806cf180915c20f9971d33da8ff2d663c1

commit r11-260-g10a9bf806cf180915c20f9971d33da8ff2d663c1
Author: Martin Liska <mliska@suse.cz>
Date:   Mon May 11 09:25:46 2020 +0200

    Add caveat about parsing of .gcda and .gcno files.

            PR gcov-profile/94928
            * gcov-io.h: Add caveat about coverage format parsing and
            possible outdated documentation.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #19 from Martin Liška <marxin at gcc dot gnu.org> ---
Looking at the source file paths for tramp3d, there are quite some system
headers:

gcov tramp3d-v4.ii 2>/dev/null | sort | grep find_source | uniq -c | sort -n
      2 find_source: /usr/include/c++/6/iostream
      3 find_source: /usr/include/c++/6/bits/char_traits.h
      3 find_source: /usr/include/c++/6/ext/type_traits.h
      3 find_source: /usr/include/c++/6/iomanip
      5 find_source: /usr/include/c++/6/new
      6 find_source: /usr/include/c++/6/cmath
      6 find_source: /usr/include/c++/6/limits
     15 find_source: /usr/include/c++/6/bits/stl_function.h
     15 find_source: /usr/include/c++/6/bits/stl_iterator_base_types.h
     20 find_source: /usr/include/c++/6/bits/allocated_ptr.h
     22 find_source: /usr/include/c++/6/bits/basic_string.tcc
     22 find_source: /usr/include/c++/6/bits/predefined_ops.h
     32 find_source: /usr/include/c++/6/bits/basic_string.h
     32 find_source: /usr/include/c++/6/bits/list.tcc
     48 find_source: /usr/include/c++/6/ext/aligned_buffer.h
     53 find_source: /usr/include/c++/6/bits/stl_map.h
     55 find_source: /usr/include/c++/6/bits/stl_pair.h
     60 find_source: /usr/include/c++/6/ext/alloc_traits.h
     68 find_source: /usr/include/c++/6/bits/stl_iterator_base_funcs.h
     76 find_source: /usr/include/c++/6/bits/stl_heap.h
     78 find_source: /usr/include/c++/6/bits/cpp_type_traits.h
    130 find_source: /usr/include/c++/6/bits/stl_algo.h
    198 find_source: /usr/include/c++/6/bits/allocator.h
    205 find_source: /usr/include/c++/6/bits/stl_list.h
    253 find_source: /usr/include/c++/6/bits/move.h
    269 find_source: /usr/include/c++/6/bits/stl_bvector.h
    443 find_source: /usr/include/c++/6/bits/alloc_traits.h
    445 find_source: /usr/include/c++/6/bits/stl_construct.h
    582 find_source: /usr/include/c++/6/bits/stl_tree.h
    604 find_source: /usr/include/c++/6/ext/new_allocator.h
    779 find_source: /usr/include/c++/6/bits/stl_algobase.h
    909 find_source: /usr/include/c++/6/bits/stl_iterator.h
    992 find_source: /usr/include/c++/6/bits/vector.tcc
   1034 find_source: /usr/include/c++/6/bits/stl_uninitialized.h
   2145 find_source: /usr/include/c++/6/bits/stl_vector.h
  59079 find_source: tramp3d-v4.cpp

I bet one doesn't want to prefix all the system files with a prefix. That said,
I am not planning to add the option for prefixes.
Please open a separate bug with a feature request if you really need it. It
seems to me that merging the 3 components is an easy task (compared to parsing
of .gcda/.gcno files).

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #20 from Myron Walker <myron.walker at gmail dot com> ---
So I will probably continue working on the python script or package that can do
much of the enhanced processing.  One of the benefits of the python is I can
process the code coverage data outside of a build environment altogether.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #21 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Myron Walker from comment #20)
> So I will probably continue working on the python script or package that can
> do much of the enhanced processing.  One of the benefits of the python is I
> can process the code coverage data outside of a build environment altogether.

Can you please show me what the script does? I might be interested and for the
future I may add the options.
Right now, it's quite low on my TODO list..

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[myron.walker at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #22 from Myron Walker <myron.walker at gmail dot com> ---
It does the same things a gcov and lcov  combined but in python.  It also does
merging of data but in a different way than gcov-tool.  I might need to change
that.  Another part of it is to allow access to different types of resource
location hinting.  so a gcov prefix for a source code might be a github url and
token.  A gcno file hint might be a web url or sub or nfs share.  Like wise the
data file hints might be http, smb or nfs prefixes.

https://github.com/myronww/pycover

Still a work in progress though.

[Bug gcov-profile/94928] Doc comments in gcov-io.h do not show cwd and unexec blocks in the Notes file format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94928

--- Comment #23 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Myron Walker from comment #22)
> It does the same things a gcov and lcov  combined but in python.  It also
> does merging of data but in a different way than gcov-tool.  I might need to
> change that.

Yes, please use gcov-tool for the merging.

> Another part of it is to allow access to different types of
> resource location hinting.  so a gcov prefix for a source code might be a
> github url and token.  A gcno file hint might be a web url or sub or nfs
> share.  Like wise the data file hints might be http, smb or nfs prefixes.

That should be done by your script. I realized that for the JSON format you
only need to put together .gcda and .gcno files:

$ ls
tramp3d-v4.gcda  tramp3d-v4.gcno
$ gcov tramp3d-v4.gcda -i
...
$ gunzip tramp3d-v4.gcda.gcov.json.gz
$ cat tramp3d-v4.gcda.gcov.json | python -m json.tool | grep '"file"'
            "file": "tramp3d-v4.cpp",
            "file": "/usr/include/c++/6/ext/new_allocator.h",
            "file": "/usr/include/c++/6/ext/aligned_buffer.h",
            "file": "/usr/include/c++/6/bits/move.h",
            "file": "/usr/include/c++/6/bits/alloc_traits.h",
            "file": "/usr/include/c++/6/bits/stl_list.h",
            "file": "/usr/include/c++/6/bits/allocator.h",
            "file": "/usr/include/c++/6/bits/allocated_ptr.h",
            "file": "/usr/include/c++/6/bits/stl_iterator.h",
            "file": "/usr/include/c++/6/bits/list.tcc",
            "file": "/usr/include/c++/6/bits/stl_vector.h",
            "file": "/usr/include/c++/6/iostream",
            "file": "/usr/include/c++/6/bits/stl_construct.h",
            "file": "/usr/include/c++/6/bits/stl_uninitialized.h",
            "file": "/usr/include/c++/6/bits/vector.tcc",
            "file": "/usr/include/c++/6/bits/stl_algobase.h",
            "file": "/usr/include/c++/6/bits/stl_pair.h",
            "file": "/usr/include/c++/6/bits/cpp_type_traits.h",
            "file": "/usr/include/c++/6/bits/stl_bvector.h",
            "file": "/usr/include/c++/6/ext/alloc_traits.h",
            "file": "/usr/include/c++/6/bits/predefined_ops.h",
            "file": "/usr/include/c++/6/bits/stl_heap.h",
            "file": "/usr/include/c++/6/bits/stl_iterator_base_funcs.h",
            "file": "/usr/include/c++/6/bits/stl_iterator_base_types.h",
            "file": "/usr/include/c++/6/bits/stl_tree.h",
            "file": "/usr/include/c++/6/bits/stl_algo.h",
            "file": "/usr/include/c++/6/ext/type_traits.h",
            "file": "/usr/include/c++/6/bits/stl_function.h",
            "file": "/usr/include/c++/6/bits/basic_string.tcc",
            "file": "/usr/include/c++/6/bits/basic_string.h",
            "file": "/usr/include/c++/6/bits/stl_map.h",
            "file": "/usr/include/c++/6/iomanip",
            "file": "/usr/include/c++/6/limits",
            "file": "/usr/include/c++/6/new",
            "file": "/usr/include/c++/6/bits/char_traits.h",
            "file": "/usr/include/c++/6/cmath",

And now your script can find and get the corresponding source files.

> 
> https://github.com/myronww/pycover
> 
> Still a work in progress though.