[Bug gcov-profile/96534] New: keep gcov intermediate format

[Bug gcov-profile/96534] New: keep gcov intermediate format

[xlwu at synopsys dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96534

            Bug ID: 96534
           Summary: keep gcov intermediate format
           Product: gcc
           Version: 9.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: gcov-profile
          Assignee: unassigned at gcc dot gnu.org
          Reporter: xlwu at synopsys dot com
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

since gcc9, the gcov did not support intermediate format and replace with json
format , our application deeply depend on intermediate format , is it possible
to restore the intermediate format ? or could you let me know any workaround ? 

background :
I am a software engineer from SNPS , we have an application that extract all
file+functions and it's related lines per test case, and we save this data into
DB which help us to predict the test to verify when RnD updated any lines in
source code before their check in. we call this application as "smart
regression". 

now , when gcov move to json file, it increase the size a lot which affect the
efficiency to parse the data , what's worse, we had to revise our code to
support the new json format while we need to support the old format in the same
time , as our company have many products and each product have many live
branches , some of them still using gcc6 version.

I tried to use older gcov version on the new gcc instructed gcda and gcno file
, it did not work.

[Bug gcov-profile/96534] keep gcov intermediate format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96534

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |INVALID

--- Comment #1 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to xlwu from comment #0)
> since gcc9, the gcov did not support intermediate format and replace with
> json format , our application deeply depend on intermediate format , is it
> possible to restore the intermediate format ?

No, the support was removed from the GCC.

> or could you let me know any
> workaround ? 

Sure, it's very easy. One can get the very same information from the new JSON
format (which is fairly simple):

https://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html

> 
> background :
> I am a software engineer from SNPS , we have an application that extract all
> file+functions and it's related lines per test case, and we save this data
> into DB which help us to predict the test to verify when RnD updated any
> lines in source code before their check in. we call this application as
> "smart regression". 

That's a nice usage of the GCOV and the new format should simplify your life.

> 
> now , when gcov move to json file, it increase the size a lot which affect
> the efficiency to parse the data

I'm aware of the limitation. Note that the JSON format is compressed with gzip.
Can you please share more statistics?

> , what's worse, we had to revise our code
> to support the new json format while we need to support the old format in
> the same time , as our company have many products and each product have many
> live branches , some of them still using gcc6 version.

I see. As mentioned, porting to the new format should be fairly simple.

> 
> I tried to use older gcov version on the new gcc instructed gcda and gcno
> file , it did not work.

Please don't do it.

[Bug gcov-profile/96534] keep gcov intermediate format

[xlwu at synopsys dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96534

--- Comment #2 from xlwu at synopsys dot com ---
(In reply to Martin Liška from comment #1)
> (In reply to xlwu from comment #0)
> > since gcc9, the gcov did not support intermediate format and replace with
> > json format , our application deeply depend on intermediate format , is it
> > possible to restore the intermediate format ?
> 
> No, the support was removed from the GCC.
> 
> > or could you let me know any
> > workaround ? 
> 
> Sure, it's very easy. One can get the very same information from the new
> JSON format (which is fairly simple):
> 
> https://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html
> 
> > 
> > background :
> > I am a software engineer from SNPS , we have an application that extract all
> > file+functions and it's related lines per test case, and we save this data
> > into DB which help us to predict the test to verify when RnD updated any
> > lines in source code before their check in. we call this application as
> > "smart regression". 
> 
> That's a nice usage of the GCOV and the new format should simplify your life.
> 
> > 
> > now , when gcov move to json file, it increase the size a lot which affect
> > the efficiency to parse the data
> 
> I'm aware of the limitation. Note that the JSON format is compressed with
> gzip.
> Can you please share more statistics?

I did not have the real statistics yet as our company is not yet moved to gcc9
(maybe end of this year ).

and even the size of the compressed file is smaller , but we have to unzip and
parse it , right? it just increase the workload . 

however, the stdout option ( -t ) could be very helpful , we don't need to
write to disk and just parse the stdout content.

well, I also notice that the "function_name" in "lines" object is mangled even
I use "--demangled-names" option, can you print the demangled_name when "-m"
option given ? ( yes I know there is demangled name in "functions" part ) . the
reason is: we need to know the exact function name and it's lines, in current
format , we have to find all demangled function name and it's start and end
line number, then check the lines part to assign each lines into functions .

> 
> > , what's worse, we had to revise our code
> > to support the new json format while we need to support the old format in
> > the same time , as our company have many products and each product have many
> > live branches , some of them still using gcc6 version.
> 
> I see. As mentioned, porting to the new format should be fairly simple.
> 
> > 
> > I tried to use older gcov version on the new gcc instructed gcda and gcno
> > file , it did not work.
> 
> Please don't do it.

[Bug gcov-profile/96534] keep gcov intermediate format

[xlwu at synopsys dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96534

--- Comment #3 from xlwu at synopsys dot com ---
(In reply to xlwu from comment #2)
> (In reply to Martin Liška from comment #1)
> > (In reply to xlwu from comment #0)
> > > since gcc9, the gcov did not support intermediate format and replace with
> > > json format , our application deeply depend on intermediate format , is it
> > > possible to restore the intermediate format ?
> > 
> > No, the support was removed from the GCC.
> > 
> > > or could you let me know any
> > > workaround ? 
> > 
> > Sure, it's very easy. One can get the very same information from the new
> > JSON format (which is fairly simple):
> > 
> > https://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html
> > 
> > > 
> > > background :
> > > I am a software engineer from SNPS , we have an application that extract all
> > > file+functions and it's related lines per test case, and we save this data
> > > into DB which help us to predict the test to verify when RnD updated any
> > > lines in source code before their check in. we call this application as
> > > "smart regression". 
> > 
> > That's a nice usage of the GCOV and the new format should simplify your life.
> > 
> > > 
> > > now , when gcov move to json file, it increase the size a lot which affect
> > > the efficiency to parse the data
> > 
> > I'm aware of the limitation. Note that the JSON format is compressed with
> > gzip.
> > Can you please share more statistics?
> 
> I did not have the real statistics yet as our company is not yet moved to
> gcc9 (maybe end of this year ).
> 
> and even the size of the compressed file is smaller , but we have to unzip
> and parse it , right? it just increase the workload . 
> 
> however, the stdout option ( -t ) could be very helpful , we don't need to
> write to disk and just parse the stdout content.
> 
> well, I also notice that the "function_name" in "lines" object is mangled
> even I use "--demangled-names" option, can you print the demangled_name when
> "-m" option given ? ( yes I know there is demangled name in "functions" part
> ) . the reason is: we need to know the exact function name and it's lines,
> in current format , we have to find all demangled function name and it's
> start and end line number, then check the lines part to assign each lines
> into functions .

OK, maybe this is not a good idea , I guess this will increase the size much
larger 

> > 
> > > , what's worse, we had to revise our code
> > > to support the new json format while we need to support the old format in
> > > the same time , as our company have many products and each product have many
> > > live branches , some of them still using gcc6 version.
> > 
> > I see. As mentioned, porting to the new format should be fairly simple.
> > 
> > > 
> > > I tried to use older gcov version on the new gcc instructed gcda and gcno
> > > file , it did not work.
> > 
> > Please don't do it.

[Bug gcov-profile/96534] keep gcov intermediate format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96534

--- Comment #4 from Martin Liška <marxin at gcc dot gnu.org> ---
> I did not have the real statistics yet as our company is not yet moved to
> gcc9 (maybe end of this year ).
> 
> and even the size of the compressed file is smaller , but we have to unzip
> and parse it , right? it just increase the workload . 

Yes, but if it's problem we can start using e.g. zstd that has super-fast
decompression.

> 
> however, the stdout option ( -t ) could be very helpful , we don't need to
> write to disk and just parse the stdout content.

Yes, that be quite useful for your test-case. Or you may consider using:
https://github.com/RPGillespie6/fastcov

> 
> well, I also notice that the "function_name" in "lines" object is mangled
> even I use "--demangled-names" option, can you print the demangled_name when
> "-m" option given ? ( yes I know there is demangled name in "functions" part
> ) . the reason is: we need to know the exact function name and it's lines,
> in current format , we have to find all demangled function name and it's
> start and end line number, then check the lines part to assign each lines
> into functions .

It's handy to use mangled name as a unique key from "lines" to "functions".

[Bug gcov-profile/96534] keep gcov intermediate format

[xlwu at synopsys dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96534

--- Comment #5 from xlwu at synopsys dot com ---
A bug found in the JSON format:

in the lines.count and functions.execute_count fields, when the number is
extremely large , it will be shown as something like:  8.12123199e+08 , even
become a minus number , like -8.1332322e+08 . this is not a legal JSON format ,
when I try to unmarshel in golang , it just crash, I believe other language
will crash too

another question is:  in our environment , the gcda and gcno files are not
under same dir , to run the gcov , I had to copy them into a temp dir , is it
possible to provide options to give the gcda and gcno path in command line?

[Bug gcov-profile/96534] keep gcov intermediate format

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96534

--- Comment #6 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to xlwu from comment #5)
> A bug found in the JSON format:
> 
> in the lines.count and functions.execute_count fields, when the number is
> extremely large , it will be shown as something like:  8.12123199e+08 , even
> become a minus number , like -8.1332322e+08 . this is not a legal JSON
> format , when I try to unmarshel in golang , it just crash, I believe other
> language will crash too

That should be fixed in r10-4051-g076222782e4558e0bd22711c6fdcc0faf0882daf and
part of GCC 10.x release. Can you please verify that?

> 
> another question is:  in our environment , the gcda and gcno files are not
> under same dir

How that happens?

> , to run the gcov , I had to copy them into a temp dir , is
> it possible to provide options to give the gcda and gcno path in command
> line?

Right now, it's not possible. Can you please experiment with -o option?