[Bug gcov-profile/115047] New: Inconsistent MC/DC reported by GCC and LLVM

[Bug gcov-profile/115047] New: Inconsistent MC/DC reported by GCC and LLVM

[wentaoz5 at illinois dot edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115047

            Bug ID: 115047
           Summary: Inconsistent MC/DC reported by GCC and LLVM
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: gcov-profile
          Assignee: unassigned at gcc dot gnu.org
          Reporter: wentaoz5 at illinois dot edu
  Target Milestone: ---

How to reproduce:

$ gcc -v

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/gcc-latest/libexec/gcc/x86_64-pc-linux-gnu/14.0.1/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: ../configure --prefix=/opt/gcc-latest --enable-languages=c,c++
--enable-libstdcxx-debug --enable-libstdcxx-backtrace --disable-bootstrap
--disable-multilib --disable-libvtv --with-system-zlib --without-isl
--enable-multiarch
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 14.0.1 20240421 (experimental) (GCC) 

$ clang -v

Ubuntu clang version 18.1.3
(++20240322073153+ef6d1ec07c69-1~exp1~20240322193300.86)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-18/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64

$ cat > test.c << EOF
void f(int a, int b, int c) { a && b || c; }
int main(void) {
    f(0,0,1);
    f(1,0,0);
    return 0;
}
EOF
$ gcc --coverage -fcondition-coverage test.c -o test
$ ./test
$ gcov --conditions test.c
$ cat test.c.gcov

File 'test.c'
Lines executed:100.00% of 5
Condition outcomes covered:50.00% of 6
Creating 'test.c.gcov'

Lines executed:100.00% of 5
        -:    0:Source:test.c
        -:    0:Graph:test.gcno
        -:    0:Data:test.gcda
        -:    0:Runs:1
        2:    1:void f(int a, int b, int c) { a && b || c; }
condition outcomes covered 3/6
condition  0 not covered (true false)
condition  1 not covered (true)
        1:    2:int main(void) {
        1:    3:    f(0,0,1);
        1:    4:    f(1,0,0);
        1:    5:    return 0;
        -:    6:}

$ clang -Wno-unused-value -fprofile-instr-generate -fcoverage-mapping
-fcoverage-mcdc test.c -o test
$ ./test
$ llvm-profdata merge default.profraw -o default.profdata
$ llvm-cov show --show-mcdc -instr-profile default.profdata test

    1|      2|void f(int a, int b, int c) { a && b || c; }
  ------------------
  |---> MC/DC Decision Region (1:31) to (1:42)
  |
  |  Number of Conditions: 3
  |     Condition C1 --> (1:31)
  |     Condition C2 --> (1:36)
  |     Condition C3 --> (1:41)
  |
  |  Executed MC/DC Test Vectors:
  |
  |     C1, C2, C3    Result
  |  1 { T,  F,  F  = F      }
  |  2 { F,  -,  T  = T      }
  |
  |  C1-Pair: not covered
  |  C2-Pair: not covered
  |  C3-Pair: not covered
  |  MC/DC Coverage for Decision: 0.00%
  |
  ------------------
    2|      1|int main(void) {
    3|      1|    f(0,0,1);
    4|      1|    f(1,0,0);
    5|      1|    return 0;
    6|      1|}

In this example, gcov believes condition "c" is covered but LLVM reports that
none of the conditions is covered.

[Bug gcov-profile/115047] Inconsistent MC/DC reported by GCC and LLVM

[j at lambda dot is]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115047

--- Comment #1 from Jørgen Kvalsvik <j at lambda dot is> ---
gcc is not wrong here - f(1, 0, 0) = false, while f(1, 0, 1) = true so clearly
c has an independent effect on the outcome here, and both its values has been
observed.

I think clang measures unique-cause and not masking MC/DC which probably
explains the difference (at least clang's documentation [1] does not specify
which form it measures. Since more than one variable change between the two
test vectors I assume that would be it. I don't have clang 19 available right
now, does f(0, 0, 1); f(0, 0, 0); cover c?

[1]
https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#mc-dc-instrumentation

[Bug gcov-profile/115047] Inconsistent MC/DC reported by GCC and LLVM

[j at lambda dot is]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115047

--- Comment #2 from Jørgen Kvalsvik <j at lambda dot is> ---
> f(1, 0, 0) = false, while f(1, 0, 1) = true

Sorry, I meant:

f(0, 0, 1) = true
f(0, 0, 0) = false
and
f(1, 0, 1) = true
f(1, 0, 0) = false

It does not take away from the conclusion when doing masking MC/DC.

[Bug gcov-profile/115047] Inconsistent MC/DC reported by GCC and LLVM

[wentaoz5 at illinois dot edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115047

Wentao Zhang <wentaoz5 at illinois dot edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #3 from Wentao Zhang <wentaoz5 at illinois dot edu> ---
Jørgen, thank you for the info! I just realized this had been somewhat
discussed
between you and Alan on LLVM discourse [2]. From Alan's description I think
LLVM
is adopting unique-cause MC/DC. I might explicitly ask them for confirmation.

> does f(0, 0, 1); f(0, 0, 0); cover c?

Yes

    1|      2|void f(int a, int b, int c) { a && b || c; }
  ------------------
  |---> MC/DC Decision Region (1:31) to (1:42)
  |
  |  Number of Conditions: 3
  |     Condition C1 --> (1:31)
  |     Condition C2 --> (1:36)
  |     Condition C3 --> (1:41)
  |
  |  Executed MC/DC Test Vectors:
  |
  |     C1, C2, C3    Result
  |  1 { F,  -,  F  = F      }
  |  2 { F,  -,  T  = T      }
  |
  |  C1-Pair: not covered
  |  C2-Pair: not covered
  |  C3-Pair: covered: (1,2)
  |  MC/DC Coverage for Decision: 33.33%
  |
  ------------------
    2|      1|int main(void) {
    3|      1|    f(0,0,1);
    4|      1|    f(0,0,0);
    5|      1|    return 0;
    6|      1|}
    7|       |

I guess it would be desirable to (1) let LLVM support masking MC/DC and (2) let
gcov support unique-cause MC/DC. The first seems easier and I might try
implementing a prototype.

[2] https://discourse.llvm.org/t/rfc-source-based-mc-dc-code-coverage/59244/9

Thanks,
Wentao