[Bug lto/94311] New: LTO produces line info entries with invalid line numbers

[Bug lto/94311] New: LTO produces line info entries with invalid line numbers

[mpolacek at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

            Bug ID: 94311
           Summary: LTO produces line info entries with invalid line
                    numbers
           Product: gcc
           Version: 10.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
          Assignee: unassigned at gcc dot gnu.org
          Reporter: mpolacek at gcc dot gnu.org
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

Unfortunately this doesn't have a simple reproducer, but can be seen when
compiling valgrind:

$ wget https://sourceware.org/pub/valgrind/valgrind-3.15.0.tar.bz2
$ tar -xf valgrind-3.15.0.tar.bz2
$ cd valgrind-3.15
$ ./autogen.sh
$ ./configure --prefix=`pwd`/install --enable-only64bit --enable-lto
$ make install

then

$ ./install/bin/valgrind -q date

produces warnings like

   ==14497== warning: Can't handle line info entry with line number 177277754
greater than 1048575
   ==14497== (Nb: this message is only shown once)
   ==14497== warning: Can't handle inlined call info entry with line number
177277750 greater than 1048575 
   ==14497== (Nb: this message is only shown once)
Tue 24 Mar 2020 03:54:34 PM EDT

while with GCC 8 these warnings weren't issued.

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rguenth at gcc dot gnu.org

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
Err, but isn't this interpreting the dwarf from 'date'?  So doesn't this mean
that valgrind is "miscompiled" with --enable-lto rather than wrong debug?

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2020-03-25

--- Comment #2 from Martin Liška <marxin at gcc dot gnu.org> ---
I've just tested GCC 9 and current master and it works for me:

$ ./install/bin/valgrind -q ~/bin/gcc/bin/gcc --version
gcc (GCC) 10.0.1 20200320 (experimental)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ /install/bin/valgrind -q date
Wed 25 Mar 2020 11:43:10 AM CET

$ ./install/bin/valgrind /tmp/a.out
==32006== Memcheck, a memory error detector
==32006== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==32006== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==32006== Command: /tmp/a.out
==32006== 
==32006== 
==32006== Process terminating with default action of signal 6 (SIGABRT):
dumping core
==32006==    at 0x48BEEA1: raise (raise.c:51)
==32006==    by 0x48A853C: abort (abort.c:79)
==32006==    by 0x40112D: main (foo.c:6)

Here you can see that foo.c:6 is properly read from debuginfo.
Note that one can remove -flto-partition=one from CFLAGS. That rapidly speeds
up the build.

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[mark at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #3 from Mark Wielaard <mark at gcc dot gnu.org> ---
(In reply to Richard Biener from comment #1)
> Err, but isn't this interpreting the dwarf from 'date'?  So doesn't this
> mean that valgrind is "miscompiled" with --enable-lto rather than wrong
> debug?

The error message isn't very clear, but valgrind also reads its own code/binary
(which is inserted into the process), which is build with LTO. It is that
library that has the wrong line numbers. Which can also be seen in gdb:

      ./install/bin/valgrind -q date ==48528== warning: Can't handle line info
entry with line number 177277754 greater than 1048575 ==48528== (Nb: this
message is only shown once) ==48528== warning: Can't handle inlined call info
entry with line number 177277750 greater than 1048575 ==48528== (Nb: this
message is only shown once)

      #### Double check that valgrind debug info reader is correct:
      gdb ./.in_place/memcheck-amd64-linux
      Reading symbols from ./.in_place/memcheck-amd64-linux...
      (gdb) info line guest_amd64_toIR.c:177277754 Line number 177277754 is out
of range for "priv/guest_amd64_toIR.c".
      Line 177277754 of "priv/guest_amd64_toIR.c" starts at address 0x58069001
<dis_ESC_0F__SSE4+17> and ends at 0x58069005 <dis_ESC_0F__SSE4+21>.
      (gdb)
      You can also try:
      (gdb) disass /s dis_ESC_0F__SSE4
      and you zillions of useless lines.

      If needed, you can ask valgrind to show more info about what it is
doing/reading by doing e.g.
      ./install/bin/valgrind -v -v -v -v -d -d -d -d --debug-dump=line  date |&
tee d.out

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #4 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to Mark Wielaard from comment #3)
> (In reply to Richard Biener from comment #1)
> > Err, but isn't this interpreting the dwarf from 'date'?  So doesn't this
> > mean that valgrind is "miscompiled" with --enable-lto rather than wrong
> > debug?
> 
> The error message isn't very clear, but valgrind also reads its own
> code/binary (which is inserted into the process), which is build with LTO.
> It is that library that has the wrong line numbers. Which can also be seen
> in gdb:
> 
>       ./install/bin/valgrind -q date ==48528== warning: Can't handle line
> info entry with line number 177277754 greater than 1048575 ==48528== (Nb:
> this message is only shown once) ==48528== warning: Can't handle inlined
> call info entry with line number 177277750 greater than 1048575 ==48528==
> (Nb: this message is only shown once)
> 
>       #### Double check that valgrind debug info reader is correct:
>       gdb ./.in_place/memcheck-amd64-linux
>       Reading symbols from ./.in_place/memcheck-amd64-linux...
>       (gdb) info line guest_amd64_toIR.c:177277754 Line number 177277754 is
> out of range for "priv/guest_amd64_toIR.c".
>       Line 177277754 of "priv/guest_amd64_toIR.c" starts at address
> 0x58069001 <dis_ESC_0F__SSE4+17> and ends at 0x58069005
> <dis_ESC_0F__SSE4+21>.
>       (gdb)
>       You can also try:
>       (gdb) disass /s dis_ESC_0F__SSE4
>       and you zillions of useless lines.
> 
>       If needed, you can ask valgrind to show more info about what it is
> doing/reading by doing e.g.
>       ./install/bin/valgrind -v -v -v -v -d -d -d -d --debug-dump=line  date
> |& tee d.out

So the error must be visible with readelf as well.  Can you upload the
valgrind binary somewhere (does the issue only reproduce with separate
debug and/or dwz?)?

It's also a quite odd error unless the whole .debug_line section looks
garbled.

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[mark at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

Mark Wielaard <mark at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW

--- Comment #5 from Mark Wielaard <mark at gcc dot gnu.org> ---
I can no longer replicate the issue of the bad line numbers with gcc (GCC)
10.1.1 20200507 (Red Hat 10.1.1-1) on fedora 32. With either 3.15.0 or current
valgrind git.

But when building with LTO it looks like the (relative) paths to the VEX
library linked into memcheck-amd64-linux doesn't come out correctly. Both
valgrind and gdb cannot find the actual files, even when in the build dir.

You can see the same issue with elfutils eu-readelf --debug-dump=decodedline
./install/lib/valgrind/memcheck-amd64-linux

Without lto you'll get full (absolute) path names:

 CU [b] mc_leakcheck.c
  line:col SBPE* disc isa op address (Statement Block Prologue Epilogue *End)
  /tmp/valgrind-3.15.0/memcheck/mc_leakcheck.c (mtime: 0, length: 0)
   253:1   S        0   0  0 0x0000000058001070 <compare_MC_Chunks>
   254:4   S        0   0  0 0x0000000058001070 <compare_MC_Chunks>
   255:4   S        0   0  0 0x0000000058001070 <compare_MC_Chunks>
   256:4   S        0   0  0 0x0000000058001070 <compare_MC_Chunks>
   256:11           0   0  0 0x0000000058001070 <compare_MC_Chunks>
   256:23           0   0  0 0x0000000058001073 <compare_MC_Chunks+0x3>
   256:7            0   0  0 0x0000000058001076 <compare_MC_Chunks+0x6>
   257:7            0   0  0 0x000000005800107e <compare_MC_Chunks+0xe>
   259:1            0   0  0 0x000000005800108c <compare_MC_Chunks+0x1c>

While with lto some of the paths seem relative (but to what?):

 CU [b] <artificial>
  line:col SBPE* disc isa op address (Statement Block Prologue Epilogue *End)
  priv/guest_amd64_toIR.c (mtime: 0, length: 0)
  2036:0            0   0  0 0x0000000058001000 <sameIRExprs_aux2.cold>
  2446:0            0   0  0 0x0000000058001007 <show_hwcaps.cold>
  /home/mark/valgrind/memcheck/m_mallocfree.c (mtime: 0, length: 0)
  2643:0            0   0  0 0x000000005800100e <umsg_arg.cold>
  /home/mark/valgrind/memcheck/m_libcprint.c (mtime: 0, length: 0)
    61:0   S        0   0  0 0x000000005800100e <umsg_arg.cold>
    61:0   S        0   0  0 0x000000005800100e <umsg_arg.cold>
    61:0   S        0   0  0 0x000000005800100e <umsg_arg.cold>
  /home/mark/valgrind/memcheck/m_mallocfree.c (mtime: 0, length: 0)
  2643:0            0   0  0 0x0000000058001018 <xml_arg.cold>
  /home/mark/valgrind/memcheck/m_libcprint.c (mtime: 0, length: 0)
    61:0   S        0   0  0 0x0000000058001018 <xml_arg.cold>
    61:0   S        0   0  0 0x0000000058001018 <xml_arg.cold>
    61:0   S        0   0  0 0x0000000058001018 <xml_arg.cold>
    61:0   S   *    0   0  0 0x0000000058001021 <xml_arg.cold+0x9>

  priv/guest_amd64_toIR.c (mtime: 0, length: 0)
  7007:0   S        0   0  0 0x0000000058001150 <compare_MC_Chunks>
  7011:0   S        0   0  0 0x0000000058001150 <compare_MC_Chunks>
  7012:0   S        0   0  0 0x0000000058001150 <compare_MC_Chunks>
  7013:0   S        0   0  0 0x0000000058001150 <compare_MC_Chunks>
  7014:0            0   0  0 0x000000005800115e <compare_MC_Chunks+0xe>
  7010:0            0   0  0 0x000000005800116c <compare_MC_Chunks+0x1c>

This looks similar to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93865
(".debug_line with LTO refers to bogus file-names")

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[mark at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #6 from Mark Wielaard <mark at gcc dot gnu.org> ---
(In reply to Mark Wielaard from comment #5)
> I can no longer replicate the issue of the bad line numbers with gcc (GCC)
> 10.1.1 20200507 (Red Hat 10.1.1-1) on fedora 32. With either 3.15.0 or
> current valgrind git.

Note that current valgrind git hides these warnings:

commit 5920eb0c4302015f3648354e4f9c059f899194b7
Author: Philippe Waroquiers <philippe.waroquiers@skynet.be>
Date:   Tue Feb 18 21:35:44 2020 +0100

    Improve line info tracing, in particular when using lto.

    With gcc 9 and --enable-lto, we now have spurious warnings telling
    that the line information in the debug info has huge line numbers,
    greater than the (valgrind) maximum of 2^20.

    These spurious warnings make that all tests are failing.

    This change modifies the tracing/debugging of the line info to:
      * disable by default the warning for line info greater than 2^20.
        When using -d, such warnings are however still shown (once).
      * allow to see all such warnings, when using at least -d -d -d -d

And I am able to replicate with valgrind git on Fedora 32 with gcc (GCC) 10.2.1
20200723 (Red Hat 10.2.1-1) gcc-10.2.1-1.fc32.x86_64

 $ { ./autogen.sh; ./configure --prefix=`pwd`/install --enable-only64bit
--enable-lto; make -j8 install; } >& build.log
 $ ./install/bin/valgrind -d date 2>&1 | grep info\ entry
==110351== warning: Can't handle line info entry with line number 23270919
greater than 1048575
==110351== warning: Can't handle inlined call info entry with line number
23270918 greater than 1048575

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[mark at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #7 from Mark Wielaard <mark at gcc dot gnu.org> ---
Note that the above is in ./install/lib/valgrind/memcheck-amd64-linux

Which has .debug_line entries like:

  [0x00098404]  Set is_stmt to 0
  [0x00098405]  Advance PC by constant 17 to 0x5809993c
  [0x00098406]  Special opcode 103: advance Address by 7 to 0x58099943 and Line 
by 0 to 13372
  [0x00098407]  Set is_stmt to 1
  [0x00098408]  Advance Line by 23257547 to 23270919
  [0x0009840d]  Special opcode 145: advance Address by 10 to 0x5809994d and
Line by 0 to 23270919
  [0x0009840e]  Advance Line by 58034 to 23328953
  [0x00098412]  Special opcode 117: advance Address by 8 to 0x58099955 and Line
by 0 to 23328953
  [0x00098413]  Advance Line by 75462 to 23404415
  [0x00098417]  Special opcode 131: advance Address by 9 to 0x5809995e and Line
by 0 to 23404415
  [0x00098418]  Advance Line by -23388426 to 15989
  [0x0009841d]  Special opcode 187: advance Address by 13 to 0x5809996b and
Line by 0 to 15989

with readelf --debug-dump=decodedline we see:

priv/guest_amd64_toIR.c:
guest_amd64_toIR.c                          1505          0x580998b8           
   x
guest_amd64_toIR.c                         23769          0x580998d8           
   x
guest_amd64_toIR.c                         10611          0x580998f1           
   x
guest_amd64_toIR.c                         10611          0x580998f8        
guest_amd64_toIR.c                         10610          0x580998f8       1   
   x
guest_amd64_toIR.c                         24067          0x58099900           
   x
guest_amd64_toIR.c                         24067          0x58099900       1
guest_amd64_toIR.c                         13368          0x58099917           
   x
guest_amd64_toIR.c                         13368          0x5809991e        
guest_amd64_toIR.c                         14968          0x5809991e       1   
   x
guest_amd64_toIR.c                         13368          0x58099926           
   x
guest_amd64_toIR.c                         13368          0x5809992b        
guest_amd64_toIR.c                         13372          0x5809992b       1   
   x
guest_amd64_toIR.c                         13372          0x58099943        
guest_amd64_toIR.c                      23270919          0x5809994d           
   x
guest_amd64_toIR.c                      23328953          0x58099955           
   x
guest_amd64_toIR.c                      23404415          0x5809995e           
   x
guest_amd64_toIR.c                         15989          0x5809996b           
   x
guest_amd64_toIR.c                         15989          0x58099970        
guest_amd64_toIR.c                         15055          0x58099970       1   
   x
guest_amd64_toIR.c                          1254          0x5809997d           
   x
guest_amd64_toIR.c                          1253          0x58099980           
   x
guest_amd64_toIR.c                           226          0x58099982           
   x
guest_amd64_toIR.c                           226          0x58099987        
guest_amd64_toIR.c                         14316          0x58099987       1   
   x
guest_amd64_toIR.c                           226          0x5809998c           
   x
guest_amd64_toIR.c                           226          0x58099993        
guest_amd64_toIR.c                         14316          0x58099993       1   
   x
guest_amd64_toIR.c                         14315          0x58099996           
   x
guest_amd64_toIR.c                         14316          0x58099999           
   x
guest_amd64_toIR.c                           226          0x5809999c           
   x
guest_amd64_toIR.c                           226          0x580999a0        
guest_amd64_toIR.c                         14316          0x580999a0       1   
   x
guest_amd64_toIR.c                         14316          0x580999a2        
guest_amd64_toIR.c                           226          0x580999a2       1   
   x
guest_amd64_toIR.c                           226          0x580999a7        
guest_amd64_toIR.c                           226          0x580999b4        
guest_amd64_toIR.c                           226          0x580999c5        
guest_amd64_toIR.c                           226          0x580999ca        
guest_amd64_toIR.c                           226          0x580999d7        
guest_amd64_toIR.c                           226          0x580999dc        
guest_amd64_toIR.c                          5664          0x580999dc       1   
   x
guest_amd64_toIR.c                           226          0x580999e0           
   x
guest_amd64_toIR.c                           226          0x580999e4        
guest_amd64_toIR.c                          5664          0x580999e4       1   
   x

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[mark at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #8 from Mark Wielaard <mark at gcc dot gnu.org> ---
Note that VEX/priv/guest_arm64_toIR.c is fairly big (1.2M):
$ wc VEX/priv/guest_amd64_toIR.c
  32655  127564 1189783 VEX/priv/guest_amd64_toIR.c
(but still less than 2^15 lines)

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dmalcolm at gcc dot gnu.org,
                   |                            |hubicka at gcc dot gnu.org,
                   |                            |jakub at gcc dot gnu.org,
                   |                            |law at gcc dot gnu.org

--- Comment #9 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Ok, reproduced with last night's trunk too.
/home/jakub/src/gcc/obj68i/usr/local/bin/lto-dump -dump-body=dis_ESC_0F3A__VEX
-dump-level=lineno libvex_amd64_linux_a-guest_amd64_toIR.o 2>&1 | grep
'toIR.c:[0-9][0-9][0-9][0-9][0-9][0-9]'
prints nothing, so the LTO bytecode looks ok (largest lineno in that file below
33000 or so).
E.g. I see
  <bb 582> [local count: 456515]:
  [priv/guest_amd64_toIR.c:2248:0] _3193 = [priv/guest_amd64_toIR.c:2248:0]
xmm_names[_356];
  [priv/guest_amd64_toIR.c:25336:0] # DEBUG xmmreg => NULL
  [priv/guest_amd64_toIR.c:25336:0] vex_printf
([priv/guest_amd64_toIR.c:25336:0] "vcvtsd2ss %s,%s,%s\n", _3193, _3194,
_3195);
  goto <bb 585>; [100.00%]

  <bb 583> [local count: 691966]:
  [priv/guest_amd64_toIR.c:25339:0] addr_2000 = disAMode
([priv/guest_amd64_toIR.c:25339:0] &alen, vbi_925(D), pfx_922(D), delta_919,
[priv/guest_amd64_toIR.c:25339:0] &dis_buf, 0);
  [priv/guest_amd64_toIR.c:25339:0] # DEBUG addr => addr_2000
  [priv/guest_amd64_toIR.c:25340:0] # DEBUG tmp => addr_2000
  [priv/guest_amd64_toIR.c:258:0] _3197 = IRExpr_RdTmp (addr_2000);
in the lto-dump, but when I look at -fdump-ipa-cp-lineno from the memcheck
-flto link, I see in memcheck-amd64-linux.ltrans0.ltrans.076i.cp
  <bb 582> [local count: 456515]:
  [priv/guest_amd64_toIR.c:47085:0] _1627 = [priv/guest_amd64_toIR.c:47085:0]
xmm_names[_1626];
  [priv/guest_amd64_toIR.c:47086:0] # DEBUG xmmreg => NULL
  [priv/guest_amd64_toIR.c:47086:0] vex_printf
([priv/guest_amd64_toIR.c:47086:0] "vcvtsd2ss %s,%s,%s\n", _1627, _1625,
_1623);
  goto <bb 585>; [100.00%]

  <bb 583> [local count: 691966]:
  [priv/guest_amd64_toIR.c:47089:0] addr_1628 = disAMode
([priv/guest_amd64_toIR.c:47089:0] &alen, vbi_15(D), pfx_9(D), delta_6,
[priv/guest_amd64_toIR.c:47089:0] &dis_buf, 0);
  [priv/guest_amd64_toIR.c:47089:0] # DEBUG addr => addr_1628
  [priv/guest_amd64_toIR.c:47090:0] # DEBUG tmp => addr_1628
  [priv/guest_amd64_toIR.c:95985:0] _1629 = IRExpr_RdTmp (addr_1628);
So it looks like most of the line number are just totally bogus.

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #10 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
When I put a breakpoint on ../../libcpp/line-map.c:1737 with xloc.line > 40000
(i.e. near end of linemap_expand_location after xloc.line is set), I see
(gdb) p/x line_table->highest_line
$16 = 0x66e48c7c
(gdb) p/x line_table->highest_location
$17 = 0x66e48c7c
(gdb) p/x loc
$18 = 0x6894a440
in the caller during final notice_source_line -> insn_location ->
expand_location.
I guess I should use -fdump-ipa-cp-lineno to catch it earlier than during
final, anyway, having loc higher than highest_location seems very wrong.

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #11 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
I think the problem is just we ran out of the locations.
We hit both
  if (start_location >= LINE_MAP_MAX_LOCATION)
    /* We ran out of line map space.   */
    start_location = 0;
in linemap_add and
    overflowed:
      /* Remember we overflowed.  */
      set->highest_line = set->highest_location = LINE_MAP_MAX_LOCATION - 1;
      /* No column numbers!  */
      set->max_column_hint = 1;
      return 0;
in linemap_line_start.  While the latter is not that bad, we just get
UNKNOWN_LOCATION, the former is really bad, it just ignores whatever was before
and starts reusing those locations for something different.
Both of these spots are encountered once during this lto1 compilation, on a
total of 149763 linemap_add calls and 267678 linemap_line_start calls, so I'm
surprised that it got so quickly depleted.
Now, looking why it happens so quickly, I think because the line numbers in the
same file keep jumping back and forth extremely often, I see
#0  linemap_line_start (set=0x7ffff7ffb000, to_line=256, max_column_hint=17) at
../../libcpp/include/line-map.h:952
#1  0x0000000000d50886 in lto_location_cache::apply_location_cache
(this=0x2dbb868) at ../../gcc/lto-streamer-in.c:191
#2  0x0000000000d50de2 in stream_input_location_now (bp=<optimized out>,
data_in=0x2dbb840) at ../../gcc/lto-streamer-in.c:301
#3  0x000000000193d125 in input_gimple_stmt (tag=365, data_in=0x2dbb840,
ib=0x7fffffffcef0) at ../../gcc/gimple-streamer-in.c:111
#4  input_bb (ib=0x7fffffffcef0, tag=365, data_in=0x2dbb840, fn=0x7fffe7bd6630,
count_materialization_scale=<optimized out>)
    at ../../gcc/gimple-streamer-in.c:283
and
#0  linemap_line_start (set=0x7ffff7ffb000, to_line=8483, max_column_hint=8) at
../../libcpp/include/line-map.h:952
#1  0x0000000000d50886 in lto_location_cache::apply_location_cache
(this=0x2dbb868) at ../../gcc/lto-streamer-in.c:191
#2  0x0000000000d50de2 in stream_input_location_now (bp=<optimized out>,
data_in=0x2dbb840) at ../../gcc/lto-streamer-in.c:301
#3  0x000000000193d125 in input_gimple_stmt (tag=365, data_in=0x2dbb840,
ib=0x7fffffffcef0) at ../../gcc/gimple-streamer-in.c:111
#4  input_bb (ib=0x7fffffffcef0, tag=365, data_in=0x2dbb840, fn=0x7fffe7bd6630,
count_materialization_scale=<optimized out>)
    at ../../gcc/gimple-streamer-in.c:283
alternating all the time for quite a while (minor details on the exact to_line
numbers, but the important is going back and forth between 8NNN and 2NN
numbers).
Now, the backwards jumps result in the linemap_add calls too from (because
line_delta is negative), and when doing the big step forward, while line_delta
is large (larger than 8000).
      || (line_delta > 10
          && line_delta * map->m_column_and_range_bits > 1000)
is false as map->m_column_and_range_bits is 0, but one of the later condition
is true, so we add_map too, but then run into:
      /* Allocate the new line_map.  However, if the current map only has a
         single line we can sometimes just increase its column_bits instead. */
      if (line_delta < 0
          || last_line != ORDINARY_MAP_STARTING_LINE_NUMBER (map)
          || SOURCE_COLUMN (map, highest) >= (1U << (column_bits - range_bits))
          || ( /* We can't reuse the map if the line offset is sufficiently
                  large to cause overflow when computing location_t values.  */
              (to_line - ORDINARY_MAP_STARTING_LINE_NUMBER (map))
              >= (((uint64_t) 1)
                  << (CHAR_BIT * sizeof (linenum_type) - column_bits)))
          || range_bits < map->m_range_bits)
line_delta is > 8000, last_line is equal to ORDINARY_MAP_STARTING_LINE_NUMBER
(map), in the previous linemap_line_start we've called linemap_add because
line_delta was negative, column_bits and range_bits and SOURCE_COLUMN (map,
highest) are all 0 and to_line - ORDINARY_MAP_STARTING_LINE_NUMBER (map) is
that 8000-ish number, which is smaller than 1ULL << (8 * 4 - 0) which is 4G.

I think to resolve this we can do multiple things:
1) if possible avoid the stream_input_location_now on each gimple stmt,
that seems to be the killer.
The rationale is:
  /* Read location information.  Caching here makes no sense until streamer
     cache can handle the following gimple_set_block.  */
  gimple_set_location (stmt, stream_input_location_now (&bp, data_in));

  /* Read lexical block reference.  */
  gimple_set_block (stmt, stream_read_tree (ib, data_in));
So perhaps have a way add the block to the location cache, and maybe also avoid
it on PHIs too?

And/or try to limit the to_line jumps much more if we are already at the no
column bits stage.
I can try to create a small testcase tomorrow...

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #12 from Richard Biener <rguenth at gcc dot gnu.org> ---
For PHIs we could use the cache but I see we're simply dropping the BLOCKs
there (now that block location is encoded in the location we _do_ have
a location BLOCK there as well).

At least

      /* Do not cache a location - we do not have API to get pointer to the
         location in PHI statement and we may trigger reallocation.  */

the comment is wrong in both regards.  The API is gimple_phi_arg
which yields a phi_arg_d * which has the location member.  And re-allocation
shouldn't happen because

  len = EDGE_COUNT (bb->preds);
  result = create_phi_node (phi_result, bb);

so preds are set up and create_phi_node ensures there's enough capacity.

So it boils down to the same issue that BLOCKs are not handled by the
location cache.

I don't remember exactly but in principle we do not need any BLOCKs
on stmts in LTRANS (inline diagnostics uses it though).

Amending the location cache with a BLOCK pointer should be possible
but I wonder how to sort them?  I guess same location usually means
same BLOCK.  Otherwise sort after BLOCK_NUMBER?

That is, the idea would then be to apply the location cache only
at the end of input_function.

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #13 from Richard Biener <rguenth at gcc dot gnu.org> ---
Note we can still overflow locations so I wonder if we can have a more
forgiving failure mode?  Return UNKNOWN_LOCATION maybe.

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #14 from Richard Biener <rguenth at gcc dot gnu.org> ---
so

void
lto_location_cache::input_location_and_block (location_t *loc, tree *block,
                                    struct bitpack_d *bp,
                                    class lto_input_block *ib,
                                    class data_in *data_in);

and use that.  Should be able to drop stream_input_location_now completely
then.  In apply location cache then handle the case of NULL and non-NULL
block pointers, I think we can mix those freely.

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #15 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Short reproducer:

$ cat pr94311.c
#line 210
static inline __attribute__((always_inline)) void
foo (void)
{
  int a = 23;
}
#line 65570
volatile int v;
#define A { int b = 42; foo (); }
#define B A A A A A A A A A A
#define C B B B B B B B B B B
#define D C C C C C C C C C C
__attribute__((used, noipa)) void
baz (void)
{
  D D D
  v++;
  D
  v++;
}

int
main ()
{
  baz ();
}
$ ./xgcc -B ./ -flto -O2 -g pr94311.c -o pr94311 -save-temps; readelf -wl
pr94311 | grep 'Advance Line by [0-9][0-9][0-9][0-9][0-9][0-9]'
  [0x0000a56c]  Advance Line by 254236696 to 254302276

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #16 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
One possible way at the libcpp side is to make sure we don't deplete the
location_t stuff too much once we are above highest >
LINE_MAP_MAX_LOCATION_WITH_COLS.  I mean, the comment above this if block says:
      /* Allocate the new line_map.  However, if the current map only has a
         single line we can sometimes just increase its column_bits instead. */
and with these > 0x60000000 locations, column_bits and range_bits is 0 anyway
and in that case we are not going to increase its column_bits.
Now, I've bootstrapped a slightly different version overnight, which instead
had || column_bits == 0, and that regressed the g++.dg/diagnostic/pr77949.C
testcase which has a 4KB+ line and thus disables columns and expects no fixit
hint.  Or perhaps use || (highest > ... && (to_line -
ORDINARY_MAP_STARTING_LINE_NUMBER (map)) > 100) or something similar, allow
some growth, but not unlimited?  David, your thoughts on this?

2020-09-02  Jakub Jelinek  <jakub@redhat.com>

        PR lto/94311
        * line-map.c (linemap_line_start): For single-line map don't try
        to extend it if highest > LINE_MAP_MAX_LOCATION_WITH_COLS.

--- libcpp/line-map.c.jj        2020-07-28 15:39:10.127754591 +0200
+++ libcpp/line-map.c   2020-09-02 11:08:50.421816848 +0200
@@ -742,7 +742,8 @@ linemap_line_start (line_maps *set, line
              (to_line - ORDINARY_MAP_STARTING_LINE_NUMBER (map))
              >= (((uint64_t) 1)
                  << (CHAR_BIT * sizeof (linenum_type) - column_bits)))
-         || range_bits < map->m_range_bits)
+         || range_bits < map->m_range_bits
+         || highest > LINE_MAP_MAX_LOCATION_WITH_COLS)
        map = linemap_check_ordinary
                (const_cast <line_map *>
                  (linemap_add (set, LC_RENAME,

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #17 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Created attachment 49175
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=49175&action=edit
gcc11-pr94311.patch

Untested patch which implements location and block caching and the above test
passes with it (even with the libcpp patch reverted).

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #18 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Jakub Jelinek <jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:3536ff2de8317c430546fd97574d44c5146cef2b

commit r11-2995-g3536ff2de8317c430546fd97574d44c5146cef2b
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Thu Sep 3 12:51:01 2020 +0200

    lto: Cache location_ts including BLOCKs in GIMPLE streaming [PR94311]

    As mentioned in the PR, when compiling valgrind even on fairly small
    testcase where in one larger function the location keeps oscillating
    between a small line number and 8000-ish line number in the same file
    we very quickly run out of all possible location_t numbers and because of
    that emit non-sensical line numbers in .debug_line.
    There are ways how to decrease speed of depleting location_t numbers
    in libcpp, but the main reason of this is that we use
    stream_input_location_now for streaming in location_t for gimple_location
    and phi arg locations.  libcpp strongly prefers that the locations
    it is given are sorted by the different files and by line numbers in
    ascending order, otherwise it depletes quickly no matter what and is much
    more costly (many extra file changes etc.).
    The reason for not caching those were the BLOCKs that were streamed
    immediately after the location and encoded into the locations (and for PHIs
    we failed to stream the BLOCKs altogether).
    This patch enhances the location cache to handle also BLOCKs (but not for
    everything, only for the spots we care about the BLOCKs) and also optimizes
    the size of the LTO stream by emitting a single bit into a pack whether the
    BLOCK changed from last case and only streaming the BLOCK tree if it
    changed.

    2020-09-03  Jakub Jelinek  <jakub@redhat.com>

            PR lto/94311
            * gimple.h (gimple_location_ptr, gimple_phi_arg_location_ptr): New
            functions.
            * streamer-hooks.h (struct streamer_hooks): Add
            output_location_and_block callback.  Fix up formatting for
            output_location.
            (stream_output_location_and_block): Define.
            * lto-streamer.h (class lto_location_cache): Fix comment typo.  Add
            current_block member.
            (lto_location_cache::input_location_and_block): New method.
            (lto_location_cache::lto_location_cache): Initialize current_block.
            (lto_location_cache::cached_location): Add block member.
            (struct output_block): Add current_block member.
            (lto_output_location): Formatting fix.
            (lto_output_location_and_block): Declare.
            * lto-streamer.c (lto_streamer_hooks_init): Initialize
            streamer_hooks.output_location_and_block.
            * lto-streamer-in.c (lto_location_cache::cmp_loc): Also compare
            block members.
            (lto_location_cache::apply_location_cache): Handle blocks.
            (lto_location_cache::accept_location_cache,
            lto_location_cache::revert_location_cache): Fix up function
comments.
            (lto_location_cache::input_location_and_block): New method.
            (lto_location_cache::input_location): Implement using
            input_location_and_block.
            (input_function): Invoke apply_location_cache after streaming in
all
            bbs.
            * lto-streamer-out.c (clear_line_info): Set current_block.
            (lto_output_location_1): New function, moved from
lto_output_location,
            added block handling.
            (lto_output_location): Implement using lto_output_location_1.
            (lto_output_location_and_block): New function.
            * gimple-streamer-in.c (input_phi): Use input_location_and_block
            to input and cache both location and block.
            (input_gimple_stmt): Likewise.
            * gimple-streamer-out.c (output_phi): Use
            stream_output_location_and_block.
            (output_gimple_stmt): Likewise.

[Bug lto/94311] LTO produces line info entries with invalid line numbers

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94311

--- Comment #19 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-10 branch has been updated by Jakub Jelinek
<jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:591a05e2027d202e2d28ef6fd5df63e7c7f9556d

commit r10-8730-g591a05e2027d202e2d28ef6fd5df63e7c7f9556d
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Thu Sep 3 12:51:01 2020 +0200

    lto: Cache location_ts including BLOCKs in GIMPLE streaming [PR94311]

    As mentioned in the PR, when compiling valgrind even on fairly small
    testcase where in one larger function the location keeps oscillating
    between a small line number and 8000-ish line number in the same file
    we very quickly run out of all possible location_t numbers and because of
    that emit non-sensical line numbers in .debug_line.
    There are ways how to decrease speed of depleting location_t numbers
    in libcpp, but the main reason of this is that we use
    stream_input_location_now for streaming in location_t for gimple_location
    and phi arg locations.  libcpp strongly prefers that the locations
    it is given are sorted by the different files and by line numbers in
    ascending order, otherwise it depletes quickly no matter what and is much
    more costly (many extra file changes etc.).
    The reason for not caching those were the BLOCKs that were streamed
    immediately after the location and encoded into the locations (and for PHIs
    we failed to stream the BLOCKs altogether).
    This patch enhances the location cache to handle also BLOCKs (but not for
    everything, only for the spots we care about the BLOCKs) and also optimizes
    the size of the LTO stream by emitting a single bit into a pack whether the
    BLOCK changed from last case and only streaming the BLOCK tree if it
    changed.

    2020-09-03  Jakub Jelinek  <jakub@redhat.com>

            PR lto/94311
            * gimple.h (gimple_location_ptr, gimple_phi_arg_location_ptr): New
            functions.
            * streamer-hooks.h (struct streamer_hooks): Add
            output_location_and_block callback.  Fix up formatting for
            output_location.
            (stream_output_location_and_block): Define.
            * lto-streamer.h (class lto_location_cache): Fix comment typo.  Add
            current_block member.
            (lto_location_cache::input_location_and_block): New method.
            (lto_location_cache::lto_location_cache): Initialize current_block.
            (lto_location_cache::cached_location): Add block member.
            (struct output_block): Add current_block member.
            (lto_output_location): Formatting fix.
            (lto_output_location_and_block): Declare.
            * lto-streamer.c (lto_streamer_hooks_init): Initialize
            streamer_hooks.output_location_and_block.
            * lto-streamer-in.c (lto_location_cache::cmp_loc): Also compare
            block members.
            (lto_location_cache::apply_location_cache): Handle blocks.
            (lto_location_cache::accept_location_cache,
            lto_location_cache::revert_location_cache): Fix up function
comments.
            (lto_location_cache::input_location_and_block): New method.
            (lto_location_cache::input_location): Implement using
            input_location_and_block.
            (input_function): Invoke apply_location_cache after streaming in
all
            bbs.
            * lto-streamer-out.c (clear_line_info): Set current_block.
            (lto_output_location_1): New function, moved from
lto_output_location,
            added block handling.
            (lto_output_location): Implement using lto_output_location_1.
            (lto_output_location_and_block): New function.
            * gimple-streamer-in.c (input_phi): Use input_location_and_block
            to input and cache both location and block.
            (input_gimple_stmt): Likewise.
            * gimple-streamer-out.c (output_phi): Use
            stream_output_location_and_block.
            (output_gimple_stmt): Likewise.

    (cherry picked from commit 3536ff2de8317c430546fd97574d44c5146cef2b)