[Bug preprocessor/58580] New: preprocessor goes OOM with warning for zero literals

[Bug preprocessor/58580] New: preprocessor goes OOM with warning for zero literals

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

            Bug ID: 58580
           Summary: preprocessor goes OOM with warning for zero literals
           Product: gcc
           Version: 4.8.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: preprocessor
          Assignee: unassigned at gcc dot gnu.org
          Reporter: rguenth at gcc dot gnu.org

Created attachment 30940
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=30940&action=edit
testcase

> g++-4.8  -E test.cpp 
# 1 "test.cpp"
# 1 "<command-line>"
# 1 "test.cpp"
test.cpp:1:1: warning: null character(s) preserved in literal [enabled by
default]
 """""
 ^
test.cpp:2:1: warning: null character(s) preserved in literal [enabled by
default]
""
test.cpp:3:1: warning: null character(s) preserved in literal [enabled by
default]
""
test.cpp:4:1: warning: null character(s) preserved in literal [enabled by
default]
""

cc1plus: out of memory allocating 1677721600 bytes after a total of 405504
bytes

use a suitable ulimit -v to force the error.

[Bug preprocessor/58580] [4.8/4.9 Regression] preprocessor goes OOM with warning for zero literals

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |4.8.2
            Summary|preprocessor goes OOM with  |[4.8/4.9 Regression]
                   |warning for zero literals   |preprocessor goes OOM with
                   |                            |warning for zero literals

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
Works with 4.7.

[Bug preprocessor/58580] [4.8/4.9 Regression] preprocessor goes OOM with warning for zero literals

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.8.2                       |4.8.3

--- Comment #3 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
GCC 4.8.2 has been released.

[Bug preprocessor/58580] [4.8/4.9 Regression] preprocessor goes OOM with warning for zero literals

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dodji at gcc dot gnu.org

--- Comment #4 from Richard Biener <rguenth at gcc dot gnu.org> ---
Dodji, can you look at this?

[Bug preprocessor/58580] [4.8/4.9 Regression] preprocessor goes OOM with warning for zero literals

[dodji at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Dodji Seketeli <dodji at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
   Last reconfirmed|                            |2013-10-29
           Assignee|unassigned at gcc dot gnu.org      |dodji at gcc dot gnu.org
     Ever confirmed|0                           |1

--- Comment #5 from Dodji Seketeli <dodji at gcc dot gnu.org> ---
I am looking into this.

[Bug preprocessor/58580] [4.8/4.9 Regression] preprocessor goes OOM with warning for zero literals

[dodji at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

--- Comment #6 from Dodji Seketeli <dodji at gcc dot gnu.org> ---
A candidate patch was sent to
http://gcc.gnu.org/ml/gcc-patches/2013-10/msg02676.html

[Bug preprocessor/58580] [4.8/4.9 Regression] preprocessor goes OOM with warning for zero literals

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P3                          |P2

[Bug preprocessor/58580] [4.8/4.9 Regression] preprocessor goes OOM with warning for zero literals

[dodji at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Dodji Seketeli <dodji at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #7 from Dodji Seketeli <dodji at gcc dot gnu.org> ---
This should be fixed in trunk for 4.9 by commit
http://gcc.gnu.org/git/?p=gcc.git;a=commit;h=fc3eff8854861fcd70d33d26095b17fe456fae31


I'll give it some testing time, and if it turns out to work OK, I'll consider
backporting the change to 4.8.

What do you think, Richi?

[Bug preprocessor/58580] [4.8 Regression] preprocessor goes OOM with warning for zero literals

[rguenther at suse dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

--- Comment #9 from rguenther at suse dot de <rguenther at suse dot de> ---
On Wed, 6 Nov 2013, dodji at gcc dot gnu.org wrote:

> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580
> 
> Dodji Seketeli <dodji at gcc dot gnu.org> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>              Status|ASSIGNED                    |RESOLVED
>          Resolution|---                         |FIXED
> 
> --- Comment #7 from Dodji Seketeli <dodji at gcc dot gnu.org> ---
> This should be fixed in trunk for 4.9 by commit
> http://gcc.gnu.org/git/?p=gcc.git;a=commit;h=fc3eff8854861fcd70d33d26095b17fe456fae31
> 
> 
> I'll give it some testing time, and if it turns out to work OK, I'll consider
> backporting the change to 4.8.
> 
> What do you think, Richi?

Yes, backporting would be nice - it affects a few packages for us.

[Bug preprocessor/58580] [4.8 Regression] preprocessor goes OOM with warning for zero literals

[dodji at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

--- Comment #10 from Dodji Seketeli <dodji at gcc dot gnu.org> ---
Author: dodji
Date: Thu Jan 23 09:13:08 2014
New Revision: 206957

URL: http://gcc.gnu.org/viewcvs?rev=206957&root=gcc&view=rev
Log:
PR preprocessor/58580 - preprocessor goes OOM with warning for zero literals

In this problem report, the compiler is fed a (bogus) translation unit
in which some literals contain bytes whose value is zero.  The
preprocessor detects that and proceeds to emit diagnostics for that
king of bogus literals.  But then when the diagnostics machinery
re-reads the input file again to display the bogus literals with a
caret, it attempts to calculate the length of each of the lines it got
using fgets.  The line length calculation is done using strlen.  But
that doesn't work well when the content of the line can have several
zero bytes.  The result is that the read_line never sees the end of
the line because strlen repeatedly reports that the line ends before
the end-of-line character; so read_line thinks its buffer for reading
the line is too small; it thus increases the buffer, leading to a huge
memory consumption and disaster.

Here is what this patch does.

location_get_source_line is modified to return the length of a source
line that can now contain bytes with zero value.
diagnostic_show_locus() is then modified to consider that a line can
have characters of value zero, and so just shows a white space when
instructed to display one of these characters.

Additionally location_get_source_line is modified to avoid re-reading
each and every line from the beginning of the file until it reaches
the line number N that it is instructed to get; this was leading to
annoying quadratic behaviour when reading adjacent lines near the end
of (big) files.  So a cache is now associated to the file opened in
text mode.  When the content of the file is read, that content is
stashed in the file cache.  That file cache is searched for line
delimiters.  A number of line positions are saved in the cache and a
number of file caches are kept in memory.  That way when
location_get_source_line is asked to read line N + 1, it just has to
start reading from line N that it has already read.

libcpp/ChangeLog:

    * include/line-map.h (linemap_get_file_highest_location): Declare
    new function.
    * line-map.c (linemap_get_file_highest_location): Define it.

gcc/ChangeLog:

    * input.h (location_get_source_line): Take an additional line_size
    parameter.
    (void diagnostics_file_cache_fini): Declare new function.
    * input.c (struct fcache): New type.
    (fcache_tab_size, fcache_buffer_size, fcache_line_record_size):
    New static constants.
    (diagnostic_file_cache_init, total_lines_num)
    (lookup_file_in_cache_tab, evicted_cache_tab_entry)
    (add_file_to_cache_tab, lookup_or_add_file_to_cache_tab)
    (needs_read, needs_grow, maybe_grow, read_data, maybe_read_data)
    (get_next_line, read_next_line, goto_next_line, read_line_num):
    New static function definitions.
    (diagnostic_file_cache_fini): New function.
    (location_get_source_line): Take an additional output line_len
    parameter.  Re-write using lookup_or_add_file_to_cache_tab and
    read_line_num.
    * diagnostic.c (diagnostic_finish): Call
    diagnostic_file_cache_fini.
    (adjust_line): Take an additional input parameter for the length
    of the line, rather than calculating it with strlen.
    (diagnostic_show_locus): Adjust the use of
    location_get_source_line and adjust_line with respect to their new
    signature.  While displaying a line now, do not stop at the first
    null byte.  Rather, display the zero byte as a space and keep
    going until we reach the size of the line.
    * Makefile.in: Add vec.o to OBJS-libcommon

gcc/testsuite/ChangeLog:

    * c-c++-common/cpp/warning-zero-in-literals-1.c: New test file.

Signed-off-by: Dodji Seketeli <dodji@seketeli.org>

Added:
    trunk/gcc/testsuite/c-c++-common/cpp/warning-zero-in-literals-1.c
Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/Makefile.in
    trunk/gcc/diagnostic.c
    trunk/gcc/diagnostic.h
    trunk/gcc/input.c
    trunk/gcc/input.h
    trunk/gcc/testsuite/ChangeLog
    trunk/libcpp/ChangeLog
    trunk/libcpp/include/line-map.h
    trunk/libcpp/line-map.c

[Bug preprocessor/58580] [4.8 Regression] preprocessor goes OOM with warning for zero literals

[pthaugen at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Pat Haugen <pthaugen at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pthaugen at gcc dot gnu.org

--- Comment #11 from Pat Haugen <pthaugen at gcc dot gnu.org> ---
CPU2000 benchmark 253.perlbmk started failing to build for me starting with
revision 206957. Reduced down to following example.

[pthaugen@igoo 00000001]$ cat junk.c
# define HAS_FGETPOS
[pthaugen@igoo 00000001]$ /home/pthaugen/install/gcc/gcc_hunt/bin/gcc -c junk.c
-DHAS_FGETPOS
junk.c:1:0: warning: "HAS_FGETPOS" redefined [enabled by default]
 # define HAS_FGETPOS
 ^
<command-line>:0:0: note: this is the location of the previous definition
junk.c:1:0: internal compiler error: Segmentation fault
 # define HAS_FGETPOS
 ^
0x10667507 crash_signal
    /home/pthaugen/src/gcc/gcc_hunt/gcc/gcc/toplev.c:337
0x10cab5b7 add_file_to_cache_tab
    /home/pthaugen/src/gcc/gcc_hunt/gcc/gcc/input.c:296
0x10cab5b7 lookup_or_add_file_to_cache_tab
    /home/pthaugen/src/gcc/gcc_hunt/gcc/gcc/input.c:330
0x10cab5b7 location_get_source_line(expanded_location, int*)
    /home/pthaugen/src/gcc/gcc_hunt/gcc/gcc/input.c:704
0x10ca5167 diagnostic_show_locus(diagnostic_context*, diagnostic_info const*)
    /home/pthaugen/src/gcc/gcc_hunt/gcc/gcc/diagnostic.c:304
0x10ca580b diagnostic_report_diagnostic(diagnostic_context*, diagnostic_info*)
    /home/pthaugen/src/gcc/gcc_hunt/gcc/gcc/diagnostic.c:800
0x10156397 c_cpp_error(cpp_reader*, int, int, unsigned int, unsigned int, char
const*, char**)
    /home/pthaugen/src/gcc/gcc_hunt/gcc/gcc/c-family/c-common.c:9638
0x10cb85af cpp_diagnostic_with_line
    /home/pthaugen/src/gcc/gcc_hunt/gcc/libcpp/errors.c:145
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.

[Bug preprocessor/58580] [4.8 Regression] preprocessor goes OOM with warning for zero literals

[dodji at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Dodji Seketeli <dodji at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #12 from Dodji Seketeli <dodji at gcc dot gnu.org> ---
This fall-out seems fixed now on trunk by commit r207046.

Sorry for the inconvenience.

[Bug preprocessor/58580] [4.8 Regression] preprocessor goes OOM with warning for zero literals

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |---

--- Comment #13 from Richard Biener <rguenth at gcc dot gnu.org> ---
Still broken on the branch as far as I can see.

[Bug preprocessor/58580] [4.8 Regression] preprocessor goes OOM with warning for zero literals

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.8.3                       |4.8.4

--- Comment #14 from Richard Biener <rguenth at gcc dot gnu.org> ---
GCC 4.8.3 is being released, adjusting target milestone.

[Bug preprocessor/58580] [4.8 Regression] preprocessor goes OOM with warning for zero literals

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.8.4                       |4.8.5

--- Comment #15 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
GCC 4.8.4 has been released.

[Bug preprocessor/58580] [4.8 Regression] preprocessor goes OOM with warning for zero literals

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58580

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|---                         |FIXED
   Target Milestone|4.8.5                       |4.9.0
      Known to fail|                            |4.8.5

--- Comment #16 from Richard Biener <rguenth at gcc dot gnu.org> ---
Fixed for 4.9.0.