[Bug c++/106241] New: compiler can't handle large array of strings

[Bug c++/106241] New: compiler can't handle large array of strings

[robert.durkacz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106241

            Bug ID: 106241
           Summary: compiler can't handle large array of strings
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: robert.durkacz at gmail dot com
  Target Milestone: ---

Compiler fails with an internal error when given an unreasonably large source
file. This is because I am simply trying to get a large string array put direct
into object code. My personal interest in reporting this as a bug, whether it
is or not, is to get any advice what is a better way to do what I am trying to
do.

Source code is included here in this description.

Bug report at request of compiler
=================================
Transcript:
$ g++ -g -std=c++11 -Wall -fPIC -o obj/maps.cpp.o maps.cpp.cpp
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.

Source code:
maps.cpp.cpp, which includes ...
#include "maps.include.cpp"

Compiler version
$ g++ --version
g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

System type
$ uname -a
Linux Satellite-Pro-L50-B 5.4.0-121-generic #137~18.04.1-Ubuntu SMP Mon Jun 20
07:25:24 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Text of source files.

===============================maps.cpp.cpp=====================================
#include <iostream>
#include <string>
using namespace std;

struct La {string lemma; string a;};
  struct Form_map {
    string s;
    struct La la;
  };

  struct Form_map form_array[]= {
#include "maps.include.cpp"
  };

int main()
{
  cout <<form_array[1].s <<" ";
  cout <<form_array[1].la.lemma <<" ";
  cout <<form_array[1].la.a <<endl;
}
===============================eof==============================================

maps.include.cpp consists of 5,200,000 lines similar to this one:
  {"ADP", {"ADP", "s{NGDAILV}|mq_"}},

[Bug c++/106241] compiler can't handle large array of strings

[schwab@linux-m68k.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106241

--- Comment #1 from Andreas Schwab <schwab@linux-m68k.org> ---
That's the OOM killer.

[Bug c++/106241] compiler can't handle large array of strings

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106241

--- Comment #2 from Jonathan Wakely <redi at gcc dot gnu.org> ---
(In reply to Robert Durkacz from comment #0)
> My personal interest in reporting this as a
> bug, whether it is or not, is to get any advice what is a better way to do
> what I am trying to do.

Get more memory, and use const char* in your structs, not std::string.

It compiles easily if you have an array with trivial types.

[Bug c++/106241] compiler can't handle large array of strings

[robert.durkacz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106241

--- Comment #3 from Robert Durkacz <robert.durkacz at gmail dot com> ---
The suggestions from Johnathon Wakely would work but it is changing the
problem. Rather than using simple types, the application is aimed at using more
elaborate types (std:multimap).
My reason for thinking that the compiler "should" be able to handle it somehow,
is that the full array or multimap can be built up at runtime by reading the
data in from a text file very similar to the source code file. So I guess the
compiler just does not address this particular kind of use case but it seems to
me that, on the contrary, there should be a compilation capability to do this
kind of thing.

[Bug c++/106241] compiler can't handle large array of strings

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106241

--- Comment #4 from Jonathan Wakely <redi at gcc dot gnu.org> ---
Yes, like https://wg21.link/p1967r7

But in the meantime, you  can create a static array of trivial types (not
std::string) and then iterate over it at runtime to copy the values into the
data structure you want to use (a multimap containing strings, or whatever).

[Bug c++/106241] compiler can't handle large array of strings

[robert.durkacz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106241

--- Comment #5 from Robert Durkacz <robert.durkacz at gmail dot com> ---
That is a good idea.

On Sun, 10 Jul 2022 at 19:56, redi at gcc dot gnu.org <
gcc-bugzilla@gcc.gnu.org> wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106241
>
> --- Comment #4 from Jonathan Wakely <redi at gcc dot gnu.org> ---
> Yes, like https://wg21.link/p1967r7
>
> But in the meantime, you  can create a static array of trivial types (not
> std::string) and then iterate over it at runtime to copy the values into
> the
> data structure you want to use (a multimap containing strings, or
> whatever).
>
> --
> You are receiving this mail because:
> You reported the bug.

[Bug c++/106241] compiler can't handle large array of strings

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106241

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |DUPLICATE

--- Comment #6 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Dup of bug 105838 (and there might be others).

*** This bug has been marked as a duplicate of bug 105838 ***

[Bug c++/106241] compiler can't handle large array of strings

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106241

--- Comment #7 from Jonathan Wakely <redi at gcc dot gnu.org> ---
(In reply to Robert Durkacz from comment #3)
> So I guess the compiler just does not address this particular kind of use
> case but it seems to me that, on the contrary, there should be a compilation
> capability to do this kind of thing.

The problem is that "this kind of thing" is completely unreasonable. The
element type of the array (Form_map) has a non-trivial destructor that needs to
deallocate the strings' memory. That means that the initializer for the array
compiles to code like:

try {
  form_array[0] = {"ADP", {"ADP", "s{NGDAILV}|mq_"}};
  try {
    form_array[1] = {"ADP", {"ADP", "s{NGDAILV}|mq_"}};
    try {
      form_array[2] = {"ADP", {"ADP", "s{NGDAILV}|mq_"}};
      // ... 5.2 million more try-catch blocks
    } catch (...) {
      form_array[2].~Form_map();
      throw;
    }
  } catch (...) {
    form_array[1].~Form_map();
    throw;
  }
} catch (...) {
  form_array[0].~Form_map();
  throw;
}

The code to do that 5.2 million times is not easy to compile. And it's not
really a sensible thing to try to compile. Just because all the construction
and destruction is hidden inside std::string ctors and dtors doesn't make it a
good idea. You're still asking for all that work to be done in a global
initializer before main() even starts.

Creating a trivial type that just holds a const char* is easy, there is no
allocation, no non-trivial destructor, so no exceptions possible and so nothing
needs to be caught or rethrown.