Re: Compilation of lengthy C++ Files

[David Brown <david@westcontrol.com>]

On 18/10/2023 18:04, Kai Song via Gcc-help wrote:
> Dear GCC Developers,
> 
> I am unsuccessfully using g++ 12.0.4 to compile lengthy c++ codes. Those
> codes are automatically generated from my own code-generator tools that
> depend on parameters p.
> Typical applications are:
> - Taylor series of order p inserted into consistency conditions of
> numerical schemes, to determine optimal method parameters (think of, e.g.,
> Runge-Kutta methods)
> - recursive automatic code transformation (think of adjoints of adjoints of
> adjoints...) of recursion level p
> - Hilbert curves or other space-filling curves to generate code that
> simulates cache utilization in a Monte-Carlo context
> 
> I verify that for small p the codes compile and execute to the expected
> result. However, there is always a threshold for p so that the generated
> cpp file is so long that the compiler will just terminate after ~10min
> without monitor output but return the value +1.
> My cpp files range from 600k LOC up to 1Bio LOC. Often, the file comprises
> of one single c++ template class member function definition that relies on
> a few thousand lines of template-classes.
> 
> I would like to know:
> 1) Am I doing something wrong in that GCC should be able to compile lengthy
> codes?
> 2) Is it known that GCC is unable to compile lengthy codes?
> 3) Is it acknowledged that a compiler's ability to compile large files is
> relevant?
> 4) Are possible roots known for this inability and are these deliberate?
> 

I am curious to know why you are generating code like this.  I can see 
how some code generators for mathematical code could easily produce 
large amounts of code, but it is rarely ideal for real-world uses.  Such 
flattened code can reduce overheads and improve optimisation 
opportunities (like inlining, constant folding, function cloning, etc.) 
for small cases, but then they get impractical for compiling while the 
costs for cache misses outweigh the overhead cost for the loops or 
recursion needed for general solutions.

Any compiler is going to be targeted and tuned towards "normal" or 
"typical" code.  That means primarily hand-written code, or smaller 
generated code.  I know that some systems generate very large functions 
or large files, but those are primarily C code, and the code is often 
very simple and "flat".  (Applications here include compilers that use C 
as a intermediary target language, and simulators of various kinds.)  It 
typically makes sense to disable certain optimisation passes here, and a 
number of passes scale badly (quadratic or perhaps worse) with function 
size.

However, if you are generating huge templates in C++, you are going a 
big step beyond that - templates are, in a sense, code generators 
themselves that run at compile time as an interpreted meta-language.  I 
don't expect that there has been a deliberate decision to limit GCC's 
handling of larger files, but I can't imagine that huge templates are a 
major focus for the compiler development.  And I would expect enormous 
memory use and compile times even when it does work.