[Bug other/29442] New: insn-attrtab has grown too large

[Bug other/29442] New: insn-attrtab has grown too large

[rick at hartmantech dot com]

When compiling gcc-4.1.1, there is at least a serious slowdown, and on many
machines a failure, when insn-attrtab is compiled. There are several threads on
the Gentoo forums about this, some involving machines with 512M of memory. This
file seems to be automatically generated, and has become so large that one user
graphically describes it as a "swapfest". Compiling on a machine with the
specified Gentoo minimum of 64M is not practical.

Is it possible to split this file?


-- 
           Summary: insn-attrtab has grown too large
           Product: gcc
           Version: 4.1.1
            Status: UNCONFIRMED
          Severity: enhancement
          Priority: P3
         Component: other
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: rick at hartmantech dot com
 GCC build triplet: i686-pc-linux-gnu-4.1.1
  GCC host triplet: i686-pc-linux-gnu-4.1.1
GCC target triplet: i686-pc-linux-gnu-4.1.1


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[pinskia at gcc dot gnu dot org]

------- Comment #1 from pinskia at gcc dot gnu dot org  2006-10-14 03:39 -------
> Compiling on a machine with the
> specified Gentoo minimum of 64M is not practical.

HEHEHEHEHEHEHEHE. 64MB is very very small ammount of memory really.  Even the
PS3 has 256MB.  Oh and compiling GCC should not have a memory requirement so
gentoo is doing something wrong.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[vapier at gentoo dot org]

------- Comment #2 from vapier at gentoo dot org  2006-10-14 04:16 -------
the "minimum" is a suggestion in our install docs ... it's like the "minimum"
you see on the side of boxed software in the store

the 64M has nothing to do with GCC


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[JoeGTN1 at NetZero dot net]

------- Comment #3 from JoeGTN1 at NetZero dot net  2006-10-25 12:44 -------
I think the pertinent part of this "bug" is:
> Is it possible to split this file?

  It took 6-8 hours to compile this file alone on my xbox, which only has 64m
of memory but should be considered a sufficiently modern machine.
  My understanding is that this is a generated file, containing information
about lots of different architectures.  Might it be possible to generate a file
per architecture, or per group of architectures?

  This is not a new problem, this file is what made me give up on Linux for the
Dreamcast (Which might not be considered a sufficiently modern machine.)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[rick at hartmantech dot com]

------- Comment #4 from rick at hartmantech dot com  2006-10-25 16:23 -------
The problem is not only confined to "sufficiently modern machines". Some of us
have a conservationist sort of outlook, and are happy to find a use for an
older machine as a router, firewall, Kerberos server or whatever. Linux
supports this sort of thing well - a floppy firewall with no hard drive will
run just fine on an 8M machine, and a 486/33 will keep up with all that an
ordinary ADSL connection can transmit. With insn-attrtab excluded, gcc-4.1.1
compiles quickly and well on 52M, and quite possibly on smaller machines,
though I have no very recent experience to prove it. It seems a pity that a
single source file, and an automatically-generated one at that, should
essentially block an excellent compiler from a lot of smaller machines, new and
old, and cause enough of a slowdown even on a 512M machine to arouse concern.

If there is a rational reason for this file to be huge, well and good, but if
it is readily split that would seem like a worthwhile thing to do.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[konig12 at gmail dot com]

------- Comment #5 from konig12 at gmail dot com  2007-07-23 05:53 -------
I have experienced the same problem.  I am compiling gcc on a machine with 32
mb ram.  Even with over 2 Gb swap it will work for hours and then finally crash
with an out of memory exception.  I have worked around the problem by compiling
gcc on another machine, and successfully resumed the  build.  (no word if it
worked yet).  Are there any patches that fix/ alleviate this problem?


-- 

konig12 at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |konig12 at gmail dot com


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[rask at gcc dot gnu dot org]

------- Comment #6 from rask at gcc dot gnu dot org  2007-10-27 12:46 -------
As far as I can tell (from running cc1 in a debugger), the problem is not so
much the size of the file, but that it contains two large functions and GCC
leaks memory. After compiling the first large function, the process size is 886
megs (basic-block reordering alone being responsible for about 200 megs),
probably littered with pieces of garbage. That'll cause all following
allocations to become scattered thoughout memory and the swapfest begins if you
don't have lots of memory. 256 megs is not enough. The RTL passes in particular
seem to be affected and several df_* functions take ages to run.
I'm working on a patch to split insn-attrtab.c into more files.


-- 

rask at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|unassigned at gcc dot gnu   |rask at gcc dot gnu dot org
                   |dot org                     |
             Status|UNCONFIRMED                 |ASSIGNED
     Ever Confirmed|0                           |1
      Known to fail|                            |4.3.0
   Last reconfirmed|0000-00-00 00:00:00         |2007-10-27 12:46:05
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[rask at gcc dot gnu dot org]

------- Comment #7 from rask at gcc dot gnu dot org  2007-10-28 12:40 -------
Created an attachment (id=14420)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=14420&action=view)
Reenable alloca() on non-GCC compilers

The memory fragmentation problem is to be caused by libiberty which disables
alloca() if you bootstrap with a non-GCC compiler[1] (which I do). Instead you
get a malloc() based replacment. The df_* functions use alloca() and are called
a lot.
With this patch on top of the next one, process size tops at around 185 MB
compared to the 900+ MB it reached before, at which point I gave up with 256 MB
of RAM.
Just wondering, which compiler are you using to bootstrap GCC?

[1] http://gcc.gnu.org/ml/gcc-patches/2001-03/msg00194.html


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[rask at gcc dot gnu dot org]

------- Comment #8 from rask at gcc dot gnu dot org  2007-10-28 12:57 -------
Created an attachment (id=14421)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=14421&action=view)
split insn-attrtab.c into three files

Here's the patch to split insn-attrtab.c into smaller pieces. The result:
$ wc -l insn-*tab.c
  14566 insn-attrtab.c
  44454 insn-dfatab.c
  36815 insn-latencytab.c
  95835 total

Please tell me if it makes a difference.

(It is necessary to run autoheader and autoconf in the gcc directory after
applying the patches.)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[rask at gcc dot gnu dot org]

------- Comment #9 from rask at gcc dot gnu dot org  2007-10-28 17:48 -------
I just tried with only the alloca patch, and despite the unsplit insn-attrtab.c
file, process size tops at just 205 MB. It looks like GCC 4.3.0 is in a much
better shape than GCC 4.1.1, so I'm letting go of the bug.

Just to clarify, the process sizes where taken when the stage1 compiler was
compiling the stage2 compiler.


-- 

rask at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|rask at gcc dot gnu dot org |unassigned at gcc dot gnu
                   |                            |dot org
             Status|ASSIGNED                    |NEW
      Known to fail|4.3.0                       |
      Known to work|                            |4.3.0


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[steven at gcc dot gnu dot org]

------- Comment #10 from steven at gcc dot gnu dot org  2009-02-22 14:19 -------
What happened with the alloca patch? Looks like it was never committed.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

[Bug other/29442] insn-attrtab has grown too large

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

--- Comment #19 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-05-07 12:36:21 UTC ---
(In reply to comment #18)
> (In reply to comment #15)
> Meanwhile Jakub implemented a different mean which doesn't call the get_attr_
> functions dynamically more often (which my approach potentially does), but
> which as result isn't nearly as effective in reducing size and compile and
> generation time.

For reducing compile time (but not size/generation), we could just use the
cache vars with some magic unset value (probably the cache vars would need to
be int
instead of enum for that).  Then if genattrtab finds out that there is no
dominating call to the get_attr_* function, but still potentially for one insn
the same get_attr_* might be called more than once in certain code path, then
for that code path it could just use (cache_FOO != -1 ? (enum ...) cache_FOO :
(cache_FOO = get_attr_FOO (...))).

[Bug other/29442] insn-attrtab has grown too large

[matz at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

--- Comment #18 from Michael Matz <matz at gcc dot gnu.org> 2012-05-07 12:27:18 UTC ---
(In reply to comment #15)
> While looking for ways to speed up genattrtab itself, I found this patch, which
> doesn't speed up genattrtab, but would make the insn-*tab.c files smaller:
> 
> http://gcc.gnu.org/ml/gcc-patches/2003-09/msg00316.html
> 
> Micha, this appears to have been dropped. Any reason why you never asked for an
> OK to commit this patch?

I did on multiple occasions (last was
http://gcc.gnu.org/ml/gcc-patches/2010-06/msg01521.html) , never got an okay. 

Meanwhile Jakub implemented a different mean which doesn't call the get_attr_
functions dynamically more often (which my approach potentially does), but
which as result isn't nearly as effective in reducing size and compile and
generation time.

Some of the hunks of my patch still are useful cleanups/speedups, lemme
submit them.

[Bug other/29442] insn-attrtab has grown too large

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

--- Comment #17 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-05-07 07:53:46 UTC ---
Seems my memory doesn't serve me well already, I've even implemented that:
http://gcc.gnu.org/ml/gcc-patches/2010-06/msg01908.html

[Bug other/29442] insn-attrtab has grown too large

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jakub at gcc dot gnu.org

--- Comment #16 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-05-07 07:26:11 UTC ---
I must say I don't like the calling of potentially expensive get_attr_*
upfront.
Can't we instead introduce inlines, get_attr_*_cached (or macros), which would
take another argument by reference, initially initialized to a value don't know
by genattrtab, and they would cache the attribute value in that variable.
genattrtab would emit those if there is more than one call to that get_attr_*
for the insn.

[Bug other/29442] insn-attrtab has grown too large

[steven at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

Steven Bosscher <steven at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |matz at gcc dot gnu.org

--- Comment #15 from Steven Bosscher <steven at gcc dot gnu.org> 2012-05-06 23:39:46 UTC ---
While looking for ways to speed up genattrtab itself, I found this patch, which
doesn't speed up genattrtab, but would make the insn-*tab.c files smaller:

http://gcc.gnu.org/ml/gcc-patches/2003-09/msg00316.html

Micha, this appears to have been dropped. Any reason why you never asked for an
OK to commit this patch?

[Bug other/29442] insn-attrtab has grown too large

[steven at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

Steven Bosscher <steven at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |steven at gcc dot gnu.org
         Resolution|                            |FIXED

--- Comment #14 from Steven Bosscher <steven at gcc dot gnu.org> 2012-05-04 20:16:57 UTC ---
The patch of comment #13 and the observation of comment #8 are Good Enough for
me to consider this problem fixed.

[Bug other/29442] insn-attrtab has grown too large

[steven at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

--- Comment #13 from Steven Bosscher <steven at gcc dot gnu.org> 2012-05-04 20:04:56 UTC ---
Author: steven
Date: Fri May  4 20:04:47 2012
New Revision: 187181

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=187181
Log:
    PR other/29442
    * read-md.c (fprint_md_ptr_loc, fprint_c_condition): New functions.
    (print_md_ptr_loc, print_c_condition): Use them.
    * read-md.h (fprint_md_ptr_loc, fprint_c_condition): New prototypes.
    * genattrtab.c (attr_file_name, dfa_file_name, latency_file_name,
    attr_file, dfa_file, latency_file): New global variables.
    (write_attr_valueq, write_attr_set, write_attr_case, write_attr_value,
    write_upcase, write_indent, write_length_unit_log, write_test_expr,
    write_attr_get, write_insn_cases, write_eligible_delay,
    write_const_num_delay_slots): Accept FILE pointer and toss it around.
    Update all callers.
    (write_header, open_outfile, handle_arg): New funcions.
    (make_automaton_attrs): Write prototypes as extern to the output
    files.
    (main): Use init_rtx_reader_args_cb with handle_arg to take 3 file
    names from the command line.  Open the output files and write out
    internal functions for DFA functions to dfa_file_name, insn latency
    functions to latency_file_name, and everything else to attr_file.
    * Makefile.in (OBJS): Add insn-dfatab.o and insn-latencytab.o.
    (BACKEND): Build libbackend first.
    (MOSTLYCLEANFILES): Add insn-dfatab.c and insn-latencytab.c.
    (.PRECIOUS): Likewise.
    (insn-dfatab.o): New rule.
    (insn-latencytab.o): New rule.
    (simple_rtl_generated_c): Do not include insn-attrtab.c.
    (s-attrtab): New rule.


Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/Makefile.in
    trunk/gcc/genattrtab.c
    trunk/gcc/read-md.c
    trunk/gcc/read-md.h

[Bug other/29442] insn-attrtab has grown too large

[manu at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

Manuel López-Ibáñez <manu at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |manu at gcc dot gnu.org

--- Comment #12 from Manuel López-Ibáñez <manu at gcc dot gnu.org> 2012-05-02 16:53:43 UTC ---
(In reply to comment #11)
> I can confirm this is still a problem on 2.5.3. Compiling on a system with

Gašper, nothing will change unless people like you that care about such systems
contribute patches to fix this situation: http://gcc.gnu.org/contribute.html

[Bug other/29442] insn-attrtab has grown too large

[gasper.azman at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

Gašper Ažman <gasper.azman at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |gasper.azman at gmail dot
                   |                            |com

--- Comment #11 from Gašper Ažman <gasper.azman at gmail dot com> 2012-05-02 16:28:52 UTC ---
I can confirm this is still a problem on 2.5.3. Compiling on a system with
256MB memory (hired, remote, virtualized) is not possible without an extra 1GB
of memory and almost nothing else running (the memory usage peaked at full RAM
and 522MB swap used, with gcc using everything but 60MB).