[Bug default/25024] dwz: Multifile temporary files too large

[Bug default/25024] dwz: Multifile temporary files too large

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #3 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Jan Kratochvil from comment #2)
> I can upload it but it is 30GB Uncompressed, 8GB xz -1.

Yes please.

Please upload to ftp://ftp.suse.com/pub/incoming/ or provide me with a download
link.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[jan.kratochvil at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #2 from Jan Kratochvil <jan.kratochvil at redhat dot com> ---
I think you just need more RAM, wasn't it OOM-killed? Running it on a 64GB
host.
Maybe switching ld->gold would build it for you.
I can upload it but it is 30GB Uncompressed, 8GB xz -1.

But then from a higher point of view I do not think fixing this issue is too
much important. xz should rather recode -fdebug-types-section as then the
linker already no longer has to build some gigantic intermediate files.

Just -fdebug-types-section currently crashes GCC so I could not test that more:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91887

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #1 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Jan Kratochvil from comment #0)
> Created attachment 11998 [details]
> lldb-experimental.spec
> 
> Tried to check how fast is DWZ but it refuses to run on LLDB debuginfo:
> 
> + dwz -h -q -r -m .dwz/lldb-experimental-10.0.0-0.20190817snap5.fc30.x86_64
> -l 1000000000 -L 2000000000
> ./opt/lldb-experimental/bin/c-index-test-10.0.0-0.20190817snap5.fc30.x86_64.
> debug ...debug
> dwz: Multifile temporary files too large
> 
> real    39m51.803s
> user    35m22.622s
> sys     1m38.881s

I installed a fedora-30 distro in a virtualbox VM, and I've been trying to
build this spec today, but I run into:
...
collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.
...

Is there any way you can make available the files required to reproduce this?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #6 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Jan Kratochvil from comment #4)
> https://www.jankratochvil.net/t/dwz-too-large.tar.xz
(In reply to Jan Kratochvil from comment #5)
> dwz -h -q -r -m merged -l 1000000000 -L 2000000000 `find -name "*.debug"`

Reproduced:
...
$ dwz -h -q -r -m merged -l 1000000000 -L 2000000000 $(find -name "*.debug")
dwz: Multifile temporary files too large
...

The offending file is /tmp/dwz.debug_info.LhYCFp at ~3.8 GB.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[jan.kratochvil at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #4 from Jan Kratochvil <jan.kratochvil at redhat dot com> ---
https://www.jankratochvil.net/t/dwz-too-large.tar.xz

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[jan.kratochvil at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #5 from Jan Kratochvil <jan.kratochvil at redhat dot com> ---
dwz -h -q -r -m merged -l 1000000000 -L 2000000000 `find -name "*.debug"`

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] New: dwz: Multifile temporary files too large

[jan.kratochvil at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

            Bug ID: 25024
           Summary: dwz: Multifile temporary files too large
           Product: dwz
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: default
          Assignee: nobody at sourceware dot org
          Reporter: jan.kratochvil at redhat dot com
                CC: dwz at sourceware dot org
  Target Milestone: ---

Created attachment 11998
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11998&action=edit
lldb-experimental.spec

Tried to check how fast is DWZ but it refuses to run on LLDB debuginfo:

+ dwz -h -q -r -m .dwz/lldb-experimental-10.0.0-0.20190817snap5.fc30.x86_64 -l
1000000000 -L 2000000000
./opt/lldb-experimental/bin/c-index-test-10.0.0-0.20190817snap5.fc30.x86_64.debug
...debug
dwz: Multifile temporary files too large

real    39m51.803s
user    35m22.622s
sys     1m38.881s

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[jan.kratochvil at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #7 from Jan Kratochvil <jan.kratochvil at redhat dot com> ---
Just the bug is dictated by the standard. I think it would need to switch DWZ
to DWARF-5 as there is DW_FORM_ref_sup8; currently DW_FORM_GNU_*_alt are both
32-bit only.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #11 from Tom de Vries <vries at gcc dot gnu.org> ---
I'm currently trying out the patch:
...
diff --git a/dwz.c b/dwz.c
index 3c886d6..804746c 100644
--- a/dwz.c
+++ b/dwz.c
@@ -11942,7 +11942,7 @@ write_multifile (DSO *dso)
                  < multi_macro_off)
        {
          error (0, 0, "Multifile temporary files too large");
-         multifile = NULL;
          ret = 1;
        }
       else
...
that implements the 'continue multifile optimization' strategy.

Currently /tmp/dwz.debug_info.xxxxxx is 4180719431, which is at 97% of the
maximum:
...
$ lsof | egrep '^COMMAND|/tmp/dwz.debug_info.'
COMMAND       PID     TID       USER   FD      TYPE             DEVICE  
SIZE/OFF       NODE NAME
dwz         28874           tdevries    3u      REG                8,2
4180719431   23298280 /tmp/dwz.debug_info.G8DTik (deleted)
...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #8 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Jan Kratochvil from comment #7)
> Just the bug is dictated by the standard. I think it would need to switch
> DWZ to DWARF-5 as there is DW_FORM_ref_sup8; currently DW_FORM_GNU_*_alt are
> both 32-bit only.

You're right about the limitation of DW_FORM_GNU_*_alt, but AFAIU that's not
the boundary we're running into here.

In multifile mode, DWZ first processes one file at a time, optimizes it in
single-file mode, and dumps multifile-eligible DWARF into temporary files, one
per debug section. Subsequently, these temporary files are mmapped, and common
DWARF is copied to the multifile.

We run here into a size limit for the temporary files.  The DW_FORM_GNU_*_alt
restriction limits the size of the multifile.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #10 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Tom de Vries from comment #9)
> There are a few things to mention here:
> - there is no clear reason why this should be an error, we could also warn
>   about not being able to add to the temporary files, and continue the
>   multifile optimization

I ran this again using --devel-trace to get a bit more information, and found
out that actually there's no error.  The "dwz: Multifile temporary files too
large" message is a warning, telling the user in a _very_ indirect way that dwz
switches off multifile optimization and will continue to process the remaining
files in single-file mode.

This is part of a larger issue in dwz where both warnings and errors are
generated using a call to 'error', and consequently it's not immediately clear
from the source code which is an error and which is a warning.

Furthermore, in write_multifile when running into this warning we immediately
return with return value 1, suggesting that processing is stopped, but the
return value is actually ignored.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #13 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Tom de Vries from comment #12)
> (In reply to Tom de Vries from comment #11)
> > I'm currently trying out the patch:
> > that implements the 'continue multifile optimization' strategy.
> 
> Result:
> ...
> $ ./reproduce.sh
> maxmem: 27212104
> real: 5215.06
> user: 4418.86
> system: 400.27
> ...
> So, this took 87 minutes real (74 minutes user, 7 minutes sys) and 26 GB (on
> a server with 256GB).
> 
> The resulting multifile is 823MB:
> ...
> $ du -h merged
> 823M    merged
> ...

Compared to without the patch:
...
maxmem: 17332160
real: 2539.16
user: 2063.37
system: 201.93
...
So, 42 minutes real (34 minutes user, 3 minutes user) and 16.5 GB.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement

--- Comment #9 from Tom de Vries <vries at gcc dot gnu.org> ---
There are a few things to mention here:
- there is no clear reason why this should be an error, we could also warn
  about not being able to add to the temporary files, and continue the
  multifile optimization
- it should be possible to continue writing at the point we run into an error
  now, by switching to 64-bit dwarf, but reading 64-bit dwarf is currently not
  supported.
- the setup of collecting all the info for the multifile optimization into
  one temporary file per section is very convenient because it follows the way
  single-file optimization is done closely.  But things don't have to be setup
  like that. We could also have one temporary file per section per input file,
  which would fix the limitation we're running into here.  Of course we'd
  spent more effort juggling things around once we start reading in those
files,
  but there is also the possibility that we'd be able to manage memory in
  a more fine-grained way, which could possibly reduce peak memory usage.

For now, I'm classifying this as enhancement, given that we're running into an
implementation-defined limitation, which dwz is correctly reporting.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/25024] dwz: Multifile temporary files too large

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25024

--- Comment #12 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Tom de Vries from comment #11)
> I'm currently trying out the patch:
> that implements the 'continue multifile optimization' strategy.

Result:
...
$ ./reproduce.sh
maxmem: 27212104
real: 5215.06
user: 4418.86
system: 400.27
...
So, this took 87 minutes real (74 minutes user, 7 minutes sys) and 26 GB (on a
server with 256GB).

The resulting multifile is 823MB:
...
$ du -h merged
823M    merged
...

-- 
You are receiving this mail because:
You are on the CC list for the bug.