[PATCH] Fix .debug_line reference above end of section

[PATCH] Fix .debug_line reference above end of section

[Tom de Vries]

Hi,

Consider the file test-import-repeatedly from PR26738, combined with a trivial
a.out:
...
$ cp test-import-repeatedly 1
$ gcc -m32 -g ~/hello.c
$ cp a.out 2
...

When doing multifile optimization, we run into:
...
$ dwz -m 3 2 1
dwz: 3: .debug_line reference above end of section
...

Using --devel-save-temps and src/contrib/gen-dwz-debug-all.sh we get the
unoptimized multifile, and find there a CU:
...
  Compilation Unit @ offset 0x371:
   Length:        0x6d (32-bit)
   Version:       4
   Abbrev Offset: 0x14d
   Pointer Size:  4
 <0><37c>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <37d>   DW_AT_stmt_list   : 0xda
    <381>   DW_AT_language    : 0       (Unknown: 0)
    <382>   DW_AT_comp_dir    : ./build-3.8
...
which refers to a .debug_line offset 0xda, but the debug_line section only
contains an entry at offset 0x0.

This can be explained as follows.  The DIEs written into the CU do not
contain a single DW_AT_decl_file.  Consequently, htab_line will be NULL once
we get here in write_multifile:
...
             || (line_htab != NULL && write_multifile_line ()))
...
and no .debug_line contribution will be added.  However, the CU DIE
referencing the .debug_line contribution is written regardless.

We could fix this by not writing the DW_AT_stmt_list for the CU DIE in this
case, but that would require changing the order in which abbreviations are
computed.

Instead, fix this conservatively, by removing "line_htab != NULL &&", such we
get a minimal .debug_line contribution.

Any comments?

Thanks,
- Tom

Fix .debug_line reference above end of section

2021-03-02  Tom de Vries  <tdevries@suse.de>

	PR dwz/26738
	* dwz.c (write_multifile): Also call write_multifile_line when
	line_htab == NULL.

---
 dwz.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/dwz.c b/dwz.c
index af1c5af..5effbc1 100644
--- a/dwz.c
+++ b/dwz.c
@@ -15150,7 +15150,7 @@ write_multifile (DSO *dso)
 			    debug_sections[DEBUG_MACRO].new_size)
 		     != (ssize_t) debug_sections[DEBUG_MACRO].new_size)
 	      || (strp_htab != NULL && write_multifile_strp ())
-	      || (line_htab != NULL && write_multifile_line ()))
+	      || write_multifile_line ())
 	    {
 	      error (0, 0, "Error writing multi-file temporary files");
 	      ret = 1;

Re: [PATCH] Fix .debug_line reference above end of section

[Mark Wielaard]

Hi Tom,

On Tue, Mar 02, 2021 at 02:57:14PM +0100, Tom de Vries wrote:
> Consider the file test-import-repeatedly from PR26738, combined with a trivial
> a.out:
> ...
> $ cp test-import-repeatedly 1
> $ gcc -m32 -g ~/hello.c
> $ cp a.out 2
> ...
> 
> When doing multifile optimization, we run into:
> ...
> $ dwz -m 3 2 1
> dwz: 3: .debug_line reference above end of section
> ...
> 
> Using --devel-save-temps and src/contrib/gen-dwz-debug-all.sh we get the
> unoptimized multifile, and find there a CU:
> ...
>   Compilation Unit @ offset 0x371:
>    Length:        0x6d (32-bit)
>    Version:       4
>    Abbrev Offset: 0x14d
>    Pointer Size:  4
>  <0><37c>: Abbrev Number: 1 (DW_TAG_compile_unit)
>     <37d>   DW_AT_stmt_list   : 0xda
>     <381>   DW_AT_language    : 0       (Unknown: 0)
>     <382>   DW_AT_comp_dir    : ./build-3.8
> ...
> which refers to a .debug_line offset 0xda, but the debug_line section only
> contains an entry at offset 0x0.

I was unable to replicate this, so it is a little harder to comment in
this. I don't fully understand why this happens, is this because of
something in the input file or because we move all DIEs with file
attributes?

Would it be possible to attach the various temp files to the bug
report?

> This can be explained as follows.  The DIEs written into the CU do not
> contain a single DW_AT_decl_file.  Consequently, htab_line will be NULL once
> we get here in write_multifile:
> ...
>              || (line_htab != NULL && write_multifile_line ()))
> ...
> and no .debug_line contribution will be added.  However, the CU DIE
> referencing the .debug_line contribution is written regardless.
> 
> We could fix this by not writing the DW_AT_stmt_list for the CU DIE in this
> case, but that would require changing the order in which abbreviations are
> computed.
> 
> Instead, fix this conservatively, by removing "line_htab != NULL &&", such we
> get a minimal .debug_line contribution.

This does look technically OK. write_multifile_line does handle line_htab == NULL
correctly and sets up multi_line_off which will be used to write the
DW_AT_stmt_list.

But if the CU doesn't have a DW_AT_stmt_list this will produce an
unnecessary line table (stub). I cannot immediately see whether this
is an odd corner case that normally wouldn't happen in practice or if
that could happen more often.

Cheers,

Mark

Re: [PATCH] Fix .debug_line reference above end of section

[Tom de Vries]

On 3/4/21 1:33 AM, Mark Wielaard wrote:
> Hi Tom,
> 
> On Tue, Mar 02, 2021 at 02:57:14PM +0100, Tom de Vries wrote:
>> Consider the file test-import-repeatedly from PR26738, combined with a trivial
>> a.out:
>> ...
>> $ cp test-import-repeatedly 1
>> $ gcc -m32 -g ~/hello.c
>> $ cp a.out 2
>> ...
>>
>> When doing multifile optimization, we run into:
>> ...
>> $ dwz -m 3 2 1
>> dwz: 3: .debug_line reference above end of section
>> ...
>>
>> Using --devel-save-temps and src/contrib/gen-dwz-debug-all.sh we get the
>> unoptimized multifile, and find there a CU:
>> ...
>>   Compilation Unit @ offset 0x371:
>>    Length:        0x6d (32-bit)
>>    Version:       4
>>    Abbrev Offset: 0x14d
>>    Pointer Size:  4
>>  <0><37c>: Abbrev Number: 1 (DW_TAG_compile_unit)
>>     <37d>   DW_AT_stmt_list   : 0xda
>>     <381>   DW_AT_language    : 0       (Unknown: 0)
>>     <382>   DW_AT_comp_dir    : ./build-3.8
>> ...
>> which refers to a .debug_line offset 0xda, but the debug_line section only
>> contains an entry at offset 0x0.
> 
> I was unable to replicate this, so it is a little harder to comment in
> this. I don't fully understand why this happens, is this because of
> something in the input file or because we move all DIEs with file
> attributes?
> 

This is because all the DIEs from the input file that are written into
the unoptimized multifile don't have any attributes referring to the
file table.

> Would it be possible to attach the various temp files to the bug
> report?
> 

Done.

>> This can be explained as follows.  The DIEs written into the CU do not
>> contain a single DW_AT_decl_file.  Consequently, htab_line will be NULL once
>> we get here in write_multifile:
>> ...
>>              || (line_htab != NULL && write_multifile_line ()))
>> ...
>> and no .debug_line contribution will be added.  However, the CU DIE
>> referencing the .debug_line contribution is written regardless.
>>
>> We could fix this by not writing the DW_AT_stmt_list for the CU DIE in this
>> case, but that would require changing the order in which abbreviations are
>> computed.
>>
>> Instead, fix this conservatively, by removing "line_htab != NULL &&", such we
>> get a minimal .debug_line contribution.
> 
> This does look technically OK. write_multifile_line does handle line_htab == NULL
> correctly and sets up multi_line_off which will be used to write the
> DW_AT_stmt_list.
> 
> But if the CU doesn't have a DW_AT_stmt_list this will produce an
> unnecessary line table (stub). 

Ack.  The fix is conservative.

> I cannot immediately see whether this
> is an odd corner case that normally wouldn't happen in practice or if
> that could happen more often.

I've seen this error before, so it's not completely cornercase.

Thanks,
- Tom

Re: [PATCH] Fix .debug_line reference above end of section

[Mark Wielaard]

Hi Tom,

On Thu, Mar 04, 2021 at 09:33:43AM +0100, Tom de Vries wrote:
> On 3/4/21 1:33 AM, Mark Wielaard wrote:
> > I was unable to replicate this, so it is a little harder to comment in
> > this. I don't fully understand why this happens, is this because of
> > something in the input file or because we move all DIEs with file
> > attributes?
> > 
> 
> This is because all the DIEs from the input file that are written into
> the unoptimized multifile don't have any attributes referring to the
> file table.
> 
> > Would it be possible to attach the various temp files to the bug
> > report?
> > 
> 
> Done.

Thanks. It turns out I was confused because I was using eu-readelf and
that doesn't handle DWARF Units which contain zero DIEs. I'll write a
patch for elfutils.

> > This does look technically OK. write_multifile_line does handle line_htab == NULL
> > correctly and sets up multi_line_off which will be used to write the
> > DW_AT_stmt_list.
> > 
> > But if the CU doesn't have a DW_AT_stmt_list this will produce an
> > unnecessary line table (stub). 
> 
> Ack.  The fix is conservative.
> 
> > I cannot immediately see whether this
> > is an odd corner case that normally wouldn't happen in practice or if
> > that could happen more often.
> 
> I've seen this error before, so it's not completely cornercase.

I tried to come up with something different, but your fix is much
simpler. So please push what you have.

This is what I came up with, which makes sure there always is a
line_htab, even if it is empty. But even in write_unit_die we don't
yet know if there will be actual files. So we still end up writing the
DW_AT_stmt_list with the current multi_line_off, so we must produce a
debug_line entry anyway.

diff --git a/dwz.c b/dwz.c
index af1c5af..c493c1d 100644
--- a/dwz.c
+++ b/dwz.c
@@ -9929,6 +9929,16 @@ line_eq (const void *p, const void *q)
   return s1->file->time == s2->file->time && s1->file->size == s2->file->size;
 }
 
+static htab_t
+line_htab_create (void)
+{
+  htab_t htab = htab_try_create (50, line_hash, line_eq, NULL);
+  if (htab == NULL)
+    dwz_oom ();
+  max_line_id = 1;
+  return htab;
+}
+
 /* Map original file ID to new file ID.  */
 static unsigned int
 line_htab_lookup (dw_cu_ref cu, unsigned int id)
@@ -9947,12 +9957,7 @@ line_htab_lookup (dw_cu_ref cu, unsigned int id)
   if (le.file->dir)
     h = iterative_hash (le.file->dir, strlen (le.file->dir) + 1, h);
   if (line_htab == NULL)
-    {
-      line_htab = htab_try_create (50, line_hash, line_eq, NULL);
-      if (line_htab == NULL)
-	dwz_oom ();
-      max_line_id = 1;
-    }
+    line_htab = line_htab_create ();
   le.hash = h;
   slot = htab_find_slot_with_hash (line_htab, &le, h, INSERT);
   if (slot == NULL)
@@ -12222,7 +12227,11 @@ write_unit_die (unsigned char *ptr, dw_die_ref die, dw_die_ref origin)
 	    assert (p && (form == DW_FORM_sec_offset
 			  || form == DW_FORM_data4));
 	    if (wr_multifile)
-	      write_32 (ptr, multi_line_off);
+	      {
+		write_32 (ptr, multi_line_off);
+		if (line_htab == NULL)
+		  line_htab = line_htab_create ();
+	      }
 	    else if (op_multifile)
 	      write_32 (ptr, 0);
 	    else

Cheers,

Mark