Re: [PATCH] gas: Add --force-compress-debug-sections

[Tom de Vries <tdevries@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 3597 bytes --]

On 2/23/23 14:44, Jan Beulich wrote:
> On 23.02.2023 14:27, Tom de Vries wrote:
>> On 2/23/23 14:08, Jan Beulich wrote:
>>> On 23.02.2023 13:45, Tom de Vries via Binutils wrote:
>>>> Gas has an option --compress-debug-sections that allows it to generate
>>>> compressed debug sections.
>>>>
>>>> That does not guarantee that the debug sections are in fact compressed:
>>>> ...
>>>> $ gcc ~/hello.c -Wa,-gdwarf-5 -c -Wa,--compress-debug-sections=zstd
>>>> $ readelf -S -W hello.o | grep " .debug"
>>>>     [ 9] .debug_line       PROGBITS         0000a8 000053 00      0   0  1
>>>>     [11] .debug_line_str   PROGBITS         0000fb 000025 01  MS  0   0  1
>>>>     [12] .debug_info       PROGBITS         000120 000039 00      0   0  1
>>>>     [14] .debug_abbrev     PROGBITS         000159 000028 00      0   0  1
>>>>     [15] .debug_aranges    PROGBITS         000190 000030 00      0   0 16
>>>>     [17] .debug_str        PROGBITS         0001c0 000039 01  MS  0   0  1
>>>> ...
>>>>
>>>> Sensibly so, they're only compressed if that provides a size benefit.
>>>>
>>>> However, for the purposes of testing components consuming dwarf
>>>> we may want the sections to be compressed regardless.
>>>>
>>>> Add a new option --force-compress-debug-sections that ignores the size
>>>> heuristic, such that we have instead:
>>>> ...
>>>> $ gcc ~/hello.c -Wa,-gdwarf-5 -c -Wa,--compress-debug-sections=zstd \
>>>>     -Wa,--force-compress-debug-sections
>>>> $ readelf -S -W hello.o | grep " .debug"
>>>>     [ 9] .debug_line       PROGBITS         0000a8 000064 00   C  0   0  8
>>>>     [11] .debug_line_str   PROGBITS         000110 000046 01 MSC  0   0  8
>>>>     [12] .debug_info       PROGBITS         000158 000046 00   C  0   0  8
>>>>     [14] .debug_abbrev     PROGBITS         0001a0 000049 00   C  0   0  8
>>>>     [15] .debug_aranges    PROGBITS         0001f0 000034 00   C  0   0  8
>>>>     [17] .debug_str        PROGBITS         000228 00005a 01 MSC  0   0  8
>>>> ...
>>>>
>>>> Advertised as:
>>>> ...
>>>> $ as --help 2>&1 | grep compress
>>>>     --compress-debug-sections[={none|zlib|zlib-gnu|zlib-gabi|zstd}]
>>>>                             compress DWARF debug sections
>>>>     --nocompress-debug-sections
>>>>                             don't compress DWARF debug sections
>>>>     --force-compress-debug-sections
>>>>                             force compression of DWARF debug sections
>>>
>>> No objection in principle, but have you considered making this a new
>>> sub-option to --compress-debug-sections, i.e. compress-debug-sections=force?
>>
>> I did consider adding a "force-" prefix variant for all the non-none
>> sub-options, but decided to go with the simplest solution first.
>>
>> Your suggestion, --compress-debug-sections=force is more orthogonal,
>> though it breaks the pattern that all the sub-options are mutually
>> exclusive.
>>
>> We could have it be standalone, so you'd do:
>> --compress-debug-sections=zstd --compress-debug-sections=force.
>>
>> Or instead combined: --compress-debug-sections=force,zstd.  Harder to
>> parse though, I suppose.
> 
> I think both should be allowed. In a complex build system it may be
> different entities setting "how" and "whether". (To me "none" falls in
> the "whether" category together with "force", and it also can be seen
> as falling in the "how" category together with "zlib" etc. In Linux
> Kconfig, for example, I'd see this being expressed as first a "whether"
> choice [yes/maybe/forced] and then a "how" choice dependent upon
> "whether != none".)
> 

I gave this approach a try.

Thanks,
- Tom


[-- Attachment #2: 0001-gas-Add-compress-debug-sections-force.patch --]
[-- Type: text/x-patch, Size: 11188 bytes --]

From 792f364b885126b83ad50619085426aca1bc308b Mon Sep 17 00:00:00 2001
From: Tom de Vries <tdevries@suse.de>
Date: Thu, 23 Feb 2023 12:53:40 +0100
Subject: [PATCH] gas: Add --compress-debug-sections=force

Gas has an option --compress-debug-sections that allows it to generate
compressed debug sections.

That does not guarantee that the debug sections are in fact compressed:
...
$ gcc hello.c -Wa,-gdwarf-5 -c -Wa,--compress-debug-sections=zstd
$ readelf -S -W hello.o | grep " .debug"
  [ 9] .debug_line       PROGBITS         0000a8 000053 00      0   0  1
  [11] .debug_line_str   PROGBITS         0000fb 000025 01  MS  0   0  1
  [12] .debug_info       PROGBITS         000120 000039 00      0   0  1
  [14] .debug_abbrev     PROGBITS         000159 000028 00      0   0  1
  [15] .debug_aranges    PROGBITS         000190 000030 00      0   0 16
  [17] .debug_str        PROGBITS         0001c0 000039 01  MS  0   0  1
...

Sensibly so, they're only compressed if that provides a size benefit.

However, for the purpose of testing components consuming dwarf
we may want the sections to be compressed regardless.

Add a new suboption --compress-debug-sections=force that ignores the size
heuristic, such that we have instead:
...
$ gcc hello.c -Wa,-gdwarf-5 -c -Wa,--compress-debug-sections=zstd+force
$ readelf -S -W hello.o | grep " .debug"
  [ 9] .debug_line       PROGBITS         0000a8 000064 00   C  0   0  8
  [11] .debug_line_str   PROGBITS         000110 000046 01 MSC  0   0  8
  [12] .debug_info       PROGBITS         000158 000046 00   C  0   0  8
  [14] .debug_abbrev     PROGBITS         0001a0 000049 00   C  0   0  8
  [15] .debug_aranges    PROGBITS         0001f0 000034 00   C  0   0  8
  [17] .debug_str        PROGBITS         000228 00005a 01 MSC  0   0  8
...

Advertised as:
...
$ as --help
  ...
  --compress-debug-sections[={none|<format>|force|force+<format>}]
                          where <format> is {zlib|zlib-gnu|zlib-gabi|zstd}
                          compress DWARF debug sections
                            Default: zstd
...

Tested on x86_64-linux.
---
 gas/as.c        | 158 ++++++++++++++++++++++++++++++++++++++++++------
 gas/as.h        |   4 ++
 gas/doc/as.texi |  10 ++-
 gas/write.c     |   4 +-
 4 files changed, 153 insertions(+), 23 deletions(-)

diff --git a/gas/as.c b/gas/as.c
index 598bfd56cf5..3c38b7a45e9 100644
--- a/gas/as.c
+++ b/gas/as.c
@@ -230,6 +230,18 @@ enum compressed_debug_section_type flag_compress_debug
   = DEFAULT_COMPRESSED_DEBUG_ALGORITHM;
 #endif
 
+enum compress_debug_action
+{
+  cda_default,
+  cda_none,
+  cda_force,
+  cda_yes,
+};
+static enum compress_debug_action compress_debug_action
+  = cda_default;
+
+bool force_compress_debug = false;
+
 static void
 show_usage (FILE * stream)
 {
@@ -252,7 +264,8 @@ Options:\n\
   fprintf (stream, _("\
   --alternate             initially turn on alternate macro syntax\n"));
   fprintf (stream, _("\
-  --compress-debug-sections[={none|zlib|zlib-gnu|zlib-gabi|zstd}]\n\
+  --compress-debug-sections[={none|<format>|force|force+<format>}]\n\
+                          where <format> is {zlib|zlib-gnu|zlib-gabi|zstd}\n\
                           compress DWARF debug sections\n")),
   fprintf (stream, _("\
 		            Default: %s\n"),
@@ -418,6 +431,126 @@ Options:\n\
     fprintf (stream, _("Report bugs to %s\n"), REPORT_BUGS_TO);
 }
 
+static void
+parse_compress_debug_optarg_1 (const char *optarg, unsigned int *none_cnt,
+			       unsigned int *force_cnt,
+			       unsigned int *format_cnt,
+			       enum compressed_debug_section_type *format)
+{
+  gas_assert (optarg != NULL);
+
+  if (strcmp (optarg, "force") == 0)
+    {
+      ++*force_cnt;
+      return;
+    }
+
+  enum compressed_debug_section_type tmp
+    = bfd_get_compression_algorithm (optarg);
+
+#ifndef HAVE_ZSTD
+  if (tmp == COMPRESS_DEBUG_ZSTD)
+    as_fatal (_ ("--compress-debug-sections=zstd: gas is not "
+		 "built with zstd support"));
+#endif
+
+  if (tmp == COMPRESS_UNKNOWN)
+    as_fatal (_("Invalid --compress-debug-sections option: `%s'"),
+	      optarg);
+
+  if (tmp == COMPRESS_DEBUG_NONE)
+    {
+      ++*none_cnt;
+      return;
+    }
+
+  *format = tmp;
+  ++*format_cnt;
+}
+
+static void
+parse_compress_debug_optarg (const char *optarg)
+{
+  const char *orig_optarg = optarg;
+
+#if !defined OBJ_ELF && !defined OBJ_MAYBE_ELF
+  as_fatal (_("--compress-debug-sections=%s is unsupported"),
+	    optarg);
+#endif
+
+  /* Tokenize subopts seperated by '+' and pass to
+     parse_compress_debug_optarg_1.  */
+  unsigned int none_cnt = 0;
+  unsigned int force_cnt = 0;
+  unsigned int format_cnt = 0;
+  enum compressed_debug_section_type format = COMPRESS_UNKNOWN;
+  while (true)
+    {
+      const char *idx = optarg;
+      while (*idx != '\0' && *idx != '+')
+	idx++;
+
+      size_t len = idx - optarg;
+      if (len == 0)
+	{
+	  /* Generate error.  */
+	  parse_compress_debug_optarg_1 ("", NULL, NULL, NULL, NULL);
+	  break;
+	}
+
+      char *tmp = xstrndup (optarg, len);
+      parse_compress_debug_optarg_1 (tmp, &none_cnt, &force_cnt, &format_cnt,
+				     &format);
+      free (tmp);
+
+      if (*idx == '\0')
+	break;
+
+      /* Step over '+' and continue tokenizing.  */
+      gas_assert (*idx == '+');
+      optarg = idx + 1;
+  }
+
+  bool valid_p = true;
+  if (force_cnt >= 2 || none_cnt >= 2 || format_cnt >= 2)
+    /* Not allowed: force+force, none+none, zstd+zstd, zstd+zlib.  */
+    valid_p = false;
+  else if (none_cnt == 1 && (force_cnt > 0 || format_cnt > 0))
+    /* Not allowed: none+force, none+zstd.  */
+    valid_p = false;
+	
+  if (!valid_p)
+    as_fatal (_("Invalid --compress-debug-sections option combination: `%s'"),
+	      orig_optarg);
+
+  if (none_cnt == 1)
+    /* Let --compress-debug-section=none override
+       --compress-debug-section and --compress-debug-section=force.  */
+    compress_debug_action = cda_none;
+  else if (force_cnt == 1)
+    /* Let --compress-debug-section=force override
+       --compress-debug-section and --compress-debug-section=none.  */
+    compress_debug_action = cda_force;
+  else
+    /* Let --compress-debug-section override
+       --compress-debug-section=none and --compress-debug-section=force.  */
+    compress_debug_action = cda_yes;
+
+  if (format_cnt == 1)
+    /* Let --compress-debug-section=zlib override
+       --compress-debug-section=zstd.  */
+    flag_compress_debug = format;
+}
+
+static void
+finalize_parse_compress_debug_optarg (void)
+{
+  if (compress_debug_action == cda_none)
+    flag_compress_debug = COMPRESS_DEBUG_NONE;
+  else if (compress_debug_action == cda_force)
+    force_compress_debug = true;
+}
+
 /* Since it is easy to do here we interpret the special arg "-"
    to mean "use stdin" and we set that argv[] pointing to "".
    After we have munged argv[], the only things left are source file
@@ -747,28 +880,13 @@ This program has absolutely no warranty.\n"));
 
 	case OPTION_COMPRESS_DEBUG:
 	  if (optarg)
-	    {
-#if defined OBJ_ELF || defined OBJ_MAYBE_ELF
-	      flag_compress_debug = bfd_get_compression_algorithm (optarg);
-#ifndef HAVE_ZSTD
-	      if (flag_compress_debug == COMPRESS_DEBUG_ZSTD)
-		  as_fatal (_ ("--compress-debug-sections=zstd: gas is not "
-			       "built with zstd support"));
-#endif
-	      if (flag_compress_debug == COMPRESS_UNKNOWN)
-		as_fatal (_("Invalid --compress-debug-sections option: `%s'"),
-			  optarg);
-#else
-	      as_fatal (_("--compress-debug-sections=%s is unsupported"),
-			optarg);
-#endif
-	    }
+	    parse_compress_debug_optarg (optarg);
 	  else
-	    flag_compress_debug = COMPRESS_DEBUG_GABI_ZLIB;
+	    parse_compress_debug_optarg ("zlib-gabi");
 	  break;
 
 	case OPTION_NOCOMPRESS_DEBUG:
-	  flag_compress_debug = COMPRESS_DEBUG_NONE;
+	  parse_compress_debug_optarg ("none");
 	  break;
 
 	case OPTION_DEBUG_PREFIX_MAP:
@@ -1136,6 +1254,8 @@ This program has absolutely no warranty.\n"));
   *pargc = new_argc;
   *pargv = new_argv;
 
+  finalize_parse_compress_debug_optarg ();
+
 #ifdef md_after_parse_args
   md_after_parse_args ();
 #endif
diff --git a/gas/as.h b/gas/as.h
index 4c5fa9ecf7d..115af019815 100644
--- a/gas/as.h
+++ b/gas/as.h
@@ -331,6 +331,10 @@ COMMON int flag_traditional_format;
 /* Type of compressed debug sections we should generate.   */
 COMMON enum compressed_debug_section_type flag_compress_debug;
 
+/* True if we want to generate compressed debug sections, even if it
+   doesn't make them smaller.  */
+COMMON bool force_compress_debug;
+
 /* TRUE if .note.GNU-stack section with SEC_CODE should be created */
 COMMON int flag_execstack;
 
diff --git a/gas/doc/as.texi b/gas/doc/as.texi
index bbdfa4bfdca..d25559141c7 100644
--- a/gas/doc/as.texi
+++ b/gas/doc/as.texi
@@ -718,7 +718,8 @@ Begin in alternate macro mode.
 Compress DWARF debug sections using zlib with SHF_COMPRESSED from the
 ELF ABI.  The resulting object file may not be compatible with older
 linkers and object file utilities.  Note if compression would make a
-given section @emph{larger} then it is not compressed.
+given section @emph{larger} then it is not compressed, unless
+@option{--compress-debug-section=force} is used.
 
 @ifset ELF
 @cindex @samp{--compress-debug-sections=} option
@@ -727,6 +728,7 @@ given section @emph{larger} then it is not compressed.
 @itemx --compress-debug-sections=zlib-gnu
 @itemx --compress-debug-sections=zlib-gabi
 @itemx --compress-debug-sections=zstd
+@itemx --compress-debug-sections=force
 These options control how DWARF debug sections are compressed.
 @option{--compress-debug-sections=none} is equivalent to
 @option{--nocompress-debug-sections}.
@@ -738,7 +740,11 @@ using the obsoleted zlib-gnu format.  The debug sections are renamed to begin
 with @samp{.zdebug}.
 @option{--compress-debug-sections=zstd} compresses DWARF debug
 sections using zstd.  Note - if compression would actually make a section
-@emph{larger}, then it is not compressed nor renamed.
+@emph{larger}, then it is not compressed nor renamed, unless
+@option{--compress-debug-section=force} is used.
+@option{--compress-debug-sections=force} compresses DWARF debug sections,
+even if this does not reduce size.  It can be used in conjunction with a format
+selection, for instance @option{--compress-debug-section=zstd+force}.
 
 @end ifset
 
diff --git a/gas/write.c b/gas/write.c
index 8273b7a42f1..39bcea23fac 100644
--- a/gas/write.c
+++ b/gas/write.c
@@ -1465,7 +1465,7 @@ compress_debug (bfd *abfd, asection *sec, void *xxx ATTRIBUTE_UNUSED)
   flagword flags = bfd_section_flags (sec);
 
   if (seginfo == NULL
-      || uncompressed_size < 32
+      || (!force_compress_debug && uncompressed_size < 32)
       || (flags & SEC_HAS_CONTENTS) == 0)
     return;
 
@@ -1582,7 +1582,7 @@ compress_debug (bfd *abfd, asection *sec, void *xxx ATTRIBUTE_UNUSED)
 
   /* PR binutils/18087: If compression didn't make the section smaller,
      just keep it uncompressed.  */
-  if (compressed_size >= uncompressed_size)
+  if (!force_compress_debug && compressed_size >= uncompressed_size)
     return;
 
   /* Replace the uncompressed frag list with the compressed frag list.  */
-- 
2.35.3