[committed][PR rtl-optimization/115876] Avoid ubsan in ext-dce.cc

[committed][PR rtl-optimization/115876] Avoid ubsan in ext-dce.cc

[Jeff Law]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

This fixes two general ubsan issues in ext-dce, both related to use-side 
processsing of modes > DImode.

In ext_dce_process_uses we can be presented with something like this as 
a use (subreg:SI (reg:TF) 12)

That will result in an out of range shift for a HOST_WIDE_INT object. 
Where this happens is safe to just break from the SET context and 
process the subjects.  This will ultimately result in seeing (reg:TF) 
and we'll mark all bit groups as live.

In carry_backpropagate we can be presented with a TImode shift (for 
example) and the shift count can be > 63 for such a shift.  This 
naturally trips ubsan as well as we're operating on 64 bit objects.

We can just return mmask in this case noting that every bit group is live.

The combination of these two fixes eliminates all the reported ubsan 
issues in ext-dce seen in a bootstrap and regression test on x86.

While I was in there I went ahead and fixed the various hardcoded 63/64 
values to be HOST_BITS_PER_WIDE_INT based.

Bootstrapped and regression tested on x86 with no regressions.  Also 
built with ubsan enabled and verified the build logs and testsuite logs 
don't call out any issues in ext-dce anymore.

Pushing to the trunk.

Jeff

[-- Attachment #2: P --]
[-- Type: text/plain, Size: 5181 bytes --]

commit f10d2ee95356b9de6c44d701c4dfa8fb088714d2
Author: Jeff Law <jlaw@ventanamicro.com>
Date:   Sun Aug 18 16:55:52 2024 -0600

    [PR rtl-optimization/115876] Avoid ubsan in ext-dce.cc
    
    This fixes two general ubsan issues in ext-dce, both related to use-side
    processsing of modes > DImode.
    
    In ext_dce_process_uses we can be presented with something like this as a use
    (subreg:SI (reg:TF) 12)
    
    That will result in an out of range shift for a HOST_WIDE_INT object.  Where
    this happens is safe to just break from the SET context and process the
    subjects.  This will ultimately result in seeing (reg:TF) and we'll mark all
    bit groups as live.
    
    In carry_backpropagate we can be presented with a TImode shift (for example)
    and the shift count can be > 63 for such a shift.  This naturally trips ubsan
    as well as we're operating on 64 bit objects.
    
    We can just return mmask in this case noting that every bit group is live.
    
    The combination of these two fixes eliminates all the reported ubsan issues in
    ext-dce seen in a bootstrap and regression test on x86.
    
    While I was in there I went ahead and fixed the various hardcoded 63/64 values
    to be HOST_BITS_PER_WIDE_INT based.
    
    Bootstrapped and regression tested on x86 with no regressions.  Also built with
    ubsan enabled and verified the build logs and testsuite logs don't call out any
    issues in ext-dce anymore.
    
    Pushing to the trunk.
    
            PR rtl-optimization/115876
    gcc
            * ext-dce.cc (ext_dce_process_sets): Replace hardcoded 63/64 instances
            with HOST_BITS_PER_WIDE_INT based values.
            (carry_backpropagate): Handle modes with more bits than
            HOST_BITS_PER_WIDE_INT gracefully, avoiding undefined behavior.
            (ext_dce_process_uses): Handle subreg offsets which would result
            in ubsan shifts gracefully, avoiding undefined behavior.

diff --git a/gcc/ext-dce.cc b/gcc/ext-dce.cc
index 017e2de000d..eee9208f0d6 100644
--- a/gcc/ext-dce.cc
+++ b/gcc/ext-dce.cc
@@ -207,7 +207,7 @@ ext_dce_process_sets (rtx_insn *insn, rtx obj, bitmap live_tmp)
 	     wider than DImode.  */
 	  scalar_int_mode outer_mode;
 	  if (!is_a <scalar_int_mode> (GET_MODE (x), &outer_mode)
-	      || GET_MODE_BITSIZE (outer_mode) > 64)
+	      || GET_MODE_BITSIZE (outer_mode) > HOST_BITS_PER_WIDE_INT)
 	    {
 	      /* Skip the subrtxs of this destination.  There is
 		 little value in iterating into the subobjects, so
@@ -239,7 +239,7 @@ ext_dce_process_sets (rtx_insn *insn, rtx obj, bitmap live_tmp)
 		 that case.  Remember, we can not just continue to process
 		 the inner RTXs due to the STRICT_LOW_PART.  */
 	      if (!is_a <scalar_int_mode> (GET_MODE (SUBREG_REG (x)), &outer_mode)
-		  || GET_MODE_BITSIZE (outer_mode) > 64)
+		  || GET_MODE_BITSIZE (outer_mode) > HOST_BITS_PER_WIDE_INT)
 		{
 		  /* Skip the subrtxs of the STRICT_LOW_PART.  We can't
 		     process them because it'll set objects as no longer
@@ -293,7 +293,7 @@ ext_dce_process_sets (rtx_insn *insn, rtx obj, bitmap live_tmp)
 		 the top of the loop which just complicates the flow even
 		 more.  */
 	      if (!is_a <scalar_int_mode> (GET_MODE (SUBREG_REG (x)), &outer_mode)
-		  || GET_MODE_BITSIZE (outer_mode) > 64)
+		  || GET_MODE_BITSIZE (outer_mode) > HOST_BITS_PER_WIDE_INT)
 		{
 		  skipped_dest = true;
 		  iter.skip_subrtxes ();
@@ -329,7 +329,7 @@ ext_dce_process_sets (rtx_insn *insn, rtx obj, bitmap live_tmp)
 	    }
 
 	  /* BIT >= 64 indicates something went horribly wrong.  */
-	  gcc_assert (bit <= 63);
+	  gcc_assert (bit <= HOST_BITS_PER_WIDE_INT - 1);
 
 	  /* Now handle the actual object that was changed.  */
 	  if (REG_P (x))
@@ -483,6 +483,17 @@ carry_backpropagate (unsigned HOST_WIDE_INT mask, enum rtx_code code, rtx x)
 
   enum machine_mode mode = GET_MODE_INNER (GET_MODE (x));
   unsigned HOST_WIDE_INT mmask = GET_MODE_MASK (mode);
+
+  /* While we don't try to optimize operations on types larger
+     than 64 bits, we do want to make sure not to invoke undefined
+     behavior when presented with such operations during use
+     processing.  The safe thing to do is to just return mmask
+     for that scenario indicating every possible chunk is life.  */
+  scalar_int_mode smode;
+  if (!is_a <scalar_int_mode> (mode, &smode)
+      || GET_MODE_BITSIZE (smode) > HOST_BITS_PER_WIDE_INT)
+    return mmask;
+
   switch (code)
     {
     case PLUS:
@@ -733,8 +744,17 @@ ext_dce_process_uses (rtx_insn *insn, rtx obj,
 				      && SUBREG_PROMOTED_UNSIGNED_P (y)))))
 			break;
 
-		      /* The SUBREG's mode determine the live width.  */
 		      bit = subreg_lsb (y).to_constant ();
+
+		      /* If this is a wide object (more bits than we can fit
+			 in a HOST_WIDE_INT), then just break from the SET
+			 context.   That will cause the iterator to walk down
+			 into the subrtx and if we land on a REG we'll mark
+			 the whole think live.  */
+		      if (bit >= HOST_BITS_PER_WIDE_INT)
+			break;
+
+		      /* The SUBREG's mode determines the live width.  */
 		      if (dst_mask)
 			{
 			  dst_mask <<= bit;