loc_mentioned_in_p invokes undefined behavior

[Alexandre Oliva <aoliva@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

I'd noticed that loc_mentioned_in_p() wouldn't find loc if it was
mentioned as one of the XVECEXPs of say a concatn, a parallel or
somesuch.  Debugging this, I realized this function would access every
fld of an RTX as if it was a pointer to an RTX itself.  This is wrong:
it makes room for false positives and it accesses as pointers fields
that might have been initialized as narrower or otherwise non-pointer
values, thus invoking undefined behavior.

This patch, that I've already tested in the vta branch, and I'm not
re-testing in mainline (x86_64-linux-gnu for both), should fix this
problem.  Ok to install?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gcc-rtlanal-loc-mentioned-in-vecexp.patch --]
[-- Type: text/x-patch, Size: 968 bytes --]

for  gcc/ChangeLog .vta?
from  Alexandre Oliva  <aoliva@redhat.com>

	* rtlanal.c (loc_mentioned_in_p): Test XVECEXPs correctly.

Index: gcc/rtlanal.c
===================================================================
--- gcc/rtlanal.c.orig	2007-11-05 04:26:09.000000000 -0200
+++ gcc/rtlanal.c	2007-11-05 04:27:22.000000000 -0200
@@ -2989,16 +2989,15 @@ loc_mentioned_in_p (rtx *loc, const_rtx 
   fmt = GET_RTX_FORMAT (code);
   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
     {
-      if (loc == &in->u.fld[i].rt_rtx)
-	return 1;
       if (fmt[i] == 'e')
 	{
-	  if (loc_mentioned_in_p (loc, XEXP (in, i)))
+	  if (loc == &XEXP (in, i) || loc_mentioned_in_p (loc, XEXP (in, i)))
 	    return 1;
 	}
       else if (fmt[i] == 'E')
 	for (j = XVECLEN (in, i) - 1; j >= 0; j--)
-	  if (loc_mentioned_in_p (loc, XVECEXP (in, i, j)))
+	  if (loc == &XVECEXP (in, i, j)
+	      || loc_mentioned_in_p (loc, XVECEXP (in, i, j)))
 	    return 1;
     }
   return 0;

[-- Attachment #3: Type: text/plain, Size: 249 bytes --]


-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
FSF Latin America Board Member         http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}