[PATCH] gdb fix for count overflow check (correct)

[PATCH] gdb fix for count overflow check (correct)

[Eric Lubin]

UPDATE: i apologize for the last patch, I forgot to include the library that defines INT_MAX. This one will work


There is a bug in the overflow check. The overflow check tries to assume that signed integers will wrap around on overflow, and thus a number that wraps around after a multiplication by 10 should no longer be divisible by 10. Unfortunately, signed integer overflow is undefined behavior (see section 2.3 of http://pdos.csail.mit.edu/papers/ub:apsys12.pdf). and a check for signed integer overflow can be optimized out by the compiler. Thus, the check for whether count is not divisible by 10 can be optimized to false by compilers, therefore rendering the error check useless. To mitigate this problem, we check whether the multiplication will overflow, before the operation, by comparing count to INT_MAX/ 10. An integer will overflow when multiplied by 10 if and only if that integer is larger than floor(INT_MAX / 10).


CHANGELOG:
2013-12-07  Eric Lubin  <eric@lubin.us>

	* libiberty/cplus-dem.c: Fixed an integer overflow check.


patch:

index e948487..5e6662b 100644
--- a/libiberty/cplus-dem.c
+++ b/libiberty/cplus-dem.c
@@ -44,7 +44,7 @@ Boston, MA 02110-1301, USA.  */
 #endif
 
 #include "safe-ctype.h"
-
+#include <limits.h>
 #include <sys/types.h>
 #include <string.h>
 #include <stdio.h>
@@ -484,6 +484,8 @@ recursively_demangle (struct work_stuff *, const char **, string *, int);
 
    Overflow consumes the rest of the digits, and returns -1.  */
 
+const int overflow = INT_MAX / 10;
+
 static int
 consume_count (const char **type)
 {
@@ -494,20 +496,15 @@ consume_count (const char **type)
 
   while (ISDIGIT ((unsigned char)**type))
     {
-      count *= 10;
 
-      /* Check for overflow.
-        We assume that count is represented using two's-complement;
-        no power of two is divisible by ten, so if an overflow occurs
-        when multiplying by ten, the result will not be a multiple of
-        ten.  */
-      if ((count % 10) != 0)
+      /* Check for overflow.*/
+      if (count > overflow)
        {
          while (ISDIGIT ((unsigned char) **type))
            (*type)++;
          return -1;
        }
-
+      count *= 10;
       count += **type - '0';
       (*type)++;
     }

Re: [PATCH] gdb fix for count overflow check (correct)

[Pedro Alves]

Hi,

Thanks.

libiberty is maintained by the gcc folks (see the
toplevel MAINTAINERS file).  Can you resend this to the
gcc-patches list please?

-- 
Pedro Alves