[Bug other/54620] New: sha1.c has incorrect math if sizeof(size_t) is 8

["pikegadgets+gcc at gmail dot com" <gcc-bugzilla@gcc.gnu.org>]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54620

             Bug #: 54620
           Summary: sha1.c has incorrect math if sizeof(size_t) is 8
    Classification: Unclassified
           Product: gcc
           Version: 4.8.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: other
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: pikegadgets+gcc@gmail.com


In libiberty/sha1.*, I don't see anything saying that sha1_process_block() has
a size limit on its input buffer, and if the length of the buffer is big (e.g.,
2^32 on a 64-bit machine) then this code won't correctly add a 64-bit number to
64-bit number:

  /* First increment the byte count.  RFC 1321 specifies the possible
     length of the file up to 2^64 bits.  Here we only compute the
     number of bytes.  Do a double word increment.  */
  ctx->total[0] += len;
  if (ctx->total[0] < len)
    ++ctx->total[1];

The above is at sha1.c around line 302.

Also, Florian Weimer pointed out that code nearby uses "len & ~63" when it
perhaps should use something like "len & (~(size_t)63)".

Similar bug(s) are in md5.*. See bug 39064 for details.