[binutils-gdb] Fix float to LONGEST conversion.

[binutils-gdb] Fix float to LONGEST conversion.

[Ali Tamur]

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=9cab7ecda21c6499b0559f6e676fe222e26141be

commit 9cab7ecda21c6499b0559f6e676fe222e26141be
Author: Ali Tamur <tamur@google.com>
Date:   Tue Aug 27 18:58:57 2019 -0700

    Fix float to LONGEST conversion.
    
    The code used to have undefined behaviour when template parameter is float and
    host_float is NaN, because it attempted to convert NaN value to LONGEST at the
    last statement. This frequently caused crashes on tests that checked "info
    all-registers" (at least when the code is compiled with clang; I didn't test
    with gdb).
    
    gdb/ChangeLog:
    
    	*target-float.c (host_float_ops<T>::to_longest): Update
    	implementation.

Diff:
---
 gdb/ChangeLog      |  5 +++++
 gdb/target-float.c | 17 +++++++++++------
 2 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index 3b28884..eccd4ef 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,3 +1,8 @@
+2019-09-11  Ali Tamur  <tamur@google.com>
+
+	*gdb/target-float.c (host_float_ops<T>::to_longest): Update
+	implementation.
+
 2019-09-11  Christian Biesinger  <cbiesinger@google.com>
 
 	* dbxread.c (read_dbx_symtab): Update.
diff --git a/gdb/target-float.c b/gdb/target-float.c
index 39abb12..0fd71c0 100644
--- a/gdb/target-float.c
+++ b/gdb/target-float.c
@@ -1007,13 +1007,18 @@ host_float_ops<T>::to_longest (const gdb_byte *addr,
 {
   T host_float;
   from_target (type, addr, &host_float);
-  /* Converting an out-of-range value is undefined behavior in C, but we
-     prefer to return a defined value here.  */
-  if (host_float > std::numeric_limits<LONGEST>::max())
-    return std::numeric_limits<LONGEST>::max();
-  if (host_float < std::numeric_limits<LONGEST>::min())
+  T min_possible_range = static_cast<T>(std::numeric_limits<LONGEST>::min());
+  T max_possible_range = -min_possible_range;
+  /* host_float can be converted to an integer as long as it's in
+     the range [min_possible_range, max_possible_range). If not, it is either
+     too large, or too small, or is NaN; in this case return the maximum or
+     minimum possible value.  */
+  if (host_float < max_possible_range && host_float >= min_possible_range)
+    return static_cast<LONGEST> (host_float);
+  if (host_float < min_possible_range)
     return std::numeric_limits<LONGEST>::min();
-  return (LONGEST) host_float;
+  /* This line will be executed if host_float is NaN.  */
+  return std::numeric_limits<LONGEST>::max();
 }
 
 /* Convert signed integer VAL to a target floating-number of type TYPE