[PATCH] Improvements to valid range checks in debug mode

[PATCH] Improvements to valid range checks in debug mode

[Jonathan Wakely]

[-- Attachment #1: Type: text/plain, Size: 298 bytes --]

These should wait for stage 1 but I'm posting them now for comment.

With the change to __gnu_debug::__valid_range we now get a debug
assertion for:

   std::string s;
   std::min_element(std::string::iterator{}, s.end());

where previously it would just crash with undefined behaviour.


[-- Attachment #2: patch.txt --]
[-- Type: text/x-patch, Size: 7010 bytes --]

commit 77a610b7e88635ee7c63d82cc30fad9c80abebea
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Thu Feb 27 11:17:31 2020 +0000

    libstdc++: Minor optimization for min/max/minmax
    
    By calling the internal __min_element (or __max_element or
    __minmax_element) function directly we avoid a function call and the
    valid range checks that are redundant when the range is defined by an
    initializer_list.
    
            * include/bits/stl_algo.h (min(initializer_list<T>))
            (min(initializer_list<T>, Compare)): Call __min_element directly to
            avoid redundant debug checks for valid ranges.
            (max(initializer_list<T>), max(initializer_list<T>, Compare)):
            Likewise, for __max_element.
            (minmax(initializer_list<T>), minmax(initializer_list<T>, Compare)):
            Likewise, for __minmax_element.

diff --git a/libstdc++-v3/include/bits/stl_algo.h b/libstdc++-v3/include/bits/stl_algo.h
index 6503d1518d3..d5eed9c47f6 100644
--- a/libstdc++-v3/include/bits/stl_algo.h
+++ b/libstdc++-v3/include/bits/stl_algo.h
@@ -3543,38 +3543,15 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION
 				   __gnu_cxx::__ops::__iter_comp_iter(__comp));
     }
 
-  // N2722 + DR 915.
-  template<typename _Tp>
-    _GLIBCXX14_CONSTEXPR
-    inline _Tp
-    min(initializer_list<_Tp> __l)
-    { return *std::min_element(__l.begin(), __l.end()); }
-
-  template<typename _Tp, typename _Compare>
-    _GLIBCXX14_CONSTEXPR
-    inline _Tp
-    min(initializer_list<_Tp> __l, _Compare __comp)
-    { return *std::min_element(__l.begin(), __l.end(), __comp); }
-
-  template<typename _Tp>
-    _GLIBCXX14_CONSTEXPR
-    inline _Tp
-    max(initializer_list<_Tp> __l)
-    { return *std::max_element(__l.begin(), __l.end()); }
-
-  template<typename _Tp, typename _Compare>
-    _GLIBCXX14_CONSTEXPR
-    inline _Tp
-    max(initializer_list<_Tp> __l, _Compare __comp)
-    { return *std::max_element(__l.begin(), __l.end(), __comp); }
-
   template<typename _Tp>
     _GLIBCXX14_CONSTEXPR
     inline pair<_Tp, _Tp>
     minmax(initializer_list<_Tp> __l)
     {
+      __glibcxx_requires_irreflexive(__l.begin(), __l.end);
       pair<const _Tp*, const _Tp*> __p =
-	std::minmax_element(__l.begin(), __l.end());
+	std::__minmax_element(__l.begin(), __l.end(),
+			      __gnu_cxx::__ops::__iter_less_iter());
       return std::make_pair(*__p.first, *__p.second);
     }
 
@@ -3583,8 +3560,10 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION
     inline pair<_Tp, _Tp>
     minmax(initializer_list<_Tp> __l, _Compare __comp)
     {
+      __glibcxx_requires_irreflexive_pred(__l.begin(), __l.end, __comp);
       pair<const _Tp*, const _Tp*> __p =
-	std::minmax_element(__l.begin(), __l.end(), __comp);
+	std::__minmax_element(__l.begin(), __l.end(),
+			      __gnu_cxx::__ops::__iter_comp_iter(__comp));
       return std::make_pair(*__p.first, *__p.second);
     }
 
@@ -3959,7 +3938,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION
       for (_RandomAccessIterator __i = __first + 1; __i != __last; ++__i)
 	std::iter_swap(__i, __first + __d(__g, __p_type(0, __i - __first)));
     }
-#endif
+#endif // USE C99_STDINT
 
 #endif // C++11
 
@@ -5902,6 +5881,49 @@ _GLIBCXX_BEGIN_NAMESPACE_ALGO
 				__gnu_cxx::__ops::__iter_comp_iter(__comp));
     }
 
+#if __cplusplus >= 201103L
+  // N2722 + DR 915.
+  template<typename _Tp>
+    _GLIBCXX14_CONSTEXPR
+    inline _Tp
+    min(initializer_list<_Tp> __l)
+    {
+      __glibcxx_requires_irreflexive(__l.begin(), __l.end);
+      return *_GLIBCXX_STD_A::__min_element(__l.begin(), __l.end(),
+	  __gnu_cxx::__ops::__iter_less_iter());
+    }
+
+  template<typename _Tp, typename _Compare>
+    _GLIBCXX14_CONSTEXPR
+    inline _Tp
+    min(initializer_list<_Tp> __l, _Compare __comp)
+    {
+      __glibcxx_requires_irreflexive_pred(__l.begin(), __l.end, __comp);
+      return *_GLIBCXX_STD_A::__min_element(__l.begin(), __l.end(),
+	  __gnu_cxx::__ops::__iter_comp_iter(__comp));
+    }
+
+  template<typename _Tp>
+    _GLIBCXX14_CONSTEXPR
+    inline _Tp
+    max(initializer_list<_Tp> __l)
+    {
+      __glibcxx_requires_irreflexive(__l.begin(), __l.end);
+      return *_GLIBCXX_STD_A::__max_element(__l.begin(), __l.end(),
+	  __gnu_cxx::__ops::__iter_less_iter());
+    }
+
+  template<typename _Tp, typename _Compare>
+    _GLIBCXX14_CONSTEXPR
+    inline _Tp
+    max(initializer_list<_Tp> __l, _Compare __comp)
+    {
+      __glibcxx_requires_irreflexive_pred(__l.begin(), __l.end, __comp);
+      return *_GLIBCXX_STD_A::__max_element(__l.begin(), __l.end(),
+	  __gnu_cxx::__ops::__iter_comp_iter(__comp));
+    }
+#endif // C++11
+
 #if __cplusplus >= 201402L
   /// Reservoir sampling algorithm.
   template<typename _InputIterator, typename _RandomAccessIterator,

commit b183bd10b7ac1f20cc4ec60ebe621c3177297d52
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Thu Feb 27 11:20:54 2020 +0000

    libstdc++: Improve check for valid forward iterator range
    
    Since C++14 we can assume that value-initialized forward iterators are
    not part of a valid range (except the special case of an empty range
    defined by two value-initialized iterators).
    
    This covers the case of null pointers which was checked by the overload
    for input iterators, but also handles non-pointers.
    
            * include/debug/helper_functions.h (__valid_range_aux): Add overload
            for forward iterators that checks for value-initialized iterators.

diff --git a/libstdc++-v3/include/debug/helper_functions.h b/libstdc++-v3/include/debug/helper_functions.h
index 251582d5922..3e016c7643e 100644
--- a/libstdc++-v3/include/debug/helper_functions.h
+++ b/libstdc++-v3/include/debug/helper_functions.h
@@ -161,6 +161,28 @@ namespace __gnu_debug
 	|| (!__check_singular(__first) && !__check_singular(__last));
     }
 
+#if __cplusplus >= 201402L
+  template<typename _ForwardIterator>
+    constexpr bool
+    __valid_range_aux(_ForwardIterator __first, _ForwardIterator __last,
+		      std::forward_iterator_tag)
+    {
+      if (__first == __last)
+	return true;
+
+      // Value-initialized forward iterators cannot form a non-empty range.
+      if (__first == _ForwardIterator() || __last == _ForwardIterator())
+	return false;
+
+      // Hook for checking if safe iterators are singular.
+      if (__check_singular_aux(std::__addressof(__first))
+	  || __check_singular_aux(std::__addressof(__last)))
+	return false;
+
+      return true;
+    }
+#endif
+
   template<typename _InputIterator>
     _GLIBCXX_CONSTEXPR
     inline bool
@@ -168,7 +190,7 @@ namespace __gnu_debug
 		      std::random_access_iterator_tag)
     {
       return
-	__valid_range_aux(__first, __last, std::input_iterator_tag())
+	__valid_range_aux(__first, __last, std::forward_iterator_tag())
 	&& __first <= __last;
     }
 

Re: [PATCH] Improvements to valid range checks in debug mode

[François Dumont]

I just come back to this email and it sounds like the right moment to 
commit it.

On 27/02/20 2:06 pm, Jonathan Wakely wrote:
> These should wait for stage 1 but I'm posting them now for comment.
>
> With the change to __gnu_debug::__valid_range we now get a debug
> assertion for:
>
>   std::string s;
>   std::min_element(std::string::iterator{}, s.end());
>
> where previously it would just crash with undefined behaviour.
>

Re: [PATCH] Improvements to valid range checks in debug mode

[Jonathan Wakely]

On Fri, 13 Aug 2021 at 18:10, François Dumont via Libstdc++
<libstdc++@gcc.gnu.org> wrote:
>
> I just come back to this email and it sounds like the right moment to
> commit it.

Ah yes, thanks for the reminder!

I'll push it tomorrow.

>
> On 27/02/20 2:06 pm, Jonathan Wakely wrote:
> > These should wait for stage 1 but I'm posting them now for comment.
> >
> > With the change to __gnu_debug::__valid_range we now get a debug
> > assertion for:
> >
> >   std::string s;
> >   std::min_element(std::string::iterator{}, s.end());
> >
> > where previously it would just crash with undefined behaviour.
> >
>

Re: [PATCH] Improvements to valid range checks in debug mode

[Jonathan Wakely]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

On Mon, 16 Aug 2021 at 20:26, Jonathan Wakely via Libstdc++
<libstdc++@gcc.gnu.org> wrote:
>
> On Fri, 13 Aug 2021 at 18:10, François Dumont via Libstdc++
> <libstdc++@gcc.gnu.org> wrote:
> >
> > I just come back to this email and it sounds like the right moment to
> > commit it.
>
> Ah yes, thanks for the reminder!
>
> I'll push it tomorrow.
>
> >
> > On 27/02/20 2:06 pm, Jonathan Wakely wrote:
> > > These should wait for stage 1 but I'm posting them now for comment.
> > >
> > > With the change to __gnu_debug::__valid_range we now get a debug
> > > assertion for:
> > >
> > >   std::string s;
> > >   std::min_element(std::string::iterator{}, s.end());
> > >
> > > where previously it would just crash with undefined behaviour.

Actually, that change doesn't work. Some of our container iterators
use a value-initialized iterator as the past-the-end value, so the
check in the new __valid_range_aux function incorrectly rejects some
valid ranges. Maybe I can make it work for bidirectional iterators,
which must be attached to a container to be valid.

For now I've only pushed the second half of the patch, optimizing the
std::min/max/minmax overloads taking an initializer_list (as
attached).

Tested powerpc64le-linux, pushed to trunk.

[-- Attachment #2: patch.txt --]
[-- Type: text/plain, Size: 5080 bytes --]

commit de44eee5d52f5980b6b2b2120940f70cc2fa007e
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Wed Aug 18 12:24:12 2021

    libstdc++: Minor optimization for min/max/minmax
    
    The debug mode checks for a valid range are redundant when we have an
    initializer_list argument, because we know it's a valid range already.
    By making std::min(initialier_list<T>) call the internal __min_element
    function directly we avoid a function call and skip those checks. The
    same can be done for the overload taking a comparison function, and also
    for the std::max and std::minmax overloads for initializer_list
    arguments.
    
    Signed-off-by: Jonathan Wakely <jwakely@redhat.com>
    
    libstdc++-v3/ChangeLog:
    
            * include/bits/stl_algo.h (min(initializer_list<T>))
            (min(initializer_list<T>, Compare)): Call __min_element directly to
            avoid redundant debug checks for valid ranges.
            (max(initializer_list<T>), max(initializer_list<T>, Compare)):
            Likewise, for __max_element.
            (minmax(initializer_list<T>), minmax(initializer_list<T>, Compare)):
            Likewise, for __minmax_element.

diff --git a/libstdc++-v3/include/bits/stl_algo.h b/libstdc++-v3/include/bits/stl_algo.h
index 54ad383711f..ac4f2d0f721 100644
--- a/libstdc++-v3/include/bits/stl_algo.h
+++ b/libstdc++-v3/include/bits/stl_algo.h
@@ -3445,38 +3445,15 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION
 				   __gnu_cxx::__ops::__iter_comp_iter(__comp));
     }
 
-  // N2722 + DR 915.
-  template<typename _Tp>
-    _GLIBCXX14_CONSTEXPR
-    inline _Tp
-    min(initializer_list<_Tp> __l)
-    { return *std::min_element(__l.begin(), __l.end()); }
-
-  template<typename _Tp, typename _Compare>
-    _GLIBCXX14_CONSTEXPR
-    inline _Tp
-    min(initializer_list<_Tp> __l, _Compare __comp)
-    { return *std::min_element(__l.begin(), __l.end(), __comp); }
-
-  template<typename _Tp>
-    _GLIBCXX14_CONSTEXPR
-    inline _Tp
-    max(initializer_list<_Tp> __l)
-    { return *std::max_element(__l.begin(), __l.end()); }
-
-  template<typename _Tp, typename _Compare>
-    _GLIBCXX14_CONSTEXPR
-    inline _Tp
-    max(initializer_list<_Tp> __l, _Compare __comp)
-    { return *std::max_element(__l.begin(), __l.end(), __comp); }
-
   template<typename _Tp>
     _GLIBCXX14_CONSTEXPR
     inline pair<_Tp, _Tp>
     minmax(initializer_list<_Tp> __l)
     {
+      __glibcxx_requires_irreflexive(__l.begin(), __l.end());
       pair<const _Tp*, const _Tp*> __p =
-	std::minmax_element(__l.begin(), __l.end());
+	std::__minmax_element(__l.begin(), __l.end(),
+			      __gnu_cxx::__ops::__iter_less_iter());
       return std::make_pair(*__p.first, *__p.second);
     }
 
@@ -3485,8 +3462,10 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION
     inline pair<_Tp, _Tp>
     minmax(initializer_list<_Tp> __l, _Compare __comp)
     {
+      __glibcxx_requires_irreflexive_pred(__l.begin(), __l.end(), __comp);
       pair<const _Tp*, const _Tp*> __p =
-	std::minmax_element(__l.begin(), __l.end(), __comp);
+	std::__minmax_element(__l.begin(), __l.end(),
+			      __gnu_cxx::__ops::__iter_comp_iter(__comp));
       return std::make_pair(*__p.first, *__p.second);
     }
 
@@ -3793,7 +3772,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION
       for (_RandomAccessIterator __i = __first + 1; __i != __last; ++__i)
 	std::iter_swap(__i, __first + __d(__g, __p_type(0, __i - __first)));
     }
-#endif
+#endif // USE C99_STDINT
 
 #endif // C++11
 
@@ -5746,6 +5725,49 @@ _GLIBCXX_BEGIN_NAMESPACE_ALGO
 				__gnu_cxx::__ops::__iter_comp_iter(__comp));
     }
 
+#if __cplusplus >= 201103L
+  // N2722 + DR 915.
+  template<typename _Tp>
+    _GLIBCXX14_CONSTEXPR
+    inline _Tp
+    min(initializer_list<_Tp> __l)
+    {
+      __glibcxx_requires_irreflexive(__l.begin(), __l.end());
+      return *_GLIBCXX_STD_A::__min_element(__l.begin(), __l.end(),
+	  __gnu_cxx::__ops::__iter_less_iter());
+    }
+
+  template<typename _Tp, typename _Compare>
+    _GLIBCXX14_CONSTEXPR
+    inline _Tp
+    min(initializer_list<_Tp> __l, _Compare __comp)
+    {
+      __glibcxx_requires_irreflexive_pred(__l.begin(), __l.end(), __comp);
+      return *_GLIBCXX_STD_A::__min_element(__l.begin(), __l.end(),
+	  __gnu_cxx::__ops::__iter_comp_iter(__comp));
+    }
+
+  template<typename _Tp>
+    _GLIBCXX14_CONSTEXPR
+    inline _Tp
+    max(initializer_list<_Tp> __l)
+    {
+      __glibcxx_requires_irreflexive(__l.begin(), __l.end());
+      return *_GLIBCXX_STD_A::__max_element(__l.begin(), __l.end(),
+	  __gnu_cxx::__ops::__iter_less_iter());
+    }
+
+  template<typename _Tp, typename _Compare>
+    _GLIBCXX14_CONSTEXPR
+    inline _Tp
+    max(initializer_list<_Tp> __l, _Compare __comp)
+    {
+      __glibcxx_requires_irreflexive_pred(__l.begin(), __l.end(), __comp);
+      return *_GLIBCXX_STD_A::__max_element(__l.begin(), __l.end(),
+	  __gnu_cxx::__ops::__iter_comp_iter(__comp));
+    }
+#endif // C++11
+
 #if __cplusplus >= 201402L
   /// Reservoir sampling algorithm.
   template<typename _InputIterator, typename _RandomAccessIterator,

Re: [PATCH] Improvements to valid range checks in debug mode

[Jonathan Wakely]

[-- Attachment #1: Type: text/plain, Size: 1420 bytes --]

On Wed, 18 Aug 2021 at 15:40, Jonathan Wakely wrote:
>
> On Mon, 16 Aug 2021 at 20:26, Jonathan Wakely via Libstdc++
> <libstdc++@gcc.gnu.org> wrote:
> >
> > On Fri, 13 Aug 2021 at 18:10, François Dumont via Libstdc++
> > <libstdc++@gcc.gnu.org> wrote:
> > >
> > > I just come back to this email and it sounds like the right moment to
> > > commit it.
> >
> > Ah yes, thanks for the reminder!
> >
> > I'll push it tomorrow.
> >
> > >
> > > On 27/02/20 2:06 pm, Jonathan Wakely wrote:
> > > > These should wait for stage 1 but I'm posting them now for comment.
> > > >
> > > > With the change to __gnu_debug::__valid_range we now get a debug
> > > > assertion for:
> > > >
> > > >   std::string s;
> > > >   std::min_element(std::string::iterator{}, s.end());
> > > >
> > > > where previously it would just crash with undefined behaviour.
>
> Actually, that change doesn't work. Some of our container iterators
> use a value-initialized iterator as the past-the-end value, so the
> check in the new __valid_range_aux function incorrectly rejects some
> valid ranges. Maybe I can make it work for bidirectional iterators,
> which must be attached to a container to be valid.

The attached patch is good enough for the original motivation, as shown above.

This makes the debug checks work with basic_string iterators, and for
all other container types we already have safe iterators.

[-- Attachment #2: patch.txt --]
[-- Type: text/plain, Size: 1698 bytes --]

commit fb061814aa724727c5761b53e5a60007355da210
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Thu Feb 27 11:20:54 2020

    libstdc++: Detect null __normal_iterator in valid range checks
    
    Enahnce debug mode checks for valid ranges to detect value-initialized
    __normal_iterator objects, so that basic_string iterators can be
    checked.
    
    Signed-off-by: Jonathan Wakely <jwakely@redhat.com>
    
    libstdc++-v3/ChangeLog:
    
            * include/debug/helper_functions.h (__check_singular): Add overload
            for __normal_iterator that checks for value-initialized iterators.

diff --git a/libstdc++-v3/include/debug/helper_functions.h b/libstdc++-v3/include/debug/helper_functions.h
index c0144ced979..e0c69151e3a 100644
--- a/libstdc++-v3/include/debug/helper_functions.h
+++ b/libstdc++-v3/include/debug/helper_functions.h
@@ -36,6 +36,13 @@
 
 #include <bits/stl_pair.h>			// for pair
 
+namespace __gnu_cxx
+{
+_GLIBCXX_BEGIN_NAMESPACE_VERSION
+  template<typename _Iterator, typename _Container> class __normal_iterator;
+_GLIBCXX_END_NAMESPACE_VERSION
+}
+
 namespace __gnu_debug
 {
   template<typename _Iterator, typename _Sequence, typename _Category>
@@ -144,7 +151,13 @@ namespace __gnu_debug
 	__ptr == 0;
     }
 
-  /** We say that integral types for a valid range, and defer to other
+  template<typename _Tp, typename _Cont>
+    _GLIBCXX_CONSTEXPR
+    inline bool
+    __check_singular(__gnu_cxx::__normal_iterator<_Tp*, _Cont> const& __x)
+    { return __x.base() == 0; }
+
+  /** We say that integral types form a valid range, and defer to other
    *  routines to realize what to do with integral types instead of
    *  iterators.
   */

Re: [PATCH] Improvements to valid range checks in debug mode

[François Dumont]

On 18/08/21 5:55 pm, Jonathan Wakely wrote:
> On Wed, 18 Aug 2021 at 15:40, Jonathan Wakely wrote:
>> On Mon, 16 Aug 2021 at 20:26, Jonathan Wakely via Libstdc++
>> <libstdc++@gcc.gnu.org> wrote:
>>> On Fri, 13 Aug 2021 at 18:10, François Dumont via Libstdc++
>>> <libstdc++@gcc.gnu.org> wrote:
>>>> I just come back to this email and it sounds like the right moment to
>>>> commit it.
>>> Ah yes, thanks for the reminder!
>>>
>>> I'll push it tomorrow.
>>>
>>>> On 27/02/20 2:06 pm, Jonathan Wakely wrote:
>>>>> These should wait for stage 1 but I'm posting them now for comment.
>>>>>
>>>>> With the change to __gnu_debug::__valid_range we now get a debug
>>>>> assertion for:
>>>>>
>>>>>    std::string s;
>>>>>    std::min_element(std::string::iterator{}, s.end());
>>>>>
>>>>> where previously it would just crash with undefined behaviour.
>> Actually, that change doesn't work. Some of our container iterators
>> use a value-initialized iterator as the past-the-end value, so the
>> check in the new __valid_range_aux function incorrectly rejects some
>> valid ranges. Maybe I can make it work for bidirectional iterators,
>> which must be attached to a container to be valid.
> The attached patch is good enough for the original motivation, as shown above.
>
> This makes the debug checks work with basic_string iterators, and for
> all other container types we already have safe iterators.

It would be great, especially considering my proposal to activate 
__valid_range check in _GLIBCXX_ASSERTIONS mode.