[PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 10053 bytes --]

Hi.

As slightly discussed here: https://gcc.gnu.org/ml/gcc-patches/2018-10/msg01674.html I fixed
a situation where an equal operator of a hash table returns true, while corresponding hash
value of a pair of elements is different. That's inconsistent and can probably cause issues
in different areas of compiler.

I wrote a simple patch that verifies for a newly added element into a hash table that there's
no other equal element with a different hash. That's O(n) for each insert, so that it's only
enabled with -fchecking=3. Apart from that the way how it's enable is a bit cumbersome, but
that's caused by usage of hash-table also in generated files.

Anyway, there are first places where I see violation of the sanity check:

1) cselib_lookup_1:

$ cat ice.c
a() { b(); }

$ /dev/shm/objdir/gcc/xgcc -B/dev/shm/objdir/gcc/ ice.c -g -c -fchecking=3 -O
hash table checking failed: equal operator returns true for a pair of values with a different hash valueduring RTL pass: vartrack
ice.c:1:1: internal compiler error: in find_slot_with_hash, at hash-table.h:905
    1 | a() { b(); }
      | ^
0x9680b5 hash_table<cselib_hasher, xcallocator>::find_slot_with_hash(cselib_hasher::key* const&, unsigned int, insert_option)
	/home/marxin/Programming/gcc/gcc/hash-table.h:905
0x962518 cselib_find_slot
	/home/marxin/Programming/gcc/gcc/cselib.c:584
0x9625d4 cselib_lookup_1
	/home/marxin/Programming/gcc/gcc/cselib.c:2097
0x9625d4 cselib_lookup(rtx_def*, machine_mode, int, machine_mode)
	/home/marxin/Programming/gcc/gcc/cselib.c:2141
0x965ee7 cselib_record_sets
	/home/marxin/Programming/gcc/gcc/cselib.c:2593
0x9670a9 cselib_process_insn(rtx_insn*)
	/home/marxin/Programming/gcc/gcc/cselib.c:2790
0x1036b73 vt_initialize
	/home/marxin/Programming/gcc/gcc/var-tracking.c:10231
0x103b98a variable_tracking_main_1
	/home/marxin/Programming/gcc/gcc/var-tracking.c:10460
0x103b98a variable_tracking_main()
	/home/marxin/Programming/gcc/gcc/var-tracking.c:10513

2) gfc_find_module

$ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gfortran.dg/coarray/alloc_comp_2.f90 -fcoarray=single -fchecking=3
hash table checking failed: equal operator returns true for a pair of values with a different hash valuef951: internal compiler error: in find_slot_with_hash, at hash-table.h:905
0x8e5e86 hash_table<module_hasher, xcallocator>::find_slot_with_hash(char const* const&, unsigned int, insert_option)
	/home/marxin/Programming/gcc/gcc/hash-table.h:905
0x8e2c2c gfc_find_module(char const*)
	/home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:4865
0x8e4f42 gfc_generate_module_vars(gfc_namespace*)
	/home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:5475
0x8b8d7e gfc_generate_module_code(gfc_namespace*)
	/home/marxin/Programming/gcc/gcc/fortran/trans.c:2190
0x868427 translate_all_program_units
	/home/marxin/Programming/gcc/gcc/fortran/parse.c:6112
0x868427 gfc_parse_file()
	/home/marxin/Programming/gcc/gcc/fortran/parse.c:6328
0x8b19cb gfc_be_parse_file
	/home/marxin/Programming/gcc/gcc/fortran/f95-lang.c:204

3) lookup_template_class_1

$ ./xg++ -B. /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C -c -fchecking=3
hash table checking failed: equal operator returns true for a pair of values with a different hash value/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C: In instantiation of ‘struct B<A>’:
/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:15:8:   required from here
/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:8:17: internal compiler error: in find_slot_with_hash, at hash-table.h:905
    8 |     friend bool foo (const B<Q>& a);
      |                 ^~~
0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option)
	/home/marxin/Programming/gcc/gcc/hash-table.h:905
0xa042ce lookup_template_class_1
	/home/marxin/Programming/gcc/gcc/cp/pt.c:9629
0xa042ce lookup_template_class(tree_node*, tree_node*, tree_node*, tree_node*, int, int)
	/home/marxin/Programming/gcc/gcc/cp/pt.c:9674
0xa03670 tsubst_aggr_type
	/home/marxin/Programming/gcc/gcc/cp/pt.c:12679
0x9fefcd tsubst(tree_node*, tree_node*, int, tree_node*)
	/home/marxin/Programming/gcc/gcc/cp/pt.c:14294
0x9fe1a9 tsubst(tree_node*, tree_node*, int, tree_node*)
	/home/marxin/Programming/gcc/gcc/cp/pt.c:14285
0xa0d8bd tsubst_arg_types
	/home/marxin/Programming/gcc/gcc/cp/pt.c:13891
0xa0dc24 tsubst_function_type
	/home/marxin/Programming/gcc/gcc/cp/pt.c:14032
0x9fe790 tsubst(tree_node*, tree_node*, int, tree_node*)
	/home/marxin/Programming/gcc/gcc/cp/pt.c:14769
0x9f2c7c tsubst_function_decl
	/home/marxin/Programming/gcc/gcc/cp/pt.c:12921
0xa02d27 tsubst_template_decl
	/home/marxin/Programming/gcc/gcc/cp/pt.c:13214
0x9f4416 tsubst_decl
	/home/marxin/Programming/gcc/gcc/cp/pt.c:13316
0x9ff0ca tsubst(tree_node*, tree_node*, int, tree_node*)
	/home/marxin/Programming/gcc/gcc/cp/pt.c:14212
0xa1dfd0 tsubst_friend_function
	/home/marxin/Programming/gcc/gcc/cp/pt.c:10310
0xa1dfd0 instantiate_class_template_1
	/home/marxin/Programming/gcc/gcc/cp/pt.c:11359
0xa1dfd0 instantiate_class_template(tree_node*)
	/home/marxin/Programming/gcc/gcc/cp/pt.c:11424
0xa66b22 complete_type(tree_node*)
	/home/marxin/Programming/gcc/gcc/cp/typeck.c:138
0x9023c7 start_decl_1(tree_node*, bool)
	/home/marxin/Programming/gcc/gcc/cp/decl.c:5278
0x92a15f start_decl(cp_declarator const*, cp_decl_specifier_seq*, int, tree_node*, tree_node*, tree_node**)
	/home/marxin/Programming/gcc/gcc/cp/decl.c:5241
0x9c1944 cp_parser_init_declarator
	/home/marxin/Programming/gcc/gcc/cp/parser.c:19750

4) register_specialization

$ ./xg++ -B. /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/cpp0x/udlit-template.C -I/dev/shm/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu -I/dev/shm/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include -I/home/marxin/Programming/gcc/libstdc++-v3/libsupc++ -I/home/marxin/Programming/gcc/libstdc++-v3/include/ba -c -fchecking=3
hash table checking failed: equal operator returns true for a pair of values with a different hash value/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/cpp0x/udlit-template.C:13:21: internal compiler error: in find_slot_with_hash, at hash-table.h:905
   13 |   operator"" _abc<>()
      |                     ^
0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option)
	/home/marxin/Programming/gcc/gcc/hash-table.h:905
0x9e35e6 register_specialization
	/home/marxin/Programming/gcc/gcc/cp/pt.c:1534
0xa22ac3 check_explicit_specialization(tree_node*, tree_node*, int, int, tree_node*)
	/home/marxin/Programming/gcc/gcc/cp/pt.c:3243
0x91552d grokfndecl
	/home/marxin/Programming/gcc/gcc/cp/decl.c:9106
0x9274bd grokdeclarator(cp_declarator const*, cp_decl_specifier_seq*, decl_context, int, tree_node**)
	/home/marxin/Programming/gcc/gcc/cp/decl.c:12607
0x92a9c6 start_function(cp_decl_specifier_seq*, cp_declarator const*, tree_node*)
	/home/marxin/Programming/gcc/gcc/cp/decl.c:15470
0x9c14a9 cp_parser_function_definition_from_specifiers_and_declarator
	/home/marxin/Programming/gcc/gcc/cp/parser.c:26853
0x9c14a9 cp_parser_init_declarator
	/home/marxin/Programming/gcc/gcc/cp/parser.c:19654
0x9c8ebd cp_parser_single_declaration
	/home/marxin/Programming/gcc/gcc/cp/parser.c:27437
0x9c9b11 cp_parser_explicit_specialization
	/home/marxin/Programming/gcc/gcc/cp/parser.c:16802
0x9cf30e cp_parser_declaration
	/home/marxin/Programming/gcc/gcc/cp/parser.c:12822
0x9cf94d cp_parser_translation_unit
	/home/marxin/Programming/gcc/gcc/cp/parser.c:4631
0x9cf94d c_parse_file()
	/home/marxin/Programming/gcc/gcc/cp/parser.c:39108
0xadde4a c_common_parse_file()
	/home/marxin/Programming/gcc/gcc/c-family/c-opts.c:1150

5) the case fixed in PR86158:

$ /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/ipa/iinline-2.c:37:1: internal compiler error: in find_slot_with_hash, at hash-table.h:905
0xba10f6 hash_table<ipa_vr_ggc_hash_traits, xcallocator>::find_slot_with_hash(value_range* const&, unsigned int, insert_option)
        /home/marxin/Programming/gcc/gcc/hash-table.h:905
0xb9e02c hash_table<ipa_vr_ggc_hash_traits, xcallocator>::find_slot(value_range* const&, insert_option)
        /home/marxin/Programming/gcc/gcc/hash-table.h:418
0xb8ea21 ipa_get_value_range
        /home/marxin/Programming/gcc/gcc/ipa-prop.c:1776
0xb8eab6 ipa_get_value_range
        /home/marxin/Programming/gcc/gcc/ipa-prop.c:1795
0xb8eae5 ipa_set_jfunc_vr
        /home/marxin/Programming/gcc/gcc/ipa-prop.c:1806
0xb8ef49 ipa_compute_jump_functions_for_edge
        /home/marxin/Programming/gcc/gcc/ipa-prop.c:1875
0xb8fb53 ipa_compute_jump_functions_for_bb
        /home/marxin/Programming/gcc/gcc/ipa-prop.c:2017
0xb912ab analysis_dom_walker::before_dom_children(basic_block_def*)
        /home/marxin/Programming/gcc/gcc/ipa-prop.c:2535
0x158f997 dom_walker::walk(basic_block_def*)
        /home/marxin/Programming/gcc/gcc/domwalk.c:353
0xb915b0 ipa_analyze_node(cgraph_node*)
        /home/marxin/Programming/gcc/gcc/ipa-prop.c:2605
0xb776bf inline_indirect_intraprocedural_analysis
        /home/marxin/Programming/gcc/gcc/ipa-fnsummary.c:3125
0xb776bf inline_analyze_function(cgraph_node*)
        /home/marxin/Programming/gcc/gcc/ipa-fnsummary.c:3145
0xb77866 ipa_fn_summary_generate
        /home/marxin/Programming/gcc/gcc/ipa-fnsummary.c:3189
0xca1751 execute_ipa_summary_passes(ipa_opt_pass_d*)
        /home/marxin/Programming/gcc/gcc/passes.c:2131
0x96907a ipa_passes
        /home/marxin/Programming/gcc/gcc/cgraphunit.c:2505
0x96907a symbol_table::compile()
        /home/marxin/Programming/gcc/gcc/cgraphunit.c:2616
0x96b055 symbol_table::finalize_compilation_unit()
        /home/marxin/Programming/gcc/gcc/cgraphunit.c:2861

My question is whether we want to have in GCC 9 time frame or should I wait with that?
Does it worth implementing?

Thanks,
Martin

---
 gcc/hash-table.c |  2 ++
 gcc/hash-table.h | 25 ++++++++++++++++++++++++-
 gcc/toplev.c     |  3 +++
 3 files changed, 29 insertions(+), 1 deletion(-)



[-- Attachment #2: 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch --]
[-- Type: text/x-patch, Size: 2084 bytes --]

diff --git a/gcc/hash-table.c b/gcc/hash-table.c
index bff9644ae81..d396a368171 100644
--- a/gcc/hash-table.c
+++ b/gcc/hash-table.c
@@ -121,3 +121,5 @@ void dump_hash_table_loc_statistics (void)
       hash_table_usage ().dump (origin);
     }
 }
+
+bool hash_table_verify_p = false;
diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index bd83345c7b8..2d740b42535 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -240,6 +240,10 @@ along with GCC; see the file COPYING3.  If not see
 #include "hash-traits.h"
 #include "hash-map-traits.h"
 
+#ifndef GENERATOR_FILE
+extern bool hash_table_verify_p;
+#endif
+
 template<typename, typename, typename> class hash_map;
 template<typename, typename> class hash_set;
 
@@ -883,12 +887,31 @@ hash_table<Descriptor, Allocator>
     expand ();
 
   m_searches++;
+  size_t size = m_size;
+
+#ifndef GENERATOR_FILE
+  if (hash_table_verify_p)
+    if (insert == INSERT)
+      for (size_t i = 0; i < size; i++)
+	{
+	  value_type *entry = &m_entries[i];
+	  if (!is_empty (*entry) && !is_deleted (*entry)
+	      && Descriptor::equal (*entry, comparable)
+	      && hash != Descriptor::hash (*entry))
+	    {
+	      fprintf (stderr, "hash table checking failed: "
+		       "equal operator returns true for a pair "
+		       "of values with a different hash value");
+	      gcc_unreachable ();
+	    }
+	}
+#endif
+// TODO: enable it also for generated files: there are failures!
 
   value_type *first_deleted_slot = NULL;
   hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
   hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
   value_type *entry = &m_entries[index];
-  size_t size = m_size;
   if (is_empty (*entry))
     goto empty_entry;
   else if (is_deleted (*entry))
diff --git a/gcc/toplev.c b/gcc/toplev.c
index d7ea11abf53..4cd3aafa2e8 100644
--- a/gcc/toplev.c
+++ b/gcc/toplev.c
@@ -2289,6 +2289,9 @@ toplev::main (int argc, char **argv)
 
   handle_common_deferred_options ();
 
+  if (flag_checking > 2)
+    hash_table_verify_p = true;
+
   init_local_tick ();
 
   initialize_plugins ();

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Alexander Monakov]

[-- Attachment #1: Type: text/plain, Size: 1455 bytes --]

On Mon, 29 Oct 2018, Martin Liška wrote:
> My question is whether we want to have in GCC 9 time frame or should I wait with that?
> Does it worth implementing?

This is cool, thanks! A few questions/comments on the patch.

I think there are places that use libiberty C-style hashtab (htab_t), would it
make sense to have this kind of checking for them as well?  I think it's going
to be more complicated though, no need to do both in one step.

I would recommend to factor out the error reporting path into a separate
non-template function, e.g. hashtab_chk_error.  See how qsort_chk_error
has "cold" and "noreturn" attributes and invokes problematic comparators:
the idea was that a developer can 'break qsort_chk_error' in GDB and then
easily step into broken comparators.

Furthermore, it might be nice to investigate if the entire checking loop can
be factored out somehow into a non-template function to avoid having multiple
instantiations of it for different hashtable template parameters.

On my first attempt to submit qsort_chk, Richi asked how much it slows down
stage2, do you have some data on the cost of this hashtable checking?

I think it is possible to optimize this a bit: instead of checking on
insertions, check on deletions and when destroying the table (with care to do
n^2/2 rather than n^2 tests). Although, such approach would miss errors on
hashtables that are never destroyed (leaked or deliberately not deleted).

Alexander

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 2443 bytes --]

On 10/29/18 2:53 PM, Alexander Monakov wrote:
> On Mon, 29 Oct 2018, Martin Liška wrote:
>> My question is whether we want to have in GCC 9 time frame or should I wait with that?
>> Does it worth implementing?
> 
> This is cool, thanks! A few questions/comments on the patch.

Hi.

Thanks for support.

> 
> I think there are places that use libiberty C-style hashtab (htab_t), would it
> make sense to have this kind of checking for them as well?  I think it's going
> to be more complicated though, no need to do both in one step.

Sure, that can be also added in the future!

> 
> I would recommend to factor out the error reporting path into a separate
> non-template function, e.g. hashtab_chk_error.  See how qsort_chk_error
> has "cold" and "noreturn" attributes and invokes problematic comparators:
> the idea was that a developer can 'break qsort_chk_error' in GDB and then
> easily step into broken comparators.

I did so.

> 
> Furthermore, it might be nice to investigate if the entire checking loop can
> be factored out somehow into a non-template function to avoid having multiple
> instantiations of it for different hashtable template parameters.

Note that Jakub is preferring to enable the checking only in non-release builds.
Thus I've guarded the checking with #if ENABLE_EXTRA_CHECKING. That said I won't
care much about saving a binary size.

The checking function is dependent on template arguments, so the avoidance is probably
not possible.

> 
> On my first attempt to submit qsort_chk, Richi asked how much it slows down
> stage2, do you have some data on the cost of this hashtable checking?

It does not survive bootstrap right now (too many cselib_find_slot failures).

> 
> I think it is possible to optimize this a bit: instead of checking on
> insertions, check on deletions and when destroying the table (with care to do
> n^2/2 rather than n^2 tests). Although, such approach would miss errors on
> hashtables that are never destroyed (leaked or deliberately not deleted).

Interesting idea. Benefit of the current approach is that the ICE is triggered
as soon as first bad element is inserted into a hast table. Which means it's easier
to debug. One problem with the destruction is that one equals function is defined on:
Descriptor::equal (Descriptor::value_type, Descriptor::compare_type). One can have
a table where these are not equal and so that equals can't be called.

Martin

> 
> Alexander
> 


[-- Attachment #2: 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch --]
[-- Type: text/x-patch, Size: 2427 bytes --]

From 9afc42d324820cc0fc8a8a8b2d9cccd218734d10 Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Mon, 29 Oct 2018 09:38:21 +0100
Subject: [PATCH] Sanitize equals and hash functions in hash-tables.

---
 gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 39 insertions(+), 1 deletion(-)

diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index bd83345c7b8..43adfac2dc0 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -503,6 +503,7 @@ private:
 
   value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
   value_type *find_empty_slot_for_expand (hashval_t);
+  void verify (const compare_type &comparable, hashval_t hash);
   bool too_empty_p (unsigned int);
   void expand ();
   static bool is_deleted (value_type &v)
@@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
   if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
     expand ();
 
-  m_searches++;
+#if ENABLE_EXTRA_CHECKING
+    if (insert == INSERT)
+      verify (comparable, hash);
+#endif
 
+  m_searches++;
   value_type *first_deleted_slot = NULL;
   hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
   hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
@@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
   return &m_entries[index];
 }
 
+#if ENABLE_EXTRA_CHECKING
+
+/* Report a hash table checking error.  */
+
+ATTRIBUTE_NORETURN ATTRIBUTE_COLD
+static void
+hashtab_chk_error ()
+{
+  fprintf (stderr, "hash table checking failed: "
+	   "equal operator returns true for a pair "
+	   "of values with a different hash value");
+  gcc_unreachable ();
+}
+
+/* Verify that all existing elements in th hash table which are
+   equal to COMPARABLE have an equal HASH value provided as argument.  */
+
+template<typename Descriptor, template<typename Type> class Allocator>
+void
+hash_table<Descriptor, Allocator>
+::verify (const compare_type &comparable, hashval_t hash)
+{
+  for (size_t i = 0; i < m_size; i++)
+    {
+      value_type *entry = &m_entries[i];
+      if (!is_empty (*entry) && !is_deleted (*entry)
+	  && Descriptor::equal (*entry, comparable)
+	  && hash != Descriptor::hash (*entry))
+	hashtab_chk_error ();
+    }
+}
+#endif
+
 /* This function deletes an element with the given COMPARABLE value
    from hash table starting with the given HASH.  If there is no
    matching element in the hash table, this function does nothing. */
-- 
2.19.0

hash-table violation in cselib.c

[Martin Liška]

> 1) cselib_lookup_1:
> 
> $ cat ice.c
> a() { b(); }
> 
> $ /dev/shm/objdir/gcc/xgcc -B/dev/shm/objdir/gcc/ ice.c -g -c -fchecking=3 -O
> hash table checking failed: equal operator returns true for a pair of values with a different hash valueduring RTL pass: vartrack
> ice.c:1:1: internal compiler error: in find_slot_with_hash, at hash-table.h:905
>     1 | a() { b(); }
>       | ^
> 0x9680b5 hash_table<cselib_hasher, xcallocator>::find_slot_with_hash(cselib_hasher::key* const&, unsigned int, insert_option)
> 	/home/marxin/Programming/gcc/gcc/hash-table.h:905
> 0x962518 cselib_find_slot
> 	/home/marxin/Programming/gcc/gcc/cselib.c:584
> 0x9625d4 cselib_lookup_1
> 	/home/marxin/Programming/gcc/gcc/cselib.c:2097
> 0x9625d4 cselib_lookup(rtx_def*, machine_mode, int, machine_mode)
> 	/home/marxin/Programming/gcc/gcc/cselib.c:2141
> 0x965ee7 cselib_record_sets
> 	/home/marxin/Programming/gcc/gcc/cselib.c:2593
> 0x9670a9 cselib_process_insn(rtx_insn*)
> 	/home/marxin/Programming/gcc/gcc/cselib.c:2790
> 0x1036b73 vt_initialize
> 	/home/marxin/Programming/gcc/gcc/var-tracking.c:10231
> 0x103b98a variable_tracking_main_1
> 	/home/marxin/Programming/gcc/gcc/var-tracking.c:10460
> 0x103b98a variable_tracking_main()
> 	/home/marxin/Programming/gcc/gcc/var-tracking.c:10513
> 

I did a small analysis and problematic back-trace is:

$ (gdb) bt
#0  new_cselib_val (hash=9, mode=E_SImode, x=0x7ffff6ad7be8) at /home/marxin/Programming/gcc/gcc/cselib.c:1323
#1  0x0000000000af86f0 in cselib_lookup_1 (x=0x7ffff6ad7be8, mode=E_SImode, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:2046
#2  0x0000000000af8c5e in cselib_lookup (x=0x7ffff6ad7be8, mode=E_SImode, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:2153
#3  0x0000000000af6119 in cselib_hash_rtx2 (x=0x7ffff6ad7be8, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:1089
#4  0x0000000000af693d in cselib_hash_rtx2 (x=0x7ffff6ad7cc0, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:1254
#5  0x0000000000af6ab4 in cselib_hash_rtx (x=0x7ffff6ad7cc0, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:1309
#6  0x0000000000af8aef in cselib_lookup_1 (x=0x7ffff6ad7cc0, mode=E_CCZmode, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:2104
#7  0x0000000000af8c5e in cselib_lookup (x=0x7ffff6ad7cc0, mode=E_CCZmode, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:2153
#8  0x0000000000af9df4 in cselib_record_sets (insn=0x7ffff69a1480) at /home/marxin/Programming/gcc/gcc/cselib.c:2605
#9  0x0000000000afa946 in cselib_process_insn (insn=0x7ffff69a1480) at /home/marxin/Programming/gcc/gcc/cselib.c:2802

as seen hash ==9, which is:
  >│2046          e = new_cselib_val (next_uid, GET_MODE (x), x);

it seems to me that there are combined real hash values (based on cselib_hash_rtx) and next_uid (which is incremented).
Anyway, somebody familiar with the code needs to analyze that.

Martin

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jakub Jelinek]

On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> +hashtab_chk_error ()
> +{
> +  fprintf (stderr, "hash table checking failed: "
> +	   "equal operator returns true for a pair "
> +	   "of values with a different hash value");

BTW, either use internal_error here, or at least if using fprintf
terminate with \n, in your recent mail I saw:
...different hash valueduring RTL pass: vartrack
                    ^^^^^^

> +  gcc_unreachable ();
> +}

	Jakub

hash-table violation in gcc/fortran/trans-decl.c

[Martin Liška]

On 10/29/18 12:04 PM, Martin Liška wrote:
> 2) gfc_find_module
> 
> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gfortran.dg/coarray/alloc_comp_2.f90 -fcoarray=single -fchecking=3
> hash table checking failed: equal operator returns true for a pair of values with a different hash valuef951: internal compiler error: in find_slot_with_hash, at hash-table.h:905
> 0x8e5e86 hash_table<module_hasher, xcallocator>::find_slot_with_hash(char const* const&, unsigned int, insert_option)
> 	/home/marxin/Programming/gcc/gcc/hash-table.h:905
> 0x8e2c2c gfc_find_module(char const*)
> 	/home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:4865
> 0x8e4f42 gfc_generate_module_vars(gfc_namespace*)
> 	/home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:5475
> 0x8b8d7e gfc_generate_module_code(gfc_namespace*)
> 	/home/marxin/Programming/gcc/gcc/fortran/trans.c:2190
> 0x868427 translate_all_program_units
> 	/home/marxin/Programming/gcc/gcc/fortran/parse.c:6112
> 0x868427 gfc_parse_file()
> 	/home/marxin/Programming/gcc/gcc/fortran/parse.c:6328
> 0x8b19cb gfc_be_parse_file
> 	/home/marxin/Programming/gcc/gcc/fortran/f95-lang.c:204

This one is real issue introduced in r216127, I'll post it to corresponding mailing list
after testing:

diff --git a/gcc/fortran/trans-decl.c b/gcc/fortran/trans-decl.c
index 88f9f570725..e8da9e11d22 100644
--- a/gcc/fortran/trans-decl.c
+++ b/gcc/fortran/trans-decl.c
@@ -4825,7 +4825,7 @@ struct module_hasher : ggc_ptr_hash<module_htab_entry>
 {
   typedef const char *compare_type;
 
-  static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s); }
+  static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s->name); }
   static bool
   equal (module_htab_entry *a, const char *b)
   {

Martin

hash-table violation in gcc/cp/pt.c

[Martin Liška]

On 10/29/18 12:04 PM, Martin Liška wrote:
> 3) lookup_template_class_1
> 
> $ ./xg++ -B. /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C -c -fchecking=3
> hash table checking failed: equal operator returns true for a pair of values with a different hash value/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C: In instantiation of ‘struct B<A>’:
> /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:15:8:   required from here
> /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:8:17: internal compiler error: in find_slot_with_hash, at hash-table.h:905
>     8 |     friend bool foo (const B<Q>& a);
>       |                 ^~~
> 0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option)
> 	/home/marxin/Programming/gcc/gcc/hash-table.h:905
> 0xa042ce lookup_template_class_1
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:9629
> 0xa042ce lookup_template_class(tree_node*, tree_node*, tree_node*, tree_node*, int, int)
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:9674
> 0xa03670 tsubst_aggr_type
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:12679
> 0x9fefcd tsubst(tree_node*, tree_node*, int, tree_node*)
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14294
> 0x9fe1a9 tsubst(tree_node*, tree_node*, int, tree_node*)
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14285
> 0xa0d8bd tsubst_arg_types
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:13891
> 0xa0dc24 tsubst_function_type
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14032
> 0x9fe790 tsubst(tree_node*, tree_node*, int, tree_node*)
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14769
> 0x9f2c7c tsubst_function_decl
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:12921
> 0xa02d27 tsubst_template_decl
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:13214
> 0x9f4416 tsubst_decl
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:13316
> 0x9ff0ca tsubst(tree_node*, tree_node*, int, tree_node*)
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14212
> 0xa1dfd0 tsubst_friend_function
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:10310
> 0xa1dfd0 instantiate_class_template_1
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:11359
> 0xa1dfd0 instantiate_class_template(tree_node*)
> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:11424
> 0xa66b22 complete_type(tree_node*)
> 	/home/marxin/Programming/gcc/gcc/cp/typeck.c:138
> 0x9023c7 start_decl_1(tree_node*, bool)
> 	/home/marxin/Programming/gcc/gcc/cp/decl.c:5278
> 0x92a15f start_decl(cp_declarator const*, cp_decl_specifier_seq*, int, tree_node*, tree_node*, tree_node**)
> 	/home/marxin/Programming/gcc/gcc/cp/decl.c:5241
> 0x9c1944 cp_parser_init_declarator
> 	/home/marxin/Programming/gcc/gcc/cp/parser.c:19750

This one is about inconsistency between:

/* Returns a hash for a template TMPL and template arguments ARGS.  */

static hashval_t
hash_tmpl_and_args (tree tmpl, tree args)
{
  hashval_t val = iterative_hash_object (DECL_UID (tmpl), 0);
  return iterative_hash_template_arg (args, val);
}

iterative_hash_template_arg is problematic, differs from:
bool
spec_hasher::equal (spec_entry *e1, spec_entry *e2)
{
  int equal;

  ++comparing_specializations;
  equal = (e1->tmpl == e2->tmpl
	   && comp_template_args (e1->args, e2->args));
...

where comp_template_args (e1->args, e2->args) returns true, but
iterative_hash_template_arg values are different.

Can please some C++ maintainer take a look?

Thanks,
Martin

Re: hash-table violation in gcc/cp/pt.c

[Martin Liška]

On 10/30/18 11:25 AM, Martin Liška wrote:
> On 10/29/18 12:04 PM, Martin Liška wrote:
>> 3) lookup_template_class_1
>>
>> $ ./xg++ -B. /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C -c -fchecking=3
>> hash table checking failed: equal operator returns true for a pair of values with a different hash value/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C: In instantiation of ‘struct B<A>’:
>> /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:15:8:   required from here
>> /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:8:17: internal compiler error: in find_slot_with_hash, at hash-table.h:905
>>     8 |     friend bool foo (const B<Q>& a);
>>       |                 ^~~
>> 0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option)
>> 	/home/marxin/Programming/gcc/gcc/hash-table.h:905
>> 0xa042ce lookup_template_class_1
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:9629
>> 0xa042ce lookup_template_class(tree_node*, tree_node*, tree_node*, tree_node*, int, int)
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:9674
>> 0xa03670 tsubst_aggr_type
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:12679
>> 0x9fefcd tsubst(tree_node*, tree_node*, int, tree_node*)
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14294
>> 0x9fe1a9 tsubst(tree_node*, tree_node*, int, tree_node*)
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14285
>> 0xa0d8bd tsubst_arg_types
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:13891
>> 0xa0dc24 tsubst_function_type
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14032
>> 0x9fe790 tsubst(tree_node*, tree_node*, int, tree_node*)
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14769
>> 0x9f2c7c tsubst_function_decl
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:12921
>> 0xa02d27 tsubst_template_decl
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:13214
>> 0x9f4416 tsubst_decl
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:13316
>> 0x9ff0ca tsubst(tree_node*, tree_node*, int, tree_node*)
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:14212
>> 0xa1dfd0 tsubst_friend_function
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:10310
>> 0xa1dfd0 instantiate_class_template_1
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:11359
>> 0xa1dfd0 instantiate_class_template(tree_node*)
>> 	/home/marxin/Programming/gcc/gcc/cp/pt.c:11424
>> 0xa66b22 complete_type(tree_node*)
>> 	/home/marxin/Programming/gcc/gcc/cp/typeck.c:138
>> 0x9023c7 start_decl_1(tree_node*, bool)
>> 	/home/marxin/Programming/gcc/gcc/cp/decl.c:5278
>> 0x92a15f start_decl(cp_declarator const*, cp_decl_specifier_seq*, int, tree_node*, tree_node*, tree_node**)
>> 	/home/marxin/Programming/gcc/gcc/cp/decl.c:5241
>> 0x9c1944 cp_parser_init_declarator
>> 	/home/marxin/Programming/gcc/gcc/cp/parser.c:19750
> 
> This one is about inconsistency between:
> 
> /* Returns a hash for a template TMPL and template arguments ARGS.  */
> 
> static hashval_t
> hash_tmpl_and_args (tree tmpl, tree args)
> {
>   hashval_t val = iterative_hash_object (DECL_UID (tmpl), 0);
>   return iterative_hash_template_arg (args, val);
> }
> 
> iterative_hash_template_arg is problematic, differs from:
> bool
> spec_hasher::equal (spec_entry *e1, spec_entry *e2)
> {
>   int equal;
> 
>   ++comparing_specializations;
>   equal = (e1->tmpl == e2->tmpl
> 	   && comp_template_args (e1->args, e2->args));
> ...
> 
> where comp_template_args (e1->args, e2->args) returns true, but
> iterative_hash_template_arg values are different.
> 
> Can please some C++ maintainer take a look?
> 
> Thanks,
> Martin
> 

The same spec_hasher type is also involved in 4)

0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option)
	/home/marxin/Programming/gcc/gcc/hash-table.h:905
0x9e35e6 register_specialization
	/home/marxin/Programming/gcc/gcc/cp/pt.c:1534
0xa22ac3 check_explicit_specialization(tree_node*, tree_node*, int, int, tree_node*)
	/home/marxin/Programming/gcc/gcc/cp/pt.c:3243
0x91552d grokfndecl
	/home/marxin/Programming/gcc/gcc/cp/decl.c:9106
...

Martin

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>> +hashtab_chk_error ()
>> +{
>> +  fprintf (stderr, "hash table checking failed: "
>> +	   "equal operator returns true for a pair "
>> +	   "of values with a different hash value");
> 
> BTW, either use internal_error here, or at least if using fprintf
> terminate with \n, in your recent mail I saw:
> ...different hash valueduring RTL pass: vartrack
>                     ^^^^^^

Sure, fixed in attached patch.

Martin

> 
>> +  gcc_unreachable ();
>> +}
> 
> 	Jakub
> 


[-- Attachment #2: 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch --]
[-- Type: text/x-patch, Size: 2429 bytes --]

From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Mon, 29 Oct 2018 09:38:21 +0100
Subject: [PATCH] Sanitize equals and hash functions in hash-tables.

---
 gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 39 insertions(+), 1 deletion(-)

diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index bd83345c7b8..694eedfc4be 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -503,6 +503,7 @@ private:
 
   value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
   value_type *find_empty_slot_for_expand (hashval_t);
+  void verify (const compare_type &comparable, hashval_t hash);
   bool too_empty_p (unsigned int);
   void expand ();
   static bool is_deleted (value_type &v)
@@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
   if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
     expand ();
 
-  m_searches++;
+#if ENABLE_EXTRA_CHECKING
+    if (insert == INSERT)
+      verify (comparable, hash);
+#endif
 
+  m_searches++;
   value_type *first_deleted_slot = NULL;
   hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
   hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
@@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
   return &m_entries[index];
 }
 
+#if ENABLE_EXTRA_CHECKING
+
+/* Report a hash table checking error.  */
+
+ATTRIBUTE_NORETURN ATTRIBUTE_COLD
+static void
+hashtab_chk_error ()
+{
+  fprintf (stderr, "hash table checking failed: "
+	   "equal operator returns true for a pair "
+	   "of values with a different hash value\n");
+  gcc_unreachable ();
+}
+
+/* Verify that all existing elements in th hash table which are
+   equal to COMPARABLE have an equal HASH value provided as argument.  */
+
+template<typename Descriptor, template<typename Type> class Allocator>
+void
+hash_table<Descriptor, Allocator>
+::verify (const compare_type &comparable, hashval_t hash)
+{
+  for (size_t i = 0; i < m_size; i++)
+    {
+      value_type *entry = &m_entries[i];
+      if (!is_empty (*entry) && !is_deleted (*entry)
+	  && hash != Descriptor::hash (*entry)
+	  && Descriptor::equal (*entry, comparable))
+	hashtab_chk_error ();
+    }
+}
+#endif
+
 /* This function deletes an element with the given COMPARABLE value
    from hash table starting with the given HASH.  If there is no
    matching element in the hash table, this function does nothing. */
-- 
2.19.0

Re: hash-table violation in gcc/fortran/trans-decl.c

[Trevor Saunders]

On Tue, Oct 30, 2018 at 11:07:16AM +0100, Martin Li??ka wrote:
> On 10/29/18 12:04 PM, Martin Li??ka wrote:
> > 2) gfc_find_module
> > 
> > $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gfortran.dg/coarray/alloc_comp_2.f90 -fcoarray=single -fchecking=3
> > hash table checking failed: equal operator returns true for a pair of values with a different hash valuef951: internal compiler error: in find_slot_with_hash, at hash-table.h:905
> > 0x8e5e86 hash_table<module_hasher, xcallocator>::find_slot_with_hash(char const* const&, unsigned int, insert_option)
> > 	/home/marxin/Programming/gcc/gcc/hash-table.h:905
> > 0x8e2c2c gfc_find_module(char const*)
> > 	/home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:4865
> > 0x8e4f42 gfc_generate_module_vars(gfc_namespace*)
> > 	/home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:5475
> > 0x8b8d7e gfc_generate_module_code(gfc_namespace*)
> > 	/home/marxin/Programming/gcc/gcc/fortran/trans.c:2190
> > 0x868427 translate_all_program_units
> > 	/home/marxin/Programming/gcc/gcc/fortran/parse.c:6112
> > 0x868427 gfc_parse_file()
> > 	/home/marxin/Programming/gcc/gcc/fortran/parse.c:6328
> > 0x8b19cb gfc_be_parse_file
> > 	/home/marxin/Programming/gcc/gcc/fortran/f95-lang.c:204
> 
> This one is real issue introduced in r216127, I'll post it to corresponding mailing list
> after testing:

Yeah, sorry and lgtm.  I wonder if we could make htab_hash_string take a
const char * to catch this sooner?

Trev

> 
> diff --git a/gcc/fortran/trans-decl.c b/gcc/fortran/trans-decl.c
> index 88f9f570725..e8da9e11d22 100644
> --- a/gcc/fortran/trans-decl.c
> +++ b/gcc/fortran/trans-decl.c
> @@ -4825,7 +4825,7 @@ struct module_hasher : ggc_ptr_hash<module_htab_entry>
>  {
>    typedef const char *compare_type;
>  
> -  static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s); }
> +  static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s->name); }
>    static bool
>    equal (module_htab_entry *a, const char *b)
>    {
> 
> Martin

Re: hash-table violation in gcc/fortran/trans-decl.c

[Martin Liška]

On 10/31/18 10:40 AM, Trevor Saunders wrote:
> On Tue, Oct 30, 2018 at 11:07:16AM +0100, Martin Li??ka wrote:
>> On 10/29/18 12:04 PM, Martin Li??ka wrote:
>>> 2) gfc_find_module
>>>
>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gfortran.dg/coarray/alloc_comp_2.f90 -fcoarray=single -fchecking=3
>>> hash table checking failed: equal operator returns true for a pair of values with a different hash valuef951: internal compiler error: in find_slot_with_hash, at hash-table.h:905
>>> 0x8e5e86 hash_table<module_hasher, xcallocator>::find_slot_with_hash(char const* const&, unsigned int, insert_option)
>>> 	/home/marxin/Programming/gcc/gcc/hash-table.h:905
>>> 0x8e2c2c gfc_find_module(char const*)
>>> 	/home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:4865
>>> 0x8e4f42 gfc_generate_module_vars(gfc_namespace*)
>>> 	/home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:5475
>>> 0x8b8d7e gfc_generate_module_code(gfc_namespace*)
>>> 	/home/marxin/Programming/gcc/gcc/fortran/trans.c:2190
>>> 0x868427 translate_all_program_units
>>> 	/home/marxin/Programming/gcc/gcc/fortran/parse.c:6112
>>> 0x868427 gfc_parse_file()
>>> 	/home/marxin/Programming/gcc/gcc/fortran/parse.c:6328
>>> 0x8b19cb gfc_be_parse_file
>>> 	/home/marxin/Programming/gcc/gcc/fortran/f95-lang.c:204
>>
>> This one is real issue introduced in r216127, I'll post it to corresponding mailing list
>> after testing:
> 
> Yeah, sorry and lgtm.  I wonder if we could make htab_hash_string take a
> const char * to catch this sooner?

That happens.

Exactly my thoughts! The issue is that old C-style htab in libiberty expects
void *, so I can come up with a specific const char * specialization that
we can leverage in gcc internally (read with C++).

Martin

> 
> Trev
> 
>>
>> diff --git a/gcc/fortran/trans-decl.c b/gcc/fortran/trans-decl.c
>> index 88f9f570725..e8da9e11d22 100644
>> --- a/gcc/fortran/trans-decl.c
>> +++ b/gcc/fortran/trans-decl.c
>> @@ -4825,7 +4825,7 @@ struct module_hasher : ggc_ptr_hash<module_htab_entry>
>>  {
>>    typedef const char *compare_type;
>>  
>> -  static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s); }
>> +  static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s->name); }
>>    static bool
>>    equal (module_htab_entry *a, const char *b)
>>    {
>>
>> Martin

Re: hash-table violation in cselib.c

[Martin Liška]

I created PR87845 for that.

Martin

Re: hash-table violation in gcc/cp/pt.c

[Martin Liška]

I created PR87847 for that.

Martin

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jeff Law]

On 10/30/18 6:28 AM, Martin Liška wrote:
> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>> +hashtab_chk_error ()
>>> +{
>>> +  fprintf (stderr, "hash table checking failed: "
>>> +	   "equal operator returns true for a pair "
>>> +	   "of values with a different hash value");
>> BTW, either use internal_error here, or at least if using fprintf
>> terminate with \n, in your recent mail I saw:
>> ...different hash valueduring RTL pass: vartrack
>>                     ^^^^^^
> Sure, fixed in attached patch.
> 
> Martin
> 
>>> +  gcc_unreachable ();
>>> +}
>> 	Jakub
>>
> 
> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> 
> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> From: marxin <mliska@suse.cz>
> Date: Mon, 29 Oct 2018 09:38:21 +0100
> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> 
> ---
>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 39 insertions(+), 1 deletion(-)
> 
> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> index bd83345c7b8..694eedfc4be 100644
> --- a/gcc/hash-table.h
> +++ b/gcc/hash-table.h
> @@ -503,6 +503,7 @@ private:
>  
>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>    value_type *find_empty_slot_for_expand (hashval_t);
> +  void verify (const compare_type &comparable, hashval_t hash);
>    bool too_empty_p (unsigned int);
>    void expand ();
>    static bool is_deleted (value_type &v)
> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>      expand ();
>  
> -  m_searches++;
> +#if ENABLE_EXTRA_CHECKING
> +    if (insert == INSERT)
> +      verify (comparable, hash);
> +#endif
>  
> +  m_searches++;
>    value_type *first_deleted_slot = NULL;
>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>    return &m_entries[index];
>  }
>  
> +#if ENABLE_EXTRA_CHECKING
> +
> +/* Report a hash table checking error.  */
> +
> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> +static void
> +hashtab_chk_error ()
> +{
> +  fprintf (stderr, "hash table checking failed: "
> +	   "equal operator returns true for a pair "
> +	   "of values with a different hash value\n");
> +  gcc_unreachable ();
> +}
I think an internal_error here is probably still better than a simple
fprintf, even if the fprintf is terminated with a \n :-)

The question then becomes can we bootstrap with this stuff enabled and
if not, are we likely to soon?  It'd be a shame to put it into
EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
because we've got too many bugs to fix.

> +
> +/* Verify that all existing elements in th hash table which are
s/th/the/


Jeff

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jakub Jelinek]

On Wed, Nov 07, 2018 at 03:23:55PM -0700, Jeff Law wrote:
> > @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >      expand ();
> >  
> > -  m_searches++;
> > +#if ENABLE_EXTRA_CHECKING
> > +    if (insert == INSERT)
> > +      verify (comparable, hash);
> > +#endif

Plus formatting, the above is indented too much.

	Jakub

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 11/7/18 11:23 PM, Jeff Law wrote:
> On 10/30/18 6:28 AM, Martin Liška wrote:
>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>> +hashtab_chk_error ()
>>>> +{
>>>> +  fprintf (stderr, "hash table checking failed: "
>>>> +	   "equal operator returns true for a pair "
>>>> +	   "of values with a different hash value");
>>> BTW, either use internal_error here, or at least if using fprintf
>>> terminate with \n, in your recent mail I saw:
>>> ...different hash valueduring RTL pass: vartrack
>>>                     ^^^^^^
>> Sure, fixed in attached patch.
>>
>> Martin
>>
>>>> +  gcc_unreachable ();
>>>> +}
>>> 	Jakub
>>>
>>
>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>
>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>> From: marxin <mliska@suse.cz>
>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>
>> ---
>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>
>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>> index bd83345c7b8..694eedfc4be 100644
>> --- a/gcc/hash-table.h
>> +++ b/gcc/hash-table.h
>> @@ -503,6 +503,7 @@ private:
>>  
>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>    value_type *find_empty_slot_for_expand (hashval_t);
>> +  void verify (const compare_type &comparable, hashval_t hash);
>>    bool too_empty_p (unsigned int);
>>    void expand ();
>>    static bool is_deleted (value_type &v)
>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>      expand ();
>>  
>> -  m_searches++;
>> +#if ENABLE_EXTRA_CHECKING
>> +    if (insert == INSERT)
>> +      verify (comparable, hash);
>> +#endif
>>  
>> +  m_searches++;
>>    value_type *first_deleted_slot = NULL;
>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>    return &m_entries[index];
>>  }
>>  
>> +#if ENABLE_EXTRA_CHECKING
>> +
>> +/* Report a hash table checking error.  */
>> +
>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>> +static void
>> +hashtab_chk_error ()
>> +{
>> +  fprintf (stderr, "hash table checking failed: "
>> +	   "equal operator returns true for a pair "
>> +	   "of values with a different hash value\n");
>> +  gcc_unreachable ();
>> +}
> I think an internal_error here is probably still better than a simple
> fprintf, even if the fprintf is terminated with a \n :-)

Fully agree with that, but I see a lot of build errors when using internal_error.

> 
> The question then becomes can we bootstrap with this stuff enabled and
> if not, are we likely to soon?  It'd be a shame to put it into
> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> because we've got too many bugs to fix.

Unfortunately it's blocked with these 2 PRs:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847

I'm fine with having the patch in in next stage1 after the problems will
be fixed.

Martin

> 
>> +
>> +/* Verify that all existing elements in th hash table which are
> s/th/the/
> 
> 
> Jeff
> 

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 4585 bytes --]

On 11/8/18 9:56 AM, Martin Liška wrote:
> On 11/7/18 11:23 PM, Jeff Law wrote:
>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>> +hashtab_chk_error ()
>>>>> +{
>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>> +	   "equal operator returns true for a pair "
>>>>> +	   "of values with a different hash value");
>>>> BTW, either use internal_error here, or at least if using fprintf
>>>> terminate with \n, in your recent mail I saw:
>>>> ...different hash valueduring RTL pass: vartrack
>>>>                     ^^^^^^
>>> Sure, fixed in attached patch.
>>>
>>> Martin
>>>
>>>>> +  gcc_unreachable ();
>>>>> +}
>>>> 	Jakub
>>>>
>>>
>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>
>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>> From: marxin <mliska@suse.cz>
>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>
>>> ---
>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>> index bd83345c7b8..694eedfc4be 100644
>>> --- a/gcc/hash-table.h
>>> +++ b/gcc/hash-table.h
>>> @@ -503,6 +503,7 @@ private:
>>>  
>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>    bool too_empty_p (unsigned int);
>>>    void expand ();
>>>    static bool is_deleted (value_type &v)
>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>      expand ();
>>>  
>>> -  m_searches++;
>>> +#if ENABLE_EXTRA_CHECKING
>>> +    if (insert == INSERT)
>>> +      verify (comparable, hash);
>>> +#endif
>>>  
>>> +  m_searches++;
>>>    value_type *first_deleted_slot = NULL;
>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>    return &m_entries[index];
>>>  }
>>>  
>>> +#if ENABLE_EXTRA_CHECKING
>>> +
>>> +/* Report a hash table checking error.  */
>>> +
>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>> +static void
>>> +hashtab_chk_error ()
>>> +{
>>> +  fprintf (stderr, "hash table checking failed: "
>>> +	   "equal operator returns true for a pair "
>>> +	   "of values with a different hash value\n");
>>> +  gcc_unreachable ();
>>> +}
>> I think an internal_error here is probably still better than a simple
>> fprintf, even if the fprintf is terminated with a \n :-)
> 
> Fully agree with that, but I see a lot of build errors when using internal_error.
> 
>>
>> The question then becomes can we bootstrap with this stuff enabled and
>> if not, are we likely to soon?  It'd be a shame to put it into
>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>> because we've got too many bugs to fix.
> 
> Unfortunately it's blocked with these 2 PRs:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847

Hi.

I've just added one more PR:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450

I'm sending updated version of the patch that provides a disablement for the 3 PRs
with a new function disable_sanitize_eq_and_hash.

With that I can bootstrap and finish tests. However, I've done that with a patch
limits maximal number of checks:

diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index dc24fea6405..57564914e31 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -1027,7 +1027,7 @@ void
 hash_table<Descriptor, Lazy, Allocator>
 ::verify (const compare_type &comparable, hashval_t hash)
 {
-  for (size_t i = 0; i < m_size; i++)
+  for (size_t i = 0; i < MIN (m_size, 1000); i++)
     {
       value_type *entry = &m_entries[i];
       if (!is_empty (*entry) && !is_deleted (*entry)

Without that it would be probably terribly slow. Moreover, one probably does not want
that with an extra checking, but with an extra-extra checking. Ideas about where to enable
it?

Would it be possible to add the sanitization with the aforementioned disablement?

Thanks,
Martin

> 
> I'm fine with having the patch in in next stage1 after the problems will
> be fixed.
> 
> Martin
> 
>>
>>> +
>>> +/* Verify that all existing elements in th hash table which are
>> s/th/the/
>>
>>
>> Jeff
>>
> 


[-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --]
[-- Type: text/x-patch, Size: 5391 bytes --]

From fee775b5f4443e6eaeb4028e9a8922ea4ec8703f Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Mon, 13 May 2019 07:16:22 +0200
Subject: [PATCH] Enable sanitization for hash tables.

---
 gcc/cp/pt.c            |  4 ++++
 gcc/cselib.c           |  2 ++
 gcc/hash-table.h       | 53 ++++++++++++++++++++++++++++++++++++++++--
 gcc/tree-ssa-loop-im.c |  2 ++
 4 files changed, 59 insertions(+), 2 deletions(-)

diff --git a/gcc/cp/pt.c b/gcc/cp/pt.c
index d6976e08690..db9c953e3dd 100644
--- a/gcc/cp/pt.c
+++ b/gcc/cp/pt.c
@@ -28320,7 +28320,11 @@ void
 init_template_processing (void)
 {
   decl_specializations = hash_table<spec_hasher>::create_ggc (37);
+  /* FIXME: enable sanitization */
+  decl_specializations->disable_sanitize_eq_and_hash ();
   type_specializations = hash_table<spec_hasher>::create_ggc (37);
+  /* FIXME: enable sanitization */
+  type_specializations->disable_sanitize_eq_and_hash ();
 
   if (cxx_dialect >= cxx11)
     declare_integer_pack ();
diff --git a/gcc/cselib.c b/gcc/cselib.c
index 84c17c23f6d..a60865c810a 100644
--- a/gcc/cselib.c
+++ b/gcc/cselib.c
@@ -2859,6 +2859,8 @@ cselib_init (int record_what)
   used_regs = XNEWVEC (unsigned int, cselib_nregs);
   n_used_regs = 0;
   cselib_hash_table = new hash_table<cselib_hasher> (31);
+  /* FIXME: enable sanitization */
+  cselib_hash_table->disable_sanitize_eq_and_hash ();
   if (cselib_preserve_constants)
     cselib_preserved_hash_table = new hash_table<cselib_hasher> (31);
   next_uid = 1;
diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index 4178616478e..dc24fea6405 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -405,6 +405,9 @@ public:
   /* Return true when there are no elements in this hash table.  */
   bool is_empty () const { return elements () == 0; }
 
+  /* Disable equal and hash function sanitization.  */
+  void disable_sanitize_eq_and_hash (void) { m_sanitize_eq_and_hash = false; }
+
   /* This function clears a specified SLOT in a hash table.  It is
      useful when you've already done the lookup and don't want to do it
      again. */
@@ -516,6 +519,7 @@ private:
 
   value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
   value_type *find_empty_slot_for_expand (hashval_t);
+  void verify (const compare_type &comparable, hashval_t hash);
   bool too_empty_p (unsigned int);
   void expand ();
   static bool is_deleted (value_type &v)
@@ -564,6 +568,9 @@ private:
   /* if m_entries is stored in ggc memory.  */
   bool m_ggc;
 
+  /* True if the table should be sanitized for equal and hash functions.  */
+  bool m_sanitize_eq_and_hash;
+
   /* If we should gather memory statistics for the table.  */
 #if GATHER_STATISTICS
   bool m_gather_mem_stats;
@@ -591,7 +598,7 @@ hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0),
-  m_ggc (ggc)
+  m_ggc (ggc), m_sanitize_eq_and_hash (true)
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -941,8 +948,12 @@ hash_table<Descriptor, Lazy, Allocator>
   if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
     expand ();
 
-  m_searches++;
+#if ENABLE_EXTRA_CHECKING
+  if (m_sanitize_eq_and_hash && insert == INSERT)
+    verify (comparable, hash);
+#endif
 
+  m_searches++;
   value_type *first_deleted_slot = NULL;
   hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
   hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
@@ -989,6 +1000,44 @@ hash_table<Descriptor, Lazy, Allocator>
   return &m_entries[index];
 }
 
+#if ENABLE_EXTRA_CHECKING
+
+/* Report a hash table checking error.  */
+
+ATTRIBUTE_NORETURN ATTRIBUTE_COLD
+static void
+hashtab_chk_error ()
+{
+  fprintf (stderr, "hash table checking failed: "
+	   "equal operator returns true for a pair "
+	   "of values with a different hash value\n");
+#ifndef GENERATOR_FILE
+  gcc_unreachable ();
+#else
+  exit (1);
+#endif
+}
+
+/* Verify that all existing elements in th hash table which are
+   equal to COMPARABLE have an equal HASH value provided as argument.  */
+
+template<typename Descriptor, bool Lazy,
+	 template<typename Type> class Allocator>
+void
+hash_table<Descriptor, Lazy, Allocator>
+::verify (const compare_type &comparable, hashval_t hash)
+{
+  for (size_t i = 0; i < m_size; i++)
+    {
+      value_type *entry = &m_entries[i];
+      if (!is_empty (*entry) && !is_deleted (*entry)
+	  && hash != Descriptor::hash (*entry)
+	  && Descriptor::equal (*entry, comparable))
+	hashtab_chk_error ();
+    }
+}
+#endif
+
 /* This function deletes an element with the given COMPARABLE value
    from hash table starting with the given HASH.  If there is no
    matching element in the hash table, this function does nothing. */
diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c
index 56d8e8e4330..1bef286d46b 100644
--- a/gcc/tree-ssa-loop-im.c
+++ b/gcc/tree-ssa-loop-im.c
@@ -2565,6 +2565,8 @@ tree_ssa_lim_initialize (void)
   alloc_aux_for_edges (0);
 
   memory_accesses.refs = new hash_table<mem_ref_hasher> (100);
+  /* FIXME: enable sanitization */
+  memory_accesses.refs->disable_sanitize_eq_and_hash ();
   memory_accesses.refs_list.create (100);
   /* Allocate a special, unanalyzable mem-ref with ID zero.  */
   memory_accesses.refs_list.quick_push
-- 
2.21.0

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jason Merrill]

On 5/13/19 3:41 AM, Martin Liška wrote:
> On 11/8/18 9:56 AM, Martin Liška wrote:
>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>> +hashtab_chk_error ()
>>>>>> +{
>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>> +	   "equal operator returns true for a pair "
>>>>>> +	   "of values with a different hash value");
>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>> terminate with \n, in your recent mail I saw:
>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>                      ^^^^^^
>>>> Sure, fixed in attached patch.
>>>>
>>>> Martin
>>>>
>>>>>> +  gcc_unreachable ();
>>>>>> +}
>>>>> 	Jakub
>>>>>
>>>>
>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>
>>>>  From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>> From: marxin <mliska@suse.cz>
>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>
>>>> ---
>>>>   gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>   1 file changed, 39 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>> index bd83345c7b8..694eedfc4be 100644
>>>> --- a/gcc/hash-table.h
>>>> +++ b/gcc/hash-table.h
>>>> @@ -503,6 +503,7 @@ private:
>>>>   
>>>>     value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>     value_type *find_empty_slot_for_expand (hashval_t);
>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>     bool too_empty_p (unsigned int);
>>>>     void expand ();
>>>>     static bool is_deleted (value_type &v)
>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>     if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>       expand ();
>>>>   
>>>> -  m_searches++;
>>>> +#if ENABLE_EXTRA_CHECKING
>>>> +    if (insert == INSERT)
>>>> +      verify (comparable, hash);
>>>> +#endif
>>>>   
>>>> +  m_searches++;
>>>>     value_type *first_deleted_slot = NULL;
>>>>     hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>     hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>     return &m_entries[index];
>>>>   }
>>>>   
>>>> +#if ENABLE_EXTRA_CHECKING
>>>> +
>>>> +/* Report a hash table checking error.  */
>>>> +
>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>> +static void
>>>> +hashtab_chk_error ()
>>>> +{
>>>> +  fprintf (stderr, "hash table checking failed: "
>>>> +	   "equal operator returns true for a pair "
>>>> +	   "of values with a different hash value\n");
>>>> +  gcc_unreachable ();
>>>> +}
>>> I think an internal_error here is probably still better than a simple
>>> fprintf, even if the fprintf is terminated with a \n :-)
>>
>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>
>>>
>>> The question then becomes can we bootstrap with this stuff enabled and
>>> if not, are we likely to soon?  It'd be a shame to put it into
>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>> because we've got too many bugs to fix.
>>
>> Unfortunately it's blocked with these 2 PRs:
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> 
> Hi.
> 
> I've just added one more PR:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> 
> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> with a new function disable_sanitize_eq_and_hash.
> 
> With that I can bootstrap and finish tests. However, I've done that with a patch
> limits maximal number of checks:
> 
> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> index dc24fea6405..57564914e31 100644
> --- a/gcc/hash-table.h
> +++ b/gcc/hash-table.h
> @@ -1027,7 +1027,7 @@ void
>   hash_table<Descriptor, Lazy, Allocator>
>   ::verify (const compare_type &comparable, hashval_t hash)
>   {
> -  for (size_t i = 0; i < m_size; i++)
> +  for (size_t i = 0; i < MIN (m_size, 1000); i++)
>       {
>         value_type *entry = &m_entries[i];
>         if (!is_empty (*entry) && !is_deleted (*entry)
> 
> Without that it would be probably terribly slow. Moreover, one probably does not want
> that with an extra checking, but with an extra-extra checking. Ideas about where to enable
> it?

As a --param?  I use --param ggc-min-heapsize=0 --param ggc-min-expand=0 
to catch GC issues, and this seems similar.

> +  /* FIXME: enable sanitization */

Please add PR numbers to these comments.

> +#if ENABLE_EXTRA_CHECKING

The documentation describes this flag as

> @samp{extra} adds for @samp{misc} checking extra checks that might affect
> code generation and should therefore not differ between stage1 and later
> stages.

which doesn't seem to apply here.

Jason

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jeff Law]

On 5/13/19 1:41 AM, Martin Liška wrote:
> On 11/8/18 9:56 AM, Martin Liška wrote:
>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>> +hashtab_chk_error ()
>>>>>> +{
>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>> +	   "equal operator returns true for a pair "
>>>>>> +	   "of values with a different hash value");
>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>> terminate with \n, in your recent mail I saw:
>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>                     ^^^^^^
>>>> Sure, fixed in attached patch.
>>>>
>>>> Martin
>>>>
>>>>>> +  gcc_unreachable ();
>>>>>> +}
>>>>> 	Jakub
>>>>>
>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>
>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>> From: marxin <mliska@suse.cz>
>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>
>>>> ---
>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>> index bd83345c7b8..694eedfc4be 100644
>>>> --- a/gcc/hash-table.h
>>>> +++ b/gcc/hash-table.h
>>>> @@ -503,6 +503,7 @@ private:
>>>>  
>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>    bool too_empty_p (unsigned int);
>>>>    void expand ();
>>>>    static bool is_deleted (value_type &v)
>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>      expand ();
>>>>  
>>>> -  m_searches++;
>>>> +#if ENABLE_EXTRA_CHECKING
>>>> +    if (insert == INSERT)
>>>> +      verify (comparable, hash);
>>>> +#endif
>>>>  
>>>> +  m_searches++;
>>>>    value_type *first_deleted_slot = NULL;
>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>    return &m_entries[index];
>>>>  }
>>>>  
>>>> +#if ENABLE_EXTRA_CHECKING
>>>> +
>>>> +/* Report a hash table checking error.  */
>>>> +
>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>> +static void
>>>> +hashtab_chk_error ()
>>>> +{
>>>> +  fprintf (stderr, "hash table checking failed: "
>>>> +	   "equal operator returns true for a pair "
>>>> +	   "of values with a different hash value\n");
>>>> +  gcc_unreachable ();
>>>> +}
>>> I think an internal_error here is probably still better than a simple
>>> fprintf, even if the fprintf is terminated with a \n :-)
>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>
>>> The question then becomes can we bootstrap with this stuff enabled and
>>> if not, are we likely to soon?  It'd be a shame to put it into
>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>> because we've got too many bugs to fix.
>> Unfortunately it's blocked with these 2 PRs:
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> Hi.
> 
> I've just added one more PR:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> 
> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> with a new function disable_sanitize_eq_and_hash.
> 
> With that I can bootstrap and finish tests. However, I've done that with a patch
> limits maximal number of checks:
So rather than call the disable_sanitize_eq_and_hash, can you have its
state set up when you instantiate the object?  It's not a huge deal,
just thinking about loud.



So how do we want to go forward, particularly the EXTRA_EXTRA checking
issue :-)


Jeff

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>
> On 5/13/19 1:41 AM, Martin Liška wrote:
> > On 11/8/18 9:56 AM, Martin Liška wrote:
> >> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>> +hashtab_chk_error ()
> >>>>>> +{
> >>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>> +           "equal operator returns true for a pair "
> >>>>>> +           "of values with a different hash value");
> >>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>> terminate with \n, in your recent mail I saw:
> >>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>                     ^^^^^^
> >>>> Sure, fixed in attached patch.
> >>>>
> >>>> Martin
> >>>>
> >>>>>> +  gcc_unreachable ();
> >>>>>> +}
> >>>>>   Jakub
> >>>>>
> >>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>
> >>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>> From: marxin <mliska@suse.cz>
> >>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>
> >>>> ---
> >>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>
> >>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>> index bd83345c7b8..694eedfc4be 100644
> >>>> --- a/gcc/hash-table.h
> >>>> +++ b/gcc/hash-table.h
> >>>> @@ -503,6 +503,7 @@ private:
> >>>>
> >>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>    value_type *find_empty_slot_for_expand (hashval_t);
> >>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>    bool too_empty_p (unsigned int);
> >>>>    void expand ();
> >>>>    static bool is_deleted (value_type &v)
> >>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>      expand ();
> >>>>
> >>>> -  m_searches++;
> >>>> +#if ENABLE_EXTRA_CHECKING
> >>>> +    if (insert == INSERT)
> >>>> +      verify (comparable, hash);
> >>>> +#endif
> >>>>
> >>>> +  m_searches++;
> >>>>    value_type *first_deleted_slot = NULL;
> >>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>    return &m_entries[index];
> >>>>  }
> >>>>
> >>>> +#if ENABLE_EXTRA_CHECKING
> >>>> +
> >>>> +/* Report a hash table checking error.  */
> >>>> +
> >>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>> +static void
> >>>> +hashtab_chk_error ()
> >>>> +{
> >>>> +  fprintf (stderr, "hash table checking failed: "
> >>>> +     "equal operator returns true for a pair "
> >>>> +     "of values with a different hash value\n");
> >>>> +  gcc_unreachable ();
> >>>> +}
> >>> I think an internal_error here is probably still better than a simple
> >>> fprintf, even if the fprintf is terminated with a \n :-)
> >> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>
> >>> The question then becomes can we bootstrap with this stuff enabled and
> >>> if not, are we likely to soon?  It'd be a shame to put it into
> >>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>> because we've got too many bugs to fix.
> >> Unfortunately it's blocked with these 2 PRs:
> >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> > Hi.
> >
> > I've just added one more PR:
> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >
> > I'm sending updated version of the patch that provides a disablement for the 3 PRs
> > with a new function disable_sanitize_eq_and_hash.
> >
> > With that I can bootstrap and finish tests. However, I've done that with a patch
> > limits maximal number of checks:
> So rather than call the disable_sanitize_eq_and_hash, can you have its
> state set up when you instantiate the object?  It's not a huge deal,
> just thinking about loud.
>
>
>
> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> issue :-)

There is at least one PR where we have a table where elements _in_ the
table are never compared against each other but always against another
object (I guess that's usual even), but the setup is in a way that the
comparison function only works with those.  With the patch we verify
hashing/comparison for something that is never used.

So - wouldn't it be more "correct" to only verify comparison/hashing
at lookup time, using the object from the lookup and verify that against
all other elements?

Richard.

>
> Jeff

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 5414 bytes --]

On 5/21/19 11:38 AM, Richard Biener wrote:
> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>
>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>> +hashtab_chk_error ()
>>>>>>>> +{
>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>> +           "of values with a different hash value");
>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>                     ^^^^^^
>>>>>> Sure, fixed in attached patch.
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>>>> +  gcc_unreachable ();
>>>>>>>> +}
>>>>>>>   Jakub
>>>>>>>
>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>
>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>> From: marxin <mliska@suse.cz>
>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>
>>>>>> ---
>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>> --- a/gcc/hash-table.h
>>>>>> +++ b/gcc/hash-table.h
>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>
>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>    bool too_empty_p (unsigned int);
>>>>>>    void expand ();
>>>>>>    static bool is_deleted (value_type &v)
>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>      expand ();
>>>>>>
>>>>>> -  m_searches++;
>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>> +    if (insert == INSERT)
>>>>>> +      verify (comparable, hash);
>>>>>> +#endif
>>>>>>
>>>>>> +  m_searches++;
>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>    return &m_entries[index];
>>>>>>  }
>>>>>>
>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>> +
>>>>>> +/* Report a hash table checking error.  */
>>>>>> +
>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>> +static void
>>>>>> +hashtab_chk_error ()
>>>>>> +{
>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>> +     "equal operator returns true for a pair "
>>>>>> +     "of values with a different hash value\n");
>>>>>> +  gcc_unreachable ();
>>>>>> +}
>>>>> I think an internal_error here is probably still better than a simple
>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>
>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>> because we've got too many bugs to fix.
>>>> Unfortunately it's blocked with these 2 PRs:
>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>> Hi.
>>>
>>> I've just added one more PR:
>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>
>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>> with a new function disable_sanitize_eq_and_hash.
>>>
>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>> limits maximal number of checks:
>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>> state set up when you instantiate the object?  It's not a huge deal,
>> just thinking about loud.
>>
>>
>>
>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>> issue :-)
> 
> There is at least one PR where we have a table where elements _in_ the
> table are never compared against each other but always against another
> object (I guess that's usual even), but the setup is in a way that the
> comparison function only works with those.  With the patch we verify
> hashing/comparison for something that is never used.
> 
> So - wouldn't it be more "correct" to only verify comparison/hashing
> at lookup time, using the object from the lookup and verify that against
> all other elements?

I don't a have problem with that. Apparently this changes fixes
PR90450 and PR87847.

Changes from previous version:
- verification happens only when an element is searched (not inserted)
- new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
- new param has been introduced hash-table-verification-limit in order
  to limit number of elements that are compared within a table
- verification happens only with flag_checking >= 2

I've been bootstrapping and testing the patch right now.

Martin

> 
> Richard.
> 
>>
>> Jeff


[-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --]
[-- Type: text/x-patch, Size: 8202 bytes --]

From 47df69feb035672056d2ad86c53eda9c286ed92a Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Mon, 13 May 2019 07:16:22 +0200
Subject: [PATCH] Enable sanitization for hash tables.

---
 gcc/cselib.c     |  9 ++++++--
 gcc/hash-set.h   |  2 +-
 gcc/hash-table.c |  3 +++
 gcc/hash-table.h | 60 ++++++++++++++++++++++++++++++++++++++++++++----
 gcc/params.def   |  6 +++++
 gcc/toplev.c     |  4 ++++
 6 files changed, 77 insertions(+), 7 deletions(-)

diff --git a/gcc/cselib.c b/gcc/cselib.c
index 84c17c23f6d..a1cbdec9718 100644
--- a/gcc/cselib.c
+++ b/gcc/cselib.c
@@ -2858,9 +2858,14 @@ cselib_init (int record_what)
     }
   used_regs = XNEWVEC (unsigned int, cselib_nregs);
   n_used_regs = 0;
-  cselib_hash_table = new hash_table<cselib_hasher> (31);
+  /* FIXME: enable sanitization (PR87845) */
+  cselib_hash_table
+    = new hash_table<cselib_hasher> (31, /* ggc */ false,
+				     /* sanitize_eq_and_hash */ false);
   if (cselib_preserve_constants)
-    cselib_preserved_hash_table = new hash_table<cselib_hasher> (31);
+    cselib_preserved_hash_table
+      = new hash_table<cselib_hasher> (31, /* ggc */ false,
+				       /* sanitize_eq_and_hash */ false);
   next_uid = 1;
 }
 
diff --git a/gcc/hash-set.h b/gcc/hash-set.h
index de3532f5f68..d891ed78297 100644
--- a/gcc/hash-set.h
+++ b/gcc/hash-set.h
@@ -28,7 +28,7 @@ class hash_set
 public:
   typedef typename Traits::value_type Key;
   explicit hash_set (size_t n = 13, bool ggc = false CXX_MEM_STAT_INFO)
-    : m_table (n, ggc, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {}
+    : m_table (n, ggc, true, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {}
 
   /* Create a hash_set in gc memory with space for at least n elements.  */
 
diff --git a/gcc/hash-table.c b/gcc/hash-table.c
index 646a7a1c497..8e86fffa36f 100644
--- a/gcc/hash-table.c
+++ b/gcc/hash-table.c
@@ -74,6 +74,9 @@ struct prime_ent const prime_tab[] = {
   { 0xfffffffb, 0x00000006, 0x00000008, 31 }
 };
 
+/* Limit number of comparisons when calling hash_table<>::verify.  */
+unsigned int hash_table_sanitize_eq_limit;
+
 /* The following function returns an index into the above table of the
    nearest prime number which is greater than N, and near a power of two. */
 
diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index 4178616478e..b3e579b8822 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -295,6 +295,8 @@ struct prime_ent
 
 extern struct prime_ent const prime_tab[];
 
+/* Limit number of comparisons when calling hash_table<>::verify.  */
+extern unsigned int hash_table_sanitize_eq_limit;
 
 /* Functions for computing hash table indexes.  */
 
@@ -371,10 +373,12 @@ class hash_table
 
 public:
   explicit hash_table (size_t, bool ggc = false,
+		       bool sanitize_eq_and_hash = true,
 		       bool gather_mem_stats = GATHER_STATISTICS,
 		       mem_alloc_origin origin = HASH_TABLE_ORIGIN
 		       CXX_MEM_STAT_INFO);
   explicit hash_table (const hash_table &, bool ggc = false,
+		       bool sanitize_eq_and_hash = true,
 		       bool gather_mem_stats = GATHER_STATISTICS,
 		       mem_alloc_origin origin = HASH_TABLE_ORIGIN
 		       CXX_MEM_STAT_INFO);
@@ -516,6 +520,7 @@ private:
 
   value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
   value_type *find_empty_slot_for_expand (hashval_t);
+  void verify (const compare_type &comparable, hashval_t hash);
   bool too_empty_p (unsigned int);
   void expand ();
   static bool is_deleted (value_type &v)
@@ -564,6 +569,9 @@ private:
   /* if m_entries is stored in ggc memory.  */
   bool m_ggc;
 
+  /* True if the table should be sanitized for equal and hash functions.  */
+  bool m_sanitize_eq_and_hash;
+
   /* If we should gather memory statistics for the table.  */
 #if GATHER_STATISTICS
   bool m_gather_mem_stats;
@@ -586,12 +594,13 @@ extern void dump_hash_table_loc_statistics (void);
 template<typename Descriptor, bool Lazy,
 	 template<typename Type> class Allocator>
 hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc,
+						     bool sanitize_eq_and_hash,
 						     bool gather_mem_stats
 						     ATTRIBUTE_UNUSED,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0),
-  m_ggc (ggc)
+  m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash)
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -617,12 +626,14 @@ template<typename Descriptor, bool Lazy,
 	 template<typename Type> class Allocator>
 hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h,
 						     bool ggc,
+						     bool sanitize_eq_and_hash,
 						     bool gather_mem_stats
 						     ATTRIBUTE_UNUSED,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted),
-  m_searches (0), m_collisions (0), m_ggc (ggc)
+  m_searches (0), m_collisions (0), m_ggc (ggc),
+  m_sanitize_eq_and_hash (sanitize_eq_and_hash)
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -912,7 +923,11 @@ hash_table<Descriptor, Lazy, Allocator>
       entry = &m_entries[index];
       if (is_empty (*entry)
           || (!is_deleted (*entry) && Descriptor::equal (*entry, comparable)))
-        return *entry;
+	{
+	  if (m_sanitize_eq_and_hash)
+	    verify (comparable, hash);
+	  return *entry;
+	}
     }
 }
 
@@ -941,8 +956,10 @@ hash_table<Descriptor, Lazy, Allocator>
   if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
     expand ();
 
-  m_searches++;
+  if (m_sanitize_eq_and_hash && insert == NO_INSERT)
+    verify (comparable, hash);
 
+  m_searches++;
   value_type *first_deleted_slot = NULL;
   hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
   hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
@@ -989,6 +1006,41 @@ hash_table<Descriptor, Lazy, Allocator>
   return &m_entries[index];
 }
 
+/* Report a hash table checking error.  */
+
+ATTRIBUTE_NORETURN ATTRIBUTE_COLD
+static void
+hashtab_chk_error ()
+{
+  fprintf (stderr, "hash table checking failed: "
+	   "equal operator returns true for a pair "
+	   "of values with a different hash value\n");
+#ifndef GENERATOR_FILE
+  gcc_unreachable ();
+#else
+  exit (1);
+#endif
+}
+
+/* Verify that all existing elements in th hash table which are
+   equal to COMPARABLE have an equal HASH value provided as argument.  */
+
+template<typename Descriptor, bool Lazy,
+	 template<typename Type> class Allocator>
+void
+hash_table<Descriptor, Lazy, Allocator>
+::verify (const compare_type &comparable, hashval_t hash)
+{
+  for (size_t i = 0; i < MIN (hash_table_sanitize_eq_limit, m_size); i++)
+    {
+      value_type *entry = &m_entries[i];
+      if (!is_empty (*entry) && !is_deleted (*entry)
+	  && hash != Descriptor::hash (*entry)
+	  && Descriptor::equal (*entry, comparable))
+	hashtab_chk_error ();
+    }
+}
+
 /* This function deletes an element with the given COMPARABLE value
    from hash table starting with the given HASH.  If there is no
    matching element in the hash table, this function does nothing. */
diff --git a/gcc/params.def b/gcc/params.def
index 6b7f7eb5bae..e5ca6ff45e4 100644
--- a/gcc/params.def
+++ b/gcc/params.def
@@ -1439,6 +1439,12 @@ DEFPARAM(PARAM_GIMPLE_FE_COMPUTED_HOT_BB_THRESHOLD,
 	 " The parameter is used only in GIMPLE FE.",
 	 0, 0, 0)
 
+DEFPARAM(PARAM_HASH_TABLE_VERIFICATION_LIMIT,
+	 "hash-table-verification-limit",
+	 "The number of elements for which hash table verification is done for "
+	 "each searched element.",
+	 100, 0, 0)
+
 /*
 
 Local variables:
diff --git a/gcc/toplev.c b/gcc/toplev.c
index d300ac2ec89..116be7be395 100644
--- a/gcc/toplev.c
+++ b/gcc/toplev.c
@@ -1799,6 +1799,10 @@ process_options (void)
   optimization_default_node = build_optimization_node (&global_options);
   optimization_current_node = optimization_default_node;
 
+  if (flag_checking >= 2)
+    hash_table_sanitize_eq_limit
+      = PARAM_VALUE (PARAM_HASH_TABLE_VERIFICATION_LIMIT);
+
   /* Please don't change global_options after this point, those changes won't
      be reflected in optimization_{default,current}_node.  */
 }
-- 
2.21.0

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 5/21/19 11:38 AM, Richard Biener wrote:
> > On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>
> >> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>> +hashtab_chk_error ()
> >>>>>>>> +{
> >>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>> +           "of values with a different hash value");
> >>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>                     ^^^^^^
> >>>>>> Sure, fixed in attached patch.
> >>>>>>
> >>>>>> Martin
> >>>>>>
> >>>>>>>> +  gcc_unreachable ();
> >>>>>>>> +}
> >>>>>>>   Jakub
> >>>>>>>
> >>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>
> >>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>> From: marxin <mliska@suse.cz>
> >>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>
> >>>>>> ---
> >>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>
> >>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>> --- a/gcc/hash-table.h
> >>>>>> +++ b/gcc/hash-table.h
> >>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>
> >>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>    bool too_empty_p (unsigned int);
> >>>>>>    void expand ();
> >>>>>>    static bool is_deleted (value_type &v)
> >>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>      expand ();
> >>>>>>
> >>>>>> -  m_searches++;
> >>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>> +    if (insert == INSERT)
> >>>>>> +      verify (comparable, hash);
> >>>>>> +#endif
> >>>>>>
> >>>>>> +  m_searches++;
> >>>>>>    value_type *first_deleted_slot = NULL;
> >>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>    return &m_entries[index];
> >>>>>>  }
> >>>>>>
> >>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>> +
> >>>>>> +/* Report a hash table checking error.  */
> >>>>>> +
> >>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>> +static void
> >>>>>> +hashtab_chk_error ()
> >>>>>> +{
> >>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>> +     "equal operator returns true for a pair "
> >>>>>> +     "of values with a different hash value\n");
> >>>>>> +  gcc_unreachable ();
> >>>>>> +}
> >>>>> I think an internal_error here is probably still better than a simple
> >>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>
> >>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>> because we've got too many bugs to fix.
> >>>> Unfortunately it's blocked with these 2 PRs:
> >>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>> Hi.
> >>>
> >>> I've just added one more PR:
> >>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>
> >>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>> with a new function disable_sanitize_eq_and_hash.
> >>>
> >>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>> limits maximal number of checks:
> >> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >> state set up when you instantiate the object?  It's not a huge deal,
> >> just thinking about loud.
> >>
> >>
> >>
> >> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >> issue :-)
> >
> > There is at least one PR where we have a table where elements _in_ the
> > table are never compared against each other but always against another
> > object (I guess that's usual even), but the setup is in a way that the
> > comparison function only works with those.  With the patch we verify
> > hashing/comparison for something that is never used.
> >
> > So - wouldn't it be more "correct" to only verify comparison/hashing
> > at lookup time, using the object from the lookup and verify that against
> > all other elements?
>
> I don't a have problem with that. Apparently this changes fixes
> PR90450 and PR87847.
>
> Changes from previous version:
> - verification happens only when an element is searched (not inserted)
> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> - new param has been introduced hash-table-verification-limit in order
>   to limit number of elements that are compared within a table
> - verification happens only with flag_checking >= 2
>
> I've been bootstrapping and testing the patch right now.

Looks like I misremembered the original patch.  The issue isn't
comparing random two elements in the table.

That it fixes PR90450 is because LIM never calls find_slot_with_hash
without INSERTing.

I guess PR90450 is "real" indeed...

Richard.

> Martin
>
> >
> > Richard.
> >
> >>
> >> Jeff
>

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 7675 bytes --]

On 5/21/19 1:51 PM, Richard Biener wrote:
> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>
>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>> +{
>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>                     ^^^^^^
>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>> +}
>>>>>>>>>   Jakub
>>>>>>>>>
>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>
>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>
>>>>>>>> ---
>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>
>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>
>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>    bool too_empty_p (unsigned int);
>>>>>>>>    void expand ();
>>>>>>>>    static bool is_deleted (value_type &v)
>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>      expand ();
>>>>>>>>
>>>>>>>> -  m_searches++;
>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>> +    if (insert == INSERT)
>>>>>>>> +      verify (comparable, hash);
>>>>>>>> +#endif
>>>>>>>>
>>>>>>>> +  m_searches++;
>>>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>    return &m_entries[index];
>>>>>>>>  }
>>>>>>>>
>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>> +
>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>> +
>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>> +static void
>>>>>>>> +hashtab_chk_error ()
>>>>>>>> +{
>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>> +  gcc_unreachable ();
>>>>>>>> +}
>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>
>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>> because we've got too many bugs to fix.
>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>> Hi.
>>>>>
>>>>> I've just added one more PR:
>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>
>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>
>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>> limits maximal number of checks:
>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>> just thinking about loud.
>>>>
>>>>
>>>>
>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>> issue :-)
>>>
>>> There is at least one PR where we have a table where elements _in_ the
>>> table are never compared against each other but always against another
>>> object (I guess that's usual even), but the setup is in a way that the
>>> comparison function only works with those.  With the patch we verify
>>> hashing/comparison for something that is never used.
>>>
>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>> at lookup time, using the object from the lookup and verify that against
>>> all other elements?
>>
>> I don't a have problem with that. Apparently this changes fixes
>> PR90450 and PR87847.
>>
>> Changes from previous version:
>> - verification happens only when an element is searched (not inserted)
>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>> - new param has been introduced hash-table-verification-limit in order
>>   to limit number of elements that are compared within a table
>> - verification happens only with flag_checking >= 2
>>
>> I've been bootstrapping and testing the patch right now.
> 
> Looks like I misremembered the original patch.  The issue isn't
> comparing random two elements in the table.
> 
> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> without INSERTing.
> 

There's updated version of the patch where I check all find operations
(both w/ and w/o insertion).

Patch can bootstrap on x86_64-linux-gnu and survives regression tests
except for:

$ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
hash table checking failed: equal operator returns true for a pair of values with a different hash value
during GIMPLE pass: lim
/home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
/home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
    6 | fn1 ()
      | ^~~
0x6c5725 hashtab_chk_error
	/home/marxin/Programming/gcc/gcc/hash-table.h:1019
0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
	/home/marxin/Programming/gcc/gcc/hash-table.h:1040
0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
	/home/marxin/Programming/gcc/gcc/hash-table.h:960
0xe504ea gather_mem_refs_stmt
	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
0xe504ea analyze_memory_references
	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
0xe504ea tree_ssa_lim
	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
0xe504ea execute
	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708

Richi: it's after your recent patch.

For some reason I don't see PR87847 issue any longer.


May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?

Thanks,
Martin

> I guess PR90450 is "real" indeed...
> 
> Richard.
> 
>> Martin
>>
>>>
>>> Richard.
>>>
>>>>
>>>> Jeff
>>


[-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --]
[-- Type: text/x-patch, Size: 8179 bytes --]

From 569165f803e35eb4f2eecdf42919a05d885e3919 Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Mon, 13 May 2019 07:16:22 +0200
Subject: [PATCH] Enable sanitization for hash tables.

---
 gcc/cselib.c     |  9 ++++++--
 gcc/hash-set.h   |  2 +-
 gcc/hash-table.c |  3 +++
 gcc/hash-table.h | 60 ++++++++++++++++++++++++++++++++++++++++++++----
 gcc/params.def   |  6 +++++
 gcc/toplev.c     |  4 ++++
 6 files changed, 77 insertions(+), 7 deletions(-)

diff --git a/gcc/cselib.c b/gcc/cselib.c
index 84c17c23f6d..a1cbdec9718 100644
--- a/gcc/cselib.c
+++ b/gcc/cselib.c
@@ -2858,9 +2858,14 @@ cselib_init (int record_what)
     }
   used_regs = XNEWVEC (unsigned int, cselib_nregs);
   n_used_regs = 0;
-  cselib_hash_table = new hash_table<cselib_hasher> (31);
+  /* FIXME: enable sanitization (PR87845) */
+  cselib_hash_table
+    = new hash_table<cselib_hasher> (31, /* ggc */ false,
+				     /* sanitize_eq_and_hash */ false);
   if (cselib_preserve_constants)
-    cselib_preserved_hash_table = new hash_table<cselib_hasher> (31);
+    cselib_preserved_hash_table
+      = new hash_table<cselib_hasher> (31, /* ggc */ false,
+				       /* sanitize_eq_and_hash */ false);
   next_uid = 1;
 }
 
diff --git a/gcc/hash-set.h b/gcc/hash-set.h
index de3532f5f68..d891ed78297 100644
--- a/gcc/hash-set.h
+++ b/gcc/hash-set.h
@@ -28,7 +28,7 @@ class hash_set
 public:
   typedef typename Traits::value_type Key;
   explicit hash_set (size_t n = 13, bool ggc = false CXX_MEM_STAT_INFO)
-    : m_table (n, ggc, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {}
+    : m_table (n, ggc, true, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {}
 
   /* Create a hash_set in gc memory with space for at least n elements.  */
 
diff --git a/gcc/hash-table.c b/gcc/hash-table.c
index 646a7a1c497..8e86fffa36f 100644
--- a/gcc/hash-table.c
+++ b/gcc/hash-table.c
@@ -74,6 +74,9 @@ struct prime_ent const prime_tab[] = {
   { 0xfffffffb, 0x00000006, 0x00000008, 31 }
 };
 
+/* Limit number of comparisons when calling hash_table<>::verify.  */
+unsigned int hash_table_sanitize_eq_limit;
+
 /* The following function returns an index into the above table of the
    nearest prime number which is greater than N, and near a power of two. */
 
diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index 4178616478e..7e62715535d 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -295,6 +295,8 @@ struct prime_ent
 
 extern struct prime_ent const prime_tab[];
 
+/* Limit number of comparisons when calling hash_table<>::verify.  */
+extern unsigned int hash_table_sanitize_eq_limit;
 
 /* Functions for computing hash table indexes.  */
 
@@ -371,10 +373,12 @@ class hash_table
 
 public:
   explicit hash_table (size_t, bool ggc = false,
+		       bool sanitize_eq_and_hash = true,
 		       bool gather_mem_stats = GATHER_STATISTICS,
 		       mem_alloc_origin origin = HASH_TABLE_ORIGIN
 		       CXX_MEM_STAT_INFO);
   explicit hash_table (const hash_table &, bool ggc = false,
+		       bool sanitize_eq_and_hash = true,
 		       bool gather_mem_stats = GATHER_STATISTICS,
 		       mem_alloc_origin origin = HASH_TABLE_ORIGIN
 		       CXX_MEM_STAT_INFO);
@@ -516,6 +520,7 @@ private:
 
   value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
   value_type *find_empty_slot_for_expand (hashval_t);
+  void verify (const compare_type &comparable, hashval_t hash);
   bool too_empty_p (unsigned int);
   void expand ();
   static bool is_deleted (value_type &v)
@@ -564,6 +569,9 @@ private:
   /* if m_entries is stored in ggc memory.  */
   bool m_ggc;
 
+  /* True if the table should be sanitized for equal and hash functions.  */
+  bool m_sanitize_eq_and_hash;
+
   /* If we should gather memory statistics for the table.  */
 #if GATHER_STATISTICS
   bool m_gather_mem_stats;
@@ -586,12 +594,13 @@ extern void dump_hash_table_loc_statistics (void);
 template<typename Descriptor, bool Lazy,
 	 template<typename Type> class Allocator>
 hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc,
+						     bool sanitize_eq_and_hash,
 						     bool gather_mem_stats
 						     ATTRIBUTE_UNUSED,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0),
-  m_ggc (ggc)
+  m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash)
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -617,12 +626,14 @@ template<typename Descriptor, bool Lazy,
 	 template<typename Type> class Allocator>
 hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h,
 						     bool ggc,
+						     bool sanitize_eq_and_hash,
 						     bool gather_mem_stats
 						     ATTRIBUTE_UNUSED,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted),
-  m_searches (0), m_collisions (0), m_ggc (ggc)
+  m_searches (0), m_collisions (0), m_ggc (ggc),
+  m_sanitize_eq_and_hash (sanitize_eq_and_hash)
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -912,7 +923,11 @@ hash_table<Descriptor, Lazy, Allocator>
       entry = &m_entries[index];
       if (is_empty (*entry)
           || (!is_deleted (*entry) && Descriptor::equal (*entry, comparable)))
-        return *entry;
+	{
+	  if (m_sanitize_eq_and_hash)
+	    verify (comparable, hash);
+	  return *entry;
+	}
     }
 }
 
@@ -941,8 +956,10 @@ hash_table<Descriptor, Lazy, Allocator>
   if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
     expand ();
 
-  m_searches++;
+  if (m_sanitize_eq_and_hash)
+    verify (comparable, hash);
 
+  m_searches++;
   value_type *first_deleted_slot = NULL;
   hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
   hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
@@ -989,6 +1006,41 @@ hash_table<Descriptor, Lazy, Allocator>
   return &m_entries[index];
 }
 
+/* Report a hash table checking error.  */
+
+ATTRIBUTE_NORETURN ATTRIBUTE_COLD
+static void
+hashtab_chk_error ()
+{
+  fprintf (stderr, "hash table checking failed: "
+	   "equal operator returns true for a pair "
+	   "of values with a different hash value\n");
+#ifndef GENERATOR_FILE
+  gcc_unreachable ();
+#else
+  exit (1);
+#endif
+}
+
+/* Verify that all existing elements in th hash table which are
+   equal to COMPARABLE have an equal HASH value provided as argument.  */
+
+template<typename Descriptor, bool Lazy,
+	 template<typename Type> class Allocator>
+void
+hash_table<Descriptor, Lazy, Allocator>
+::verify (const compare_type &comparable, hashval_t hash)
+{
+  for (size_t i = 0; i < MIN (hash_table_sanitize_eq_limit, m_size); i++)
+    {
+      value_type *entry = &m_entries[i];
+      if (!is_empty (*entry) && !is_deleted (*entry)
+	  && hash != Descriptor::hash (*entry)
+	  && Descriptor::equal (*entry, comparable))
+	hashtab_chk_error ();
+    }
+}
+
 /* This function deletes an element with the given COMPARABLE value
    from hash table starting with the given HASH.  If there is no
    matching element in the hash table, this function does nothing. */
diff --git a/gcc/params.def b/gcc/params.def
index 6b7f7eb5bae..e5ca6ff45e4 100644
--- a/gcc/params.def
+++ b/gcc/params.def
@@ -1439,6 +1439,12 @@ DEFPARAM(PARAM_GIMPLE_FE_COMPUTED_HOT_BB_THRESHOLD,
 	 " The parameter is used only in GIMPLE FE.",
 	 0, 0, 0)
 
+DEFPARAM(PARAM_HASH_TABLE_VERIFICATION_LIMIT,
+	 "hash-table-verification-limit",
+	 "The number of elements for which hash table verification is done for "
+	 "each searched element.",
+	 100, 0, 0)
+
 /*
 
 Local variables:
diff --git a/gcc/toplev.c b/gcc/toplev.c
index d300ac2ec89..116be7be395 100644
--- a/gcc/toplev.c
+++ b/gcc/toplev.c
@@ -1799,6 +1799,10 @@ process_options (void)
   optimization_default_node = build_optimization_node (&global_options);
   optimization_current_node = optimization_default_node;
 
+  if (flag_checking >= 2)
+    hash_table_sanitize_eq_limit
+      = PARAM_VALUE (PARAM_HASH_TABLE_VERIFICATION_LIMIT);
+
   /* Please don't change global_options after this point, those changes won't
      be reflected in optimization_{default,current}_node.  */
 }
-- 
2.21.0

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Wed, May 22, 2019 at 11:13 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 5/21/19 1:51 PM, Richard Biener wrote:
> > On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>
> >>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>> +{
> >>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>                     ^^^^^^
> >>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>
> >>>>>>>> Martin
> >>>>>>>>
> >>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>> +}
> >>>>>>>>>   Jakub
> >>>>>>>>>
> >>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>
> >>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>
> >>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>
> >>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>    bool too_empty_p (unsigned int);
> >>>>>>>>    void expand ();
> >>>>>>>>    static bool is_deleted (value_type &v)
> >>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>      expand ();
> >>>>>>>>
> >>>>>>>> -  m_searches++;
> >>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>> +    if (insert == INSERT)
> >>>>>>>> +      verify (comparable, hash);
> >>>>>>>> +#endif
> >>>>>>>>
> >>>>>>>> +  m_searches++;
> >>>>>>>>    value_type *first_deleted_slot = NULL;
> >>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>    return &m_entries[index];
> >>>>>>>>  }
> >>>>>>>>
> >>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>> +
> >>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>> +
> >>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>> +static void
> >>>>>>>> +hashtab_chk_error ()
> >>>>>>>> +{
> >>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>> +  gcc_unreachable ();
> >>>>>>>> +}
> >>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>
> >>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>> because we've got too many bugs to fix.
> >>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>> Hi.
> >>>>>
> >>>>> I've just added one more PR:
> >>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>
> >>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>
> >>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>> limits maximal number of checks:
> >>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>> just thinking about loud.
> >>>>
> >>>>
> >>>>
> >>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>> issue :-)
> >>>
> >>> There is at least one PR where we have a table where elements _in_ the
> >>> table are never compared against each other but always against another
> >>> object (I guess that's usual even), but the setup is in a way that the
> >>> comparison function only works with those.  With the patch we verify
> >>> hashing/comparison for something that is never used.
> >>>
> >>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>> at lookup time, using the object from the lookup and verify that against
> >>> all other elements?
> >>
> >> I don't a have problem with that. Apparently this changes fixes
> >> PR90450 and PR87847.
> >>
> >> Changes from previous version:
> >> - verification happens only when an element is searched (not inserted)
> >> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >> - new param has been introduced hash-table-verification-limit in order
> >>   to limit number of elements that are compared within a table
> >> - verification happens only with flag_checking >= 2
> >>
> >> I've been bootstrapping and testing the patch right now.
> >
> > Looks like I misremembered the original patch.  The issue isn't
> > comparing random two elements in the table.
> >
> > That it fixes PR90450 is because LIM never calls find_slot_with_hash
> > without INSERTing.
> >
>
> There's updated version of the patch where I check all find operations
> (both w/ and w/o insertion).
>
> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> except for:
>
> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> during GIMPLE pass: lim
> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>     6 | fn1 ()
>       | ^~~
> 0x6c5725 hashtab_chk_error
>         /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>         /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>         /home/marxin/Programming/gcc/gcc/hash-table.h:960
> 0xe504ea gather_mem_refs_stmt
>         /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> 0xe504ea analyze_memory_references
>         /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> 0xe504ea tree_ssa_lim
>         /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> 0xe504ea execute
>         /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>
> Richi: it's after your recent patch.

I can't reproduce this (but I have a slightly patched tree).

Richard.

> For some reason I don't see PR87847 issue any longer.
>
>
> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>
> Thanks,
> Martin
>
> > I guess PR90450 is "real" indeed...
> >
> > Richard.
> >
> >> Martin
> >>
> >>>
> >>> Richard.
> >>>
> >>>>
> >>>> Jeff
> >>
>

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 5/31/19 2:50 PM, Richard Biener wrote:
> On Wed, May 22, 2019 at 11:13 AM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>
>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>
>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>> +{
>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>                     ^^^^^^
>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>> +}
>>>>>>>>>>>   Jakub
>>>>>>>>>>>
>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>
>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>
>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>    bool too_empty_p (unsigned int);
>>>>>>>>>>    void expand ();
>>>>>>>>>>    static bool is_deleted (value_type &v)
>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>      expand ();
>>>>>>>>>>
>>>>>>>>>> -  m_searches++;
>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>> +#endif
>>>>>>>>>>
>>>>>>>>>> +  m_searches++;
>>>>>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>    return &m_entries[index];
>>>>>>>>>>  }
>>>>>>>>>>
>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>> +
>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>> +
>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>> +static void
>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>> +{
>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>> +}
>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>
>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>> Hi.
>>>>>>>
>>>>>>> I've just added one more PR:
>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>
>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>
>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>> limits maximal number of checks:
>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>> just thinking about loud.
>>>>>>
>>>>>>
>>>>>>
>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>> issue :-)
>>>>>
>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>> table are never compared against each other but always against another
>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>> comparison function only works with those.  With the patch we verify
>>>>> hashing/comparison for something that is never used.
>>>>>
>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>> at lookup time, using the object from the lookup and verify that against
>>>>> all other elements?
>>>>
>>>> I don't a have problem with that. Apparently this changes fixes
>>>> PR90450 and PR87847.
>>>>
>>>> Changes from previous version:
>>>> - verification happens only when an element is searched (not inserted)
>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>> - new param has been introduced hash-table-verification-limit in order
>>>>   to limit number of elements that are compared within a table
>>>> - verification happens only with flag_checking >= 2
>>>>
>>>> I've been bootstrapping and testing the patch right now.
>>>
>>> Looks like I misremembered the original patch.  The issue isn't
>>> comparing random two elements in the table.
>>>
>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>> without INSERTing.
>>>
>>
>> There's updated version of the patch where I check all find operations
>> (both w/ and w/o insertion).
>>
>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>> except for:
>>
>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>> during GIMPLE pass: lim
>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>     6 | fn1 ()
>>       | ^~~
>> 0x6c5725 hashtab_chk_error
>>         /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>         /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>         /home/marxin/Programming/gcc/gcc/hash-table.h:960
>> 0xe504ea gather_mem_refs_stmt
>>         /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>> 0xe504ea analyze_memory_references
>>         /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>> 0xe504ea tree_ssa_lim
>>         /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>> 0xe504ea execute
>>         /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>
>> Richi: it's after your recent patch.
> 
> I can't reproduce this (but I have a slightly patched tree).

Me neither. I'll retest the whole patch again.

Martin

> 
> Richard.
> 
>> For some reason I don't see PR87847 issue any longer.
>>
>>
>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>
>> Thanks,
>> Martin
>>
>>> I guess PR90450 is "real" indeed...
>>>
>>> Richard.
>>>
>>>> Martin
>>>>
>>>>>
>>>>> Richard.
>>>>>
>>>>>>
>>>>>> Jeff
>>>>
>>

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jeff Law]

On 5/22/19 3:13 AM, Martin Liška wrote:
> On 5/21/19 1:51 PM, Richard Biener wrote:
>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>
>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>
>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>> +{
>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>                     ^^^^^^
>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>> +}
>>>>>>>>>>   Jakub
>>>>>>>>>>
>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>
>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>
>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>
>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>    bool too_empty_p (unsigned int);
>>>>>>>>>    void expand ();
>>>>>>>>>    static bool is_deleted (value_type &v)
>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>      expand ();
>>>>>>>>>
>>>>>>>>> -  m_searches++;
>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>> +#endif
>>>>>>>>>
>>>>>>>>> +  m_searches++;
>>>>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>    return &m_entries[index];
>>>>>>>>>  }
>>>>>>>>>
>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>> +
>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>> +
>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>> +static void
>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>> +{
>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>> +}
>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>
>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>> because we've got too many bugs to fix.
>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>> Hi.
>>>>>>
>>>>>> I've just added one more PR:
>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>
>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>
>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>> limits maximal number of checks:
>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>> just thinking about loud.
>>>>>
>>>>>
>>>>>
>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>> issue :-)
>>>>
>>>> There is at least one PR where we have a table where elements _in_ the
>>>> table are never compared against each other but always against another
>>>> object (I guess that's usual even), but the setup is in a way that the
>>>> comparison function only works with those.  With the patch we verify
>>>> hashing/comparison for something that is never used.
>>>>
>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>> at lookup time, using the object from the lookup and verify that against
>>>> all other elements?
>>>
>>> I don't a have problem with that. Apparently this changes fixes
>>> PR90450 and PR87847.
>>>
>>> Changes from previous version:
>>> - verification happens only when an element is searched (not inserted)
>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>> - new param has been introduced hash-table-verification-limit in order
>>>   to limit number of elements that are compared within a table
>>> - verification happens only with flag_checking >= 2
>>>
>>> I've been bootstrapping and testing the patch right now.
>>
>> Looks like I misremembered the original patch.  The issue isn't
>> comparing random two elements in the table.
>>
>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>> without INSERTing.
>>
> 
> There's updated version of the patch where I check all find operations
> (both w/ and w/o insertion).
> 
> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> except for:
> 
> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> during GIMPLE pass: lim
> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>     6 | fn1 ()
>       | ^~~
> 0x6c5725 hashtab_chk_error
> 	/home/marxin/Programming/gcc/gcc/hash-table.h:1019
> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> 	/home/marxin/Programming/gcc/gcc/hash-table.h:1040
> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> 	/home/marxin/Programming/gcc/gcc/hash-table.h:960
> 0xe504ea gather_mem_refs_stmt
> 	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> 0xe504ea analyze_memory_references
> 	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> 0xe504ea tree_ssa_lim
> 	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> 0xe504ea execute
> 	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> 
> Richi: it's after your recent patch.
> 
> For some reason I don't see PR87847 issue any longer.
> 
> 
> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
Don't we still need to deal with the naked fprintf when there's a
failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?

jeff

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 8315 bytes --]

On 6/1/19 12:06 AM, Jeff Law wrote:
> On 5/22/19 3:13 AM, Martin Liška wrote:
>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>
>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>
>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>> +{
>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>                     ^^^^^^
>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>> +}
>>>>>>>>>>>   Jakub
>>>>>>>>>>>
>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>
>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>
>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>    bool too_empty_p (unsigned int);
>>>>>>>>>>    void expand ();
>>>>>>>>>>    static bool is_deleted (value_type &v)
>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>      expand ();
>>>>>>>>>>
>>>>>>>>>> -  m_searches++;
>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>> +#endif
>>>>>>>>>>
>>>>>>>>>> +  m_searches++;
>>>>>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>    return &m_entries[index];
>>>>>>>>>>  }
>>>>>>>>>>
>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>> +
>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>> +
>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>> +static void
>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>> +{
>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>> +}
>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>
>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>> Hi.
>>>>>>>
>>>>>>> I've just added one more PR:
>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>
>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>
>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>> limits maximal number of checks:
>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>> just thinking about loud.
>>>>>>
>>>>>>
>>>>>>
>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>> issue :-)
>>>>>
>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>> table are never compared against each other but always against another
>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>> comparison function only works with those.  With the patch we verify
>>>>> hashing/comparison for something that is never used.
>>>>>
>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>> at lookup time, using the object from the lookup and verify that against
>>>>> all other elements?
>>>>
>>>> I don't a have problem with that. Apparently this changes fixes
>>>> PR90450 and PR87847.
>>>>
>>>> Changes from previous version:
>>>> - verification happens only when an element is searched (not inserted)
>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>> - new param has been introduced hash-table-verification-limit in order
>>>>   to limit number of elements that are compared within a table
>>>> - verification happens only with flag_checking >= 2
>>>>
>>>> I've been bootstrapping and testing the patch right now.
>>>
>>> Looks like I misremembered the original patch.  The issue isn't
>>> comparing random two elements in the table.
>>>
>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>> without INSERTing.
>>>
>>
>> There's updated version of the patch where I check all find operations
>> (both w/ and w/o insertion).
>>
>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>> except for:
>>
>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>> during GIMPLE pass: lim
>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>     6 | fn1 ()
>>       | ^~~
>> 0x6c5725 hashtab_chk_error
>> 	/home/marxin/Programming/gcc/gcc/hash-table.h:1019
>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>> 	/home/marxin/Programming/gcc/gcc/hash-table.h:1040
>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>> 	/home/marxin/Programming/gcc/gcc/hash-table.h:960
>> 0xe504ea gather_mem_refs_stmt
>> 	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>> 0xe504ea analyze_memory_references
>> 	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>> 0xe504ea tree_ssa_lim
>> 	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>> 0xe504ea execute
>> 	/home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>
>> Richi: it's after your recent patch.
>>
>> For some reason I don't see PR87847 issue any longer.
>>
>>
>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> Don't we still need to deal with the naked fprintf when there's a
> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?

Good point, I've just adjusted that.

Patch can bootstrap on x86_64-linux-gnu and survives regression tests.

Ready to be installed?
Thanks,
Martin

> 
> jeff
> 


[-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --]
[-- Type: text/x-patch, Size: 8647 bytes --]

From a51fc8dc210942b87d84130bb105467112fa9967 Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Mon, 13 May 2019 07:16:22 +0200
Subject: [PATCH] Enable sanitization for hash tables.

gcc/ChangeLog:

2019-06-03  Martin Liska  <mliska@suse.cz>

	* cselib.c (cselib_init): Disable hash table
	sanitization.
	* hash-set.h: Pass new default argument to m_table.
	* hash-table.c: Add global variable with hash table
	sanitization limit.
	* hash-table.h (Allocator>::hash_table): Add new argument
	to ctor.
	(hashtab_chk_error): New.
	* params.def (PARAM_HASH_TABLE_VERIFICATION_LIMIT): New.
	* toplev.c (process_options): Set hash_table_sanitize_eq_limit
	from the PARAM_HASH_TABLE_VERIFICATION_LIMIT value.
---
 gcc/cselib.c     |  9 ++++++--
 gcc/hash-set.h   |  2 +-
 gcc/hash-table.c |  3 +++
 gcc/hash-table.h | 56 ++++++++++++++++++++++++++++++++++++++++++++----
 gcc/params.def   |  6 ++++++
 gcc/toplev.c     |  4 ++++
 6 files changed, 73 insertions(+), 7 deletions(-)

diff --git a/gcc/cselib.c b/gcc/cselib.c
index 84c17c23f6d..a1cbdec9718 100644
--- a/gcc/cselib.c
+++ b/gcc/cselib.c
@@ -2858,9 +2858,14 @@ cselib_init (int record_what)
     }
   used_regs = XNEWVEC (unsigned int, cselib_nregs);
   n_used_regs = 0;
-  cselib_hash_table = new hash_table<cselib_hasher> (31);
+  /* FIXME: enable sanitization (PR87845) */
+  cselib_hash_table
+    = new hash_table<cselib_hasher> (31, /* ggc */ false,
+				     /* sanitize_eq_and_hash */ false);
   if (cselib_preserve_constants)
-    cselib_preserved_hash_table = new hash_table<cselib_hasher> (31);
+    cselib_preserved_hash_table
+      = new hash_table<cselib_hasher> (31, /* ggc */ false,
+				       /* sanitize_eq_and_hash */ false);
   next_uid = 1;
 }
 
diff --git a/gcc/hash-set.h b/gcc/hash-set.h
index de3532f5f68..d891ed78297 100644
--- a/gcc/hash-set.h
+++ b/gcc/hash-set.h
@@ -28,7 +28,7 @@ class hash_set
 public:
   typedef typename Traits::value_type Key;
   explicit hash_set (size_t n = 13, bool ggc = false CXX_MEM_STAT_INFO)
-    : m_table (n, ggc, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {}
+    : m_table (n, ggc, true, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {}
 
   /* Create a hash_set in gc memory with space for at least n elements.  */
 
diff --git a/gcc/hash-table.c b/gcc/hash-table.c
index 646a7a1c497..8e86fffa36f 100644
--- a/gcc/hash-table.c
+++ b/gcc/hash-table.c
@@ -74,6 +74,9 @@ struct prime_ent const prime_tab[] = {
   { 0xfffffffb, 0x00000006, 0x00000008, 31 }
 };
 
+/* Limit number of comparisons when calling hash_table<>::verify.  */
+unsigned int hash_table_sanitize_eq_limit;
+
 /* The following function returns an index into the above table of the
    nearest prime number which is greater than N, and near a power of two. */
 
diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index 4178616478e..785bed14679 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -295,6 +295,8 @@ struct prime_ent
 
 extern struct prime_ent const prime_tab[];
 
+/* Limit number of comparisons when calling hash_table<>::verify.  */
+extern unsigned int hash_table_sanitize_eq_limit;
 
 /* Functions for computing hash table indexes.  */
 
@@ -371,10 +373,12 @@ class hash_table
 
 public:
   explicit hash_table (size_t, bool ggc = false,
+		       bool sanitize_eq_and_hash = true,
 		       bool gather_mem_stats = GATHER_STATISTICS,
 		       mem_alloc_origin origin = HASH_TABLE_ORIGIN
 		       CXX_MEM_STAT_INFO);
   explicit hash_table (const hash_table &, bool ggc = false,
+		       bool sanitize_eq_and_hash = true,
 		       bool gather_mem_stats = GATHER_STATISTICS,
 		       mem_alloc_origin origin = HASH_TABLE_ORIGIN
 		       CXX_MEM_STAT_INFO);
@@ -516,6 +520,7 @@ private:
 
   value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
   value_type *find_empty_slot_for_expand (hashval_t);
+  void verify (const compare_type &comparable, hashval_t hash);
   bool too_empty_p (unsigned int);
   void expand ();
   static bool is_deleted (value_type &v)
@@ -564,6 +569,9 @@ private:
   /* if m_entries is stored in ggc memory.  */
   bool m_ggc;
 
+  /* True if the table should be sanitized for equal and hash functions.  */
+  bool m_sanitize_eq_and_hash;
+
   /* If we should gather memory statistics for the table.  */
 #if GATHER_STATISTICS
   bool m_gather_mem_stats;
@@ -586,12 +594,13 @@ extern void dump_hash_table_loc_statistics (void);
 template<typename Descriptor, bool Lazy,
 	 template<typename Type> class Allocator>
 hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc,
+						     bool sanitize_eq_and_hash,
 						     bool gather_mem_stats
 						     ATTRIBUTE_UNUSED,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0),
-  m_ggc (ggc)
+  m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash)
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -617,12 +626,14 @@ template<typename Descriptor, bool Lazy,
 	 template<typename Type> class Allocator>
 hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h,
 						     bool ggc,
+						     bool sanitize_eq_and_hash,
 						     bool gather_mem_stats
 						     ATTRIBUTE_UNUSED,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted),
-  m_searches (0), m_collisions (0), m_ggc (ggc)
+  m_searches (0), m_collisions (0), m_ggc (ggc),
+  m_sanitize_eq_and_hash (sanitize_eq_and_hash)
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -912,7 +923,11 @@ hash_table<Descriptor, Lazy, Allocator>
       entry = &m_entries[index];
       if (is_empty (*entry)
           || (!is_deleted (*entry) && Descriptor::equal (*entry, comparable)))
-        return *entry;
+	{
+	  if (m_sanitize_eq_and_hash)
+	    verify (comparable, hash);
+	  return *entry;
+	}
     }
 }
 
@@ -941,8 +956,10 @@ hash_table<Descriptor, Lazy, Allocator>
   if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
     expand ();
 
-  m_searches++;
+  if (m_sanitize_eq_and_hash)
+    verify (comparable, hash);
 
+  m_searches++;
   value_type *first_deleted_slot = NULL;
   hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
   hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
@@ -989,6 +1006,37 @@ hash_table<Descriptor, Lazy, Allocator>
   return &m_entries[index];
 }
 
+/* Report a hash table checking error.  */
+
+ATTRIBUTE_NORETURN ATTRIBUTE_COLD
+static void
+hashtab_chk_error ()
+{
+  fprintf (stderr, "hash table checking failed: "
+	   "equal operator returns true for a pair "
+	   "of values with a different hash value\n");
+  gcc_unreachable ();
+}
+
+/* Verify that all existing elements in th hash table which are
+   equal to COMPARABLE have an equal HASH value provided as argument.  */
+
+template<typename Descriptor, bool Lazy,
+	 template<typename Type> class Allocator>
+void
+hash_table<Descriptor, Lazy, Allocator>
+::verify (const compare_type &comparable, hashval_t hash)
+{
+  for (size_t i = 0; i < MIN (hash_table_sanitize_eq_limit, m_size); i++)
+    {
+      value_type *entry = &m_entries[i];
+      if (!is_empty (*entry) && !is_deleted (*entry)
+	  && hash != Descriptor::hash (*entry)
+	  && Descriptor::equal (*entry, comparable))
+	hashtab_chk_error ();
+    }
+}
+
 /* This function deletes an element with the given COMPARABLE value
    from hash table starting with the given HASH.  If there is no
    matching element in the hash table, this function does nothing. */
diff --git a/gcc/params.def b/gcc/params.def
index 6b7f7eb5bae..e5ca6ff45e4 100644
--- a/gcc/params.def
+++ b/gcc/params.def
@@ -1439,6 +1439,12 @@ DEFPARAM(PARAM_GIMPLE_FE_COMPUTED_HOT_BB_THRESHOLD,
 	 " The parameter is used only in GIMPLE FE.",
 	 0, 0, 0)
 
+DEFPARAM(PARAM_HASH_TABLE_VERIFICATION_LIMIT,
+	 "hash-table-verification-limit",
+	 "The number of elements for which hash table verification is done for "
+	 "each searched element.",
+	 100, 0, 0)
+
 /*
 
 Local variables:
diff --git a/gcc/toplev.c b/gcc/toplev.c
index d300ac2ec89..116be7be395 100644
--- a/gcc/toplev.c
+++ b/gcc/toplev.c
@@ -1799,6 +1799,10 @@ process_options (void)
   optimization_default_node = build_optimization_node (&global_options);
   optimization_current_node = optimization_default_node;
 
+  if (flag_checking >= 2)
+    hash_table_sanitize_eq_limit
+      = PARAM_VALUE (PARAM_HASH_TABLE_VERIFICATION_LIMIT);
+
   /* Please don't change global_options after this point, those changes won't
      be reflected in optimization_{default,current}_node.  */
 }
-- 
2.21.0

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/1/19 12:06 AM, Jeff Law wrote:
> > On 5/22/19 3:13 AM, Martin Liška wrote:
> >> On 5/21/19 1:51 PM, Richard Biener wrote:
> >>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>
> >>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>>>
> >>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>> +{
> >>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>>>                     ^^^^^^
> >>>>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>>>
> >>>>>>>>>> Martin
> >>>>>>>>>>
> >>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>> +}
> >>>>>>>>>>>   Jakub
> >>>>>>>>>>>
> >>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>>>
> >>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>>>
> >>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>>>    bool too_empty_p (unsigned int);
> >>>>>>>>>>    void expand ();
> >>>>>>>>>>    static bool is_deleted (value_type &v)
> >>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>>>      expand ();
> >>>>>>>>>>
> >>>>>>>>>> -  m_searches++;
> >>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>> +    if (insert == INSERT)
> >>>>>>>>>> +      verify (comparable, hash);
> >>>>>>>>>> +#endif
> >>>>>>>>>>
> >>>>>>>>>> +  m_searches++;
> >>>>>>>>>>    value_type *first_deleted_slot = NULL;
> >>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>    return &m_entries[index];
> >>>>>>>>>>  }
> >>>>>>>>>>
> >>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>> +
> >>>>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>>>> +
> >>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>>>> +static void
> >>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>> +{
> >>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>> +}
> >>>>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>>>
> >>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>>>> because we've got too many bugs to fix.
> >>>>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>>>> Hi.
> >>>>>>>
> >>>>>>> I've just added one more PR:
> >>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>>>
> >>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>>>
> >>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>>>> limits maximal number of checks:
> >>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>>>> just thinking about loud.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>>>> issue :-)
> >>>>>
> >>>>> There is at least one PR where we have a table where elements _in_ the
> >>>>> table are never compared against each other but always against another
> >>>>> object (I guess that's usual even), but the setup is in a way that the
> >>>>> comparison function only works with those.  With the patch we verify
> >>>>> hashing/comparison for something that is never used.
> >>>>>
> >>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>>>> at lookup time, using the object from the lookup and verify that against
> >>>>> all other elements?
> >>>>
> >>>> I don't a have problem with that. Apparently this changes fixes
> >>>> PR90450 and PR87847.
> >>>>
> >>>> Changes from previous version:
> >>>> - verification happens only when an element is searched (not inserted)
> >>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >>>> - new param has been introduced hash-table-verification-limit in order
> >>>>   to limit number of elements that are compared within a table
> >>>> - verification happens only with flag_checking >= 2
> >>>>
> >>>> I've been bootstrapping and testing the patch right now.
> >>>
> >>> Looks like I misremembered the original patch.  The issue isn't
> >>> comparing random two elements in the table.
> >>>
> >>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> >>> without INSERTing.
> >>>
> >>
> >> There's updated version of the patch where I check all find operations
> >> (both w/ and w/o insertion).
> >>
> >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> >> except for:
> >>
> >> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> >> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> >> during GIMPLE pass: lim
> >> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> >> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> >>     6 | fn1 ()
> >>       | ^~~
> >> 0x6c5725 hashtab_chk_error
> >>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> >> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> >>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> >> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> >>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
> >> 0xe504ea gather_mem_refs_stmt
> >>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> >> 0xe504ea analyze_memory_references
> >>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> >> 0xe504ea tree_ssa_lim
> >>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> >> 0xe504ea execute
> >>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> >>
> >> Richi: it's after your recent patch.
> >>
> >> For some reason I don't see PR87847 issue any longer.
> >>
> >>
> >> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> > Don't we still need to deal with the naked fprintf when there's a
> > failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>
> Good point, I've just adjusted that.
>
> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>
> Ready to be installed?

Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
fixing it ...

One question - there's unconditional

+         if (m_sanitize_eq_and_hash)
+           verify (comparable, hash);

which will read a global variable and have (possibly not inline) call
to verify on a common path even with checking disabled.  So I think
we want to compile this checking feature out for !CHECKING_P
or at least make the if __builtin_expect (..., 0), ::verify not
inlined and marked pure () (thus, !CHECKING_P is simplest ;)).

Thanks,
Richard.

> Thanks,
> Martin
>
> >
> > jeff
> >
>

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 9584 bytes --]

On 6/7/19 10:57 AM, Richard Biener wrote:
> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>
>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>
>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>                     ^^^^^^
>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>
>>>>>>>>>>>> Martin
>>>>>>>>>>>>
>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>   Jakub
>>>>>>>>>>>>>
>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>
>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>
>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>    bool too_empty_p (unsigned int);
>>>>>>>>>>>>    void expand ();
>>>>>>>>>>>>    static bool is_deleted (value_type &v)
>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>      expand ();
>>>>>>>>>>>>
>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>> +#endif
>>>>>>>>>>>>
>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>    return &m_entries[index];
>>>>>>>>>>>>  }
>>>>>>>>>>>>
>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>> +
>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>> +
>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>> +static void
>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>> +{
>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>> +}
>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>
>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> I've just added one more PR:
>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>
>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>
>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>> limits maximal number of checks:
>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>> just thinking about loud.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>> issue :-)
>>>>>>>
>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>> table are never compared against each other but always against another
>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>> hashing/comparison for something that is never used.
>>>>>>>
>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>> all other elements?
>>>>>>
>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>> PR90450 and PR87847.
>>>>>>
>>>>>> Changes from previous version:
>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>   to limit number of elements that are compared within a table
>>>>>> - verification happens only with flag_checking >= 2
>>>>>>
>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>
>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>> comparing random two elements in the table.
>>>>>
>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>> without INSERTing.
>>>>>
>>>>
>>>> There's updated version of the patch where I check all find operations
>>>> (both w/ and w/o insertion).
>>>>
>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>> except for:
>>>>
>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>> during GIMPLE pass: lim
>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>     6 | fn1 ()
>>>>       | ^~~
>>>> 0x6c5725 hashtab_chk_error
>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>> 0xe504ea gather_mem_refs_stmt
>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>> 0xe504ea analyze_memory_references
>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>> 0xe504ea tree_ssa_lim
>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>> 0xe504ea execute
>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>
>>>> Richi: it's after your recent patch.
>>>>
>>>> For some reason I don't see PR87847 issue any longer.
>>>>
>>>>
>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>> Don't we still need to deal with the naked fprintf when there's a
>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>
>> Good point, I've just adjusted that.
>>
>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>
>> Ready to be installed?
> 
> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> fixing it ...

Yes :D It's been some time and there's no interest in the PR.

> 
> One question - there's unconditional
> 
> +         if (m_sanitize_eq_and_hash)
> +           verify (comparable, hash);
> 
> which will read a global variable and have (possibly not inline) call
> to verify on a common path even with checking disabled.  So I think
> we want to compile this checking feature out for !CHECKING_P
> or at least make the if __builtin_expect (..., 0), ::verify not
> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).

Fixed. May I install the patch? The cselib issue can be solved later..

Martin

> 
> Thanks,
> Richard.
> 
>> Thanks,
>> Martin
>>
>>>
>>> jeff
>>>
>>


[-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --]
[-- Type: text/x-patch, Size: 8670 bytes --]

From 1ccf332932661b087bd76d017f888572abca594c Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Mon, 13 May 2019 07:16:22 +0200
Subject: [PATCH] Enable sanitization for hash tables.

gcc/ChangeLog:

2019-06-03  Martin Liska  <mliska@suse.cz>

	* cselib.c (cselib_init): Disable hash table
	sanitization.
	* hash-set.h: Pass new default argument to m_table.
	* hash-table.c: Add global variable with hash table
	sanitization limit.
	* hash-table.h (Allocator>::hash_table): Add new argument
	to ctor.
	(hashtab_chk_error): New.
	* params.def (PARAM_HASH_TABLE_VERIFICATION_LIMIT): New.
	* toplev.c (process_options): Set hash_table_sanitize_eq_limit
	from the PARAM_HASH_TABLE_VERIFICATION_LIMIT value.
---
 gcc/cselib.c     |  9 ++++++--
 gcc/hash-set.h   |  2 +-
 gcc/hash-table.c |  3 +++
 gcc/hash-table.h | 58 ++++++++++++++++++++++++++++++++++++++++++++----
 gcc/params.def   |  6 +++++
 gcc/toplev.c     |  4 ++++
 6 files changed, 75 insertions(+), 7 deletions(-)

diff --git a/gcc/cselib.c b/gcc/cselib.c
index 84c17c23f6d..a1cbdec9718 100644
--- a/gcc/cselib.c
+++ b/gcc/cselib.c
@@ -2858,9 +2858,14 @@ cselib_init (int record_what)
     }
   used_regs = XNEWVEC (unsigned int, cselib_nregs);
   n_used_regs = 0;
-  cselib_hash_table = new hash_table<cselib_hasher> (31);
+  /* FIXME: enable sanitization (PR87845) */
+  cselib_hash_table
+    = new hash_table<cselib_hasher> (31, /* ggc */ false,
+				     /* sanitize_eq_and_hash */ false);
   if (cselib_preserve_constants)
-    cselib_preserved_hash_table = new hash_table<cselib_hasher> (31);
+    cselib_preserved_hash_table
+      = new hash_table<cselib_hasher> (31, /* ggc */ false,
+				       /* sanitize_eq_and_hash */ false);
   next_uid = 1;
 }
 
diff --git a/gcc/hash-set.h b/gcc/hash-set.h
index de3532f5f68..d891ed78297 100644
--- a/gcc/hash-set.h
+++ b/gcc/hash-set.h
@@ -28,7 +28,7 @@ class hash_set
 public:
   typedef typename Traits::value_type Key;
   explicit hash_set (size_t n = 13, bool ggc = false CXX_MEM_STAT_INFO)
-    : m_table (n, ggc, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {}
+    : m_table (n, ggc, true, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {}
 
   /* Create a hash_set in gc memory with space for at least n elements.  */
 
diff --git a/gcc/hash-table.c b/gcc/hash-table.c
index 646a7a1c497..8e86fffa36f 100644
--- a/gcc/hash-table.c
+++ b/gcc/hash-table.c
@@ -74,6 +74,9 @@ struct prime_ent const prime_tab[] = {
   { 0xfffffffb, 0x00000006, 0x00000008, 31 }
 };
 
+/* Limit number of comparisons when calling hash_table<>::verify.  */
+unsigned int hash_table_sanitize_eq_limit;
+
 /* The following function returns an index into the above table of the
    nearest prime number which is greater than N, and near a power of two. */
 
diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index 4178616478e..bf457f80d1b 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -295,6 +295,8 @@ struct prime_ent
 
 extern struct prime_ent const prime_tab[];
 
+/* Limit number of comparisons when calling hash_table<>::verify.  */
+extern unsigned int hash_table_sanitize_eq_limit;
 
 /* Functions for computing hash table indexes.  */
 
@@ -371,10 +373,12 @@ class hash_table
 
 public:
   explicit hash_table (size_t, bool ggc = false,
+		       bool sanitize_eq_and_hash = true,
 		       bool gather_mem_stats = GATHER_STATISTICS,
 		       mem_alloc_origin origin = HASH_TABLE_ORIGIN
 		       CXX_MEM_STAT_INFO);
   explicit hash_table (const hash_table &, bool ggc = false,
+		       bool sanitize_eq_and_hash = true,
 		       bool gather_mem_stats = GATHER_STATISTICS,
 		       mem_alloc_origin origin = HASH_TABLE_ORIGIN
 		       CXX_MEM_STAT_INFO);
@@ -516,6 +520,7 @@ private:
 
   value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
   value_type *find_empty_slot_for_expand (hashval_t);
+  void verify (const compare_type &comparable, hashval_t hash);
   bool too_empty_p (unsigned int);
   void expand ();
   static bool is_deleted (value_type &v)
@@ -564,6 +569,9 @@ private:
   /* if m_entries is stored in ggc memory.  */
   bool m_ggc;
 
+  /* True if the table should be sanitized for equal and hash functions.  */
+  bool m_sanitize_eq_and_hash;
+
   /* If we should gather memory statistics for the table.  */
 #if GATHER_STATISTICS
   bool m_gather_mem_stats;
@@ -586,12 +594,13 @@ extern void dump_hash_table_loc_statistics (void);
 template<typename Descriptor, bool Lazy,
 	 template<typename Type> class Allocator>
 hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc,
+						     bool sanitize_eq_and_hash,
 						     bool gather_mem_stats
 						     ATTRIBUTE_UNUSED,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0),
-  m_ggc (ggc)
+  m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash)
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -617,12 +626,14 @@ template<typename Descriptor, bool Lazy,
 	 template<typename Type> class Allocator>
 hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h,
 						     bool ggc,
+						     bool sanitize_eq_and_hash,
 						     bool gather_mem_stats
 						     ATTRIBUTE_UNUSED,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted),
-  m_searches (0), m_collisions (0), m_ggc (ggc)
+  m_searches (0), m_collisions (0), m_ggc (ggc),
+  m_sanitize_eq_and_hash (sanitize_eq_and_hash)
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -912,7 +923,13 @@ hash_table<Descriptor, Lazy, Allocator>
       entry = &m_entries[index];
       if (is_empty (*entry)
           || (!is_deleted (*entry) && Descriptor::equal (*entry, comparable)))
-        return *entry;
+	{
+#if CHECKING_P
+	  if (m_sanitize_eq_and_hash)
+	    verify (comparable, hash);
+#endif
+	  return *entry;
+	}
     }
 }
 
@@ -941,8 +958,10 @@ hash_table<Descriptor, Lazy, Allocator>
   if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
     expand ();
 
-  m_searches++;
+  if (m_sanitize_eq_and_hash)
+    verify (comparable, hash);
 
+  m_searches++;
   value_type *first_deleted_slot = NULL;
   hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
   hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
@@ -989,6 +1008,37 @@ hash_table<Descriptor, Lazy, Allocator>
   return &m_entries[index];
 }
 
+/* Report a hash table checking error.  */
+
+ATTRIBUTE_NORETURN ATTRIBUTE_COLD
+static void
+hashtab_chk_error ()
+{
+  fprintf (stderr, "hash table checking failed: "
+	   "equal operator returns true for a pair "
+	   "of values with a different hash value\n");
+  gcc_unreachable ();
+}
+
+/* Verify that all existing elements in th hash table which are
+   equal to COMPARABLE have an equal HASH value provided as argument.  */
+
+template<typename Descriptor, bool Lazy,
+	 template<typename Type> class Allocator>
+void
+hash_table<Descriptor, Lazy, Allocator>
+::verify (const compare_type &comparable, hashval_t hash)
+{
+  for (size_t i = 0; i < MIN (hash_table_sanitize_eq_limit, m_size); i++)
+    {
+      value_type *entry = &m_entries[i];
+      if (!is_empty (*entry) && !is_deleted (*entry)
+	  && hash != Descriptor::hash (*entry)
+	  && Descriptor::equal (*entry, comparable))
+	hashtab_chk_error ();
+    }
+}
+
 /* This function deletes an element with the given COMPARABLE value
    from hash table starting with the given HASH.  If there is no
    matching element in the hash table, this function does nothing. */
diff --git a/gcc/params.def b/gcc/params.def
index b4a4e4a4190..0db60951413 100644
--- a/gcc/params.def
+++ b/gcc/params.def
@@ -1431,6 +1431,12 @@ DEFPARAM(PARAM_GIMPLE_FE_COMPUTED_HOT_BB_THRESHOLD,
 	 " The parameter is used only in GIMPLE FE.",
 	 0, 0, 0)
 
+DEFPARAM(PARAM_HASH_TABLE_VERIFICATION_LIMIT,
+	 "hash-table-verification-limit",
+	 "The number of elements for which hash table verification is done for "
+	 "each searched element.",
+	 100, 0, 0)
+
 /*
 
 Local variables:
diff --git a/gcc/toplev.c b/gcc/toplev.c
index d300ac2ec89..116be7be395 100644
--- a/gcc/toplev.c
+++ b/gcc/toplev.c
@@ -1799,6 +1799,10 @@ process_options (void)
   optimization_default_node = build_optimization_node (&global_options);
   optimization_current_node = optimization_default_node;
 
+  if (flag_checking >= 2)
+    hash_table_sanitize_eq_limit
+      = PARAM_VALUE (PARAM_HASH_TABLE_VERIFICATION_LIMIT);
+
   /* Please don't change global_options after this point, those changes won't
      be reflected in optimization_{default,current}_node.  */
 }
-- 
2.21.0

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/7/19 10:57 AM, Richard Biener wrote:
> > On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> On 6/1/19 12:06 AM, Jeff Law wrote:
> >>> On 5/22/19 3:13 AM, Martin Liška wrote:
> >>>> On 5/21/19 1:51 PM, Richard Biener wrote:
> >>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>
> >>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>>>>>
> >>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>>>>>                     ^^^^^^
> >>>>>>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Martin
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>   Jakub
> >>>>>>>>>>>>>
> >>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>>>>>
> >>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>>>>>
> >>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>>>>>    bool too_empty_p (unsigned int);
> >>>>>>>>>>>>    void expand ();
> >>>>>>>>>>>>    static bool is_deleted (value_type &v)
> >>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>>>>>      expand ();
> >>>>>>>>>>>>
> >>>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>> +    if (insert == INSERT)
> >>>>>>>>>>>> +      verify (comparable, hash);
> >>>>>>>>>>>> +#endif
> >>>>>>>>>>>>
> >>>>>>>>>>>> +  m_searches++;
> >>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
> >>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>    return &m_entries[index];
> >>>>>>>>>>>>  }
> >>>>>>>>>>>>
> >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>> +
> >>>>>>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>>>>>> +
> >>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>>>>>> +static void
> >>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>> +{
> >>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>> +}
> >>>>>>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>>>>>
> >>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>>>>>> because we've got too many bugs to fix.
> >>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>>>>>> Hi.
> >>>>>>>>>
> >>>>>>>>> I've just added one more PR:
> >>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>>>>>
> >>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>>>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>>>>>
> >>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>>>>>> limits maximal number of checks:
> >>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>>>>>> just thinking about loud.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>>>>>> issue :-)
> >>>>>>>
> >>>>>>> There is at least one PR where we have a table where elements _in_ the
> >>>>>>> table are never compared against each other but always against another
> >>>>>>> object (I guess that's usual even), but the setup is in a way that the
> >>>>>>> comparison function only works with those.  With the patch we verify
> >>>>>>> hashing/comparison for something that is never used.
> >>>>>>>
> >>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>>>>>> at lookup time, using the object from the lookup and verify that against
> >>>>>>> all other elements?
> >>>>>>
> >>>>>> I don't a have problem with that. Apparently this changes fixes
> >>>>>> PR90450 and PR87847.
> >>>>>>
> >>>>>> Changes from previous version:
> >>>>>> - verification happens only when an element is searched (not inserted)
> >>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >>>>>> - new param has been introduced hash-table-verification-limit in order
> >>>>>>   to limit number of elements that are compared within a table
> >>>>>> - verification happens only with flag_checking >= 2
> >>>>>>
> >>>>>> I've been bootstrapping and testing the patch right now.
> >>>>>
> >>>>> Looks like I misremembered the original patch.  The issue isn't
> >>>>> comparing random two elements in the table.
> >>>>>
> >>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> >>>>> without INSERTing.
> >>>>>
> >>>>
> >>>> There's updated version of the patch where I check all find operations
> >>>> (both w/ and w/o insertion).
> >>>>
> >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> >>>> except for:
> >>>>
> >>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> >>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> >>>> during GIMPLE pass: lim
> >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> >>>>     6 | fn1 ()
> >>>>       | ^~~
> >>>> 0x6c5725 hashtab_chk_error
> >>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> >>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> >>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
> >>>> 0xe504ea gather_mem_refs_stmt
> >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> >>>> 0xe504ea analyze_memory_references
> >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> >>>> 0xe504ea tree_ssa_lim
> >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> >>>> 0xe504ea execute
> >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> >>>>
> >>>> Richi: it's after your recent patch.
> >>>>
> >>>> For some reason I don't see PR87847 issue any longer.
> >>>>
> >>>>
> >>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> >>> Don't we still need to deal with the naked fprintf when there's a
> >>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
> >>
> >> Good point, I've just adjusted that.
> >>
> >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> >>
> >> Ready to be installed?
> >
> > Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> > fixing it ...
>
> Yes :D It's been some time and there's no interest in the PR.
>
> >
> > One question - there's unconditional
> >
> > +         if (m_sanitize_eq_and_hash)
> > +           verify (comparable, hash);
> >
> > which will read a global variable and have (possibly not inline) call
> > to verify on a common path even with checking disabled.  So I think
> > we want to compile this checking feature out for !CHECKING_P
> > or at least make the if __builtin_expect (..., 0), ::verify not
> > inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>
> Fixed. May I install the patch? The cselib issue can be solved later..

You missed the second occurance

-  m_searches++;
+  if (m_sanitize_eq_and_hash)
+    verify (comparable, hash);


otherwise OK.

Richard.

> Martin
>
> >
> > Thanks,
> > Richard.
> >
> >> Thanks,
> >> Martin
> >>
> >>>
> >>> jeff
> >>>
> >>
>

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 6/7/19 2:09 PM, Richard Biener wrote:
> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>
>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>
>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>
>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>                     ^^^^^^
>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>   Jakub
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>    bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>    void expand ();
>>>>>>>>>>>>>>    static bool is_deleted (value_type &v)
>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>      expand ();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>    return &m_entries[index];
>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>> +}
>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>
>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>> Hi.
>>>>>>>>>>>
>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>
>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>
>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>> just thinking about loud.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>> issue :-)
>>>>>>>>>
>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>
>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>> all other elements?
>>>>>>>>
>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>> PR90450 and PR87847.
>>>>>>>>
>>>>>>>> Changes from previous version:
>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>   to limit number of elements that are compared within a table
>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>
>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>
>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>> comparing random two elements in the table.
>>>>>>>
>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>> without INSERTing.
>>>>>>>
>>>>>>
>>>>>> There's updated version of the patch where I check all find operations
>>>>>> (both w/ and w/o insertion).
>>>>>>
>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>> except for:
>>>>>>
>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>> during GIMPLE pass: lim
>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>     6 | fn1 ()
>>>>>>       | ^~~
>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>> 0xe504ea analyze_memory_references
>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>> 0xe504ea tree_ssa_lim
>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>> 0xe504ea execute
>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>
>>>>>> Richi: it's after your recent patch.
>>>>>>
>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>
>>>>>>
>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>
>>>> Good point, I've just adjusted that.
>>>>
>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>
>>>> Ready to be installed?
>>>
>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>> fixing it ...
>>
>> Yes :D It's been some time and there's no interest in the PR.
>>
>>>
>>> One question - there's unconditional
>>>
>>> +         if (m_sanitize_eq_and_hash)
>>> +           verify (comparable, hash);
>>>
>>> which will read a global variable and have (possibly not inline) call
>>> to verify on a common path even with checking disabled.  So I think
>>> we want to compile this checking feature out for !CHECKING_P
>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>
>> Fixed. May I install the patch? The cselib issue can be solved later..
> 
> You missed the second occurance
> 
> -  m_searches++;
> +  if (m_sanitize_eq_and_hash)
> +    verify (comparable, hash);

Yep ;) I've just install the patch.

Martin

> 
> 
> otherwise OK.
> 
> Richard.
> 
>> Martin
>>
>>>
>>> Thanks,
>>> Richard.
>>>
>>>> Thanks,
>>>> Martin
>>>>
>>>>>
>>>>> jeff
>>>>>
>>>>
>>

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Sebor]

On 6/7/19 6:13 AM, Martin Liška wrote:
> On 6/7/19 2:09 PM, Richard Biener wrote:
>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>>
>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>
>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>
>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>                      ^^^^^^
>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>    Jakub
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>   gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>   1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>     value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>     value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>     bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>     void expand ();
>>>>>>>>>>>>>>>     static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>     if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>       expand ();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>     value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>     hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>     hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>     return &m_entries[index];
>>>>>>>>>>>>>>>   }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>> Hi.
>>>>>>>>>>>>
>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>
>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>
>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>> issue :-)
>>>>>>>>>>
>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>
>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>> all other elements?
>>>>>>>>>
>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>
>>>>>>>>> Changes from previous version:
>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>    to limit number of elements that are compared within a table
>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>
>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>
>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>> comparing random two elements in the table.
>>>>>>>>
>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>> without INSERTing.
>>>>>>>>
>>>>>>>
>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>> (both w/ and w/o insertion).
>>>>>>>
>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>> except for:
>>>>>>>
>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>> during GIMPLE pass: lim
>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>      6 | fn1 ()
>>>>>>>        | ^~~
>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>       /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>       /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>       /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>> 0xe504ea execute
>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>
>>>>>>> Richi: it's after your recent patch.
>>>>>>>
>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>
>>>>>>>
>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>
>>>>> Good point, I've just adjusted that.
>>>>>
>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>
>>>>> Ready to be installed?
>>>>
>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>> fixing it ...
>>>
>>> Yes :D It's been some time and there's no interest in the PR.
>>>
>>>>
>>>> One question - there's unconditional
>>>>
>>>> +         if (m_sanitize_eq_and_hash)
>>>> +           verify (comparable, hash);
>>>>
>>>> which will read a global variable and have (possibly not inline) call
>>>> to verify on a common path even with checking disabled.  So I think
>>>> we want to compile this checking feature out for !CHECKING_P
>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>
>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>
>> You missed the second occurance
>>
>> -  m_searches++;
>> +  if (m_sanitize_eq_and_hash)
>> +    verify (comparable, hash);
> 
> Yep ;) I've just install the patch.

I think it breaks bootstrap with the error below.  I don't see
the error after reverting it so I'm pretty sure it has something
to do with it.

Martin

g++ -std=gnu++98   -O0 -g3 -DIN_GCC -fPIC    -fno-exceptions -fno-rtti 
-fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings 
-Wcast-qual -Wno-format -Wmissing-format-attribute -Woverloaded-virtual 
-pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings 
-fno-common  -DHAVE_CONFIG_H  -DGENERATOR_FILE -fno-PIE  -no-pie -o 
build/gencondmd \
     build/gencondmd.o 
../build-x86_64-pc-linux-gnu/libiberty/pic/libiberty.a
/usr/bin/ld: build/gencondmd.o: in function `hashtab_chk_error()':
/src/gcc/git-svn/gcc/hash-table.h:1022: undefined reference to 
`fancy_abort(char const*, int, char const*)'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:2865: build/gencondmd] Error 1

Martin

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jason Merrill]

On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/7/19 2:09 PM, Richard Biener wrote:
> > On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> On 6/7/19 10:57 AM, Richard Biener wrote:
> >>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>
> >>>> On 6/1/19 12:06 AM, Jeff Law wrote:
> >>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
> >>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
> >>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>
> >>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>>>>>>>                     ^^^^^^
> >>>>>>>>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Martin
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>   Jakub
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>>>>>>>    bool too_empty_p (unsigned int);
> >>>>>>>>>>>>>>    void expand ();
> >>>>>>>>>>>>>>    static bool is_deleted (value_type &v)
> >>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>>>>>>>      expand ();
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>> +    if (insert == INSERT)
> >>>>>>>>>>>>>> +      verify (comparable, hash);
> >>>>>>>>>>>>>> +#endif
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +  m_searches++;
> >>>>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
> >>>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>    return &m_entries[index];
> >>>>>>>>>>>>>>  }
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>>>>>>>> +static void
> >>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>> +}
> >>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>>>>>>>> because we've got too many bugs to fix.
> >>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>>>>>>>> Hi.
> >>>>>>>>>>>
> >>>>>>>>>>> I've just added one more PR:
> >>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>>>>>>>
> >>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>>>>>>>
> >>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>>>>>>>> limits maximal number of checks:
> >>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>>>>>>>> just thinking about loud.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>>>>>>>> issue :-)
> >>>>>>>>>
> >>>>>>>>> There is at least one PR where we have a table where elements _in_ the
> >>>>>>>>> table are never compared against each other but always against another
> >>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
> >>>>>>>>> comparison function only works with those.  With the patch we verify
> >>>>>>>>> hashing/comparison for something that is never used.
> >>>>>>>>>
> >>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>>>>>>>> at lookup time, using the object from the lookup and verify that against
> >>>>>>>>> all other elements?
> >>>>>>>>
> >>>>>>>> I don't a have problem with that. Apparently this changes fixes
> >>>>>>>> PR90450 and PR87847.
> >>>>>>>>
> >>>>>>>> Changes from previous version:
> >>>>>>>> - verification happens only when an element is searched (not inserted)
> >>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >>>>>>>> - new param has been introduced hash-table-verification-limit in order
> >>>>>>>>   to limit number of elements that are compared within a table
> >>>>>>>> - verification happens only with flag_checking >= 2
> >>>>>>>>
> >>>>>>>> I've been bootstrapping and testing the patch right now.
> >>>>>>>
> >>>>>>> Looks like I misremembered the original patch.  The issue isn't
> >>>>>>> comparing random two elements in the table.
> >>>>>>>
> >>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> >>>>>>> without INSERTing.
> >>>>>>>
> >>>>>>
> >>>>>> There's updated version of the patch where I check all find operations
> >>>>>> (both w/ and w/o insertion).
> >>>>>>
> >>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> >>>>>> except for:
> >>>>>>
> >>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> >>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> >>>>>> during GIMPLE pass: lim
> >>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> >>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> >>>>>>     6 | fn1 ()
> >>>>>>       | ^~~
> >>>>>> 0x6c5725 hashtab_chk_error
> >>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> >>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> >>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> >>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> >>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
> >>>>>> 0xe504ea gather_mem_refs_stmt
> >>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> >>>>>> 0xe504ea analyze_memory_references
> >>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> >>>>>> 0xe504ea tree_ssa_lim
> >>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> >>>>>> 0xe504ea execute
> >>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> >>>>>>
> >>>>>> Richi: it's after your recent patch.
> >>>>>>
> >>>>>> For some reason I don't see PR87847 issue any longer.
> >>>>>>
> >>>>>>
> >>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> >>>>> Don't we still need to deal with the naked fprintf when there's a
> >>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
> >>>>
> >>>> Good point, I've just adjusted that.
> >>>>
> >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> >>>>
> >>>> Ready to be installed?
> >>>
> >>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> >>> fixing it ...
> >>
> >> Yes :D It's been some time and there's no interest in the PR.
> >>
> >>>
> >>> One question - there's unconditional
> >>>
> >>> +         if (m_sanitize_eq_and_hash)
> >>> +           verify (comparable, hash);
> >>>
> >>> which will read a global variable and have (possibly not inline) call
> >>> to verify on a common path even with checking disabled.  So I think
> >>> we want to compile this checking feature out for !CHECKING_P
> >>> or at least make the if __builtin_expect (..., 0), ::verify not
> >>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
> >>
> >> Fixed. May I install the patch? The cselib issue can be solved later..
> >
> > You missed the second occurance
> >
> > -  m_searches++;
> > +  if (m_sanitize_eq_and_hash)
> > +    verify (comparable, hash);
>
> Yep ;) I've just install the patch.

This is breaking my build:

/home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
call to ‘hash_table<hash_map<mem_alloc_d\
escription<mem_usage>::mem_location_hash, mem_usage*,
simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
false, xcallocator>::hash_table(size_t&, bo\
ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
     : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}

Looks like this needs to be updated to pass an argument to the new
sanitize_eq_and_hash parameter.

Jason

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 6/7/19 11:43 PM, Jason Merrill wrote:
> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 6/7/19 2:09 PM, Richard Biener wrote:
>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>>>
>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>
>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>
>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>>                     ^^^^^^
>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>   Jakub
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>>    bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>>    void expand ();
>>>>>>>>>>>>>>>>    static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>>      expand ();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>    return &m_entries[index];
>>>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>>
>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>>> issue :-)
>>>>>>>>>>>
>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>>
>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>>> all other elements?
>>>>>>>>>>
>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>>
>>>>>>>>>> Changes from previous version:
>>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>>   to limit number of elements that are compared within a table
>>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>>
>>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>>
>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>>> comparing random two elements in the table.
>>>>>>>>>
>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>>> without INSERTing.
>>>>>>>>>
>>>>>>>>
>>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>>> (both w/ and w/o insertion).
>>>>>>>>
>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>>> except for:
>>>>>>>>
>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>>> during GIMPLE pass: lim
>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>>     6 | fn1 ()
>>>>>>>>       | ^~~
>>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>>> 0xe504ea execute
>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>>
>>>>>>>> Richi: it's after your recent patch.
>>>>>>>>
>>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>>
>>>>>>>>
>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>>
>>>>>> Good point, I've just adjusted that.
>>>>>>
>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>>
>>>>>> Ready to be installed?
>>>>>
>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>>> fixing it ...
>>>>
>>>> Yes :D It's been some time and there's no interest in the PR.
>>>>
>>>>>
>>>>> One question - there's unconditional
>>>>>
>>>>> +         if (m_sanitize_eq_and_hash)
>>>>> +           verify (comparable, hash);
>>>>>
>>>>> which will read a global variable and have (possibly not inline) call
>>>>> to verify on a common path even with checking disabled.  So I think
>>>>> we want to compile this checking feature out for !CHECKING_P
>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>>
>>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>>
>>> You missed the second occurance
>>>
>>> -  m_searches++;
>>> +  if (m_sanitize_eq_and_hash)
>>> +    verify (comparable, hash);
>>
>> Yep ;) I've just install the patch.
> 
> This is breaking my build:
> 
> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
> call to ‘hash_table<hash_map<mem_alloc_d\
> escription<mem_usage>::mem_location_hash, mem_usage*,
> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
> false, xcallocator>::hash_table(size_t&, bo\
> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
>      : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
> 
> Looks like this needs to be updated to pass an argument to the new
> sanitize_eq_and_hash parameter.
> 
> Jason
> 

Hi.

Sorry for the breakage, I've just fixed that in r272104.

Martin

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jason Merrill]

On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
> On 6/7/19 11:43 PM, Jason Merrill wrote:
> > On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
> >> On 6/7/19 2:09 PM, Richard Biener wrote:
> >>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
> >>>> On 6/7/19 10:57 AM, Richard Biener wrote:
> >>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
> >>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
> >>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
> >>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>
> >>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>>>>>>>>>                     ^^^^^^
> >>>>>>>>>>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Martin
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>   Jakub
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>>>>>>>>>    bool too_empty_p (unsigned int);
> >>>>>>>>>>>>>>>>    void expand ();
> >>>>>>>>>>>>>>>>    static bool is_deleted (value_type &v)
> >>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>>>>>>>>>      expand ();
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>>> +    if (insert == INSERT)
> >>>>>>>>>>>>>>>> +      verify (comparable, hash);
> >>>>>>>>>>>>>>>> +#endif
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +  m_searches++;
> >>>>>>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
> >>>>>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>>    return &m_entries[index];
> >>>>>>>>>>>>>>>>  }
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>>>>>>>>>> +static void
> >>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>>>>>>>>>> because we've got too many bugs to fix.
> >>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>>>>>>>>>> Hi.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I've just added one more PR:
> >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>>>>>>>>>> limits maximal number of checks:
> >>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>>>>>>>>>> just thinking about loud.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>>>>>>>>>> issue :-)
> >>>>>>>>>>>
> >>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
> >>>>>>>>>>> table are never compared against each other but always against another
> >>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
> >>>>>>>>>>> comparison function only works with those.  With the patch we verify
> >>>>>>>>>>> hashing/comparison for something that is never used.
> >>>>>>>>>>>
> >>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
> >>>>>>>>>>> all other elements?
> >>>>>>>>>>
> >>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
> >>>>>>>>>> PR90450 and PR87847.
> >>>>>>>>>>
> >>>>>>>>>> Changes from previous version:
> >>>>>>>>>> - verification happens only when an element is searched (not inserted)
> >>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
> >>>>>>>>>>   to limit number of elements that are compared within a table
> >>>>>>>>>> - verification happens only with flag_checking >= 2
> >>>>>>>>>>
> >>>>>>>>>> I've been bootstrapping and testing the patch right now.
> >>>>>>>>>
> >>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
> >>>>>>>>> comparing random two elements in the table.
> >>>>>>>>>
> >>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> >>>>>>>>> without INSERTing.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> There's updated version of the patch where I check all find operations
> >>>>>>>> (both w/ and w/o insertion).
> >>>>>>>>
> >>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> >>>>>>>> except for:
> >>>>>>>>
> >>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> >>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> >>>>>>>> during GIMPLE pass: lim
> >>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> >>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> >>>>>>>>     6 | fn1 ()
> >>>>>>>>       | ^~~
> >>>>>>>> 0x6c5725 hashtab_chk_error
> >>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> >>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> >>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> >>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> >>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
> >>>>>>>> 0xe504ea gather_mem_refs_stmt
> >>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> >>>>>>>> 0xe504ea analyze_memory_references
> >>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> >>>>>>>> 0xe504ea tree_ssa_lim
> >>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> >>>>>>>> 0xe504ea execute
> >>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> >>>>>>>>
> >>>>>>>> Richi: it's after your recent patch.
> >>>>>>>>
> >>>>>>>> For some reason I don't see PR87847 issue any longer.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> >>>>>>> Don't we still need to deal with the naked fprintf when there's a
> >>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
> >>>>>>
> >>>>>> Good point, I've just adjusted that.
> >>>>>>
> >>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> >>>>>>
> >>>>>> Ready to be installed?
> >>>>>
> >>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> >>>>> fixing it ...
> >>>>
> >>>> Yes :D It's been some time and there's no interest in the PR.
> >>>>
> >>>>>
> >>>>> One question - there's unconditional
> >>>>>
> >>>>> +         if (m_sanitize_eq_and_hash)
> >>>>> +           verify (comparable, hash);
> >>>>>
> >>>>> which will read a global variable and have (possibly not inline) call
> >>>>> to verify on a common path even with checking disabled.  So I think
> >>>>> we want to compile this checking feature out for !CHECKING_P
> >>>>> or at least make the if __builtin_expect (..., 0), ::verify not
> >>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
> >>>>
> >>>> Fixed. May I install the patch? The cselib issue can be solved later..
> >>>
> >>> You missed the second occurance
> >>>
> >>> -  m_searches++;
> >>> +  if (m_sanitize_eq_and_hash)
> >>> +    verify (comparable, hash);
> >>
> >> Yep ;) I've just install the patch.
> >
> > This is breaking my build:
> >
> > /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
> > call to ‘hash_table<hash_map<mem_alloc_d\
> > escription<mem_usage>::mem_location_hash, mem_usage*,
> > simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
> > ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
> > false, xcallocator>::hash_table(size_t&, bo\
> > ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
> >      : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
> >
> > Looks like this needs to be updated to pass an argument to the new
> > sanitize_eq_and_hash parameter.
> >
> > Jason
>
> Sorry for the breakage, I've just fixed that in r272104.

Thanks.  I'm also seeing a massive compile time hit from this:  A
constexpr testcase that I've been looking at went from compiling in 13
seconds to 78 seconds, 6 times as long.  I would expect template-heavy
code to see similar problems when sanitization is enabled for those
hash tables.  Could we keep the parameter low or 0 by default, and
just do occasional sanitize runs with it explicitly enabled?

Jason

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 6/10/19 8:21 PM, Jason Merrill wrote:
> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
>> On 6/7/19 11:43 PM, Jason Merrill wrote:
>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>>>>                     ^^^^^^
>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>   Jakub
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>>>>    bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>>>>    void expand ();
>>>>>>>>>>>>>>>>>>    static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>>>>      expand ();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>    return &m_entries[index];
>>>>>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>>>>> issue :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>>>>> all other elements?
>>>>>>>>>>>>
>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>>>>
>>>>>>>>>>>> Changes from previous version:
>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>>>>   to limit number of elements that are compared within a table
>>>>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>>>>
>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>>>>
>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>>>>> comparing random two elements in the table.
>>>>>>>>>>>
>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>>>>> without INSERTing.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>>>>> (both w/ and w/o insertion).
>>>>>>>>>>
>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>>>>> except for:
>>>>>>>>>>
>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>>>>> during GIMPLE pass: lim
>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>>>>     6 | fn1 ()
>>>>>>>>>>       | ^~~
>>>>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>>>>> 0xe504ea execute
>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>>>>
>>>>>>>>>> Richi: it's after your recent patch.
>>>>>>>>>>
>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>>>>
>>>>>>>> Good point, I've just adjusted that.
>>>>>>>>
>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>>>>
>>>>>>>> Ready to be installed?
>>>>>>>
>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>>>>> fixing it ...
>>>>>>
>>>>>> Yes :D It's been some time and there's no interest in the PR.
>>>>>>
>>>>>>>
>>>>>>> One question - there's unconditional
>>>>>>>
>>>>>>> +         if (m_sanitize_eq_and_hash)
>>>>>>> +           verify (comparable, hash);
>>>>>>>
>>>>>>> which will read a global variable and have (possibly not inline) call
>>>>>>> to verify on a common path even with checking disabled.  So I think
>>>>>>> we want to compile this checking feature out for !CHECKING_P
>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>>>>
>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>>>>
>>>>> You missed the second occurance
>>>>>
>>>>> -  m_searches++;
>>>>> +  if (m_sanitize_eq_and_hash)
>>>>> +    verify (comparable, hash);
>>>>
>>>> Yep ;) I've just install the patch.
>>>
>>> This is breaking my build:
>>>
>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
>>> call to ‘hash_table<hash_map<mem_alloc_d\
>>> escription<mem_usage>::mem_location_hash, mem_usage*,
>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
>>> false, xcallocator>::hash_table(size_t&, bo\
>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
>>>      : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
>>>
>>> Looks like this needs to be updated to pass an argument to the new
>>> sanitize_eq_and_hash parameter.
>>>
>>> Jason
>>
>> Sorry for the breakage, I've just fixed that in r272104.
> 
> Thanks.  I'm also seeing a massive compile time hit from this:  A
> constexpr testcase that I've been looking at went from compiling in 13
> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
> code to see similar problems when sanitization is enabled for those
> hash tables.  Could we keep the parameter low or 0 by default, and
> just do occasional sanitize runs with it explicitly enabled?

Makes sense to me. Can you please provide a test-case which I can measure?

Martin

> 
> Jason
> 

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jason Merrill]

On 6/11/19 3:41 AM, Martin Liška wrote:
> On 6/10/19 8:21 PM, Jason Merrill wrote:
>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
>>> On 6/7/19 11:43 PM, Jason Merrill wrote:
>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
>>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>>>>>                      ^^^^^^
>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>>    Jakub
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>  From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>   gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>>>>>   1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>     value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>>>>>     value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>>>>>     bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>>>>>     void expand ();
>>>>>>>>>>>>>>>>>>>     static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>     if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>>>>>       expand ();
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>>>>>     value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>>>>>     hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>     hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>     return &m_entries[index];
>>>>>>>>>>>>>>>>>>>   }
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>>>>>> issue :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>>>>>> all other elements?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes from previous version:
>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>>>>>    to limit number of elements that are compared within a table
>>>>>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>>>>>
>>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>>>>>> comparing random two elements in the table.
>>>>>>>>>>>>
>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>>>>>> without INSERTing.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>>>>>> (both w/ and w/o insertion).
>>>>>>>>>>>
>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>>>>>> except for:
>>>>>>>>>>>
>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>>>>>> during GIMPLE pass: lim
>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>>>>>      6 | fn1 ()
>>>>>>>>>>>        | ^~~
>>>>>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>>>>>> 0xe504ea execute
>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>>>>>
>>>>>>>>>>> Richi: it's after your recent patch.
>>>>>>>>>>>
>>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>>>>>
>>>>>>>>> Good point, I've just adjusted that.
>>>>>>>>>
>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>>>>>
>>>>>>>>> Ready to be installed?
>>>>>>>>
>>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>>>>>> fixing it ...
>>>>>>>
>>>>>>> Yes :D It's been some time and there's no interest in the PR.
>>>>>>>
>>>>>>>>
>>>>>>>> One question - there's unconditional
>>>>>>>>
>>>>>>>> +         if (m_sanitize_eq_and_hash)
>>>>>>>> +           verify (comparable, hash);
>>>>>>>>
>>>>>>>> which will read a global variable and have (possibly not inline) call
>>>>>>>> to verify on a common path even with checking disabled.  So I think
>>>>>>>> we want to compile this checking feature out for !CHECKING_P
>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>>>>>
>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>>>>>
>>>>>> You missed the second occurance
>>>>>>
>>>>>> -  m_searches++;
>>>>>> +  if (m_sanitize_eq_and_hash)
>>>>>> +    verify (comparable, hash);
>>>>>
>>>>> Yep ;) I've just install the patch.
>>>>
>>>> This is breaking my build:
>>>>
>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
>>>> call to ‘hash_table<hash_map<mem_alloc_d\
>>>> escription<mem_usage>::mem_location_hash, mem_usage*,
>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
>>>> false, xcallocator>::hash_table(size_t&, bo\
>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
>>>>       : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
>>>>
>>>> Looks like this needs to be updated to pass an argument to the new
>>>> sanitize_eq_and_hash parameter.
>>>>
>>>> Jason
>>>
>>> Sorry for the breakage, I've just fixed that in r272104.
>>
>> Thanks.  I'm also seeing a massive compile time hit from this:  A
>> constexpr testcase that I've been looking at went from compiling in 13
>> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
>> code to see similar problems when sanitization is enabled for those
>> hash tables.  Could we keep the parameter low or 0 by default, and
>> just do occasional sanitize runs with it explicitly enabled?
> 
> Makes sense to me. Can you please provide a test-case which I can measure?

This is the one I've been looking at:

   struct Int {
     constexpr Int(int v): v(v) {}
     constexpr Int& operator+=(Int b) { this->v += b.v; return *this; }
     constexpr Int& operator++() { ++this->v; return *this; }
   private:
     friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; }
     int v;
   };
   constexpr int f(int n) {
     Int i = {0};
     Int k = {0};
     k = 0;
     for (; k<10000; ++k) {
       i += k;
     }
     return n;
   }

   template<int N> struct S {
     static constexpr int sm = S<N-1>::sm+f(N);
   };
   template<> struct S<0> {
     static constexpr int sm = 0;
   };
   constexpr int r = S<20>::sm;

Jason

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 6/11/19 2:27 PM, Jason Merrill wrote:
> On 6/11/19 3:41 AM, Martin Liška wrote:
>> On 6/10/19 8:21 PM, Jason Merrill wrote:
>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
>>>> On 6/7/19 11:43 PM, Jason Merrill wrote:
>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>>>>>>                      ^^^^^^
>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>>>    Jakub
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>  From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>   gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>>>>>>   1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>>>>>>     value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>>>>>>     bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>>>>>>     void expand ();
>>>>>>>>>>>>>>>>>>>>     static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>     if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>>>>>>       expand ();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>>>>>>     value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>>>>>>     hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>>     hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>     return &m_entries[index];
>>>>>>>>>>>>>>>>>>>>   }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>>>>>>> issue :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>>>>>>> all other elements?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Changes from previous version:
>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>>>>>>    to limit number of elements that are compared within a table
>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>>>>>>> comparing random two elements in the table.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>>>>>>> without INSERTing.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>>>>>>> (both w/ and w/o insertion).
>>>>>>>>>>>>
>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>>>>>>> except for:
>>>>>>>>>>>>
>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>>>>>>> during GIMPLE pass: lim
>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>>>>>>      6 | fn1 ()
>>>>>>>>>>>>        | ^~~
>>>>>>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>>>>>>> 0xe504ea execute
>>>>>>>>>>>>       /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>>>>>>
>>>>>>>>>>>> Richi: it's after your recent patch.
>>>>>>>>>>>>
>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>>>>>>
>>>>>>>>>> Good point, I've just adjusted that.
>>>>>>>>>>
>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>>>>>>
>>>>>>>>>> Ready to be installed?
>>>>>>>>>
>>>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>>>>>>> fixing it ...
>>>>>>>>
>>>>>>>> Yes :D It's been some time and there's no interest in the PR.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> One question - there's unconditional
>>>>>>>>>
>>>>>>>>> +         if (m_sanitize_eq_and_hash)
>>>>>>>>> +           verify (comparable, hash);
>>>>>>>>>
>>>>>>>>> which will read a global variable and have (possibly not inline) call
>>>>>>>>> to verify on a common path even with checking disabled.  So I think
>>>>>>>>> we want to compile this checking feature out for !CHECKING_P
>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>>>>>>
>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>>>>>>
>>>>>>> You missed the second occurance
>>>>>>>
>>>>>>> -  m_searches++;
>>>>>>> +  if (m_sanitize_eq_and_hash)
>>>>>>> +    verify (comparable, hash);
>>>>>>
>>>>>> Yep ;) I've just install the patch.
>>>>>
>>>>> This is breaking my build:
>>>>>
>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
>>>>> call to ‘hash_table<hash_map<mem_alloc_d\
>>>>> escription<mem_usage>::mem_location_hash, mem_usage*,
>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
>>>>> false, xcallocator>::hash_table(size_t&, bo\
>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
>>>>>       : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
>>>>>
>>>>> Looks like this needs to be updated to pass an argument to the new
>>>>> sanitize_eq_and_hash parameter.
>>>>>
>>>>> Jason
>>>>
>>>> Sorry for the breakage, I've just fixed that in r272104.
>>>
>>> Thanks.  I'm also seeing a massive compile time hit from this:  A
>>> constexpr testcase that I've been looking at went from compiling in 13
>>> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
>>> code to see similar problems when sanitization is enabled for those
>>> hash tables.  Could we keep the parameter low or 0 by default, and
>>> just do occasional sanitize runs with it explicitly enabled?
>>
>> Makes sense to me. Can you please provide a test-case which I can measure?
> 
> This is the one I've been looking at:
> 
>   struct Int {
>     constexpr Int(int v): v(v) {}
>     constexpr Int& operator+=(Int b) { this->v += b.v; return *this; }
>     constexpr Int& operator++() { ++this->v; return *this; }
>   private:
>     friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; }
>     int v;
>   };
>   constexpr int f(int n) {
>     Int i = {0};
>     Int k = {0};
>     k = 0;
>     for (; k<10000; ++k) {
>       i += k;
>     }
>     return n;
>   }
> 
>   template<int N> struct S {
>     static constexpr int sm = S<N-1>::sm+f(N);
>   };
>   template<> struct S<0> {
>     static constexpr int sm = 0;
>   };
>   constexpr int r = S<20>::sm;
> 
> Jason

For the test-case provided I see:

$ time g++ time.cc -c --param hash-table-verification-limit=100

real	0m1.855s
user	0m1.829s
sys	0m0.025s

$ time g++ time.cc -c --param hash-table-verification-limit=0

real	0m1.275s
user	0m1.219s
sys	0m0.052s

$ time g++-9 time.cc -c

real	0m0.939s
user	0m0.827s
sys	0m0.109s

So it's slower, but I can't confirm the huge slowdown you see.
Is it due to r272144?

Martin

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Jason Merrill]

On 6/11/19 9:16 AM, Martin Liška wrote:
> On 6/11/19 2:27 PM, Jason Merrill wrote:
>> On 6/11/19 3:41 AM, Martin Liška wrote:
>>> On 6/10/19 8:21 PM, Jason Merrill wrote:
>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote:
>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>>>>>>>                       ^^^^^^
>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>>>>     Jakub
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>   From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>    gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>>>>>>>    1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>      value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>>>>>>>      value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>>>>>>>      bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>>>>>>>      void expand ();
>>>>>>>>>>>>>>>>>>>>>      static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>>      if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>>>>>>>        expand ();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>>>>>>>      value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>>>>>>>      hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>>>      hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>>      return &m_entries[index];
>>>>>>>>>>>>>>>>>>>>>    }
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>>>>>>>> issue :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>>>>>>>> all other elements?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Changes from previous version:
>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>>>>>>>     to limit number of elements that are compared within a table
>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>>>>>>>> comparing random two elements in the table.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>>>>>>>> without INSERTing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>>>>>>>> (both w/ and w/o insertion).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>>>>>>>> except for:
>>>>>>>>>>>>>
>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>>>>>>>> during GIMPLE pass: lim
>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>>>>>>>       6 | fn1 ()
>>>>>>>>>>>>>         | ^~~
>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>>>>>>>> 0xe504ea execute
>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>>>>>>>
>>>>>>>>>>>>> Richi: it's after your recent patch.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>>>>>>>
>>>>>>>>>>> Good point, I've just adjusted that.
>>>>>>>>>>>
>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>>>>>>>
>>>>>>>>>>> Ready to be installed?
>>>>>>>>>>
>>>>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>>>>>>>> fixing it ...
>>>>>>>>>
>>>>>>>>> Yes :D It's been some time and there's no interest in the PR.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> One question - there's unconditional
>>>>>>>>>>
>>>>>>>>>> +         if (m_sanitize_eq_and_hash)
>>>>>>>>>> +           verify (comparable, hash);
>>>>>>>>>>
>>>>>>>>>> which will read a global variable and have (possibly not inline) call
>>>>>>>>>> to verify on a common path even with checking disabled.  So I think
>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P
>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>>>>>>>
>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>>>>>>>
>>>>>>>> You missed the second occurance
>>>>>>>>
>>>>>>>> -  m_searches++;
>>>>>>>> +  if (m_sanitize_eq_and_hash)
>>>>>>>> +    verify (comparable, hash);
>>>>>>>
>>>>>>> Yep ;) I've just install the patch.
>>>>>>
>>>>>> This is breaking my build:
>>>>>>
>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
>>>>>> call to ‘hash_table<hash_map<mem_alloc_d\
>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*,
>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
>>>>>> false, xcallocator>::hash_table(size_t&, bo\
>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
>>>>>>        : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
>>>>>>
>>>>>> Looks like this needs to be updated to pass an argument to the new
>>>>>> sanitize_eq_and_hash parameter.
>>>>>>
>>>>>> Jason
>>>>>
>>>>> Sorry for the breakage, I've just fixed that in r272104.
>>>>
>>>> Thanks.  I'm also seeing a massive compile time hit from this:  A
>>>> constexpr testcase that I've been looking at went from compiling in 13
>>>> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
>>>> code to see similar problems when sanitization is enabled for those
>>>> hash tables.  Could we keep the parameter low or 0 by default, and
>>>> just do occasional sanitize runs with it explicitly enabled?
>>>
>>> Makes sense to me. Can you please provide a test-case which I can measure?
>>
>> This is the one I've been looking at:
>>
>>    struct Int {
>>      constexpr Int(int v): v(v) {}
>>      constexpr Int& operator+=(Int b) { this->v += b.v; return *this; }
>>      constexpr Int& operator++() { ++this->v; return *this; }
>>    private:
>>      friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; }
>>      int v;
>>    };
>>    constexpr int f(int n) {
>>      Int i = {0};
>>      Int k = {0};
>>      k = 0;
>>      for (; k<10000; ++k) {
>>        i += k;
>>      }
>>      return n;
>>    }
>>
>>    template<int N> struct S {
>>      static constexpr int sm = S<N-1>::sm+f(N);
>>    };
>>    template<> struct S<0> {
>>      static constexpr int sm = 0;
>>    };
>>    constexpr int r = S<20>::sm;
>>
>> Jason
> 
> For the test-case provided I see:
> 
> $ time g++ time.cc -c --param hash-table-verification-limit=100
> 
> real	0m1.855s
> user	0m1.829s
> sys	0m0.025s
> 
> $ time g++ time.cc -c --param hash-table-verification-limit=0
> 
> real	0m1.275s
> user	0m1.219s
> sys	0m0.052s
> 
> $ time g++-9 time.cc -c
> 
> real	0m0.939s
> user	0m0.827s
> sys	0m0.109s
> 
> So it's slower, but I can't confirm the huge slowdown you see.
> Is it due to r272144?

Hmm, I wonder if this is because of the 
--enable-gather-detailed-mem-stats hash tables.

Jason

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote:
>
> On 6/11/19 9:16 AM, Martin Liška wrote:
> > On 6/11/19 2:27 PM, Jason Merrill wrote:
> >> On 6/11/19 3:41 AM, Martin Liška wrote:
> >>> On 6/10/19 8:21 PM, Jason Merrill wrote:
> >>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
> >>>>> On 6/7/19 11:43 PM, Jason Merrill wrote:
> >>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
> >>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
> >>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
> >>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
> >>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
> >>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>>>>>>>>>>>>>>                       ^^^^^^
> >>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Martin
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>>>>>>     Jakub
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>   From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>    gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>>>>>>>>>>>>    1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>      value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>>>>>>>>>>>>>>      value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>>>>>>>>>>>>>>      bool too_empty_p (unsigned int);
> >>>>>>>>>>>>>>>>>>>>>      void expand ();
> >>>>>>>>>>>>>>>>>>>>>      static bool is_deleted (value_type &v)
> >>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>>>>>>>      if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>>>>>>>>>>>>>>        expand ();
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
> >>>>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
> >>>>>>>>>>>>>>>>>>>>> +#endif
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> +  m_searches++;
> >>>>>>>>>>>>>>>>>>>>>      value_type *first_deleted_slot = NULL;
> >>>>>>>>>>>>>>>>>>>>>      hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>>>>>>>>      hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>>>>>>>      return &m_entries[index];
> >>>>>>>>>>>>>>>>>>>>>    }
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>>>>>>>>>>>>>>> +static void
> >>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
> >>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>>>>>>>>>>>>>>> Hi.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I've just added one more PR:
> >>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>>>>>>>>>>>>>>> limits maximal number of checks:
> >>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>>>>>>>>>>>>>>> just thinking about loud.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>>>>>>>>>>>>>>> issue :-)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
> >>>>>>>>>>>>>>>> table are never compared against each other but always against another
> >>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
> >>>>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
> >>>>>>>>>>>>>>>> hashing/comparison for something that is never used.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
> >>>>>>>>>>>>>>>> all other elements?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
> >>>>>>>>>>>>>>> PR90450 and PR87847.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Changes from previous version:
> >>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
> >>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
> >>>>>>>>>>>>>>>     to limit number of elements that are compared within a table
> >>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
> >>>>>>>>>>>>>> comparing random two elements in the table.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> >>>>>>>>>>>>>> without INSERTing.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> There's updated version of the patch where I check all find operations
> >>>>>>>>>>>>> (both w/ and w/o insertion).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> >>>>>>>>>>>>> except for:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> >>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> >>>>>>>>>>>>> during GIMPLE pass: lim
> >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> >>>>>>>>>>>>>       6 | fn1 ()
> >>>>>>>>>>>>>         | ^~~
> >>>>>>>>>>>>> 0x6c5725 hashtab_chk_error
> >>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> >>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> >>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> >>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> >>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:960
> >>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
> >>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> >>>>>>>>>>>>> 0xe504ea analyze_memory_references
> >>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> >>>>>>>>>>>>> 0xe504ea tree_ssa_lim
> >>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> >>>>>>>>>>>>> 0xe504ea execute
> >>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Richi: it's after your recent patch.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> >>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
> >>>>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
> >>>>>>>>>>>
> >>>>>>>>>>> Good point, I've just adjusted that.
> >>>>>>>>>>>
> >>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> >>>>>>>>>>>
> >>>>>>>>>>> Ready to be installed?
> >>>>>>>>>>
> >>>>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> >>>>>>>>>> fixing it ...
> >>>>>>>>>
> >>>>>>>>> Yes :D It's been some time and there's no interest in the PR.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> One question - there's unconditional
> >>>>>>>>>>
> >>>>>>>>>> +         if (m_sanitize_eq_and_hash)
> >>>>>>>>>> +           verify (comparable, hash);
> >>>>>>>>>>
> >>>>>>>>>> which will read a global variable and have (possibly not inline) call
> >>>>>>>>>> to verify on a common path even with checking disabled.  So I think
> >>>>>>>>>> we want to compile this checking feature out for !CHECKING_P
> >>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
> >>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
> >>>>>>>>>
> >>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
> >>>>>>>>
> >>>>>>>> You missed the second occurance
> >>>>>>>>
> >>>>>>>> -  m_searches++;
> >>>>>>>> +  if (m_sanitize_eq_and_hash)
> >>>>>>>> +    verify (comparable, hash);
> >>>>>>>
> >>>>>>> Yep ;) I've just install the patch.
> >>>>>>
> >>>>>> This is breaking my build:
> >>>>>>
> >>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
> >>>>>> call to ‘hash_table<hash_map<mem_alloc_d\
> >>>>>> escription<mem_usage>::mem_location_hash, mem_usage*,
> >>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
> >>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
> >>>>>> false, xcallocator>::hash_table(size_t&, bo\
> >>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
> >>>>>>        : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
> >>>>>>
> >>>>>> Looks like this needs to be updated to pass an argument to the new
> >>>>>> sanitize_eq_and_hash parameter.
> >>>>>>
> >>>>>> Jason
> >>>>>
> >>>>> Sorry for the breakage, I've just fixed that in r272104.
> >>>>
> >>>> Thanks.  I'm also seeing a massive compile time hit from this:  A
> >>>> constexpr testcase that I've been looking at went from compiling in 13
> >>>> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
> >>>> code to see similar problems when sanitization is enabled for those
> >>>> hash tables.  Could we keep the parameter low or 0 by default, and
> >>>> just do occasional sanitize runs with it explicitly enabled?
> >>>
> >>> Makes sense to me. Can you please provide a test-case which I can measure?
> >>
> >> This is the one I've been looking at:
> >>
> >>    struct Int {
> >>      constexpr Int(int v): v(v) {}
> >>      constexpr Int& operator+=(Int b) { this->v += b.v; return *this; }
> >>      constexpr Int& operator++() { ++this->v; return *this; }
> >>    private:
> >>      friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; }
> >>      int v;
> >>    };
> >>    constexpr int f(int n) {
> >>      Int i = {0};
> >>      Int k = {0};
> >>      k = 0;
> >>      for (; k<10000; ++k) {
> >>        i += k;
> >>      }
> >>      return n;
> >>    }
> >>
> >>    template<int N> struct S {
> >>      static constexpr int sm = S<N-1>::sm+f(N);
> >>    };
> >>    template<> struct S<0> {
> >>      static constexpr int sm = 0;
> >>    };
> >>    constexpr int r = S<20>::sm;
> >>
> >> Jason
> >
> > For the test-case provided I see:
> >
> > $ time g++ time.cc -c --param hash-table-verification-limit=100
> >
> > real  0m1.855s
> > user  0m1.829s
> > sys   0m0.025s
> >
> > $ time g++ time.cc -c --param hash-table-verification-limit=0
> >
> > real  0m1.275s
> > user  0m1.219s
> > sys   0m0.052s
> >
> > $ time g++-9 time.cc -c
> >
> > real  0m0.939s
> > user  0m0.827s
> > sys   0m0.109s
> >
> > So it's slower, but I can't confirm the huge slowdown you see.
> > Is it due to r272144?
>
> Hmm, I wonder if this is because of the
> --enable-gather-detailed-mem-stats hash tables.

I wonder if we can reduce the overhead by making
hash-tables/maps using predefined traits not perform
the checking?  Thus make [the default] whether to check or not
to check part of the traits?  Surely the basic pointer-hash/map
stuff is OK and needs no extra checking.

Richard.

> Jason

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 6/12/19 9:59 AM, Richard Biener wrote:
> On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote:
>>
>> On 6/11/19 9:16 AM, Martin Liška wrote:
>>> On 6/11/19 2:27 PM, Jason Merrill wrote:
>>>> On 6/11/19 3:41 AM, Martin Liška wrote:
>>>>> On 6/10/19 8:21 PM, Jason Merrill wrote:
>>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
>>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote:
>>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
>>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>>>>>>>>>                       ^^^^^^
>>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>>>>>>     Jakub
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>   From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>    gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>>>>>>>>>    1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>      value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>>>>>>>>>      value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>>>>>>>>>      bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>>>>>>>>>      void expand ();
>>>>>>>>>>>>>>>>>>>>>>>      static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>>>>      if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>>>>>>>>>        expand ();
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>>>>>>>>>      value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>>>>>>>>>      hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>>>>>      hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>>>>      return &m_entries[index];
>>>>>>>>>>>>>>>>>>>>>>>    }
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>>>>>>>>>> issue :-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>>>>>>>>>> all other elements?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Changes from previous version:
>>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>>>>>>>>>     to limit number of elements that are compared within a table
>>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>>>>>>>>>> comparing random two elements in the table.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>>>>>>>>>> without INSERTing.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>>>>>>>>>> (both w/ and w/o insertion).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>>>>>>>>>> except for:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>>>>>>>>>> during GIMPLE pass: lim
>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>>>>>>>>>       6 | fn1 ()
>>>>>>>>>>>>>>>         | ^~~
>>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>>>>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>>>>>>>>>> 0xe504ea execute
>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Richi: it's after your recent patch.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Good point, I've just adjusted that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ready to be installed?
>>>>>>>>>>>>
>>>>>>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>>>>>>>>>> fixing it ...
>>>>>>>>>>>
>>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> One question - there's unconditional
>>>>>>>>>>>>
>>>>>>>>>>>> +         if (m_sanitize_eq_and_hash)
>>>>>>>>>>>> +           verify (comparable, hash);
>>>>>>>>>>>>
>>>>>>>>>>>> which will read a global variable and have (possibly not inline) call
>>>>>>>>>>>> to verify on a common path even with checking disabled.  So I think
>>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P
>>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>>>>>>>>>
>>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>>>>>>>>>
>>>>>>>>>> You missed the second occurance
>>>>>>>>>>
>>>>>>>>>> -  m_searches++;
>>>>>>>>>> +  if (m_sanitize_eq_and_hash)
>>>>>>>>>> +    verify (comparable, hash);
>>>>>>>>>
>>>>>>>>> Yep ;) I've just install the patch.
>>>>>>>>
>>>>>>>> This is breaking my build:
>>>>>>>>
>>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
>>>>>>>> call to ‘hash_table<hash_map<mem_alloc_d\
>>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*,
>>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
>>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
>>>>>>>> false, xcallocator>::hash_table(size_t&, bo\
>>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
>>>>>>>>        : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
>>>>>>>>
>>>>>>>> Looks like this needs to be updated to pass an argument to the new
>>>>>>>> sanitize_eq_and_hash parameter.
>>>>>>>>
>>>>>>>> Jason
>>>>>>>
>>>>>>> Sorry for the breakage, I've just fixed that in r272104.
>>>>>>
>>>>>> Thanks.  I'm also seeing a massive compile time hit from this:  A
>>>>>> constexpr testcase that I've been looking at went from compiling in 13
>>>>>> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
>>>>>> code to see similar problems when sanitization is enabled for those
>>>>>> hash tables.  Could we keep the parameter low or 0 by default, and
>>>>>> just do occasional sanitize runs with it explicitly enabled?
>>>>>
>>>>> Makes sense to me. Can you please provide a test-case which I can measure?
>>>>
>>>> This is the one I've been looking at:
>>>>
>>>>    struct Int {
>>>>      constexpr Int(int v): v(v) {}
>>>>      constexpr Int& operator+=(Int b) { this->v += b.v; return *this; }
>>>>      constexpr Int& operator++() { ++this->v; return *this; }
>>>>    private:
>>>>      friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; }
>>>>      int v;
>>>>    };
>>>>    constexpr int f(int n) {
>>>>      Int i = {0};
>>>>      Int k = {0};
>>>>      k = 0;
>>>>      for (; k<10000; ++k) {
>>>>        i += k;
>>>>      }
>>>>      return n;
>>>>    }
>>>>
>>>>    template<int N> struct S {
>>>>      static constexpr int sm = S<N-1>::sm+f(N);
>>>>    };
>>>>    template<> struct S<0> {
>>>>      static constexpr int sm = 0;
>>>>    };
>>>>    constexpr int r = S<20>::sm;
>>>>
>>>> Jason
>>>
>>> For the test-case provided I see:
>>>
>>> $ time g++ time.cc -c --param hash-table-verification-limit=100
>>>
>>> real  0m1.855s
>>> user  0m1.829s
>>> sys   0m0.025s
>>>
>>> $ time g++ time.cc -c --param hash-table-verification-limit=0
>>>
>>> real  0m1.275s
>>> user  0m1.219s
>>> sys   0m0.052s
>>>
>>> $ time g++-9 time.cc -c
>>>
>>> real  0m0.939s
>>> user  0m0.827s
>>> sys   0m0.109s
>>>
>>> So it's slower, but I can't confirm the huge slowdown you see.
>>> Is it due to r272144?
>>
>> Hmm, I wonder if this is because of the
>> --enable-gather-detailed-mem-stats hash tables.
> 
> I wonder if we can reduce the overhead by making
> hash-tables/maps using predefined traits not perform
> the checking?  Thus make [the default] whether to check or not
> to check part of the traits?  Surely the basic pointer-hash/map
> stuff is OK and needs no extra checking.

Interesting idea! I can prepare a patch. Right now I'm testing a patch
that removes sanitization for hash-tables (and maps) that track memory
allocations.

Martin

> 
> Richard.
> 
>> Jason

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 16616 bytes --]

On 6/12/19 10:02 AM, Martin Liška wrote:
> On 6/12/19 9:59 AM, Richard Biener wrote:
>> On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote:
>>>
>>> On 6/11/19 9:16 AM, Martin Liška wrote:
>>>> On 6/11/19 2:27 PM, Jason Merrill wrote:
>>>>> On 6/11/19 3:41 AM, Martin Liška wrote:
>>>>>> On 6/10/19 8:21 PM, Jason Merrill wrote:
>>>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote:
>>>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
>>>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>>>>>>>>>>                       ^^^^^^
>>>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>>>>>>>     Jakub
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>   From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>    gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>>>>>>>>>>    1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>      value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>>>>>>>>>>      value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>>>>>>>>>>      bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>>>>>>>>>>      void expand ();
>>>>>>>>>>>>>>>>>>>>>>>>      static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>>>>>      if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>>>>>>>>>>        expand ();
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>>>>>>>>>>      value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>>>>>>>>>>      hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>>>>>>      hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>>>>>      return &m_entries[index];
>>>>>>>>>>>>>>>>>>>>>>>>    }
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>>>>>>>>>>> issue :-)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>>>>>>>>>>> all other elements?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>>>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Changes from previous version:
>>>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>>>>>>>>>>     to limit number of elements that are compared within a table
>>>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>>>>>>>>>>> comparing random two elements in the table.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>>>>>>>>>>> without INSERTing.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>>>>>>>>>>> (both w/ and w/o insertion).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>>>>>>>>>>> except for:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>>>>>>>>>>> during GIMPLE pass: lim
>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>>>>>>>>>>       6 | fn1 ()
>>>>>>>>>>>>>>>>         | ^~~
>>>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>>>>>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>>>>>>>>>>> 0xe504ea execute
>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Richi: it's after your recent patch.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>>>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Good point, I've just adjusted that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ready to be installed?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>>>>>>>>>>> fixing it ...
>>>>>>>>>>>>
>>>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> One question - there's unconditional
>>>>>>>>>>>>>
>>>>>>>>>>>>> +         if (m_sanitize_eq_and_hash)
>>>>>>>>>>>>> +           verify (comparable, hash);
>>>>>>>>>>>>>
>>>>>>>>>>>>> which will read a global variable and have (possibly not inline) call
>>>>>>>>>>>>> to verify on a common path even with checking disabled.  So I think
>>>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P
>>>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>>>>>>>>>>
>>>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>>>>>>>>>>
>>>>>>>>>>> You missed the second occurance
>>>>>>>>>>>
>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>> +  if (m_sanitize_eq_and_hash)
>>>>>>>>>>> +    verify (comparable, hash);
>>>>>>>>>>
>>>>>>>>>> Yep ;) I've just install the patch.
>>>>>>>>>
>>>>>>>>> This is breaking my build:
>>>>>>>>>
>>>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
>>>>>>>>> call to ‘hash_table<hash_map<mem_alloc_d\
>>>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*,
>>>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
>>>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
>>>>>>>>> false, xcallocator>::hash_table(size_t&, bo\
>>>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
>>>>>>>>>        : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
>>>>>>>>>
>>>>>>>>> Looks like this needs to be updated to pass an argument to the new
>>>>>>>>> sanitize_eq_and_hash parameter.
>>>>>>>>>
>>>>>>>>> Jason
>>>>>>>>
>>>>>>>> Sorry for the breakage, I've just fixed that in r272104.
>>>>>>>
>>>>>>> Thanks.  I'm also seeing a massive compile time hit from this:  A
>>>>>>> constexpr testcase that I've been looking at went from compiling in 13
>>>>>>> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
>>>>>>> code to see similar problems when sanitization is enabled for those
>>>>>>> hash tables.  Could we keep the parameter low or 0 by default, and
>>>>>>> just do occasional sanitize runs with it explicitly enabled?
>>>>>>
>>>>>> Makes sense to me. Can you please provide a test-case which I can measure?
>>>>>
>>>>> This is the one I've been looking at:
>>>>>
>>>>>    struct Int {
>>>>>      constexpr Int(int v): v(v) {}
>>>>>      constexpr Int& operator+=(Int b) { this->v += b.v; return *this; }
>>>>>      constexpr Int& operator++() { ++this->v; return *this; }
>>>>>    private:
>>>>>      friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; }
>>>>>      int v;
>>>>>    };
>>>>>    constexpr int f(int n) {
>>>>>      Int i = {0};
>>>>>      Int k = {0};
>>>>>      k = 0;
>>>>>      for (; k<10000; ++k) {
>>>>>        i += k;
>>>>>      }
>>>>>      return n;
>>>>>    }
>>>>>
>>>>>    template<int N> struct S {
>>>>>      static constexpr int sm = S<N-1>::sm+f(N);
>>>>>    };
>>>>>    template<> struct S<0> {
>>>>>      static constexpr int sm = 0;
>>>>>    };
>>>>>    constexpr int r = S<20>::sm;
>>>>>
>>>>> Jason
>>>>
>>>> For the test-case provided I see:
>>>>
>>>> $ time g++ time.cc -c --param hash-table-verification-limit=100
>>>>
>>>> real  0m1.855s
>>>> user  0m1.829s
>>>> sys   0m0.025s
>>>>
>>>> $ time g++ time.cc -c --param hash-table-verification-limit=0
>>>>
>>>> real  0m1.275s
>>>> user  0m1.219s
>>>> sys   0m0.052s
>>>>
>>>> $ time g++-9 time.cc -c
>>>>
>>>> real  0m0.939s
>>>> user  0m0.827s
>>>> sys   0m0.109s
>>>>
>>>> So it's slower, but I can't confirm the huge slowdown you see.
>>>> Is it due to r272144?
>>>
>>> Hmm, I wonder if this is because of the
>>> --enable-gather-detailed-mem-stats hash tables.
>>
>> I wonder if we can reduce the overhead by making
>> hash-tables/maps using predefined traits not perform
>> the checking?  Thus make [the default] whether to check or not
>> to check part of the traits?  Surely the basic pointer-hash/map
>> stuff is OK and needs no extra checking.
> 
> Interesting idea! I can prepare a patch. Right now I'm testing a patch
> that removes sanitization for hash-tables (and maps) that track memory
> allocations.

I've got 2 patch candidates that will make it happen. It survives build
on --enable-languages=all, but one would need to build all cross-compilers
to make a proper testing.

Do you like the idea of the patch before I'll write a changelog and test it?

Thanks,
Martin

> 
> Martin
> 
>>
>> Richard.
>>
>>> Jason
> 


[-- Attachment #2: 0002-Disable-hash-table-sanitization-for-mem-stats-maps.patch --]
[-- Type: text/x-patch, Size: 2694 bytes --]

From 4888f897e579267abb6ebf94646518f5c9f2da1d Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Wed, 12 Jun 2019 11:04:11 +0200
Subject: [PATCH 2/2] Disable hash-table sanitization for mem stats maps.

---
 gcc/ggc-common.c | 2 +-
 gcc/hash-map.h   | 9 ++++++---
 gcc/mem-stats.h  | 6 +++---
 3 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/gcc/ggc-common.c b/gcc/ggc-common.c
index 2acdb6dc60c..6fb5a3d5ceb 100644
--- a/gcc/ggc-common.c
+++ b/gcc/ggc-common.c
@@ -1014,5 +1014,5 @@ ggc_prune_overhead_list (void)
       (*it).second.first->m_collected += (*it).second.second;
 
   delete ggc_mem_desc.m_reverse_object_map;
-  ggc_mem_desc.m_reverse_object_map = new map_t (13, false, false);
+  ggc_mem_desc.m_reverse_object_map = new map_t (13, false, false, false);
 }
diff --git a/gcc/hash-map.h b/gcc/hash-map.h
index 151632730e7..77f17275ec3 100644
--- a/gcc/hash-map.h
+++ b/gcc/hash-map.h
@@ -119,16 +119,19 @@ class GTY((user)) hash_map
 
 public:
   explicit hash_map (size_t n = 13, bool ggc = false,
+		     bool sanitize_eq_and_hash = true,
 		     bool gather_mem_stats = GATHER_STATISTICS
 		     CXX_MEM_STAT_INFO)
-    : m_table (n, ggc, true, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT)
+    : m_table (n, ggc, sanitize_eq_and_hash, gather_mem_stats,
+	       HASH_MAP_ORIGIN PASS_MEM_STAT)
   {
   }
 
   explicit hash_map (const hash_map &h, bool ggc = false,
+		     bool sanitize_eq_and_hash = true,
 		     bool gather_mem_stats = GATHER_STATISTICS
 		     CXX_MEM_STAT_INFO)
-    : m_table (h.m_table, ggc, true, gather_mem_stats,
+    : m_table (h.m_table, ggc, sanitize_eq_and_hash, gather_mem_stats,
 	       HASH_MAP_ORIGIN PASS_MEM_STAT) {}
 
   /* Create a hash_map in ggc memory.  */
@@ -137,7 +140,7 @@ public:
 			       CXX_MEM_STAT_INFO)
     {
       hash_map *map = ggc_alloc<hash_map> ();
-      new (map) hash_map (size, true, gather_mem_stats PASS_MEM_STAT);
+      new (map) hash_map (size, true, true, gather_mem_stats PASS_MEM_STAT);
       return map;
     }
 
diff --git a/gcc/mem-stats.h b/gcc/mem-stats.h
index 63ce8712e2b..77960595753 100644
--- a/gcc/mem-stats.h
+++ b/gcc/mem-stats.h
@@ -559,9 +559,9 @@ template <class T>
 inline
 mem_alloc_description<T>::mem_alloc_description ()
 {
-  m_map = new mem_map_t (13, false, false);
-  m_reverse_map = new reverse_mem_map_t (13, false, false);
-  m_reverse_object_map = new reverse_object_map_t (13, false, false);
+  m_map = new mem_map_t (13, false, false, false);
+  m_reverse_map = new reverse_mem_map_t (13, false, false, false);
+  m_reverse_object_map = new reverse_object_map_t (13, false, false, false);
 }
 
 /* Default destructor.  */
-- 
2.21.0


[-- Attachment #3: 0001-Add-sanitize-into-hash-traits-and-disable-it-for-obv.patch --]
[-- Type: text/x-patch, Size: 12263 bytes --]

From 7b96fb6e6d6c5d0e574440ecf0b842d71d5a8049 Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Wed, 12 Jun 2019 11:03:33 +0200
Subject: [PATCH 1/2] Add ::sanitize into hash-traits and disable it for
 obvious types.

---
 gcc/attribs.c         |  5 +++++
 gcc/cp/cp-tree.h      |  2 ++
 gcc/cp/decl2.c        |  1 +
 gcc/gcov.c            |  2 ++
 gcc/graphite.c        |  1 +
 gcc/hash-map-traits.h | 26 ++++++++++++++++++++++++++
 gcc/hash-map.h        |  1 +
 gcc/hash-table.h      |  5 +++--
 gcc/hash-traits.h     | 25 +++++++++++++++++++++++++
 gcc/ipa-devirt.c      |  2 ++
 gcc/ipa-prop.c        |  3 +++
 gcc/profile.c         |  1 +
 gcc/sanopt.c          |  2 ++
 gcc/tree-hasher.h     |  1 +
 gcc/tree-ssa-sccvn.c  |  1 +
 gcc/tree-vect-slp.c   |  1 +
 16 files changed, 77 insertions(+), 2 deletions(-)

diff --git a/gcc/attribs.c b/gcc/attribs.c
index 4441922543f..06a6b56b634 100644
--- a/gcc/attribs.c
+++ b/gcc/attribs.c
@@ -2057,6 +2057,11 @@ struct excl_hash_traits: typed_noop_remove<excl_pair>
   {
     return !*x.first && !*x.second;
   }
+
+  static bool sanitize ()
+  {
+    return true;
+  }
 };
 
 
diff --git a/gcc/cp/cp-tree.h b/gcc/cp/cp-tree.h
index 1f4e1e15554..d0ce3b68693 100644
--- a/gcc/cp/cp-tree.h
+++ b/gcc/cp/cp-tree.h
@@ -869,6 +869,7 @@ struct named_decl_hash : ggc_remove <tree>
   /* Nothing is deletable.  Everything is insertable.  */
   static bool is_deleted (value_type) { return false; }
   static void mark_deleted (value_type) { gcc_unreachable (); }
+  inline static bool sanitize () { return true; }
 };
 
 /* Simplified unique_ptr clone to release a tree vec on exit.  */
@@ -1815,6 +1816,7 @@ struct named_label_hash : ggc_remove <named_label_entry *>
   /* Nothing is deletable.  Everything is insertable.  */
   inline static bool is_deleted (value_type) { return false; }
   inline static void mark_deleted (value_type) { gcc_unreachable (); }
+  inline static bool sanitize () { return true; }
 };
 
 /* Global state pertinent to the current function.  */
diff --git a/gcc/cp/decl2.c b/gcc/cp/decl2.c
index 338db4ab6de..f4f5e878eb7 100644
--- a/gcc/cp/decl2.c
+++ b/gcc/cp/decl2.c
@@ -131,6 +131,7 @@ struct mangled_decl_hash : ggc_remove <tree>
   {
     e = reinterpret_cast <value_type> (1);
   }
+  inline static bool sanitize () { return true; }
 };
 
 /* A hash table of decls keyed by mangled name.  Used to figure out if
diff --git a/gcc/gcov.c b/gcc/gcov.c
index b06a6714c2e..015cac6247a 100644
--- a/gcc/gcov.c
+++ b/gcc/gcov.c
@@ -1234,6 +1234,8 @@ struct function_start_pair_hash : typed_noop_remove <function_start>
   {
     return ref.start_line == ~2U;
   }
+
+  inline static bool sanitize () { return true; }
 };
 
 /* Process a single input file.  */
diff --git a/gcc/graphite.c b/gcc/graphite.c
index 67202e276e2..c85f88e9e34 100644
--- a/gcc/graphite.c
+++ b/gcc/graphite.c
@@ -237,6 +237,7 @@ struct sese_scev_hash : typed_noop_remove <seir_cache_key>
   static void mark_empty (seir_cache_key &key) { key.entry_dest = 0; }
   static bool is_deleted (const seir_cache_key &key) { return !key.expr; }
   static bool is_empty (const seir_cache_key &key) { return key.entry_dest == 0; }
+  static bool sanitize () { return true; }
 };
 
 static hash_map<sese_scev_hash, tree> *seir_cache;
diff --git a/gcc/hash-map-traits.h b/gcc/hash-map-traits.h
index af66018e92c..a52c272d7e2 100644
--- a/gcc/hash-map-traits.h
+++ b/gcc/hash-map-traits.h
@@ -40,6 +40,7 @@ struct simple_hashmap_traits
   template <typename T> static inline bool is_deleted (const T &);
   template <typename T> static inline void mark_empty (T &);
   template <typename T> static inline void mark_deleted (T &);
+  static inline bool sanitize ();
 };
 
 template <typename H, typename Value>
@@ -98,6 +99,13 @@ simple_hashmap_traits <H, Value>::mark_deleted (T &entry)
   H::mark_deleted (entry.m_key);
 }
 
+template <typename H, typename Value>
+inline bool
+simple_hashmap_traits <H, Value>::sanitize ()
+{
+  return H::sanitize ();
+}
+
 template <typename H, typename Value>
 struct simple_cache_map_traits: public simple_hashmap_traits<H,Value>
 {
@@ -117,6 +125,7 @@ struct unbounded_hashmap_traits
   template <typename T> static inline bool is_deleted (const T &);
   template <typename T> static inline void mark_empty (T &);
   template <typename T> static inline void mark_deleted (T &);
+  template <typename T> static inline bool sanitize ();
 };
 
 template <typename Value>
@@ -159,6 +168,14 @@ unbounded_hashmap_traits <Value>::mark_deleted (T &entry)
   default_hash_traits <Value>::mark_deleted (entry.m_value);
 }
 
+template <typename Value>
+template <typename T>
+inline bool
+unbounded_hashmap_traits <Value>::sanitize ()
+{
+  return T::sanitize ();
+}
+
 /* Implement traits for a hash_map from integer type Key to Value in
    cases where Key has no spare values for recording empty and deleted
    slots.  */
@@ -169,6 +186,7 @@ struct unbounded_int_hashmap_traits : unbounded_hashmap_traits <Value>
   typedef Key key_type;
   static inline hashval_t hash (Key);
   static inline bool equal_keys (Key, Key);
+  static inline bool sanitize ();
 };
 
 template <typename Key, typename Value>
@@ -185,4 +203,12 @@ unbounded_int_hashmap_traits <Key, Value>::equal_keys (Key k1, Key k2)
   return k1 == k2;
 }
 
+template <typename Key, typename Value>
+inline bool
+unbounded_int_hashmap_traits <Key, Value>::sanitize ()
+{
+  return false;
+}
+
+
 #endif // HASH_MAP_TRAITS_H
diff --git a/gcc/hash-map.h b/gcc/hash-map.h
index a8eb42d5a03..151632730e7 100644
--- a/gcc/hash-map.h
+++ b/gcc/hash-map.h
@@ -55,6 +55,7 @@ class GTY((user)) hash_map
 
     static void mark_empty (hash_entry &e) { Traits::mark_empty (e); }
     static bool is_empty (const hash_entry &e) { return Traits::is_empty (e); }
+    static bool sanitize () { return Traits::sanitize (); }
 
     static void ggc_mx (hash_entry &e)
       {
diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index 4f5e150a0ac..de5ebc8ba6d 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -600,7 +600,8 @@ hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc,
 						     mem_alloc_origin origin
 						     MEM_STAT_DECL) :
   m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0),
-  m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash)
+  m_ggc (ggc),
+  m_sanitize_eq_and_hash (sanitize_eq_and_hash && Descriptor::sanitize ())
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
@@ -633,7 +634,7 @@ hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h,
 						     MEM_STAT_DECL) :
   m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted),
   m_searches (0), m_collisions (0), m_ggc (ggc),
-  m_sanitize_eq_and_hash (sanitize_eq_and_hash)
+  m_sanitize_eq_and_hash (sanitize_eq_and_hash && Descriptor::sanitize ())
 #if GATHER_STATISTICS
   , m_gather_mem_stats (gather_mem_stats)
 #endif
diff --git a/gcc/hash-traits.h b/gcc/hash-traits.h
index 2d17e2c982a..37a97c31491 100644
--- a/gcc/hash-traits.h
+++ b/gcc/hash-traits.h
@@ -91,6 +91,7 @@ struct int_hash : typed_noop_remove <Type>
   static inline void mark_empty (Type &);
   static inline bool is_deleted (Type);
   static inline bool is_empty (Type);
+  static inline bool sanitize ();
 };
 
 template <typename Type, Type Empty, Type Deleted>
@@ -136,6 +137,13 @@ int_hash <Type, Empty, Deleted>::is_empty (Type x)
   return x == Empty;
 }
 
+template <typename Type, Type Empty, Type Deleted>
+inline bool
+int_hash <Type, Empty, Deleted>::sanitize ()
+{
+  return false;
+}
+
 /* Pointer hasher based on pointer equality.  Other types of pointer hash
    can inherit this and override the hash and equal functions with some
    other form of equality (such as string equality).  */
@@ -153,6 +161,7 @@ struct pointer_hash
   static inline void mark_empty (Type *&);
   static inline bool is_deleted (Type *);
   static inline bool is_empty (Type *);
+  static inline bool sanitize ();
 };
 
 template <typename Type>
@@ -200,6 +209,13 @@ pointer_hash <Type>::is_empty (Type *e)
   return e == NULL;
 }
 
+template <typename Type>
+inline bool
+pointer_hash <Type>::sanitize ()
+{
+  return true;
+}
+
 /* Hasher for "const char *" strings, using string rather than pointer
    equality.  */
 
@@ -326,6 +342,7 @@ struct pair_hash
   static inline void mark_empty (value_type &);
   static inline bool is_deleted (const value_type &);
   static inline bool is_empty (const value_type &);
+  static inline bool sanitize ();
 };
 
 template <typename T1, typename T2>
@@ -378,6 +395,14 @@ pair_hash <T1, T2>::is_empty (const value_type &x)
   return T1::is_empty (x.first);
 }
 
+template <typename T1, typename T2>
+inline bool
+pair_hash <T1, T2>::sanitize ()
+{
+  return T1::sanitize ();
+}
+
+
 template <typename T> struct default_hash_traits : T {};
 
 template <typename T>
diff --git a/gcc/ipa-devirt.c b/gcc/ipa-devirt.c
index a4c8b0de86f..26f0d332e00 100644
--- a/gcc/ipa-devirt.c
+++ b/gcc/ipa-devirt.c
@@ -170,6 +170,8 @@ struct default_hash_traits <type_pair>
     {
       e.first = NULL;
     }
+
+  static bool sanitize () { return true; }
 };
 
 static bool odr_types_equivalent_p (tree, tree, bool, bool *,
diff --git a/gcc/ipa-prop.c b/gcc/ipa-prop.c
index d86c2f3db55..57cd864b74d 100644
--- a/gcc/ipa-prop.c
+++ b/gcc/ipa-prop.c
@@ -99,6 +99,8 @@ struct ipa_bit_ggc_hash_traits : public ggc_cache_remove <ipa_bits *>
     {
       p = reinterpret_cast<ipa_bits *> (1);
     }
+
+  static bool sanitize () { return true; }
 };
 
 /* Hash table for avoid repeated allocations of equal ipa_bits.  */
@@ -144,6 +146,7 @@ struct ipa_vr_ggc_hash_traits : public ggc_cache_remove <value_range_base *>
     {
       p = reinterpret_cast<value_range_base *> (1);
     }
+  inline static bool sanitize () { return true; }
 };
 
 /* Hash table for avoid repeated allocations of equal value_ranges.  */
diff --git a/gcc/profile.c b/gcc/profile.c
index 9aff9ef2b21..668888f89ab 100644
--- a/gcc/profile.c
+++ b/gcc/profile.c
@@ -884,6 +884,7 @@ struct location_triplet_hash : typed_noop_remove <location_triplet>
   {
     return ref.lineno == -2;
   }
+  inline static bool sanitize () { return true; }
 };
 
 
diff --git a/gcc/sanopt.c b/gcc/sanopt.c
index 5cb98e1b50e..d2f9ba5a56c 100644
--- a/gcc/sanopt.c
+++ b/gcc/sanopt.c
@@ -147,6 +147,7 @@ struct sanopt_tree_triplet_hash : typed_noop_remove <sanopt_tree_triplet>
   {
     return ref.t1 == NULL;
   }
+  inline static bool sanitize () { return true; }
 };
 
 /* Tree couple for ptr_check_map.  */
@@ -202,6 +203,7 @@ struct sanopt_tree_couple_hash : typed_noop_remove <sanopt_tree_couple>
   {
     return ref.ptr == NULL;
   }
+  inline static bool sanitize () { return true; }
 };
 
 /* This is used to carry various hash maps and variables used
diff --git a/gcc/tree-hasher.h b/gcc/tree-hasher.h
index a64f297fb2a..24da6c0a416 100644
--- a/gcc/tree-hasher.h
+++ b/gcc/tree-hasher.h
@@ -42,6 +42,7 @@ struct int_tree_hasher
   static bool is_empty (const value_type &v) { return v.to == NULL; }
   static void mark_empty (value_type &v) { v.to = NULL; }
   static void remove (value_type &) {}
+  inline static bool sanitize () { return true; }
 };
 
 /* Hash a UID in a int_tree_map.  */
diff --git a/gcc/tree-ssa-sccvn.c b/gcc/tree-ssa-sccvn.c
index b4f626000dd..69d365dc5d2 100644
--- a/gcc/tree-ssa-sccvn.c
+++ b/gcc/tree-ssa-sccvn.c
@@ -335,6 +335,7 @@ struct vn_ssa_aux_hasher : typed_noop_remove <vn_ssa_aux_t>
   static inline void mark_empty (value_type &e) { e = NULL; }
   static inline bool is_deleted (value_type &) { return false; }
   static inline bool is_empty (value_type &e) { return e == NULL; }
+  inline static bool sanitize () { return true; }
 };
 
 hashval_t
diff --git a/gcc/tree-vect-slp.c b/gcc/tree-vect-slp.c
index 930cd79784e..10ad0632853 100644
--- a/gcc/tree-vect-slp.c
+++ b/gcc/tree-vect-slp.c
@@ -1023,6 +1023,7 @@ struct bst_traits
   static inline void mark_empty (value_type &x) { x.release (); }
   static inline void mark_deleted (value_type &x) { x.release (); }
   static inline void remove (value_type &x) { x.release (); }
+  inline static bool sanitize () { return true; }
 };
 inline hashval_t
 bst_traits::hash (value_type x)
-- 
2.21.0

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Wed, Jun 12, 2019 at 11:15 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/12/19 10:02 AM, Martin Liška wrote:
> > On 6/12/19 9:59 AM, Richard Biener wrote:
> >> On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote:
> >>>
> >>> On 6/11/19 9:16 AM, Martin Liška wrote:
> >>>> On 6/11/19 2:27 PM, Jason Merrill wrote:
> >>>>> On 6/11/19 3:41 AM, Martin Liška wrote:
> >>>>>> On 6/10/19 8:21 PM, Jason Merrill wrote:
> >>>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote:
> >>>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
> >>>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
> >>>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
> >>>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
> >>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>>>>>>>>>>>>>>>>>                       ^^^^^^
> >>>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Martin
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>>>>>>>>>     Jakub
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>   From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>    gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>>>>>>>>>>>>>>>    1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>      value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>>>>>>>>>>>>>>>>>      value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>>>>>>>>>>>>>>>>>      bool too_empty_p (unsigned int);
> >>>>>>>>>>>>>>>>>>>>>>>>      void expand ();
> >>>>>>>>>>>>>>>>>>>>>>>>      static bool is_deleted (value_type &v)
> >>>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>>>>>>>>>>      if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>>>>>>>>>>>>>>>>>        expand ();
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
> >>>>>>>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
> >>>>>>>>>>>>>>>>>>>>>>>> +#endif
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> +  m_searches++;
> >>>>>>>>>>>>>>>>>>>>>>>>      value_type *first_deleted_slot = NULL;
> >>>>>>>>>>>>>>>>>>>>>>>>      hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>>>>>>>>>>>      hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>>>>>>>>>>      return &m_entries[index];
> >>>>>>>>>>>>>>>>>>>>>>>>    }
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>>>>>>>>>>>>>>>>>> +static void
> >>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
> >>>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>>>>>>>>>>>>>>>>>> Hi.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I've just added one more PR:
> >>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>>>>>>>>>>>>>>>>>> limits maximal number of checks:
> >>>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>>>>>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>>>>>>>>>>>>>>>>>> just thinking about loud.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>>>>>>>>>>>>>>>>>> issue :-)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
> >>>>>>>>>>>>>>>>>>> table are never compared against each other but always against another
> >>>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
> >>>>>>>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
> >>>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
> >>>>>>>>>>>>>>>>>>> all other elements?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
> >>>>>>>>>>>>>>>>>> PR90450 and PR87847.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Changes from previous version:
> >>>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
> >>>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >>>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
> >>>>>>>>>>>>>>>>>>     to limit number of elements that are compared within a table
> >>>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
> >>>>>>>>>>>>>>>>> comparing random two elements in the table.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> >>>>>>>>>>>>>>>>> without INSERTing.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations
> >>>>>>>>>>>>>>>> (both w/ and w/o insertion).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> >>>>>>>>>>>>>>>> except for:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> >>>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> >>>>>>>>>>>>>>>> during GIMPLE pass: lim
> >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> >>>>>>>>>>>>>>>>       6 | fn1 ()
> >>>>>>>>>>>>>>>>         | ^~~
> >>>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error
> >>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> >>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> >>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> >>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> >>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:960
> >>>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
> >>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> >>>>>>>>>>>>>>>> 0xe504ea analyze_memory_references
> >>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> >>>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim
> >>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> >>>>>>>>>>>>>>>> 0xe504ea execute
> >>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Richi: it's after your recent patch.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> >>>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
> >>>>>>>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Good point, I've just adjusted that.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ready to be installed?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> >>>>>>>>>>>>> fixing it ...
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR.
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> One question - there's unconditional
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> +         if (m_sanitize_eq_and_hash)
> >>>>>>>>>>>>> +           verify (comparable, hash);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> which will read a global variable and have (possibly not inline) call
> >>>>>>>>>>>>> to verify on a common path even with checking disabled.  So I think
> >>>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P
> >>>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
> >>>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
> >>>>>>>>>>>
> >>>>>>>>>>> You missed the second occurance
> >>>>>>>>>>>
> >>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>> +  if (m_sanitize_eq_and_hash)
> >>>>>>>>>>> +    verify (comparable, hash);
> >>>>>>>>>>
> >>>>>>>>>> Yep ;) I've just install the patch.
> >>>>>>>>>
> >>>>>>>>> This is breaking my build:
> >>>>>>>>>
> >>>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
> >>>>>>>>> call to ‘hash_table<hash_map<mem_alloc_d\
> >>>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*,
> >>>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
> >>>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
> >>>>>>>>> false, xcallocator>::hash_table(size_t&, bo\
> >>>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
> >>>>>>>>>        : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
> >>>>>>>>>
> >>>>>>>>> Looks like this needs to be updated to pass an argument to the new
> >>>>>>>>> sanitize_eq_and_hash parameter.
> >>>>>>>>>
> >>>>>>>>> Jason
> >>>>>>>>
> >>>>>>>> Sorry for the breakage, I've just fixed that in r272104.
> >>>>>>>
> >>>>>>> Thanks.  I'm also seeing a massive compile time hit from this:  A
> >>>>>>> constexpr testcase that I've been looking at went from compiling in 13
> >>>>>>> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
> >>>>>>> code to see similar problems when sanitization is enabled for those
> >>>>>>> hash tables.  Could we keep the parameter low or 0 by default, and
> >>>>>>> just do occasional sanitize runs with it explicitly enabled?
> >>>>>>
> >>>>>> Makes sense to me. Can you please provide a test-case which I can measure?
> >>>>>
> >>>>> This is the one I've been looking at:
> >>>>>
> >>>>>    struct Int {
> >>>>>      constexpr Int(int v): v(v) {}
> >>>>>      constexpr Int& operator+=(Int b) { this->v += b.v; return *this; }
> >>>>>      constexpr Int& operator++() { ++this->v; return *this; }
> >>>>>    private:
> >>>>>      friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; }
> >>>>>      int v;
> >>>>>    };
> >>>>>    constexpr int f(int n) {
> >>>>>      Int i = {0};
> >>>>>      Int k = {0};
> >>>>>      k = 0;
> >>>>>      for (; k<10000; ++k) {
> >>>>>        i += k;
> >>>>>      }
> >>>>>      return n;
> >>>>>    }
> >>>>>
> >>>>>    template<int N> struct S {
> >>>>>      static constexpr int sm = S<N-1>::sm+f(N);
> >>>>>    };
> >>>>>    template<> struct S<0> {
> >>>>>      static constexpr int sm = 0;
> >>>>>    };
> >>>>>    constexpr int r = S<20>::sm;
> >>>>>
> >>>>> Jason
> >>>>
> >>>> For the test-case provided I see:
> >>>>
> >>>> $ time g++ time.cc -c --param hash-table-verification-limit=100
> >>>>
> >>>> real  0m1.855s
> >>>> user  0m1.829s
> >>>> sys   0m0.025s
> >>>>
> >>>> $ time g++ time.cc -c --param hash-table-verification-limit=0
> >>>>
> >>>> real  0m1.275s
> >>>> user  0m1.219s
> >>>> sys   0m0.052s
> >>>>
> >>>> $ time g++-9 time.cc -c
> >>>>
> >>>> real  0m0.939s
> >>>> user  0m0.827s
> >>>> sys   0m0.109s
> >>>>
> >>>> So it's slower, but I can't confirm the huge slowdown you see.
> >>>> Is it due to r272144?
> >>>
> >>> Hmm, I wonder if this is because of the
> >>> --enable-gather-detailed-mem-stats hash tables.
> >>
> >> I wonder if we can reduce the overhead by making
> >> hash-tables/maps using predefined traits not perform
> >> the checking?  Thus make [the default] whether to check or not
> >> to check part of the traits?  Surely the basic pointer-hash/map
> >> stuff is OK and needs no extra checking.
> >
> > Interesting idea! I can prepare a patch. Right now I'm testing a patch
> > that removes sanitization for hash-tables (and maps) that track memory
> > allocations.
>
> I've got 2 patch candidates that will make it happen. It survives build
> on --enable-languages=all, but one would need to build all cross-compilers
> to make a proper testing.
>
> Do you like the idea of the patch before I'll write a changelog and test it?

The disabling for mem-stats patch looks good to me.  I don't like the
implementation of the traits one, we don't want to have to put the
default returning true everywhere.  Instead I expected some
enable_if <> magic to do select a default true if the member isn't
present.  The issue with the traits idea is also that people like
to derive from one of the standard traits and thus might inherit
'false' even though they override hash/compare methods.  That is,
I expected

+template <typename Type>
+inline bool
+pointer_hash <Type>::sanitize ()
+{
+  return true;
+}

to return false for example.  Basically for the case we
auto-detect the traits based on the key type I wanted to
disable sanitizing and for custom users leave them to
per-object decisions.

Not sure if we have enough C++ power to make that happen easily.
So let's go the route of disabling the "ovbiously" correct and performance
critical parts for now.

Richard.

> Thanks,
> Martin
>
> >
> > Martin
> >
> >>
> >> Richard.
> >>
> >>> Jason
> >
>

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 18561 bytes --]

On 6/12/19 11:41 AM, Richard Biener wrote:
> On Wed, Jun 12, 2019 at 11:15 AM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 6/12/19 10:02 AM, Martin Liška wrote:
>>> On 6/12/19 9:59 AM, Richard Biener wrote:
>>>> On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote:
>>>>>
>>>>> On 6/11/19 9:16 AM, Martin Liška wrote:
>>>>>> On 6/11/19 2:27 PM, Jason Merrill wrote:
>>>>>>> On 6/11/19 3:41 AM, Martin Liška wrote:
>>>>>>>> On 6/10/19 8:21 PM, Jason Merrill wrote:
>>>>>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote:
>>>>>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
>>>>>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>>>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>>>>>>>>>>>>                       ^^^^^^
>>>>>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>>>>>>>>>     Jakub
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>   From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>    gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>>>>>>>>>>>>    1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>      value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>>>>>>>>>>>>      value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>>>>>>>>>>>>      bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>>>>>>>>>>>>      void expand ();
>>>>>>>>>>>>>>>>>>>>>>>>>>      static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>>>>>>>      if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>>>>>>>>>>>>        expand ();
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>>>>>>>>>>>>      value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>>>>>>>>>>>>      hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>>>>>>>>      hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>>>>>>>>>>>>      return &m_entries[index];
>>>>>>>>>>>>>>>>>>>>>>>>>>    }
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>>>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>>>>>>>>>>>>> issue :-)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>>>>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>>>>>>>>>>>>> all other elements?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>>>>>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Changes from previous version:
>>>>>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>>>>>>>>>>>>     to limit number of elements that are compared within a table
>>>>>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>>>>>>>>>>>>> comparing random two elements in the table.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>>>>>>>>>>>>> without INSERTing.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>>>>>>>>>>>>> (both w/ and w/o insertion).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>>>>>>>>>>>>> except for:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>>>>>>>>>>>>> during GIMPLE pass: lim
>>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>>>>>>>>>>>>       6 | fn1 ()
>>>>>>>>>>>>>>>>>>         | ^~~
>>>>>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>>>>>>>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>>>>>>>>>>>>> 0xe504ea execute
>>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Richi: it's after your recent patch.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>>>>>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Good point, I've just adjusted that.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ready to be installed?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>>>>>>>>>>>>> fixing it ...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> One question - there's unconditional
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +         if (m_sanitize_eq_and_hash)
>>>>>>>>>>>>>>> +           verify (comparable, hash);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> which will read a global variable and have (possibly not inline) call
>>>>>>>>>>>>>>> to verify on a common path even with checking disabled.  So I think
>>>>>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P
>>>>>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>>>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>>>>>>>>>>>>
>>>>>>>>>>>>> You missed the second occurance
>>>>>>>>>>>>>
>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>> +  if (m_sanitize_eq_and_hash)
>>>>>>>>>>>>> +    verify (comparable, hash);
>>>>>>>>>>>>
>>>>>>>>>>>> Yep ;) I've just install the patch.
>>>>>>>>>>>
>>>>>>>>>>> This is breaking my build:
>>>>>>>>>>>
>>>>>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
>>>>>>>>>>> call to ‘hash_table<hash_map<mem_alloc_d\
>>>>>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*,
>>>>>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
>>>>>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
>>>>>>>>>>> false, xcallocator>::hash_table(size_t&, bo\
>>>>>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
>>>>>>>>>>>        : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
>>>>>>>>>>>
>>>>>>>>>>> Looks like this needs to be updated to pass an argument to the new
>>>>>>>>>>> sanitize_eq_and_hash parameter.
>>>>>>>>>>>
>>>>>>>>>>> Jason
>>>>>>>>>>
>>>>>>>>>> Sorry for the breakage, I've just fixed that in r272104.
>>>>>>>>>
>>>>>>>>> Thanks.  I'm also seeing a massive compile time hit from this:  A
>>>>>>>>> constexpr testcase that I've been looking at went from compiling in 13
>>>>>>>>> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
>>>>>>>>> code to see similar problems when sanitization is enabled for those
>>>>>>>>> hash tables.  Could we keep the parameter low or 0 by default, and
>>>>>>>>> just do occasional sanitize runs with it explicitly enabled?
>>>>>>>>
>>>>>>>> Makes sense to me. Can you please provide a test-case which I can measure?
>>>>>>>
>>>>>>> This is the one I've been looking at:
>>>>>>>
>>>>>>>    struct Int {
>>>>>>>      constexpr Int(int v): v(v) {}
>>>>>>>      constexpr Int& operator+=(Int b) { this->v += b.v; return *this; }
>>>>>>>      constexpr Int& operator++() { ++this->v; return *this; }
>>>>>>>    private:
>>>>>>>      friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; }
>>>>>>>      int v;
>>>>>>>    };
>>>>>>>    constexpr int f(int n) {
>>>>>>>      Int i = {0};
>>>>>>>      Int k = {0};
>>>>>>>      k = 0;
>>>>>>>      for (; k<10000; ++k) {
>>>>>>>        i += k;
>>>>>>>      }
>>>>>>>      return n;
>>>>>>>    }
>>>>>>>
>>>>>>>    template<int N> struct S {
>>>>>>>      static constexpr int sm = S<N-1>::sm+f(N);
>>>>>>>    };
>>>>>>>    template<> struct S<0> {
>>>>>>>      static constexpr int sm = 0;
>>>>>>>    };
>>>>>>>    constexpr int r = S<20>::sm;
>>>>>>>
>>>>>>> Jason
>>>>>>
>>>>>> For the test-case provided I see:
>>>>>>
>>>>>> $ time g++ time.cc -c --param hash-table-verification-limit=100
>>>>>>
>>>>>> real  0m1.855s
>>>>>> user  0m1.829s
>>>>>> sys   0m0.025s
>>>>>>
>>>>>> $ time g++ time.cc -c --param hash-table-verification-limit=0
>>>>>>
>>>>>> real  0m1.275s
>>>>>> user  0m1.219s
>>>>>> sys   0m0.052s
>>>>>>
>>>>>> $ time g++-9 time.cc -c
>>>>>>
>>>>>> real  0m0.939s
>>>>>> user  0m0.827s
>>>>>> sys   0m0.109s
>>>>>>
>>>>>> So it's slower, but I can't confirm the huge slowdown you see.
>>>>>> Is it due to r272144?
>>>>>
>>>>> Hmm, I wonder if this is because of the
>>>>> --enable-gather-detailed-mem-stats hash tables.
>>>>
>>>> I wonder if we can reduce the overhead by making
>>>> hash-tables/maps using predefined traits not perform
>>>> the checking?  Thus make [the default] whether to check or not
>>>> to check part of the traits?  Surely the basic pointer-hash/map
>>>> stuff is OK and needs no extra checking.
>>>
>>> Interesting idea! I can prepare a patch. Right now I'm testing a patch
>>> that removes sanitization for hash-tables (and maps) that track memory
>>> allocations.
>>
>> I've got 2 patch candidates that will make it happen. It survives build
>> on --enable-languages=all, but one would need to build all cross-compilers
>> to make a proper testing.
>>
>> Do you like the idea of the patch before I'll write a changelog and test it?
> 
> The disabling for mem-stats patch looks good to me.  I don't like the
> implementation of the traits one, we don't want to have to put the
> default returning true everywhere.  Instead I expected some
> enable_if <> magic to do select a default true if the member isn't
> present.  The issue with the traits idea is also that people like
> to derive from one of the standard traits and thus might inherit
> 'false' even though they override hash/compare methods.  That is,
> I expected
> 
> +template <typename Type>
> +inline bool
> +pointer_hash <Type>::sanitize ()
> +{
> +  return true;
> +}
> 
> to return false for example.  Basically for the case we
> auto-detect the traits based on the key type I wanted to
> disable sanitizing and for custom users leave them to
> per-object decisions.
> 
> Not sure if we have enough C++ power to make that happen easily.

Huh, I've tried quite hard and I was unable to make it happen. Maybe
somebody more familiar with C++ can step in?

> So let's go the route of disabling the "ovbiously" correct and performance
> critical parts for now.

I'm sending patch for it.

Martin

> 
> Richard.
> 
>> Thanks,
>> Martin
>>
>>>
>>> Martin
>>>
>>>>
>>>> Richard.
>>>>
>>>>> Jason
>>>
>>


[-- Attachment #2: 0001-Disable-hash-table-sanitization-for-mem-stats-maps.patch --]
[-- Type: text/x-patch, Size: 2978 bytes --]

From 71668e0eaf983ff05dfd52a95827cefdf24350ec Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Wed, 12 Jun 2019 11:04:11 +0200
Subject: [PATCH] Disable hash-table sanitization for mem stats maps.

gcc/ChangeLog:

2019-06-12  Martin Liska  <mliska@suse.cz>

	* ggc-common.c (ggc_prune_overhead_list): Do not sanitize
	the created map.
	* hash-map.h: Add sanitize_eq_and_hash into ::hash_map.
	* mem-stats.h (mem_alloc_description::mem_alloc_description):
	Do not sanitize created maps.
---
 gcc/ggc-common.c | 2 +-
 gcc/hash-map.h   | 9 ++++++---
 gcc/mem-stats.h  | 6 +++---
 3 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/gcc/ggc-common.c b/gcc/ggc-common.c
index 2acdb6dc60c..6fb5a3d5ceb 100644
--- a/gcc/ggc-common.c
+++ b/gcc/ggc-common.c
@@ -1014,5 +1014,5 @@ ggc_prune_overhead_list (void)
       (*it).second.first->m_collected += (*it).second.second;
 
   delete ggc_mem_desc.m_reverse_object_map;
-  ggc_mem_desc.m_reverse_object_map = new map_t (13, false, false);
+  ggc_mem_desc.m_reverse_object_map = new map_t (13, false, false, false);
 }
diff --git a/gcc/hash-map.h b/gcc/hash-map.h
index a8eb42d5a03..588dfda04fa 100644
--- a/gcc/hash-map.h
+++ b/gcc/hash-map.h
@@ -118,16 +118,19 @@ class GTY((user)) hash_map
 
 public:
   explicit hash_map (size_t n = 13, bool ggc = false,
+		     bool sanitize_eq_and_hash = true,
 		     bool gather_mem_stats = GATHER_STATISTICS
 		     CXX_MEM_STAT_INFO)
-    : m_table (n, ggc, true, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT)
+    : m_table (n, ggc, sanitize_eq_and_hash, gather_mem_stats,
+	       HASH_MAP_ORIGIN PASS_MEM_STAT)
   {
   }
 
   explicit hash_map (const hash_map &h, bool ggc = false,
+		     bool sanitize_eq_and_hash = true,
 		     bool gather_mem_stats = GATHER_STATISTICS
 		     CXX_MEM_STAT_INFO)
-    : m_table (h.m_table, ggc, true, gather_mem_stats,
+    : m_table (h.m_table, ggc, sanitize_eq_and_hash, gather_mem_stats,
 	       HASH_MAP_ORIGIN PASS_MEM_STAT) {}
 
   /* Create a hash_map in ggc memory.  */
@@ -136,7 +139,7 @@ public:
 			       CXX_MEM_STAT_INFO)
     {
       hash_map *map = ggc_alloc<hash_map> ();
-      new (map) hash_map (size, true, gather_mem_stats PASS_MEM_STAT);
+      new (map) hash_map (size, true, true, gather_mem_stats PASS_MEM_STAT);
       return map;
     }
 
diff --git a/gcc/mem-stats.h b/gcc/mem-stats.h
index 63ce8712e2b..77960595753 100644
--- a/gcc/mem-stats.h
+++ b/gcc/mem-stats.h
@@ -559,9 +559,9 @@ template <class T>
 inline
 mem_alloc_description<T>::mem_alloc_description ()
 {
-  m_map = new mem_map_t (13, false, false);
-  m_reverse_map = new reverse_mem_map_t (13, false, false);
-  m_reverse_object_map = new reverse_object_map_t (13, false, false);
+  m_map = new mem_map_t (13, false, false, false);
+  m_reverse_map = new reverse_mem_map_t (13, false, false, false);
+  m_reverse_object_map = new reverse_object_map_t (13, false, false, false);
 }
 
 /* Default destructor.  */
-- 
2.21.0

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Wed, Jun 12, 2019 at 1:45 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/12/19 11:41 AM, Richard Biener wrote:
> > On Wed, Jun 12, 2019 at 11:15 AM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> On 6/12/19 10:02 AM, Martin Liška wrote:
> >>> On 6/12/19 9:59 AM, Richard Biener wrote:
> >>>> On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote:
> >>>>>
> >>>>> On 6/11/19 9:16 AM, Martin Liška wrote:
> >>>>>> On 6/11/19 2:27 PM, Jason Merrill wrote:
> >>>>>>> On 6/11/19 3:41 AM, Martin Liška wrote:
> >>>>>>>> On 6/10/19 8:21 PM, Jason Merrill wrote:
> >>>>>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote:
> >>>>>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote:
> >>>>>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote:
> >>>>>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
> >>>>>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
> >>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>>>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>>>>>>>>>>>>>>>>>>>                       ^^^^^^
> >>>>>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Martin
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>>>>>>>>>>>     Jakub
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>   From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>    gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>>>>>>>>>>>>>>>>>    1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>      value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>>>>>>>>>>>>>>>>>>>      value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>>>>>>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>>>>>>>>>>>>>>>>>>>      bool too_empty_p (unsigned int);
> >>>>>>>>>>>>>>>>>>>>>>>>>>      void expand ();
> >>>>>>>>>>>>>>>>>>>>>>>>>>      static bool is_deleted (value_type &v)
> >>>>>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>>>>>>>>>>>>      if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>>>>>>>>>>>>>>>>>>>        expand ();
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>>>>>>>>>>>>> +    if (insert == INSERT)
> >>>>>>>>>>>>>>>>>>>>>>>>>> +      verify (comparable, hash);
> >>>>>>>>>>>>>>>>>>>>>>>>>> +#endif
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> +  m_searches++;
> >>>>>>>>>>>>>>>>>>>>>>>>>>      value_type *first_deleted_slot = NULL;
> >>>>>>>>>>>>>>>>>>>>>>>>>>      hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>>>>>>>>>>>>>      hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>>>>>>>>>>>>      return &m_entries[index];
> >>>>>>>>>>>>>>>>>>>>>>>>>>    }
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>>>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>>>>>>>>>>>>>>>>>>>> +static void
> >>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>>>>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix.
> >>>>>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>>>>>>>>>>>>>>>>>>>> Hi.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I've just added one more PR:
> >>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>>>>>>>>>>>>>>>>>>>> limits maximal number of checks:
> >>>>>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>>>>>>>>>>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>>>>>>>>>>>>>>>>>>>> just thinking about loud.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>>>>>>>>>>>>>>>>>>>> issue :-)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
> >>>>>>>>>>>>>>>>>>>>> table are never compared against each other but always against another
> >>>>>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
> >>>>>>>>>>>>>>>>>>>>> comparison function only works with those.  With the patch we verify
> >>>>>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>>>>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
> >>>>>>>>>>>>>>>>>>>>> all other elements?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
> >>>>>>>>>>>>>>>>>>>> PR90450 and PR87847.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Changes from previous version:
> >>>>>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted)
> >>>>>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >>>>>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
> >>>>>>>>>>>>>>>>>>>>     to limit number of elements that are compared within a table
> >>>>>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
> >>>>>>>>>>>>>>>>>>> comparing random two elements in the table.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> >>>>>>>>>>>>>>>>>>> without INSERTing.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations
> >>>>>>>>>>>>>>>>>> (both w/ and w/o insertion).
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> >>>>>>>>>>>>>>>>>> except for:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> >>>>>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> >>>>>>>>>>>>>>>>>> during GIMPLE pass: lim
> >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> >>>>>>>>>>>>>>>>>>       6 | fn1 ()
> >>>>>>>>>>>>>>>>>>         | ^~~
> >>>>>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error
> >>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> >>>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> >>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> >>>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> >>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/hash-table.h:960
> >>>>>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt
> >>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> >>>>>>>>>>>>>>>>>> 0xe504ea analyze_memory_references
> >>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> >>>>>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim
> >>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> >>>>>>>>>>>>>>>>>> 0xe504ea execute
> >>>>>>>>>>>>>>>>>>        /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Richi: it's after your recent patch.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> >>>>>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a
> >>>>>>>>>>>>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Good point, I've just adjusted that.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Ready to be installed?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> >>>>>>>>>>>>>>> fixing it ...
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> One question - there's unconditional
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> +         if (m_sanitize_eq_and_hash)
> >>>>>>>>>>>>>>> +           verify (comparable, hash);
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> which will read a global variable and have (possibly not inline) call
> >>>>>>>>>>>>>>> to verify on a common path even with checking disabled.  So I think
> >>>>>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P
> >>>>>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not
> >>>>>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later..
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> You missed the second occurance
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>>>> +  if (m_sanitize_eq_and_hash)
> >>>>>>>>>>>>> +    verify (comparable, hash);
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yep ;) I've just install the patch.
> >>>>>>>>>>>
> >>>>>>>>>>> This is breaking my build:
> >>>>>>>>>>>
> >>>>>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for
> >>>>>>>>>>> call to ‘hash_table<hash_map<mem_alloc_d\
> >>>>>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*,
> >>>>>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\
> >>>>>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry,
> >>>>>>>>>>> false, xcallocator>::hash_table(size_t&, bo\
> >>>>>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’
> >>>>>>>>>>>        : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {}
> >>>>>>>>>>>
> >>>>>>>>>>> Looks like this needs to be updated to pass an argument to the new
> >>>>>>>>>>> sanitize_eq_and_hash parameter.
> >>>>>>>>>>>
> >>>>>>>>>>> Jason
> >>>>>>>>>>
> >>>>>>>>>> Sorry for the breakage, I've just fixed that in r272104.
> >>>>>>>>>
> >>>>>>>>> Thanks.  I'm also seeing a massive compile time hit from this:  A
> >>>>>>>>> constexpr testcase that I've been looking at went from compiling in 13
> >>>>>>>>> seconds to 78 seconds, 6 times as long.  I would expect template-heavy
> >>>>>>>>> code to see similar problems when sanitization is enabled for those
> >>>>>>>>> hash tables.  Could we keep the parameter low or 0 by default, and
> >>>>>>>>> just do occasional sanitize runs with it explicitly enabled?
> >>>>>>>>
> >>>>>>>> Makes sense to me. Can you please provide a test-case which I can measure?
> >>>>>>>
> >>>>>>> This is the one I've been looking at:
> >>>>>>>
> >>>>>>>    struct Int {
> >>>>>>>      constexpr Int(int v): v(v) {}
> >>>>>>>      constexpr Int& operator+=(Int b) { this->v += b.v; return *this; }
> >>>>>>>      constexpr Int& operator++() { ++this->v; return *this; }
> >>>>>>>    private:
> >>>>>>>      friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; }
> >>>>>>>      int v;
> >>>>>>>    };
> >>>>>>>    constexpr int f(int n) {
> >>>>>>>      Int i = {0};
> >>>>>>>      Int k = {0};
> >>>>>>>      k = 0;
> >>>>>>>      for (; k<10000; ++k) {
> >>>>>>>        i += k;
> >>>>>>>      }
> >>>>>>>      return n;
> >>>>>>>    }
> >>>>>>>
> >>>>>>>    template<int N> struct S {
> >>>>>>>      static constexpr int sm = S<N-1>::sm+f(N);
> >>>>>>>    };
> >>>>>>>    template<> struct S<0> {
> >>>>>>>      static constexpr int sm = 0;
> >>>>>>>    };
> >>>>>>>    constexpr int r = S<20>::sm;
> >>>>>>>
> >>>>>>> Jason
> >>>>>>
> >>>>>> For the test-case provided I see:
> >>>>>>
> >>>>>> $ time g++ time.cc -c --param hash-table-verification-limit=100
> >>>>>>
> >>>>>> real  0m1.855s
> >>>>>> user  0m1.829s
> >>>>>> sys   0m0.025s
> >>>>>>
> >>>>>> $ time g++ time.cc -c --param hash-table-verification-limit=0
> >>>>>>
> >>>>>> real  0m1.275s
> >>>>>> user  0m1.219s
> >>>>>> sys   0m0.052s
> >>>>>>
> >>>>>> $ time g++-9 time.cc -c
> >>>>>>
> >>>>>> real  0m0.939s
> >>>>>> user  0m0.827s
> >>>>>> sys   0m0.109s
> >>>>>>
> >>>>>> So it's slower, but I can't confirm the huge slowdown you see.
> >>>>>> Is it due to r272144?
> >>>>>
> >>>>> Hmm, I wonder if this is because of the
> >>>>> --enable-gather-detailed-mem-stats hash tables.
> >>>>
> >>>> I wonder if we can reduce the overhead by making
> >>>> hash-tables/maps using predefined traits not perform
> >>>> the checking?  Thus make [the default] whether to check or not
> >>>> to check part of the traits?  Surely the basic pointer-hash/map
> >>>> stuff is OK and needs no extra checking.
> >>>
> >>> Interesting idea! I can prepare a patch. Right now I'm testing a patch
> >>> that removes sanitization for hash-tables (and maps) that track memory
> >>> allocations.
> >>
> >> I've got 2 patch candidates that will make it happen. It survives build
> >> on --enable-languages=all, but one would need to build all cross-compilers
> >> to make a proper testing.
> >>
> >> Do you like the idea of the patch before I'll write a changelog and test it?
> >
> > The disabling for mem-stats patch looks good to me.  I don't like the
> > implementation of the traits one, we don't want to have to put the
> > default returning true everywhere.  Instead I expected some
> > enable_if <> magic to do select a default true if the member isn't
> > present.  The issue with the traits idea is also that people like
> > to derive from one of the standard traits and thus might inherit
> > 'false' even though they override hash/compare methods.  That is,
> > I expected
> >
> > +template <typename Type>
> > +inline bool
> > +pointer_hash <Type>::sanitize ()
> > +{
> > +  return true;
> > +}
> >
> > to return false for example.  Basically for the case we
> > auto-detect the traits based on the key type I wanted to
> > disable sanitizing and for custom users leave them to
> > per-object decisions.
> >
> > Not sure if we have enough C++ power to make that happen easily.
>
> Huh, I've tried quite hard and I was unable to make it happen. Maybe
> somebody more familiar with C++ can step in?
>
> > So let's go the route of disabling the "ovbiously" correct and performance
> > critical parts for now.
>
> I'm sending patch for it.

   explicit hash_map (size_t n = 13, bool ggc = false,
+                    bool sanitize_eq_and_hash = true,
                     bool gather_mem_stats = GATHER_STATISTICS
                     CXX_MEM_STAT_INFO)

maybe not an issue here but adding defaulted params into the
middle is prone to change existing calls semantics.  Consider

 hash_map (13, false, false);

which was gather_mem_stats == false before but now is
sanitize_eq_and_hash.  I realize the original patch had the
same issue.

Patch is still OK.

Please grep for explicit constructor calls and double-check.
Ick.  I guess also

   hash_map<foo, bar> baz (13, false, false);

is affected so make sure to apply enough grep-fu.

New params should always go to the end ;)

Richard.

>
> Martin
>
> >
> > Richard.
> >
> >> Thanks,
> >> Martin
> >>
> >>>
> >>> Martin
> >>>
> >>>>
> >>>> Richard.
> >>>>
> >>>>> Jason
> >>>
> >>
>

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 6/12/19 2:50 PM, Richard Biener wrote:
> New params should always go to the end 

Ah, sorry, I'll take of that new time.

I've just changed the function signature and I can verify there are no
other calls (tested for --enable-languages=all).

Thanks,
Martin

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Ian Lance Taylor]

[-- Attachment #1: Type: text/plain, Size: 10925 bytes --]

On Fri, Jun 7, 2019 at 5:04 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/7/19 10:57 AM, Richard Biener wrote:
> > On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> On 6/1/19 12:06 AM, Jeff Law wrote:
> >>> On 5/22/19 3:13 AM, Martin Liška wrote:
> >>>> On 5/21/19 1:51 PM, Richard Biener wrote:
> >>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>
> >>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>>>>>
> >>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>>>>>                     ^^^^^^
> >>>>>>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Martin
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>   Jakub
> >>>>>>>>>>>>>
> >>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>>>>>
> >>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>>>>>
> >>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>>>>>    bool too_empty_p (unsigned int);
> >>>>>>>>>>>>    void expand ();
> >>>>>>>>>>>>    static bool is_deleted (value_type &v)
> >>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>>>>>      expand ();
> >>>>>>>>>>>>
> >>>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>> +    if (insert == INSERT)
> >>>>>>>>>>>> +      verify (comparable, hash);
> >>>>>>>>>>>> +#endif
> >>>>>>>>>>>>
> >>>>>>>>>>>> +  m_searches++;
> >>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
> >>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>    return &m_entries[index];
> >>>>>>>>>>>>  }
> >>>>>>>>>>>>
> >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>> +
> >>>>>>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>>>>>> +
> >>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>>>>>> +static void
> >>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>> +{
> >>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>> +}
> >>>>>>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>>>>>
> >>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>>>>>> because we've got too many bugs to fix.
> >>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>>>>>> Hi.
> >>>>>>>>>
> >>>>>>>>> I've just added one more PR:
> >>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>>>>>
> >>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>>>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>>>>>
> >>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>>>>>> limits maximal number of checks:
> >>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>>>>>> just thinking about loud.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>>>>>> issue :-)
> >>>>>>>
> >>>>>>> There is at least one PR where we have a table where elements _in_ the
> >>>>>>> table are never compared against each other but always against another
> >>>>>>> object (I guess that's usual even), but the setup is in a way that the
> >>>>>>> comparison function only works with those.  With the patch we verify
> >>>>>>> hashing/comparison for something that is never used.
> >>>>>>>
> >>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>>>>>> at lookup time, using the object from the lookup and verify that against
> >>>>>>> all other elements?
> >>>>>>
> >>>>>> I don't a have problem with that. Apparently this changes fixes
> >>>>>> PR90450 and PR87847.
> >>>>>>
> >>>>>> Changes from previous version:
> >>>>>> - verification happens only when an element is searched (not inserted)
> >>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >>>>>> - new param has been introduced hash-table-verification-limit in order
> >>>>>>   to limit number of elements that are compared within a table
> >>>>>> - verification happens only with flag_checking >= 2
> >>>>>>
> >>>>>> I've been bootstrapping and testing the patch right now.
> >>>>>
> >>>>> Looks like I misremembered the original patch.  The issue isn't
> >>>>> comparing random two elements in the table.
> >>>>>
> >>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> >>>>> without INSERTing.
> >>>>>
> >>>>
> >>>> There's updated version of the patch where I check all find operations
> >>>> (both w/ and w/o insertion).
> >>>>
> >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> >>>> except for:
> >>>>
> >>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> >>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> >>>> during GIMPLE pass: lim
> >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> >>>>     6 | fn1 ()
> >>>>       | ^~~
> >>>> 0x6c5725 hashtab_chk_error
> >>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> >>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> >>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
> >>>> 0xe504ea gather_mem_refs_stmt
> >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> >>>> 0xe504ea analyze_memory_references
> >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> >>>> 0xe504ea tree_ssa_lim
> >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> >>>> 0xe504ea execute
> >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> >>>>
> >>>> Richi: it's after your recent patch.
> >>>>
> >>>> For some reason I don't see PR87847 issue any longer.
> >>>>
> >>>>
> >>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> >>> Don't we still need to deal with the naked fprintf when there's a
> >>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
> >>
> >> Good point, I've just adjusted that.
> >>
> >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> >>
> >> Ready to be installed?
> >
> > Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> > fixing it ...
>
> Yes :D It's been some time and there's no interest in the PR.
>
> >
> > One question - there's unconditional
> >
> > +         if (m_sanitize_eq_and_hash)
> > +           verify (comparable, hash);
> >
> > which will read a global variable and have (possibly not inline) call
> > to verify on a common path even with checking disabled.  So I think
> > we want to compile this checking feature out for !CHECKING_P
> > or at least make the if __builtin_expect (..., 0), ::verify not
> > inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>
> Fixed. May I install the patch? The cselib issue can be solved later..

After this patch, when I do a configure with --disable-bootstrap, and
build with "gcc (Debian 7.3.0-18) 7.3.0", I get a lot of warnings of
the form

In file included from ../../gccgo3/gcc/coretypes.h:440:0,
                 from ../../gccgo3/gcc/go/go-system.h:137,
                 from ../../gccgo3/gcc/go/gofrontend/go.cc:7:
../../gccgo3/gcc/hash-table.h:1017:1: warning: ‘void
hashtab_chk_error()’ defined but not used [-Wunused-function]
 hashtab_chk_error ()
 ^~~~~~~~~~~~~~~~~

These are just warnings, since I am using --disable-bootstrap, but
they are distracting.

This patch fixes it.  OK for trunk?

Ian

2019-06-23  Ian Lance Taylor  <iant@golang.org>

* hash-table.h (hashtab_chk_error): Add ATTRIBUTE_UNUSED.

[-- Attachment #2: patch.txt --]
[-- Type: text/plain, Size: 390 bytes --]

Index: hash-table.h
===================================================================
--- hash-table.h	(revision 272609)
+++ hash-table.h	(working copy)
@@ -1012,7 +1012,7 @@ hash_table<Descriptor, Lazy, Allocator>
 
 /* Report a hash table checking error.  */
 
-ATTRIBUTE_NORETURN ATTRIBUTE_COLD
+ATTRIBUTE_NORETURN ATTRIBUTE_COLD ATTRIBUTE_UNUSED
 static void
 hashtab_chk_error ()
 {

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Mon, Jun 24, 2019 at 1:08 AM Ian Lance Taylor <iant@golang.org> wrote:
>
> On Fri, Jun 7, 2019 at 5:04 AM Martin Liška <mliska@suse.cz> wrote:
> >
> > On 6/7/19 10:57 AM, Richard Biener wrote:
> > > On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
> > >>
> > >> On 6/1/19 12:06 AM, Jeff Law wrote:
> > >>> On 5/22/19 3:13 AM, Martin Liška wrote:
> > >>>> On 5/21/19 1:51 PM, Richard Biener wrote:
> > >>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> > >>>>>>
> > >>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> > >>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> > >>>>>>>>
> > >>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> > >>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> > >>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> > >>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> > >>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> > >>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> > >>>>>>>>>>>>>> +hashtab_chk_error ()
> > >>>>>>>>>>>>>> +{
> > >>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> > >>>>>>>>>>>>>> +           "equal operator returns true for a pair "
> > >>>>>>>>>>>>>> +           "of values with a different hash value");
> > >>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> > >>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
> > >>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> > >>>>>>>>>>>>>                     ^^^^^^
> > >>>>>>>>>>>> Sure, fixed in attached patch.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Martin
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>> +  gcc_unreachable ();
> > >>>>>>>>>>>>>> +}
> > >>>>>>>>>>>>>   Jakub
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> > >>>>>>>>>>>> From: marxin <mliska@suse.cz>
> > >>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> > >>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> > >>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> > >>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> > >>>>>>>>>>>> --- a/gcc/hash-table.h
> > >>>>>>>>>>>> +++ b/gcc/hash-table.h
> > >>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> > >>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
> > >>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> > >>>>>>>>>>>>    bool too_empty_p (unsigned int);
> > >>>>>>>>>>>>    void expand ();
> > >>>>>>>>>>>>    static bool is_deleted (value_type &v)
> > >>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> > >>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> > >>>>>>>>>>>>      expand ();
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -  m_searches++;
> > >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> > >>>>>>>>>>>> +    if (insert == INSERT)
> > >>>>>>>>>>>> +      verify (comparable, hash);
> > >>>>>>>>>>>> +#endif
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +  m_searches++;
> > >>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
> > >>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> > >>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> > >>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> > >>>>>>>>>>>>    return &m_entries[index];
> > >>>>>>>>>>>>  }
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>> +/* Report a hash table checking error.  */
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> > >>>>>>>>>>>> +static void
> > >>>>>>>>>>>> +hashtab_chk_error ()
> > >>>>>>>>>>>> +{
> > >>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> > >>>>>>>>>>>> +     "equal operator returns true for a pair "
> > >>>>>>>>>>>> +     "of values with a different hash value\n");
> > >>>>>>>>>>>> +  gcc_unreachable ();
> > >>>>>>>>>>>> +}
> > >>>>>>>>>>> I think an internal_error here is probably still better than a simple
> > >>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> > >>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> > >>>>>>>>>>
> > >>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> > >>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> > >>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> > >>>>>>>>>>> because we've got too many bugs to fix.
> > >>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
> > >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> > >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> > >>>>>>>>> Hi.
> > >>>>>>>>>
> > >>>>>>>>> I've just added one more PR:
> > >>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> > >>>>>>>>>
> > >>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> > >>>>>>>>> with a new function disable_sanitize_eq_and_hash.
> > >>>>>>>>>
> > >>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> > >>>>>>>>> limits maximal number of checks:
> > >>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> > >>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
> > >>>>>>>> just thinking about loud.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> > >>>>>>>> issue :-)
> > >>>>>>>
> > >>>>>>> There is at least one PR where we have a table where elements _in_ the
> > >>>>>>> table are never compared against each other but always against another
> > >>>>>>> object (I guess that's usual even), but the setup is in a way that the
> > >>>>>>> comparison function only works with those.  With the patch we verify
> > >>>>>>> hashing/comparison for something that is never used.
> > >>>>>>>
> > >>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> > >>>>>>> at lookup time, using the object from the lookup and verify that against
> > >>>>>>> all other elements?
> > >>>>>>
> > >>>>>> I don't a have problem with that. Apparently this changes fixes
> > >>>>>> PR90450 and PR87847.
> > >>>>>>
> > >>>>>> Changes from previous version:
> > >>>>>> - verification happens only when an element is searched (not inserted)
> > >>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> > >>>>>> - new param has been introduced hash-table-verification-limit in order
> > >>>>>>   to limit number of elements that are compared within a table
> > >>>>>> - verification happens only with flag_checking >= 2
> > >>>>>>
> > >>>>>> I've been bootstrapping and testing the patch right now.
> > >>>>>
> > >>>>> Looks like I misremembered the original patch.  The issue isn't
> > >>>>> comparing random two elements in the table.
> > >>>>>
> > >>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> > >>>>> without INSERTing.
> > >>>>>
> > >>>>
> > >>>> There's updated version of the patch where I check all find operations
> > >>>> (both w/ and w/o insertion).
> > >>>>
> > >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> > >>>> except for:
> > >>>>
> > >>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> > >>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> > >>>> during GIMPLE pass: lim
> > >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> > >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> > >>>>     6 | fn1 ()
> > >>>>       | ^~~
> > >>>> 0x6c5725 hashtab_chk_error
> > >>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> > >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> > >>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> > >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> > >>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
> > >>>> 0xe504ea gather_mem_refs_stmt
> > >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> > >>>> 0xe504ea analyze_memory_references
> > >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> > >>>> 0xe504ea tree_ssa_lim
> > >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> > >>>> 0xe504ea execute
> > >>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> > >>>>
> > >>>> Richi: it's after your recent patch.
> > >>>>
> > >>>> For some reason I don't see PR87847 issue any longer.
> > >>>>
> > >>>>
> > >>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> > >>> Don't we still need to deal with the naked fprintf when there's a
> > >>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
> > >>
> > >> Good point, I've just adjusted that.
> > >>
> > >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> > >>
> > >> Ready to be installed?
> > >
> > > Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> > > fixing it ...
> >
> > Yes :D It's been some time and there's no interest in the PR.
> >
> > >
> > > One question - there's unconditional
> > >
> > > +         if (m_sanitize_eq_and_hash)
> > > +           verify (comparable, hash);
> > >
> > > which will read a global variable and have (possibly not inline) call
> > > to verify on a common path even with checking disabled.  So I think
> > > we want to compile this checking feature out for !CHECKING_P
> > > or at least make the if __builtin_expect (..., 0), ::verify not
> > > inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
> >
> > Fixed. May I install the patch? The cselib issue can be solved later..
>
> After this patch, when I do a configure with --disable-bootstrap, and
> build with "gcc (Debian 7.3.0-18) 7.3.0", I get a lot of warnings of
> the form
>
> In file included from ../../gccgo3/gcc/coretypes.h:440:0,
>                  from ../../gccgo3/gcc/go/go-system.h:137,
>                  from ../../gccgo3/gcc/go/gofrontend/go.cc:7:
> ../../gccgo3/gcc/hash-table.h:1017:1: warning: ‘void
> hashtab_chk_error()’ defined but not used [-Wunused-function]
>  hashtab_chk_error ()
>  ^~~~~~~~~~~~~~~~~
>
> These are just warnings, since I am using --disable-bootstrap, but
> they are distracting.
>
> This patch fixes it.  OK for trunk?

Hmm, the function is called exactly once.  I guess the intent was
to not emit the printf in every ::verify instance but then why not
instantiate this function in just hash-table.c and not mark it inline?

Richard.

>
> Ian
>
> 2019-06-23  Ian Lance Taylor  <iant@golang.org>
>
> * hash-table.h (hashtab_chk_error): Add ATTRIBUTE_UNUSED.

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 6/24/19 2:29 PM, Richard Biener wrote:
> On Mon, Jun 24, 2019 at 1:08 AM Ian Lance Taylor <iant@golang.org> wrote:
>>
>> On Fri, Jun 7, 2019 at 5:04 AM Martin Liška <mliska@suse.cz> wrote:
>>>
>>> On 6/7/19 10:57 AM, Richard Biener wrote:
>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>
>>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>>>
>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
>>>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
>>>>>>>>>>>>>>>>> +           "of values with a different hash value");
>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
>>>>>>>>>>>>>>>>                     ^^^^^^
>>>>>>>>>>>>>>> Sure, fixed in attached patch.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>   Jakub
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
>>>>>>>>>>>>>>> --- a/gcc/hash-table.h
>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
>>>>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
>>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
>>>>>>>>>>>>>>>    bool too_empty_p (unsigned int);
>>>>>>>>>>>>>>>    void expand ();
>>>>>>>>>>>>>>>    static bool is_deleted (value_type &v)
>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
>>>>>>>>>>>>>>>      expand ();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -  m_searches++;
>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>> +    if (insert == INSERT)
>>>>>>>>>>>>>>> +      verify (comparable, hash);
>>>>>>>>>>>>>>> +#endif
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +  m_searches++;
>>>>>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
>>>>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
>>>>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
>>>>>>>>>>>>>>>    return &m_entries[index];
>>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>>>>>>>>>>>>>>> +static void
>>>>>>>>>>>>>>> +hashtab_chk_error ()
>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
>>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
>>>>>>>>>>>>>>> +     "of values with a different hash value\n");
>>>>>>>>>>>>>>> +  gcc_unreachable ();
>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
>>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
>>>>>>>>>>>>>> because we've got too many bugs to fix.
>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
>>>>>>>>>>>> Hi.
>>>>>>>>>>>>
>>>>>>>>>>>> I've just added one more PR:
>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
>>>>>>>>>>>>
>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
>>>>>>>>>>>>
>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
>>>>>>>>>>>> limits maximal number of checks:
>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
>>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
>>>>>>>>>>> just thinking about loud.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
>>>>>>>>>>> issue :-)
>>>>>>>>>>
>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
>>>>>>>>>> table are never compared against each other but always against another
>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
>>>>>>>>>> comparison function only works with those.  With the patch we verify
>>>>>>>>>> hashing/comparison for something that is never used.
>>>>>>>>>>
>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
>>>>>>>>>> at lookup time, using the object from the lookup and verify that against
>>>>>>>>>> all other elements?
>>>>>>>>>
>>>>>>>>> I don't a have problem with that. Apparently this changes fixes
>>>>>>>>> PR90450 and PR87847.
>>>>>>>>>
>>>>>>>>> Changes from previous version:
>>>>>>>>> - verification happens only when an element is searched (not inserted)
>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
>>>>>>>>> - new param has been introduced hash-table-verification-limit in order
>>>>>>>>>   to limit number of elements that are compared within a table
>>>>>>>>> - verification happens only with flag_checking >= 2
>>>>>>>>>
>>>>>>>>> I've been bootstrapping and testing the patch right now.
>>>>>>>>
>>>>>>>> Looks like I misremembered the original patch.  The issue isn't
>>>>>>>> comparing random two elements in the table.
>>>>>>>>
>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
>>>>>>>> without INSERTing.
>>>>>>>>
>>>>>>>
>>>>>>> There's updated version of the patch where I check all find operations
>>>>>>> (both w/ and w/o insertion).
>>>>>>>
>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
>>>>>>> except for:
>>>>>>>
>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
>>>>>>> during GIMPLE pass: lim
>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
>>>>>>>     6 | fn1 ()
>>>>>>>       | ^~~
>>>>>>> 0x6c5725 hashtab_chk_error
>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
>>>>>>> 0xe504ea gather_mem_refs_stmt
>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
>>>>>>> 0xe504ea analyze_memory_references
>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
>>>>>>> 0xe504ea tree_ssa_lim
>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
>>>>>>> 0xe504ea execute
>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
>>>>>>>
>>>>>>> Richi: it's after your recent patch.
>>>>>>>
>>>>>>> For some reason I don't see PR87847 issue any longer.
>>>>>>>
>>>>>>>
>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
>>>>>> Don't we still need to deal with the naked fprintf when there's a
>>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
>>>>>
>>>>> Good point, I've just adjusted that.
>>>>>
>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>>>>>
>>>>> Ready to be installed?
>>>>
>>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
>>>> fixing it ...
>>>
>>> Yes :D It's been some time and there's no interest in the PR.
>>>
>>>>
>>>> One question - there's unconditional
>>>>
>>>> +         if (m_sanitize_eq_and_hash)
>>>> +           verify (comparable, hash);
>>>>
>>>> which will read a global variable and have (possibly not inline) call
>>>> to verify on a common path even with checking disabled.  So I think
>>>> we want to compile this checking feature out for !CHECKING_P
>>>> or at least make the if __builtin_expect (..., 0), ::verify not
>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
>>>
>>> Fixed. May I install the patch? The cselib issue can be solved later..
>>
>> After this patch, when I do a configure with --disable-bootstrap, and
>> build with "gcc (Debian 7.3.0-18) 7.3.0", I get a lot of warnings of
>> the form
>>
>> In file included from ../../gccgo3/gcc/coretypes.h:440:0,
>>                  from ../../gccgo3/gcc/go/go-system.h:137,
>>                  from ../../gccgo3/gcc/go/gofrontend/go.cc:7:
>> ../../gccgo3/gcc/hash-table.h:1017:1: warning: ‘void
>> hashtab_chk_error()’ defined but not used [-Wunused-function]
>>  hashtab_chk_error ()
>>  ^~~~~~~~~~~~~~~~~
>>
>> These are just warnings, since I am using --disable-bootstrap, but
>> they are distracting.
>>
>> This patch fixes it.  OK for trunk?
> 
> Hmm, the function is called exactly once.  I guess the intent was
> to not emit the printf in every ::verify instance but then why not
> instantiate this function in just hash-table.c and not mark it inline?

I marked the function ATTRIBUTE_COLD, so it should not be inlined
into ::verify.

  1013  /* Report a hash table checking error.  */
  1014  
  1015  ATTRIBUTE_NORETURN ATTRIBUTE_COLD
  1016  static void
  1017  hashtab_chk_error ()
  1018  {
  1019    fprintf (stderr, "hash table checking failed: "
  1020             "equal operator returns true for a pair "
  1021             "of values with a different hash value\n");
  1022    gcc_unreachable ();
  1023  }

Martin

> 
> Richard.
> 
>>
>> Ian
>>
>> 2019-06-23  Ian Lance Taylor  <iant@golang.org>
>>
>> * hash-table.h (hashtab_chk_error): Add ATTRIBUTE_UNUSED.

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Mon, Jun 24, 2019 at 3:51 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/24/19 2:29 PM, Richard Biener wrote:
> > On Mon, Jun 24, 2019 at 1:08 AM Ian Lance Taylor <iant@golang.org> wrote:
> >>
> >> On Fri, Jun 7, 2019 at 5:04 AM Martin Liška <mliska@suse.cz> wrote:
> >>>
> >>> On 6/7/19 10:57 AM, Richard Biener wrote:
> >>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>
> >>>>> On 6/1/19 12:06 AM, Jeff Law wrote:
> >>>>>> On 5/22/19 3:13 AM, Martin Liška wrote:
> >>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote:
> >>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>>>>
> >>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote:
> >>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote:
> >>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote:
> >>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote:
> >>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote:
> >>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote:
> >>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote:
> >>>>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>>>> +           "equal operator returns true for a pair "
> >>>>>>>>>>>>>>>>> +           "of values with a different hash value");
> >>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf
> >>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw:
> >>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack
> >>>>>>>>>>>>>>>>                     ^^^^^^
> >>>>>>>>>>>>>>> Sure, fixed in attached patch.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Martin
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>   Jakub
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001
> >>>>>>>>>>>>>>> From: marxin <mliska@suse.cz>
> >>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100
> >>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h
> >>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644
> >>>>>>>>>>>>>>> --- a/gcc/hash-table.h
> >>>>>>>>>>>>>>> +++ b/gcc/hash-table.h
> >>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const;
> >>>>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t);
> >>>>>>>>>>>>>>> +  void verify (const compare_type &comparable, hashval_t hash);
> >>>>>>>>>>>>>>>    bool too_empty_p (unsigned int);
> >>>>>>>>>>>>>>>    void expand ();
> >>>>>>>>>>>>>>>    static bool is_deleted (value_type &v)
> >>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4)
> >>>>>>>>>>>>>>>      expand ();
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -  m_searches++;
> >>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>> +    if (insert == INSERT)
> >>>>>>>>>>>>>>> +      verify (comparable, hash);
> >>>>>>>>>>>>>>> +#endif
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> +  m_searches++;
> >>>>>>>>>>>>>>>    value_type *first_deleted_slot = NULL;
> >>>>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index);
> >>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator>
> >>>>>>>>>>>>>>>    return &m_entries[index];
> >>>>>>>>>>>>>>>  }
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING
> >>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>> +/* Report a hash table checking error.  */
> >>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD
> >>>>>>>>>>>>>>> +static void
> >>>>>>>>>>>>>>> +hashtab_chk_error ()
> >>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>> +  fprintf (stderr, "hash table checking failed: "
> >>>>>>>>>>>>>>> +     "equal operator returns true for a pair "
> >>>>>>>>>>>>>>> +     "of values with a different hash value\n");
> >>>>>>>>>>>>>>> +  gcc_unreachable ();
> >>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple
> >>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-)
> >>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and
> >>>>>>>>>>>>>> if not, are we likely to soon?  It'd be a shame to put it into
> >>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING
> >>>>>>>>>>>>>> because we've got too many bugs to fix.
> >>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs:
> >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845
> >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847
> >>>>>>>>>>>> Hi.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I've just added one more PR:
> >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs
> >>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash.
> >>>>>>>>>>>>
> >>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch
> >>>>>>>>>>>> limits maximal number of checks:
> >>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its
> >>>>>>>>>>> state set up when you instantiate the object?  It's not a huge deal,
> >>>>>>>>>>> just thinking about loud.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking
> >>>>>>>>>>> issue :-)
> >>>>>>>>>>
> >>>>>>>>>> There is at least one PR where we have a table where elements _in_ the
> >>>>>>>>>> table are never compared against each other but always against another
> >>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the
> >>>>>>>>>> comparison function only works with those.  With the patch we verify
> >>>>>>>>>> hashing/comparison for something that is never used.
> >>>>>>>>>>
> >>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing
> >>>>>>>>>> at lookup time, using the object from the lookup and verify that against
> >>>>>>>>>> all other elements?
> >>>>>>>>>
> >>>>>>>>> I don't a have problem with that. Apparently this changes fixes
> >>>>>>>>> PR90450 and PR87847.
> >>>>>>>>>
> >>>>>>>>> Changes from previous version:
> >>>>>>>>> - verification happens only when an element is searched (not inserted)
> >>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table
> >>>>>>>>> - new param has been introduced hash-table-verification-limit in order
> >>>>>>>>>   to limit number of elements that are compared within a table
> >>>>>>>>> - verification happens only with flag_checking >= 2
> >>>>>>>>>
> >>>>>>>>> I've been bootstrapping and testing the patch right now.
> >>>>>>>>
> >>>>>>>> Looks like I misremembered the original patch.  The issue isn't
> >>>>>>>> comparing random two elements in the table.
> >>>>>>>>
> >>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash
> >>>>>>>> without INSERTing.
> >>>>>>>>
> >>>>>>>
> >>>>>>> There's updated version of the patch where I check all find operations
> >>>>>>> (both w/ and w/o insertion).
> >>>>>>>
> >>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests
> >>>>>>> except for:
> >>>>>>>
> >>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c
> >>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value
> >>>>>>> during GIMPLE pass: lim
> >>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’:
> >>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019
> >>>>>>>     6 | fn1 ()
> >>>>>>>       | ^~~
> >>>>>>> 0x6c5725 hashtab_chk_error
> >>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019
> >>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int)
> >>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040
> >>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option)
> >>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960
> >>>>>>> 0xe504ea gather_mem_refs_stmt
> >>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501
> >>>>>>> 0xe504ea analyze_memory_references
> >>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625
> >>>>>>> 0xe504ea tree_ssa_lim
> >>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646
> >>>>>>> 0xe504ea execute
> >>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708
> >>>>>>>
> >>>>>>> Richi: it's after your recent patch.
> >>>>>>>
> >>>>>>> For some reason I don't see PR87847 issue any longer.
> >>>>>>>
> >>>>>>>
> >>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ?
> >>>>>> Don't we still need to deal with the naked fprintf when there's a
> >>>>>> failure.  ie, shouldn't we be raising it with a gcc_assert or somesuch?
> >>>>>
> >>>>> Good point, I've just adjusted that.
> >>>>>
> >>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> >>>>>
> >>>>> Ready to be installed?
> >>>>
> >>>> Ugh, the cselib one is really bad.  But I don't hold my breath for anyone
> >>>> fixing it ...
> >>>
> >>> Yes :D It's been some time and there's no interest in the PR.
> >>>
> >>>>
> >>>> One question - there's unconditional
> >>>>
> >>>> +         if (m_sanitize_eq_and_hash)
> >>>> +           verify (comparable, hash);
> >>>>
> >>>> which will read a global variable and have (possibly not inline) call
> >>>> to verify on a common path even with checking disabled.  So I think
> >>>> we want to compile this checking feature out for !CHECKING_P
> >>>> or at least make the if __builtin_expect (..., 0), ::verify not
> >>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)).
> >>>
> >>> Fixed. May I install the patch? The cselib issue can be solved later..
> >>
> >> After this patch, when I do a configure with --disable-bootstrap, and
> >> build with "gcc (Debian 7.3.0-18) 7.3.0", I get a lot of warnings of
> >> the form
> >>
> >> In file included from ../../gccgo3/gcc/coretypes.h:440:0,
> >>                  from ../../gccgo3/gcc/go/go-system.h:137,
> >>                  from ../../gccgo3/gcc/go/gofrontend/go.cc:7:
> >> ../../gccgo3/gcc/hash-table.h:1017:1: warning: ‘void
> >> hashtab_chk_error()’ defined but not used [-Wunused-function]
> >>  hashtab_chk_error ()
> >>  ^~~~~~~~~~~~~~~~~
> >>
> >> These are just warnings, since I am using --disable-bootstrap, but
> >> they are distracting.
> >>
> >> This patch fixes it.  OK for trunk?
> >
> > Hmm, the function is called exactly once.  I guess the intent was
> > to not emit the printf in every ::verify instance but then why not
> > instantiate this function in just hash-table.c and not mark it inline?
>
> I marked the function ATTRIBUTE_COLD, so it should not be inlined
> into ::verify.

You still get one instance in each TU ...

>   1013  /* Report a hash table checking error.  */
>   1014
>   1015  ATTRIBUTE_NORETURN ATTRIBUTE_COLD
>   1016  static void
>   1017  hashtab_chk_error ()
>   1018  {
>   1019    fprintf (stderr, "hash table checking failed: "
>   1020             "equal operator returns true for a pair "
>   1021             "of values with a different hash value\n");
>   1022    gcc_unreachable ();
>   1023  }
>
> Martin
>
> >
> > Richard.
> >
> >>
> >> Ian
> >>
> >> 2019-06-23  Ian Lance Taylor  <iant@golang.org>
> >>
> >> * hash-table.h (hashtab_chk_error): Add ATTRIBUTE_UNUSED.
>

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 230 bytes --]

On 6/24/19 4:09 PM, Richard Biener wrote:
> You still get one instance in each TU ...

Right, fixed in attached patch.

Patch can bootstrap on x86_64-linux-gnu and survives regression tests.

Ready to be installed?
Thanks,
Martin

[-- Attachment #2: 0001-Put-hashtab_chk_error-into-hash-table.c.patch --]
[-- Type: text/x-patch, Size: 2083 bytes --]

From aa5ea14a8665b14aa60245c42bd4c9809d0bf81a Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Tue, 25 Jun 2019 10:33:39 +0200
Subject: [PATCH] Put hashtab_chk_error into hash-table.c.

gcc/ChangeLog:

2019-06-25  Martin Liska  <mliska@suse.cz>

	* hash-table.c (hashtab_chk_error): Move here from ...
	* hash-table.h (hashtab_chk_error): ... here.
---
 gcc/hash-table.c | 12 ++++++++++++
 gcc/hash-table.h | 14 ++------------
 2 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/gcc/hash-table.c b/gcc/hash-table.c
index 8e86fffa36f..e3b5d3da09e 100644
--- a/gcc/hash-table.c
+++ b/gcc/hash-table.c
@@ -124,3 +124,15 @@ void dump_hash_table_loc_statistics (void)
       hash_table_usage ().dump (origin);
     }
 }
+
+/* Report a hash table checking error.  */
+
+ATTRIBUTE_NORETURN ATTRIBUTE_COLD
+void
+hashtab_chk_error ()
+{
+  fprintf (stderr, "hash table checking failed: "
+	   "equal operator returns true for a pair "
+	   "of values with a different hash value\n");
+  gcc_unreachable ();
+}
diff --git a/gcc/hash-table.h b/gcc/hash-table.h
index 4f5e150a0ac..a39fb942158 100644
--- a/gcc/hash-table.h
+++ b/gcc/hash-table.h
@@ -303,6 +303,8 @@ extern unsigned int hash_table_sanitize_eq_limit;
 extern unsigned int hash_table_higher_prime_index (unsigned long n)
    ATTRIBUTE_PURE;
 
+extern ATTRIBUTE_NORETURN ATTRIBUTE_COLD void hashtab_chk_error ();
+
 /* Return X % Y using multiplicative inverse values INV and SHIFT.
 
    The multiplicative inverses computed above are for 32-bit types,
@@ -1010,18 +1012,6 @@ hash_table<Descriptor, Lazy, Allocator>
   return &m_entries[index];
 }
 
-/* Report a hash table checking error.  */
-
-ATTRIBUTE_NORETURN ATTRIBUTE_COLD
-static void
-hashtab_chk_error ()
-{
-  fprintf (stderr, "hash table checking failed: "
-	   "equal operator returns true for a pair "
-	   "of values with a different hash value\n");
-  gcc_unreachable ();
-}
-
 /* Verify that all existing elements in th hash table which are
    equal to COMPARABLE have an equal HASH value provided as argument.  */
 
-- 
2.21.0

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Martin Liška]

On 6/25/19 12:25 PM, Martin Liška wrote:
> On 6/24/19 4:09 PM, Richard Biener wrote:
>> You still get one instance in each TU ...
> 
> Right, fixed in attached patch.
> 
> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> 
> Ready to be installed?
> Thanks,
> Martin
> 

Btw. clang provides a reasonable warning for that:

/home/marxin/Programming/gcc/gcc/hash-table.h:1017:1: warning: 'static' function 'hashtab_chk_error' declared in header file should be declared 'static inline' [-Wunneeded-internal-declaration]

Martin

Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables.

[Richard Biener]

On Tue, Jun 25, 2019 at 12:25 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/24/19 4:09 PM, Richard Biener wrote:
> > You still get one instance in each TU ...
>
> Right, fixed in attached patch.
>
> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
>
> Ready to be installed?

Yes.

Thanks,
Richard.

> Thanks,
> Martin