* [PATCH][RFC] Sanitize equals and hash functions in hash-tables. @ 2018-10-29 12:02 Martin Liška 2018-10-29 14:28 ` Alexander Monakov ` (3 more replies) 0 siblings, 4 replies; 53+ messages in thread From: Martin Liška @ 2018-10-29 12:02 UTC (permalink / raw) To: gcc-patches Cc: Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 10053 bytes --] Hi. As slightly discussed here: https://gcc.gnu.org/ml/gcc-patches/2018-10/msg01674.html I fixed a situation where an equal operator of a hash table returns true, while corresponding hash value of a pair of elements is different. That's inconsistent and can probably cause issues in different areas of compiler. I wrote a simple patch that verifies for a newly added element into a hash table that there's no other equal element with a different hash. That's O(n) for each insert, so that it's only enabled with -fchecking=3. Apart from that the way how it's enable is a bit cumbersome, but that's caused by usage of hash-table also in generated files. Anyway, there are first places where I see violation of the sanity check: 1) cselib_lookup_1: $ cat ice.c a() { b(); } $ /dev/shm/objdir/gcc/xgcc -B/dev/shm/objdir/gcc/ ice.c -g -c -fchecking=3 -O hash table checking failed: equal operator returns true for a pair of values with a different hash valueduring RTL pass: vartrack ice.c:1:1: internal compiler error: in find_slot_with_hash, at hash-table.h:905 1 | a() { b(); } | ^ 0x9680b5 hash_table<cselib_hasher, xcallocator>::find_slot_with_hash(cselib_hasher::key* const&, unsigned int, insert_option) /home/marxin/Programming/gcc/gcc/hash-table.h:905 0x962518 cselib_find_slot /home/marxin/Programming/gcc/gcc/cselib.c:584 0x9625d4 cselib_lookup_1 /home/marxin/Programming/gcc/gcc/cselib.c:2097 0x9625d4 cselib_lookup(rtx_def*, machine_mode, int, machine_mode) /home/marxin/Programming/gcc/gcc/cselib.c:2141 0x965ee7 cselib_record_sets /home/marxin/Programming/gcc/gcc/cselib.c:2593 0x9670a9 cselib_process_insn(rtx_insn*) /home/marxin/Programming/gcc/gcc/cselib.c:2790 0x1036b73 vt_initialize /home/marxin/Programming/gcc/gcc/var-tracking.c:10231 0x103b98a variable_tracking_main_1 /home/marxin/Programming/gcc/gcc/var-tracking.c:10460 0x103b98a variable_tracking_main() /home/marxin/Programming/gcc/gcc/var-tracking.c:10513 2) gfc_find_module $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gfortran.dg/coarray/alloc_comp_2.f90 -fcoarray=single -fchecking=3 hash table checking failed: equal operator returns true for a pair of values with a different hash valuef951: internal compiler error: in find_slot_with_hash, at hash-table.h:905 0x8e5e86 hash_table<module_hasher, xcallocator>::find_slot_with_hash(char const* const&, unsigned int, insert_option) /home/marxin/Programming/gcc/gcc/hash-table.h:905 0x8e2c2c gfc_find_module(char const*) /home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:4865 0x8e4f42 gfc_generate_module_vars(gfc_namespace*) /home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:5475 0x8b8d7e gfc_generate_module_code(gfc_namespace*) /home/marxin/Programming/gcc/gcc/fortran/trans.c:2190 0x868427 translate_all_program_units /home/marxin/Programming/gcc/gcc/fortran/parse.c:6112 0x868427 gfc_parse_file() /home/marxin/Programming/gcc/gcc/fortran/parse.c:6328 0x8b19cb gfc_be_parse_file /home/marxin/Programming/gcc/gcc/fortran/f95-lang.c:204 3) lookup_template_class_1 $ ./xg++ -B. /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C -c -fchecking=3 hash table checking failed: equal operator returns true for a pair of values with a different hash value/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C: In instantiation of âstruct B<A>â: /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:15:8: required from here /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:8:17: internal compiler error: in find_slot_with_hash, at hash-table.h:905 8 | friend bool foo (const B<Q>& a); | ^~~ 0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option) /home/marxin/Programming/gcc/gcc/hash-table.h:905 0xa042ce lookup_template_class_1 /home/marxin/Programming/gcc/gcc/cp/pt.c:9629 0xa042ce lookup_template_class(tree_node*, tree_node*, tree_node*, tree_node*, int, int) /home/marxin/Programming/gcc/gcc/cp/pt.c:9674 0xa03670 tsubst_aggr_type /home/marxin/Programming/gcc/gcc/cp/pt.c:12679 0x9fefcd tsubst(tree_node*, tree_node*, int, tree_node*) /home/marxin/Programming/gcc/gcc/cp/pt.c:14294 0x9fe1a9 tsubst(tree_node*, tree_node*, int, tree_node*) /home/marxin/Programming/gcc/gcc/cp/pt.c:14285 0xa0d8bd tsubst_arg_types /home/marxin/Programming/gcc/gcc/cp/pt.c:13891 0xa0dc24 tsubst_function_type /home/marxin/Programming/gcc/gcc/cp/pt.c:14032 0x9fe790 tsubst(tree_node*, tree_node*, int, tree_node*) /home/marxin/Programming/gcc/gcc/cp/pt.c:14769 0x9f2c7c tsubst_function_decl /home/marxin/Programming/gcc/gcc/cp/pt.c:12921 0xa02d27 tsubst_template_decl /home/marxin/Programming/gcc/gcc/cp/pt.c:13214 0x9f4416 tsubst_decl /home/marxin/Programming/gcc/gcc/cp/pt.c:13316 0x9ff0ca tsubst(tree_node*, tree_node*, int, tree_node*) /home/marxin/Programming/gcc/gcc/cp/pt.c:14212 0xa1dfd0 tsubst_friend_function /home/marxin/Programming/gcc/gcc/cp/pt.c:10310 0xa1dfd0 instantiate_class_template_1 /home/marxin/Programming/gcc/gcc/cp/pt.c:11359 0xa1dfd0 instantiate_class_template(tree_node*) /home/marxin/Programming/gcc/gcc/cp/pt.c:11424 0xa66b22 complete_type(tree_node*) /home/marxin/Programming/gcc/gcc/cp/typeck.c:138 0x9023c7 start_decl_1(tree_node*, bool) /home/marxin/Programming/gcc/gcc/cp/decl.c:5278 0x92a15f start_decl(cp_declarator const*, cp_decl_specifier_seq*, int, tree_node*, tree_node*, tree_node**) /home/marxin/Programming/gcc/gcc/cp/decl.c:5241 0x9c1944 cp_parser_init_declarator /home/marxin/Programming/gcc/gcc/cp/parser.c:19750 4) register_specialization $ ./xg++ -B. /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/cpp0x/udlit-template.C -I/dev/shm/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu -I/dev/shm/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include -I/home/marxin/Programming/gcc/libstdc++-v3/libsupc++ -I/home/marxin/Programming/gcc/libstdc++-v3/include/ba -c -fchecking=3 hash table checking failed: equal operator returns true for a pair of values with a different hash value/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/cpp0x/udlit-template.C:13:21: internal compiler error: in find_slot_with_hash, at hash-table.h:905 13 | operator"" _abc<>() | ^ 0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option) /home/marxin/Programming/gcc/gcc/hash-table.h:905 0x9e35e6 register_specialization /home/marxin/Programming/gcc/gcc/cp/pt.c:1534 0xa22ac3 check_explicit_specialization(tree_node*, tree_node*, int, int, tree_node*) /home/marxin/Programming/gcc/gcc/cp/pt.c:3243 0x91552d grokfndecl /home/marxin/Programming/gcc/gcc/cp/decl.c:9106 0x9274bd grokdeclarator(cp_declarator const*, cp_decl_specifier_seq*, decl_context, int, tree_node**) /home/marxin/Programming/gcc/gcc/cp/decl.c:12607 0x92a9c6 start_function(cp_decl_specifier_seq*, cp_declarator const*, tree_node*) /home/marxin/Programming/gcc/gcc/cp/decl.c:15470 0x9c14a9 cp_parser_function_definition_from_specifiers_and_declarator /home/marxin/Programming/gcc/gcc/cp/parser.c:26853 0x9c14a9 cp_parser_init_declarator /home/marxin/Programming/gcc/gcc/cp/parser.c:19654 0x9c8ebd cp_parser_single_declaration /home/marxin/Programming/gcc/gcc/cp/parser.c:27437 0x9c9b11 cp_parser_explicit_specialization /home/marxin/Programming/gcc/gcc/cp/parser.c:16802 0x9cf30e cp_parser_declaration /home/marxin/Programming/gcc/gcc/cp/parser.c:12822 0x9cf94d cp_parser_translation_unit /home/marxin/Programming/gcc/gcc/cp/parser.c:4631 0x9cf94d c_parse_file() /home/marxin/Programming/gcc/gcc/cp/parser.c:39108 0xadde4a c_common_parse_file() /home/marxin/Programming/gcc/gcc/c-family/c-opts.c:1150 5) the case fixed in PR86158: $ /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/ipa/iinline-2.c:37:1: internal compiler error: in find_slot_with_hash, at hash-table.h:905 0xba10f6 hash_table<ipa_vr_ggc_hash_traits, xcallocator>::find_slot_with_hash(value_range* const&, unsigned int, insert_option) /home/marxin/Programming/gcc/gcc/hash-table.h:905 0xb9e02c hash_table<ipa_vr_ggc_hash_traits, xcallocator>::find_slot(value_range* const&, insert_option) /home/marxin/Programming/gcc/gcc/hash-table.h:418 0xb8ea21 ipa_get_value_range /home/marxin/Programming/gcc/gcc/ipa-prop.c:1776 0xb8eab6 ipa_get_value_range /home/marxin/Programming/gcc/gcc/ipa-prop.c:1795 0xb8eae5 ipa_set_jfunc_vr /home/marxin/Programming/gcc/gcc/ipa-prop.c:1806 0xb8ef49 ipa_compute_jump_functions_for_edge /home/marxin/Programming/gcc/gcc/ipa-prop.c:1875 0xb8fb53 ipa_compute_jump_functions_for_bb /home/marxin/Programming/gcc/gcc/ipa-prop.c:2017 0xb912ab analysis_dom_walker::before_dom_children(basic_block_def*) /home/marxin/Programming/gcc/gcc/ipa-prop.c:2535 0x158f997 dom_walker::walk(basic_block_def*) /home/marxin/Programming/gcc/gcc/domwalk.c:353 0xb915b0 ipa_analyze_node(cgraph_node*) /home/marxin/Programming/gcc/gcc/ipa-prop.c:2605 0xb776bf inline_indirect_intraprocedural_analysis /home/marxin/Programming/gcc/gcc/ipa-fnsummary.c:3125 0xb776bf inline_analyze_function(cgraph_node*) /home/marxin/Programming/gcc/gcc/ipa-fnsummary.c:3145 0xb77866 ipa_fn_summary_generate /home/marxin/Programming/gcc/gcc/ipa-fnsummary.c:3189 0xca1751 execute_ipa_summary_passes(ipa_opt_pass_d*) /home/marxin/Programming/gcc/gcc/passes.c:2131 0x96907a ipa_passes /home/marxin/Programming/gcc/gcc/cgraphunit.c:2505 0x96907a symbol_table::compile() /home/marxin/Programming/gcc/gcc/cgraphunit.c:2616 0x96b055 symbol_table::finalize_compilation_unit() /home/marxin/Programming/gcc/gcc/cgraphunit.c:2861 My question is whether we want to have in GCC 9 time frame or should I wait with that? Does it worth implementing? Thanks, Martin --- gcc/hash-table.c | 2 ++ gcc/hash-table.h | 25 ++++++++++++++++++++++++- gcc/toplev.c | 3 +++ 3 files changed, 29 insertions(+), 1 deletion(-) [-- Attachment #2: 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch --] [-- Type: text/x-patch, Size: 2084 bytes --] diff --git a/gcc/hash-table.c b/gcc/hash-table.c index bff9644ae81..d396a368171 100644 --- a/gcc/hash-table.c +++ b/gcc/hash-table.c @@ -121,3 +121,5 @@ void dump_hash_table_loc_statistics (void) hash_table_usage ().dump (origin); } } + +bool hash_table_verify_p = false; diff --git a/gcc/hash-table.h b/gcc/hash-table.h index bd83345c7b8..2d740b42535 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -240,6 +240,10 @@ along with GCC; see the file COPYING3. If not see #include "hash-traits.h" #include "hash-map-traits.h" +#ifndef GENERATOR_FILE +extern bool hash_table_verify_p; +#endif + template<typename, typename, typename> class hash_map; template<typename, typename> class hash_set; @@ -883,12 +887,31 @@ hash_table<Descriptor, Allocator> expand (); m_searches++; + size_t size = m_size; + +#ifndef GENERATOR_FILE + if (hash_table_verify_p) + if (insert == INSERT) + for (size_t i = 0; i < size; i++) + { + value_type *entry = &m_entries[i]; + if (!is_empty (*entry) && !is_deleted (*entry) + && Descriptor::equal (*entry, comparable) + && hash != Descriptor::hash (*entry)) + { + fprintf (stderr, "hash table checking failed: " + "equal operator returns true for a pair " + "of values with a different hash value"); + gcc_unreachable (); + } + } +#endif +// TODO: enable it also for generated files: there are failures! value_type *first_deleted_slot = NULL; hashval_t index = hash_table_mod1 (hash, m_size_prime_index); hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); value_type *entry = &m_entries[index]; - size_t size = m_size; if (is_empty (*entry)) goto empty_entry; else if (is_deleted (*entry)) diff --git a/gcc/toplev.c b/gcc/toplev.c index d7ea11abf53..4cd3aafa2e8 100644 --- a/gcc/toplev.c +++ b/gcc/toplev.c @@ -2289,6 +2289,9 @@ toplev::main (int argc, char **argv) handle_common_deferred_options (); + if (flag_checking > 2) + hash_table_verify_p = true; + init_local_tick (); initialize_plugins (); ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2018-10-29 12:02 [PATCH][RFC] Sanitize equals and hash functions in hash-tables Martin Liška @ 2018-10-29 14:28 ` Alexander Monakov 2018-10-29 15:56 ` Martin Liška 2018-10-30 10:25 ` hash-table violation in cselib.c Martin Liška ` (2 subsequent siblings) 3 siblings, 1 reply; 53+ messages in thread From: Alexander Monakov @ 2018-10-29 14:28 UTC (permalink / raw) To: Martin Liška Cc: gcc-patches, Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 1455 bytes --] On Mon, 29 Oct 2018, Martin LiÅ¡ka wrote: > My question is whether we want to have in GCC 9 time frame or should I wait with that? > Does it worth implementing? This is cool, thanks! A few questions/comments on the patch. I think there are places that use libiberty C-style hashtab (htab_t), would it make sense to have this kind of checking for them as well? I think it's going to be more complicated though, no need to do both in one step. I would recommend to factor out the error reporting path into a separate non-template function, e.g. hashtab_chk_error. See how qsort_chk_error has "cold" and "noreturn" attributes and invokes problematic comparators: the idea was that a developer can 'break qsort_chk_error' in GDB and then easily step into broken comparators. Furthermore, it might be nice to investigate if the entire checking loop can be factored out somehow into a non-template function to avoid having multiple instantiations of it for different hashtable template parameters. On my first attempt to submit qsort_chk, Richi asked how much it slows down stage2, do you have some data on the cost of this hashtable checking? I think it is possible to optimize this a bit: instead of checking on insertions, check on deletions and when destroying the table (with care to do n^2/2 rather than n^2 tests). Although, such approach would miss errors on hashtables that are never destroyed (leaked or deliberately not deleted). Alexander ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2018-10-29 14:28 ` Alexander Monakov @ 2018-10-29 15:56 ` Martin Liška 2018-10-30 10:32 ` Jakub Jelinek 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2018-10-29 15:56 UTC (permalink / raw) To: Alexander Monakov Cc: gcc-patches, Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 2443 bytes --] On 10/29/18 2:53 PM, Alexander Monakov wrote: > On Mon, 29 Oct 2018, Martin LiÅ¡ka wrote: >> My question is whether we want to have in GCC 9 time frame or should I wait with that? >> Does it worth implementing? > > This is cool, thanks! A few questions/comments on the patch. Hi. Thanks for support. > > I think there are places that use libiberty C-style hashtab (htab_t), would it > make sense to have this kind of checking for them as well? I think it's going > to be more complicated though, no need to do both in one step. Sure, that can be also added in the future! > > I would recommend to factor out the error reporting path into a separate > non-template function, e.g. hashtab_chk_error. See how qsort_chk_error > has "cold" and "noreturn" attributes and invokes problematic comparators: > the idea was that a developer can 'break qsort_chk_error' in GDB and then > easily step into broken comparators. I did so. > > Furthermore, it might be nice to investigate if the entire checking loop can > be factored out somehow into a non-template function to avoid having multiple > instantiations of it for different hashtable template parameters. Note that Jakub is preferring to enable the checking only in non-release builds. Thus I've guarded the checking with #if ENABLE_EXTRA_CHECKING. That said I won't care much about saving a binary size. The checking function is dependent on template arguments, so the avoidance is probably not possible. > > On my first attempt to submit qsort_chk, Richi asked how much it slows down > stage2, do you have some data on the cost of this hashtable checking? It does not survive bootstrap right now (too many cselib_find_slot failures). > > I think it is possible to optimize this a bit: instead of checking on > insertions, check on deletions and when destroying the table (with care to do > n^2/2 rather than n^2 tests). Although, such approach would miss errors on > hashtables that are never destroyed (leaked or deliberately not deleted). Interesting idea. Benefit of the current approach is that the ICE is triggered as soon as first bad element is inserted into a hast table. Which means it's easier to debug. One problem with the destruction is that one equals function is defined on: Descriptor::equal (Descriptor::value_type, Descriptor::compare_type). One can have a table where these are not equal and so that equals can't be called. Martin > > Alexander > [-- Attachment #2: 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch --] [-- Type: text/x-patch, Size: 2427 bytes --] From 9afc42d324820cc0fc8a8a8b2d9cccd218734d10 Mon Sep 17 00:00:00 2001 From: marxin <mliska@suse.cz> Date: Mon, 29 Oct 2018 09:38:21 +0100 Subject: [PATCH] Sanitize equals and hash functions in hash-tables. --- gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/gcc/hash-table.h b/gcc/hash-table.h index bd83345c7b8..43adfac2dc0 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -503,6 +503,7 @@ private: value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; value_type *find_empty_slot_for_expand (hashval_t); + void verify (const compare_type &comparable, hashval_t hash); bool too_empty_p (unsigned int); void expand (); static bool is_deleted (value_type &v) @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) expand (); - m_searches++; +#if ENABLE_EXTRA_CHECKING + if (insert == INSERT) + verify (comparable, hash); +#endif + m_searches++; value_type *first_deleted_slot = NULL; hashval_t index = hash_table_mod1 (hash, m_size_prime_index); hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> return &m_entries[index]; } +#if ENABLE_EXTRA_CHECKING + +/* Report a hash table checking error. */ + +ATTRIBUTE_NORETURN ATTRIBUTE_COLD +static void +hashtab_chk_error () +{ + fprintf (stderr, "hash table checking failed: " + "equal operator returns true for a pair " + "of values with a different hash value"); + gcc_unreachable (); +} + +/* Verify that all existing elements in th hash table which are + equal to COMPARABLE have an equal HASH value provided as argument. */ + +template<typename Descriptor, template<typename Type> class Allocator> +void +hash_table<Descriptor, Allocator> +::verify (const compare_type &comparable, hashval_t hash) +{ + for (size_t i = 0; i < m_size; i++) + { + value_type *entry = &m_entries[i]; + if (!is_empty (*entry) && !is_deleted (*entry) + && Descriptor::equal (*entry, comparable) + && hash != Descriptor::hash (*entry)) + hashtab_chk_error (); + } +} +#endif + /* This function deletes an element with the given COMPARABLE value from hash table starting with the given HASH. If there is no matching element in the hash table, this function does nothing. */ -- 2.19.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2018-10-29 15:56 ` Martin Liška @ 2018-10-30 10:32 ` Jakub Jelinek 2018-10-30 14:17 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Jakub Jelinek @ 2018-10-30 10:32 UTC (permalink / raw) To: Martin Liška Cc: Alexander Monakov, gcc-patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: > +hashtab_chk_error () > +{ > + fprintf (stderr, "hash table checking failed: " > + "equal operator returns true for a pair " > + "of values with a different hash value"); BTW, either use internal_error here, or at least if using fprintf terminate with \n, in your recent mail I saw: ...different hash valueduring RTL pass: vartrack ^^^^^^ > + gcc_unreachable (); > +} Jakub ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2018-10-30 10:32 ` Jakub Jelinek @ 2018-10-30 14:17 ` Martin Liška 2018-11-07 22:24 ` Jeff Law 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2018-10-30 14:17 UTC (permalink / raw) To: Jakub Jelinek Cc: Alexander Monakov, gcc-patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 586 bytes --] On 10/30/18 11:03 AM, Jakub Jelinek wrote: > On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >> +hashtab_chk_error () >> +{ >> + fprintf (stderr, "hash table checking failed: " >> + "equal operator returns true for a pair " >> + "of values with a different hash value"); > > BTW, either use internal_error here, or at least if using fprintf > terminate with \n, in your recent mail I saw: > ...different hash valueduring RTL pass: vartrack > ^^^^^^ Sure, fixed in attached patch. Martin > >> + gcc_unreachable (); >> +} > > Jakub > [-- Attachment #2: 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch --] [-- Type: text/x-patch, Size: 2429 bytes --] From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 From: marxin <mliska@suse.cz> Date: Mon, 29 Oct 2018 09:38:21 +0100 Subject: [PATCH] Sanitize equals and hash functions in hash-tables. --- gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/gcc/hash-table.h b/gcc/hash-table.h index bd83345c7b8..694eedfc4be 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -503,6 +503,7 @@ private: value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; value_type *find_empty_slot_for_expand (hashval_t); + void verify (const compare_type &comparable, hashval_t hash); bool too_empty_p (unsigned int); void expand (); static bool is_deleted (value_type &v) @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) expand (); - m_searches++; +#if ENABLE_EXTRA_CHECKING + if (insert == INSERT) + verify (comparable, hash); +#endif + m_searches++; value_type *first_deleted_slot = NULL; hashval_t index = hash_table_mod1 (hash, m_size_prime_index); hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> return &m_entries[index]; } +#if ENABLE_EXTRA_CHECKING + +/* Report a hash table checking error. */ + +ATTRIBUTE_NORETURN ATTRIBUTE_COLD +static void +hashtab_chk_error () +{ + fprintf (stderr, "hash table checking failed: " + "equal operator returns true for a pair " + "of values with a different hash value\n"); + gcc_unreachable (); +} + +/* Verify that all existing elements in th hash table which are + equal to COMPARABLE have an equal HASH value provided as argument. */ + +template<typename Descriptor, template<typename Type> class Allocator> +void +hash_table<Descriptor, Allocator> +::verify (const compare_type &comparable, hashval_t hash) +{ + for (size_t i = 0; i < m_size; i++) + { + value_type *entry = &m_entries[i]; + if (!is_empty (*entry) && !is_deleted (*entry) + && hash != Descriptor::hash (*entry) + && Descriptor::equal (*entry, comparable)) + hashtab_chk_error (); + } +} +#endif + /* This function deletes an element with the given COMPARABLE value from hash table starting with the given HASH. If there is no matching element in the hash table, this function does nothing. */ -- 2.19.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2018-10-30 14:17 ` Martin Liška @ 2018-11-07 22:24 ` Jeff Law 2018-11-07 22:44 ` Jakub Jelinek 2018-11-08 8:56 ` Martin Liška 0 siblings, 2 replies; 53+ messages in thread From: Jeff Law @ 2018-11-07 22:24 UTC (permalink / raw) To: Martin Liška, Jakub Jelinek Cc: Alexander Monakov, gcc-patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: > On 10/30/18 11:03 AM, Jakub Jelinek wrote: >> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>> +hashtab_chk_error () >>> +{ >>> + fprintf (stderr, "hash table checking failed: " >>> + "equal operator returns true for a pair " >>> + "of values with a different hash value"); >> BTW, either use internal_error here, or at least if using fprintf >> terminate with \n, in your recent mail I saw: >> ...different hash valueduring RTL pass: vartrack >> ^^^^^^ > Sure, fixed in attached patch. > > Martin > >>> + gcc_unreachable (); >>> +} >> Jakub >> > > 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > > From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > From: marxin <mliska@suse.cz> > Date: Mon, 29 Oct 2018 09:38:21 +0100 > Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > > --- > gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > 1 file changed, 39 insertions(+), 1 deletion(-) > > diff --git a/gcc/hash-table.h b/gcc/hash-table.h > index bd83345c7b8..694eedfc4be 100644 > --- a/gcc/hash-table.h > +++ b/gcc/hash-table.h > @@ -503,6 +503,7 @@ private: > > value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > value_type *find_empty_slot_for_expand (hashval_t); > + void verify (const compare_type &comparable, hashval_t hash); > bool too_empty_p (unsigned int); > void expand (); > static bool is_deleted (value_type &v) > @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > expand (); > > - m_searches++; > +#if ENABLE_EXTRA_CHECKING > + if (insert == INSERT) > + verify (comparable, hash); > +#endif > > + m_searches++; > value_type *first_deleted_slot = NULL; > hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > return &m_entries[index]; > } > > +#if ENABLE_EXTRA_CHECKING > + > +/* Report a hash table checking error. */ > + > +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > +static void > +hashtab_chk_error () > +{ > + fprintf (stderr, "hash table checking failed: " > + "equal operator returns true for a pair " > + "of values with a different hash value\n"); > + gcc_unreachable (); > +} I think an internal_error here is probably still better than a simple fprintf, even if the fprintf is terminated with a \n :-) The question then becomes can we bootstrap with this stuff enabled and if not, are we likely to soon? It'd be a shame to put it into EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING because we've got too many bugs to fix. > + > +/* Verify that all existing elements in th hash table which are s/th/the/ Jeff ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2018-11-07 22:24 ` Jeff Law @ 2018-11-07 22:44 ` Jakub Jelinek 2018-11-08 8:56 ` Martin Liška 1 sibling, 0 replies; 53+ messages in thread From: Jakub Jelinek @ 2018-11-07 22:44 UTC (permalink / raw) To: Jeff Law Cc: Martin Liška, Alexander Monakov, gcc-patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Wed, Nov 07, 2018 at 03:23:55PM -0700, Jeff Law wrote: > > @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > > if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > > expand (); > > > > - m_searches++; > > +#if ENABLE_EXTRA_CHECKING > > + if (insert == INSERT) > > + verify (comparable, hash); > > +#endif Plus formatting, the above is indented too much. Jakub ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2018-11-07 22:24 ` Jeff Law 2018-11-07 22:44 ` Jakub Jelinek @ 2018-11-08 8:56 ` Martin Liška 2019-05-13 7:42 ` Martin Liška 1 sibling, 1 reply; 53+ messages in thread From: Martin Liška @ 2018-11-08 8:56 UTC (permalink / raw) To: Jeff Law, Jakub Jelinek Cc: Alexander Monakov, gcc-patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On 11/7/18 11:23 PM, Jeff Law wrote: > On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>> +hashtab_chk_error () >>>> +{ >>>> + fprintf (stderr, "hash table checking failed: " >>>> + "equal operator returns true for a pair " >>>> + "of values with a different hash value"); >>> BTW, either use internal_error here, or at least if using fprintf >>> terminate with \n, in your recent mail I saw: >>> ...different hash valueduring RTL pass: vartrack >>> ^^^^^^ >> Sure, fixed in attached patch. >> >> Martin >> >>>> + gcc_unreachable (); >>>> +} >>> Jakub >>> >> >> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >> >> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >> From: marxin <mliska@suse.cz> >> Date: Mon, 29 Oct 2018 09:38:21 +0100 >> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >> >> --- >> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >> 1 file changed, 39 insertions(+), 1 deletion(-) >> >> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >> index bd83345c7b8..694eedfc4be 100644 >> --- a/gcc/hash-table.h >> +++ b/gcc/hash-table.h >> @@ -503,6 +503,7 @@ private: >> >> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >> value_type *find_empty_slot_for_expand (hashval_t); >> + void verify (const compare_type &comparable, hashval_t hash); >> bool too_empty_p (unsigned int); >> void expand (); >> static bool is_deleted (value_type &v) >> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >> expand (); >> >> - m_searches++; >> +#if ENABLE_EXTRA_CHECKING >> + if (insert == INSERT) >> + verify (comparable, hash); >> +#endif >> >> + m_searches++; >> value_type *first_deleted_slot = NULL; >> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >> return &m_entries[index]; >> } >> >> +#if ENABLE_EXTRA_CHECKING >> + >> +/* Report a hash table checking error. */ >> + >> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >> +static void >> +hashtab_chk_error () >> +{ >> + fprintf (stderr, "hash table checking failed: " >> + "equal operator returns true for a pair " >> + "of values with a different hash value\n"); >> + gcc_unreachable (); >> +} > I think an internal_error here is probably still better than a simple > fprintf, even if the fprintf is terminated with a \n :-) Fully agree with that, but I see a lot of build errors when using internal_error. > > The question then becomes can we bootstrap with this stuff enabled and > if not, are we likely to soon? It'd be a shame to put it into > EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > because we've got too many bugs to fix. Unfortunately it's blocked with these 2 PRs: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 I'm fine with having the patch in in next stage1 after the problems will be fixed. Martin > >> + >> +/* Verify that all existing elements in th hash table which are > s/th/the/ > > > Jeff > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2018-11-08 8:56 ` Martin Liška @ 2019-05-13 7:42 ` Martin Liška 2019-05-20 17:26 ` Jason Merrill 2019-05-20 22:07 ` Jeff Law 0 siblings, 2 replies; 53+ messages in thread From: Martin Liška @ 2019-05-13 7:42 UTC (permalink / raw) To: Jeff Law, Jakub Jelinek Cc: Alexander Monakov, gcc-patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 4585 bytes --] On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: > On 11/7/18 11:23 PM, Jeff Law wrote: >> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>> +hashtab_chk_error () >>>>> +{ >>>>> + fprintf (stderr, "hash table checking failed: " >>>>> + "equal operator returns true for a pair " >>>>> + "of values with a different hash value"); >>>> BTW, either use internal_error here, or at least if using fprintf >>>> terminate with \n, in your recent mail I saw: >>>> ...different hash valueduring RTL pass: vartrack >>>> ^^^^^^ >>> Sure, fixed in attached patch. >>> >>> Martin >>> >>>>> + gcc_unreachable (); >>>>> +} >>>> Jakub >>>> >>> >>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>> >>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>> From: marxin <mliska@suse.cz> >>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>> >>> --- >>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>> 1 file changed, 39 insertions(+), 1 deletion(-) >>> >>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>> index bd83345c7b8..694eedfc4be 100644 >>> --- a/gcc/hash-table.h >>> +++ b/gcc/hash-table.h >>> @@ -503,6 +503,7 @@ private: >>> >>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>> value_type *find_empty_slot_for_expand (hashval_t); >>> + void verify (const compare_type &comparable, hashval_t hash); >>> bool too_empty_p (unsigned int); >>> void expand (); >>> static bool is_deleted (value_type &v) >>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>> expand (); >>> >>> - m_searches++; >>> +#if ENABLE_EXTRA_CHECKING >>> + if (insert == INSERT) >>> + verify (comparable, hash); >>> +#endif >>> >>> + m_searches++; >>> value_type *first_deleted_slot = NULL; >>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>> return &m_entries[index]; >>> } >>> >>> +#if ENABLE_EXTRA_CHECKING >>> + >>> +/* Report a hash table checking error. */ >>> + >>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>> +static void >>> +hashtab_chk_error () >>> +{ >>> + fprintf (stderr, "hash table checking failed: " >>> + "equal operator returns true for a pair " >>> + "of values with a different hash value\n"); >>> + gcc_unreachable (); >>> +} >> I think an internal_error here is probably still better than a simple >> fprintf, even if the fprintf is terminated with a \n :-) > > Fully agree with that, but I see a lot of build errors when using internal_error. > >> >> The question then becomes can we bootstrap with this stuff enabled and >> if not, are we likely to soon? It'd be a shame to put it into >> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >> because we've got too many bugs to fix. > > Unfortunately it's blocked with these 2 PRs: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 Hi. I've just added one more PR: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 I'm sending updated version of the patch that provides a disablement for the 3 PRs with a new function disable_sanitize_eq_and_hash. With that I can bootstrap and finish tests. However, I've done that with a patch limits maximal number of checks: diff --git a/gcc/hash-table.h b/gcc/hash-table.h index dc24fea6405..57564914e31 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -1027,7 +1027,7 @@ void hash_table<Descriptor, Lazy, Allocator> ::verify (const compare_type &comparable, hashval_t hash) { - for (size_t i = 0; i < m_size; i++) + for (size_t i = 0; i < MIN (m_size, 1000); i++) { value_type *entry = &m_entries[i]; if (!is_empty (*entry) && !is_deleted (*entry) Without that it would be probably terribly slow. Moreover, one probably does not want that with an extra checking, but with an extra-extra checking. Ideas about where to enable it? Would it be possible to add the sanitization with the aforementioned disablement? Thanks, Martin > > I'm fine with having the patch in in next stage1 after the problems will > be fixed. > > Martin > >> >>> + >>> +/* Verify that all existing elements in th hash table which are >> s/th/the/ >> >> >> Jeff >> > [-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --] [-- Type: text/x-patch, Size: 5391 bytes --] From fee775b5f4443e6eaeb4028e9a8922ea4ec8703f Mon Sep 17 00:00:00 2001 From: marxin <mliska@suse.cz> Date: Mon, 13 May 2019 07:16:22 +0200 Subject: [PATCH] Enable sanitization for hash tables. --- gcc/cp/pt.c | 4 ++++ gcc/cselib.c | 2 ++ gcc/hash-table.h | 53 ++++++++++++++++++++++++++++++++++++++++-- gcc/tree-ssa-loop-im.c | 2 ++ 4 files changed, 59 insertions(+), 2 deletions(-) diff --git a/gcc/cp/pt.c b/gcc/cp/pt.c index d6976e08690..db9c953e3dd 100644 --- a/gcc/cp/pt.c +++ b/gcc/cp/pt.c @@ -28320,7 +28320,11 @@ void init_template_processing (void) { decl_specializations = hash_table<spec_hasher>::create_ggc (37); + /* FIXME: enable sanitization */ + decl_specializations->disable_sanitize_eq_and_hash (); type_specializations = hash_table<spec_hasher>::create_ggc (37); + /* FIXME: enable sanitization */ + type_specializations->disable_sanitize_eq_and_hash (); if (cxx_dialect >= cxx11) declare_integer_pack (); diff --git a/gcc/cselib.c b/gcc/cselib.c index 84c17c23f6d..a60865c810a 100644 --- a/gcc/cselib.c +++ b/gcc/cselib.c @@ -2859,6 +2859,8 @@ cselib_init (int record_what) used_regs = XNEWVEC (unsigned int, cselib_nregs); n_used_regs = 0; cselib_hash_table = new hash_table<cselib_hasher> (31); + /* FIXME: enable sanitization */ + cselib_hash_table->disable_sanitize_eq_and_hash (); if (cselib_preserve_constants) cselib_preserved_hash_table = new hash_table<cselib_hasher> (31); next_uid = 1; diff --git a/gcc/hash-table.h b/gcc/hash-table.h index 4178616478e..dc24fea6405 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -405,6 +405,9 @@ public: /* Return true when there are no elements in this hash table. */ bool is_empty () const { return elements () == 0; } + /* Disable equal and hash function sanitization. */ + void disable_sanitize_eq_and_hash (void) { m_sanitize_eq_and_hash = false; } + /* This function clears a specified SLOT in a hash table. It is useful when you've already done the lookup and don't want to do it again. */ @@ -516,6 +519,7 @@ private: value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; value_type *find_empty_slot_for_expand (hashval_t); + void verify (const compare_type &comparable, hashval_t hash); bool too_empty_p (unsigned int); void expand (); static bool is_deleted (value_type &v) @@ -564,6 +568,9 @@ private: /* if m_entries is stored in ggc memory. */ bool m_ggc; + /* True if the table should be sanitized for equal and hash functions. */ + bool m_sanitize_eq_and_hash; + /* If we should gather memory statistics for the table. */ #if GATHER_STATISTICS bool m_gather_mem_stats; @@ -591,7 +598,7 @@ hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0), - m_ggc (ggc) + m_ggc (ggc), m_sanitize_eq_and_hash (true) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -941,8 +948,12 @@ hash_table<Descriptor, Lazy, Allocator> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) expand (); - m_searches++; +#if ENABLE_EXTRA_CHECKING + if (m_sanitize_eq_and_hash && insert == INSERT) + verify (comparable, hash); +#endif + m_searches++; value_type *first_deleted_slot = NULL; hashval_t index = hash_table_mod1 (hash, m_size_prime_index); hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); @@ -989,6 +1000,44 @@ hash_table<Descriptor, Lazy, Allocator> return &m_entries[index]; } +#if ENABLE_EXTRA_CHECKING + +/* Report a hash table checking error. */ + +ATTRIBUTE_NORETURN ATTRIBUTE_COLD +static void +hashtab_chk_error () +{ + fprintf (stderr, "hash table checking failed: " + "equal operator returns true for a pair " + "of values with a different hash value\n"); +#ifndef GENERATOR_FILE + gcc_unreachable (); +#else + exit (1); +#endif +} + +/* Verify that all existing elements in th hash table which are + equal to COMPARABLE have an equal HASH value provided as argument. */ + +template<typename Descriptor, bool Lazy, + template<typename Type> class Allocator> +void +hash_table<Descriptor, Lazy, Allocator> +::verify (const compare_type &comparable, hashval_t hash) +{ + for (size_t i = 0; i < m_size; i++) + { + value_type *entry = &m_entries[i]; + if (!is_empty (*entry) && !is_deleted (*entry) + && hash != Descriptor::hash (*entry) + && Descriptor::equal (*entry, comparable)) + hashtab_chk_error (); + } +} +#endif + /* This function deletes an element with the given COMPARABLE value from hash table starting with the given HASH. If there is no matching element in the hash table, this function does nothing. */ diff --git a/gcc/tree-ssa-loop-im.c b/gcc/tree-ssa-loop-im.c index 56d8e8e4330..1bef286d46b 100644 --- a/gcc/tree-ssa-loop-im.c +++ b/gcc/tree-ssa-loop-im.c @@ -2565,6 +2565,8 @@ tree_ssa_lim_initialize (void) alloc_aux_for_edges (0); memory_accesses.refs = new hash_table<mem_ref_hasher> (100); + /* FIXME: enable sanitization */ + memory_accesses.refs->disable_sanitize_eq_and_hash (); memory_accesses.refs_list.create (100); /* Allocate a special, unanalyzable mem-ref with ID zero. */ memory_accesses.refs_list.quick_push -- 2.21.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-13 7:42 ` Martin Liška @ 2019-05-20 17:26 ` Jason Merrill 2019-05-20 22:07 ` Jeff Law 1 sibling, 0 replies; 53+ messages in thread From: Jason Merrill @ 2019-05-20 17:26 UTC (permalink / raw) To: Martin Liška, Jeff Law, Jakub Jelinek Cc: Alexander Monakov, gcc-patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On 5/13/19 3:41 AM, Martin LiÅ¡ka wrote: > On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >> On 11/7/18 11:23 PM, Jeff Law wrote: >>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>> +hashtab_chk_error () >>>>>> +{ >>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>> + "equal operator returns true for a pair " >>>>>> + "of values with a different hash value"); >>>>> BTW, either use internal_error here, or at least if using fprintf >>>>> terminate with \n, in your recent mail I saw: >>>>> ...different hash valueduring RTL pass: vartrack >>>>> ^^^^^^ >>>> Sure, fixed in attached patch. >>>> >>>> Martin >>>> >>>>>> + gcc_unreachable (); >>>>>> +} >>>>> Jakub >>>>> >>>> >>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>> >>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>> From: marxin <mliska@suse.cz> >>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>> >>>> --- >>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>> index bd83345c7b8..694eedfc4be 100644 >>>> --- a/gcc/hash-table.h >>>> +++ b/gcc/hash-table.h >>>> @@ -503,6 +503,7 @@ private: >>>> >>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>> value_type *find_empty_slot_for_expand (hashval_t); >>>> + void verify (const compare_type &comparable, hashval_t hash); >>>> bool too_empty_p (unsigned int); >>>> void expand (); >>>> static bool is_deleted (value_type &v) >>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>> expand (); >>>> >>>> - m_searches++; >>>> +#if ENABLE_EXTRA_CHECKING >>>> + if (insert == INSERT) >>>> + verify (comparable, hash); >>>> +#endif >>>> >>>> + m_searches++; >>>> value_type *first_deleted_slot = NULL; >>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>> return &m_entries[index]; >>>> } >>>> >>>> +#if ENABLE_EXTRA_CHECKING >>>> + >>>> +/* Report a hash table checking error. */ >>>> + >>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>> +static void >>>> +hashtab_chk_error () >>>> +{ >>>> + fprintf (stderr, "hash table checking failed: " >>>> + "equal operator returns true for a pair " >>>> + "of values with a different hash value\n"); >>>> + gcc_unreachable (); >>>> +} >>> I think an internal_error here is probably still better than a simple >>> fprintf, even if the fprintf is terminated with a \n :-) >> >> Fully agree with that, but I see a lot of build errors when using internal_error. >> >>> >>> The question then becomes can we bootstrap with this stuff enabled and >>> if not, are we likely to soon? It'd be a shame to put it into >>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>> because we've got too many bugs to fix. >> >> Unfortunately it's blocked with these 2 PRs: >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > > Hi. > > I've just added one more PR: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > > I'm sending updated version of the patch that provides a disablement for the 3 PRs > with a new function disable_sanitize_eq_and_hash. > > With that I can bootstrap and finish tests. However, I've done that with a patch > limits maximal number of checks: > > diff --git a/gcc/hash-table.h b/gcc/hash-table.h > index dc24fea6405..57564914e31 100644 > --- a/gcc/hash-table.h > +++ b/gcc/hash-table.h > @@ -1027,7 +1027,7 @@ void > hash_table<Descriptor, Lazy, Allocator> > ::verify (const compare_type &comparable, hashval_t hash) > { > - for (size_t i = 0; i < m_size; i++) > + for (size_t i = 0; i < MIN (m_size, 1000); i++) > { > value_type *entry = &m_entries[i]; > if (!is_empty (*entry) && !is_deleted (*entry) > > Without that it would be probably terribly slow. Moreover, one probably does not want > that with an extra checking, but with an extra-extra checking. Ideas about where to enable > it? As a --param? I use --param ggc-min-heapsize=0 --param ggc-min-expand=0 to catch GC issues, and this seems similar. > + /* FIXME: enable sanitization */ Please add PR numbers to these comments. > +#if ENABLE_EXTRA_CHECKING The documentation describes this flag as > @samp{extra} adds for @samp{misc} checking extra checks that might affect > code generation and should therefore not differ between stage1 and later > stages. which doesn't seem to apply here. Jason ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-13 7:42 ` Martin Liška 2019-05-20 17:26 ` Jason Merrill @ 2019-05-20 22:07 ` Jeff Law 2019-05-21 9:38 ` Richard Biener 1 sibling, 1 reply; 53+ messages in thread From: Jeff Law @ 2019-05-20 22:07 UTC (permalink / raw) To: Martin Liška, Jakub Jelinek Cc: Alexander Monakov, gcc-patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: > On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >> On 11/7/18 11:23 PM, Jeff Law wrote: >>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>> +hashtab_chk_error () >>>>>> +{ >>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>> + "equal operator returns true for a pair " >>>>>> + "of values with a different hash value"); >>>>> BTW, either use internal_error here, or at least if using fprintf >>>>> terminate with \n, in your recent mail I saw: >>>>> ...different hash valueduring RTL pass: vartrack >>>>> ^^^^^^ >>>> Sure, fixed in attached patch. >>>> >>>> Martin >>>> >>>>>> + gcc_unreachable (); >>>>>> +} >>>>> Jakub >>>>> >>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>> >>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>> From: marxin <mliska@suse.cz> >>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>> >>>> --- >>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>> index bd83345c7b8..694eedfc4be 100644 >>>> --- a/gcc/hash-table.h >>>> +++ b/gcc/hash-table.h >>>> @@ -503,6 +503,7 @@ private: >>>> >>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>> value_type *find_empty_slot_for_expand (hashval_t); >>>> + void verify (const compare_type &comparable, hashval_t hash); >>>> bool too_empty_p (unsigned int); >>>> void expand (); >>>> static bool is_deleted (value_type &v) >>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>> expand (); >>>> >>>> - m_searches++; >>>> +#if ENABLE_EXTRA_CHECKING >>>> + if (insert == INSERT) >>>> + verify (comparable, hash); >>>> +#endif >>>> >>>> + m_searches++; >>>> value_type *first_deleted_slot = NULL; >>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>> return &m_entries[index]; >>>> } >>>> >>>> +#if ENABLE_EXTRA_CHECKING >>>> + >>>> +/* Report a hash table checking error. */ >>>> + >>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>> +static void >>>> +hashtab_chk_error () >>>> +{ >>>> + fprintf (stderr, "hash table checking failed: " >>>> + "equal operator returns true for a pair " >>>> + "of values with a different hash value\n"); >>>> + gcc_unreachable (); >>>> +} >>> I think an internal_error here is probably still better than a simple >>> fprintf, even if the fprintf is terminated with a \n :-) >> Fully agree with that, but I see a lot of build errors when using internal_error. >> >>> The question then becomes can we bootstrap with this stuff enabled and >>> if not, are we likely to soon? It'd be a shame to put it into >>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>> because we've got too many bugs to fix. >> Unfortunately it's blocked with these 2 PRs: >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > Hi. > > I've just added one more PR: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > > I'm sending updated version of the patch that provides a disablement for the 3 PRs > with a new function disable_sanitize_eq_and_hash. > > With that I can bootstrap and finish tests. However, I've done that with a patch > limits maximal number of checks: So rather than call the disable_sanitize_eq_and_hash, can you have its state set up when you instantiate the object? It's not a huge deal, just thinking about loud. So how do we want to go forward, particularly the EXTRA_EXTRA checking issue :-) Jeff ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-20 22:07 ` Jeff Law @ 2019-05-21 9:38 ` Richard Biener 2019-05-21 11:02 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-05-21 9:38 UTC (permalink / raw) To: Jeff Law Cc: Martin Liška, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > > On 5/13/19 1:41 AM, Martin Liška wrote: > > On 11/8/18 9:56 AM, Martin Liška wrote: > >> On 11/7/18 11:23 PM, Jeff Law wrote: > >>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>> +hashtab_chk_error () > >>>>>> +{ > >>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>> + "equal operator returns true for a pair " > >>>>>> + "of values with a different hash value"); > >>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>> terminate with \n, in your recent mail I saw: > >>>>> ...different hash valueduring RTL pass: vartrack > >>>>> ^^^^^^ > >>>> Sure, fixed in attached patch. > >>>> > >>>> Martin > >>>> > >>>>>> + gcc_unreachable (); > >>>>>> +} > >>>>> Jakub > >>>>> > >>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>> > >>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>> From: marxin <mliska@suse.cz> > >>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>> > >>>> --- > >>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>> > >>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>> index bd83345c7b8..694eedfc4be 100644 > >>>> --- a/gcc/hash-table.h > >>>> +++ b/gcc/hash-table.h > >>>> @@ -503,6 +503,7 @@ private: > >>>> > >>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>> bool too_empty_p (unsigned int); > >>>> void expand (); > >>>> static bool is_deleted (value_type &v) > >>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>> expand (); > >>>> > >>>> - m_searches++; > >>>> +#if ENABLE_EXTRA_CHECKING > >>>> + if (insert == INSERT) > >>>> + verify (comparable, hash); > >>>> +#endif > >>>> > >>>> + m_searches++; > >>>> value_type *first_deleted_slot = NULL; > >>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>> return &m_entries[index]; > >>>> } > >>>> > >>>> +#if ENABLE_EXTRA_CHECKING > >>>> + > >>>> +/* Report a hash table checking error. */ > >>>> + > >>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>> +static void > >>>> +hashtab_chk_error () > >>>> +{ > >>>> + fprintf (stderr, "hash table checking failed: " > >>>> + "equal operator returns true for a pair " > >>>> + "of values with a different hash value\n"); > >>>> + gcc_unreachable (); > >>>> +} > >>> I think an internal_error here is probably still better than a simple > >>> fprintf, even if the fprintf is terminated with a \n :-) > >> Fully agree with that, but I see a lot of build errors when using internal_error. > >> > >>> The question then becomes can we bootstrap with this stuff enabled and > >>> if not, are we likely to soon? It'd be a shame to put it into > >>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>> because we've got too many bugs to fix. > >> Unfortunately it's blocked with these 2 PRs: > >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > > Hi. > > > > I've just added one more PR: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > > > > I'm sending updated version of the patch that provides a disablement for the 3 PRs > > with a new function disable_sanitize_eq_and_hash. > > > > With that I can bootstrap and finish tests. However, I've done that with a patch > > limits maximal number of checks: > So rather than call the disable_sanitize_eq_and_hash, can you have its > state set up when you instantiate the object? It's not a huge deal, > just thinking about loud. > > > > So how do we want to go forward, particularly the EXTRA_EXTRA checking > issue :-) There is at least one PR where we have a table where elements _in_ the table are never compared against each other but always against another object (I guess that's usual even), but the setup is in a way that the comparison function only works with those. With the patch we verify hashing/comparison for something that is never used. So - wouldn't it be more "correct" to only verify comparison/hashing at lookup time, using the object from the lookup and verify that against all other elements? Richard. > > Jeff ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-21 9:38 ` Richard Biener @ 2019-05-21 11:02 ` Martin Liška 2019-05-21 11:52 ` Richard Biener 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2019-05-21 11:02 UTC (permalink / raw) To: Richard Biener, Jeff Law Cc: Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 5414 bytes --] On 5/21/19 11:38 AM, Richard Biener wrote: > On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >> >> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>> +hashtab_chk_error () >>>>>>>> +{ >>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>> + "equal operator returns true for a pair " >>>>>>>> + "of values with a different hash value"); >>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>> terminate with \n, in your recent mail I saw: >>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>> ^^^^^^ >>>>>> Sure, fixed in attached patch. >>>>>> >>>>>> Martin >>>>>> >>>>>>>> + gcc_unreachable (); >>>>>>>> +} >>>>>>> Jakub >>>>>>> >>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>> >>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>> From: marxin <mliska@suse.cz> >>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>> >>>>>> --- >>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>> >>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>> --- a/gcc/hash-table.h >>>>>> +++ b/gcc/hash-table.h >>>>>> @@ -503,6 +503,7 @@ private: >>>>>> >>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>> bool too_empty_p (unsigned int); >>>>>> void expand (); >>>>>> static bool is_deleted (value_type &v) >>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>> expand (); >>>>>> >>>>>> - m_searches++; >>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>> + if (insert == INSERT) >>>>>> + verify (comparable, hash); >>>>>> +#endif >>>>>> >>>>>> + m_searches++; >>>>>> value_type *first_deleted_slot = NULL; >>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>> return &m_entries[index]; >>>>>> } >>>>>> >>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>> + >>>>>> +/* Report a hash table checking error. */ >>>>>> + >>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>> +static void >>>>>> +hashtab_chk_error () >>>>>> +{ >>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>> + "equal operator returns true for a pair " >>>>>> + "of values with a different hash value\n"); >>>>>> + gcc_unreachable (); >>>>>> +} >>>>> I think an internal_error here is probably still better than a simple >>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>> >>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>> because we've got too many bugs to fix. >>>> Unfortunately it's blocked with these 2 PRs: >>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>> Hi. >>> >>> I've just added one more PR: >>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>> >>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>> with a new function disable_sanitize_eq_and_hash. >>> >>> With that I can bootstrap and finish tests. However, I've done that with a patch >>> limits maximal number of checks: >> So rather than call the disable_sanitize_eq_and_hash, can you have its >> state set up when you instantiate the object? It's not a huge deal, >> just thinking about loud. >> >> >> >> So how do we want to go forward, particularly the EXTRA_EXTRA checking >> issue :-) > > There is at least one PR where we have a table where elements _in_ the > table are never compared against each other but always against another > object (I guess that's usual even), but the setup is in a way that the > comparison function only works with those. With the patch we verify > hashing/comparison for something that is never used. > > So - wouldn't it be more "correct" to only verify comparison/hashing > at lookup time, using the object from the lookup and verify that against > all other elements? I don't a have problem with that. Apparently this changes fixes PR90450 and PR87847. Changes from previous version: - verification happens only when an element is searched (not inserted) - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table - new param has been introduced hash-table-verification-limit in order to limit number of elements that are compared within a table - verification happens only with flag_checking >= 2 I've been bootstrapping and testing the patch right now. Martin > > Richard. > >> >> Jeff [-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --] [-- Type: text/x-patch, Size: 8202 bytes --] From 47df69feb035672056d2ad86c53eda9c286ed92a Mon Sep 17 00:00:00 2001 From: marxin <mliska@suse.cz> Date: Mon, 13 May 2019 07:16:22 +0200 Subject: [PATCH] Enable sanitization for hash tables. --- gcc/cselib.c | 9 ++++++-- gcc/hash-set.h | 2 +- gcc/hash-table.c | 3 +++ gcc/hash-table.h | 60 ++++++++++++++++++++++++++++++++++++++++++++---- gcc/params.def | 6 +++++ gcc/toplev.c | 4 ++++ 6 files changed, 77 insertions(+), 7 deletions(-) diff --git a/gcc/cselib.c b/gcc/cselib.c index 84c17c23f6d..a1cbdec9718 100644 --- a/gcc/cselib.c +++ b/gcc/cselib.c @@ -2858,9 +2858,14 @@ cselib_init (int record_what) } used_regs = XNEWVEC (unsigned int, cselib_nregs); n_used_regs = 0; - cselib_hash_table = new hash_table<cselib_hasher> (31); + /* FIXME: enable sanitization (PR87845) */ + cselib_hash_table + = new hash_table<cselib_hasher> (31, /* ggc */ false, + /* sanitize_eq_and_hash */ false); if (cselib_preserve_constants) - cselib_preserved_hash_table = new hash_table<cselib_hasher> (31); + cselib_preserved_hash_table + = new hash_table<cselib_hasher> (31, /* ggc */ false, + /* sanitize_eq_and_hash */ false); next_uid = 1; } diff --git a/gcc/hash-set.h b/gcc/hash-set.h index de3532f5f68..d891ed78297 100644 --- a/gcc/hash-set.h +++ b/gcc/hash-set.h @@ -28,7 +28,7 @@ class hash_set public: typedef typename Traits::value_type Key; explicit hash_set (size_t n = 13, bool ggc = false CXX_MEM_STAT_INFO) - : m_table (n, ggc, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {} + : m_table (n, ggc, true, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {} /* Create a hash_set in gc memory with space for at least n elements. */ diff --git a/gcc/hash-table.c b/gcc/hash-table.c index 646a7a1c497..8e86fffa36f 100644 --- a/gcc/hash-table.c +++ b/gcc/hash-table.c @@ -74,6 +74,9 @@ struct prime_ent const prime_tab[] = { { 0xfffffffb, 0x00000006, 0x00000008, 31 } }; +/* Limit number of comparisons when calling hash_table<>::verify. */ +unsigned int hash_table_sanitize_eq_limit; + /* The following function returns an index into the above table of the nearest prime number which is greater than N, and near a power of two. */ diff --git a/gcc/hash-table.h b/gcc/hash-table.h index 4178616478e..b3e579b8822 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -295,6 +295,8 @@ struct prime_ent extern struct prime_ent const prime_tab[]; +/* Limit number of comparisons when calling hash_table<>::verify. */ +extern unsigned int hash_table_sanitize_eq_limit; /* Functions for computing hash table indexes. */ @@ -371,10 +373,12 @@ class hash_table public: explicit hash_table (size_t, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS, mem_alloc_origin origin = HASH_TABLE_ORIGIN CXX_MEM_STAT_INFO); explicit hash_table (const hash_table &, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS, mem_alloc_origin origin = HASH_TABLE_ORIGIN CXX_MEM_STAT_INFO); @@ -516,6 +520,7 @@ private: value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; value_type *find_empty_slot_for_expand (hashval_t); + void verify (const compare_type &comparable, hashval_t hash); bool too_empty_p (unsigned int); void expand (); static bool is_deleted (value_type &v) @@ -564,6 +569,9 @@ private: /* if m_entries is stored in ggc memory. */ bool m_ggc; + /* True if the table should be sanitized for equal and hash functions. */ + bool m_sanitize_eq_and_hash; + /* If we should gather memory statistics for the table. */ #if GATHER_STATISTICS bool m_gather_mem_stats; @@ -586,12 +594,13 @@ extern void dump_hash_table_loc_statistics (void); template<typename Descriptor, bool Lazy, template<typename Type> class Allocator> hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc, + bool sanitize_eq_and_hash, bool gather_mem_stats ATTRIBUTE_UNUSED, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0), - m_ggc (ggc) + m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -617,12 +626,14 @@ template<typename Descriptor, bool Lazy, template<typename Type> class Allocator> hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h, bool ggc, + bool sanitize_eq_and_hash, bool gather_mem_stats ATTRIBUTE_UNUSED, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted), - m_searches (0), m_collisions (0), m_ggc (ggc) + m_searches (0), m_collisions (0), m_ggc (ggc), + m_sanitize_eq_and_hash (sanitize_eq_and_hash) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -912,7 +923,11 @@ hash_table<Descriptor, Lazy, Allocator> entry = &m_entries[index]; if (is_empty (*entry) || (!is_deleted (*entry) && Descriptor::equal (*entry, comparable))) - return *entry; + { + if (m_sanitize_eq_and_hash) + verify (comparable, hash); + return *entry; + } } } @@ -941,8 +956,10 @@ hash_table<Descriptor, Lazy, Allocator> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) expand (); - m_searches++; + if (m_sanitize_eq_and_hash && insert == NO_INSERT) + verify (comparable, hash); + m_searches++; value_type *first_deleted_slot = NULL; hashval_t index = hash_table_mod1 (hash, m_size_prime_index); hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); @@ -989,6 +1006,41 @@ hash_table<Descriptor, Lazy, Allocator> return &m_entries[index]; } +/* Report a hash table checking error. */ + +ATTRIBUTE_NORETURN ATTRIBUTE_COLD +static void +hashtab_chk_error () +{ + fprintf (stderr, "hash table checking failed: " + "equal operator returns true for a pair " + "of values with a different hash value\n"); +#ifndef GENERATOR_FILE + gcc_unreachable (); +#else + exit (1); +#endif +} + +/* Verify that all existing elements in th hash table which are + equal to COMPARABLE have an equal HASH value provided as argument. */ + +template<typename Descriptor, bool Lazy, + template<typename Type> class Allocator> +void +hash_table<Descriptor, Lazy, Allocator> +::verify (const compare_type &comparable, hashval_t hash) +{ + for (size_t i = 0; i < MIN (hash_table_sanitize_eq_limit, m_size); i++) + { + value_type *entry = &m_entries[i]; + if (!is_empty (*entry) && !is_deleted (*entry) + && hash != Descriptor::hash (*entry) + && Descriptor::equal (*entry, comparable)) + hashtab_chk_error (); + } +} + /* This function deletes an element with the given COMPARABLE value from hash table starting with the given HASH. If there is no matching element in the hash table, this function does nothing. */ diff --git a/gcc/params.def b/gcc/params.def index 6b7f7eb5bae..e5ca6ff45e4 100644 --- a/gcc/params.def +++ b/gcc/params.def @@ -1439,6 +1439,12 @@ DEFPARAM(PARAM_GIMPLE_FE_COMPUTED_HOT_BB_THRESHOLD, " The parameter is used only in GIMPLE FE.", 0, 0, 0) +DEFPARAM(PARAM_HASH_TABLE_VERIFICATION_LIMIT, + "hash-table-verification-limit", + "The number of elements for which hash table verification is done for " + "each searched element.", + 100, 0, 0) + /* Local variables: diff --git a/gcc/toplev.c b/gcc/toplev.c index d300ac2ec89..116be7be395 100644 --- a/gcc/toplev.c +++ b/gcc/toplev.c @@ -1799,6 +1799,10 @@ process_options (void) optimization_default_node = build_optimization_node (&global_options); optimization_current_node = optimization_default_node; + if (flag_checking >= 2) + hash_table_sanitize_eq_limit + = PARAM_VALUE (PARAM_HASH_TABLE_VERIFICATION_LIMIT); + /* Please don't change global_options after this point, those changes won't be reflected in optimization_{default,current}_node. */ } -- 2.21.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-21 11:02 ` Martin Liška @ 2019-05-21 11:52 ` Richard Biener 2019-05-22 9:13 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-05-21 11:52 UTC (permalink / raw) To: Martin Liška Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > > On 5/21/19 11:38 AM, Richard Biener wrote: > > On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >> > >> On 5/13/19 1:41 AM, Martin Liška wrote: > >>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>> +hashtab_chk_error () > >>>>>>>> +{ > >>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>> + "equal operator returns true for a pair " > >>>>>>>> + "of values with a different hash value"); > >>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>> ^^^^^^ > >>>>>> Sure, fixed in attached patch. > >>>>>> > >>>>>> Martin > >>>>>> > >>>>>>>> + gcc_unreachable (); > >>>>>>>> +} > >>>>>>> Jakub > >>>>>>> > >>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>> > >>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>> From: marxin <mliska@suse.cz> > >>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>> > >>>>>> --- > >>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>> > >>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>> --- a/gcc/hash-table.h > >>>>>> +++ b/gcc/hash-table.h > >>>>>> @@ -503,6 +503,7 @@ private: > >>>>>> > >>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>> bool too_empty_p (unsigned int); > >>>>>> void expand (); > >>>>>> static bool is_deleted (value_type &v) > >>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>> expand (); > >>>>>> > >>>>>> - m_searches++; > >>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>> + if (insert == INSERT) > >>>>>> + verify (comparable, hash); > >>>>>> +#endif > >>>>>> > >>>>>> + m_searches++; > >>>>>> value_type *first_deleted_slot = NULL; > >>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>> return &m_entries[index]; > >>>>>> } > >>>>>> > >>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>> + > >>>>>> +/* Report a hash table checking error. */ > >>>>>> + > >>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>> +static void > >>>>>> +hashtab_chk_error () > >>>>>> +{ > >>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>> + "equal operator returns true for a pair " > >>>>>> + "of values with a different hash value\n"); > >>>>>> + gcc_unreachable (); > >>>>>> +} > >>>>> I think an internal_error here is probably still better than a simple > >>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>> > >>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>> because we've got too many bugs to fix. > >>>> Unfortunately it's blocked with these 2 PRs: > >>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>> Hi. > >>> > >>> I've just added one more PR: > >>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>> > >>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>> with a new function disable_sanitize_eq_and_hash. > >>> > >>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>> limits maximal number of checks: > >> So rather than call the disable_sanitize_eq_and_hash, can you have its > >> state set up when you instantiate the object? It's not a huge deal, > >> just thinking about loud. > >> > >> > >> > >> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >> issue :-) > > > > There is at least one PR where we have a table where elements _in_ the > > table are never compared against each other but always against another > > object (I guess that's usual even), but the setup is in a way that the > > comparison function only works with those. With the patch we verify > > hashing/comparison for something that is never used. > > > > So - wouldn't it be more "correct" to only verify comparison/hashing > > at lookup time, using the object from the lookup and verify that against > > all other elements? > > I don't a have problem with that. Apparently this changes fixes > PR90450 and PR87847. > > Changes from previous version: > - verification happens only when an element is searched (not inserted) > - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > - new param has been introduced hash-table-verification-limit in order > to limit number of elements that are compared within a table > - verification happens only with flag_checking >= 2 > > I've been bootstrapping and testing the patch right now. Looks like I misremembered the original patch. The issue isn't comparing random two elements in the table. That it fixes PR90450 is because LIM never calls find_slot_with_hash without INSERTing. I guess PR90450 is "real" indeed... Richard. > Martin > > > > > Richard. > > > >> > >> Jeff > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-21 11:52 ` Richard Biener @ 2019-05-22 9:13 ` Martin Liška 2019-05-31 13:23 ` Richard Biener 2019-05-31 22:10 ` Jeff Law 0 siblings, 2 replies; 53+ messages in thread From: Martin Liška @ 2019-05-22 9:13 UTC (permalink / raw) To: Richard Biener Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 7675 bytes --] On 5/21/19 1:51 PM, Richard Biener wrote: > On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> On 5/21/19 11:38 AM, Richard Biener wrote: >>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>> >>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>> +hashtab_chk_error () >>>>>>>>>> +{ >>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>> + "of values with a different hash value"); >>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>> ^^^^^^ >>>>>>>> Sure, fixed in attached patch. >>>>>>>> >>>>>>>> Martin >>>>>>>> >>>>>>>>>> + gcc_unreachable (); >>>>>>>>>> +} >>>>>>>>> Jakub >>>>>>>>> >>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>> >>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>> >>>>>>>> --- >>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>> >>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>> --- a/gcc/hash-table.h >>>>>>>> +++ b/gcc/hash-table.h >>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>> >>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>> bool too_empty_p (unsigned int); >>>>>>>> void expand (); >>>>>>>> static bool is_deleted (value_type &v) >>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>> expand (); >>>>>>>> >>>>>>>> - m_searches++; >>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>> + if (insert == INSERT) >>>>>>>> + verify (comparable, hash); >>>>>>>> +#endif >>>>>>>> >>>>>>>> + m_searches++; >>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>> return &m_entries[index]; >>>>>>>> } >>>>>>>> >>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>> + >>>>>>>> +/* Report a hash table checking error. */ >>>>>>>> + >>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>> +static void >>>>>>>> +hashtab_chk_error () >>>>>>>> +{ >>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>> + "equal operator returns true for a pair " >>>>>>>> + "of values with a different hash value\n"); >>>>>>>> + gcc_unreachable (); >>>>>>>> +} >>>>>>> I think an internal_error here is probably still better than a simple >>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>> >>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>> because we've got too many bugs to fix. >>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>> Hi. >>>>> >>>>> I've just added one more PR: >>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>> >>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>> with a new function disable_sanitize_eq_and_hash. >>>>> >>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>> limits maximal number of checks: >>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>> state set up when you instantiate the object? It's not a huge deal, >>>> just thinking about loud. >>>> >>>> >>>> >>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>> issue :-) >>> >>> There is at least one PR where we have a table where elements _in_ the >>> table are never compared against each other but always against another >>> object (I guess that's usual even), but the setup is in a way that the >>> comparison function only works with those. With the patch we verify >>> hashing/comparison for something that is never used. >>> >>> So - wouldn't it be more "correct" to only verify comparison/hashing >>> at lookup time, using the object from the lookup and verify that against >>> all other elements? >> >> I don't a have problem with that. Apparently this changes fixes >> PR90450 and PR87847. >> >> Changes from previous version: >> - verification happens only when an element is searched (not inserted) >> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >> - new param has been introduced hash-table-verification-limit in order >> to limit number of elements that are compared within a table >> - verification happens only with flag_checking >= 2 >> >> I've been bootstrapping and testing the patch right now. > > Looks like I misremembered the original patch. The issue isn't > comparing random two elements in the table. > > That it fixes PR90450 is because LIM never calls find_slot_with_hash > without INSERTing. > There's updated version of the patch where I check all find operations (both w/ and w/o insertion). Patch can bootstrap on x86_64-linux-gnu and survives regression tests except for: $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c hash table checking failed: equal operator returns true for a pair of values with a different hash value during GIMPLE pass: lim /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 6 | fn1 () | ^~~ 0x6c5725 hashtab_chk_error /home/marxin/Programming/gcc/gcc/hash-table.h:1019 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) /home/marxin/Programming/gcc/gcc/hash-table.h:1040 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) /home/marxin/Programming/gcc/gcc/hash-table.h:960 0xe504ea gather_mem_refs_stmt /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 0xe504ea analyze_memory_references /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 0xe504ea tree_ssa_lim /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 0xe504ea execute /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 Richi: it's after your recent patch. For some reason I don't see PR87847 issue any longer. May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? Thanks, Martin > I guess PR90450 is "real" indeed... > > Richard. > >> Martin >> >>> >>> Richard. >>> >>>> >>>> Jeff >> [-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --] [-- Type: text/x-patch, Size: 8179 bytes --] From 569165f803e35eb4f2eecdf42919a05d885e3919 Mon Sep 17 00:00:00 2001 From: marxin <mliska@suse.cz> Date: Mon, 13 May 2019 07:16:22 +0200 Subject: [PATCH] Enable sanitization for hash tables. --- gcc/cselib.c | 9 ++++++-- gcc/hash-set.h | 2 +- gcc/hash-table.c | 3 +++ gcc/hash-table.h | 60 ++++++++++++++++++++++++++++++++++++++++++++---- gcc/params.def | 6 +++++ gcc/toplev.c | 4 ++++ 6 files changed, 77 insertions(+), 7 deletions(-) diff --git a/gcc/cselib.c b/gcc/cselib.c index 84c17c23f6d..a1cbdec9718 100644 --- a/gcc/cselib.c +++ b/gcc/cselib.c @@ -2858,9 +2858,14 @@ cselib_init (int record_what) } used_regs = XNEWVEC (unsigned int, cselib_nregs); n_used_regs = 0; - cselib_hash_table = new hash_table<cselib_hasher> (31); + /* FIXME: enable sanitization (PR87845) */ + cselib_hash_table + = new hash_table<cselib_hasher> (31, /* ggc */ false, + /* sanitize_eq_and_hash */ false); if (cselib_preserve_constants) - cselib_preserved_hash_table = new hash_table<cselib_hasher> (31); + cselib_preserved_hash_table + = new hash_table<cselib_hasher> (31, /* ggc */ false, + /* sanitize_eq_and_hash */ false); next_uid = 1; } diff --git a/gcc/hash-set.h b/gcc/hash-set.h index de3532f5f68..d891ed78297 100644 --- a/gcc/hash-set.h +++ b/gcc/hash-set.h @@ -28,7 +28,7 @@ class hash_set public: typedef typename Traits::value_type Key; explicit hash_set (size_t n = 13, bool ggc = false CXX_MEM_STAT_INFO) - : m_table (n, ggc, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {} + : m_table (n, ggc, true, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {} /* Create a hash_set in gc memory with space for at least n elements. */ diff --git a/gcc/hash-table.c b/gcc/hash-table.c index 646a7a1c497..8e86fffa36f 100644 --- a/gcc/hash-table.c +++ b/gcc/hash-table.c @@ -74,6 +74,9 @@ struct prime_ent const prime_tab[] = { { 0xfffffffb, 0x00000006, 0x00000008, 31 } }; +/* Limit number of comparisons when calling hash_table<>::verify. */ +unsigned int hash_table_sanitize_eq_limit; + /* The following function returns an index into the above table of the nearest prime number which is greater than N, and near a power of two. */ diff --git a/gcc/hash-table.h b/gcc/hash-table.h index 4178616478e..7e62715535d 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -295,6 +295,8 @@ struct prime_ent extern struct prime_ent const prime_tab[]; +/* Limit number of comparisons when calling hash_table<>::verify. */ +extern unsigned int hash_table_sanitize_eq_limit; /* Functions for computing hash table indexes. */ @@ -371,10 +373,12 @@ class hash_table public: explicit hash_table (size_t, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS, mem_alloc_origin origin = HASH_TABLE_ORIGIN CXX_MEM_STAT_INFO); explicit hash_table (const hash_table &, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS, mem_alloc_origin origin = HASH_TABLE_ORIGIN CXX_MEM_STAT_INFO); @@ -516,6 +520,7 @@ private: value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; value_type *find_empty_slot_for_expand (hashval_t); + void verify (const compare_type &comparable, hashval_t hash); bool too_empty_p (unsigned int); void expand (); static bool is_deleted (value_type &v) @@ -564,6 +569,9 @@ private: /* if m_entries is stored in ggc memory. */ bool m_ggc; + /* True if the table should be sanitized for equal and hash functions. */ + bool m_sanitize_eq_and_hash; + /* If we should gather memory statistics for the table. */ #if GATHER_STATISTICS bool m_gather_mem_stats; @@ -586,12 +594,13 @@ extern void dump_hash_table_loc_statistics (void); template<typename Descriptor, bool Lazy, template<typename Type> class Allocator> hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc, + bool sanitize_eq_and_hash, bool gather_mem_stats ATTRIBUTE_UNUSED, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0), - m_ggc (ggc) + m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -617,12 +626,14 @@ template<typename Descriptor, bool Lazy, template<typename Type> class Allocator> hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h, bool ggc, + bool sanitize_eq_and_hash, bool gather_mem_stats ATTRIBUTE_UNUSED, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted), - m_searches (0), m_collisions (0), m_ggc (ggc) + m_searches (0), m_collisions (0), m_ggc (ggc), + m_sanitize_eq_and_hash (sanitize_eq_and_hash) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -912,7 +923,11 @@ hash_table<Descriptor, Lazy, Allocator> entry = &m_entries[index]; if (is_empty (*entry) || (!is_deleted (*entry) && Descriptor::equal (*entry, comparable))) - return *entry; + { + if (m_sanitize_eq_and_hash) + verify (comparable, hash); + return *entry; + } } } @@ -941,8 +956,10 @@ hash_table<Descriptor, Lazy, Allocator> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) expand (); - m_searches++; + if (m_sanitize_eq_and_hash) + verify (comparable, hash); + m_searches++; value_type *first_deleted_slot = NULL; hashval_t index = hash_table_mod1 (hash, m_size_prime_index); hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); @@ -989,6 +1006,41 @@ hash_table<Descriptor, Lazy, Allocator> return &m_entries[index]; } +/* Report a hash table checking error. */ + +ATTRIBUTE_NORETURN ATTRIBUTE_COLD +static void +hashtab_chk_error () +{ + fprintf (stderr, "hash table checking failed: " + "equal operator returns true for a pair " + "of values with a different hash value\n"); +#ifndef GENERATOR_FILE + gcc_unreachable (); +#else + exit (1); +#endif +} + +/* Verify that all existing elements in th hash table which are + equal to COMPARABLE have an equal HASH value provided as argument. */ + +template<typename Descriptor, bool Lazy, + template<typename Type> class Allocator> +void +hash_table<Descriptor, Lazy, Allocator> +::verify (const compare_type &comparable, hashval_t hash) +{ + for (size_t i = 0; i < MIN (hash_table_sanitize_eq_limit, m_size); i++) + { + value_type *entry = &m_entries[i]; + if (!is_empty (*entry) && !is_deleted (*entry) + && hash != Descriptor::hash (*entry) + && Descriptor::equal (*entry, comparable)) + hashtab_chk_error (); + } +} + /* This function deletes an element with the given COMPARABLE value from hash table starting with the given HASH. If there is no matching element in the hash table, this function does nothing. */ diff --git a/gcc/params.def b/gcc/params.def index 6b7f7eb5bae..e5ca6ff45e4 100644 --- a/gcc/params.def +++ b/gcc/params.def @@ -1439,6 +1439,12 @@ DEFPARAM(PARAM_GIMPLE_FE_COMPUTED_HOT_BB_THRESHOLD, " The parameter is used only in GIMPLE FE.", 0, 0, 0) +DEFPARAM(PARAM_HASH_TABLE_VERIFICATION_LIMIT, + "hash-table-verification-limit", + "The number of elements for which hash table verification is done for " + "each searched element.", + 100, 0, 0) + /* Local variables: diff --git a/gcc/toplev.c b/gcc/toplev.c index d300ac2ec89..116be7be395 100644 --- a/gcc/toplev.c +++ b/gcc/toplev.c @@ -1799,6 +1799,10 @@ process_options (void) optimization_default_node = build_optimization_node (&global_options); optimization_current_node = optimization_default_node; + if (flag_checking >= 2) + hash_table_sanitize_eq_limit + = PARAM_VALUE (PARAM_HASH_TABLE_VERIFICATION_LIMIT); + /* Please don't change global_options after this point, those changes won't be reflected in optimization_{default,current}_node. */ } -- 2.21.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-22 9:13 ` Martin Liška @ 2019-05-31 13:23 ` Richard Biener 2019-05-31 13:35 ` Martin Liška 2019-05-31 22:10 ` Jeff Law 1 sibling, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-05-31 13:23 UTC (permalink / raw) To: Martin Liška Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Wed, May 22, 2019 at 11:13 AM Martin Liška <mliska@suse.cz> wrote: > > On 5/21/19 1:51 PM, Richard Biener wrote: > > On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >> > >> On 5/21/19 11:38 AM, Richard Biener wrote: > >>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>> > >>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>> +{ > >>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>> ^^^^^^ > >>>>>>>> Sure, fixed in attached patch. > >>>>>>>> > >>>>>>>> Martin > >>>>>>>> > >>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>> +} > >>>>>>>>> Jakub > >>>>>>>>> > >>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>> > >>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>> > >>>>>>>> --- > >>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>> > >>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>> --- a/gcc/hash-table.h > >>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>> > >>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>> void expand (); > >>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>> expand (); > >>>>>>>> > >>>>>>>> - m_searches++; > >>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>> + if (insert == INSERT) > >>>>>>>> + verify (comparable, hash); > >>>>>>>> +#endif > >>>>>>>> > >>>>>>>> + m_searches++; > >>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>> return &m_entries[index]; > >>>>>>>> } > >>>>>>>> > >>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>> + > >>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>> + > >>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>> +static void > >>>>>>>> +hashtab_chk_error () > >>>>>>>> +{ > >>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>> + "equal operator returns true for a pair " > >>>>>>>> + "of values with a different hash value\n"); > >>>>>>>> + gcc_unreachable (); > >>>>>>>> +} > >>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>> > >>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>> because we've got too many bugs to fix. > >>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>> Hi. > >>>>> > >>>>> I've just added one more PR: > >>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>> > >>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>> with a new function disable_sanitize_eq_and_hash. > >>>>> > >>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>> limits maximal number of checks: > >>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>> state set up when you instantiate the object? It's not a huge deal, > >>>> just thinking about loud. > >>>> > >>>> > >>>> > >>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>> issue :-) > >>> > >>> There is at least one PR where we have a table where elements _in_ the > >>> table are never compared against each other but always against another > >>> object (I guess that's usual even), but the setup is in a way that the > >>> comparison function only works with those. With the patch we verify > >>> hashing/comparison for something that is never used. > >>> > >>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>> at lookup time, using the object from the lookup and verify that against > >>> all other elements? > >> > >> I don't a have problem with that. Apparently this changes fixes > >> PR90450 and PR87847. > >> > >> Changes from previous version: > >> - verification happens only when an element is searched (not inserted) > >> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >> - new param has been introduced hash-table-verification-limit in order > >> to limit number of elements that are compared within a table > >> - verification happens only with flag_checking >= 2 > >> > >> I've been bootstrapping and testing the patch right now. > > > > Looks like I misremembered the original patch. The issue isn't > > comparing random two elements in the table. > > > > That it fixes PR90450 is because LIM never calls find_slot_with_hash > > without INSERTing. > > > > There's updated version of the patch where I check all find operations > (both w/ and w/o insertion). > > Patch can bootstrap on x86_64-linux-gnu and survives regression tests > except for: > > $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > hash table checking failed: equal operator returns true for a pair of values with a different hash value > during GIMPLE pass: lim > /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > 6 | fn1 () > | ^~~ > 0x6c5725 hashtab_chk_error > /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > /home/marxin/Programming/gcc/gcc/hash-table.h:960 > 0xe504ea gather_mem_refs_stmt > /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > 0xe504ea analyze_memory_references > /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > 0xe504ea tree_ssa_lim > /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > 0xe504ea execute > /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > > Richi: it's after your recent patch. I can't reproduce this (but I have a slightly patched tree). Richard. > For some reason I don't see PR87847 issue any longer. > > > May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > > Thanks, > Martin > > > I guess PR90450 is "real" indeed... > > > > Richard. > > > >> Martin > >> > >>> > >>> Richard. > >>> > >>>> > >>>> Jeff > >> > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-31 13:23 ` Richard Biener @ 2019-05-31 13:35 ` Martin Liška 0 siblings, 0 replies; 53+ messages in thread From: Martin Liška @ 2019-05-31 13:35 UTC (permalink / raw) To: Richard Biener Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On 5/31/19 2:50 PM, Richard Biener wrote: > On Wed, May 22, 2019 at 11:13 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> On 5/21/19 1:51 PM, Richard Biener wrote: >>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>> >>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>> >>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>> +{ >>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>> ^^^^^^ >>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>> >>>>>>>>>> Martin >>>>>>>>>> >>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>> +} >>>>>>>>>>> Jakub >>>>>>>>>>> >>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>> >>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>> >>>>>>>>>> --- >>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>> >>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>> >>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>> void expand (); >>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>> expand (); >>>>>>>>>> >>>>>>>>>> - m_searches++; >>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>> + if (insert == INSERT) >>>>>>>>>> + verify (comparable, hash); >>>>>>>>>> +#endif >>>>>>>>>> >>>>>>>>>> + m_searches++; >>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>> return &m_entries[index]; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>> + >>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>> + >>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>> +static void >>>>>>>>>> +hashtab_chk_error () >>>>>>>>>> +{ >>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>> + gcc_unreachable (); >>>>>>>>>> +} >>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>> >>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>> because we've got too many bugs to fix. >>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>> Hi. >>>>>>> >>>>>>> I've just added one more PR: >>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>> >>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>> >>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>> limits maximal number of checks: >>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>> just thinking about loud. >>>>>> >>>>>> >>>>>> >>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>> issue :-) >>>>> >>>>> There is at least one PR where we have a table where elements _in_ the >>>>> table are never compared against each other but always against another >>>>> object (I guess that's usual even), but the setup is in a way that the >>>>> comparison function only works with those. With the patch we verify >>>>> hashing/comparison for something that is never used. >>>>> >>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>> at lookup time, using the object from the lookup and verify that against >>>>> all other elements? >>>> >>>> I don't a have problem with that. Apparently this changes fixes >>>> PR90450 and PR87847. >>>> >>>> Changes from previous version: >>>> - verification happens only when an element is searched (not inserted) >>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>> - new param has been introduced hash-table-verification-limit in order >>>> to limit number of elements that are compared within a table >>>> - verification happens only with flag_checking >= 2 >>>> >>>> I've been bootstrapping and testing the patch right now. >>> >>> Looks like I misremembered the original patch. The issue isn't >>> comparing random two elements in the table. >>> >>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>> without INSERTing. >>> >> >> There's updated version of the patch where I check all find operations >> (both w/ and w/o insertion). >> >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >> except for: >> >> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >> hash table checking failed: equal operator returns true for a pair of values with a different hash value >> during GIMPLE pass: lim >> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >> 6 | fn1 () >> | ^~~ >> 0x6c5725 hashtab_chk_error >> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >> 0xe504ea gather_mem_refs_stmt >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >> 0xe504ea analyze_memory_references >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >> 0xe504ea tree_ssa_lim >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >> 0xe504ea execute >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >> >> Richi: it's after your recent patch. > > I can't reproduce this (but I have a slightly patched tree). Me neither. I'll retest the whole patch again. Martin > > Richard. > >> For some reason I don't see PR87847 issue any longer. >> >> >> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >> >> Thanks, >> Martin >> >>> I guess PR90450 is "real" indeed... >>> >>> Richard. >>> >>>> Martin >>>> >>>>> >>>>> Richard. >>>>> >>>>>> >>>>>> Jeff >>>> >> ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-22 9:13 ` Martin Liška 2019-05-31 13:23 ` Richard Biener @ 2019-05-31 22:10 ` Jeff Law 2019-06-03 13:35 ` Martin Liška 1 sibling, 1 reply; 53+ messages in thread From: Jeff Law @ 2019-05-31 22:10 UTC (permalink / raw) To: Martin Liška, Richard Biener Cc: Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: > On 5/21/19 1:51 PM, Richard Biener wrote: >> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>> >>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>> >>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>> +{ >>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>> ^^^^^^ >>>>>>>>> Sure, fixed in attached patch. >>>>>>>>> >>>>>>>>> Martin >>>>>>>>> >>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>> +} >>>>>>>>>> Jakub >>>>>>>>>> >>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>> >>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>> >>>>>>>>> --- >>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>> >>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>> >>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>> void expand (); >>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>> expand (); >>>>>>>>> >>>>>>>>> - m_searches++; >>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>> + if (insert == INSERT) >>>>>>>>> + verify (comparable, hash); >>>>>>>>> +#endif >>>>>>>>> >>>>>>>>> + m_searches++; >>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>> return &m_entries[index]; >>>>>>>>> } >>>>>>>>> >>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>> + >>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>> + >>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>> +static void >>>>>>>>> +hashtab_chk_error () >>>>>>>>> +{ >>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>> + gcc_unreachable (); >>>>>>>>> +} >>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>> >>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>> because we've got too many bugs to fix. >>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>> Hi. >>>>>> >>>>>> I've just added one more PR: >>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>> >>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>> >>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>> limits maximal number of checks: >>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>> state set up when you instantiate the object? It's not a huge deal, >>>>> just thinking about loud. >>>>> >>>>> >>>>> >>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>> issue :-) >>>> >>>> There is at least one PR where we have a table where elements _in_ the >>>> table are never compared against each other but always against another >>>> object (I guess that's usual even), but the setup is in a way that the >>>> comparison function only works with those. With the patch we verify >>>> hashing/comparison for something that is never used. >>>> >>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>> at lookup time, using the object from the lookup and verify that against >>>> all other elements? >>> >>> I don't a have problem with that. Apparently this changes fixes >>> PR90450 and PR87847. >>> >>> Changes from previous version: >>> - verification happens only when an element is searched (not inserted) >>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>> - new param has been introduced hash-table-verification-limit in order >>> to limit number of elements that are compared within a table >>> - verification happens only with flag_checking >= 2 >>> >>> I've been bootstrapping and testing the patch right now. >> >> Looks like I misremembered the original patch. The issue isn't >> comparing random two elements in the table. >> >> That it fixes PR90450 is because LIM never calls find_slot_with_hash >> without INSERTing. >> > > There's updated version of the patch where I check all find operations > (both w/ and w/o insertion). > > Patch can bootstrap on x86_64-linux-gnu and survives regression tests > except for: > > $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > hash table checking failed: equal operator returns true for a pair of values with a different hash value > during GIMPLE pass: lim > /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: > /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > 6 | fn1 () > | ^~~ > 0x6c5725 hashtab_chk_error > /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > /home/marxin/Programming/gcc/gcc/hash-table.h:960 > 0xe504ea gather_mem_refs_stmt > /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > 0xe504ea analyze_memory_references > /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > 0xe504ea tree_ssa_lim > /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > 0xe504ea execute > /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > > Richi: it's after your recent patch. > > For some reason I don't see PR87847 issue any longer. > > > May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? Don't we still need to deal with the naked fprintf when there's a failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? jeff ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-05-31 22:10 ` Jeff Law @ 2019-06-03 13:35 ` Martin Liška 2019-06-07 8:57 ` Richard Biener 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2019-06-03 13:35 UTC (permalink / raw) To: Jeff Law, Richard Biener Cc: Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 8315 bytes --] On 6/1/19 12:06 AM, Jeff Law wrote: > On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >> On 5/21/19 1:51 PM, Richard Biener wrote: >>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>> >>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>> >>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>> +{ >>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>> ^^^^^^ >>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>> >>>>>>>>>> Martin >>>>>>>>>> >>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>> +} >>>>>>>>>>> Jakub >>>>>>>>>>> >>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>> >>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>> >>>>>>>>>> --- >>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>> >>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>> >>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>> void expand (); >>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>> expand (); >>>>>>>>>> >>>>>>>>>> - m_searches++; >>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>> + if (insert == INSERT) >>>>>>>>>> + verify (comparable, hash); >>>>>>>>>> +#endif >>>>>>>>>> >>>>>>>>>> + m_searches++; >>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>> return &m_entries[index]; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>> + >>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>> + >>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>> +static void >>>>>>>>>> +hashtab_chk_error () >>>>>>>>>> +{ >>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>> + gcc_unreachable (); >>>>>>>>>> +} >>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>> >>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>> because we've got too many bugs to fix. >>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>> Hi. >>>>>>> >>>>>>> I've just added one more PR: >>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>> >>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>> >>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>> limits maximal number of checks: >>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>> just thinking about loud. >>>>>> >>>>>> >>>>>> >>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>> issue :-) >>>>> >>>>> There is at least one PR where we have a table where elements _in_ the >>>>> table are never compared against each other but always against another >>>>> object (I guess that's usual even), but the setup is in a way that the >>>>> comparison function only works with those. With the patch we verify >>>>> hashing/comparison for something that is never used. >>>>> >>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>> at lookup time, using the object from the lookup and verify that against >>>>> all other elements? >>>> >>>> I don't a have problem with that. Apparently this changes fixes >>>> PR90450 and PR87847. >>>> >>>> Changes from previous version: >>>> - verification happens only when an element is searched (not inserted) >>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>> - new param has been introduced hash-table-verification-limit in order >>>> to limit number of elements that are compared within a table >>>> - verification happens only with flag_checking >= 2 >>>> >>>> I've been bootstrapping and testing the patch right now. >>> >>> Looks like I misremembered the original patch. The issue isn't >>> comparing random two elements in the table. >>> >>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>> without INSERTing. >>> >> >> There's updated version of the patch where I check all find operations >> (both w/ and w/o insertion). >> >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >> except for: >> >> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >> hash table checking failed: equal operator returns true for a pair of values with a different hash value >> during GIMPLE pass: lim >> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >> 6 | fn1 () >> | ^~~ >> 0x6c5725 hashtab_chk_error >> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >> 0xe504ea gather_mem_refs_stmt >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >> 0xe504ea analyze_memory_references >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >> 0xe504ea tree_ssa_lim >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >> 0xe504ea execute >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >> >> Richi: it's after your recent patch. >> >> For some reason I don't see PR87847 issue any longer. >> >> >> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > Don't we still need to deal with the naked fprintf when there's a > failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? Good point, I've just adjusted that. Patch can bootstrap on x86_64-linux-gnu and survives regression tests. Ready to be installed? Thanks, Martin > > jeff > [-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --] [-- Type: text/x-patch, Size: 8647 bytes --] From a51fc8dc210942b87d84130bb105467112fa9967 Mon Sep 17 00:00:00 2001 From: marxin <mliska@suse.cz> Date: Mon, 13 May 2019 07:16:22 +0200 Subject: [PATCH] Enable sanitization for hash tables. gcc/ChangeLog: 2019-06-03 Martin Liska <mliska@suse.cz> * cselib.c (cselib_init): Disable hash table sanitization. * hash-set.h: Pass new default argument to m_table. * hash-table.c: Add global variable with hash table sanitization limit. * hash-table.h (Allocator>::hash_table): Add new argument to ctor. (hashtab_chk_error): New. * params.def (PARAM_HASH_TABLE_VERIFICATION_LIMIT): New. * toplev.c (process_options): Set hash_table_sanitize_eq_limit from the PARAM_HASH_TABLE_VERIFICATION_LIMIT value. --- gcc/cselib.c | 9 ++++++-- gcc/hash-set.h | 2 +- gcc/hash-table.c | 3 +++ gcc/hash-table.h | 56 ++++++++++++++++++++++++++++++++++++++++++++---- gcc/params.def | 6 ++++++ gcc/toplev.c | 4 ++++ 6 files changed, 73 insertions(+), 7 deletions(-) diff --git a/gcc/cselib.c b/gcc/cselib.c index 84c17c23f6d..a1cbdec9718 100644 --- a/gcc/cselib.c +++ b/gcc/cselib.c @@ -2858,9 +2858,14 @@ cselib_init (int record_what) } used_regs = XNEWVEC (unsigned int, cselib_nregs); n_used_regs = 0; - cselib_hash_table = new hash_table<cselib_hasher> (31); + /* FIXME: enable sanitization (PR87845) */ + cselib_hash_table + = new hash_table<cselib_hasher> (31, /* ggc */ false, + /* sanitize_eq_and_hash */ false); if (cselib_preserve_constants) - cselib_preserved_hash_table = new hash_table<cselib_hasher> (31); + cselib_preserved_hash_table + = new hash_table<cselib_hasher> (31, /* ggc */ false, + /* sanitize_eq_and_hash */ false); next_uid = 1; } diff --git a/gcc/hash-set.h b/gcc/hash-set.h index de3532f5f68..d891ed78297 100644 --- a/gcc/hash-set.h +++ b/gcc/hash-set.h @@ -28,7 +28,7 @@ class hash_set public: typedef typename Traits::value_type Key; explicit hash_set (size_t n = 13, bool ggc = false CXX_MEM_STAT_INFO) - : m_table (n, ggc, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {} + : m_table (n, ggc, true, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {} /* Create a hash_set in gc memory with space for at least n elements. */ diff --git a/gcc/hash-table.c b/gcc/hash-table.c index 646a7a1c497..8e86fffa36f 100644 --- a/gcc/hash-table.c +++ b/gcc/hash-table.c @@ -74,6 +74,9 @@ struct prime_ent const prime_tab[] = { { 0xfffffffb, 0x00000006, 0x00000008, 31 } }; +/* Limit number of comparisons when calling hash_table<>::verify. */ +unsigned int hash_table_sanitize_eq_limit; + /* The following function returns an index into the above table of the nearest prime number which is greater than N, and near a power of two. */ diff --git a/gcc/hash-table.h b/gcc/hash-table.h index 4178616478e..785bed14679 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -295,6 +295,8 @@ struct prime_ent extern struct prime_ent const prime_tab[]; +/* Limit number of comparisons when calling hash_table<>::verify. */ +extern unsigned int hash_table_sanitize_eq_limit; /* Functions for computing hash table indexes. */ @@ -371,10 +373,12 @@ class hash_table public: explicit hash_table (size_t, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS, mem_alloc_origin origin = HASH_TABLE_ORIGIN CXX_MEM_STAT_INFO); explicit hash_table (const hash_table &, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS, mem_alloc_origin origin = HASH_TABLE_ORIGIN CXX_MEM_STAT_INFO); @@ -516,6 +520,7 @@ private: value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; value_type *find_empty_slot_for_expand (hashval_t); + void verify (const compare_type &comparable, hashval_t hash); bool too_empty_p (unsigned int); void expand (); static bool is_deleted (value_type &v) @@ -564,6 +569,9 @@ private: /* if m_entries is stored in ggc memory. */ bool m_ggc; + /* True if the table should be sanitized for equal and hash functions. */ + bool m_sanitize_eq_and_hash; + /* If we should gather memory statistics for the table. */ #if GATHER_STATISTICS bool m_gather_mem_stats; @@ -586,12 +594,13 @@ extern void dump_hash_table_loc_statistics (void); template<typename Descriptor, bool Lazy, template<typename Type> class Allocator> hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc, + bool sanitize_eq_and_hash, bool gather_mem_stats ATTRIBUTE_UNUSED, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0), - m_ggc (ggc) + m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -617,12 +626,14 @@ template<typename Descriptor, bool Lazy, template<typename Type> class Allocator> hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h, bool ggc, + bool sanitize_eq_and_hash, bool gather_mem_stats ATTRIBUTE_UNUSED, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted), - m_searches (0), m_collisions (0), m_ggc (ggc) + m_searches (0), m_collisions (0), m_ggc (ggc), + m_sanitize_eq_and_hash (sanitize_eq_and_hash) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -912,7 +923,11 @@ hash_table<Descriptor, Lazy, Allocator> entry = &m_entries[index]; if (is_empty (*entry) || (!is_deleted (*entry) && Descriptor::equal (*entry, comparable))) - return *entry; + { + if (m_sanitize_eq_and_hash) + verify (comparable, hash); + return *entry; + } } } @@ -941,8 +956,10 @@ hash_table<Descriptor, Lazy, Allocator> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) expand (); - m_searches++; + if (m_sanitize_eq_and_hash) + verify (comparable, hash); + m_searches++; value_type *first_deleted_slot = NULL; hashval_t index = hash_table_mod1 (hash, m_size_prime_index); hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); @@ -989,6 +1006,37 @@ hash_table<Descriptor, Lazy, Allocator> return &m_entries[index]; } +/* Report a hash table checking error. */ + +ATTRIBUTE_NORETURN ATTRIBUTE_COLD +static void +hashtab_chk_error () +{ + fprintf (stderr, "hash table checking failed: " + "equal operator returns true for a pair " + "of values with a different hash value\n"); + gcc_unreachable (); +} + +/* Verify that all existing elements in th hash table which are + equal to COMPARABLE have an equal HASH value provided as argument. */ + +template<typename Descriptor, bool Lazy, + template<typename Type> class Allocator> +void +hash_table<Descriptor, Lazy, Allocator> +::verify (const compare_type &comparable, hashval_t hash) +{ + for (size_t i = 0; i < MIN (hash_table_sanitize_eq_limit, m_size); i++) + { + value_type *entry = &m_entries[i]; + if (!is_empty (*entry) && !is_deleted (*entry) + && hash != Descriptor::hash (*entry) + && Descriptor::equal (*entry, comparable)) + hashtab_chk_error (); + } +} + /* This function deletes an element with the given COMPARABLE value from hash table starting with the given HASH. If there is no matching element in the hash table, this function does nothing. */ diff --git a/gcc/params.def b/gcc/params.def index 6b7f7eb5bae..e5ca6ff45e4 100644 --- a/gcc/params.def +++ b/gcc/params.def @@ -1439,6 +1439,12 @@ DEFPARAM(PARAM_GIMPLE_FE_COMPUTED_HOT_BB_THRESHOLD, " The parameter is used only in GIMPLE FE.", 0, 0, 0) +DEFPARAM(PARAM_HASH_TABLE_VERIFICATION_LIMIT, + "hash-table-verification-limit", + "The number of elements for which hash table verification is done for " + "each searched element.", + 100, 0, 0) + /* Local variables: diff --git a/gcc/toplev.c b/gcc/toplev.c index d300ac2ec89..116be7be395 100644 --- a/gcc/toplev.c +++ b/gcc/toplev.c @@ -1799,6 +1799,10 @@ process_options (void) optimization_default_node = build_optimization_node (&global_options); optimization_current_node = optimization_default_node; + if (flag_checking >= 2) + hash_table_sanitize_eq_limit + = PARAM_VALUE (PARAM_HASH_TABLE_VERIFICATION_LIMIT); + /* Please don't change global_options after this point, those changes won't be reflected in optimization_{default,current}_node. */ } -- 2.21.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-03 13:35 ` Martin Liška @ 2019-06-07 8:57 ` Richard Biener 2019-06-07 12:04 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-06-07 8:57 UTC (permalink / raw) To: Martin Liška Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > > On 6/1/19 12:06 AM, Jeff Law wrote: > > On 5/22/19 3:13 AM, Martin Liška wrote: > >> On 5/21/19 1:51 PM, Richard Biener wrote: > >>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >>>> > >>>> On 5/21/19 11:38 AM, Richard Biener wrote: > >>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>>>> > >>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>> +{ > >>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>>>> ^^^^^^ > >>>>>>>>>> Sure, fixed in attached patch. > >>>>>>>>>> > >>>>>>>>>> Martin > >>>>>>>>>> > >>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>> +} > >>>>>>>>>>> Jakub > >>>>>>>>>>> > >>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>>>> > >>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>>>> > >>>>>>>>>> --- > >>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>>>> > >>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>>>> --- a/gcc/hash-table.h > >>>>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>>>> > >>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>>>> void expand (); > >>>>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>>>> expand (); > >>>>>>>>>> > >>>>>>>>>> - m_searches++; > >>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>> + if (insert == INSERT) > >>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>> +#endif > >>>>>>>>>> > >>>>>>>>>> + m_searches++; > >>>>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>> return &m_entries[index]; > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>> + > >>>>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>>>> + > >>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>>>> +static void > >>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>> +{ > >>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>> + "of values with a different hash value\n"); > >>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>> +} > >>>>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>>>> > >>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>>>> because we've got too many bugs to fix. > >>>>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>>>> Hi. > >>>>>>> > >>>>>>> I've just added one more PR: > >>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>>>> > >>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>>>> with a new function disable_sanitize_eq_and_hash. > >>>>>>> > >>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>>>> limits maximal number of checks: > >>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>>>> state set up when you instantiate the object? It's not a huge deal, > >>>>>> just thinking about loud. > >>>>>> > >>>>>> > >>>>>> > >>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>>>> issue :-) > >>>>> > >>>>> There is at least one PR where we have a table where elements _in_ the > >>>>> table are never compared against each other but always against another > >>>>> object (I guess that's usual even), but the setup is in a way that the > >>>>> comparison function only works with those. With the patch we verify > >>>>> hashing/comparison for something that is never used. > >>>>> > >>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>>>> at lookup time, using the object from the lookup and verify that against > >>>>> all other elements? > >>>> > >>>> I don't a have problem with that. Apparently this changes fixes > >>>> PR90450 and PR87847. > >>>> > >>>> Changes from previous version: > >>>> - verification happens only when an element is searched (not inserted) > >>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >>>> - new param has been introduced hash-table-verification-limit in order > >>>> to limit number of elements that are compared within a table > >>>> - verification happens only with flag_checking >= 2 > >>>> > >>>> I've been bootstrapping and testing the patch right now. > >>> > >>> Looks like I misremembered the original patch. The issue isn't > >>> comparing random two elements in the table. > >>> > >>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > >>> without INSERTing. > >>> > >> > >> There's updated version of the patch where I check all find operations > >> (both w/ and w/o insertion). > >> > >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > >> except for: > >> > >> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > >> hash table checking failed: equal operator returns true for a pair of values with a different hash value > >> during GIMPLE pass: lim > >> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > >> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > >> 6 | fn1 () > >> | ^~~ > >> 0x6c5725 hashtab_chk_error > >> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > >> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > >> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > >> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > >> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > >> 0xe504ea gather_mem_refs_stmt > >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > >> 0xe504ea analyze_memory_references > >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > >> 0xe504ea tree_ssa_lim > >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > >> 0xe504ea execute > >> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > >> > >> Richi: it's after your recent patch. > >> > >> For some reason I don't see PR87847 issue any longer. > >> > >> > >> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > > Don't we still need to deal with the naked fprintf when there's a > > failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > > Good point, I've just adjusted that. > > Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > > Ready to be installed? Ugh, the cselib one is really bad. But I don't hold my breath for anyone fixing it ... One question - there's unconditional + if (m_sanitize_eq_and_hash) + verify (comparable, hash); which will read a global variable and have (possibly not inline) call to verify on a common path even with checking disabled. So I think we want to compile this checking feature out for !CHECKING_P or at least make the if __builtin_expect (..., 0), ::verify not inlined and marked pure () (thus, !CHECKING_P is simplest ;)). Thanks, Richard. > Thanks, > Martin > > > > > jeff > > > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-07 8:57 ` Richard Biener @ 2019-06-07 12:04 ` Martin Liška 2019-06-07 12:09 ` Richard Biener 2019-06-23 23:08 ` Ian Lance Taylor 0 siblings, 2 replies; 53+ messages in thread From: Martin Liška @ 2019-06-07 12:04 UTC (permalink / raw) To: Richard Biener Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 9584 bytes --] On 6/7/19 10:57 AM, Richard Biener wrote: > On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> On 6/1/19 12:06 AM, Jeff Law wrote: >>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>> >>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>> >>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>> +{ >>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>> >>>>>>>>>>>> Martin >>>>>>>>>>>> >>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>> +} >>>>>>>>>>>>> Jakub >>>>>>>>>>>>> >>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>> >>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>> >>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>> >>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>> void expand (); >>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>> expand (); >>>>>>>>>>>> >>>>>>>>>>>> - m_searches++; >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>> +#endif >>>>>>>>>>>> >>>>>>>>>>>> + m_searches++; >>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>> + >>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>> + >>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>> +static void >>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>> +{ >>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>> +} >>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>> >>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>> Hi. >>>>>>>>> >>>>>>>>> I've just added one more PR: >>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>> >>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>> >>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>> limits maximal number of checks: >>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>> just thinking about loud. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>> issue :-) >>>>>>> >>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>> table are never compared against each other but always against another >>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>> comparison function only works with those. With the patch we verify >>>>>>> hashing/comparison for something that is never used. >>>>>>> >>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>> all other elements? >>>>>> >>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>> PR90450 and PR87847. >>>>>> >>>>>> Changes from previous version: >>>>>> - verification happens only when an element is searched (not inserted) >>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>> to limit number of elements that are compared within a table >>>>>> - verification happens only with flag_checking >= 2 >>>>>> >>>>>> I've been bootstrapping and testing the patch right now. >>>>> >>>>> Looks like I misremembered the original patch. The issue isn't >>>>> comparing random two elements in the table. >>>>> >>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>> without INSERTing. >>>>> >>>> >>>> There's updated version of the patch where I check all find operations >>>> (both w/ and w/o insertion). >>>> >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>> except for: >>>> >>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>> during GIMPLE pass: lim >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>> 6 | fn1 () >>>> | ^~~ >>>> 0x6c5725 hashtab_chk_error >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>> 0xe504ea gather_mem_refs_stmt >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>> 0xe504ea analyze_memory_references >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>> 0xe504ea tree_ssa_lim >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>> 0xe504ea execute >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>> >>>> Richi: it's after your recent patch. >>>> >>>> For some reason I don't see PR87847 issue any longer. >>>> >>>> >>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>> Don't we still need to deal with the naked fprintf when there's a >>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >> >> Good point, I've just adjusted that. >> >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >> >> Ready to be installed? > > Ugh, the cselib one is really bad. But I don't hold my breath for anyone > fixing it ... Yes :D It's been some time and there's no interest in the PR. > > One question - there's unconditional > > + if (m_sanitize_eq_and_hash) > + verify (comparable, hash); > > which will read a global variable and have (possibly not inline) call > to verify on a common path even with checking disabled. So I think > we want to compile this checking feature out for !CHECKING_P > or at least make the if __builtin_expect (..., 0), ::verify not > inlined and marked pure () (thus, !CHECKING_P is simplest ;)). Fixed. May I install the patch? The cselib issue can be solved later.. Martin > > Thanks, > Richard. > >> Thanks, >> Martin >> >>> >>> jeff >>> >> [-- Attachment #2: 0001-Enable-sanitization-for-hash-tables.patch --] [-- Type: text/x-patch, Size: 8670 bytes --] From 1ccf332932661b087bd76d017f888572abca594c Mon Sep 17 00:00:00 2001 From: marxin <mliska@suse.cz> Date: Mon, 13 May 2019 07:16:22 +0200 Subject: [PATCH] Enable sanitization for hash tables. gcc/ChangeLog: 2019-06-03 Martin Liska <mliska@suse.cz> * cselib.c (cselib_init): Disable hash table sanitization. * hash-set.h: Pass new default argument to m_table. * hash-table.c: Add global variable with hash table sanitization limit. * hash-table.h (Allocator>::hash_table): Add new argument to ctor. (hashtab_chk_error): New. * params.def (PARAM_HASH_TABLE_VERIFICATION_LIMIT): New. * toplev.c (process_options): Set hash_table_sanitize_eq_limit from the PARAM_HASH_TABLE_VERIFICATION_LIMIT value. --- gcc/cselib.c | 9 ++++++-- gcc/hash-set.h | 2 +- gcc/hash-table.c | 3 +++ gcc/hash-table.h | 58 ++++++++++++++++++++++++++++++++++++++++++++---- gcc/params.def | 6 +++++ gcc/toplev.c | 4 ++++ 6 files changed, 75 insertions(+), 7 deletions(-) diff --git a/gcc/cselib.c b/gcc/cselib.c index 84c17c23f6d..a1cbdec9718 100644 --- a/gcc/cselib.c +++ b/gcc/cselib.c @@ -2858,9 +2858,14 @@ cselib_init (int record_what) } used_regs = XNEWVEC (unsigned int, cselib_nregs); n_used_regs = 0; - cselib_hash_table = new hash_table<cselib_hasher> (31); + /* FIXME: enable sanitization (PR87845) */ + cselib_hash_table + = new hash_table<cselib_hasher> (31, /* ggc */ false, + /* sanitize_eq_and_hash */ false); if (cselib_preserve_constants) - cselib_preserved_hash_table = new hash_table<cselib_hasher> (31); + cselib_preserved_hash_table + = new hash_table<cselib_hasher> (31, /* ggc */ false, + /* sanitize_eq_and_hash */ false); next_uid = 1; } diff --git a/gcc/hash-set.h b/gcc/hash-set.h index de3532f5f68..d891ed78297 100644 --- a/gcc/hash-set.h +++ b/gcc/hash-set.h @@ -28,7 +28,7 @@ class hash_set public: typedef typename Traits::value_type Key; explicit hash_set (size_t n = 13, bool ggc = false CXX_MEM_STAT_INFO) - : m_table (n, ggc, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {} + : m_table (n, ggc, true, GATHER_STATISTICS, HASH_SET_ORIGIN PASS_MEM_STAT) {} /* Create a hash_set in gc memory with space for at least n elements. */ diff --git a/gcc/hash-table.c b/gcc/hash-table.c index 646a7a1c497..8e86fffa36f 100644 --- a/gcc/hash-table.c +++ b/gcc/hash-table.c @@ -74,6 +74,9 @@ struct prime_ent const prime_tab[] = { { 0xfffffffb, 0x00000006, 0x00000008, 31 } }; +/* Limit number of comparisons when calling hash_table<>::verify. */ +unsigned int hash_table_sanitize_eq_limit; + /* The following function returns an index into the above table of the nearest prime number which is greater than N, and near a power of two. */ diff --git a/gcc/hash-table.h b/gcc/hash-table.h index 4178616478e..bf457f80d1b 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -295,6 +295,8 @@ struct prime_ent extern struct prime_ent const prime_tab[]; +/* Limit number of comparisons when calling hash_table<>::verify. */ +extern unsigned int hash_table_sanitize_eq_limit; /* Functions for computing hash table indexes. */ @@ -371,10 +373,12 @@ class hash_table public: explicit hash_table (size_t, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS, mem_alloc_origin origin = HASH_TABLE_ORIGIN CXX_MEM_STAT_INFO); explicit hash_table (const hash_table &, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS, mem_alloc_origin origin = HASH_TABLE_ORIGIN CXX_MEM_STAT_INFO); @@ -516,6 +520,7 @@ private: value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; value_type *find_empty_slot_for_expand (hashval_t); + void verify (const compare_type &comparable, hashval_t hash); bool too_empty_p (unsigned int); void expand (); static bool is_deleted (value_type &v) @@ -564,6 +569,9 @@ private: /* if m_entries is stored in ggc memory. */ bool m_ggc; + /* True if the table should be sanitized for equal and hash functions. */ + bool m_sanitize_eq_and_hash; + /* If we should gather memory statistics for the table. */ #if GATHER_STATISTICS bool m_gather_mem_stats; @@ -586,12 +594,13 @@ extern void dump_hash_table_loc_statistics (void); template<typename Descriptor, bool Lazy, template<typename Type> class Allocator> hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc, + bool sanitize_eq_and_hash, bool gather_mem_stats ATTRIBUTE_UNUSED, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0), - m_ggc (ggc) + m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -617,12 +626,14 @@ template<typename Descriptor, bool Lazy, template<typename Type> class Allocator> hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h, bool ggc, + bool sanitize_eq_and_hash, bool gather_mem_stats ATTRIBUTE_UNUSED, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted), - m_searches (0), m_collisions (0), m_ggc (ggc) + m_searches (0), m_collisions (0), m_ggc (ggc), + m_sanitize_eq_and_hash (sanitize_eq_and_hash) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -912,7 +923,13 @@ hash_table<Descriptor, Lazy, Allocator> entry = &m_entries[index]; if (is_empty (*entry) || (!is_deleted (*entry) && Descriptor::equal (*entry, comparable))) - return *entry; + { +#if CHECKING_P + if (m_sanitize_eq_and_hash) + verify (comparable, hash); +#endif + return *entry; + } } } @@ -941,8 +958,10 @@ hash_table<Descriptor, Lazy, Allocator> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) expand (); - m_searches++; + if (m_sanitize_eq_and_hash) + verify (comparable, hash); + m_searches++; value_type *first_deleted_slot = NULL; hashval_t index = hash_table_mod1 (hash, m_size_prime_index); hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); @@ -989,6 +1008,37 @@ hash_table<Descriptor, Lazy, Allocator> return &m_entries[index]; } +/* Report a hash table checking error. */ + +ATTRIBUTE_NORETURN ATTRIBUTE_COLD +static void +hashtab_chk_error () +{ + fprintf (stderr, "hash table checking failed: " + "equal operator returns true for a pair " + "of values with a different hash value\n"); + gcc_unreachable (); +} + +/* Verify that all existing elements in th hash table which are + equal to COMPARABLE have an equal HASH value provided as argument. */ + +template<typename Descriptor, bool Lazy, + template<typename Type> class Allocator> +void +hash_table<Descriptor, Lazy, Allocator> +::verify (const compare_type &comparable, hashval_t hash) +{ + for (size_t i = 0; i < MIN (hash_table_sanitize_eq_limit, m_size); i++) + { + value_type *entry = &m_entries[i]; + if (!is_empty (*entry) && !is_deleted (*entry) + && hash != Descriptor::hash (*entry) + && Descriptor::equal (*entry, comparable)) + hashtab_chk_error (); + } +} + /* This function deletes an element with the given COMPARABLE value from hash table starting with the given HASH. If there is no matching element in the hash table, this function does nothing. */ diff --git a/gcc/params.def b/gcc/params.def index b4a4e4a4190..0db60951413 100644 --- a/gcc/params.def +++ b/gcc/params.def @@ -1431,6 +1431,12 @@ DEFPARAM(PARAM_GIMPLE_FE_COMPUTED_HOT_BB_THRESHOLD, " The parameter is used only in GIMPLE FE.", 0, 0, 0) +DEFPARAM(PARAM_HASH_TABLE_VERIFICATION_LIMIT, + "hash-table-verification-limit", + "The number of elements for which hash table verification is done for " + "each searched element.", + 100, 0, 0) + /* Local variables: diff --git a/gcc/toplev.c b/gcc/toplev.c index d300ac2ec89..116be7be395 100644 --- a/gcc/toplev.c +++ b/gcc/toplev.c @@ -1799,6 +1799,10 @@ process_options (void) optimization_default_node = build_optimization_node (&global_options); optimization_current_node = optimization_default_node; + if (flag_checking >= 2) + hash_table_sanitize_eq_limit + = PARAM_VALUE (PARAM_HASH_TABLE_VERIFICATION_LIMIT); + /* Please don't change global_options after this point, those changes won't be reflected in optimization_{default,current}_node. */ } -- 2.21.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-07 12:04 ` Martin Liška @ 2019-06-07 12:09 ` Richard Biener 2019-06-07 12:13 ` Martin Liška 2019-06-23 23:08 ` Ian Lance Taylor 1 sibling, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-06-07 12:09 UTC (permalink / raw) To: Martin Liška Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote: > > On 6/7/19 10:57 AM, Richard Biener wrote: > > On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > >> > >> On 6/1/19 12:06 AM, Jeff Law wrote: > >>> On 5/22/19 3:13 AM, Martin Liška wrote: > >>>> On 5/21/19 1:51 PM, Richard Biener wrote: > >>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>> > >>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: > >>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>>>>>> > >>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>>>>>> ^^^^^^ > >>>>>>>>>>>> Sure, fixed in attached patch. > >>>>>>>>>>>> > >>>>>>>>>>>> Martin > >>>>>>>>>>>> > >>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>> +} > >>>>>>>>>>>>> Jakub > >>>>>>>>>>>>> > >>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>>>>>> > >>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>>>>>> > >>>>>>>>>>>> --- > >>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>>>>>> > >>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>>>>>> --- a/gcc/hash-table.h > >>>>>>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>>>>>> > >>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>>>>>> void expand (); > >>>>>>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>>>>>> expand (); > >>>>>>>>>>>> > >>>>>>>>>>>> - m_searches++; > >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>> + if (insert == INSERT) > >>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>> +#endif > >>>>>>>>>>>> > >>>>>>>>>>>> + m_searches++; > >>>>>>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>> return &m_entries[index]; > >>>>>>>>>>>> } > >>>>>>>>>>>> > >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>> + > >>>>>>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>>>>>> + > >>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>>>>>> +static void > >>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>> +{ > >>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>> + "of values with a different hash value\n"); > >>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>> +} > >>>>>>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>>>>>> > >>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>>>>>> because we've got too many bugs to fix. > >>>>>>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>>>>>> Hi. > >>>>>>>>> > >>>>>>>>> I've just added one more PR: > >>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>>>>>> > >>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>>>>>> with a new function disable_sanitize_eq_and_hash. > >>>>>>>>> > >>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>>>>>> limits maximal number of checks: > >>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>>>>>> state set up when you instantiate the object? It's not a huge deal, > >>>>>>>> just thinking about loud. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>>>>>> issue :-) > >>>>>>> > >>>>>>> There is at least one PR where we have a table where elements _in_ the > >>>>>>> table are never compared against each other but always against another > >>>>>>> object (I guess that's usual even), but the setup is in a way that the > >>>>>>> comparison function only works with those. With the patch we verify > >>>>>>> hashing/comparison for something that is never used. > >>>>>>> > >>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>>>>>> at lookup time, using the object from the lookup and verify that against > >>>>>>> all other elements? > >>>>>> > >>>>>> I don't a have problem with that. Apparently this changes fixes > >>>>>> PR90450 and PR87847. > >>>>>> > >>>>>> Changes from previous version: > >>>>>> - verification happens only when an element is searched (not inserted) > >>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >>>>>> - new param has been introduced hash-table-verification-limit in order > >>>>>> to limit number of elements that are compared within a table > >>>>>> - verification happens only with flag_checking >= 2 > >>>>>> > >>>>>> I've been bootstrapping and testing the patch right now. > >>>>> > >>>>> Looks like I misremembered the original patch. The issue isn't > >>>>> comparing random two elements in the table. > >>>>> > >>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > >>>>> without INSERTing. > >>>>> > >>>> > >>>> There's updated version of the patch where I check all find operations > >>>> (both w/ and w/o insertion). > >>>> > >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > >>>> except for: > >>>> > >>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > >>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value > >>>> during GIMPLE pass: lim > >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > >>>> 6 | fn1 () > >>>> | ^~~ > >>>> 0x6c5725 hashtab_chk_error > >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > >>>> 0xe504ea gather_mem_refs_stmt > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > >>>> 0xe504ea analyze_memory_references > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > >>>> 0xe504ea tree_ssa_lim > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > >>>> 0xe504ea execute > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > >>>> > >>>> Richi: it's after your recent patch. > >>>> > >>>> For some reason I don't see PR87847 issue any longer. > >>>> > >>>> > >>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > >>> Don't we still need to deal with the naked fprintf when there's a > >>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > >> > >> Good point, I've just adjusted that. > >> > >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >> > >> Ready to be installed? > > > > Ugh, the cselib one is really bad. But I don't hold my breath for anyone > > fixing it ... > > Yes :D It's been some time and there's no interest in the PR. > > > > > One question - there's unconditional > > > > + if (m_sanitize_eq_and_hash) > > + verify (comparable, hash); > > > > which will read a global variable and have (possibly not inline) call > > to verify on a common path even with checking disabled. So I think > > we want to compile this checking feature out for !CHECKING_P > > or at least make the if __builtin_expect (..., 0), ::verify not > > inlined and marked pure () (thus, !CHECKING_P is simplest ;)). > > Fixed. May I install the patch? The cselib issue can be solved later.. You missed the second occurance - m_searches++; + if (m_sanitize_eq_and_hash) + verify (comparable, hash); otherwise OK. Richard. > Martin > > > > > Thanks, > > Richard. > > > >> Thanks, > >> Martin > >> > >>> > >>> jeff > >>> > >> > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-07 12:09 ` Richard Biener @ 2019-06-07 12:13 ` Martin Liška 2019-06-07 14:48 ` Martin Sebor 2019-06-07 21:43 ` Jason Merrill 0 siblings, 2 replies; 53+ messages in thread From: Martin Liška @ 2019-06-07 12:13 UTC (permalink / raw) To: Richard Biener Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On 6/7/19 2:09 PM, Richard Biener wrote: > On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> On 6/7/19 10:57 AM, Richard Biener wrote: >>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>> >>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>> >>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>> >>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Martin >>>>>>>>>>>>>> >>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>> Jakub >>>>>>>>>>>>>>> >>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>> >>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>> >>>>>>>>>>>>>> --- >>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>> >>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>> >>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>>>> void expand (); >>>>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>> expand (); >>>>>>>>>>>>>> >>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>> >>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>>>> } >>>>>>>>>>>>>> >>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>> + >>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>> + >>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>> +static void >>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>> +{ >>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>> +} >>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>> >>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>> Hi. >>>>>>>>>>> >>>>>>>>>>> I've just added one more PR: >>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>> >>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>> >>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>> limits maximal number of checks: >>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>> just thinking about loud. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>> issue :-) >>>>>>>>> >>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>> table are never compared against each other but always against another >>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>> >>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>> all other elements? >>>>>>>> >>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>> PR90450 and PR87847. >>>>>>>> >>>>>>>> Changes from previous version: >>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>> to limit number of elements that are compared within a table >>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>> >>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>> >>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>> comparing random two elements in the table. >>>>>>> >>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>> without INSERTing. >>>>>>> >>>>>> >>>>>> There's updated version of the patch where I check all find operations >>>>>> (both w/ and w/o insertion). >>>>>> >>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>> except for: >>>>>> >>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>> during GIMPLE pass: lim >>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>> 6 | fn1 () >>>>>> | ^~~ >>>>>> 0x6c5725 hashtab_chk_error >>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>> 0xe504ea gather_mem_refs_stmt >>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>> 0xe504ea analyze_memory_references >>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>> 0xe504ea tree_ssa_lim >>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>> 0xe504ea execute >>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>> >>>>>> Richi: it's after your recent patch. >>>>>> >>>>>> For some reason I don't see PR87847 issue any longer. >>>>>> >>>>>> >>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>> Don't we still need to deal with the naked fprintf when there's a >>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>> >>>> Good point, I've just adjusted that. >>>> >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>> >>>> Ready to be installed? >>> >>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>> fixing it ... >> >> Yes :D It's been some time and there's no interest in the PR. >> >>> >>> One question - there's unconditional >>> >>> + if (m_sanitize_eq_and_hash) >>> + verify (comparable, hash); >>> >>> which will read a global variable and have (possibly not inline) call >>> to verify on a common path even with checking disabled. So I think >>> we want to compile this checking feature out for !CHECKING_P >>> or at least make the if __builtin_expect (..., 0), ::verify not >>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >> >> Fixed. May I install the patch? The cselib issue can be solved later.. > > You missed the second occurance > > - m_searches++; > + if (m_sanitize_eq_and_hash) > + verify (comparable, hash); Yep ;) I've just install the patch. Martin > > > otherwise OK. > > Richard. > >> Martin >> >>> >>> Thanks, >>> Richard. >>> >>>> Thanks, >>>> Martin >>>> >>>>> >>>>> jeff >>>>> >>>> >> ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-07 12:13 ` Martin Liška @ 2019-06-07 14:48 ` Martin Sebor 2019-06-07 21:43 ` Jason Merrill 1 sibling, 0 replies; 53+ messages in thread From: Martin Sebor @ 2019-06-07 14:48 UTC (permalink / raw) To: Martin Liška, Richard Biener Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On 6/7/19 6:13 AM, Martin LiÅ¡ka wrote: > On 6/7/19 2:09 PM, Richard Biener wrote: >> On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>> >>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>> >>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>> >>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>> >>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>> Jakub >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>>>>> void expand (); >>>>>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>> expand (); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>> +} >>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>> >>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>> Hi. >>>>>>>>>>>> >>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>> >>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>> >>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>> just thinking about loud. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>> issue :-) >>>>>>>>>> >>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>> >>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>> all other elements? >>>>>>>>> >>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>> PR90450 and PR87847. >>>>>>>>> >>>>>>>>> Changes from previous version: >>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>> to limit number of elements that are compared within a table >>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>> >>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>> >>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>> comparing random two elements in the table. >>>>>>>> >>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>> without INSERTing. >>>>>>>> >>>>>>> >>>>>>> There's updated version of the patch where I check all find operations >>>>>>> (both w/ and w/o insertion). >>>>>>> >>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>> except for: >>>>>>> >>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>> during GIMPLE pass: lim >>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>> 6 | fn1 () >>>>>>> | ^~~ >>>>>>> 0x6c5725 hashtab_chk_error >>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>> 0xe504ea analyze_memory_references >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>> 0xe504ea tree_ssa_lim >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>> 0xe504ea execute >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>> >>>>>>> Richi: it's after your recent patch. >>>>>>> >>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>> >>>>>>> >>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>> >>>>> Good point, I've just adjusted that. >>>>> >>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>> >>>>> Ready to be installed? >>>> >>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>> fixing it ... >>> >>> Yes :D It's been some time and there's no interest in the PR. >>> >>>> >>>> One question - there's unconditional >>>> >>>> + if (m_sanitize_eq_and_hash) >>>> + verify (comparable, hash); >>>> >>>> which will read a global variable and have (possibly not inline) call >>>> to verify on a common path even with checking disabled. So I think >>>> we want to compile this checking feature out for !CHECKING_P >>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>> >>> Fixed. May I install the patch? The cselib issue can be solved later.. >> >> You missed the second occurance >> >> - m_searches++; >> + if (m_sanitize_eq_and_hash) >> + verify (comparable, hash); > > Yep ;) I've just install the patch. I think it breaks bootstrap with the error below. I don't see the error after reverting it so I'm pretty sure it has something to do with it. Martin g++ -std=gnu++98 -O0 -g3 -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -DHAVE_CONFIG_H -DGENERATOR_FILE -fno-PIE -no-pie -o build/gencondmd \ build/gencondmd.o ../build-x86_64-pc-linux-gnu/libiberty/pic/libiberty.a /usr/bin/ld: build/gencondmd.o: in function `hashtab_chk_error()': /src/gcc/git-svn/gcc/hash-table.h:1022: undefined reference to `fancy_abort(char const*, int, char const*)' collect2: error: ld returned 1 exit status make[2]: *** [Makefile:2865: build/gencondmd] Error 1 Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-07 12:13 ` Martin Liška 2019-06-07 14:48 ` Martin Sebor @ 2019-06-07 21:43 ` Jason Merrill 2019-06-10 7:08 ` Martin Liška 1 sibling, 1 reply; 53+ messages in thread From: Jason Merrill @ 2019-06-07 21:43 UTC (permalink / raw) To: Martin Liška Cc: Richard Biener, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote: > > On 6/7/19 2:09 PM, Richard Biener wrote: > > On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote: > >> > >> On 6/7/19 10:57 AM, Richard Biener wrote: > >>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > >>>> > >>>> On 6/1/19 12:06 AM, Jeff Law wrote: > >>>>> On 5/22/19 3:13 AM, Martin Liška wrote: > >>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: > >>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>> > >>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: > >>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>>>>>>>> > >>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>>>>>>>> ^^^^^^ > >>>>>>>>>>>>>> Sure, fixed in attached patch. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Martin > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>> Jakub > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> --- > >>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>>>>>>>> --- a/gcc/hash-table.h > >>>>>>>>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>>>>>>>> void expand (); > >>>>>>>>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>>>>>>>> expand (); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> - m_searches++; > >>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>> + if (insert == INSERT) > >>>>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>>>> +#endif > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> + m_searches++; > >>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>> return &m_entries[index]; > >>>>>>>>>>>>>> } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>> + > >>>>>>>>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>>>>>>>> + > >>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>>>>>>>> +static void > >>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>> + "of values with a different hash value\n"); > >>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>> +} > >>>>>>>>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>>>>>>>> > >>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>>>>>>>> because we've got too many bugs to fix. > >>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>>>>>>>> Hi. > >>>>>>>>>>> > >>>>>>>>>>> I've just added one more PR: > >>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>>>>>>>> > >>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. > >>>>>>>>>>> > >>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>>>>>>>> limits maximal number of checks: > >>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, > >>>>>>>>>> just thinking about loud. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>>>>>>>> issue :-) > >>>>>>>>> > >>>>>>>>> There is at least one PR where we have a table where elements _in_ the > >>>>>>>>> table are never compared against each other but always against another > >>>>>>>>> object (I guess that's usual even), but the setup is in a way that the > >>>>>>>>> comparison function only works with those. With the patch we verify > >>>>>>>>> hashing/comparison for something that is never used. > >>>>>>>>> > >>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>>>>>>>> at lookup time, using the object from the lookup and verify that against > >>>>>>>>> all other elements? > >>>>>>>> > >>>>>>>> I don't a have problem with that. Apparently this changes fixes > >>>>>>>> PR90450 and PR87847. > >>>>>>>> > >>>>>>>> Changes from previous version: > >>>>>>>> - verification happens only when an element is searched (not inserted) > >>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >>>>>>>> - new param has been introduced hash-table-verification-limit in order > >>>>>>>> to limit number of elements that are compared within a table > >>>>>>>> - verification happens only with flag_checking >= 2 > >>>>>>>> > >>>>>>>> I've been bootstrapping and testing the patch right now. > >>>>>>> > >>>>>>> Looks like I misremembered the original patch. The issue isn't > >>>>>>> comparing random two elements in the table. > >>>>>>> > >>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > >>>>>>> without INSERTing. > >>>>>>> > >>>>>> > >>>>>> There's updated version of the patch where I check all find operations > >>>>>> (both w/ and w/o insertion). > >>>>>> > >>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > >>>>>> except for: > >>>>>> > >>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > >>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value > >>>>>> during GIMPLE pass: lim > >>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > >>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > >>>>>> 6 | fn1 () > >>>>>> | ^~~ > >>>>>> 0x6c5725 hashtab_chk_error > >>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > >>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > >>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > >>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > >>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > >>>>>> 0xe504ea gather_mem_refs_stmt > >>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > >>>>>> 0xe504ea analyze_memory_references > >>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > >>>>>> 0xe504ea tree_ssa_lim > >>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > >>>>>> 0xe504ea execute > >>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > >>>>>> > >>>>>> Richi: it's after your recent patch. > >>>>>> > >>>>>> For some reason I don't see PR87847 issue any longer. > >>>>>> > >>>>>> > >>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > >>>>> Don't we still need to deal with the naked fprintf when there's a > >>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > >>>> > >>>> Good point, I've just adjusted that. > >>>> > >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >>>> > >>>> Ready to be installed? > >>> > >>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone > >>> fixing it ... > >> > >> Yes :D It's been some time and there's no interest in the PR. > >> > >>> > >>> One question - there's unconditional > >>> > >>> + if (m_sanitize_eq_and_hash) > >>> + verify (comparable, hash); > >>> > >>> which will read a global variable and have (possibly not inline) call > >>> to verify on a common path even with checking disabled. So I think > >>> we want to compile this checking feature out for !CHECKING_P > >>> or at least make the if __builtin_expect (..., 0), ::verify not > >>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). > >> > >> Fixed. May I install the patch? The cselib issue can be solved later.. > > > > You missed the second occurance > > > > - m_searches++; > > + if (m_sanitize_eq_and_hash) > > + verify (comparable, hash); > > Yep ;) I've just install the patch. This is breaking my build: /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for call to ‘hash_table<hash_map<mem_alloc_d\ escription<mem_usage>::mem_location_hash, mem_usage*, simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, false, xcallocator>::hash_table(size_t&, bo\ ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’ : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} Looks like this needs to be updated to pass an argument to the new sanitize_eq_and_hash parameter. Jason ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-07 21:43 ` Jason Merrill @ 2019-06-10 7:08 ` Martin Liška 2019-06-10 18:22 ` Jason Merrill 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2019-06-10 7:08 UTC (permalink / raw) To: Jason Merrill Cc: Richard Biener, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On 6/7/19 11:43 PM, Jason Merrill wrote: > On Fri, Jun 7, 2019 at 8:14 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> On 6/7/19 2:09 PM, Richard Biener wrote: >>> On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>> >>>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>> >>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>> >>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>> Jakub >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>>>>>> void expand (); >>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>>> expand (); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>>> Hi. >>>>>>>>>>>>> >>>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>>> >>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>>> >>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>>> just thinking about loud. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>>> issue :-) >>>>>>>>>>> >>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>>> >>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>>> all other elements? >>>>>>>>>> >>>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>>> PR90450 and PR87847. >>>>>>>>>> >>>>>>>>>> Changes from previous version: >>>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>>> to limit number of elements that are compared within a table >>>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>>> >>>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>>> >>>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>>> comparing random two elements in the table. >>>>>>>>> >>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>>> without INSERTing. >>>>>>>>> >>>>>>>> >>>>>>>> There's updated version of the patch where I check all find operations >>>>>>>> (both w/ and w/o insertion). >>>>>>>> >>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>>> except for: >>>>>>>> >>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>>> during GIMPLE pass: lim >>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>>> 6 | fn1 () >>>>>>>> | ^~~ >>>>>>>> 0x6c5725 hashtab_chk_error >>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>>> 0xe504ea analyze_memory_references >>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>>> 0xe504ea tree_ssa_lim >>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>>> 0xe504ea execute >>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>>> >>>>>>>> Richi: it's after your recent patch. >>>>>>>> >>>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>>> >>>>>>>> >>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>>> >>>>>> Good point, I've just adjusted that. >>>>>> >>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>> >>>>>> Ready to be installed? >>>>> >>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>>> fixing it ... >>>> >>>> Yes :D It's been some time and there's no interest in the PR. >>>> >>>>> >>>>> One question - there's unconditional >>>>> >>>>> + if (m_sanitize_eq_and_hash) >>>>> + verify (comparable, hash); >>>>> >>>>> which will read a global variable and have (possibly not inline) call >>>>> to verify on a common path even with checking disabled. So I think >>>>> we want to compile this checking feature out for !CHECKING_P >>>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>>> >>>> Fixed. May I install the patch? The cselib issue can be solved later.. >>> >>> You missed the second occurance >>> >>> - m_searches++; >>> + if (m_sanitize_eq_and_hash) >>> + verify (comparable, hash); >> >> Yep ;) I've just install the patch. > > This is breaking my build: > > /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for > call to âhash_table<hash_map<mem_alloc_d\ > escription<mem_usage>::mem_location_hash, mem_usage*, > simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ > ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, > false, xcallocator>::hash_table(size_t&, bo\ > ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)â > : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} > > Looks like this needs to be updated to pass an argument to the new > sanitize_eq_and_hash parameter. > > Jason > Hi. Sorry for the breakage, I've just fixed that in r272104. Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-10 7:08 ` Martin Liška @ 2019-06-10 18:22 ` Jason Merrill 2019-06-11 7:41 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Jason Merrill @ 2019-06-10 18:22 UTC (permalink / raw) To: Martin Liška Cc: Richard Biener, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote: > On 6/7/19 11:43 PM, Jason Merrill wrote: > > On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote: > >> On 6/7/19 2:09 PM, Richard Biener wrote: > >>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote: > >>>> On 6/7/19 10:57 AM, Richard Biener wrote: > >>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: > >>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote: > >>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: > >>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>> > >>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: > >>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>>>>>>>>>> ^^^^^^ > >>>>>>>>>>>>>>>> Sure, fixed in attached patch. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Martin > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>>>> Jakub > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> --- > >>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>>>>>>>>>> --- a/gcc/hash-table.h > >>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>>>>>>>>>> void expand (); > >>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>>>>>>>>>> expand (); > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> - m_searches++; > >>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>>> + if (insert == INSERT) > >>>>>>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>>>>>> +#endif > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> + m_searches++; > >>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>>> return &m_entries[index]; > >>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>>>>>>>>>> +static void > >>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>> + "of values with a different hash value\n"); > >>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>>>>>>>>>> because we've got too many bugs to fix. > >>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>>>>>>>>>> Hi. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I've just added one more PR: > >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>>>>>>>>>> > >>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. > >>>>>>>>>>>>> > >>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>>>>>>>>>> limits maximal number of checks: > >>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, > >>>>>>>>>>>> just thinking about loud. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>>>>>>>>>> issue :-) > >>>>>>>>>>> > >>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the > >>>>>>>>>>> table are never compared against each other but always against another > >>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the > >>>>>>>>>>> comparison function only works with those. With the patch we verify > >>>>>>>>>>> hashing/comparison for something that is never used. > >>>>>>>>>>> > >>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>>>>>>>>>> at lookup time, using the object from the lookup and verify that against > >>>>>>>>>>> all other elements? > >>>>>>>>>> > >>>>>>>>>> I don't a have problem with that. Apparently this changes fixes > >>>>>>>>>> PR90450 and PR87847. > >>>>>>>>>> > >>>>>>>>>> Changes from previous version: > >>>>>>>>>> - verification happens only when an element is searched (not inserted) > >>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >>>>>>>>>> - new param has been introduced hash-table-verification-limit in order > >>>>>>>>>> to limit number of elements that are compared within a table > >>>>>>>>>> - verification happens only with flag_checking >= 2 > >>>>>>>>>> > >>>>>>>>>> I've been bootstrapping and testing the patch right now. > >>>>>>>>> > >>>>>>>>> Looks like I misremembered the original patch. The issue isn't > >>>>>>>>> comparing random two elements in the table. > >>>>>>>>> > >>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > >>>>>>>>> without INSERTing. > >>>>>>>>> > >>>>>>>> > >>>>>>>> There's updated version of the patch where I check all find operations > >>>>>>>> (both w/ and w/o insertion). > >>>>>>>> > >>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > >>>>>>>> except for: > >>>>>>>> > >>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > >>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value > >>>>>>>> during GIMPLE pass: lim > >>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > >>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > >>>>>>>> 6 | fn1 () > >>>>>>>> | ^~~ > >>>>>>>> 0x6c5725 hashtab_chk_error > >>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > >>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > >>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > >>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > >>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > >>>>>>>> 0xe504ea gather_mem_refs_stmt > >>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > >>>>>>>> 0xe504ea analyze_memory_references > >>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > >>>>>>>> 0xe504ea tree_ssa_lim > >>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > >>>>>>>> 0xe504ea execute > >>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > >>>>>>>> > >>>>>>>> Richi: it's after your recent patch. > >>>>>>>> > >>>>>>>> For some reason I don't see PR87847 issue any longer. > >>>>>>>> > >>>>>>>> > >>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > >>>>>>> Don't we still need to deal with the naked fprintf when there's a > >>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > >>>>>> > >>>>>> Good point, I've just adjusted that. > >>>>>> > >>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >>>>>> > >>>>>> Ready to be installed? > >>>>> > >>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone > >>>>> fixing it ... > >>>> > >>>> Yes :D It's been some time and there's no interest in the PR. > >>>> > >>>>> > >>>>> One question - there's unconditional > >>>>> > >>>>> + if (m_sanitize_eq_and_hash) > >>>>> + verify (comparable, hash); > >>>>> > >>>>> which will read a global variable and have (possibly not inline) call > >>>>> to verify on a common path even with checking disabled. So I think > >>>>> we want to compile this checking feature out for !CHECKING_P > >>>>> or at least make the if __builtin_expect (..., 0), ::verify not > >>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). > >>>> > >>>> Fixed. May I install the patch? The cselib issue can be solved later.. > >>> > >>> You missed the second occurance > >>> > >>> - m_searches++; > >>> + if (m_sanitize_eq_and_hash) > >>> + verify (comparable, hash); > >> > >> Yep ;) I've just install the patch. > > > > This is breaking my build: > > > > /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for > > call to ‘hash_table<hash_map<mem_alloc_d\ > > escription<mem_usage>::mem_location_hash, mem_usage*, > > simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ > > ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, > > false, xcallocator>::hash_table(size_t&, bo\ > > ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’ > > : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} > > > > Looks like this needs to be updated to pass an argument to the new > > sanitize_eq_and_hash parameter. > > > > Jason > > Sorry for the breakage, I've just fixed that in r272104. Thanks. I'm also seeing a massive compile time hit from this: A constexpr testcase that I've been looking at went from compiling in 13 seconds to 78 seconds, 6 times as long. I would expect template-heavy code to see similar problems when sanitization is enabled for those hash tables. Could we keep the parameter low or 0 by default, and just do occasional sanitize runs with it explicitly enabled? Jason ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-10 18:22 ` Jason Merrill @ 2019-06-11 7:41 ` Martin Liška 2019-06-11 12:28 ` Jason Merrill 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2019-06-11 7:41 UTC (permalink / raw) To: Jason Merrill Cc: Richard Biener, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On 6/10/19 8:21 PM, Jason Merrill wrote: > On Mon, Jun 10, 2019 at 3:08 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> On 6/7/19 11:43 PM, Jason Merrill wrote: >>> On Fri, Jun 7, 2019 at 8:14 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>> On 6/7/19 2:09 PM, Richard Biener wrote: >>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>> Jakub >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>>>>>>>> void expand (); >>>>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>>>>> expand (); >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>>>>> just thinking about loud. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>>>>> issue :-) >>>>>>>>>>>>> >>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>>>>> >>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>>>>> all other elements? >>>>>>>>>>>> >>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>>>>> PR90450 and PR87847. >>>>>>>>>>>> >>>>>>>>>>>> Changes from previous version: >>>>>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>>>>> to limit number of elements that are compared within a table >>>>>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>>>>> >>>>>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>>>>> >>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>>>>> comparing random two elements in the table. >>>>>>>>>>> >>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>>>>> without INSERTing. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> There's updated version of the patch where I check all find operations >>>>>>>>>> (both w/ and w/o insertion). >>>>>>>>>> >>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>>>>> except for: >>>>>>>>>> >>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>>>>> during GIMPLE pass: lim >>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>>>>> 6 | fn1 () >>>>>>>>>> | ^~~ >>>>>>>>>> 0x6c5725 hashtab_chk_error >>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>>>>> 0xe504ea analyze_memory_references >>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>>>>> 0xe504ea tree_ssa_lim >>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>>>>> 0xe504ea execute >>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>>>>> >>>>>>>>>> Richi: it's after your recent patch. >>>>>>>>>> >>>>>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>>>>> >>>>>>>> Good point, I've just adjusted that. >>>>>>>> >>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>>>> >>>>>>>> Ready to be installed? >>>>>>> >>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>>>>> fixing it ... >>>>>> >>>>>> Yes :D It's been some time and there's no interest in the PR. >>>>>> >>>>>>> >>>>>>> One question - there's unconditional >>>>>>> >>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>> + verify (comparable, hash); >>>>>>> >>>>>>> which will read a global variable and have (possibly not inline) call >>>>>>> to verify on a common path even with checking disabled. So I think >>>>>>> we want to compile this checking feature out for !CHECKING_P >>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>>>>> >>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. >>>>> >>>>> You missed the second occurance >>>>> >>>>> - m_searches++; >>>>> + if (m_sanitize_eq_and_hash) >>>>> + verify (comparable, hash); >>>> >>>> Yep ;) I've just install the patch. >>> >>> This is breaking my build: >>> >>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for >>> call to âhash_table<hash_map<mem_alloc_d\ >>> escription<mem_usage>::mem_location_hash, mem_usage*, >>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ >>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, >>> false, xcallocator>::hash_table(size_t&, bo\ >>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)â >>> : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} >>> >>> Looks like this needs to be updated to pass an argument to the new >>> sanitize_eq_and_hash parameter. >>> >>> Jason >> >> Sorry for the breakage, I've just fixed that in r272104. > > Thanks. I'm also seeing a massive compile time hit from this: A > constexpr testcase that I've been looking at went from compiling in 13 > seconds to 78 seconds, 6 times as long. I would expect template-heavy > code to see similar problems when sanitization is enabled for those > hash tables. Could we keep the parameter low or 0 by default, and > just do occasional sanitize runs with it explicitly enabled? Makes sense to me. Can you please provide a test-case which I can measure? Martin > > Jason > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-11 7:41 ` Martin Liška @ 2019-06-11 12:28 ` Jason Merrill 2019-06-11 13:16 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Jason Merrill @ 2019-06-11 12:28 UTC (permalink / raw) To: Martin Liška Cc: Richard Biener, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On 6/11/19 3:41 AM, Martin LiÅ¡ka wrote: > On 6/10/19 8:21 PM, Jason Merrill wrote: >> On Mon, Jun 10, 2019 at 3:08 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>> On 6/7/19 11:43 PM, Jason Merrill wrote: >>>> On Fri, Jun 7, 2019 at 8:14 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>> On 6/7/19 2:09 PM, Richard Biener wrote: >>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>> Jakub >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>>>>>>>>> void expand (); >>>>>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>>>>>> expand (); >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>>>>>> just thinking about loud. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>>>>>> issue :-) >>>>>>>>>>>>>> >>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>>>>>> all other elements? >>>>>>>>>>>>> >>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>>>>>> PR90450 and PR87847. >>>>>>>>>>>>> >>>>>>>>>>>>> Changes from previous version: >>>>>>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>>>>>> to limit number of elements that are compared within a table >>>>>>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>>>>>> >>>>>>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>>>>>> >>>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>>>>>> comparing random two elements in the table. >>>>>>>>>>>> >>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>>>>>> without INSERTing. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> There's updated version of the patch where I check all find operations >>>>>>>>>>> (both w/ and w/o insertion). >>>>>>>>>>> >>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>>>>>> except for: >>>>>>>>>>> >>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>>>>>> during GIMPLE pass: lim >>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>>>>>> 6 | fn1 () >>>>>>>>>>> | ^~~ >>>>>>>>>>> 0x6c5725 hashtab_chk_error >>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>>>>>> 0xe504ea analyze_memory_references >>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>>>>>> 0xe504ea tree_ssa_lim >>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>>>>>> 0xe504ea execute >>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>>>>>> >>>>>>>>>>> Richi: it's after your recent patch. >>>>>>>>>>> >>>>>>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>>>>>> >>>>>>>>> Good point, I've just adjusted that. >>>>>>>>> >>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>>>>> >>>>>>>>> Ready to be installed? >>>>>>>> >>>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>>>>>> fixing it ... >>>>>>> >>>>>>> Yes :D It's been some time and there's no interest in the PR. >>>>>>> >>>>>>>> >>>>>>>> One question - there's unconditional >>>>>>>> >>>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>>> + verify (comparable, hash); >>>>>>>> >>>>>>>> which will read a global variable and have (possibly not inline) call >>>>>>>> to verify on a common path even with checking disabled. So I think >>>>>>>> we want to compile this checking feature out for !CHECKING_P >>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>>>>>> >>>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. >>>>>> >>>>>> You missed the second occurance >>>>>> >>>>>> - m_searches++; >>>>>> + if (m_sanitize_eq_and_hash) >>>>>> + verify (comparable, hash); >>>>> >>>>> Yep ;) I've just install the patch. >>>> >>>> This is breaking my build: >>>> >>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for >>>> call to âhash_table<hash_map<mem_alloc_d\ >>>> escription<mem_usage>::mem_location_hash, mem_usage*, >>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ >>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, >>>> false, xcallocator>::hash_table(size_t&, bo\ >>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)â >>>> : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} >>>> >>>> Looks like this needs to be updated to pass an argument to the new >>>> sanitize_eq_and_hash parameter. >>>> >>>> Jason >>> >>> Sorry for the breakage, I've just fixed that in r272104. >> >> Thanks. I'm also seeing a massive compile time hit from this: A >> constexpr testcase that I've been looking at went from compiling in 13 >> seconds to 78 seconds, 6 times as long. I would expect template-heavy >> code to see similar problems when sanitization is enabled for those >> hash tables. Could we keep the parameter low or 0 by default, and >> just do occasional sanitize runs with it explicitly enabled? > > Makes sense to me. Can you please provide a test-case which I can measure? This is the one I've been looking at: struct Int { constexpr Int(int v): v(v) {} constexpr Int& operator+=(Int b) { this->v += b.v; return *this; } constexpr Int& operator++() { ++this->v; return *this; } private: friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; } int v; }; constexpr int f(int n) { Int i = {0}; Int k = {0}; k = 0; for (; k<10000; ++k) { i += k; } return n; } template<int N> struct S { static constexpr int sm = S<N-1>::sm+f(N); }; template<> struct S<0> { static constexpr int sm = 0; }; constexpr int r = S<20>::sm; Jason ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-11 12:28 ` Jason Merrill @ 2019-06-11 13:16 ` Martin Liška 2019-06-11 19:02 ` Jason Merrill 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2019-06-11 13:16 UTC (permalink / raw) To: Jason Merrill Cc: Richard Biener, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On 6/11/19 2:27 PM, Jason Merrill wrote: > On 6/11/19 3:41 AM, Martin LiÅ¡ka wrote: >> On 6/10/19 8:21 PM, Jason Merrill wrote: >>> On Mon, Jun 10, 2019 at 3:08 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>> On 6/7/19 11:43 PM, Jason Merrill wrote: >>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>> On 6/7/19 2:09 PM, Richard Biener wrote: >>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>>>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>>> +          "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>>> +          "of values with a different hash value"); >>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>>>>>>>                     ^^^^^^ >>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>>>   Jakub >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>  From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>>>>>>>    bool too_empty_p (unsigned int); >>>>>>>>>>>>>>>>>>>>    void expand (); >>>>>>>>>>>>>>>>>>>>    static bool is_deleted (value_type &v) >>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>>>>>>>      expand (); >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>> +   if (insert == INSERT) >>>>>>>>>>>>>>>>>>>> +     verify (comparable, hash); >>>>>>>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>>>>>>>    value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>    return &m_entries[index]; >>>>>>>>>>>>>>>>>>>>  } >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>> +    "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>> +    "of values with a different hash value\n"); >>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>>>>>>> just thinking about loud. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>>>>>>> issue :-) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>>>>>>> all other elements? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>>>>>>> PR90450 and PR87847. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Changes from previous version: >>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>>>>>>>   to limit number of elements that are compared within a table >>>>>>>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>>>>>>> >>>>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>>>>>>> comparing random two elements in the table. >>>>>>>>>>>>> >>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>>>>>>> without INSERTing. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> There's updated version of the patch where I check all find operations >>>>>>>>>>>> (both w/ and w/o insertion). >>>>>>>>>>>> >>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>>>>>>> except for: >>>>>>>>>>>> >>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>>>>>>> during GIMPLE pass: lim >>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>>>>>>>     6 | fn1 () >>>>>>>>>>>>       | ^~~ >>>>>>>>>>>> 0x6c5725 hashtab_chk_error >>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>>>>>>> 0xe504ea analyze_memory_references >>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>>>>>>> 0xe504ea tree_ssa_lim >>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>>>>>>> 0xe504ea execute >>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>>>>>>> >>>>>>>>>>>> Richi: it's after your recent patch. >>>>>>>>>>>> >>>>>>>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>>>>>>> >>>>>>>>>> Good point, I've just adjusted that. >>>>>>>>>> >>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>>>>>> >>>>>>>>>> Ready to be installed? >>>>>>>>> >>>>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>>>>>>> fixing it ... >>>>>>>> >>>>>>>> Yes :D It's been some time and there's no interest in the PR. >>>>>>>> >>>>>>>>> >>>>>>>>> One question - there's unconditional >>>>>>>>> >>>>>>>>> +        if (m_sanitize_eq_and_hash) >>>>>>>>> +          verify (comparable, hash); >>>>>>>>> >>>>>>>>> which will read a global variable and have (possibly not inline) call >>>>>>>>> to verify on a common path even with checking disabled. So I think >>>>>>>>> we want to compile this checking feature out for !CHECKING_P >>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>>>>>>> >>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. >>>>>>> >>>>>>> You missed the second occurance >>>>>>> >>>>>>> - m_searches++; >>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>> +   verify (comparable, hash); >>>>>> >>>>>> Yep ;) I've just install the patch. >>>>> >>>>> This is breaking my build: >>>>> >>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for >>>>> call to âhash_table<hash_map<mem_alloc_d\ >>>>> escription<mem_usage>::mem_location_hash, mem_usage*, >>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ >>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, >>>>> false, xcallocator>::hash_table(size_t&, bo\ >>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)â >>>>>      : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} >>>>> >>>>> Looks like this needs to be updated to pass an argument to the new >>>>> sanitize_eq_and_hash parameter. >>>>> >>>>> Jason >>>> >>>> Sorry for the breakage, I've just fixed that in r272104. >>> >>> Thanks. I'm also seeing a massive compile time hit from this: A >>> constexpr testcase that I've been looking at went from compiling in 13 >>> seconds to 78 seconds, 6 times as long. I would expect template-heavy >>> code to see similar problems when sanitization is enabled for those >>> hash tables. Could we keep the parameter low or 0 by default, and >>> just do occasional sanitize runs with it explicitly enabled? >> >> Makes sense to me. Can you please provide a test-case which I can measure? > > This is the one I've been looking at: > >  struct Int { >    constexpr Int(int v): v(v) {} >    constexpr Int& operator+=(Int b) { this->v += b.v; return *this; } >    constexpr Int& operator++() { ++this->v; return *this; } >  private: >    friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; } >    int v; >  }; >  constexpr int f(int n) { >    Int i = {0}; >    Int k = {0}; >    k = 0; >    for (; k<10000; ++k) { >      i += k; >    } >    return n; >  } > >  template<int N> struct S { >    static constexpr int sm = S<N-1>::sm+f(N); >  }; >  template<> struct S<0> { >    static constexpr int sm = 0; >  }; >  constexpr int r = S<20>::sm; > > Jason For the test-case provided I see: $ time g++ time.cc -c --param hash-table-verification-limit=100 real 0m1.855s user 0m1.829s sys 0m0.025s $ time g++ time.cc -c --param hash-table-verification-limit=0 real 0m1.275s user 0m1.219s sys 0m0.052s $ time g++-9 time.cc -c real 0m0.939s user 0m0.827s sys 0m0.109s So it's slower, but I can't confirm the huge slowdown you see. Is it due to r272144? Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-11 13:16 ` Martin Liška @ 2019-06-11 19:02 ` Jason Merrill 2019-06-12 7:59 ` Richard Biener 0 siblings, 1 reply; 53+ messages in thread From: Jason Merrill @ 2019-06-11 19:02 UTC (permalink / raw) To: Martin Liška Cc: Richard Biener, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On 6/11/19 9:16 AM, Martin LiÅ¡ka wrote: > On 6/11/19 2:27 PM, Jason Merrill wrote: >> On 6/11/19 3:41 AM, Martin LiÅ¡ka wrote: >>> On 6/10/19 8:21 PM, Jason Merrill wrote: >>>> On Mon, Jun 10, 2019 at 3:08 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>> On 6/7/19 11:43 PM, Jason Merrill wrote: >>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote: >>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>>>>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>>>> +          "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>>>> +          "of values with a different hash value"); >>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>>>>>>>>                     ^^^^^^ >>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>>>>   Jakub >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>  From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>>  gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>>>>>>>>  1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>    value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>>>>>>>>    value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>>>>>>>>    bool too_empty_p (unsigned int); >>>>>>>>>>>>>>>>>>>>>    void expand (); >>>>>>>>>>>>>>>>>>>>>    static bool is_deleted (value_type &v) >>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>>    if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>>>>>>>>      expand (); >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>> +   if (insert == INSERT) >>>>>>>>>>>>>>>>>>>>> +     verify (comparable, hash); >>>>>>>>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>>>>>>>>    value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>>>>>>>>    hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>>>    hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>>    return &m_entries[index]; >>>>>>>>>>>>>>>>>>>>>  } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>> +    "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>> +    "of values with a different hash value\n"); >>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>>>>>>>> just thinking about loud. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>>>>>>>> issue :-) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>>>>>>>> all other elements? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>>>>>>>> PR90450 and PR87847. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Changes from previous version: >>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>>>>>>>>   to limit number of elements that are compared within a table >>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>>>>>>>> comparing random two elements in the table. >>>>>>>>>>>>>> >>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>>>>>>>> without INSERTing. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> There's updated version of the patch where I check all find operations >>>>>>>>>>>>> (both w/ and w/o insertion). >>>>>>>>>>>>> >>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>>>>>>>> except for: >>>>>>>>>>>>> >>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>>>>>>>> during GIMPLE pass: lim >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>>>>>>>>     6 | fn1 () >>>>>>>>>>>>>       | ^~~ >>>>>>>>>>>>> 0x6c5725 hashtab_chk_error >>>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>>>>>>>> 0xe504ea analyze_memory_references >>>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>>>>>>>> 0xe504ea tree_ssa_lim >>>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>>>>>>>> 0xe504ea execute >>>>>>>>>>>>>      /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>>>>>>>> >>>>>>>>>>>>> Richi: it's after your recent patch. >>>>>>>>>>>>> >>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>>>>>>>> >>>>>>>>>>> Good point, I've just adjusted that. >>>>>>>>>>> >>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>>>>>>> >>>>>>>>>>> Ready to be installed? >>>>>>>>>> >>>>>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>>>>>>>> fixing it ... >>>>>>>>> >>>>>>>>> Yes :D It's been some time and there's no interest in the PR. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> One question - there's unconditional >>>>>>>>>> >>>>>>>>>> +        if (m_sanitize_eq_and_hash) >>>>>>>>>> +          verify (comparable, hash); >>>>>>>>>> >>>>>>>>>> which will read a global variable and have (possibly not inline) call >>>>>>>>>> to verify on a common path even with checking disabled. So I think >>>>>>>>>> we want to compile this checking feature out for !CHECKING_P >>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>>>>>>>> >>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. >>>>>>>> >>>>>>>> You missed the second occurance >>>>>>>> >>>>>>>> - m_searches++; >>>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>>> +   verify (comparable, hash); >>>>>>> >>>>>>> Yep ;) I've just install the patch. >>>>>> >>>>>> This is breaking my build: >>>>>> >>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for >>>>>> call to âhash_table<hash_map<mem_alloc_d\ >>>>>> escription<mem_usage>::mem_location_hash, mem_usage*, >>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ >>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, >>>>>> false, xcallocator>::hash_table(size_t&, bo\ >>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)â >>>>>>      : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} >>>>>> >>>>>> Looks like this needs to be updated to pass an argument to the new >>>>>> sanitize_eq_and_hash parameter. >>>>>> >>>>>> Jason >>>>> >>>>> Sorry for the breakage, I've just fixed that in r272104. >>>> >>>> Thanks. I'm also seeing a massive compile time hit from this: A >>>> constexpr testcase that I've been looking at went from compiling in 13 >>>> seconds to 78 seconds, 6 times as long. I would expect template-heavy >>>> code to see similar problems when sanitization is enabled for those >>>> hash tables. Could we keep the parameter low or 0 by default, and >>>> just do occasional sanitize runs with it explicitly enabled? >>> >>> Makes sense to me. Can you please provide a test-case which I can measure? >> >> This is the one I've been looking at: >> >>  struct Int { >>    constexpr Int(int v): v(v) {} >>    constexpr Int& operator+=(Int b) { this->v += b.v; return *this; } >>    constexpr Int& operator++() { ++this->v; return *this; } >>  private: >>    friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; } >>    int v; >>  }; >>  constexpr int f(int n) { >>    Int i = {0}; >>    Int k = {0}; >>    k = 0; >>    for (; k<10000; ++k) { >>      i += k; >>    } >>    return n; >>  } >> >>  template<int N> struct S { >>    static constexpr int sm = S<N-1>::sm+f(N); >>  }; >>  template<> struct S<0> { >>    static constexpr int sm = 0; >>  }; >>  constexpr int r = S<20>::sm; >> >> Jason > > For the test-case provided I see: > > $ time g++ time.cc -c --param hash-table-verification-limit=100 > > real 0m1.855s > user 0m1.829s > sys 0m0.025s > > $ time g++ time.cc -c --param hash-table-verification-limit=0 > > real 0m1.275s > user 0m1.219s > sys 0m0.052s > > $ time g++-9 time.cc -c > > real 0m0.939s > user 0m0.827s > sys 0m0.109s > > So it's slower, but I can't confirm the huge slowdown you see. > Is it due to r272144? Hmm, I wonder if this is because of the --enable-gather-detailed-mem-stats hash tables. Jason ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-11 19:02 ` Jason Merrill @ 2019-06-12 7:59 ` Richard Biener 2019-06-12 8:02 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-06-12 7:59 UTC (permalink / raw) To: Jason Merrill Cc: Martin Liška, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote: > > On 6/11/19 9:16 AM, Martin Liška wrote: > > On 6/11/19 2:27 PM, Jason Merrill wrote: > >> On 6/11/19 3:41 AM, Martin Liška wrote: > >>> On 6/10/19 8:21 PM, Jason Merrill wrote: > >>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote: > >>>>> On 6/7/19 11:43 PM, Jason Merrill wrote: > >>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote: > >>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote: > >>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: > >>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: > >>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote: > >>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: > >>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: > >>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>>>>>>>>>>>>>>> ^^^^^^ > >>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Martin > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>>>>>>>>> Jakub > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> --- > >>>>>>>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h > >>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>>>>>>>>>>>>>>> void expand (); > >>>>>>>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>>>>>>>>>>>>>>> expand (); > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> - m_searches++; > >>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>>>>>>>> + if (insert == INSERT) > >>>>>>>>>>>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>>>>>>>>>>> +#endif > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> + m_searches++; > >>>>>>>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>>>>>>>> return &m_entries[index]; > >>>>>>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>>>>>>>>>>>>>>> +static void > >>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value\n"); > >>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix. > >>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>>>>>>>>>>>>>>> Hi. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I've just added one more PR: > >>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>>>>>>>>>>>>>>> limits maximal number of checks: > >>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>>>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, > >>>>>>>>>>>>>>>>> just thinking about loud. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>>>>>>>>>>>>>>> issue :-) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the > >>>>>>>>>>>>>>>> table are never compared against each other but always against another > >>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the > >>>>>>>>>>>>>>>> comparison function only works with those. With the patch we verify > >>>>>>>>>>>>>>>> hashing/comparison for something that is never used. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against > >>>>>>>>>>>>>>>> all other elements? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes > >>>>>>>>>>>>>>> PR90450 and PR87847. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Changes from previous version: > >>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted) > >>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order > >>>>>>>>>>>>>>> to limit number of elements that are compared within a table > >>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't > >>>>>>>>>>>>>> comparing random two elements in the table. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > >>>>>>>>>>>>>> without INSERTing. > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> There's updated version of the patch where I check all find operations > >>>>>>>>>>>>> (both w/ and w/o insertion). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > >>>>>>>>>>>>> except for: > >>>>>>>>>>>>> > >>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > >>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value > >>>>>>>>>>>>> during GIMPLE pass: lim > >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > >>>>>>>>>>>>> 6 | fn1 () > >>>>>>>>>>>>> | ^~~ > >>>>>>>>>>>>> 0x6c5725 hashtab_chk_error > >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > >>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > >>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > >>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt > >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > >>>>>>>>>>>>> 0xe504ea analyze_memory_references > >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > >>>>>>>>>>>>> 0xe504ea tree_ssa_lim > >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > >>>>>>>>>>>>> 0xe504ea execute > >>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > >>>>>>>>>>>>> > >>>>>>>>>>>>> Richi: it's after your recent patch. > >>>>>>>>>>>>> > >>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > >>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a > >>>>>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > >>>>>>>>>>> > >>>>>>>>>>> Good point, I've just adjusted that. > >>>>>>>>>>> > >>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >>>>>>>>>>> > >>>>>>>>>>> Ready to be installed? > >>>>>>>>>> > >>>>>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone > >>>>>>>>>> fixing it ... > >>>>>>>>> > >>>>>>>>> Yes :D It's been some time and there's no interest in the PR. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> One question - there's unconditional > >>>>>>>>>> > >>>>>>>>>> + if (m_sanitize_eq_and_hash) > >>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>> > >>>>>>>>>> which will read a global variable and have (possibly not inline) call > >>>>>>>>>> to verify on a common path even with checking disabled. So I think > >>>>>>>>>> we want to compile this checking feature out for !CHECKING_P > >>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not > >>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). > >>>>>>>>> > >>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. > >>>>>>>> > >>>>>>>> You missed the second occurance > >>>>>>>> > >>>>>>>> - m_searches++; > >>>>>>>> + if (m_sanitize_eq_and_hash) > >>>>>>>> + verify (comparable, hash); > >>>>>>> > >>>>>>> Yep ;) I've just install the patch. > >>>>>> > >>>>>> This is breaking my build: > >>>>>> > >>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for > >>>>>> call to ‘hash_table<hash_map<mem_alloc_d\ > >>>>>> escription<mem_usage>::mem_location_hash, mem_usage*, > >>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ > >>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, > >>>>>> false, xcallocator>::hash_table(size_t&, bo\ > >>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’ > >>>>>> : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} > >>>>>> > >>>>>> Looks like this needs to be updated to pass an argument to the new > >>>>>> sanitize_eq_and_hash parameter. > >>>>>> > >>>>>> Jason > >>>>> > >>>>> Sorry for the breakage, I've just fixed that in r272104. > >>>> > >>>> Thanks. I'm also seeing a massive compile time hit from this: A > >>>> constexpr testcase that I've been looking at went from compiling in 13 > >>>> seconds to 78 seconds, 6 times as long. I would expect template-heavy > >>>> code to see similar problems when sanitization is enabled for those > >>>> hash tables. Could we keep the parameter low or 0 by default, and > >>>> just do occasional sanitize runs with it explicitly enabled? > >>> > >>> Makes sense to me. Can you please provide a test-case which I can measure? > >> > >> This is the one I've been looking at: > >> > >> struct Int { > >> constexpr Int(int v): v(v) {} > >> constexpr Int& operator+=(Int b) { this->v += b.v; return *this; } > >> constexpr Int& operator++() { ++this->v; return *this; } > >> private: > >> friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; } > >> int v; > >> }; > >> constexpr int f(int n) { > >> Int i = {0}; > >> Int k = {0}; > >> k = 0; > >> for (; k<10000; ++k) { > >> i += k; > >> } > >> return n; > >> } > >> > >> template<int N> struct S { > >> static constexpr int sm = S<N-1>::sm+f(N); > >> }; > >> template<> struct S<0> { > >> static constexpr int sm = 0; > >> }; > >> constexpr int r = S<20>::sm; > >> > >> Jason > > > > For the test-case provided I see: > > > > $ time g++ time.cc -c --param hash-table-verification-limit=100 > > > > real 0m1.855s > > user 0m1.829s > > sys 0m0.025s > > > > $ time g++ time.cc -c --param hash-table-verification-limit=0 > > > > real 0m1.275s > > user 0m1.219s > > sys 0m0.052s > > > > $ time g++-9 time.cc -c > > > > real 0m0.939s > > user 0m0.827s > > sys 0m0.109s > > > > So it's slower, but I can't confirm the huge slowdown you see. > > Is it due to r272144? > > Hmm, I wonder if this is because of the > --enable-gather-detailed-mem-stats hash tables. I wonder if we can reduce the overhead by making hash-tables/maps using predefined traits not perform the checking? Thus make [the default] whether to check or not to check part of the traits? Surely the basic pointer-hash/map stuff is OK and needs no extra checking. Richard. > Jason ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-12 7:59 ` Richard Biener @ 2019-06-12 8:02 ` Martin Liška 2019-06-12 9:15 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2019-06-12 8:02 UTC (permalink / raw) To: Richard Biener, Jason Merrill Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On 6/12/19 9:59 AM, Richard Biener wrote: > On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote: >> >> On 6/11/19 9:16 AM, Martin LiÅ¡ka wrote: >>> On 6/11/19 2:27 PM, Jason Merrill wrote: >>>> On 6/11/19 3:41 AM, Martin LiÅ¡ka wrote: >>>>> On 6/10/19 8:21 PM, Jason Merrill wrote: >>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote: >>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote: >>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>>>>>> Jakub >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>>>>>>>>>>>>> void expand (); >>>>>>>>>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>>>>>>>>>> expand (); >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>>>>>>>>>> just thinking about loud. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>>>>>>>>>> issue :-) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>>>>>>>>>> all other elements? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>>>>>>>>>> PR90450 and PR87847. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Changes from previous version: >>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>>>>>>>>>> to limit number of elements that are compared within a table >>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>>>>>>>>>> comparing random two elements in the table. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>>>>>>>>>> without INSERTing. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations >>>>>>>>>>>>>>> (both w/ and w/o insertion). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>>>>>>>>>> except for: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>>>>>>>>>> during GIMPLE pass: lim >>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>>>>>>>>>> 6 | fn1 () >>>>>>>>>>>>>>> | ^~~ >>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error >>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>>>>>>>>>> 0xe504ea analyze_memory_references >>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim >>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>>>>>>>>>> 0xe504ea execute >>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Richi: it's after your recent patch. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>>>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>>>>>>>>>> >>>>>>>>>>>>> Good point, I've just adjusted that. >>>>>>>>>>>>> >>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>>>>>>>>> >>>>>>>>>>>>> Ready to be installed? >>>>>>>>>>>> >>>>>>>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>>>>>>>>>> fixing it ... >>>>>>>>>>> >>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> One question - there's unconditional >>>>>>>>>>>> >>>>>>>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>> >>>>>>>>>>>> which will read a global variable and have (possibly not inline) call >>>>>>>>>>>> to verify on a common path even with checking disabled. So I think >>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P >>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>>>>>>>>>> >>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. >>>>>>>>>> >>>>>>>>>> You missed the second occurance >>>>>>>>>> >>>>>>>>>> - m_searches++; >>>>>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>>>>> + verify (comparable, hash); >>>>>>>>> >>>>>>>>> Yep ;) I've just install the patch. >>>>>>>> >>>>>>>> This is breaking my build: >>>>>>>> >>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for >>>>>>>> call to âhash_table<hash_map<mem_alloc_d\ >>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*, >>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ >>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, >>>>>>>> false, xcallocator>::hash_table(size_t&, bo\ >>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)â >>>>>>>> : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} >>>>>>>> >>>>>>>> Looks like this needs to be updated to pass an argument to the new >>>>>>>> sanitize_eq_and_hash parameter. >>>>>>>> >>>>>>>> Jason >>>>>>> >>>>>>> Sorry for the breakage, I've just fixed that in r272104. >>>>>> >>>>>> Thanks. I'm also seeing a massive compile time hit from this: A >>>>>> constexpr testcase that I've been looking at went from compiling in 13 >>>>>> seconds to 78 seconds, 6 times as long. I would expect template-heavy >>>>>> code to see similar problems when sanitization is enabled for those >>>>>> hash tables. Could we keep the parameter low or 0 by default, and >>>>>> just do occasional sanitize runs with it explicitly enabled? >>>>> >>>>> Makes sense to me. Can you please provide a test-case which I can measure? >>>> >>>> This is the one I've been looking at: >>>> >>>> struct Int { >>>> constexpr Int(int v): v(v) {} >>>> constexpr Int& operator+=(Int b) { this->v += b.v; return *this; } >>>> constexpr Int& operator++() { ++this->v; return *this; } >>>> private: >>>> friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; } >>>> int v; >>>> }; >>>> constexpr int f(int n) { >>>> Int i = {0}; >>>> Int k = {0}; >>>> k = 0; >>>> for (; k<10000; ++k) { >>>> i += k; >>>> } >>>> return n; >>>> } >>>> >>>> template<int N> struct S { >>>> static constexpr int sm = S<N-1>::sm+f(N); >>>> }; >>>> template<> struct S<0> { >>>> static constexpr int sm = 0; >>>> }; >>>> constexpr int r = S<20>::sm; >>>> >>>> Jason >>> >>> For the test-case provided I see: >>> >>> $ time g++ time.cc -c --param hash-table-verification-limit=100 >>> >>> real 0m1.855s >>> user 0m1.829s >>> sys 0m0.025s >>> >>> $ time g++ time.cc -c --param hash-table-verification-limit=0 >>> >>> real 0m1.275s >>> user 0m1.219s >>> sys 0m0.052s >>> >>> $ time g++-9 time.cc -c >>> >>> real 0m0.939s >>> user 0m0.827s >>> sys 0m0.109s >>> >>> So it's slower, but I can't confirm the huge slowdown you see. >>> Is it due to r272144? >> >> Hmm, I wonder if this is because of the >> --enable-gather-detailed-mem-stats hash tables. > > I wonder if we can reduce the overhead by making > hash-tables/maps using predefined traits not perform > the checking? Thus make [the default] whether to check or not > to check part of the traits? Surely the basic pointer-hash/map > stuff is OK and needs no extra checking. Interesting idea! I can prepare a patch. Right now I'm testing a patch that removes sanitization for hash-tables (and maps) that track memory allocations. Martin > > Richard. > >> Jason ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-12 8:02 ` Martin Liška @ 2019-06-12 9:15 ` Martin Liška 2019-06-12 9:41 ` Richard Biener 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2019-06-12 9:15 UTC (permalink / raw) To: Richard Biener, Jason Merrill Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 16616 bytes --] On 6/12/19 10:02 AM, Martin LiÅ¡ka wrote: > On 6/12/19 9:59 AM, Richard Biener wrote: >> On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote: >>> >>> On 6/11/19 9:16 AM, Martin LiÅ¡ka wrote: >>>> On 6/11/19 2:27 PM, Jason Merrill wrote: >>>>> On 6/11/19 3:41 AM, Martin LiÅ¡ka wrote: >>>>>> On 6/10/19 8:21 PM, Jason Merrill wrote: >>>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote: >>>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote: >>>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>>>>>>> Jakub >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>>>>>>>>>>>>>> void expand (); >>>>>>>>>>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>>>>>>>>>>> expand (); >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>>>>>>>>>>> just thinking about loud. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>>>>>>>>>>> issue :-) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>>>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>>>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>>>>>>>>>>> all other elements? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>>>>>>>>>>> PR90450 and PR87847. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Changes from previous version: >>>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>>>>>>>>>>> to limit number of elements that are compared within a table >>>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>>>>>>>>>>> comparing random two elements in the table. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>>>>>>>>>>> without INSERTing. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations >>>>>>>>>>>>>>>> (both w/ and w/o insertion). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>>>>>>>>>>> except for: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>>>>>>>>>>> during GIMPLE pass: lim >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>>>>>>>>>>> 6 | fn1 () >>>>>>>>>>>>>>>> | ^~~ >>>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>>>>>>>>>>> 0xe504ea analyze_memory_references >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>>>>>>>>>>> 0xe504ea execute >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Richi: it's after your recent patch. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>>>>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Good point, I've just adjusted that. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ready to be installed? >>>>>>>>>>>>> >>>>>>>>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>>>>>>>>>>> fixing it ... >>>>>>>>>>>> >>>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> One question - there's unconditional >>>>>>>>>>>>> >>>>>>>>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>> >>>>>>>>>>>>> which will read a global variable and have (possibly not inline) call >>>>>>>>>>>>> to verify on a common path even with checking disabled. So I think >>>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P >>>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>>>>>>>>>>> >>>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. >>>>>>>>>>> >>>>>>>>>>> You missed the second occurance >>>>>>>>>>> >>>>>>>>>>> - m_searches++; >>>>>>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>> >>>>>>>>>> Yep ;) I've just install the patch. >>>>>>>>> >>>>>>>>> This is breaking my build: >>>>>>>>> >>>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for >>>>>>>>> call to âhash_table<hash_map<mem_alloc_d\ >>>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*, >>>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ >>>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, >>>>>>>>> false, xcallocator>::hash_table(size_t&, bo\ >>>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)â >>>>>>>>> : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} >>>>>>>>> >>>>>>>>> Looks like this needs to be updated to pass an argument to the new >>>>>>>>> sanitize_eq_and_hash parameter. >>>>>>>>> >>>>>>>>> Jason >>>>>>>> >>>>>>>> Sorry for the breakage, I've just fixed that in r272104. >>>>>>> >>>>>>> Thanks. I'm also seeing a massive compile time hit from this: A >>>>>>> constexpr testcase that I've been looking at went from compiling in 13 >>>>>>> seconds to 78 seconds, 6 times as long. I would expect template-heavy >>>>>>> code to see similar problems when sanitization is enabled for those >>>>>>> hash tables. Could we keep the parameter low or 0 by default, and >>>>>>> just do occasional sanitize runs with it explicitly enabled? >>>>>> >>>>>> Makes sense to me. Can you please provide a test-case which I can measure? >>>>> >>>>> This is the one I've been looking at: >>>>> >>>>> struct Int { >>>>> constexpr Int(int v): v(v) {} >>>>> constexpr Int& operator+=(Int b) { this->v += b.v; return *this; } >>>>> constexpr Int& operator++() { ++this->v; return *this; } >>>>> private: >>>>> friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; } >>>>> int v; >>>>> }; >>>>> constexpr int f(int n) { >>>>> Int i = {0}; >>>>> Int k = {0}; >>>>> k = 0; >>>>> for (; k<10000; ++k) { >>>>> i += k; >>>>> } >>>>> return n; >>>>> } >>>>> >>>>> template<int N> struct S { >>>>> static constexpr int sm = S<N-1>::sm+f(N); >>>>> }; >>>>> template<> struct S<0> { >>>>> static constexpr int sm = 0; >>>>> }; >>>>> constexpr int r = S<20>::sm; >>>>> >>>>> Jason >>>> >>>> For the test-case provided I see: >>>> >>>> $ time g++ time.cc -c --param hash-table-verification-limit=100 >>>> >>>> real 0m1.855s >>>> user 0m1.829s >>>> sys 0m0.025s >>>> >>>> $ time g++ time.cc -c --param hash-table-verification-limit=0 >>>> >>>> real 0m1.275s >>>> user 0m1.219s >>>> sys 0m0.052s >>>> >>>> $ time g++-9 time.cc -c >>>> >>>> real 0m0.939s >>>> user 0m0.827s >>>> sys 0m0.109s >>>> >>>> So it's slower, but I can't confirm the huge slowdown you see. >>>> Is it due to r272144? >>> >>> Hmm, I wonder if this is because of the >>> --enable-gather-detailed-mem-stats hash tables. >> >> I wonder if we can reduce the overhead by making >> hash-tables/maps using predefined traits not perform >> the checking? Thus make [the default] whether to check or not >> to check part of the traits? Surely the basic pointer-hash/map >> stuff is OK and needs no extra checking. > > Interesting idea! I can prepare a patch. Right now I'm testing a patch > that removes sanitization for hash-tables (and maps) that track memory > allocations. I've got 2 patch candidates that will make it happen. It survives build on --enable-languages=all, but one would need to build all cross-compilers to make a proper testing. Do you like the idea of the patch before I'll write a changelog and test it? Thanks, Martin > > Martin > >> >> Richard. >> >>> Jason > [-- Attachment #2: 0002-Disable-hash-table-sanitization-for-mem-stats-maps.patch --] [-- Type: text/x-patch, Size: 2694 bytes --] From 4888f897e579267abb6ebf94646518f5c9f2da1d Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Wed, 12 Jun 2019 11:04:11 +0200 Subject: [PATCH 2/2] Disable hash-table sanitization for mem stats maps. --- gcc/ggc-common.c | 2 +- gcc/hash-map.h | 9 ++++++--- gcc/mem-stats.h | 6 +++--- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/gcc/ggc-common.c b/gcc/ggc-common.c index 2acdb6dc60c..6fb5a3d5ceb 100644 --- a/gcc/ggc-common.c +++ b/gcc/ggc-common.c @@ -1014,5 +1014,5 @@ ggc_prune_overhead_list (void) (*it).second.first->m_collected += (*it).second.second; delete ggc_mem_desc.m_reverse_object_map; - ggc_mem_desc.m_reverse_object_map = new map_t (13, false, false); + ggc_mem_desc.m_reverse_object_map = new map_t (13, false, false, false); } diff --git a/gcc/hash-map.h b/gcc/hash-map.h index 151632730e7..77f17275ec3 100644 --- a/gcc/hash-map.h +++ b/gcc/hash-map.h @@ -119,16 +119,19 @@ class GTY((user)) hash_map public: explicit hash_map (size_t n = 13, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS CXX_MEM_STAT_INFO) - : m_table (n, ggc, true, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) + : m_table (n, ggc, sanitize_eq_and_hash, gather_mem_stats, + HASH_MAP_ORIGIN PASS_MEM_STAT) { } explicit hash_map (const hash_map &h, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS CXX_MEM_STAT_INFO) - : m_table (h.m_table, ggc, true, gather_mem_stats, + : m_table (h.m_table, ggc, sanitize_eq_and_hash, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} /* Create a hash_map in ggc memory. */ @@ -137,7 +140,7 @@ public: CXX_MEM_STAT_INFO) { hash_map *map = ggc_alloc<hash_map> (); - new (map) hash_map (size, true, gather_mem_stats PASS_MEM_STAT); + new (map) hash_map (size, true, true, gather_mem_stats PASS_MEM_STAT); return map; } diff --git a/gcc/mem-stats.h b/gcc/mem-stats.h index 63ce8712e2b..77960595753 100644 --- a/gcc/mem-stats.h +++ b/gcc/mem-stats.h @@ -559,9 +559,9 @@ template <class T> inline mem_alloc_description<T>::mem_alloc_description () { - m_map = new mem_map_t (13, false, false); - m_reverse_map = new reverse_mem_map_t (13, false, false); - m_reverse_object_map = new reverse_object_map_t (13, false, false); + m_map = new mem_map_t (13, false, false, false); + m_reverse_map = new reverse_mem_map_t (13, false, false, false); + m_reverse_object_map = new reverse_object_map_t (13, false, false, false); } /* Default destructor. */ -- 2.21.0 [-- Attachment #3: 0001-Add-sanitize-into-hash-traits-and-disable-it-for-obv.patch --] [-- Type: text/x-patch, Size: 12263 bytes --] From 7b96fb6e6d6c5d0e574440ecf0b842d71d5a8049 Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Wed, 12 Jun 2019 11:03:33 +0200 Subject: [PATCH 1/2] Add ::sanitize into hash-traits and disable it for obvious types. --- gcc/attribs.c | 5 +++++ gcc/cp/cp-tree.h | 2 ++ gcc/cp/decl2.c | 1 + gcc/gcov.c | 2 ++ gcc/graphite.c | 1 + gcc/hash-map-traits.h | 26 ++++++++++++++++++++++++++ gcc/hash-map.h | 1 + gcc/hash-table.h | 5 +++-- gcc/hash-traits.h | 25 +++++++++++++++++++++++++ gcc/ipa-devirt.c | 2 ++ gcc/ipa-prop.c | 3 +++ gcc/profile.c | 1 + gcc/sanopt.c | 2 ++ gcc/tree-hasher.h | 1 + gcc/tree-ssa-sccvn.c | 1 + gcc/tree-vect-slp.c | 1 + 16 files changed, 77 insertions(+), 2 deletions(-) diff --git a/gcc/attribs.c b/gcc/attribs.c index 4441922543f..06a6b56b634 100644 --- a/gcc/attribs.c +++ b/gcc/attribs.c @@ -2057,6 +2057,11 @@ struct excl_hash_traits: typed_noop_remove<excl_pair> { return !*x.first && !*x.second; } + + static bool sanitize () + { + return true; + } }; diff --git a/gcc/cp/cp-tree.h b/gcc/cp/cp-tree.h index 1f4e1e15554..d0ce3b68693 100644 --- a/gcc/cp/cp-tree.h +++ b/gcc/cp/cp-tree.h @@ -869,6 +869,7 @@ struct named_decl_hash : ggc_remove <tree> /* Nothing is deletable. Everything is insertable. */ static bool is_deleted (value_type) { return false; } static void mark_deleted (value_type) { gcc_unreachable (); } + inline static bool sanitize () { return true; } }; /* Simplified unique_ptr clone to release a tree vec on exit. */ @@ -1815,6 +1816,7 @@ struct named_label_hash : ggc_remove <named_label_entry *> /* Nothing is deletable. Everything is insertable. */ inline static bool is_deleted (value_type) { return false; } inline static void mark_deleted (value_type) { gcc_unreachable (); } + inline static bool sanitize () { return true; } }; /* Global state pertinent to the current function. */ diff --git a/gcc/cp/decl2.c b/gcc/cp/decl2.c index 338db4ab6de..f4f5e878eb7 100644 --- a/gcc/cp/decl2.c +++ b/gcc/cp/decl2.c @@ -131,6 +131,7 @@ struct mangled_decl_hash : ggc_remove <tree> { e = reinterpret_cast <value_type> (1); } + inline static bool sanitize () { return true; } }; /* A hash table of decls keyed by mangled name. Used to figure out if diff --git a/gcc/gcov.c b/gcc/gcov.c index b06a6714c2e..015cac6247a 100644 --- a/gcc/gcov.c +++ b/gcc/gcov.c @@ -1234,6 +1234,8 @@ struct function_start_pair_hash : typed_noop_remove <function_start> { return ref.start_line == ~2U; } + + inline static bool sanitize () { return true; } }; /* Process a single input file. */ diff --git a/gcc/graphite.c b/gcc/graphite.c index 67202e276e2..c85f88e9e34 100644 --- a/gcc/graphite.c +++ b/gcc/graphite.c @@ -237,6 +237,7 @@ struct sese_scev_hash : typed_noop_remove <seir_cache_key> static void mark_empty (seir_cache_key &key) { key.entry_dest = 0; } static bool is_deleted (const seir_cache_key &key) { return !key.expr; } static bool is_empty (const seir_cache_key &key) { return key.entry_dest == 0; } + static bool sanitize () { return true; } }; static hash_map<sese_scev_hash, tree> *seir_cache; diff --git a/gcc/hash-map-traits.h b/gcc/hash-map-traits.h index af66018e92c..a52c272d7e2 100644 --- a/gcc/hash-map-traits.h +++ b/gcc/hash-map-traits.h @@ -40,6 +40,7 @@ struct simple_hashmap_traits template <typename T> static inline bool is_deleted (const T &); template <typename T> static inline void mark_empty (T &); template <typename T> static inline void mark_deleted (T &); + static inline bool sanitize (); }; template <typename H, typename Value> @@ -98,6 +99,13 @@ simple_hashmap_traits <H, Value>::mark_deleted (T &entry) H::mark_deleted (entry.m_key); } +template <typename H, typename Value> +inline bool +simple_hashmap_traits <H, Value>::sanitize () +{ + return H::sanitize (); +} + template <typename H, typename Value> struct simple_cache_map_traits: public simple_hashmap_traits<H,Value> { @@ -117,6 +125,7 @@ struct unbounded_hashmap_traits template <typename T> static inline bool is_deleted (const T &); template <typename T> static inline void mark_empty (T &); template <typename T> static inline void mark_deleted (T &); + template <typename T> static inline bool sanitize (); }; template <typename Value> @@ -159,6 +168,14 @@ unbounded_hashmap_traits <Value>::mark_deleted (T &entry) default_hash_traits <Value>::mark_deleted (entry.m_value); } +template <typename Value> +template <typename T> +inline bool +unbounded_hashmap_traits <Value>::sanitize () +{ + return T::sanitize (); +} + /* Implement traits for a hash_map from integer type Key to Value in cases where Key has no spare values for recording empty and deleted slots. */ @@ -169,6 +186,7 @@ struct unbounded_int_hashmap_traits : unbounded_hashmap_traits <Value> typedef Key key_type; static inline hashval_t hash (Key); static inline bool equal_keys (Key, Key); + static inline bool sanitize (); }; template <typename Key, typename Value> @@ -185,4 +203,12 @@ unbounded_int_hashmap_traits <Key, Value>::equal_keys (Key k1, Key k2) return k1 == k2; } +template <typename Key, typename Value> +inline bool +unbounded_int_hashmap_traits <Key, Value>::sanitize () +{ + return false; +} + + #endif // HASH_MAP_TRAITS_H diff --git a/gcc/hash-map.h b/gcc/hash-map.h index a8eb42d5a03..151632730e7 100644 --- a/gcc/hash-map.h +++ b/gcc/hash-map.h @@ -55,6 +55,7 @@ class GTY((user)) hash_map static void mark_empty (hash_entry &e) { Traits::mark_empty (e); } static bool is_empty (const hash_entry &e) { return Traits::is_empty (e); } + static bool sanitize () { return Traits::sanitize (); } static void ggc_mx (hash_entry &e) { diff --git a/gcc/hash-table.h b/gcc/hash-table.h index 4f5e150a0ac..de5ebc8ba6d 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -600,7 +600,8 @@ hash_table<Descriptor, Lazy, Allocator>::hash_table (size_t size, bool ggc, mem_alloc_origin origin MEM_STAT_DECL) : m_n_elements (0), m_n_deleted (0), m_searches (0), m_collisions (0), - m_ggc (ggc), m_sanitize_eq_and_hash (sanitize_eq_and_hash) + m_ggc (ggc), + m_sanitize_eq_and_hash (sanitize_eq_and_hash && Descriptor::sanitize ()) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif @@ -633,7 +634,7 @@ hash_table<Descriptor, Lazy, Allocator>::hash_table (const hash_table &h, MEM_STAT_DECL) : m_n_elements (h.m_n_elements), m_n_deleted (h.m_n_deleted), m_searches (0), m_collisions (0), m_ggc (ggc), - m_sanitize_eq_and_hash (sanitize_eq_and_hash) + m_sanitize_eq_and_hash (sanitize_eq_and_hash && Descriptor::sanitize ()) #if GATHER_STATISTICS , m_gather_mem_stats (gather_mem_stats) #endif diff --git a/gcc/hash-traits.h b/gcc/hash-traits.h index 2d17e2c982a..37a97c31491 100644 --- a/gcc/hash-traits.h +++ b/gcc/hash-traits.h @@ -91,6 +91,7 @@ struct int_hash : typed_noop_remove <Type> static inline void mark_empty (Type &); static inline bool is_deleted (Type); static inline bool is_empty (Type); + static inline bool sanitize (); }; template <typename Type, Type Empty, Type Deleted> @@ -136,6 +137,13 @@ int_hash <Type, Empty, Deleted>::is_empty (Type x) return x == Empty; } +template <typename Type, Type Empty, Type Deleted> +inline bool +int_hash <Type, Empty, Deleted>::sanitize () +{ + return false; +} + /* Pointer hasher based on pointer equality. Other types of pointer hash can inherit this and override the hash and equal functions with some other form of equality (such as string equality). */ @@ -153,6 +161,7 @@ struct pointer_hash static inline void mark_empty (Type *&); static inline bool is_deleted (Type *); static inline bool is_empty (Type *); + static inline bool sanitize (); }; template <typename Type> @@ -200,6 +209,13 @@ pointer_hash <Type>::is_empty (Type *e) return e == NULL; } +template <typename Type> +inline bool +pointer_hash <Type>::sanitize () +{ + return true; +} + /* Hasher for "const char *" strings, using string rather than pointer equality. */ @@ -326,6 +342,7 @@ struct pair_hash static inline void mark_empty (value_type &); static inline bool is_deleted (const value_type &); static inline bool is_empty (const value_type &); + static inline bool sanitize (); }; template <typename T1, typename T2> @@ -378,6 +395,14 @@ pair_hash <T1, T2>::is_empty (const value_type &x) return T1::is_empty (x.first); } +template <typename T1, typename T2> +inline bool +pair_hash <T1, T2>::sanitize () +{ + return T1::sanitize (); +} + + template <typename T> struct default_hash_traits : T {}; template <typename T> diff --git a/gcc/ipa-devirt.c b/gcc/ipa-devirt.c index a4c8b0de86f..26f0d332e00 100644 --- a/gcc/ipa-devirt.c +++ b/gcc/ipa-devirt.c @@ -170,6 +170,8 @@ struct default_hash_traits <type_pair> { e.first = NULL; } + + static bool sanitize () { return true; } }; static bool odr_types_equivalent_p (tree, tree, bool, bool *, diff --git a/gcc/ipa-prop.c b/gcc/ipa-prop.c index d86c2f3db55..57cd864b74d 100644 --- a/gcc/ipa-prop.c +++ b/gcc/ipa-prop.c @@ -99,6 +99,8 @@ struct ipa_bit_ggc_hash_traits : public ggc_cache_remove <ipa_bits *> { p = reinterpret_cast<ipa_bits *> (1); } + + static bool sanitize () { return true; } }; /* Hash table for avoid repeated allocations of equal ipa_bits. */ @@ -144,6 +146,7 @@ struct ipa_vr_ggc_hash_traits : public ggc_cache_remove <value_range_base *> { p = reinterpret_cast<value_range_base *> (1); } + inline static bool sanitize () { return true; } }; /* Hash table for avoid repeated allocations of equal value_ranges. */ diff --git a/gcc/profile.c b/gcc/profile.c index 9aff9ef2b21..668888f89ab 100644 --- a/gcc/profile.c +++ b/gcc/profile.c @@ -884,6 +884,7 @@ struct location_triplet_hash : typed_noop_remove <location_triplet> { return ref.lineno == -2; } + inline static bool sanitize () { return true; } }; diff --git a/gcc/sanopt.c b/gcc/sanopt.c index 5cb98e1b50e..d2f9ba5a56c 100644 --- a/gcc/sanopt.c +++ b/gcc/sanopt.c @@ -147,6 +147,7 @@ struct sanopt_tree_triplet_hash : typed_noop_remove <sanopt_tree_triplet> { return ref.t1 == NULL; } + inline static bool sanitize () { return true; } }; /* Tree couple for ptr_check_map. */ @@ -202,6 +203,7 @@ struct sanopt_tree_couple_hash : typed_noop_remove <sanopt_tree_couple> { return ref.ptr == NULL; } + inline static bool sanitize () { return true; } }; /* This is used to carry various hash maps and variables used diff --git a/gcc/tree-hasher.h b/gcc/tree-hasher.h index a64f297fb2a..24da6c0a416 100644 --- a/gcc/tree-hasher.h +++ b/gcc/tree-hasher.h @@ -42,6 +42,7 @@ struct int_tree_hasher static bool is_empty (const value_type &v) { return v.to == NULL; } static void mark_empty (value_type &v) { v.to = NULL; } static void remove (value_type &) {} + inline static bool sanitize () { return true; } }; /* Hash a UID in a int_tree_map. */ diff --git a/gcc/tree-ssa-sccvn.c b/gcc/tree-ssa-sccvn.c index b4f626000dd..69d365dc5d2 100644 --- a/gcc/tree-ssa-sccvn.c +++ b/gcc/tree-ssa-sccvn.c @@ -335,6 +335,7 @@ struct vn_ssa_aux_hasher : typed_noop_remove <vn_ssa_aux_t> static inline void mark_empty (value_type &e) { e = NULL; } static inline bool is_deleted (value_type &) { return false; } static inline bool is_empty (value_type &e) { return e == NULL; } + inline static bool sanitize () { return true; } }; hashval_t diff --git a/gcc/tree-vect-slp.c b/gcc/tree-vect-slp.c index 930cd79784e..10ad0632853 100644 --- a/gcc/tree-vect-slp.c +++ b/gcc/tree-vect-slp.c @@ -1023,6 +1023,7 @@ struct bst_traits static inline void mark_empty (value_type &x) { x.release (); } static inline void mark_deleted (value_type &x) { x.release (); } static inline void remove (value_type &x) { x.release (); } + inline static bool sanitize () { return true; } }; inline hashval_t bst_traits::hash (value_type x) -- 2.21.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-12 9:15 ` Martin Liška @ 2019-06-12 9:41 ` Richard Biener 2019-06-12 11:45 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-06-12 9:41 UTC (permalink / raw) To: Martin Liška Cc: Jason Merrill, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On Wed, Jun 12, 2019 at 11:15 AM Martin Liška <mliska@suse.cz> wrote: > > On 6/12/19 10:02 AM, Martin Liška wrote: > > On 6/12/19 9:59 AM, Richard Biener wrote: > >> On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote: > >>> > >>> On 6/11/19 9:16 AM, Martin Liška wrote: > >>>> On 6/11/19 2:27 PM, Jason Merrill wrote: > >>>>> On 6/11/19 3:41 AM, Martin Liška wrote: > >>>>>> On 6/10/19 8:21 PM, Jason Merrill wrote: > >>>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote: > >>>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote: > >>>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: > >>>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: > >>>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: > >>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: > >>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>>>>>>>>>>>>>>>>>> ^^^^^^ > >>>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Martin > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>>>>>>>>>>>> Jakub > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> --- > >>>>>>>>>>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>>>>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h > >>>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>>>>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>>>>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>>>>>>>>>>>>>>>>>> void expand (); > >>>>>>>>>>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>>>>>>>>>>>>>>>>>> expand (); > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> - m_searches++; > >>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>>>>>>>>>>> + if (insert == INSERT) > >>>>>>>>>>>>>>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>>>>>>>>>>>>>> +#endif > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> + m_searches++; > >>>>>>>>>>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>>>>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>>>>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>>>>>>>>>>> return &m_entries[index]; > >>>>>>>>>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>>>>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>>>>>>>>>>>>>>>>>> +static void > >>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value\n"); > >>>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix. > >>>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>>>>>>>>>>>>>>>>>> Hi. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I've just added one more PR: > >>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>>>>>>>>>>>>>>>>>> limits maximal number of checks: > >>>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>>>>>>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, > >>>>>>>>>>>>>>>>>>>> just thinking about loud. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>>>>>>>>>>>>>>>>>> issue :-) > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the > >>>>>>>>>>>>>>>>>>> table are never compared against each other but always against another > >>>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the > >>>>>>>>>>>>>>>>>>> comparison function only works with those. With the patch we verify > >>>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against > >>>>>>>>>>>>>>>>>>> all other elements? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes > >>>>>>>>>>>>>>>>>> PR90450 and PR87847. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Changes from previous version: > >>>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted) > >>>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >>>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order > >>>>>>>>>>>>>>>>>> to limit number of elements that are compared within a table > >>>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't > >>>>>>>>>>>>>>>>> comparing random two elements in the table. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > >>>>>>>>>>>>>>>>> without INSERTing. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations > >>>>>>>>>>>>>>>> (both w/ and w/o insertion). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > >>>>>>>>>>>>>>>> except for: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > >>>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value > >>>>>>>>>>>>>>>> during GIMPLE pass: lim > >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > >>>>>>>>>>>>>>>> 6 | fn1 () > >>>>>>>>>>>>>>>> | ^~~ > >>>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error > >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > >>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > >>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > >>>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt > >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > >>>>>>>>>>>>>>>> 0xe504ea analyze_memory_references > >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > >>>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim > >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > >>>>>>>>>>>>>>>> 0xe504ea execute > >>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Richi: it's after your recent patch. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > >>>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a > >>>>>>>>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Good point, I've just adjusted that. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Ready to be installed? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone > >>>>>>>>>>>>> fixing it ... > >>>>>>>>>>>> > >>>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR. > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> One question - there's unconditional > >>>>>>>>>>>>> > >>>>>>>>>>>>> + if (m_sanitize_eq_and_hash) > >>>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>>> > >>>>>>>>>>>>> which will read a global variable and have (possibly not inline) call > >>>>>>>>>>>>> to verify on a common path even with checking disabled. So I think > >>>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P > >>>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not > >>>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). > >>>>>>>>>>>> > >>>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. > >>>>>>>>>>> > >>>>>>>>>>> You missed the second occurance > >>>>>>>>>>> > >>>>>>>>>>> - m_searches++; > >>>>>>>>>>> + if (m_sanitize_eq_and_hash) > >>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>> > >>>>>>>>>> Yep ;) I've just install the patch. > >>>>>>>>> > >>>>>>>>> This is breaking my build: > >>>>>>>>> > >>>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for > >>>>>>>>> call to ‘hash_table<hash_map<mem_alloc_d\ > >>>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*, > >>>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ > >>>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, > >>>>>>>>> false, xcallocator>::hash_table(size_t&, bo\ > >>>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’ > >>>>>>>>> : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} > >>>>>>>>> > >>>>>>>>> Looks like this needs to be updated to pass an argument to the new > >>>>>>>>> sanitize_eq_and_hash parameter. > >>>>>>>>> > >>>>>>>>> Jason > >>>>>>>> > >>>>>>>> Sorry for the breakage, I've just fixed that in r272104. > >>>>>>> > >>>>>>> Thanks. I'm also seeing a massive compile time hit from this: A > >>>>>>> constexpr testcase that I've been looking at went from compiling in 13 > >>>>>>> seconds to 78 seconds, 6 times as long. I would expect template-heavy > >>>>>>> code to see similar problems when sanitization is enabled for those > >>>>>>> hash tables. Could we keep the parameter low or 0 by default, and > >>>>>>> just do occasional sanitize runs with it explicitly enabled? > >>>>>> > >>>>>> Makes sense to me. Can you please provide a test-case which I can measure? > >>>>> > >>>>> This is the one I've been looking at: > >>>>> > >>>>> struct Int { > >>>>> constexpr Int(int v): v(v) {} > >>>>> constexpr Int& operator+=(Int b) { this->v += b.v; return *this; } > >>>>> constexpr Int& operator++() { ++this->v; return *this; } > >>>>> private: > >>>>> friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; } > >>>>> int v; > >>>>> }; > >>>>> constexpr int f(int n) { > >>>>> Int i = {0}; > >>>>> Int k = {0}; > >>>>> k = 0; > >>>>> for (; k<10000; ++k) { > >>>>> i += k; > >>>>> } > >>>>> return n; > >>>>> } > >>>>> > >>>>> template<int N> struct S { > >>>>> static constexpr int sm = S<N-1>::sm+f(N); > >>>>> }; > >>>>> template<> struct S<0> { > >>>>> static constexpr int sm = 0; > >>>>> }; > >>>>> constexpr int r = S<20>::sm; > >>>>> > >>>>> Jason > >>>> > >>>> For the test-case provided I see: > >>>> > >>>> $ time g++ time.cc -c --param hash-table-verification-limit=100 > >>>> > >>>> real 0m1.855s > >>>> user 0m1.829s > >>>> sys 0m0.025s > >>>> > >>>> $ time g++ time.cc -c --param hash-table-verification-limit=0 > >>>> > >>>> real 0m1.275s > >>>> user 0m1.219s > >>>> sys 0m0.052s > >>>> > >>>> $ time g++-9 time.cc -c > >>>> > >>>> real 0m0.939s > >>>> user 0m0.827s > >>>> sys 0m0.109s > >>>> > >>>> So it's slower, but I can't confirm the huge slowdown you see. > >>>> Is it due to r272144? > >>> > >>> Hmm, I wonder if this is because of the > >>> --enable-gather-detailed-mem-stats hash tables. > >> > >> I wonder if we can reduce the overhead by making > >> hash-tables/maps using predefined traits not perform > >> the checking? Thus make [the default] whether to check or not > >> to check part of the traits? Surely the basic pointer-hash/map > >> stuff is OK and needs no extra checking. > > > > Interesting idea! I can prepare a patch. Right now I'm testing a patch > > that removes sanitization for hash-tables (and maps) that track memory > > allocations. > > I've got 2 patch candidates that will make it happen. It survives build > on --enable-languages=all, but one would need to build all cross-compilers > to make a proper testing. > > Do you like the idea of the patch before I'll write a changelog and test it? The disabling for mem-stats patch looks good to me. I don't like the implementation of the traits one, we don't want to have to put the default returning true everywhere. Instead I expected some enable_if <> magic to do select a default true if the member isn't present. The issue with the traits idea is also that people like to derive from one of the standard traits and thus might inherit 'false' even though they override hash/compare methods. That is, I expected +template <typename Type> +inline bool +pointer_hash <Type>::sanitize () +{ + return true; +} to return false for example. Basically for the case we auto-detect the traits based on the key type I wanted to disable sanitizing and for custom users leave them to per-object decisions. Not sure if we have enough C++ power to make that happen easily. So let's go the route of disabling the "ovbiously" correct and performance critical parts for now. Richard. > Thanks, > Martin > > > > > Martin > > > >> > >> Richard. > >> > >>> Jason > > > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-12 9:41 ` Richard Biener @ 2019-06-12 11:45 ` Martin Liška 2019-06-12 12:50 ` Richard Biener 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2019-06-12 11:45 UTC (permalink / raw) To: Richard Biener Cc: Jason Merrill, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 18561 bytes --] On 6/12/19 11:41 AM, Richard Biener wrote: > On Wed, Jun 12, 2019 at 11:15 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> On 6/12/19 10:02 AM, Martin LiÅ¡ka wrote: >>> On 6/12/19 9:59 AM, Richard Biener wrote: >>>> On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote: >>>>> >>>>> On 6/11/19 9:16 AM, Martin LiÅ¡ka wrote: >>>>>> On 6/11/19 2:27 PM, Jason Merrill wrote: >>>>>>> On 6/11/19 3:41 AM, Martin LiÅ¡ka wrote: >>>>>>>> On 6/10/19 8:21 PM, Jason Merrill wrote: >>>>>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote: >>>>>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote: >>>>>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>>>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>>>>>>>>> Jakub >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>>>>>>>>>>>>>>>> void expand (); >>>>>>>>>>>>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>>>>>>>>>>>>> expand (); >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>>>>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>>>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>>>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>>>>>>>>>>>>> just thinking about loud. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>>>>>>>>>>>>> issue :-) >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>>>>>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>>>>>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>>>>>>>>>>>>> all other elements? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>>>>>>>>>>>>> PR90450 and PR87847. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Changes from previous version: >>>>>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>>>>>>>>>>>>> to limit number of elements that are compared within a table >>>>>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>>>>>>>>>>>>> comparing random two elements in the table. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>>>>>>>>>>>>> without INSERTing. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations >>>>>>>>>>>>>>>>>> (both w/ and w/o insertion). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>>>>>>>>>>>>> except for: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>>>>>>>>>>>>> during GIMPLE pass: lim >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>>>>>>>>>>>>> 6 | fn1 () >>>>>>>>>>>>>>>>>> | ^~~ >>>>>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>>>>>>>>>>>>> 0xe504ea analyze_memory_references >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>>>>>>>>>>>>> 0xe504ea execute >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Richi: it's after your recent patch. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>>>>>>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Good point, I've just adjusted that. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ready to be installed? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>>>>>>>>>>>>> fixing it ... >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> One question - there's unconditional >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> which will read a global variable and have (possibly not inline) call >>>>>>>>>>>>>>> to verify on a common path even with checking disabled. So I think >>>>>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P >>>>>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>>>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. >>>>>>>>>>>>> >>>>>>>>>>>>> You missed the second occurance >>>>>>>>>>>>> >>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>> + if (m_sanitize_eq_and_hash) >>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>> >>>>>>>>>>>> Yep ;) I've just install the patch. >>>>>>>>>>> >>>>>>>>>>> This is breaking my build: >>>>>>>>>>> >>>>>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for >>>>>>>>>>> call to âhash_table<hash_map<mem_alloc_d\ >>>>>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*, >>>>>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ >>>>>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, >>>>>>>>>>> false, xcallocator>::hash_table(size_t&, bo\ >>>>>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)â >>>>>>>>>>> : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} >>>>>>>>>>> >>>>>>>>>>> Looks like this needs to be updated to pass an argument to the new >>>>>>>>>>> sanitize_eq_and_hash parameter. >>>>>>>>>>> >>>>>>>>>>> Jason >>>>>>>>>> >>>>>>>>>> Sorry for the breakage, I've just fixed that in r272104. >>>>>>>>> >>>>>>>>> Thanks. I'm also seeing a massive compile time hit from this: A >>>>>>>>> constexpr testcase that I've been looking at went from compiling in 13 >>>>>>>>> seconds to 78 seconds, 6 times as long. I would expect template-heavy >>>>>>>>> code to see similar problems when sanitization is enabled for those >>>>>>>>> hash tables. Could we keep the parameter low or 0 by default, and >>>>>>>>> just do occasional sanitize runs with it explicitly enabled? >>>>>>>> >>>>>>>> Makes sense to me. Can you please provide a test-case which I can measure? >>>>>>> >>>>>>> This is the one I've been looking at: >>>>>>> >>>>>>> struct Int { >>>>>>> constexpr Int(int v): v(v) {} >>>>>>> constexpr Int& operator+=(Int b) { this->v += b.v; return *this; } >>>>>>> constexpr Int& operator++() { ++this->v; return *this; } >>>>>>> private: >>>>>>> friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; } >>>>>>> int v; >>>>>>> }; >>>>>>> constexpr int f(int n) { >>>>>>> Int i = {0}; >>>>>>> Int k = {0}; >>>>>>> k = 0; >>>>>>> for (; k<10000; ++k) { >>>>>>> i += k; >>>>>>> } >>>>>>> return n; >>>>>>> } >>>>>>> >>>>>>> template<int N> struct S { >>>>>>> static constexpr int sm = S<N-1>::sm+f(N); >>>>>>> }; >>>>>>> template<> struct S<0> { >>>>>>> static constexpr int sm = 0; >>>>>>> }; >>>>>>> constexpr int r = S<20>::sm; >>>>>>> >>>>>>> Jason >>>>>> >>>>>> For the test-case provided I see: >>>>>> >>>>>> $ time g++ time.cc -c --param hash-table-verification-limit=100 >>>>>> >>>>>> real 0m1.855s >>>>>> user 0m1.829s >>>>>> sys 0m0.025s >>>>>> >>>>>> $ time g++ time.cc -c --param hash-table-verification-limit=0 >>>>>> >>>>>> real 0m1.275s >>>>>> user 0m1.219s >>>>>> sys 0m0.052s >>>>>> >>>>>> $ time g++-9 time.cc -c >>>>>> >>>>>> real 0m0.939s >>>>>> user 0m0.827s >>>>>> sys 0m0.109s >>>>>> >>>>>> So it's slower, but I can't confirm the huge slowdown you see. >>>>>> Is it due to r272144? >>>>> >>>>> Hmm, I wonder if this is because of the >>>>> --enable-gather-detailed-mem-stats hash tables. >>>> >>>> I wonder if we can reduce the overhead by making >>>> hash-tables/maps using predefined traits not perform >>>> the checking? Thus make [the default] whether to check or not >>>> to check part of the traits? Surely the basic pointer-hash/map >>>> stuff is OK and needs no extra checking. >>> >>> Interesting idea! I can prepare a patch. Right now I'm testing a patch >>> that removes sanitization for hash-tables (and maps) that track memory >>> allocations. >> >> I've got 2 patch candidates that will make it happen. It survives build >> on --enable-languages=all, but one would need to build all cross-compilers >> to make a proper testing. >> >> Do you like the idea of the patch before I'll write a changelog and test it? > > The disabling for mem-stats patch looks good to me. I don't like the > implementation of the traits one, we don't want to have to put the > default returning true everywhere. Instead I expected some > enable_if <> magic to do select a default true if the member isn't > present. The issue with the traits idea is also that people like > to derive from one of the standard traits and thus might inherit > 'false' even though they override hash/compare methods. That is, > I expected > > +template <typename Type> > +inline bool > +pointer_hash <Type>::sanitize () > +{ > + return true; > +} > > to return false for example. Basically for the case we > auto-detect the traits based on the key type I wanted to > disable sanitizing and for custom users leave them to > per-object decisions. > > Not sure if we have enough C++ power to make that happen easily. Huh, I've tried quite hard and I was unable to make it happen. Maybe somebody more familiar with C++ can step in? > So let's go the route of disabling the "ovbiously" correct and performance > critical parts for now. I'm sending patch for it. Martin > > Richard. > >> Thanks, >> Martin >> >>> >>> Martin >>> >>>> >>>> Richard. >>>> >>>>> Jason >>> >> [-- Attachment #2: 0001-Disable-hash-table-sanitization-for-mem-stats-maps.patch --] [-- Type: text/x-patch, Size: 2978 bytes --] From 71668e0eaf983ff05dfd52a95827cefdf24350ec Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Wed, 12 Jun 2019 11:04:11 +0200 Subject: [PATCH] Disable hash-table sanitization for mem stats maps. gcc/ChangeLog: 2019-06-12 Martin Liska <mliska@suse.cz> * ggc-common.c (ggc_prune_overhead_list): Do not sanitize the created map. * hash-map.h: Add sanitize_eq_and_hash into ::hash_map. * mem-stats.h (mem_alloc_description::mem_alloc_description): Do not sanitize created maps. --- gcc/ggc-common.c | 2 +- gcc/hash-map.h | 9 ++++++--- gcc/mem-stats.h | 6 +++--- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/gcc/ggc-common.c b/gcc/ggc-common.c index 2acdb6dc60c..6fb5a3d5ceb 100644 --- a/gcc/ggc-common.c +++ b/gcc/ggc-common.c @@ -1014,5 +1014,5 @@ ggc_prune_overhead_list (void) (*it).second.first->m_collected += (*it).second.second; delete ggc_mem_desc.m_reverse_object_map; - ggc_mem_desc.m_reverse_object_map = new map_t (13, false, false); + ggc_mem_desc.m_reverse_object_map = new map_t (13, false, false, false); } diff --git a/gcc/hash-map.h b/gcc/hash-map.h index a8eb42d5a03..588dfda04fa 100644 --- a/gcc/hash-map.h +++ b/gcc/hash-map.h @@ -118,16 +118,19 @@ class GTY((user)) hash_map public: explicit hash_map (size_t n = 13, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS CXX_MEM_STAT_INFO) - : m_table (n, ggc, true, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) + : m_table (n, ggc, sanitize_eq_and_hash, gather_mem_stats, + HASH_MAP_ORIGIN PASS_MEM_STAT) { } explicit hash_map (const hash_map &h, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS CXX_MEM_STAT_INFO) - : m_table (h.m_table, ggc, true, gather_mem_stats, + : m_table (h.m_table, ggc, sanitize_eq_and_hash, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} /* Create a hash_map in ggc memory. */ @@ -136,7 +139,7 @@ public: CXX_MEM_STAT_INFO) { hash_map *map = ggc_alloc<hash_map> (); - new (map) hash_map (size, true, gather_mem_stats PASS_MEM_STAT); + new (map) hash_map (size, true, true, gather_mem_stats PASS_MEM_STAT); return map; } diff --git a/gcc/mem-stats.h b/gcc/mem-stats.h index 63ce8712e2b..77960595753 100644 --- a/gcc/mem-stats.h +++ b/gcc/mem-stats.h @@ -559,9 +559,9 @@ template <class T> inline mem_alloc_description<T>::mem_alloc_description () { - m_map = new mem_map_t (13, false, false); - m_reverse_map = new reverse_mem_map_t (13, false, false); - m_reverse_object_map = new reverse_object_map_t (13, false, false); + m_map = new mem_map_t (13, false, false, false); + m_reverse_map = new reverse_mem_map_t (13, false, false, false); + m_reverse_object_map = new reverse_object_map_t (13, false, false, false); } /* Default destructor. */ -- 2.21.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-12 11:45 ` Martin Liška @ 2019-06-12 12:50 ` Richard Biener 2019-06-12 13:05 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-06-12 12:50 UTC (permalink / raw) To: Martin Liška Cc: Jason Merrill, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On Wed, Jun 12, 2019 at 1:45 PM Martin Liška <mliska@suse.cz> wrote: > > On 6/12/19 11:41 AM, Richard Biener wrote: > > On Wed, Jun 12, 2019 at 11:15 AM Martin Liška <mliska@suse.cz> wrote: > >> > >> On 6/12/19 10:02 AM, Martin Liška wrote: > >>> On 6/12/19 9:59 AM, Richard Biener wrote: > >>>> On Tue, Jun 11, 2019 at 9:02 PM Jason Merrill <jason@redhat.com> wrote: > >>>>> > >>>>> On 6/11/19 9:16 AM, Martin Liška wrote: > >>>>>> On 6/11/19 2:27 PM, Jason Merrill wrote: > >>>>>>> On 6/11/19 3:41 AM, Martin Liška wrote: > >>>>>>>> On 6/10/19 8:21 PM, Jason Merrill wrote: > >>>>>>>>> On Mon, Jun 10, 2019 at 3:08 AM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>> On 6/7/19 11:43 PM, Jason Merrill wrote: > >>>>>>>>>>> On Fri, Jun 7, 2019 at 8:14 AM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>>>> On 6/7/19 2:09 PM, Richard Biener wrote: > >>>>>>>>>>>>> On Fri, Jun 7, 2019 at 2:03 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>>>>>> On 6/7/19 10:57 AM, Richard Biener wrote: > >>>>>>>>>>>>>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>>>>>>>> On 6/1/19 12:06 AM, Jeff Law wrote: > >>>>>>>>>>>>>>>>> On 5/22/19 3:13 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: > >>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: > >>>>>>>>>>>>>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>>>>>>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>>>>>>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>>>>>>>>>>>>>>>>>>>> ^^^^^^ > >>>>>>>>>>>>>>>>>>>>>>>>>> Sure, fixed in attached patch. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Martin > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jakub > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>>>>>>>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>>>>>>>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>>>>>>>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> --- > >>>>>>>>>>>>>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>>>>>>>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>>>>>>>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>>>>>>>>>>>>>>>>>>>> --- a/gcc/hash-table.h > >>>>>>>>>>>>>>>>>>>>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>>>>>>>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>>>>>>>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>>>>>>>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>>>>>>>>>>>>>>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>>>>>>>>>>>>>>>>>>>> void expand (); > >>>>>>>>>>>>>>>>>>>>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>>>>>>>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>>>>>>>>>>>>>>>>>>>> expand (); > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> - m_searches++; > >>>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>>>>>>>>>>>>> + if (insert == INSERT) > >>>>>>>>>>>>>>>>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>>>>>>>>>>>>>>>> +#endif > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> + m_searches++; > >>>>>>>>>>>>>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>>>>>>>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>>>>>>>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>>>>>>>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>>>>>>>>>>>>> return &m_entries[index]; > >>>>>>>>>>>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>>>>>>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>>>>>>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>>>>>>>>>>>>>>>>>>>> +static void > >>>>>>>>>>>>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>>>>>>>>>>>> + "of values with a different hash value\n"); > >>>>>>>>>>>>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>>>>>>>>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>>>>>>>>>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>>>>>>>>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>>>>>>>>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>>>>>>>>>>>>>>>>>>>> because we've got too many bugs to fix. > >>>>>>>>>>>>>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>>>>>>>>>>>>>>>>>>>> Hi. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I've just added one more PR: > >>>>>>>>>>>>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>>>>>>>>>>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>>>>>>>>>>>>>>>>>>>> limits maximal number of checks: > >>>>>>>>>>>>>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>>>>>>>>>>>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, > >>>>>>>>>>>>>>>>>>>>>> just thinking about loud. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>>>>>>>>>>>>>>>>>>>> issue :-) > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> There is at least one PR where we have a table where elements _in_ the > >>>>>>>>>>>>>>>>>>>>> table are never compared against each other but always against another > >>>>>>>>>>>>>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the > >>>>>>>>>>>>>>>>>>>>> comparison function only works with those. With the patch we verify > >>>>>>>>>>>>>>>>>>>>> hashing/comparison for something that is never used. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>>>>>>>>>>>>>>>>>>>> at lookup time, using the object from the lookup and verify that against > >>>>>>>>>>>>>>>>>>>>> all other elements? > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I don't a have problem with that. Apparently this changes fixes > >>>>>>>>>>>>>>>>>>>> PR90450 and PR87847. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Changes from previous version: > >>>>>>>>>>>>>>>>>>>> - verification happens only when an element is searched (not inserted) > >>>>>>>>>>>>>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >>>>>>>>>>>>>>>>>>>> - new param has been introduced hash-table-verification-limit in order > >>>>>>>>>>>>>>>>>>>> to limit number of elements that are compared within a table > >>>>>>>>>>>>>>>>>>>> - verification happens only with flag_checking >= 2 > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I've been bootstrapping and testing the patch right now. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Looks like I misremembered the original patch. The issue isn't > >>>>>>>>>>>>>>>>>>> comparing random two elements in the table. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > >>>>>>>>>>>>>>>>>>> without INSERTing. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> There's updated version of the patch where I check all find operations > >>>>>>>>>>>>>>>>>> (both w/ and w/o insertion). > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > >>>>>>>>>>>>>>>>>> except for: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > >>>>>>>>>>>>>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value > >>>>>>>>>>>>>>>>>> during GIMPLE pass: lim > >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > >>>>>>>>>>>>>>>>>> 6 | fn1 () > >>>>>>>>>>>>>>>>>> | ^~~ > >>>>>>>>>>>>>>>>>> 0x6c5725 hashtab_chk_error > >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > >>>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > >>>>>>>>>>>>>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > >>>>>>>>>>>>>>>>>> 0xe504ea gather_mem_refs_stmt > >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > >>>>>>>>>>>>>>>>>> 0xe504ea analyze_memory_references > >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > >>>>>>>>>>>>>>>>>> 0xe504ea tree_ssa_lim > >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > >>>>>>>>>>>>>>>>>> 0xe504ea execute > >>>>>>>>>>>>>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Richi: it's after your recent patch. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> For some reason I don't see PR87847 issue any longer. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > >>>>>>>>>>>>>>>>> Don't we still need to deal with the naked fprintf when there's a > >>>>>>>>>>>>>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Good point, I've just adjusted that. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Ready to be installed? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone > >>>>>>>>>>>>>>> fixing it ... > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Yes :D It's been some time and there's no interest in the PR. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> One question - there's unconditional > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> + if (m_sanitize_eq_and_hash) > >>>>>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> which will read a global variable and have (possibly not inline) call > >>>>>>>>>>>>>>> to verify on a common path even with checking disabled. So I think > >>>>>>>>>>>>>>> we want to compile this checking feature out for !CHECKING_P > >>>>>>>>>>>>>>> or at least make the if __builtin_expect (..., 0), ::verify not > >>>>>>>>>>>>>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Fixed. May I install the patch? The cselib issue can be solved later.. > >>>>>>>>>>>>> > >>>>>>>>>>>>> You missed the second occurance > >>>>>>>>>>>>> > >>>>>>>>>>>>> - m_searches++; > >>>>>>>>>>>>> + if (m_sanitize_eq_and_hash) > >>>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>> > >>>>>>>>>>>> Yep ;) I've just install the patch. > >>>>>>>>>>> > >>>>>>>>>>> This is breaking my build: > >>>>>>>>>>> > >>>>>>>>>>> /home/jason/gt/gcc/hash-map.h:123:71: error: no matching function for > >>>>>>>>>>> call to ‘hash_table<hash_map<mem_alloc_d\ > >>>>>>>>>>> escription<mem_usage>::mem_location_hash, mem_usage*, > >>>>>>>>>>> simple_hashmap_traits<default_hash_traits<mem_alloc_desc\ > >>>>>>>>>>> ription<mem_usage>::mem_location_hash>, mem_usage*> >::hash_entry, > >>>>>>>>>>> false, xcallocator>::hash_table(size_t&, bo\ > >>>>>>>>>>> ol&, bool&, mem_alloc_origin, const char*&, int&, const char*&)’ > >>>>>>>>>>> : m_table (n, ggc, gather_mem_stats, HASH_MAP_ORIGIN PASS_MEM_STAT) {} > >>>>>>>>>>> > >>>>>>>>>>> Looks like this needs to be updated to pass an argument to the new > >>>>>>>>>>> sanitize_eq_and_hash parameter. > >>>>>>>>>>> > >>>>>>>>>>> Jason > >>>>>>>>>> > >>>>>>>>>> Sorry for the breakage, I've just fixed that in r272104. > >>>>>>>>> > >>>>>>>>> Thanks. I'm also seeing a massive compile time hit from this: A > >>>>>>>>> constexpr testcase that I've been looking at went from compiling in 13 > >>>>>>>>> seconds to 78 seconds, 6 times as long. I would expect template-heavy > >>>>>>>>> code to see similar problems when sanitization is enabled for those > >>>>>>>>> hash tables. Could we keep the parameter low or 0 by default, and > >>>>>>>>> just do occasional sanitize runs with it explicitly enabled? > >>>>>>>> > >>>>>>>> Makes sense to me. Can you please provide a test-case which I can measure? > >>>>>>> > >>>>>>> This is the one I've been looking at: > >>>>>>> > >>>>>>> struct Int { > >>>>>>> constexpr Int(int v): v(v) {} > >>>>>>> constexpr Int& operator+=(Int b) { this->v += b.v; return *this; } > >>>>>>> constexpr Int& operator++() { ++this->v; return *this; } > >>>>>>> private: > >>>>>>> friend constexpr bool operator<(Int a, Int b) { return a.v < b.v; } > >>>>>>> int v; > >>>>>>> }; > >>>>>>> constexpr int f(int n) { > >>>>>>> Int i = {0}; > >>>>>>> Int k = {0}; > >>>>>>> k = 0; > >>>>>>> for (; k<10000; ++k) { > >>>>>>> i += k; > >>>>>>> } > >>>>>>> return n; > >>>>>>> } > >>>>>>> > >>>>>>> template<int N> struct S { > >>>>>>> static constexpr int sm = S<N-1>::sm+f(N); > >>>>>>> }; > >>>>>>> template<> struct S<0> { > >>>>>>> static constexpr int sm = 0; > >>>>>>> }; > >>>>>>> constexpr int r = S<20>::sm; > >>>>>>> > >>>>>>> Jason > >>>>>> > >>>>>> For the test-case provided I see: > >>>>>> > >>>>>> $ time g++ time.cc -c --param hash-table-verification-limit=100 > >>>>>> > >>>>>> real 0m1.855s > >>>>>> user 0m1.829s > >>>>>> sys 0m0.025s > >>>>>> > >>>>>> $ time g++ time.cc -c --param hash-table-verification-limit=0 > >>>>>> > >>>>>> real 0m1.275s > >>>>>> user 0m1.219s > >>>>>> sys 0m0.052s > >>>>>> > >>>>>> $ time g++-9 time.cc -c > >>>>>> > >>>>>> real 0m0.939s > >>>>>> user 0m0.827s > >>>>>> sys 0m0.109s > >>>>>> > >>>>>> So it's slower, but I can't confirm the huge slowdown you see. > >>>>>> Is it due to r272144? > >>>>> > >>>>> Hmm, I wonder if this is because of the > >>>>> --enable-gather-detailed-mem-stats hash tables. > >>>> > >>>> I wonder if we can reduce the overhead by making > >>>> hash-tables/maps using predefined traits not perform > >>>> the checking? Thus make [the default] whether to check or not > >>>> to check part of the traits? Surely the basic pointer-hash/map > >>>> stuff is OK and needs no extra checking. > >>> > >>> Interesting idea! I can prepare a patch. Right now I'm testing a patch > >>> that removes sanitization for hash-tables (and maps) that track memory > >>> allocations. > >> > >> I've got 2 patch candidates that will make it happen. It survives build > >> on --enable-languages=all, but one would need to build all cross-compilers > >> to make a proper testing. > >> > >> Do you like the idea of the patch before I'll write a changelog and test it? > > > > The disabling for mem-stats patch looks good to me. I don't like the > > implementation of the traits one, we don't want to have to put the > > default returning true everywhere. Instead I expected some > > enable_if <> magic to do select a default true if the member isn't > > present. The issue with the traits idea is also that people like > > to derive from one of the standard traits and thus might inherit > > 'false' even though they override hash/compare methods. That is, > > I expected > > > > +template <typename Type> > > +inline bool > > +pointer_hash <Type>::sanitize () > > +{ > > + return true; > > +} > > > > to return false for example. Basically for the case we > > auto-detect the traits based on the key type I wanted to > > disable sanitizing and for custom users leave them to > > per-object decisions. > > > > Not sure if we have enough C++ power to make that happen easily. > > Huh, I've tried quite hard and I was unable to make it happen. Maybe > somebody more familiar with C++ can step in? > > > So let's go the route of disabling the "ovbiously" correct and performance > > critical parts for now. > > I'm sending patch for it. explicit hash_map (size_t n = 13, bool ggc = false, + bool sanitize_eq_and_hash = true, bool gather_mem_stats = GATHER_STATISTICS CXX_MEM_STAT_INFO) maybe not an issue here but adding defaulted params into the middle is prone to change existing calls semantics. Consider hash_map (13, false, false); which was gather_mem_stats == false before but now is sanitize_eq_and_hash. I realize the original patch had the same issue. Patch is still OK. Please grep for explicit constructor calls and double-check. Ick. I guess also hash_map<foo, bar> baz (13, false, false); is affected so make sure to apply enough grep-fu. New params should always go to the end ;) Richard. > > Martin > > > > > Richard. > > > >> Thanks, > >> Martin > >> > >>> > >>> Martin > >>> > >>>> > >>>> Richard. > >>>> > >>>>> Jason > >>> > >> > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-12 12:50 ` Richard Biener @ 2019-06-12 13:05 ` Martin Liška 0 siblings, 0 replies; 53+ messages in thread From: Martin Liška @ 2019-06-12 13:05 UTC (permalink / raw) To: Richard Biener Cc: Jason Merrill, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Paul Richard Thomas, Martin Jambor On 6/12/19 2:50 PM, Richard Biener wrote: > New params should always go to the end Ah, sorry, I'll take of that new time. I've just changed the function signature and I can verify there are no other calls (tested for --enable-languages=all). Thanks, Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-07 12:04 ` Martin Liška 2019-06-07 12:09 ` Richard Biener @ 2019-06-23 23:08 ` Ian Lance Taylor 2019-06-24 12:29 ` Richard Biener 1 sibling, 1 reply; 53+ messages in thread From: Ian Lance Taylor @ 2019-06-23 23:08 UTC (permalink / raw) To: Martin Liška Cc: Richard Biener, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 10925 bytes --] On Fri, Jun 7, 2019 at 5:04 AM Martin Liška <mliska@suse.cz> wrote: > > On 6/7/19 10:57 AM, Richard Biener wrote: > > On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > >> > >> On 6/1/19 12:06 AM, Jeff Law wrote: > >>> On 5/22/19 3:13 AM, Martin Liška wrote: > >>>> On 5/21/19 1:51 PM, Richard Biener wrote: > >>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>> > >>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: > >>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>>>>>> > >>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>>>>>> ^^^^^^ > >>>>>>>>>>>> Sure, fixed in attached patch. > >>>>>>>>>>>> > >>>>>>>>>>>> Martin > >>>>>>>>>>>> > >>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>> +} > >>>>>>>>>>>>> Jakub > >>>>>>>>>>>>> > >>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>>>>>> > >>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>>>>>> > >>>>>>>>>>>> --- > >>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>>>>>> > >>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>>>>>> --- a/gcc/hash-table.h > >>>>>>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>>>>>> > >>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>>>>>> void expand (); > >>>>>>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>>>>>> expand (); > >>>>>>>>>>>> > >>>>>>>>>>>> - m_searches++; > >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>> + if (insert == INSERT) > >>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>> +#endif > >>>>>>>>>>>> > >>>>>>>>>>>> + m_searches++; > >>>>>>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>> return &m_entries[index]; > >>>>>>>>>>>> } > >>>>>>>>>>>> > >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>> + > >>>>>>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>>>>>> + > >>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>>>>>> +static void > >>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>> +{ > >>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>> + "of values with a different hash value\n"); > >>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>> +} > >>>>>>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>>>>>> > >>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>>>>>> because we've got too many bugs to fix. > >>>>>>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>>>>>> Hi. > >>>>>>>>> > >>>>>>>>> I've just added one more PR: > >>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>>>>>> > >>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>>>>>> with a new function disable_sanitize_eq_and_hash. > >>>>>>>>> > >>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>>>>>> limits maximal number of checks: > >>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>>>>>> state set up when you instantiate the object? It's not a huge deal, > >>>>>>>> just thinking about loud. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>>>>>> issue :-) > >>>>>>> > >>>>>>> There is at least one PR where we have a table where elements _in_ the > >>>>>>> table are never compared against each other but always against another > >>>>>>> object (I guess that's usual even), but the setup is in a way that the > >>>>>>> comparison function only works with those. With the patch we verify > >>>>>>> hashing/comparison for something that is never used. > >>>>>>> > >>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>>>>>> at lookup time, using the object from the lookup and verify that against > >>>>>>> all other elements? > >>>>>> > >>>>>> I don't a have problem with that. Apparently this changes fixes > >>>>>> PR90450 and PR87847. > >>>>>> > >>>>>> Changes from previous version: > >>>>>> - verification happens only when an element is searched (not inserted) > >>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >>>>>> - new param has been introduced hash-table-verification-limit in order > >>>>>> to limit number of elements that are compared within a table > >>>>>> - verification happens only with flag_checking >= 2 > >>>>>> > >>>>>> I've been bootstrapping and testing the patch right now. > >>>>> > >>>>> Looks like I misremembered the original patch. The issue isn't > >>>>> comparing random two elements in the table. > >>>>> > >>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > >>>>> without INSERTing. > >>>>> > >>>> > >>>> There's updated version of the patch where I check all find operations > >>>> (both w/ and w/o insertion). > >>>> > >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > >>>> except for: > >>>> > >>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > >>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value > >>>> during GIMPLE pass: lim > >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > >>>> 6 | fn1 () > >>>> | ^~~ > >>>> 0x6c5725 hashtab_chk_error > >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > >>>> 0xe504ea gather_mem_refs_stmt > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > >>>> 0xe504ea analyze_memory_references > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > >>>> 0xe504ea tree_ssa_lim > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > >>>> 0xe504ea execute > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > >>>> > >>>> Richi: it's after your recent patch. > >>>> > >>>> For some reason I don't see PR87847 issue any longer. > >>>> > >>>> > >>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > >>> Don't we still need to deal with the naked fprintf when there's a > >>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > >> > >> Good point, I've just adjusted that. > >> > >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >> > >> Ready to be installed? > > > > Ugh, the cselib one is really bad. But I don't hold my breath for anyone > > fixing it ... > > Yes :D It's been some time and there's no interest in the PR. > > > > > One question - there's unconditional > > > > + if (m_sanitize_eq_and_hash) > > + verify (comparable, hash); > > > > which will read a global variable and have (possibly not inline) call > > to verify on a common path even with checking disabled. So I think > > we want to compile this checking feature out for !CHECKING_P > > or at least make the if __builtin_expect (..., 0), ::verify not > > inlined and marked pure () (thus, !CHECKING_P is simplest ;)). > > Fixed. May I install the patch? The cselib issue can be solved later.. After this patch, when I do a configure with --disable-bootstrap, and build with "gcc (Debian 7.3.0-18) 7.3.0", I get a lot of warnings of the form In file included from ../../gccgo3/gcc/coretypes.h:440:0, from ../../gccgo3/gcc/go/go-system.h:137, from ../../gccgo3/gcc/go/gofrontend/go.cc:7: ../../gccgo3/gcc/hash-table.h:1017:1: warning: ‘void hashtab_chk_error()’ defined but not used [-Wunused-function] hashtab_chk_error () ^~~~~~~~~~~~~~~~~ These are just warnings, since I am using --disable-bootstrap, but they are distracting. This patch fixes it. OK for trunk? Ian 2019-06-23 Ian Lance Taylor <iant@golang.org> * hash-table.h (hashtab_chk_error): Add ATTRIBUTE_UNUSED. [-- Attachment #2: patch.txt --] [-- Type: text/plain, Size: 390 bytes --] Index: hash-table.h =================================================================== --- hash-table.h (revision 272609) +++ hash-table.h (working copy) @@ -1012,7 +1012,7 @@ hash_table<Descriptor, Lazy, Allocator> /* Report a hash table checking error. */ -ATTRIBUTE_NORETURN ATTRIBUTE_COLD +ATTRIBUTE_NORETURN ATTRIBUTE_COLD ATTRIBUTE_UNUSED static void hashtab_chk_error () { ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-23 23:08 ` Ian Lance Taylor @ 2019-06-24 12:29 ` Richard Biener 2019-06-24 13:51 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-06-24 12:29 UTC (permalink / raw) To: Ian Lance Taylor Cc: Martin Liška, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Mon, Jun 24, 2019 at 1:08 AM Ian Lance Taylor <iant@golang.org> wrote: > > On Fri, Jun 7, 2019 at 5:04 AM Martin Liška <mliska@suse.cz> wrote: > > > > On 6/7/19 10:57 AM, Richard Biener wrote: > > > On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > > >> > > >> On 6/1/19 12:06 AM, Jeff Law wrote: > > >>> On 5/22/19 3:13 AM, Martin Liška wrote: > > >>>> On 5/21/19 1:51 PM, Richard Biener wrote: > > >>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > > >>>>>> > > >>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: > > >>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > > >>>>>>>> > > >>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > > >>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > > >>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > > >>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > > >>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > > >>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > > >>>>>>>>>>>>>> +hashtab_chk_error () > > >>>>>>>>>>>>>> +{ > > >>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > > >>>>>>>>>>>>>> + "equal operator returns true for a pair " > > >>>>>>>>>>>>>> + "of values with a different hash value"); > > >>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > > >>>>>>>>>>>>> terminate with \n, in your recent mail I saw: > > >>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > > >>>>>>>>>>>>> ^^^^^^ > > >>>>>>>>>>>> Sure, fixed in attached patch. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Martin > > >>>>>>>>>>>> > > >>>>>>>>>>>>>> + gcc_unreachable (); > > >>>>>>>>>>>>>> +} > > >>>>>>>>>>>>> Jakub > > >>>>>>>>>>>>> > > >>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > > >>>>>>>>>>>> > > >>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > > >>>>>>>>>>>> From: marxin <mliska@suse.cz> > > >>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > > >>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > > >>>>>>>>>>>> > > >>>>>>>>>>>> --- > > >>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > > >>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > > >>>>>>>>>>>> > > >>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > > >>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > > >>>>>>>>>>>> --- a/gcc/hash-table.h > > >>>>>>>>>>>> +++ b/gcc/hash-table.h > > >>>>>>>>>>>> @@ -503,6 +503,7 @@ private: > > >>>>>>>>>>>> > > >>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > > >>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > > >>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > > >>>>>>>>>>>> bool too_empty_p (unsigned int); > > >>>>>>>>>>>> void expand (); > > >>>>>>>>>>>> static bool is_deleted (value_type &v) > > >>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > > >>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > > >>>>>>>>>>>> expand (); > > >>>>>>>>>>>> > > >>>>>>>>>>>> - m_searches++; > > >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > > >>>>>>>>>>>> + if (insert == INSERT) > > >>>>>>>>>>>> + verify (comparable, hash); > > >>>>>>>>>>>> +#endif > > >>>>>>>>>>>> > > >>>>>>>>>>>> + m_searches++; > > >>>>>>>>>>>> value_type *first_deleted_slot = NULL; > > >>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > > >>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > > >>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > > >>>>>>>>>>>> return &m_entries[index]; > > >>>>>>>>>>>> } > > >>>>>>>>>>>> > > >>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > > >>>>>>>>>>>> + > > >>>>>>>>>>>> +/* Report a hash table checking error. */ > > >>>>>>>>>>>> + > > >>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > > >>>>>>>>>>>> +static void > > >>>>>>>>>>>> +hashtab_chk_error () > > >>>>>>>>>>>> +{ > > >>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > > >>>>>>>>>>>> + "equal operator returns true for a pair " > > >>>>>>>>>>>> + "of values with a different hash value\n"); > > >>>>>>>>>>>> + gcc_unreachable (); > > >>>>>>>>>>>> +} > > >>>>>>>>>>> I think an internal_error here is probably still better than a simple > > >>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > > >>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > > >>>>>>>>>> > > >>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > > >>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > > >>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > > >>>>>>>>>>> because we've got too many bugs to fix. > > >>>>>>>>>> Unfortunately it's blocked with these 2 PRs: > > >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > > >>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > > >>>>>>>>> Hi. > > >>>>>>>>> > > >>>>>>>>> I've just added one more PR: > > >>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > > >>>>>>>>> > > >>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > > >>>>>>>>> with a new function disable_sanitize_eq_and_hash. > > >>>>>>>>> > > >>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > > >>>>>>>>> limits maximal number of checks: > > >>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > > >>>>>>>> state set up when you instantiate the object? It's not a huge deal, > > >>>>>>>> just thinking about loud. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > > >>>>>>>> issue :-) > > >>>>>>> > > >>>>>>> There is at least one PR where we have a table where elements _in_ the > > >>>>>>> table are never compared against each other but always against another > > >>>>>>> object (I guess that's usual even), but the setup is in a way that the > > >>>>>>> comparison function only works with those. With the patch we verify > > >>>>>>> hashing/comparison for something that is never used. > > >>>>>>> > > >>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > > >>>>>>> at lookup time, using the object from the lookup and verify that against > > >>>>>>> all other elements? > > >>>>>> > > >>>>>> I don't a have problem with that. Apparently this changes fixes > > >>>>>> PR90450 and PR87847. > > >>>>>> > > >>>>>> Changes from previous version: > > >>>>>> - verification happens only when an element is searched (not inserted) > > >>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > > >>>>>> - new param has been introduced hash-table-verification-limit in order > > >>>>>> to limit number of elements that are compared within a table > > >>>>>> - verification happens only with flag_checking >= 2 > > >>>>>> > > >>>>>> I've been bootstrapping and testing the patch right now. > > >>>>> > > >>>>> Looks like I misremembered the original patch. The issue isn't > > >>>>> comparing random two elements in the table. > > >>>>> > > >>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > > >>>>> without INSERTing. > > >>>>> > > >>>> > > >>>> There's updated version of the patch where I check all find operations > > >>>> (both w/ and w/o insertion). > > >>>> > > >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > > >>>> except for: > > >>>> > > >>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > > >>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value > > >>>> during GIMPLE pass: lim > > >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > > >>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > > >>>> 6 | fn1 () > > >>>> | ^~~ > > >>>> 0x6c5725 hashtab_chk_error > > >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > > >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > > >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > > >>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > > >>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > > >>>> 0xe504ea gather_mem_refs_stmt > > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > > >>>> 0xe504ea analyze_memory_references > > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > > >>>> 0xe504ea tree_ssa_lim > > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > > >>>> 0xe504ea execute > > >>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > > >>>> > > >>>> Richi: it's after your recent patch. > > >>>> > > >>>> For some reason I don't see PR87847 issue any longer. > > >>>> > > >>>> > > >>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > > >>> Don't we still need to deal with the naked fprintf when there's a > > >>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > > >> > > >> Good point, I've just adjusted that. > > >> > > >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > > >> > > >> Ready to be installed? > > > > > > Ugh, the cselib one is really bad. But I don't hold my breath for anyone > > > fixing it ... > > > > Yes :D It's been some time and there's no interest in the PR. > > > > > > > > One question - there's unconditional > > > > > > + if (m_sanitize_eq_and_hash) > > > + verify (comparable, hash); > > > > > > which will read a global variable and have (possibly not inline) call > > > to verify on a common path even with checking disabled. So I think > > > we want to compile this checking feature out for !CHECKING_P > > > or at least make the if __builtin_expect (..., 0), ::verify not > > > inlined and marked pure () (thus, !CHECKING_P is simplest ;)). > > > > Fixed. May I install the patch? The cselib issue can be solved later.. > > After this patch, when I do a configure with --disable-bootstrap, and > build with "gcc (Debian 7.3.0-18) 7.3.0", I get a lot of warnings of > the form > > In file included from ../../gccgo3/gcc/coretypes.h:440:0, > from ../../gccgo3/gcc/go/go-system.h:137, > from ../../gccgo3/gcc/go/gofrontend/go.cc:7: > ../../gccgo3/gcc/hash-table.h:1017:1: warning: ‘void > hashtab_chk_error()’ defined but not used [-Wunused-function] > hashtab_chk_error () > ^~~~~~~~~~~~~~~~~ > > These are just warnings, since I am using --disable-bootstrap, but > they are distracting. > > This patch fixes it. OK for trunk? Hmm, the function is called exactly once. I guess the intent was to not emit the printf in every ::verify instance but then why not instantiate this function in just hash-table.c and not mark it inline? Richard. > > Ian > > 2019-06-23 Ian Lance Taylor <iant@golang.org> > > * hash-table.h (hashtab_chk_error): Add ATTRIBUTE_UNUSED. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-24 12:29 ` Richard Biener @ 2019-06-24 13:51 ` Martin Liška 2019-06-24 14:10 ` Richard Biener 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2019-06-24 13:51 UTC (permalink / raw) To: Richard Biener, Ian Lance Taylor Cc: Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On 6/24/19 2:29 PM, Richard Biener wrote: > On Mon, Jun 24, 2019 at 1:08 AM Ian Lance Taylor <iant@golang.org> wrote: >> >> On Fri, Jun 7, 2019 at 5:04 AM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>> >>> On 6/7/19 10:57 AM, Richard Biener wrote: >>>> On Mon, Jun 3, 2019 at 3:35 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>> >>>>> On 6/1/19 12:06 AM, Jeff Law wrote: >>>>>> On 5/22/19 3:13 AM, Martin LiÅ¡ka wrote: >>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: >>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>>>>> >>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: >>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: >>>>>>>>>>> >>>>>>>>>>> On 5/13/19 1:41 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>> On 11/8/18 9:56 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: >>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: >>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin LiÅ¡ka wrote: >>>>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>>>> + "of values with a different hash value"); >>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf >>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: >>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack >>>>>>>>>>>>>>>> ^^^^^^ >>>>>>>>>>>>>>> Sure, fixed in attached patch. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>>>> +} >>>>>>>>>>>>>>>> Jakub >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 >>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> >>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 >>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- >>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h >>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 >>>>>>>>>>>>>>> --- a/gcc/hash-table.h >>>>>>>>>>>>>>> +++ b/gcc/hash-table.h >>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; >>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); >>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); >>>>>>>>>>>>>>> bool too_empty_p (unsigned int); >>>>>>>>>>>>>>> void expand (); >>>>>>>>>>>>>>> static bool is_deleted (value_type &v) >>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) >>>>>>>>>>>>>>> expand (); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - m_searches++; >>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>> + if (insert == INSERT) >>>>>>>>>>>>>>> + verify (comparable, hash); >>>>>>>>>>>>>>> +#endif >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> + m_searches++; >>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; >>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); >>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); >>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> >>>>>>>>>>>>>>> return &m_entries[index]; >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> +/* Report a hash table checking error. */ >>>>>>>>>>>>>>> + >>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD >>>>>>>>>>>>>>> +static void >>>>>>>>>>>>>>> +hashtab_chk_error () >>>>>>>>>>>>>>> +{ >>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " >>>>>>>>>>>>>>> + "equal operator returns true for a pair " >>>>>>>>>>>>>>> + "of values with a different hash value\n"); >>>>>>>>>>>>>>> + gcc_unreachable (); >>>>>>>>>>>>>>> +} >>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple >>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) >>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. >>>>>>>>>>>>> >>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and >>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into >>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING >>>>>>>>>>>>>> because we've got too many bugs to fix. >>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 >>>>>>>>>>>> Hi. >>>>>>>>>>>> >>>>>>>>>>>> I've just added one more PR: >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 >>>>>>>>>>>> >>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs >>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. >>>>>>>>>>>> >>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch >>>>>>>>>>>> limits maximal number of checks: >>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its >>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, >>>>>>>>>>> just thinking about loud. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking >>>>>>>>>>> issue :-) >>>>>>>>>> >>>>>>>>>> There is at least one PR where we have a table where elements _in_ the >>>>>>>>>> table are never compared against each other but always against another >>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the >>>>>>>>>> comparison function only works with those. With the patch we verify >>>>>>>>>> hashing/comparison for something that is never used. >>>>>>>>>> >>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing >>>>>>>>>> at lookup time, using the object from the lookup and verify that against >>>>>>>>>> all other elements? >>>>>>>>> >>>>>>>>> I don't a have problem with that. Apparently this changes fixes >>>>>>>>> PR90450 and PR87847. >>>>>>>>> >>>>>>>>> Changes from previous version: >>>>>>>>> - verification happens only when an element is searched (not inserted) >>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table >>>>>>>>> - new param has been introduced hash-table-verification-limit in order >>>>>>>>> to limit number of elements that are compared within a table >>>>>>>>> - verification happens only with flag_checking >= 2 >>>>>>>>> >>>>>>>>> I've been bootstrapping and testing the patch right now. >>>>>>>> >>>>>>>> Looks like I misremembered the original patch. The issue isn't >>>>>>>> comparing random two elements in the table. >>>>>>>> >>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash >>>>>>>> without INSERTing. >>>>>>>> >>>>>>> >>>>>>> There's updated version of the patch where I check all find operations >>>>>>> (both w/ and w/o insertion). >>>>>>> >>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests >>>>>>> except for: >>>>>>> >>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c >>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value >>>>>>> during GIMPLE pass: lim >>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function âfn1â: >>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 >>>>>>> 6 | fn1 () >>>>>>> | ^~~ >>>>>>> 0x6c5725 hashtab_chk_error >>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 >>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) >>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 >>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) >>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 >>>>>>> 0xe504ea gather_mem_refs_stmt >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 >>>>>>> 0xe504ea analyze_memory_references >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 >>>>>>> 0xe504ea tree_ssa_lim >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 >>>>>>> 0xe504ea execute >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 >>>>>>> >>>>>>> Richi: it's after your recent patch. >>>>>>> >>>>>>> For some reason I don't see PR87847 issue any longer. >>>>>>> >>>>>>> >>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? >>>>>> Don't we still need to deal with the naked fprintf when there's a >>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? >>>>> >>>>> Good point, I've just adjusted that. >>>>> >>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>> >>>>> Ready to be installed? >>>> >>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone >>>> fixing it ... >>> >>> Yes :D It's been some time and there's no interest in the PR. >>> >>>> >>>> One question - there's unconditional >>>> >>>> + if (m_sanitize_eq_and_hash) >>>> + verify (comparable, hash); >>>> >>>> which will read a global variable and have (possibly not inline) call >>>> to verify on a common path even with checking disabled. So I think >>>> we want to compile this checking feature out for !CHECKING_P >>>> or at least make the if __builtin_expect (..., 0), ::verify not >>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). >>> >>> Fixed. May I install the patch? The cselib issue can be solved later.. >> >> After this patch, when I do a configure with --disable-bootstrap, and >> build with "gcc (Debian 7.3.0-18) 7.3.0", I get a lot of warnings of >> the form >> >> In file included from ../../gccgo3/gcc/coretypes.h:440:0, >> from ../../gccgo3/gcc/go/go-system.h:137, >> from ../../gccgo3/gcc/go/gofrontend/go.cc:7: >> ../../gccgo3/gcc/hash-table.h:1017:1: warning: âvoid >> hashtab_chk_error()â defined but not used [-Wunused-function] >> hashtab_chk_error () >> ^~~~~~~~~~~~~~~~~ >> >> These are just warnings, since I am using --disable-bootstrap, but >> they are distracting. >> >> This patch fixes it. OK for trunk? > > Hmm, the function is called exactly once. I guess the intent was > to not emit the printf in every ::verify instance but then why not > instantiate this function in just hash-table.c and not mark it inline? I marked the function ATTRIBUTE_COLD, so it should not be inlined into ::verify. 1013 /* Report a hash table checking error. */ 1014 1015 ATTRIBUTE_NORETURN ATTRIBUTE_COLD 1016 static void 1017 hashtab_chk_error () 1018 { 1019 fprintf (stderr, "hash table checking failed: " 1020 "equal operator returns true for a pair " 1021 "of values with a different hash value\n"); 1022 gcc_unreachable (); 1023 } Martin > > Richard. > >> >> Ian >> >> 2019-06-23 Ian Lance Taylor <iant@golang.org> >> >> * hash-table.h (hashtab_chk_error): Add ATTRIBUTE_UNUSED. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-24 13:51 ` Martin Liška @ 2019-06-24 14:10 ` Richard Biener 2019-06-25 10:25 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Richard Biener @ 2019-06-24 14:10 UTC (permalink / raw) To: Martin Liška Cc: Ian Lance Taylor, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Mon, Jun 24, 2019 at 3:51 PM Martin Liška <mliska@suse.cz> wrote: > > On 6/24/19 2:29 PM, Richard Biener wrote: > > On Mon, Jun 24, 2019 at 1:08 AM Ian Lance Taylor <iant@golang.org> wrote: > >> > >> On Fri, Jun 7, 2019 at 5:04 AM Martin Liška <mliska@suse.cz> wrote: > >>> > >>> On 6/7/19 10:57 AM, Richard Biener wrote: > >>>> On Mon, Jun 3, 2019 at 3:35 PM Martin Liška <mliska@suse.cz> wrote: > >>>>> > >>>>> On 6/1/19 12:06 AM, Jeff Law wrote: > >>>>>> On 5/22/19 3:13 AM, Martin Liška wrote: > >>>>>>> On 5/21/19 1:51 PM, Richard Biener wrote: > >>>>>>>> On Tue, May 21, 2019 at 1:02 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>>>> > >>>>>>>>> On 5/21/19 11:38 AM, Richard Biener wrote: > >>>>>>>>>> On Tue, May 21, 2019 at 12:07 AM Jeff Law <law@redhat.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>> On 5/13/19 1:41 AM, Martin Liška wrote: > >>>>>>>>>>>> On 11/8/18 9:56 AM, Martin Liška wrote: > >>>>>>>>>>>>> On 11/7/18 11:23 PM, Jeff Law wrote: > >>>>>>>>>>>>>> On 10/30/18 6:28 AM, Martin Liška wrote: > >>>>>>>>>>>>>>> On 10/30/18 11:03 AM, Jakub Jelinek wrote: > >>>>>>>>>>>>>>>> On Mon, Oct 29, 2018 at 04:14:21PM +0100, Martin Liška wrote: > >>>>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>>>> + "of values with a different hash value"); > >>>>>>>>>>>>>>>> BTW, either use internal_error here, or at least if using fprintf > >>>>>>>>>>>>>>>> terminate with \n, in your recent mail I saw: > >>>>>>>>>>>>>>>> ...different hash valueduring RTL pass: vartrack > >>>>>>>>>>>>>>>> ^^^^^^ > >>>>>>>>>>>>>>> Sure, fixed in attached patch. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Martin > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>>> Jakub > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 0001-Sanitize-equals-and-hash-functions-in-hash-tables.patch > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> From 0d9c979c845580a98767b83c099053d36eb49bb9 Mon Sep 17 00:00:00 2001 > >>>>>>>>>>>>>>> From: marxin <mliska@suse.cz> > >>>>>>>>>>>>>>> Date: Mon, 29 Oct 2018 09:38:21 +0100 > >>>>>>>>>>>>>>> Subject: [PATCH] Sanitize equals and hash functions in hash-tables. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> --- > >>>>>>>>>>>>>>> gcc/hash-table.h | 40 +++++++++++++++++++++++++++++++++++++++- > >>>>>>>>>>>>>>> 1 file changed, 39 insertions(+), 1 deletion(-) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> diff --git a/gcc/hash-table.h b/gcc/hash-table.h > >>>>>>>>>>>>>>> index bd83345c7b8..694eedfc4be 100644 > >>>>>>>>>>>>>>> --- a/gcc/hash-table.h > >>>>>>>>>>>>>>> +++ b/gcc/hash-table.h > >>>>>>>>>>>>>>> @@ -503,6 +503,7 @@ private: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> value_type *alloc_entries (size_t n CXX_MEM_STAT_INFO) const; > >>>>>>>>>>>>>>> value_type *find_empty_slot_for_expand (hashval_t); > >>>>>>>>>>>>>>> + void verify (const compare_type &comparable, hashval_t hash); > >>>>>>>>>>>>>>> bool too_empty_p (unsigned int); > >>>>>>>>>>>>>>> void expand (); > >>>>>>>>>>>>>>> static bool is_deleted (value_type &v) > >>>>>>>>>>>>>>> @@ -882,8 +883,12 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>> if (insert == INSERT && m_size * 3 <= m_n_elements * 4) > >>>>>>>>>>>>>>> expand (); > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - m_searches++; > >>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>> + if (insert == INSERT) > >>>>>>>>>>>>>>> + verify (comparable, hash); > >>>>>>>>>>>>>>> +#endif > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> + m_searches++; > >>>>>>>>>>>>>>> value_type *first_deleted_slot = NULL; > >>>>>>>>>>>>>>> hashval_t index = hash_table_mod1 (hash, m_size_prime_index); > >>>>>>>>>>>>>>> hashval_t hash2 = hash_table_mod2 (hash, m_size_prime_index); > >>>>>>>>>>>>>>> @@ -930,6 +935,39 @@ hash_table<Descriptor, Allocator> > >>>>>>>>>>>>>>> return &m_entries[index]; > >>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> +#if ENABLE_EXTRA_CHECKING > >>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>> +/* Report a hash table checking error. */ > >>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>> +ATTRIBUTE_NORETURN ATTRIBUTE_COLD > >>>>>>>>>>>>>>> +static void > >>>>>>>>>>>>>>> +hashtab_chk_error () > >>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>> + fprintf (stderr, "hash table checking failed: " > >>>>>>>>>>>>>>> + "equal operator returns true for a pair " > >>>>>>>>>>>>>>> + "of values with a different hash value\n"); > >>>>>>>>>>>>>>> + gcc_unreachable (); > >>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>> I think an internal_error here is probably still better than a simple > >>>>>>>>>>>>>> fprintf, even if the fprintf is terminated with a \n :-) > >>>>>>>>>>>>> Fully agree with that, but I see a lot of build errors when using internal_error. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> The question then becomes can we bootstrap with this stuff enabled and > >>>>>>>>>>>>>> if not, are we likely to soon? It'd be a shame to put it into > >>>>>>>>>>>>>> EXTRA_CHECKING, but then not be able to really use EXTRA_CHECKING > >>>>>>>>>>>>>> because we've got too many bugs to fix. > >>>>>>>>>>>>> Unfortunately it's blocked with these 2 PRs: > >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87845 > >>>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87847 > >>>>>>>>>>>> Hi. > >>>>>>>>>>>> > >>>>>>>>>>>> I've just added one more PR: > >>>>>>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90450 > >>>>>>>>>>>> > >>>>>>>>>>>> I'm sending updated version of the patch that provides a disablement for the 3 PRs > >>>>>>>>>>>> with a new function disable_sanitize_eq_and_hash. > >>>>>>>>>>>> > >>>>>>>>>>>> With that I can bootstrap and finish tests. However, I've done that with a patch > >>>>>>>>>>>> limits maximal number of checks: > >>>>>>>>>>> So rather than call the disable_sanitize_eq_and_hash, can you have its > >>>>>>>>>>> state set up when you instantiate the object? It's not a huge deal, > >>>>>>>>>>> just thinking about loud. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> So how do we want to go forward, particularly the EXTRA_EXTRA checking > >>>>>>>>>>> issue :-) > >>>>>>>>>> > >>>>>>>>>> There is at least one PR where we have a table where elements _in_ the > >>>>>>>>>> table are never compared against each other but always against another > >>>>>>>>>> object (I guess that's usual even), but the setup is in a way that the > >>>>>>>>>> comparison function only works with those. With the patch we verify > >>>>>>>>>> hashing/comparison for something that is never used. > >>>>>>>>>> > >>>>>>>>>> So - wouldn't it be more "correct" to only verify comparison/hashing > >>>>>>>>>> at lookup time, using the object from the lookup and verify that against > >>>>>>>>>> all other elements? > >>>>>>>>> > >>>>>>>>> I don't a have problem with that. Apparently this changes fixes > >>>>>>>>> PR90450 and PR87847. > >>>>>>>>> > >>>>>>>>> Changes from previous version: > >>>>>>>>> - verification happens only when an element is searched (not inserted) > >>>>>>>>> - new argument 'sanitize_eq_and_hash' added for hash_table::hash_table > >>>>>>>>> - new param has been introduced hash-table-verification-limit in order > >>>>>>>>> to limit number of elements that are compared within a table > >>>>>>>>> - verification happens only with flag_checking >= 2 > >>>>>>>>> > >>>>>>>>> I've been bootstrapping and testing the patch right now. > >>>>>>>> > >>>>>>>> Looks like I misremembered the original patch. The issue isn't > >>>>>>>> comparing random two elements in the table. > >>>>>>>> > >>>>>>>> That it fixes PR90450 is because LIM never calls find_slot_with_hash > >>>>>>>> without INSERTing. > >>>>>>>> > >>>>>>> > >>>>>>> There's updated version of the patch where I check all find operations > >>>>>>> (both w/ and w/o insertion). > >>>>>>> > >>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests > >>>>>>> except for: > >>>>>>> > >>>>>>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c -O2 -c > >>>>>>> hash table checking failed: equal operator returns true for a pair of values with a different hash value > >>>>>>> during GIMPLE pass: lim > >>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c: In function ‘fn1’: > >>>>>>> /home/marxin/Programming/gcc/gcc/testsuite/gcc.dg/torture/pr63941.c:6:1: internal compiler error: in hashtab_chk_error, at hash-table.h:1019 > >>>>>>> 6 | fn1 () > >>>>>>> | ^~~ > >>>>>>> 0x6c5725 hashtab_chk_error > >>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1019 > >>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::verify(ao_ref* const&, unsigned int) > >>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:1040 > >>>>>>> 0xe504ea hash_table<mem_ref_hasher, false, xcallocator>::find_slot_with_hash(ao_ref* const&, unsigned int, insert_option) > >>>>>>> /home/marxin/Programming/gcc/gcc/hash-table.h:960 > >>>>>>> 0xe504ea gather_mem_refs_stmt > >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1501 > >>>>>>> 0xe504ea analyze_memory_references > >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:1625 > >>>>>>> 0xe504ea tree_ssa_lim > >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2646 > >>>>>>> 0xe504ea execute > >>>>>>> /home/marxin/Programming/gcc/gcc/tree-ssa-loop-im.c:2708 > >>>>>>> > >>>>>>> Richi: it's after your recent patch. > >>>>>>> > >>>>>>> For some reason I don't see PR87847 issue any longer. > >>>>>>> > >>>>>>> > >>>>>>> May I install the patch with disabled sanitization in tree-ssa-loop-im.c ? > >>>>>> Don't we still need to deal with the naked fprintf when there's a > >>>>>> failure. ie, shouldn't we be raising it with a gcc_assert or somesuch? > >>>>> > >>>>> Good point, I've just adjusted that. > >>>>> > >>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >>>>> > >>>>> Ready to be installed? > >>>> > >>>> Ugh, the cselib one is really bad. But I don't hold my breath for anyone > >>>> fixing it ... > >>> > >>> Yes :D It's been some time and there's no interest in the PR. > >>> > >>>> > >>>> One question - there's unconditional > >>>> > >>>> + if (m_sanitize_eq_and_hash) > >>>> + verify (comparable, hash); > >>>> > >>>> which will read a global variable and have (possibly not inline) call > >>>> to verify on a common path even with checking disabled. So I think > >>>> we want to compile this checking feature out for !CHECKING_P > >>>> or at least make the if __builtin_expect (..., 0), ::verify not > >>>> inlined and marked pure () (thus, !CHECKING_P is simplest ;)). > >>> > >>> Fixed. May I install the patch? The cselib issue can be solved later.. > >> > >> After this patch, when I do a configure with --disable-bootstrap, and > >> build with "gcc (Debian 7.3.0-18) 7.3.0", I get a lot of warnings of > >> the form > >> > >> In file included from ../../gccgo3/gcc/coretypes.h:440:0, > >> from ../../gccgo3/gcc/go/go-system.h:137, > >> from ../../gccgo3/gcc/go/gofrontend/go.cc:7: > >> ../../gccgo3/gcc/hash-table.h:1017:1: warning: ‘void > >> hashtab_chk_error()’ defined but not used [-Wunused-function] > >> hashtab_chk_error () > >> ^~~~~~~~~~~~~~~~~ > >> > >> These are just warnings, since I am using --disable-bootstrap, but > >> they are distracting. > >> > >> This patch fixes it. OK for trunk? > > > > Hmm, the function is called exactly once. I guess the intent was > > to not emit the printf in every ::verify instance but then why not > > instantiate this function in just hash-table.c and not mark it inline? > > I marked the function ATTRIBUTE_COLD, so it should not be inlined > into ::verify. You still get one instance in each TU ... > 1013 /* Report a hash table checking error. */ > 1014 > 1015 ATTRIBUTE_NORETURN ATTRIBUTE_COLD > 1016 static void > 1017 hashtab_chk_error () > 1018 { > 1019 fprintf (stderr, "hash table checking failed: " > 1020 "equal operator returns true for a pair " > 1021 "of values with a different hash value\n"); > 1022 gcc_unreachable (); > 1023 } > > Martin > > > > > Richard. > > > >> > >> Ian > >> > >> 2019-06-23 Ian Lance Taylor <iant@golang.org> > >> > >> * hash-table.h (hashtab_chk_error): Add ATTRIBUTE_UNUSED. > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-24 14:10 ` Richard Biener @ 2019-06-25 10:25 ` Martin Liška 2019-06-25 11:59 ` Martin Liška 2019-06-25 14:23 ` Richard Biener 0 siblings, 2 replies; 53+ messages in thread From: Martin Liška @ 2019-06-25 10:25 UTC (permalink / raw) To: Richard Biener Cc: Ian Lance Taylor, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor [-- Attachment #1: Type: text/plain, Size: 230 bytes --] On 6/24/19 4:09 PM, Richard Biener wrote: > You still get one instance in each TU ... Right, fixed in attached patch. Patch can bootstrap on x86_64-linux-gnu and survives regression tests. Ready to be installed? Thanks, Martin [-- Attachment #2: 0001-Put-hashtab_chk_error-into-hash-table.c.patch --] [-- Type: text/x-patch, Size: 2083 bytes --] From aa5ea14a8665b14aa60245c42bd4c9809d0bf81a Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Tue, 25 Jun 2019 10:33:39 +0200 Subject: [PATCH] Put hashtab_chk_error into hash-table.c. gcc/ChangeLog: 2019-06-25 Martin Liska <mliska@suse.cz> * hash-table.c (hashtab_chk_error): Move here from ... * hash-table.h (hashtab_chk_error): ... here. --- gcc/hash-table.c | 12 ++++++++++++ gcc/hash-table.h | 14 ++------------ 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/gcc/hash-table.c b/gcc/hash-table.c index 8e86fffa36f..e3b5d3da09e 100644 --- a/gcc/hash-table.c +++ b/gcc/hash-table.c @@ -124,3 +124,15 @@ void dump_hash_table_loc_statistics (void) hash_table_usage ().dump (origin); } } + +/* Report a hash table checking error. */ + +ATTRIBUTE_NORETURN ATTRIBUTE_COLD +void +hashtab_chk_error () +{ + fprintf (stderr, "hash table checking failed: " + "equal operator returns true for a pair " + "of values with a different hash value\n"); + gcc_unreachable (); +} diff --git a/gcc/hash-table.h b/gcc/hash-table.h index 4f5e150a0ac..a39fb942158 100644 --- a/gcc/hash-table.h +++ b/gcc/hash-table.h @@ -303,6 +303,8 @@ extern unsigned int hash_table_sanitize_eq_limit; extern unsigned int hash_table_higher_prime_index (unsigned long n) ATTRIBUTE_PURE; +extern ATTRIBUTE_NORETURN ATTRIBUTE_COLD void hashtab_chk_error (); + /* Return X % Y using multiplicative inverse values INV and SHIFT. The multiplicative inverses computed above are for 32-bit types, @@ -1010,18 +1012,6 @@ hash_table<Descriptor, Lazy, Allocator> return &m_entries[index]; } -/* Report a hash table checking error. */ - -ATTRIBUTE_NORETURN ATTRIBUTE_COLD -static void -hashtab_chk_error () -{ - fprintf (stderr, "hash table checking failed: " - "equal operator returns true for a pair " - "of values with a different hash value\n"); - gcc_unreachable (); -} - /* Verify that all existing elements in th hash table which are equal to COMPARABLE have an equal HASH value provided as argument. */ -- 2.21.0 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-25 10:25 ` Martin Liška @ 2019-06-25 11:59 ` Martin Liška 2019-06-25 14:23 ` Richard Biener 1 sibling, 0 replies; 53+ messages in thread From: Martin Liška @ 2019-06-25 11:59 UTC (permalink / raw) To: Richard Biener Cc: Ian Lance Taylor, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On 6/25/19 12:25 PM, Martin LiÅ¡ka wrote: > On 6/24/19 4:09 PM, Richard Biener wrote: >> You still get one instance in each TU ... > > Right, fixed in attached patch. > > Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > > Ready to be installed? > Thanks, > Martin > Btw. clang provides a reasonable warning for that: /home/marxin/Programming/gcc/gcc/hash-table.h:1017:1: warning: 'static' function 'hashtab_chk_error' declared in header file should be declared 'static inline' [-Wunneeded-internal-declaration] Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH][RFC] Sanitize equals and hash functions in hash-tables. 2019-06-25 10:25 ` Martin Liška 2019-06-25 11:59 ` Martin Liška @ 2019-06-25 14:23 ` Richard Biener 1 sibling, 0 replies; 53+ messages in thread From: Richard Biener @ 2019-06-25 14:23 UTC (permalink / raw) To: Martin Liška Cc: Ian Lance Taylor, Jeff Law, Jakub Jelinek, Alexander Monakov, GCC Patches, Nathan Sidwell, Jason Merrill, Paul Richard Thomas, Martin Jambor On Tue, Jun 25, 2019 at 12:25 PM Martin Liška <mliska@suse.cz> wrote: > > On 6/24/19 4:09 PM, Richard Biener wrote: > > You still get one instance in each TU ... > > Right, fixed in attached patch. > > Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > > Ready to be installed? Yes. Thanks, Richard. > Thanks, > Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* hash-table violation in cselib.c 2018-10-29 12:02 [PATCH][RFC] Sanitize equals and hash functions in hash-tables Martin Liška 2018-10-29 14:28 ` Alexander Monakov @ 2018-10-30 10:25 ` Martin Liška 2018-11-01 11:57 ` Martin Liška 2018-10-30 10:46 ` hash-table violation in gcc/fortran/trans-decl.c Martin Liška 2018-10-30 11:07 ` hash-table violation in gcc/cp/pt.c Martin Liška 3 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2018-10-30 10:25 UTC (permalink / raw) To: gcc-patches Cc: Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor > 1) cselib_lookup_1: > > $ cat ice.c > a() { b(); } > > $ /dev/shm/objdir/gcc/xgcc -B/dev/shm/objdir/gcc/ ice.c -g -c -fchecking=3 -O > hash table checking failed: equal operator returns true for a pair of values with a different hash valueduring RTL pass: vartrack > ice.c:1:1: internal compiler error: in find_slot_with_hash, at hash-table.h:905 > 1 | a() { b(); } > | ^ > 0x9680b5 hash_table<cselib_hasher, xcallocator>::find_slot_with_hash(cselib_hasher::key* const&, unsigned int, insert_option) > /home/marxin/Programming/gcc/gcc/hash-table.h:905 > 0x962518 cselib_find_slot > /home/marxin/Programming/gcc/gcc/cselib.c:584 > 0x9625d4 cselib_lookup_1 > /home/marxin/Programming/gcc/gcc/cselib.c:2097 > 0x9625d4 cselib_lookup(rtx_def*, machine_mode, int, machine_mode) > /home/marxin/Programming/gcc/gcc/cselib.c:2141 > 0x965ee7 cselib_record_sets > /home/marxin/Programming/gcc/gcc/cselib.c:2593 > 0x9670a9 cselib_process_insn(rtx_insn*) > /home/marxin/Programming/gcc/gcc/cselib.c:2790 > 0x1036b73 vt_initialize > /home/marxin/Programming/gcc/gcc/var-tracking.c:10231 > 0x103b98a variable_tracking_main_1 > /home/marxin/Programming/gcc/gcc/var-tracking.c:10460 > 0x103b98a variable_tracking_main() > /home/marxin/Programming/gcc/gcc/var-tracking.c:10513 > I did a small analysis and problematic back-trace is: $ (gdb) bt #0 new_cselib_val (hash=9, mode=E_SImode, x=0x7ffff6ad7be8) at /home/marxin/Programming/gcc/gcc/cselib.c:1323 #1 0x0000000000af86f0 in cselib_lookup_1 (x=0x7ffff6ad7be8, mode=E_SImode, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:2046 #2 0x0000000000af8c5e in cselib_lookup (x=0x7ffff6ad7be8, mode=E_SImode, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:2153 #3 0x0000000000af6119 in cselib_hash_rtx2 (x=0x7ffff6ad7be8, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:1089 #4 0x0000000000af693d in cselib_hash_rtx2 (x=0x7ffff6ad7cc0, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:1254 #5 0x0000000000af6ab4 in cselib_hash_rtx (x=0x7ffff6ad7cc0, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:1309 #6 0x0000000000af8aef in cselib_lookup_1 (x=0x7ffff6ad7cc0, mode=E_CCZmode, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:2104 #7 0x0000000000af8c5e in cselib_lookup (x=0x7ffff6ad7cc0, mode=E_CCZmode, create=1, memmode=E_VOIDmode) at /home/marxin/Programming/gcc/gcc/cselib.c:2153 #8 0x0000000000af9df4 in cselib_record_sets (insn=0x7ffff69a1480) at /home/marxin/Programming/gcc/gcc/cselib.c:2605 #9 0x0000000000afa946 in cselib_process_insn (insn=0x7ffff69a1480) at /home/marxin/Programming/gcc/gcc/cselib.c:2802 as seen hash ==9, which is: >â2046 e = new_cselib_val (next_uid, GET_MODE (x), x); it seems to me that there are combined real hash values (based on cselib_hash_rtx) and next_uid (which is incremented). Anyway, somebody familiar with the code needs to analyze that. Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: hash-table violation in cselib.c 2018-10-30 10:25 ` hash-table violation in cselib.c Martin Liška @ 2018-11-01 11:57 ` Martin Liška 0 siblings, 0 replies; 53+ messages in thread From: Martin Liška @ 2018-11-01 11:57 UTC (permalink / raw) To: gcc-patches Cc: Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor I created PR87845 for that. Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* hash-table violation in gcc/fortran/trans-decl.c 2018-10-29 12:02 [PATCH][RFC] Sanitize equals and hash functions in hash-tables Martin Liška 2018-10-29 14:28 ` Alexander Monakov 2018-10-30 10:25 ` hash-table violation in cselib.c Martin Liška @ 2018-10-30 10:46 ` Martin Liška 2018-10-31 10:00 ` Trevor Saunders 2018-10-30 11:07 ` hash-table violation in gcc/cp/pt.c Martin Liška 3 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2018-10-30 10:46 UTC (permalink / raw) To: gcc-patches Cc: Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor, Trevor Saunders On 10/29/18 12:04 PM, Martin LiÅ¡ka wrote: > 2) gfc_find_module > > $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gfortran.dg/coarray/alloc_comp_2.f90 -fcoarray=single -fchecking=3 > hash table checking failed: equal operator returns true for a pair of values with a different hash valuef951: internal compiler error: in find_slot_with_hash, at hash-table.h:905 > 0x8e5e86 hash_table<module_hasher, xcallocator>::find_slot_with_hash(char const* const&, unsigned int, insert_option) > /home/marxin/Programming/gcc/gcc/hash-table.h:905 > 0x8e2c2c gfc_find_module(char const*) > /home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:4865 > 0x8e4f42 gfc_generate_module_vars(gfc_namespace*) > /home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:5475 > 0x8b8d7e gfc_generate_module_code(gfc_namespace*) > /home/marxin/Programming/gcc/gcc/fortran/trans.c:2190 > 0x868427 translate_all_program_units > /home/marxin/Programming/gcc/gcc/fortran/parse.c:6112 > 0x868427 gfc_parse_file() > /home/marxin/Programming/gcc/gcc/fortran/parse.c:6328 > 0x8b19cb gfc_be_parse_file > /home/marxin/Programming/gcc/gcc/fortran/f95-lang.c:204 This one is real issue introduced in r216127, I'll post it to corresponding mailing list after testing: diff --git a/gcc/fortran/trans-decl.c b/gcc/fortran/trans-decl.c index 88f9f570725..e8da9e11d22 100644 --- a/gcc/fortran/trans-decl.c +++ b/gcc/fortran/trans-decl.c @@ -4825,7 +4825,7 @@ struct module_hasher : ggc_ptr_hash<module_htab_entry> { typedef const char *compare_type; - static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s); } + static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s->name); } static bool equal (module_htab_entry *a, const char *b) { Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: hash-table violation in gcc/fortran/trans-decl.c 2018-10-30 10:46 ` hash-table violation in gcc/fortran/trans-decl.c Martin Liška @ 2018-10-31 10:00 ` Trevor Saunders 2018-10-31 10:18 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Trevor Saunders @ 2018-10-31 10:00 UTC (permalink / raw) To: Martin Li??ka Cc: gcc-patches, Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor On Tue, Oct 30, 2018 at 11:07:16AM +0100, Martin Li??ka wrote: > On 10/29/18 12:04 PM, Martin Li??ka wrote: > > 2) gfc_find_module > > > > $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gfortran.dg/coarray/alloc_comp_2.f90 -fcoarray=single -fchecking=3 > > hash table checking failed: equal operator returns true for a pair of values with a different hash valuef951: internal compiler error: in find_slot_with_hash, at hash-table.h:905 > > 0x8e5e86 hash_table<module_hasher, xcallocator>::find_slot_with_hash(char const* const&, unsigned int, insert_option) > > /home/marxin/Programming/gcc/gcc/hash-table.h:905 > > 0x8e2c2c gfc_find_module(char const*) > > /home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:4865 > > 0x8e4f42 gfc_generate_module_vars(gfc_namespace*) > > /home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:5475 > > 0x8b8d7e gfc_generate_module_code(gfc_namespace*) > > /home/marxin/Programming/gcc/gcc/fortran/trans.c:2190 > > 0x868427 translate_all_program_units > > /home/marxin/Programming/gcc/gcc/fortran/parse.c:6112 > > 0x868427 gfc_parse_file() > > /home/marxin/Programming/gcc/gcc/fortran/parse.c:6328 > > 0x8b19cb gfc_be_parse_file > > /home/marxin/Programming/gcc/gcc/fortran/f95-lang.c:204 > > This one is real issue introduced in r216127, I'll post it to corresponding mailing list > after testing: Yeah, sorry and lgtm. I wonder if we could make htab_hash_string take a const char * to catch this sooner? Trev > > diff --git a/gcc/fortran/trans-decl.c b/gcc/fortran/trans-decl.c > index 88f9f570725..e8da9e11d22 100644 > --- a/gcc/fortran/trans-decl.c > +++ b/gcc/fortran/trans-decl.c > @@ -4825,7 +4825,7 @@ struct module_hasher : ggc_ptr_hash<module_htab_entry> > { > typedef const char *compare_type; > > - static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s); } > + static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s->name); } > static bool > equal (module_htab_entry *a, const char *b) > { > > Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: hash-table violation in gcc/fortran/trans-decl.c 2018-10-31 10:00 ` Trevor Saunders @ 2018-10-31 10:18 ` Martin Liška 0 siblings, 0 replies; 53+ messages in thread From: Martin Liška @ 2018-10-31 10:18 UTC (permalink / raw) To: Trevor Saunders Cc: gcc-patches, Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor On 10/31/18 10:40 AM, Trevor Saunders wrote: > On Tue, Oct 30, 2018 at 11:07:16AM +0100, Martin Li??ka wrote: >> On 10/29/18 12:04 PM, Martin Li??ka wrote: >>> 2) gfc_find_module >>> >>> $ ./xgcc -B. /home/marxin/Programming/gcc/gcc/testsuite/gfortran.dg/coarray/alloc_comp_2.f90 -fcoarray=single -fchecking=3 >>> hash table checking failed: equal operator returns true for a pair of values with a different hash valuef951: internal compiler error: in find_slot_with_hash, at hash-table.h:905 >>> 0x8e5e86 hash_table<module_hasher, xcallocator>::find_slot_with_hash(char const* const&, unsigned int, insert_option) >>> /home/marxin/Programming/gcc/gcc/hash-table.h:905 >>> 0x8e2c2c gfc_find_module(char const*) >>> /home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:4865 >>> 0x8e4f42 gfc_generate_module_vars(gfc_namespace*) >>> /home/marxin/Programming/gcc/gcc/fortran/trans-decl.c:5475 >>> 0x8b8d7e gfc_generate_module_code(gfc_namespace*) >>> /home/marxin/Programming/gcc/gcc/fortran/trans.c:2190 >>> 0x868427 translate_all_program_units >>> /home/marxin/Programming/gcc/gcc/fortran/parse.c:6112 >>> 0x868427 gfc_parse_file() >>> /home/marxin/Programming/gcc/gcc/fortran/parse.c:6328 >>> 0x8b19cb gfc_be_parse_file >>> /home/marxin/Programming/gcc/gcc/fortran/f95-lang.c:204 >> >> This one is real issue introduced in r216127, I'll post it to corresponding mailing list >> after testing: > > Yeah, sorry and lgtm. I wonder if we could make htab_hash_string take a > const char * to catch this sooner? That happens. Exactly my thoughts! The issue is that old C-style htab in libiberty expects void *, so I can come up with a specific const char * specialization that we can leverage in gcc internally (read with C++). Martin > > Trev > >> >> diff --git a/gcc/fortran/trans-decl.c b/gcc/fortran/trans-decl.c >> index 88f9f570725..e8da9e11d22 100644 >> --- a/gcc/fortran/trans-decl.c >> +++ b/gcc/fortran/trans-decl.c >> @@ -4825,7 +4825,7 @@ struct module_hasher : ggc_ptr_hash<module_htab_entry> >> { >> typedef const char *compare_type; >> >> - static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s); } >> + static hashval_t hash (module_htab_entry *s) { return htab_hash_string (s->name); } >> static bool >> equal (module_htab_entry *a, const char *b) >> { >> >> Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* hash-table violation in gcc/cp/pt.c 2018-10-29 12:02 [PATCH][RFC] Sanitize equals and hash functions in hash-tables Martin Liška ` (2 preceding siblings ...) 2018-10-30 10:46 ` hash-table violation in gcc/fortran/trans-decl.c Martin Liška @ 2018-10-30 11:07 ` Martin Liška 2018-10-30 11:21 ` Martin Liška 3 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2018-10-30 11:07 UTC (permalink / raw) To: gcc-patches Cc: Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor, Marek Polacek On 10/29/18 12:04 PM, Martin LiÅ¡ka wrote: > 3) lookup_template_class_1 > > $ ./xg++ -B. /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C -c -fchecking=3 > hash table checking failed: equal operator returns true for a pair of values with a different hash value/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C: In instantiation of âstruct B<A>â: > /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:15:8: required from here > /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:8:17: internal compiler error: in find_slot_with_hash, at hash-table.h:905 > 8 | friend bool foo (const B<Q>& a); > | ^~~ > 0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option) > /home/marxin/Programming/gcc/gcc/hash-table.h:905 > 0xa042ce lookup_template_class_1 > /home/marxin/Programming/gcc/gcc/cp/pt.c:9629 > 0xa042ce lookup_template_class(tree_node*, tree_node*, tree_node*, tree_node*, int, int) > /home/marxin/Programming/gcc/gcc/cp/pt.c:9674 > 0xa03670 tsubst_aggr_type > /home/marxin/Programming/gcc/gcc/cp/pt.c:12679 > 0x9fefcd tsubst(tree_node*, tree_node*, int, tree_node*) > /home/marxin/Programming/gcc/gcc/cp/pt.c:14294 > 0x9fe1a9 tsubst(tree_node*, tree_node*, int, tree_node*) > /home/marxin/Programming/gcc/gcc/cp/pt.c:14285 > 0xa0d8bd tsubst_arg_types > /home/marxin/Programming/gcc/gcc/cp/pt.c:13891 > 0xa0dc24 tsubst_function_type > /home/marxin/Programming/gcc/gcc/cp/pt.c:14032 > 0x9fe790 tsubst(tree_node*, tree_node*, int, tree_node*) > /home/marxin/Programming/gcc/gcc/cp/pt.c:14769 > 0x9f2c7c tsubst_function_decl > /home/marxin/Programming/gcc/gcc/cp/pt.c:12921 > 0xa02d27 tsubst_template_decl > /home/marxin/Programming/gcc/gcc/cp/pt.c:13214 > 0x9f4416 tsubst_decl > /home/marxin/Programming/gcc/gcc/cp/pt.c:13316 > 0x9ff0ca tsubst(tree_node*, tree_node*, int, tree_node*) > /home/marxin/Programming/gcc/gcc/cp/pt.c:14212 > 0xa1dfd0 tsubst_friend_function > /home/marxin/Programming/gcc/gcc/cp/pt.c:10310 > 0xa1dfd0 instantiate_class_template_1 > /home/marxin/Programming/gcc/gcc/cp/pt.c:11359 > 0xa1dfd0 instantiate_class_template(tree_node*) > /home/marxin/Programming/gcc/gcc/cp/pt.c:11424 > 0xa66b22 complete_type(tree_node*) > /home/marxin/Programming/gcc/gcc/cp/typeck.c:138 > 0x9023c7 start_decl_1(tree_node*, bool) > /home/marxin/Programming/gcc/gcc/cp/decl.c:5278 > 0x92a15f start_decl(cp_declarator const*, cp_decl_specifier_seq*, int, tree_node*, tree_node*, tree_node**) > /home/marxin/Programming/gcc/gcc/cp/decl.c:5241 > 0x9c1944 cp_parser_init_declarator > /home/marxin/Programming/gcc/gcc/cp/parser.c:19750 This one is about inconsistency between: /* Returns a hash for a template TMPL and template arguments ARGS. */ static hashval_t hash_tmpl_and_args (tree tmpl, tree args) { hashval_t val = iterative_hash_object (DECL_UID (tmpl), 0); return iterative_hash_template_arg (args, val); } iterative_hash_template_arg is problematic, differs from: bool spec_hasher::equal (spec_entry *e1, spec_entry *e2) { int equal; ++comparing_specializations; equal = (e1->tmpl == e2->tmpl && comp_template_args (e1->args, e2->args)); ... where comp_template_args (e1->args, e2->args) returns true, but iterative_hash_template_arg values are different. Can please some C++ maintainer take a look? Thanks, Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: hash-table violation in gcc/cp/pt.c 2018-10-30 11:07 ` hash-table violation in gcc/cp/pt.c Martin Liška @ 2018-10-30 11:21 ` Martin Liška 2018-11-01 12:06 ` Martin Liška 0 siblings, 1 reply; 53+ messages in thread From: Martin Liška @ 2018-10-30 11:21 UTC (permalink / raw) To: gcc-patches Cc: Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor, Marek Polacek On 10/30/18 11:25 AM, Martin LiÅ¡ka wrote: > On 10/29/18 12:04 PM, Martin LiÅ¡ka wrote: >> 3) lookup_template_class_1 >> >> $ ./xg++ -B. /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C -c -fchecking=3 >> hash table checking failed: equal operator returns true for a pair of values with a different hash value/home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C: In instantiation of âstruct B<A>â: >> /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:15:8: required from here >> /home/marxin/Programming/gcc/gcc/testsuite/g++.dg/template/ttp23.C:8:17: internal compiler error: in find_slot_with_hash, at hash-table.h:905 >> 8 | friend bool foo (const B<Q>& a); >> | ^~~ >> 0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option) >> /home/marxin/Programming/gcc/gcc/hash-table.h:905 >> 0xa042ce lookup_template_class_1 >> /home/marxin/Programming/gcc/gcc/cp/pt.c:9629 >> 0xa042ce lookup_template_class(tree_node*, tree_node*, tree_node*, tree_node*, int, int) >> /home/marxin/Programming/gcc/gcc/cp/pt.c:9674 >> 0xa03670 tsubst_aggr_type >> /home/marxin/Programming/gcc/gcc/cp/pt.c:12679 >> 0x9fefcd tsubst(tree_node*, tree_node*, int, tree_node*) >> /home/marxin/Programming/gcc/gcc/cp/pt.c:14294 >> 0x9fe1a9 tsubst(tree_node*, tree_node*, int, tree_node*) >> /home/marxin/Programming/gcc/gcc/cp/pt.c:14285 >> 0xa0d8bd tsubst_arg_types >> /home/marxin/Programming/gcc/gcc/cp/pt.c:13891 >> 0xa0dc24 tsubst_function_type >> /home/marxin/Programming/gcc/gcc/cp/pt.c:14032 >> 0x9fe790 tsubst(tree_node*, tree_node*, int, tree_node*) >> /home/marxin/Programming/gcc/gcc/cp/pt.c:14769 >> 0x9f2c7c tsubst_function_decl >> /home/marxin/Programming/gcc/gcc/cp/pt.c:12921 >> 0xa02d27 tsubst_template_decl >> /home/marxin/Programming/gcc/gcc/cp/pt.c:13214 >> 0x9f4416 tsubst_decl >> /home/marxin/Programming/gcc/gcc/cp/pt.c:13316 >> 0x9ff0ca tsubst(tree_node*, tree_node*, int, tree_node*) >> /home/marxin/Programming/gcc/gcc/cp/pt.c:14212 >> 0xa1dfd0 tsubst_friend_function >> /home/marxin/Programming/gcc/gcc/cp/pt.c:10310 >> 0xa1dfd0 instantiate_class_template_1 >> /home/marxin/Programming/gcc/gcc/cp/pt.c:11359 >> 0xa1dfd0 instantiate_class_template(tree_node*) >> /home/marxin/Programming/gcc/gcc/cp/pt.c:11424 >> 0xa66b22 complete_type(tree_node*) >> /home/marxin/Programming/gcc/gcc/cp/typeck.c:138 >> 0x9023c7 start_decl_1(tree_node*, bool) >> /home/marxin/Programming/gcc/gcc/cp/decl.c:5278 >> 0x92a15f start_decl(cp_declarator const*, cp_decl_specifier_seq*, int, tree_node*, tree_node*, tree_node**) >> /home/marxin/Programming/gcc/gcc/cp/decl.c:5241 >> 0x9c1944 cp_parser_init_declarator >> /home/marxin/Programming/gcc/gcc/cp/parser.c:19750 > > This one is about inconsistency between: > > /* Returns a hash for a template TMPL and template arguments ARGS. */ > > static hashval_t > hash_tmpl_and_args (tree tmpl, tree args) > { > hashval_t val = iterative_hash_object (DECL_UID (tmpl), 0); > return iterative_hash_template_arg (args, val); > } > > iterative_hash_template_arg is problematic, differs from: > bool > spec_hasher::equal (spec_entry *e1, spec_entry *e2) > { > int equal; > > ++comparing_specializations; > equal = (e1->tmpl == e2->tmpl > && comp_template_args (e1->args, e2->args)); > ... > > where comp_template_args (e1->args, e2->args) returns true, but > iterative_hash_template_arg values are different. > > Can please some C++ maintainer take a look? > > Thanks, > Martin > The same spec_hasher type is also involved in 4) 0xa265a4 hash_table<spec_hasher, xcallocator>::find_slot_with_hash(spec_entry* const&, unsigned int, insert_option) /home/marxin/Programming/gcc/gcc/hash-table.h:905 0x9e35e6 register_specialization /home/marxin/Programming/gcc/gcc/cp/pt.c:1534 0xa22ac3 check_explicit_specialization(tree_node*, tree_node*, int, int, tree_node*) /home/marxin/Programming/gcc/gcc/cp/pt.c:3243 0x91552d grokfndecl /home/marxin/Programming/gcc/gcc/cp/decl.c:9106 ... Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: hash-table violation in gcc/cp/pt.c 2018-10-30 11:21 ` Martin Liška @ 2018-11-01 12:06 ` Martin Liška 0 siblings, 0 replies; 53+ messages in thread From: Martin Liška @ 2018-11-01 12:06 UTC (permalink / raw) To: gcc-patches Cc: Nathan Sidwell, Jason Merrill, Jakub Jelinek, Paul Richard Thomas, Martin Jambor, Marek Polacek I created PR87847 for that. Martin ^ permalink raw reply [flat|nested] 53+ messages in thread
end of thread, other threads:[~2019-06-25 14:23 UTC | newest] Thread overview: 53+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-29 12:02 [PATCH][RFC] Sanitize equals and hash functions in hash-tables Martin Liška 2018-10-29 14:28 ` Alexander Monakov 2018-10-29 15:56 ` Martin Liška 2018-10-30 10:32 ` Jakub Jelinek 2018-10-30 14:17 ` Martin Liška 2018-11-07 22:24 ` Jeff Law 2018-11-07 22:44 ` Jakub Jelinek 2018-11-08 8:56 ` Martin Liška 2019-05-13 7:42 ` Martin Liška 2019-05-20 17:26 ` Jason Merrill 2019-05-20 22:07 ` Jeff Law 2019-05-21 9:38 ` Richard Biener 2019-05-21 11:02 ` Martin Liška 2019-05-21 11:52 ` Richard Biener 2019-05-22 9:13 ` Martin Liška 2019-05-31 13:23 ` Richard Biener 2019-05-31 13:35 ` Martin Liška 2019-05-31 22:10 ` Jeff Law 2019-06-03 13:35 ` Martin Liška 2019-06-07 8:57 ` Richard Biener 2019-06-07 12:04 ` Martin Liška 2019-06-07 12:09 ` Richard Biener 2019-06-07 12:13 ` Martin Liška 2019-06-07 14:48 ` Martin Sebor 2019-06-07 21:43 ` Jason Merrill 2019-06-10 7:08 ` Martin Liška 2019-06-10 18:22 ` Jason Merrill 2019-06-11 7:41 ` Martin Liška 2019-06-11 12:28 ` Jason Merrill 2019-06-11 13:16 ` Martin Liška 2019-06-11 19:02 ` Jason Merrill 2019-06-12 7:59 ` Richard Biener 2019-06-12 8:02 ` Martin Liška 2019-06-12 9:15 ` Martin Liška 2019-06-12 9:41 ` Richard Biener 2019-06-12 11:45 ` Martin Liška 2019-06-12 12:50 ` Richard Biener 2019-06-12 13:05 ` Martin Liška 2019-06-23 23:08 ` Ian Lance Taylor 2019-06-24 12:29 ` Richard Biener 2019-06-24 13:51 ` Martin Liška 2019-06-24 14:10 ` Richard Biener 2019-06-25 10:25 ` Martin Liška 2019-06-25 11:59 ` Martin Liška 2019-06-25 14:23 ` Richard Biener 2018-10-30 10:25 ` hash-table violation in cselib.c Martin Liška 2018-11-01 11:57 ` Martin Liška 2018-10-30 10:46 ` hash-table violation in gcc/fortran/trans-decl.c Martin Liška 2018-10-31 10:00 ` Trevor Saunders 2018-10-31 10:18 ` Martin Liška 2018-10-30 11:07 ` hash-table violation in gcc/cp/pt.c Martin Liška 2018-10-30 11:21 ` Martin Liška 2018-11-01 12:06 ` Martin Liška
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).