False ODR violation positives on anonymous namespace types

[Jan Hubicka <hubicka@ucw.cz>]

Jason,
I got my firefox tree building again and noticed that my patch to enable
ODR merging on non-class types caused false positives:

/aux/hubicka/trunk-install/include/c++/6.0.0/ext/new_allocator.h:66:26: warning: type ‘(anonymous namespace)::ObservationWithStack const&’ violates one definition rule [-Wodr]
       typedef const _Tp& const_reference;
                          ^
/aux/hubicka/trunk-install/include/c++/6.0.0/ext/alloc_traits.h:110:53: note: it is defined as a pointer to different type in another translation unit
     typedef const value_type&                       const_reference;
                                                     ^
/aux/hubicka/firefox8/xpcom/build/MainThreadIOLogger.cpp:28:8: note: type ‘const struct ObservationWithStack’ should match type ‘const struct value_type’
 struct ObservationWithStack
        ^
/aux/hubicka/trunk-install/include/c++/6.0.0/ext/alloc_traits.h:103:53: note: the incompatible type is defined here
     typedef typename _Base_type::value_type         value_type;
                                                     ^

Here the obvious problem is that we try to merge type in anonymous namespace.
This is because at LTO time odr_type_p returns true and
type_in_anonymous_namespace returns false.  odr_type_p basically checks that we
computed mangled type name for it which is done in free_lang_data as follows:

{
  /* We use DECL_ASSEMBLER_NAME to hold mangled type names for One Definition Rule
     merging.  */
  if (flag_lto_odr_type_mering
      && TREE_CODE (decl) == TYPE_DECL
      && DECL_NAME (decl)
      && decl == TYPE_NAME (TREE_TYPE (decl))
      && !is_lang_specific (TREE_TYPE (decl))
      /* Save some work. Names of builtin types are always derived from
         properties of its main variant.  A special case are integer types
         where mangling do make differences between char/signed char/unsigned
         char etc.  Storing name for these makes e.g.
         -fno-signed-char/-fsigned-char mismatches to be handled well.

         See cp/mangle.c:write_builtin_type for details.  */
      && (TREE_CODE (TREE_TYPE (decl)) != VOID_TYPE
          && TREE_CODE (TREE_TYPE (decl)) != BOOLEAN_TYPE
          && TREE_CODE (TREE_TYPE (decl)) != REAL_TYPE
          && TREE_CODE (TREE_TYPE (decl)) != FIXED_POINT_TYPE)
      && !TYPE_ARTIFICIAL (TREE_TYPE (decl))
      && !variably_modified_type_p (TREE_TYPE (decl), NULL_TREE)
      && !type_in_anonymous_namespace_p (TREE_TYPE (decl)))
    return !DECL_ASSEMBLER_NAME_SET_P (decl);

and obviously we are not intended to mangle tyhis type and it should be
caught by type_in_anonymous_namespace_p check:

bool
type_in_anonymous_namespace_p (const_tree t)
{
  /* TREE_PUBLIC of TYPE_STUB_DECL may not be properly set for
     bulitin types; those have CONTEXT NULL.  */
  if (!TYPE_CONTEXT (t))
    return false;
  return (TYPE_STUB_DECL (t) && !TREE_PUBLIC (TYPE_STUB_DECL (t)));
}

We already discussed earlier that type_in_anonymous_namespace_p is not working
on compund types, because these do not have TYPE_STUB_DECL.  I tought those are
!TYPE_NAME types.  What is reason for !TYPE_NAME type with no TYPE_STUB_DECL?
Is it always a compound type with typedef name?

One possible solution is to avoid producing mangled names so these won't
be odr_type_p at LTO time:

Index: tree.c
===================================================================
--- tree.c      (revision 222991)
+++ tree.c      (working copy)
@@ -5173,6 +5173,7 @@ need_assembler_name_p (tree decl)
          && TREE_CODE (TREE_TYPE (decl)) != REAL_TYPE
          && TREE_CODE (TREE_TYPE (decl)) != FIXED_POINT_TYPE)
       && !TYPE_ARTIFICIAL (TREE_TYPE (decl))
+      && TYPE_STUB_DECL (TREE_TYPE (decl))
       && !variably_modified_type_p (TREE_TYPE (decl), NULL_TREE)
       && !type_in_anonymous_namespace_p (TREE_TYPE (decl)))
     return !DECL_ASSEMBLER_NAME_SET_P (decl);

which silence the warnings, but I think it also may cause false negatives.
Ohter option is to make type_in_anonymous_namespace_p work.  The types in
question are anonymous because they are defined within anonymous template,
would the following work in type_in_anonymous_namespace_p?

+  /* Types (such as pointer_type) defined within class types may not
+     have their own TYPE_STUB_DECL.  Look for the outer type.  */
+  while (!TYPE_STUB_DECL (t) && TYPE_NAME (t)
+        && TREE_CODE (TYPE_NAME (t)) == TYPE_DECL
+        && DECL_CONTEXT (TYPE_NAME (t))
+        && TYPE_P (DECL_CONTEXT (TYPE_NAME (t))))
+    t = DECL_CONTEXT (TYPE_NAME (t));

Last option would be make anonymous_namespace_p to walk to compound type bases.
I made variant of this in
https://gcc.gnu.org/ml/gcc-patches/2015-05/msg00880.html
(odr_or_derived_type_p, modifying type_in_anonymous_namespace_p to work same
way is definitly easy)

What would be a preffered fix for this?

Honza