[PATCH] Partial fix for asan on big endian targets (PR sanitizer/88289)

[Jakub Jelinek <jakub@redhat.com>]

On Fri, Nov 30, 2018 at 12:44:04PM +0100, Martin Liška wrote:
> Ok, I'm sending updated version of the patch. I factored out the shadow memory
> byte emission into a class, it's responsible for underlying flushing and guarantees
> that stores are 4B aligned (when beginning of stack vars is properly aligned
> to ASAN_RED_ZONE_SIZE).
> 
> So far I tested the patch on x86_64-linux-gnu and ppc64le-linux-gnu machine.

The patch broke most of the asan tests on powerpc64-linux unfortunately,
sorry for not catching it during patch review.

The following patch fixes most of them, bootstrapped/regtested on
powerpc64-linux, committed to trunk as obvious.

I'm still seeing some regressions, ICEs like:
FAIL: c-c++-common/asan/clone-test-1.c   -O1  (internal compiler error)
FAIL: c-c++-common/asan/clone-test-1.c   -O1  (test for excess errors)
Excess errors:
during RTL pass: expand
/home/jakub/gcc2/gcc/testsuite/c-c++-common/asan/clone-test-1.c:25:5: internal compiler error: in asan_clear_shadow, at asan.c:1181
0x10c52833 asan_clear_shadow
        ../../gcc/asan.c:1181
0x10c618b7 asan_emit_stack_protection(rtx_def*, rtx_def*, unsigned int, long*, tree_node**, int)
        ../../gcc/asan.c:1676
0x10621fe7 expand_used_vars
        ../../gcc/cfgexpand.c:2277
0x1062708f execute
        ../../gcc/cfgexpand.c:6338
That is
  gcc_assert ((len & 3) == 0);
To be debugged later.

2018-12-01  Jakub Jelinek  <jakub@redhat.com>

	PR sanitizer/88289
	* asan.c (asan_redzone_buffer::flush_redzone_payload): Fix up
	an off-by-one for BYTES_BIG_ENDIAN.

--- gcc/asan.c.jj	2018-11-30 19:59:59.675789930 +0100
+++ gcc/asan.c	2018-11-30 23:19:55.336780260 +0100
@@ -1326,7 +1326,7 @@ asan_redzone_buffer::flush_redzone_paylo
   for (unsigned i = 0; i < RZ_BUFFER_SIZE; i++)
     {
       unsigned char v
-	= m_shadow_bytes[BYTES_BIG_ENDIAN ? RZ_BUFFER_SIZE - i : i];
+	= m_shadow_bytes[BYTES_BIG_ENDIAN ? RZ_BUFFER_SIZE - i - 1 : i];
       val |= (unsigned HOST_WIDE_INT)v << (BITS_PER_UNIT * i);
       if (dump_file && (dump_flags & TDF_DETAILS))
 	fprintf (dump_file, "%02x ", v);


	Jakub