Re: [PATCH] AArch64: Improve A64FX memcpy

[Wilco Dijkstra <Wilco.Dijkstra@arm.com>]

Hi Naohiro,

> The following Google Sheet Graph [1] shows the most distinctive data
> without noise among the performance data I measured some times.

I do see a lot of noise (easily 10-20% from run to run), but it reduces a lot
if you execute a few iterations before the benchmark timing starts.

> - memset-default shows that the update patch performance is better
>  from 64 byte to 1024 byte, but the master performance is better from
>  1024 byte to 8192 byte. 
>
> I think we need to do a lot of trial and error to keep the same
> performance with reducing the code size.
> Do you have any idea how to further update the patch?
> I believe that memset 1024 byte to 8192 byte performance has something
> to do with removing unroll32.

I don't see unroll32 making any difference. After you unroll by about 4 times,
any loop overhead is fully hidden by the unrolled code and thus further unrolling
cannot make any difference. I do sometimes see a small difference between 4x
and 8x unrolling - for smaller sizes up to about 4KB using unroll of 4x is faster,
but for 32-64KB unroll of 8x wins by 5-10%. I've left it at 8x for now.

The difference is due to handling of the last 512 bytes - for copies around 1KB
it can be a measurable overhead to always do an extra 512 bytes from the end.
So I've added some branches to do this in smaller chunks, but this also adds
extra branch mispredictions if the size varies from call to call (which is what
these microbenchmarks don't simulate).

I also fixed the zero size issue with memset - it's almost twice as fast to zero
memory with DC ZVA, so that is used when possible for large copies.
For non-zero copies it turns out to be a bad idea to use DC ZVA plus a second
store. The unroll8 loop is about 10% faster, so the new version just uses that.

Cheers,
Wilco