public inbox for
 help / color / mirror / Atom feed
From: "Ondřej Bílka" <>
To: Carlos O'Donell <>
Cc: Will Newton <>,
	"" <>,
	Patch Tracking <>,
	Siddhesh Poyarekar <>
Subject: Re: [PATCH] sysdeps/arm/armv7/multiarch/memcpy_impl.S: Improve performance.
Date: Tue, 03 Sep 2013 18:57:00 -0000	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On Tue, Sep 03, 2013 at 01:52:34PM -0400, Carlos O'Donell wrote:
> On 09/03/2013 01:37 PM, Ondřej Bílka wrote:
> >> We have one, it's the glibc microbenchmark, and we want to expand it,
> >> otherwise when ACME comes with their patch for ARM and breaks performance
> >> for targets that Linaro cares about I have no way to reject the patch
> >> objectively :-)
> >>
> > Carlos, you are asking for impossible. When you publish benchmark people
> > will try to maximize benchmark number. After certain point this becomes
> > possible only by employing shady accounting: Move part of time to place
> > wehre it will not be measured by benchmark (for example by having
> > function that is 4kb large, on benchmarks it will fit into instruction
> > cache but that does not happen in reality). 
> What is it that I'm asking that is impossible?
Having static set of benchmarks that can say if implementation is

We are shooting to moving target, architectures change and as what we write
will code that will come to users with considerable delay and factors we 
used for decision will change in meantime.

Once implementation reach certain quality question what is better
becomes dependent on program used. Until we could decide from profile
feedback we will lose some percents by having to use single
> > Taking care of common factors that can cause that is about ten times
> > more complex than whole system benchmarking, analysis will be quite
> > difficult as you will get twenty numbers and you will need to decide
> > which ones could made real impact and which wont.
> Sorry, could you clarify this a bit more, exactly what is ten times
> more complex?
Having benchmark suite that will catch all relevant factors that can
affect performance. Some are hard to qualify for them we need to know
how average program stresses resources.

Take instruction cache usage, a function will occupy cache lines and we
can accurately measure probability and cost of cache misses inside
function. What is hard to estimate is how this will affect rest of
program. For this we would need to know average probability that cache
line will be referenced in future.
> If we have N tests and they produce N numbers, for a given target,
> for a given device, for a given workload, there is a set of importance
> weights on N that should give you some kind of relevance.
You are jumping to case when we will have these weights. Problematic
part is getting those.

> We should be able to come up with some kind of framework from which
> we can clearly say "this patch is better than this other patch", even
> if not automated, it should be possible to reason from the results,
> and that reasoning recorded as a discussion on this list.
What is possible is to say that some patch is significantly worse based
on some criteria. There is lot of gray area where decision is unclear.

> >>> The key advantage of the cortex-strings framework is that it allows
> >>> graphing the results of benchmarks. Often changes to string function
> >>> performance can only really be analysed graphically as otherwise you
> >>> end up with a huge soup of numbers, some going up, some going down and
> >>> it is very hard to separate the signal from the noise.
> >>
> >> I disagree strongly. You *must* come up with a measurable answer and
> >> looking at a graph is never a solution I'm going to accept.
> >>
> > You can have that opinion.
> > Looking at performance graphs is most powerful technique how to
> > understand performance. I got most of my improvements from analyzing
> > these.
> That is a different use for the graphs. I do not disagree that graphing
> is a *powerful* way to display information and using that information to
> produce a new routine is useful. What I disagree with is using such graphs
> to argue qualitatively that your patch is better than the existing
> implementation.
> There is always a quantitative way to say X is better than Y, but it
> requires breaking down your expectations and documenting them e.g.
> should be faster with X alignment on sizes from N bytes to M bytes, and
> then ranking based on those criteria.
> >> You need to statistically analyze the numbers, assign weights to ranges,
> >> and come up with some kind of number that evaluates the results based
> >> on *some* formula. That is the only way we are going to keep moving
> >> performance forward (against some kind of criteria).
> >>
> > These accurate assigning weigths is best done by taking program running
> > it and measuring time. Without taking this into account weigths will not
> > tell much, as you will likely just optimize cold code at expense of hot
> > code.
> I don't disagree with you here.
> Cheers,
> Carlos.

  reply	other threads:[~2013-09-03 18:57 UTC|newest]

Thread overview: 37+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-08-12  7:55 Will Newton
2013-08-27  7:46 ` Will Newton
2013-08-30 17:14   ` Carlos O'Donell
2013-08-30 18:48     ` Will Newton
2013-08-30 19:26       ` Carlos O'Donell
2013-09-02 14:18         ` Will Newton
2013-09-03 16:14           ` Carlos O'Donell
     [not found]         ` <>
2013-09-02 14:18           ` benchmark improvements (Was: Re: [PATCH] sysdeps/arm/armv7/multiarch/memcpy_impl.S: Improve performance.) Siddhesh Poyarekar
2013-09-03 13:46             ` Will Newton
2013-09-03 17:48               ` Ondřej Bílka
2013-09-02 19:57           ` [PATCH] sysdeps/arm/armv7/multiarch/memcpy_impl.S: Improve performance Ondřej Bílka
2013-09-03 16:18           ` Carlos O'Donell
2013-09-03 17:37             ` Ondřej Bílka
2013-09-03 17:52               ` Carlos O'Donell
2013-09-03 18:57                 ` Ondřej Bílka [this message]
2013-09-03 19:15                   ` Carlos O'Donell
2013-09-04  7:27                     ` Siddhesh Poyarekar
2013-09-04 11:03                       ` Ondřej Bílka
2013-09-04 11:43                         ` Siddhesh Poyarekar
2013-09-04 17:37                         ` Ryan S. Arnold
2013-09-05  8:04                           ` Ondřej Bílka
2013-09-04 15:30                       ` Carlos O'Donell
2013-09-04 17:35                       ` Ryan S. Arnold
2013-09-05 11:07                         ` Ondřej Bílka
2013-09-05 11:54                         ` Joseph S. Myers
2013-09-03 19:34               ` Ryan S. Arnold
2013-09-07 11:55                 ` Ondřej Bílka
2013-09-03 19:31             ` Ryan S. Arnold
2013-09-03 19:54               ` Carlos O'Donell
2013-09-03 20:56                 ` Ryan S. Arnold
2013-09-03 23:29                   ` Ondřej Bílka
2013-09-03 23:31                   ` Carlos O'Donell
2013-09-03 22:27               ` Ondřej Bílka
2013-08-29 23:58 ` Joseph S. Myers
2013-08-30 14:56   ` Will Newton
2013-08-30 15:18     ` Joseph S. Myers
2013-08-30 18:46       ` Will Newton

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).