[PATCH] Limit perf data buffer during profiling

[PATCH] Limit perf data buffer during profiling

[Andi Kleen]

From: Andi Kleen <ak@linux.intel.com>

With high -j parallelism the autofdo tests can randomly fail.
autofdo uses Linux perf to record profiling data.
Linux perf uses a locked perf buffer. By default it has
around 516k buffer per uid (/proc/sys/kernel/perf_event_mlock_kb).

An individual perf record tries to grab the full 516k,
which makes parallel perf record fail.

This patch limits the perf buffer for individual perf record to 8k.
With the default settings this allows a parallelism of the test
cases of 16, which is hopefully good enough

(if not would need to add some kind of semaphore, or ask
the user to increase the limit as root)

I also removed an unneeded -o perf.data option

Thanks to Marcin to finally spotting the problem.

Passes bootstrap and test on x86_64-linux. Ok for trunk?

gcc/testsuite/:

2017-05-12  Andi Kleen  <ak@linux.intel.com>

	PR testsuite/77684
	* lib/target-supports.exp (profopt-perf-wrapper): Remove
	-p perf.data option. Add -m8 option.
---
 gcc/testsuite/lib/target-supports.exp | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/gcc/testsuite/lib/target-supports.exp b/gcc/testsuite/lib/target-supports.exp
index 83e7f2670e6..a22657767db 100644
--- a/gcc/testsuite/lib/target-supports.exp
+++ b/gcc/testsuite/lib/target-supports.exp
@@ -522,9 +522,16 @@ proc check_effective_target_keeps_null_pointer_checks { } {
 
 # Return the autofdo profile wrapper
 
+# Linux by default allows 516KB of perf event buffers
+# in /proc/sys/kernel/perfe_event_mlock_kb
+# Each individual perf tries to grab it
+# This causes problems with parallel test suite runs. Instead
+# limit us to 8 pages (32K), which should be good enough
+# for the small test programs. With the default settings 
+# this allows parallelism of 16 and higher of parallel gcc-auto-profile
 proc profopt-perf-wrapper { } {
     global srcdir
-    return "$srcdir/../config/i386/gcc-auto-profile -o perf.data "
+    return "$srcdir/../config/i386/gcc-auto-profile -m8 "
 }
 
 # Return true if profiling is supported on the target.
-- 
2.12.2

Re: [PATCH] Limit perf data buffer during profiling

[Richard Biener]

On Fri, May 12, 2017 at 11:38 AM, Andi Kleen <andi@firstfloor.org> wrote:
> From: Andi Kleen <ak@linux.intel.com>
>
> With high -j parallelism the autofdo tests can randomly fail.
> autofdo uses Linux perf to record profiling data.
> Linux perf uses a locked perf buffer. By default it has
> around 516k buffer per uid (/proc/sys/kernel/perf_event_mlock_kb).
>
> An individual perf record tries to grab the full 516k,
> which makes parallel perf record fail.
>
> This patch limits the perf buffer for individual perf record to 8k.
> With the default settings this allows a parallelism of the test
> cases of 16, which is hopefully good enough
>
> (if not would need to add some kind of semaphore, or ask
> the user to increase the limit as root)
>
> I also removed an unneeded -o perf.data option
>
> Thanks to Marcin to finally spotting the problem.
>
> Passes bootstrap and test on x86_64-linux. Ok for trunk?

Ok.  Can you retain -o perf.data (even if that's the default)?

Thanks,
Richard.

> gcc/testsuite/:
>
> 2017-05-12  Andi Kleen  <ak@linux.intel.com>
>
>         PR testsuite/77684
>         * lib/target-supports.exp (profopt-perf-wrapper): Remove
>         -p perf.data option. Add -m8 option.
> ---
>  gcc/testsuite/lib/target-supports.exp | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/gcc/testsuite/lib/target-supports.exp b/gcc/testsuite/lib/target-supports.exp
> index 83e7f2670e6..a22657767db 100644
> --- a/gcc/testsuite/lib/target-supports.exp
> +++ b/gcc/testsuite/lib/target-supports.exp
> @@ -522,9 +522,16 @@ proc check_effective_target_keeps_null_pointer_checks { } {
>
>  # Return the autofdo profile wrapper
>
> +# Linux by default allows 516KB of perf event buffers
> +# in /proc/sys/kernel/perfe_event_mlock_kb
> +# Each individual perf tries to grab it
> +# This causes problems with parallel test suite runs. Instead
> +# limit us to 8 pages (32K), which should be good enough
> +# for the small test programs. With the default settings
> +# this allows parallelism of 16 and higher of parallel gcc-auto-profile
>  proc profopt-perf-wrapper { } {
>      global srcdir
> -    return "$srcdir/../config/i386/gcc-auto-profile -o perf.data "
> +    return "$srcdir/../config/i386/gcc-auto-profile -m8 "
>  }
>
>  # Return true if profiling is supported on the target.
> --
> 2.12.2
>

Re: [PATCH] Limit perf data buffer during profiling

[Jeff Law]

On 05/12/2017 03:38 AM, Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> With high -j parallelism the autofdo tests can randomly fail.
> autofdo uses Linux perf to record profiling data.
> Linux perf uses a locked perf buffer. By default it has
> around 516k buffer per uid (/proc/sys/kernel/perf_event_mlock_kb).
> 
> An individual perf record tries to grab the full 516k,
> which makes parallel perf record fail.
> 
> This patch limits the perf buffer for individual perf record to 8k.
> With the default settings this allows a parallelism of the test
> cases of 16, which is hopefully good enough
I'm routinely building & testing with -j56 these days.  We'll see if 
it's sufficient.

> 
> (if not would need to add some kind of semaphore, or ask
> the user to increase the limit as root)
> 
> I also removed an unneeded -o perf.data option
> 
> Thanks to Marcin to finally spotting the problem.
> 
> Passes bootstrap and test on x86_64-linux. Ok for trunk?
> 
> gcc/testsuite/:
> 
> 2017-05-12  Andi Kleen  <ak@linux.intel.com>
> 
> 	PR testsuite/77684
> 	* lib/target-supports.exp (profopt-perf-wrapper): Remove
> 	-p perf.data option. Add -m8 option.
OK.  But be aware that it may not ultimately be sufficient.

jeff

Re: [PATCH] Limit perf data buffer during profiling

[Aldy Hernandez]

Andi Kleen <andi@firstfloor.org> writes:

> From: Andi Kleen <ak@linux.intel.com>
>
> With high -j parallelism the autofdo tests can randomly fail.
> autofdo uses Linux perf to record profiling data.
> Linux perf uses a locked perf buffer. By default it has
> around 516k buffer per uid (/proc/sys/kernel/perf_event_mlock_kb).
>
> An individual perf record tries to grab the full 516k,
> which makes parallel perf record fail.
>
> This patch limits the perf buffer for individual perf record to 8k.
> With the default settings this allows a parallelism of the test
> cases of 16, which is hopefully good enough

So for -jN > 16 it would silently fail again?

I think we should warn when the -jN is sufficiently large such that
tests will randomly fail, and perhaps suggest workarounds with
ulimit/etc.

Aldy

Re: [PATCH] Limit perf data buffer during profiling

[Richard Biener]

On Thu, May 18, 2017 at 10:35 AM, Aldy Hernandez <aldyh@redhat.com> wrote:
> Andi Kleen <andi@firstfloor.org> writes:
>
>> From: Andi Kleen <ak@linux.intel.com>
>>
>> With high -j parallelism the autofdo tests can randomly fail.
>> autofdo uses Linux perf to record profiling data.
>> Linux perf uses a locked perf buffer. By default it has
>> around 516k buffer per uid (/proc/sys/kernel/perf_event_mlock_kb).
>>
>> An individual perf record tries to grab the full 516k,
>> which makes parallel perf record fail.
>>
>> This patch limits the perf buffer for individual perf record to 8k.
>> With the default settings this allows a parallelism of the test
>> cases of 16, which is hopefully good enough
>
> So for -jN > 16 it would silently fail again?
>
> I think we should warn when the -jN is sufficiently large such that
> tests will randomly fail, and perhaps suggest workarounds with
> ulimit/etc.

given that make check parallelism is somewhat "explicit" can't we
simply arrange for the chunks to never get more than, say, 4 tree-prof.exp
testcases run at the same time?

Richard.

> Aldy

Re: [PATCH] Limit perf data buffer during profiling

[Mike Stump]

On May 18, 2017, at 1:35 AM, Aldy Hernandez <aldyh@redhat.com> wrote:
> 
> Andi Kleen <andi@firstfloor.org> writes:
> 
>> From: Andi Kleen <ak@linux.intel.com>
>> 
>> With high -j parallelism the autofdo tests can randomly fail.
>> autofdo uses Linux perf to record profiling data.
>> Linux perf uses a locked perf buffer. By default it has
>> around 516k buffer per uid (/proc/sys/kernel/perf_event_mlock_kb).
>> 
>> An individual perf record tries to grab the full 516k,
>> which makes parallel perf record fail.
>> 
>> This patch limits the perf buffer for individual perf record to 8k.
>> With the default settings this allows a parallelism of the test
>> cases of 16, which is hopefully good enough
> 
> So for -jN > 16 it would silently fail again?
> 
> I think we should warn when the -jN is sufficiently large such that
> tests will randomly fail, and perhaps suggest workarounds with
> ulimit/etc.

Not a big fan of warning.  I'd rather smell the max, and divide by n, or limit them to 4 (or 16 or any fix constant) and pass appropriate arguments or anything else that just fixes it.

Re: [PATCH] Limit perf data buffer during profiling

[Aldy Hernandez]

On 05/18/2017 12:09 PM, Mike Stump wrote:
> On May 18, 2017, at 1:35 AM, Aldy Hernandez <aldyh@redhat.com> wrote:
>>
>> Andi Kleen <andi@firstfloor.org> writes:
>>
>>> From: Andi Kleen <ak@linux.intel.com>
>>>
>>> With high -j parallelism the autofdo tests can randomly fail.
>>> autofdo uses Linux perf to record profiling data.
>>> Linux perf uses a locked perf buffer. By default it has
>>> around 516k buffer per uid (/proc/sys/kernel/perf_event_mlock_kb).
>>>
>>> An individual perf record tries to grab the full 516k,
>>> which makes parallel perf record fail.
>>>
>>> This patch limits the perf buffer for individual perf record to 8k.
>>> With the default settings this allows a parallelism of the test
>>> cases of 16, which is hopefully good enough
>>
>> So for -jN > 16 it would silently fail again?
>>
>> I think we should warn when the -jN is sufficiently large such that
>> tests will randomly fail, and perhaps suggest workarounds with
>> ulimit/etc.
> 
> Not a big fan of warning.  I'd rather smell the max, and divide by n, or limit them to 4 (or 16 or any fix constant) and pass appropriate arguments or anything else that just fixes it.
> 

You'll hear no complaints from me :).
Aldy

Re: [PATCH] Limit perf data buffer during profiling

[Jeff Law]

On 05/18/2017 02:39 AM, Richard Biener wrote:
> On Thu, May 18, 2017 at 10:35 AM, Aldy Hernandez <aldyh@redhat.com> wrote:
>> Andi Kleen <andi@firstfloor.org> writes:
>>
>>> From: Andi Kleen <ak@linux.intel.com>
>>>
>>> With high -j parallelism the autofdo tests can randomly fail.
>>> autofdo uses Linux perf to record profiling data.
>>> Linux perf uses a locked perf buffer. By default it has
>>> around 516k buffer per uid (/proc/sys/kernel/perf_event_mlock_kb).
>>>
>>> An individual perf record tries to grab the full 516k,
>>> which makes parallel perf record fail.
>>>
>>> This patch limits the perf buffer for individual perf record to 8k.
>>> With the default settings this allows a parallelism of the test
>>> cases of 16, which is hopefully good enough
>>
>> So for -jN > 16 it would silently fail again?
>>
>> I think we should warn when the -jN is sufficiently large such that
>> tests will randomly fail, and perhaps suggest workarounds with
>> ulimit/etc.
> 
> given that make check parallelism is somewhat "explicit" can't we
> simply arrange for the chunks to never get more than, say, 4 tree-prof.exp
> testcases run at the same time?
That would be significantly better than warning.

jeff