GCC usage script

GCC usage script

[Martin Liška]

Hey.

Some of you may know Honza's images about CPU utilization and memory usage
when using LTO [1]. Last week I played with Chromium and Firefox and I wanted
to visualize their utilization. That's why I came up with a new script [2].

You can easily wrap a command and the script generates graphs for you:.
The script uses modern psutil library and plots through a great matplotlib:
$ usage-wrapper.py -v -t postgresql 'make -j16'
...
$ eog usage.svg

There's a gallery of the collected reports for Firefox and Chromium:
https://gist.github.com/marxin/223890df4d8d8e490b6b2918b77dacad

and yes, we have a LTO regression since GCC 9 that I'm going to take closer to.

Martin

[1] https://4.bp.blogspot.com/-fvrkYSMBqug/XMsTqg4HEkI/AAAAAAAAGl8/8sp1GWv6Oe8tfBfL6aO8Nbq5j3hExurpwCEwYBhgL/s1600/llvm.png
[2] https://github.com/marxin/script-misc/blob/master/usage-wrapper.py

Re: GCC usage script

[Jan Hubicka]

> Hey.
> 
> Some of you may know Honza's images about CPU utilization and memory usage
> when using LTO [1]. Last week I played with Chromium and Firefox and I wanted
> to visualize their utilization. That's why I came up with a new script [2].
> 
> You can easily wrap a command and the script generates graphs for you:.
> The script uses modern psutil library and plots through a great matplotlib:
> $ usage-wrapper.py -v -t postgresql 'make -j16'
> ...
> $ eog usage.svg
> 
> There's a gallery of the collected reports for Firefox and Chromium:
> https://gist.github.com/marxin/223890df4d8d8e490b6b2918b77dacad

Thanks, it looks nice.
I am not sure what the blue line means - if it is overall usage of
vmstat then I guess you do not calculate COW of the stream out processes
correctly and that is why you get so high peak on them.
> 
> and yes, we have a LTO regression since GCC 9 that I'm going to take closer to.

Where do you see the regression? (just trying to make sense of the
numbers).
For Firefox I see you get around 16GB streaming memory peak and 32GB for
ltranses which seems kind of acceptable for such a large parallelism.

Do you have time report for Cromium WPA? It seems to be stuck on
something for quite a while.
I would say that simple time report + ideally perf profile should make
it possible to track this down.

Honza
> 
> Martin
> 
> [1] https://4.bp.blogspot.com/-fvrkYSMBqug/XMsTqg4HEkI/AAAAAAAAGl8/8sp1GWv6Oe8tfBfL6aO8Nbq5j3hExurpwCEwYBhgL/s1600/llvm.png
> [2] https://github.com/marxin/script-misc/blob/master/usage-wrapper.py

Re: GCC usage script

[Martin Liška]

On 7/7/20 12:27 PM, Jan Hubicka wrote:
>> Hey.
>>
>> Some of you may know Honza's images about CPU utilization and memory usage
>> when using LTO [1]. Last week I played with Chromium and Firefox and I wanted
>> to visualize their utilization. That's why I came up with a new script [2].
>>
>> You can easily wrap a command and the script generates graphs for you:.
>> The script uses modern psutil library and plots through a great matplotlib:
>> $ usage-wrapper.py -v -t postgresql 'make -j16'
>> ...
>> $ eog usage.svg
>>
>> There's a gallery of the collected reports for Firefox and Chromium:
>> https://gist.github.com/marxin/223890df4d8d8e490b6b2918b77dacad
> 
> Thanks, it looks nice.

Thanks.

> I am not sure what the blue line means - if it is overall usage of
> vmstat then I guess you do not calculate COW of the stream out processes
> correctly and that is why you get so high peak on them.

I use:
rss: aka “Resident Set Size”, this is the non-swapped physical memory a process has used. On UNIX it matches “top“‘s RES column).

So I guess it's not computed correctly, but it's likely hard to identify memory
that is actually shared in between processes (and doesn't need copying).

>>
>> and yes, we have a LTO regression since GCC 9 that I'm going to take closer to.

I basically speak about Firefox WPA (which is about 2-3x slower with GCC 9).
For chromium, the situation is even worse (WPA is 7x slower).

> 
> Where do you see the regression? (just trying to make sense of the
> numbers).
> For Firefox I see you get around 16GB streaming memory peak and 32GB for
> ltranses which seems kind of acceptable for such a large parallelism.
> 
> Do you have time report for Cromium WPA?

No.

> It seems to be stuck on
> something for quite a while.

Yep.

Martin

> I would say that simple time report + ideally perf profile should make
> it possible to track this down.
> 
> Honza
>>
>> Martin
>>
>> [1] https://4.bp.blogspot.com/-fvrkYSMBqug/XMsTqg4HEkI/AAAAAAAAGl8/8sp1GWv6Oe8tfBfL6aO8Nbq5j3hExurpwCEwYBhgL/s1600/llvm.png
>> [2] https://github.com/marxin/script-misc/blob/master/usage-wrapper.py