Excessive memory usage by associative arrays

Excessive memory usage by associative arrays

[Sergey Klyaus]

Hello!

OS: CentOS 7.0 with stock kernel
SystemTap: 2.4

I created script that contains plenty of associative arrays and
statistics. OOM Killer kills stapio even before module is initialized.
I've added debug print to a _stp_map_new and noticed that it
allocation is really huge:

- By default MAXMAPENTRIES is 2048
- Each string is stored statically and MAP_STRING_LENGTH is 256 (i
have 2 keys so its 256)
- Plus statistics data gives us 1088 bytes per entry
- SystemTap initializes map on per-cpu basis and do it with
for_each_possible_cpu() (128 in my case)

So this simple script:
stap -e ' global a;
     probe timer.s(1) {
             a["aaaa", "bbbb"] <<< 1;
     }
     probe timer.s(60) {
             println(@count(a["aaaa", "bbbb"]));
     }'
would eat about 300 Mb of memory.

Of course, I may reduce MAXMAPENTRIES, but I think there is something
wrong with such greedy allocation...
For the record, similiar DTrace script works perfectly (after I
removed leak of records :) ).

Thanks in advance.
BR, Sergey.

Re: Excessive memory usage by associative arrays

[Frank Ch. Eigler]

Sergey Klyaus <myautneko@gmail.com> writes:

> [...]
> - Each string is stored statically and MAP_STRING_LENGTH is 256 (i
> have 2 keys so its 256)
> - Plus statistics data gives us 1088 bytes per entry
> - SystemTap initializes map on per-cpu basis and do it with
> for_each_possible_cpu() (128 in my case)

Yes, unfortunately all those factors multiply.  You can control
MAXMAPENTRIES and MAXSTRINGLEN directly via -D options or other ways.
How many CPUs does your machine actually have?

See also
https://sourceware.org/systemtap/wiki/TipExhaustedResourceErrors 


> Of course, I may reduce MAXMAPENTRIES, but I think there is
> something wrong with such greedy allocation...

Its purpose is to ensure that no dynamic memory allocation occurs
during runtime, so is a safety measure.


> For the record, similiar DTrace script works perfectly (after I
> removed leak of records :) ).

Understood, though their situation is much simpler, and that toy
script can be made to work on stap in many ways.


- FChE

Re: Excessive memory usage by associative arrays

[Josh Stone]

On 01/19/2015 12:07 PM, Frank Ch. Eigler wrote:
> Sergey Klyaus <myautneko@gmail.com> writes:
> 
>> [...]
>> - Each string is stored statically and MAP_STRING_LENGTH is 256 (i
>> have 2 keys so its 256)
>> - Plus statistics data gives us 1088 bytes per entry
>> - SystemTap initializes map on per-cpu basis and do it with
>> for_each_possible_cpu() (128 in my case)
> 
> Yes, unfortunately all those factors multiply.  You can control
> MAXMAPENTRIES and MAXSTRINGLEN directly via -D options or other ways.
> How many CPUs does your machine actually have?
> 
> See also
> https://sourceware.org/systemtap/wiki/TipExhaustedResourceErrors 

Especially declare "global foo[SIZE]" if you know a static size limit
for a particular array.

On stats, if you're just using @count and you don't expect this value to
have highly-concurrent updates, then a simple "+=" counter will save
memory by not reserving per-cpu space.

Also if your probe has any other global writes that aren't stats, then
you'll be taking an exclusive write lock anyway, so stats don't help.
That is, a probe is only concurrent if all its globals are concurrent.

Re: Excessive memory usage by associative arrays

[Sergey Klyaus]

Hello.

> Especially declare "global foo[SIZE]" if you know a static size limit for a particular array.
Thanks for that note, I reduced per-map size, that works for me, now
script consumes only ~150 Mbytes.
For the case, here it is: https://gist.github.com/myaut/d31f6a2a6f2472c860b3

> How many CPUs does your machine actually have?
My VM has only 2 vCPUs. Entire machine has only 24 hardware threads.
I've noticed that dyninst runtime actually uses number of online CPUs
(_stp_runtime_num_contexts)

Also, I find situation when OOM need to interfere when SystemTap is
working is frightening.
STP_MEMORY is not defined by default, and after I do it, stap fails
with nicer error message:
ERROR: global variable 'a' allocation failed


BR, Sergey.

2015-01-19 23:28 GMT+03:00 Josh Stone <jistone@redhat.com>:
> On 01/19/2015 12:07 PM, Frank Ch. Eigler wrote:
>> Sergey Klyaus <myautneko@gmail.com> writes:
>>
>>> [...]
>>> - Each string is stored statically and MAP_STRING_LENGTH is 256 (i
>>> have 2 keys so its 256)
>>> - Plus statistics data gives us 1088 bytes per entry
>>> - SystemTap initializes map on per-cpu basis and do it with
>>> for_each_possible_cpu() (128 in my case)
>>
>> Yes, unfortunately all those factors multiply.  You can control
>> MAXMAPENTRIES and MAXSTRINGLEN directly via -D options or other ways.
>> How many CPUs does your machine actually have?
>>
>> See also
>> https://sourceware.org/systemtap/wiki/TipExhaustedResourceErrors
>
> Especially declare "global foo[SIZE]" if you know a static size limit
> for a particular array.
>
> On stats, if you're just using @count and you don't expect this value to
> have highly-concurrent updates, then a simple "+=" counter will save
> memory by not reserving per-cpu space.
>
> Also if your probe has any other global writes that aren't stats, then
> you'll be taking an exclusive write lock anyway, so stats don't help.
> That is, a probe is only concurrent if all its globals are concurrent.
>