Re: Optimising cygwin fork performance

[Hamish McIntyre-Bhatty <hamishmb@live.co.uk>]

[-- Attachment #1.1.1: Type: text/plain, Size: 4139 bytes --]

On 16/12/2020 20:37, Brian Inglis wrote:
> On 2020-12-16 10:36, Marco Atzeri via Cygwin-apps wrote:
>> On 16.12.2020 13:13, Hamish McIntyre-Bhatty via Cygwin-apps wrote:
>>> So I know it's been mentioned a lot that fork is slow on Cygwin, but
>>> compared to other people's machines, eg when building, it seems way
>>> slower for me.
>>>
>>> First I'd like to know if there's a good way to measure this that
>>> anyone
>>> has found, because I'm not sure how to measure it. If I print multiple
>>> lines with echo in a script, I can see it printing maybe 2-3 a second -
>>> it's very slow.
>>>
>>> I think this might be because I'm using a Virtual Machine with
>>> VirtualBox, and QEMU/KVM might be quicker. I'm using Avira Antivurus,
>>> with exceptions for the cygwin install folders (C:\cygwin64,
>>> C:\cygwin).
>>>
>>> It might be nice if we could so some comparisons so I can figure out
>>> what's wrong.
>
> Running strace on your forking executable should give you accurate
> numbers in the output, with some work to extract the relevant values.
>
>> Same AV here, W10 64bit (no VM), 2 year old Laptop
>>
>> model name      : Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz
>> 4 cores
>>
>> https://github.com/mondalaci/fork-benchmark
>> it seems there is a aging effect
>>
>> $ ./fork-benchmark.exe 1000
>> Forked, executed and destroyed 1000 processes in 39.928576 seconds.
>>
>> $ ./fork-benchmark.exe 1000
>> Forked, executed and destroyed 1000 processes in 42.701295 seconds.
>>
>> $ ./fork-benchmark.exe 1000
>> Forked, executed and destroyed 1000 processes in 49.890909 seconds.
>>
>> $ ./fork-benchmark.exe 1000
>> Forked, executed and destroyed 1000 processes in 61.657031 seconds.
>
> It's all part of your current process tree.
> Running cygserver may help this with appropriate process settings:
>
> $ grep -C1 'kern\.srv\..*_.*' /etc/cygserver.conf
>
> # kern.srv.cleanup_threads: No. of cygserver threads used for cleanup
> tasks.
> # Default: 2, Min: 1, Max: 16, command line option -c, --cleanup-threads
> #kern.srv.cleanup_threads 2
> kern.srv.cleanup_threads 16
>
> # kern.srv.request_threads: No. of cygserver threads used to serve
> #                           application requests.
> # Default: 10, Min: 1, Max: 310, command line option -r,
> --request-threads
> #kern.srv.request_threads 10
> kern.srv.request_threads 310
>
> # kern.srv.process_cache_size: No. of concurrent processes which can
> be handled
> #                              by Cygserver concurrently.
> # Default: 62, Min: 1, Max: 310, command line option -p, --process-cache
> #kern.srv.process_cache_size 62
> kern.srv.process_cache_size 310
>
> $ ./fork-benchmark 1000
> Forked, executed and destroyed 1000 processes in 33.397727 seconds.
> $ ./fork-benchmark 1000
> Forked, executed and destroyed 1000 processes in 34.70389 seconds.
> $ ./fork-benchmark 1000
> Forked, executed and destroyed 1000 processes in 34.186709 seconds.
> $ ./fork-benchmark 1000
> Forked, executed and destroyed 1000 processes in 33.65649 seconds.
> $ sed -En '/^model name|^cpu MHz/p;/MHz/q' /proc/cpuinfo
> model name      : AMD A10-9700 RADEON R7, 10 COMPUTE CORES 4C+6G
> cpu MHz         : 3500.000
>
> $ strace -o fork-benchmark.strace ./fork-benchmark 10
> $ egrep '^--- Process [0-9]+ (crea|\(pid: [0-9]+\) exi)ted|dwProcessId
> [0-9]+|ExitProcess n 0x' fork-benchmark.strace > fork-benchmark.log
> $ awk '/ [0-9]+! pinfo::exit: /{t+=$2};END{print t/10000"ms"}'
> fork-benchmark.log
> 34.1855ms
>
> Faster CPUs, faster memory, bigger caches, SSD drive may help.

Just running in KVM seems to have helped me.

My times are more like 13 seconds for 1000 forks now. There appears to
be drop-off, but after a big package build and some idling, it then gets
back down to 13 seconds.

Is there a way for me to check that the cygserver options are working?

Either way, my package builds are about twice as quick as they were
before now, possibly just from a reinstall and using KVM.

Hamish


[-- Attachment #1.1.2: 0x87B761FE07F548D6.asc --]
[-- Type: application/pgp-keys, Size: 3235 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]