Re: posix thread scaling issue

[jeff <jeff@jeffunit.com>]

On 9/2/2023 12:59, Brian Inglis wrote:
> On 2023-09-02 12:27, jeff via Cygwin wrote:
>> On 9/2/2023 10:56, Brian Inglis wrote:
>>> On 2023-09-02 08:57, jeff via Cygwin wrote:
>>>> I have a program that is embarrassing parallel.
>>>> On my older computer which has an epyc 7302 (16 cores,  32 threads) 
>>>> it scales very well using cygwin, and fully utilized all threads.
>>>> On my new computer which has an epyc 7B13 (64 cores, 128 threads) 
>>>> it does not scale very well.
>>>> According to the windows task manager, it only uses 74% of the cpu 
>>>> resources.
>>>> The time it takes the program to run on windows is 166 seconds.
>>>> Using the same hardware on a recent version of linux, I can get 
>>>> 100% cpu utilization and the program takes 100 seconds to run.
>>>> I suspect there may be something in cygwin that doesn't scale well 
>>>> with lots of posix threads.
>
> Both Windows and Cygwin support multiple processor groups, as some 
> developers, maintainers, and users need support on such systems, and 
> the process and thread support has been added to Cygwin.
>
>>>> I know this is a bit of an unusual situation, but you can buy a 128 
>>>> core / 256 thread system now.
>>>> Enclosed is the output of cygcheck.
>>>> I updated my version of cygwin to be current as of today, Sep 2 2023.
>
>>> What Windows edition and version are you running?
>>> For details run:
>>>
>>> $ reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows 
>>> NT\CurrentVersion" \
>>> | sed '/^\s\+\.*\s/!d;/^.\{80,\}/d'
>>>
>>> Some retail editions limit you to 64 threads and that seems to be 
>>> your case:
>>>
>>>     NUMBER_OF_PROCESSORS = '64'
>>>
>>> To make full use of your processors, you may have to upgrade your 
>>> Windows to a commercial licence (and installation) of Windows 10/11 
>>> Pro for Workstations, enabling server features on non-server 
>>> "Worskations" ~ HEDTs (High-End DeskTops); see:
>>>
>>> https://www.anandtech.com/show/15483/amd-threadripper-3990x-review/3
>>>
>>> or just run Linux!
>>>
>>> Watch out for terms misused like processor == socket on some sites!
>>>
>>> Also, you have to consider these are server systems, mainly designed 
>>> for VM not HPC (High Performance Computing) parallelism.
>>>
>>> Your older system has higher base and boost/turbo clocks 3.0-3.3GHz: 
>>> your newer system has lower clocks 2.25-2.65/3/3.5GHz which seems to 
>>> depend on OEM target.
>>>
>>> You may also need to upgrade your memory, as each core could run 
>>> ~10GB/s instructions, and these workstations are often provisioned 
>>> with 128-256GB (2-4GB/core), so that may also need a Windows edition 
>>> upgrade.
>
>> I am running windows 10 professional. Using the task manager, 64 
>> cores and 128 threads shows up for my processor.
>
> As the linked AnandTech article shows and explains with Task Manager/ 
> Performance tab, Win 10 Pro may think you have dual sockets, that 
> limits the maximum thread parallelism you can achieve:
According to the task manager, it says 'Sockets: 1'.

jeff