More oddities with multiple processor groups

More oddities with multiple processor groups

[Achim Gratz]

I seem to be the first to try Cygwin on a box that has multiple
processor groups, which seems odd.  Anyway, I've already noticed two
more things that indicate that Cygwin and/or Cygwin applications
currently don't deal well with the situation:

1. Trying to run top, it shows only the first 16 processors and then
exits with an error that "openproc failed".  Interestingly enough it
will keep running despite this error when running under strace (still
only showing the processors from one group, presumably the one that top
got started in).

2. Git will correctly determine that it can use 32 threads for garbage
collection, but it starts them all in the same processor group.

The problem here is that on Linux you don't need to do anything extra to
use any of the advertised logical processors from a single application,
while on Windows you need to first create a thread and set it's affinity
to a different group than where your process was started in, then assign
each new thread an affinity to one of the available groups.  If you
don't do that, all threads will be restricted to the original group.


Some more info on these differences (and already a bit outdated
w.r.t. the way processor groups are formed):

https://software.intel.com/en-us/blogs/2010/12/16/tbb-30-high-end-many-cores-and-windows-processor-groups

If it would be possible for Cygwin to hide that ugliness from Cygwin
applications I think that'd be highly welcome.  Otherwise there might
need to be some option to restrict Cygwin to a single processor group
for some applications to work (correctly).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: More oddities with multiple processor groups

[L A Walsh]

Achim Gratz wrote:
> The problem here is that on Linux you don't need to do anything extra to
> use any of the advertised logical processors from a single application,
> while on Windows you need to first create a thread and set it's affinity
> to a different group than where your process was started in, then assign
> each new thread an affinity to one of the available groups.  If you
> don't do that, all threads will be restricted to the original group.
----
	Not exactly true.  They are not *restricted* -- it's a *feature*
of the Windows scheduler, in that future procs/threads inherit the
cpu of the parent.  Linux's scheduler is more advanced as well as
being replaceable.  MS doesn't want you to do that

> there might
> need to be some option to restrict Cygwin to a single processor group
> for some applications to work (correctly).
---
	There is.  Start them all on a single cpu & set the cpu 
mask.  Pretty much the same way you restrict procs on linux --
you can run them with a specific cpu mask, and most programs will
keep running w/that mask.

	Unfortunately, AFAIK, I don't think POSIX specifies
a way to set affinities, so I'm not sure how cygwin would do it.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: More oddities with multiple processor groups

[Brian Inglis]

On 2018-04-11 12:05, Achim Gratz wrote:
> I seem to be the first to try Cygwin on a box that has multiple
> processor groups, which seems odd.  Anyway, I've already noticed two
> more things that indicate that Cygwin and/or Cygwin applications
> currently don't deal well with the situation:
> 
> 1. Trying to run top, it shows only the first 16 processors and then
> exits with an error that "openproc failed".  Interestingly enough it
> will keep running despite this error when running under strace (still
> only showing the processors from one group, presumably the one that top
> got started in).
> 
> 2. Git will correctly determine that it can use 32 threads for garbage
> collection, but it starts them all in the same processor group.
> 
> The problem here is that on Linux you don't need to do anything extra to
> use any of the advertised logical processors from a single application,
> while on Windows you need to first create a thread and set it's affinity
> to a different group than where your process was started in, then assign
> each new thread an affinity to one of the available groups.  If you
> don't do that, all threads will be restricted to the original group.
> 
> 
> Some more info on these differences (and already a bit outdated
> w.r.t. the way processor groups are formed):
> 
> https://software.intel.com/en-us/blogs/2010/12/16/tbb-30-high-end-many-cores-and-windows-processor-groups
> 
> If it would be possible for Cygwin to hide that ugliness from Cygwin
> applications I think that'd be highly welcome.  Otherwise there might
> need to be some option to restrict Cygwin to a single processor group
> for some applications to work (correctly).

Have you tried installing and running hwloc package to find out how it sees your
system?
If you run lstopo under X, you get a pretty diagram, but you can also run
lstopo-no-graphics aka hwloc-ls without X. Running "apropos hwloc" lists
commands you can use to manipulate the topology.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: More oddities with multiple processor groups

[Brian Inglis]

On 2018-04-13 08:12, L A Walsh wrote:
> Achim Gratz wrote:
>> The problem here is that on Linux you don't need to do anything extra to 
>> use any of the advertised logical processors from a single application, 
>> while on Windows you need to first create a thread and set it's affinity to
>> a different group than where your process was started in, then assign each
>> new thread an affinity to one of the available groups.  If you don't do
>> that, all threads will be restricted to the original group.
> Not exactly true.  They are not *restricted* -- it's a *feature* of the
> Windows scheduler, in that future procs/threads inherit the cpu of the
> parent.  Linux's scheduler is more advanced as well as being replaceable.  MS
> doesn't want you to do that
>> there might need to be some option to restrict Cygwin to a single processor
>> group for some applications to work (correctly).
> There is.  Start them all on a single cpu & set the cpu mask.  Pretty much 
> the same way you restrict procs on linux -- you can run them with a specific
> cpu mask, and most programs will keep running w/that mask.
> Unfortunately, AFAIK, I don't think POSIX specifies a way to set affinities,
> so I'm not sure how cygwin would do it.

Glibc adds {pthread,sched}_...affinity... functions.
Linux uses namespaces, control groups (cgroups), cpusets, sysfs/kernfs:
util-linux provides unshare and taskset (not on Cygwin).
BSDs use cpu or processor sets functions and commands.
Mac OS X/Darwin supports either same or different thread cache affinity set and
tag /hints/ (which could be used like cpu sets), but only cache sharing cpu
counts, no explicit cpu identification or control.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: More oddities with multiple processor groups

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1988 bytes --]

On Apr 13 12:29, Brian Inglis wrote:
> On 2018-04-13 08:12, L A Walsh wrote:
> > Achim Gratz wrote:
> >> The problem here is that on Linux you don't need to do anything extra to 
> >> use any of the advertised logical processors from a single application, 
> >> while on Windows you need to first create a thread and set it's affinity to
> >> a different group than where your process was started in, then assign each
> >> new thread an affinity to one of the available groups.  If you don't do
> >> that, all threads will be restricted to the original group.
> > Not exactly true.  They are not *restricted* -- it's a *feature* of the
> > Windows scheduler, in that future procs/threads inherit the cpu of the
> > parent.  Linux's scheduler is more advanced as well as being replaceable.  MS
> > doesn't want you to do that
> >> there might need to be some option to restrict Cygwin to a single processor
> >> group for some applications to work (correctly).
> > There is.  Start them all on a single cpu & set the cpu mask.  Pretty much 
> > the same way you restrict procs on linux -- you can run them with a specific
> > cpu mask, and most programs will keep running w/that mask.
> > Unfortunately, AFAIK, I don't think POSIX specifies a way to set affinities,
> > so I'm not sure how cygwin would do it.
> 
> Glibc adds {pthread,sched}_...affinity... functions.
> Linux uses namespaces, control groups (cgroups), cpusets, sysfs/kernfs:

We would only be able to support it partially.  Windows doesn't
allow to set thread affinity across CPU groups.  There's no simple
function to set process CPU group affinity, only a function to set
process affinity within a single group.  And it has a restriction
in that all current threads have to run in the same CPU group.
The API is pretty tricky.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: More oddities with multiple processor groups

[Achim Gratz]

Brian Inglis writes:
> Have you tried installing and running hwloc package to find out how it sees your
> system?

Yes.  That is OK, but it doesn't change the fact that an application in
Cygwin can see N processors, but then can't actually run on all of them.
If Cygwin would switch the process to processor group aware state by
default then my understanding is that at least the processor group would
have to be explicitly selected for each thread, while on Linux the
scheduler would eventually use all of the NUMA nodes unless the
application tells it prefers different affinities.

> If you run lstopo under X, you get a pretty diagram, but you can also run
> lstopo-no-graphics aka hwloc-ls without X. Running "apropos hwloc" lists
> commands you can use to manipulate the topology.

I'll have that machine switched to "flat mode" which will result in a
single processor group (still with two NUMA nodes, so things like MPI
still know what to do).  I have no actual use for the current topology
and don't want to deal with the obvious complications, especially as I
seem to be the only person who'd have them.  If I can free up some time
I will want to find out why top crashes, but for me that'l be the end of
it if I even get that far.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple