[gomp] GOMP_CPU_AFFINITY support

[gomp] GOMP_CPU_AFFINITY support

[Jakub Jelinek]

Hi!

I have committed following patch to libgomp, it is basically what I posted
last May, except with a couple of bugs fixed.
See http://gcc.gnu.org/ml/gcc-patches/2006-05/msg00982.html
for details.  Basically for those that really want to have total control
on which CPUs run which threads this allows them to do so, otherwise has no
effect.
Tested on x86_64-linux.

2007-04-04  Jakub Jelinek  <jakub@redhat.com>

	* libgomp.h (gomp_cpu_affinity, gomp_cpu_affinity_len): New extern
	decls.
	(gomp_init_affinity, gomp_init_thread_affinity): New prototypes.
	* env.c (gomp_cpu_affinity, gomp_cpu_affinity_len): New variables.
	(parse_affinity): New function.
	(initialize_env): Call it and gomp_init_affinity.
	* team.c (gomp_team_start): If gomp_cpu_affinity != NULL,
	create new pthread_attr_t and call gomp_init_thread_affinity
	on it for each thread before passing the attribute to pthread_create.
	* config/linux/affinity.c: New file.
	* config/posix/affinity.c: New file.
	* configure.ac (HAVE_PTHREAD_AFFINITY_NP): New test.
	* configure: Rebuilt.
	* config.h.in: Rebuilt.
	* Makefile.am (libgomp_la_SOURCES): Add affinity.c.
	* Makefile.in: Rebuilt.

--- libgomp/config.h.in.jj	2006-12-28 10:46:25.000000000 +0100
+++ libgomp/config.h.in	2007-04-04 15:51:18.000000000 +0200
@@ -24,6 +24,9 @@
 /* Define to 1 if you have the <memory.h> header file. */
 #undef HAVE_MEMORY_H
 
+/* Define if pthread_{,attr_}{g,s}etaffinity_np is supported. */
+#undef HAVE_PTHREAD_AFFINITY_NP
+
 /* Define to 1 if you have the <semaphore.h> header file. */
 #undef HAVE_SEMAPHORE_H
 
--- libgomp/config/linux/affinity.c.jj	2007-04-04 15:51:13.000000000 +0200
+++ libgomp/config/linux/affinity.c	2007-04-04 15:53:04.000000000 +0200
@@ -0,0 +1,107 @@
+/* Copyright (C) 2006, 2007 Free Software Foundation, Inc.
+   Contributed by Jakub Jelinek <jakub@redhat.com>.
+
+   This file is part of the GNU OpenMP Library (libgomp).
+
+   Libgomp is free software; you can redistribute it and/or modify it
+   under the terms of the GNU Lesser General Public License as published by
+   the Free Software Foundation; either version 2.1 of the License, or
+   (at your option) any later version.
+
+   Libgomp is distributed in the hope that it will be useful, but WITHOUT ANY
+   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+   FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for
+   more details.
+
+   You should have received a copy of the GNU Lesser General Public License
+   along with libgomp; see the file COPYING.LIB.  If not, write to the
+   Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+   MA 02110-1301, USA.  */
+
+/* As a special exception, if you link this library with other files, some
+   of which are compiled with GCC, to produce an executable, this library
+   does not by itself cause the resulting executable to be covered by the
+   GNU General Public License.  This exception does not however invalidate
+   any other reasons why the executable file might be covered by the GNU
+   General Public License.  */
+
+/* This is a Linux specific implementation of a CPU affinity setting.  */
+
+#ifndef _GNU_SOURCE
+#define _GNU_SOURCE 1
+#endif
+#include "libgomp.h"
+#include <sched.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#ifdef HAVE_PTHREAD_AFFINITY_NP
+
+static unsigned int affinity_counter;
+#ifndef HAVE_SYNC_BUILTINS
+static gomp_mutex_t affinity_lock;
+#endif
+
+void
+gomp_init_affinity (void)
+{
+  cpu_set_t cpuset;
+  size_t idx, widx;
+
+  if (pthread_getaffinity_np (pthread_self (), sizeof (cpuset), &cpuset))
+    {
+      gomp_error ("could not get CPU affinity set");
+      free (gomp_cpu_affinity);
+      gomp_cpu_affinity = NULL;
+      gomp_cpu_affinity_len = 0;
+      return;
+    }
+
+  for (widx = idx = 0; idx < gomp_cpu_affinity_len; idx++)
+    if (gomp_cpu_affinity[idx] < CPU_SETSIZE
+	&& CPU_ISSET (gomp_cpu_affinity[idx], &cpuset))
+      gomp_cpu_affinity[widx++] = gomp_cpu_affinity[idx];
+
+  if (widx == 0)
+    {
+      gomp_error ("no CPUs left for affinity setting");
+      free (gomp_cpu_affinity);
+      gomp_cpu_affinity = NULL;
+      gomp_cpu_affinity_len = 0;
+      return;
+    }
+
+  gomp_cpu_affinity_len = widx;
+  CPU_ZERO (&cpuset);
+  CPU_SET (gomp_cpu_affinity[0], &cpuset);
+  pthread_setaffinity_np (pthread_self (), sizeof (cpuset), &cpuset);
+  affinity_counter = 1;
+#ifndef HAVE_SYNC_BUILTINS
+  gomp_mutex_init (&affinity_lock);
+#endif
+}
+
+void
+gomp_init_thread_affinity (pthread_attr_t *attr)
+{
+  unsigned int cpu;
+  cpu_set_t cpuset;
+
+#ifdef HAVE_SYNC_BUILTINS
+  cpu = __sync_fetch_and_add (&affinity_counter, 1);
+#else
+  gomp_mutex_lock (&affinity_lock);
+  cpu = affinity_counter++;
+  gomp_mutex_unlock (&affinity_lock);
+#endif
+  cpu %= gomp_cpu_affinity_len;
+  CPU_ZERO (&cpuset);
+  CPU_SET (gomp_cpu_affinity[cpu], &cpuset);
+  pthread_attr_setaffinity_np (attr, sizeof (cpu_set_t), &cpuset);
+}
+
+#else
+
+#include "../posix/affinity.c"
+
+#endif
--- libgomp/config/posix/affinity.c.jj	2007-04-04 15:51:18.000000000 +0200
+++ libgomp/config/posix/affinity.c	2007-04-04 15:51:18.000000000 +0200
@@ -0,0 +1,41 @@
+/* Copyright (C) 2006 Free Software Foundation, Inc.
+   Contributed by Jakub Jelinek <jakub@redhat.com>.
+
+   This file is part of the GNU OpenMP Library (libgomp).
+
+   Libgomp is free software; you can redistribute it and/or modify it
+   under the terms of the GNU Lesser General Public License as published by
+   the Free Software Foundation; either version 2.1 of the License, or
+   (at your option) any later version.
+
+   Libgomp is distributed in the hope that it will be useful, but WITHOUT ANY
+   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+   FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for
+   more details.
+
+   You should have received a copy of the GNU Lesser General Public License
+   along with libgomp; see the file COPYING.LIB.  If not, write to the
+   Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+   MA 02110-1301, USA.  */
+
+/* As a special exception, if you link this library with other files, some
+   of which are compiled with GCC, to produce an executable, this library
+   does not by itself cause the resulting executable to be covered by the
+   GNU General Public License.  This exception does not however invalidate
+   any other reasons why the executable file might be covered by the GNU
+   General Public License.  */
+
+/* This is a generic stub implementation of a CPU affinity setting.  */
+
+#include "libgomp.h"
+
+void
+gomp_init_affinity (void)
+{
+}
+
+void
+gomp_init_thread_affinity (pthread_attr_t *attr)
+{
+  (void) attr;
+}
--- libgomp/team.c.jj	2006-10-05 00:24:40.000000000 +0200
+++ libgomp/team.c	2007-04-04 16:19:59.000000000 +0200
@@ -1,4 +1,4 @@
-/* Copyright (C) 2005 Free Software Foundation, Inc.
+/* Copyright (C) 2005, 2006, 2007 Free Software Foundation, Inc.
    Contributed by Richard Henderson <rth@redhat.com>.
 
    This file is part of the GNU OpenMP Library (libgomp).
@@ -183,6 +183,7 @@ gomp_team_start (void (*fn) (void *), vo
   struct gomp_team *team;
   bool nested;
   unsigned i, n, old_threads_used = 0;
+  pthread_attr_t thread_attr, *attr;
 
   thr = gomp_thread ();
   nested = thr->ts.team != NULL;
@@ -265,6 +266,17 @@ gomp_team_start (void (*fn) (void *), vo
 	}
     }
 
+  attr = &gomp_thread_attr;
+  if (gomp_cpu_affinity != NULL)
+    {
+      size_t stacksize;
+      pthread_attr_init (&thread_attr);
+      pthread_attr_setdetachstate (&thread_attr, PTHREAD_CREATE_DETACHED);
+      if (! pthread_attr_getstacksize (&thread_attr, &stacksize))
+	pthread_attr_setstacksize (&thread_attr, stacksize);
+      attr = &thread_attr;
+    }
+
   start_data = gomp_alloca (sizeof (struct gomp_thread_start_data)
 			    * (nthreads-i));
 
@@ -283,12 +295,17 @@ gomp_team_start (void (*fn) (void *), vo
       start_data->fn_data = data;
       start_data->nested = nested;
 
-      err = pthread_create (&pt, &gomp_thread_attr,
-			    gomp_thread_start, start_data);
+      if (gomp_cpu_affinity != NULL)
+	gomp_init_thread_affinity (attr);
+
+      err = pthread_create (&pt, attr, gomp_thread_start, start_data);
       if (err != 0)
 	gomp_fatal ("Thread creation failed: %s", strerror (err));
     }
 
+  if (gomp_cpu_affinity != NULL)
+    pthread_attr_destroy (&thread_attr);
+
  do_release:
   gomp_barrier_wait (nested ? &team->barrier : &gomp_threads_dock);
 
--- libgomp/Makefile.in.jj	2007-03-12 17:18:23.000000000 +0100
+++ libgomp/Makefile.in	2007-04-04 15:51:18.000000000 +0200
@@ -79,7 +79,7 @@ libgomp_la_LIBADD =
 am_libgomp_la_OBJECTS = alloc.lo barrier.lo critical.lo env.lo \
 	error.lo iter.lo loop.lo ordered.lo parallel.lo sections.lo \
 	single.lo team.lo work.lo lock.lo mutex.lo proc.lo sem.lo \
-	bar.lo time.lo fortran.lo
+	bar.lo time.lo fortran.lo affinity.lo
 libgomp_la_OBJECTS = $(am_libgomp_la_OBJECTS)
 DEFAULT_INCLUDES = -I. -I$(srcdir) -I.
 depcomp = $(SHELL) $(top_srcdir)/../depcomp
@@ -279,7 +279,7 @@ libgomp_version_info = -version-info $(l
 libgomp_la_LDFLAGS = $(libgomp_version_info) $(libgomp_version_script)
 libgomp_la_SOURCES = alloc.c barrier.c critical.c env.c error.c iter.c \
 	loop.c ordered.c parallel.c sections.c single.c team.c work.c \
-	lock.c mutex.c proc.c sem.c bar.c time.c fortran.c
+	lock.c mutex.c proc.c sem.c bar.c time.c fortran.c affinity.c
 
 nodist_noinst_HEADERS = libgomp_f.h
 nodist_libsubinclude_HEADERS = omp.h
@@ -406,6 +406,7 @@ mostlyclean-compile:
 distclean-compile:
 	-rm -f *.tab.c
 
+@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/affinity.Plo@am__quote@
 @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/alloc.Plo@am__quote@
 @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/bar.Plo@am__quote@
 @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/barrier.Plo@am__quote@
--- libgomp/libgomp.h.jj	2006-10-05 00:24:40.000000000 +0200
+++ libgomp/libgomp.h	2007-04-04 16:20:53.000000000 +0200
@@ -1,4 +1,4 @@
-/* Copyright (C) 2005 Free Software Foundation, Inc.
+/* Copyright (C) 2005, 2007 Free Software Foundation, Inc.
    Contributed by Richard Henderson <rth@redhat.com>.
 
    This file is part of the GNU OpenMP Library (libgomp).
@@ -246,8 +246,18 @@ extern unsigned long gomp_run_sched_chun
 /* The attributes to be used during thread creation.  */
 extern pthread_attr_t gomp_thread_attr;
 
+/* Other variables.  */
+
+extern unsigned short *gomp_cpu_affinity;
+extern size_t gomp_cpu_affinity_len;
+
 /* Function prototypes.  */
 
+/* affinity.c */
+
+extern void gomp_init_affinity (void);
+extern void gomp_init_thread_affinity (pthread_attr_t *);
+
 /* alloc.c */
 
 extern void *gomp_malloc (size_t) __attribute__((malloc));
--- libgomp/configure.ac.jj	2007-02-02 11:55:50.000000000 +0100
+++ libgomp/configure.ac	2007-04-04 15:51:18.000000000 +0200
@@ -236,6 +236,25 @@ If so, please configure with --disable-l
     ;;
 esac
 
+# Check for pthread_{,attr_}[sg]etaffinity_np.
+AC_LINK_IFELSE(
+ [AC_LANG_PROGRAM(
+  [#define _GNU_SOURCE
+   #include <pthread.h>],
+  [cpu_set_t cpuset;
+   pthread_attr_t attr;
+   pthread_getaffinity_np (pthread_self (), sizeof (cpu_set_t), &cpuset);
+   if (CPU_ISSET (0, &cpuset))
+     CPU_SET (1, &cpuset);
+   else
+     CPU_ZERO (&cpuset);
+   pthread_setaffinity_np (pthread_self (), sizeof (cpu_set_t), &cpuset);
+   pthread_attr_init (&attr);
+   pthread_attr_getaffinity_np (&attr, sizeof (cpu_set_t), &cpuset);
+   pthread_attr_setaffinity_np (&attr, sizeof (cpu_set_t), &cpuset);])],
+  AC_DEFINE(HAVE_PTHREAD_AFFINITY_NP, 1,
+[	Define if pthread_{,attr_}{g,s}etaffinity_np is supported.]))
+
 # At least for glibc, clock_gettime is in librt.  But don't pull that
 # in if it still doesn't give us the function we want.
 if test $ac_cv_func_clock_gettime = no; then
--- libgomp/configure.jj	2007-03-19 20:46:29.000000000 +0100
+++ libgomp/configure	2007-04-04 15:51:18.000000000 +0200
@@ -8851,6 +8851,68 @@ rm -f conftest.err conftest.$ac_objext \
     ;;
 esac
 
+# Check for pthread_{,attr_}[sg]etaffinity_np.
+cat >conftest.$ac_ext <<_ACEOF
+/* confdefs.h.  */
+_ACEOF
+cat confdefs.h >>conftest.$ac_ext
+cat >>conftest.$ac_ext <<_ACEOF
+/* end confdefs.h.  */
+#define _GNU_SOURCE
+   #include <pthread.h>
+int
+main ()
+{
+cpu_set_t cpuset;
+   pthread_attr_t attr;
+   pthread_getaffinity_np (pthread_self (), sizeof (cpu_set_t), &cpuset);
+   if (CPU_ISSET (0, &cpuset))
+     CPU_SET (1, &cpuset);
+   else
+     CPU_ZERO (&cpuset);
+   pthread_setaffinity_np (pthread_self (), sizeof (cpu_set_t), &cpuset);
+   pthread_attr_init (&attr);
+   pthread_attr_getaffinity_np (&attr, sizeof (cpu_set_t), &cpuset);
+   pthread_attr_setaffinity_np (&attr, sizeof (cpu_set_t), &cpuset);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:$LINENO: \"$ac_link\"") >&5
+  (eval $ac_link) 2>conftest.er1
+  ac_status=$?
+  grep -v '^ *+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:$LINENO: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+	 { ac_try='test -z "$ac_c_werror_flag"
+			 || test ! -s conftest.err'
+  { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:$LINENO: \$? = $ac_status" >&5
+  (exit $ac_status); }; } &&
+	 { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:$LINENO: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+
+cat >>confdefs.h <<\_ACEOF
+#define HAVE_PTHREAD_AFFINITY_NP 1
+_ACEOF
+
+else
+  echo "$as_me: failed program was:" >&5
+sed 's/^/| /' conftest.$ac_ext >&5
+
+fi
+rm -f conftest.err conftest.$ac_objext \
+      conftest$ac_exeext conftest.$ac_ext
+
 # At least for glibc, clock_gettime is in librt.  But don't pull that
 # in if it still doesn't give us the function we want.
 if test $ac_cv_func_clock_gettime = no; then
--- libgomp/Makefile.am.jj	2007-03-12 17:18:23.000000000 +0100
+++ libgomp/Makefile.am	2007-04-04 15:51:18.000000000 +0200
@@ -33,7 +33,7 @@ libgomp_la_LDFLAGS = $(libgomp_version_i
 
 libgomp_la_SOURCES = alloc.c barrier.c critical.c env.c error.c iter.c \
 	loop.c ordered.c parallel.c sections.c single.c team.c work.c \
-	lock.c mutex.c proc.c sem.c bar.c time.c fortran.c
+	lock.c mutex.c proc.c sem.c bar.c time.c fortran.c affinity.c
 
 nodist_noinst_HEADERS = libgomp_f.h
 nodist_libsubinclude_HEADERS = omp.h
--- libgomp/env.c.jj	2006-12-08 15:57:45.000000000 +0100
+++ libgomp/env.c	2007-04-04 17:01:00.000000000 +0200
@@ -1,4 +1,4 @@
-/* Copyright (C) 2005, 2006 Free Software Foundation, Inc.
+/* Copyright (C) 2005, 2006, 2007 Free Software Foundation, Inc.
    Contributed by Richard Henderson <rth@redhat.com>.
 
    This file is part of the GNU OpenMP Library (libgomp).
@@ -42,6 +42,8 @@ bool gomp_dyn_var = false;
 bool gomp_nest_var = false;
 enum gomp_schedule_type gomp_run_sched_var = GFS_DYNAMIC;
 unsigned long gomp_run_sched_chunk = 1;
+unsigned short *gomp_cpu_affinity;
+size_t gomp_cpu_affinity_len;
 
 /* Parse the OMP_SCHEDULE environment variable.  */
 
@@ -177,6 +179,97 @@ parse_boolean (const char *name, bool *v
     gomp_error ("Invalid value for environment variable %s", name);
 }
 
+/* Parse the GOMP_CPU_AFFINITY environment varible.  Return true if one was
+   present and it was successfully parsed.  */
+
+static bool
+parse_affinity (void)
+{
+  char *env, *end;
+  unsigned long cpu_beg, cpu_end, cpu_stride;
+  unsigned short *cpus = NULL;
+  size_t allocated = 0, used = 0, needed;
+
+  env = getenv ("GOMP_CPU_AFFINITY");
+  if (env == NULL)
+    return false;
+
+  do
+    {
+      while (*env == ' ' || *env == '\t')
+	env++;
+
+      cpu_beg = strtoul (env, &end, 0);
+      cpu_end = cpu_beg;
+      cpu_stride = 1;
+      if (env == end || cpu_beg >= 65536)
+	goto invalid;
+
+      env = end;
+      if (*env == '-')
+	{
+	  cpu_end = strtoul (++env, &end, 0);
+	  if (env == end || cpu_end >= 65536 || cpu_end < cpu_beg)
+	    goto invalid;
+
+	  env = end;
+	  if (*env == ':')
+	    {
+	      cpu_stride = strtoul (++env, &end, 0);
+	      if (env == end || cpu_stride == 0 || cpu_stride >= 65536)
+		goto invalid;
+
+	      env = end;
+	    }
+	}
+
+      needed = (cpu_end - cpu_beg) / cpu_stride + 1;
+      if (used + needed >= allocated)
+	{
+	  unsigned short *new_cpus;
+
+	  if (allocated < 64)
+	    allocated = 64;
+	  if (allocated > needed)
+	    allocated <<= 1;
+	  else
+	    allocated += 2 * needed;
+	  new_cpus = realloc (cpus, allocated * sizeof (unsigned short));
+	  if (new_cpus == NULL)
+	    {
+	      free (cpus);
+	      gomp_error ("not enough memory to store GOMP_CPU_AFFINITY list");
+	      return false;
+	    }
+
+	  cpus = new_cpus;
+	}
+
+      while (needed--)
+	{
+	  cpus[used++] = cpu_beg;
+	  cpu_beg += cpu_stride;
+	}
+
+      while (*env == ' ' || *env == '\t')
+	env++;
+
+      if (*env == ',')
+	env++;
+      else if (*env == '\0')
+	break;
+    }
+  while (1);
+
+  gomp_cpu_affinity = cpus;
+  gomp_cpu_affinity_len = used;
+  return true;
+
+ invalid:
+  gomp_error ("Invalid value for enviroment variable GOMP_CPU_AFFINITY");
+  return false;
+}
+
 static void __attribute__((constructor))
 initialize_env (void)
 {
@@ -190,6 +283,8 @@ initialize_env (void)
   parse_boolean ("OMP_NESTED", &gomp_nest_var);
   if (!parse_unsigned_long ("OMP_NUM_THREADS", &gomp_nthreads_var))
     gomp_init_num_threads ();
+  if (parse_affinity ())
+    gomp_init_affinity ();
 
   /* Not strictly environment related, but ordering constructors is tricky.  */
   pthread_attr_init (&gomp_thread_attr);

	Jakub

Re: [gomp] GOMP_CPU_AFFINITY support

[Daniel Franke]

On Wednesday 04 April 2007 17:54:36 Jakub Jelinek wrote:
> I have committed following patch to libgomp, it is basically what I posted
> last May, except with a couple of bugs fixed.
> See http://gcc.gnu.org/ml/gcc-patches/2006-05/msg00982.html
> for details.  Basically for those that really want to have total control
> on which CPUs run which threads this allows them to do so, otherwise has no
> effect.

Jakub,

I played with it and while preparing the docs, some questions arose:

 a) is there any way to query the affinity settings from within the app?
 b) is there any way to disable the affinity settings after startup?
 c) is there any way to disable this feature completely?
 d) if GOMP_CPU_AFFINITY="0 1 4" is used on a 4 CPU machine, an error is 
printed. How do you handle it internally? Does the implementation discard the 
settings completely, or does it ignore the erroneous part only? From tests, I 
would assume the former?!
 e) will it be ported to 4.2 prior to release?

Questions (a)-(c) relate to inventative people who set GOMP_CPU_AFFINITY=0 (or 
similar) for their collegues and those wonder what happend to their nice and 
shiny parallelized application that only runs on a single CPU ...

Thanks in advance

	Daniel

Re: [gomp] GOMP_CPU_AFFINITY support

[Jakub Jelinek]

On Thu, Apr 05, 2007 at 11:04:52PM +0200, Daniel Franke wrote:
> I played with it and while preparing the docs, some questions arose:
> 
>  a) is there any way to query the affinity settings from within the app?

You can getenv ("GOMP_CPU_AFFINITY"), or non-portably query affinity.

>  b) is there any way to disable the affinity settings after startup?
>  c) is there any way to disable this feature completely?

No+No.

>  d) if GOMP_CPU_AFFINITY="0 1 4" is used on a 4 CPU machine, an error is 
> printed. How do you handle it internally? Does the implementation discard the 
> settings completely, or does it ignore the erroneous part only? From tests, I 
> would assume the former?!

No error is printed for GOMP_CPU_AFFINITY="0 1 4", all it does is discard
non-existent CPUs (or CPUs disallowed by previous affinity setting, e.g.
through taskset(1)) from the list, so if you have CPUs 0-3,
GOMP_CPU_AFFINITY="0 1 4" is the same as GOMP_CPU_AFFINITY=0,1 , i.e.
threads are assigned to CPUs 0 (initial thread), 1, 0, 1, 0, 1 ...
A warning is only printed if no CPUs are left, say if
you try say GOMP_CPU_AFFINITY=4-7:2 and only have CPUs 0-3.

>  e) will it be ported to 4.2 prior to release?

No.  It can't be considered a bugfix.

> Questions (a)-(c) relate to inventative people who set GOMP_CPU_AFFINITY=0 (or 
> similar) for their collegues and those wonder what happend to their nice and 
> shiny parallelized application that only runs on a single CPU ...

How is that different from inventive people that start other people's shells with
taskset -c 0 /bin/sh?

	Jakub

Re: [gomp] GOMP_CPU_AFFINITY support

[Daniel Franke]

[-- Attachment #1: Type: text/plain, Size: 284 bytes --]


Jakub, 

thanks for answering my questions. Please find attached the updated 
documentation for libgomp. Tested info and dvi targets on i686-pc-linux-gnu.


2007-04-11  Daniel Franke  <franke.daniel@gmail.com>

	* libgomp.texi (GOMP_CPU_AFFINITY): Updated.


Ok for trunk?

	Daniel


[-- Attachment #2: libgomp.texi.diff --]
[-- Type: text/x-diff, Size: 1673 bytes --]

Index: libgomp.texi
===================================================================
--- libgomp.texi	(revision 123524)
+++ libgomp.texi	(working copy)
@@ -895,14 +895,24 @@
 @cindex Environment Variable
 @table @asis
 @item @emph{Description}:
-A patch for this extension has been submitted, but was not yet applied at the
-time of writing.
+Binds threads to specific CPUs. The variable should contain space or
+comma separated list of CPUs, e. g. either a zero based CPU number, or a
+range of CPUs (M-N) or a range with some stride (M-N:S). For example,
+@code{GOMP_CPU_AFFINITY="0 3 1-2 4-15:2"} will bind the initial thread
+to CPU 0, the second to CPU 3, the third to CPU 1, the fourth to 
+CPU 2, the fifth to CPU 4, the sixth to CPU 6, the seventh to CPU 8, ... 
+tenth to CPU 14 and then starts assigning back from the beginning of the 
+list. @code{GOMP_CPU_AFFINITY=0} binds all threads to CPU 0.
 
-@item @emph{Reference}: 
-@uref{http://gcc.gnu.org/ml/gcc-patches/2006-05/msg00982.html, 
-GCC Patches Mailinglist}
-@uref{http://gcc.gnu.org/ml/gcc-patches/2006-05/msg01133.html,
-GCC Patches Mailinglist}
+There is no GNU OpenMP library routine to determine whether CPU affinity 
+is in effect. As a workaround, language specific library functions, e. g. 
+@code{getenv} in C or @code{GET_ENVIRONMENT_VARIABLE} in Fortran, may 
+be used to query the setting of the environment variable. A defined CPU 
+affinity on startup can not be changed or disabled during the runtime of 
+the application.
+
+If this environment variable is omitted, the host system while handle the 
+assignment of threads to CPUs. 
 @end table
 
 

Re: [gomp] GOMP_CPU_AFFINITY support

[Brooks Moses]

Daniel Franke wrote:
> Ok for trunk?

Some minor typo fixes and comments:

> @@ -895,14 +895,24 @@
>  @cindex Environment Variable
>  @table @asis
>  @item @emph{Description}:
> -A patch for this extension has been submitted, but was not yet applied at the
> -time of writing.
> +Binds threads to specific CPUs. The variable should contain space or

"contain a space"

> +comma separated list of CPUs, e. g. either a zero based CPU number, or a
> +range of CPUs (M-N) or a range with some stride (M-N:S). For example,

Those don't seem like examples of "space or comma separated lists of 
CPUs"; I think this could be written a little more clearly.  Also, I 
would put "The CPU numbers are zero-based" as a separate sentence, 
rather than tucking it into a clause of this list.

Also, pedantically, that should be "space- or comma-separated"; compound 
adjectives get hyphens.  And I think (but am not sure) that house style 
for "e.g." is to omit the space, and always follow it with a comma.

> +@code{GOMP_CPU_AFFINITY="0 3 1-2 4-15:2"} will bind the initial thread
> +to CPU 0, the second to CPU 3, the third to CPU 1, the fourth to 
> +CPU 2, the fifth to CPU 4, the sixth to CPU 6, the seventh to CPU 8, ... 
> +tenth to CPU 14 and then starts assigning back from the beginning of the 

"then start assigning".

The "..." looks a bit too sketchy for my tastes; maybe "the sixth 
through tenth to CPUs 6, 8, 10, 12, and 14 respectively, and then..."? 
(At least, make that "the tenth to".)

> +list. @code{GOMP_CPU_AFFINITY=0} binds all threads to CPU 0.

Giving examples like this is definitely a good thing; well done!  (I 
would put the simple example first, though.)

> -@item @emph{Reference}: 
> -@uref{http://gcc.gnu.org/ml/gcc-patches/2006-05/msg00982.html, 
> -GCC Patches Mailinglist}
> -@uref{http://gcc.gnu.org/ml/gcc-patches/2006-05/msg01133.html,
> -GCC Patches Mailinglist}
> +There is no GNU OpenMP library routine to determine whether CPU affinity 
> +is in effect. As a workaround, language specific library functions, e. g. 

See earlier comment about "e.g.".  I'd also word that as "a CPU affinity 
specification" or "a CPU affinitiy environment variable", rather than 
just "CPU affinity".

Also: "language-specific".  (Compound adjective, again.)

> +@code{getenv} in C or @code{GET_ENVIRONMENT_VARIABLE} in Fortran, may 
> +be used to query the setting of the environment variable. A defined CPU 

"the @code{GOMP_CPU_AFFINITY} environment variable", I'd say, just to be 
completely clear.  Or maybe just "this environment variable".

> +affinity on startup can not be changed or disabled during the runtime of 
> +the application.

"cannot".

I would also suggest "while the application is running", but that's just 
opinion.  :)

> +If this environment variable is omitted, the host system while handle the 
> +assignment of threads to CPUs. 
>  @end table

"system will handle".

I do want to point out that I don't think the quantity of virtual red 
ink I just applied to this is any reflection on the quality -- it looks 
well-written and clear to me!  :)

- Brooks

Re: [gomp] GOMP_CPU_AFFINITY support

[Brooks Moses]

(This may be a duplicate; my mail server glitched and I can't tell if 
this got sent to everyone or not.  Sigh.)

Daniel Franke wrote:
> Ok for trunk?

Some minor typo fixes and comments:

> @@ -895,14 +895,24 @@
>  @cindex Environment Variable
>  @table @asis
>  @item @emph{Description}:
> -A patch for this extension has been submitted, but was not yet applied at the
> -time of writing.
> +Binds threads to specific CPUs. The variable should contain space or

"contain a space"

> +comma separated list of CPUs, e. g. either a zero based CPU number, or a
> +range of CPUs (M-N) or a range with some stride (M-N:S). For example,

Those don't seem like examples of "space or comma separated lists of 
CPUs"; I think this could be written a little more clearly.  Also, I 
would put "The CPU numbers are zero-based" as a separate sentence, 
rather than tucking it into a clause of this list.

Also, pedantically, that should be "space- or comma-separated"; compound 
adjectives get hyphens.  And I think (but am not sure) that house style 
for "e.g." is to omit the space, and always follow it with a comma.

> +@code{GOMP_CPU_AFFINITY="0 3 1-2 4-15:2"} will bind the initial thread
> +to CPU 0, the second to CPU 3, the third to CPU 1, the fourth to 
> +CPU 2, the fifth to CPU 4, the sixth to CPU 6, the seventh to CPU 8, ... 
> +tenth to CPU 14 and then starts assigning back from the beginning of the 

"then start assigning".

The "..." looks a bit too sketchy for my tastes; maybe "the sixth 
through tenth to CPUs 6, 8, 10, 12, and 14 respectively, and then..."? 
(At least, make that "the tenth to".)

> +list. @code{GOMP_CPU_AFFINITY=0} binds all threads to CPU 0.

Giving examples like this is definitely a good thing; well done!  (I 
would put the simple example first, though.)

> -@item @emph{Reference}: 
> -@uref{http://gcc.gnu.org/ml/gcc-patches/2006-05/msg00982.html, 
> -GCC Patches Mailinglist}
> -@uref{http://gcc.gnu.org/ml/gcc-patches/2006-05/msg01133.html,
> -GCC Patches Mailinglist}
> +There is no GNU OpenMP library routine to determine whether CPU affinity 
> +is in effect. As a workaround, language specific library functions, e. g. 

See earlier comment about "e.g.".  I'd also word that as "a CPU affinity 
specification" or "a CPU affinitiy environment variable", rather than 
just "CPU affinity".

Also: "language-specific".  (Compound adjective, again.)

> +@code{getenv} in C or @code{GET_ENVIRONMENT_VARIABLE} in Fortran, may 
> +be used to query the setting of the environment variable. A defined CPU 

"the @code{GOMP_CPU_AFFINITY} environment variable", I'd say, just to be 
completely clear.  Or maybe just "this environment variable".

> +affinity on startup can not be changed or disabled during the runtime of 
> +the application.

"cannot".

I would also suggest "while the application is running", but that's just 
opinion.  :)

> +If this environment variable is omitted, the host system while handle the 
> +assignment of threads to CPUs. 
>  @end table

"system will handle".

I do want to point out that I don't think the quantity of virtual red 
ink I just applied to this is any reflection on the quality -- it looks 
well-written and clear to me!  :)

- Brooks

Re: [gomp] GOMP_CPU_AFFINITY support

[Daniel Franke]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

On Thursday 12 April 2007 06:52:21 Brooks Moses wrote:

Brooks, 

thanks for your helpful comments, I applied them as suggested.


> > +list. @code{GOMP_CPU_AFFINITY=0} binds all threads to CPU 0.
>
> Giving examples like this is definitely a good thing; well done!  (I
> would put the simple example first, though.)

> I do want to point out that I don't think the quantity of virtual red
> ink I just applied to this is any reflection on the quality -- it looks
> well-written and clear to me!  :)

Thank you very much, but your praise has to go to Jakub. Most of this was
based on his explanation given when he first proposed the patch last year :)

Please find an updated version of the patch attached.

Regards
	Daniel



[-- Attachment #2: libgomp.texi.diff --]
[-- Type: text/x-diff, Size: 1759 bytes --]

Index: libgomp.texi
===================================================================
--- libgomp.texi	(revision 123524)
+++ libgomp.texi	(working copy)
@@ -895,14 +895,25 @@
 @cindex Environment Variable
 @table @asis
 @item @emph{Description}:
-A patch for this extension has been submitted, but was not yet applied at the
-time of writing.
+Binds threads to specific CPUs. The variable should contain a space- or
+comma-separated list of CPUs. This list may contain different kind of 
+entries: either single CPU numbers in any order, a range of CPUs (M-N) 
+or a range with some stride (M-N:S). CPU numbers are zero based. For example,
+@code{GOMP_CPU_AFFINITY="0 3 1-2 4-15:2"} will bind the initial thread
+to CPU 0, the second to CPU 3, the third to CPU 1, the fourth to 
+CPU 2, the fifth to CPU 4, the sixth through tenth to CPUs 6, 8, 10, 12,
+and 14 respectively and then start assigning back from the beginning of
+the list. @code{GOMP_CPU_AFFINITY=0} binds all threads to CPU 0.
 
-@item @emph{Reference}: 
-@uref{http://gcc.gnu.org/ml/gcc-patches/2006-05/msg00982.html, 
-GCC Patches Mailinglist}
-@uref{http://gcc.gnu.org/ml/gcc-patches/2006-05/msg01133.html,
-GCC Patches Mailinglist}
+There is no GNU OpenMP library routine to determine whether a CPU affinity 
+specification is in effect. As a workaround, language-specific library 
+functions, e.g., @code{getenv} in C or @code{GET_ENVIRONMENT_VARIABLE} in 
+Fortran, may be used to query the setting of the @code{GOMP_CPU_AFFINITY} 
+environment variable. A defined CPU affinity on startup cannot be changed 
+or disabled during the runtime of the application.
+
+If this environment variable is omitted, the host system will handle the 
+assignment of threads to CPUs. 
 @end table
 
 

Re: [gomp] GOMP_CPU_AFFINITY support

[Daniel Franke]

On Thursday 12 April 2007 20:17:40 I wrote:

http://gcc.gnu.org/ml/gcc-patches/2007-04/msg00712.html

> Please find an updated version of the patch attached.

Maybe I should add: 

Tested info and dvi targets on i686-pc-linux-gnu. 
Ok for trunk?

	Daniel

Re: [gomp] GOMP_CPU_AFFINITY support

[Jakub Jelinek]

On Thu, Apr 12, 2007 at 08:17:40PM +0200, Daniel Franke wrote:
> Please find an updated version of the patch attached.

Ok, with a proper ChangeLog entry.

	Jakub