[PATCH] contrib: modernize gen_autofdo_event.py

[PATCH] contrib: modernize gen_autofdo_event.py

[Xi Ruoyao]

Python 2 has been EOL'ed for two years.  egrep has been deprecated
for many years and the next grep release will start to print warning if
it is used.

-E option may be unsupported by some non-POSIX grep implementations, but
gcc-auto-profile won't work on non-Linux systems anyway.

contrib/ChangeLog:

	* gen_autofdo_event.py: Port to Python 3, and use grep -E
	instead of egrep.

gcc/ChangeLog:

	* config/i386/gcc-auto-profile: Regenerate.
---
 contrib/gen_autofdo_event.py     | 80 ++++++++++++++++----------------
 gcc/config/i386/gcc-auto-profile | 31 +++++++------
 2 files changed, 57 insertions(+), 54 deletions(-)

diff --git a/contrib/gen_autofdo_event.py b/contrib/gen_autofdo_event.py
index 1eb6f1d6d85..7da2876530d 100755
--- a/contrib/gen_autofdo_event.py
+++ b/contrib/gen_autofdo_event.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python3
 # Generate Intel taken branches Linux perf event script for autofdo profiling.
 
 # Copyright (C) 2016 Free Software Foundation, Inc.
@@ -26,18 +26,19 @@
 # Requires internet (https) access. This may require setting up a proxy
 # with export https_proxy=...
 #
-import urllib2
+import urllib.request
 import sys
 import json
 import argparse
 import collections
+import os
 
 baseurl = "https://download.01.org/perfmon"
 
-target_events = (u'BR_INST_RETIRED.NEAR_TAKEN',
-                 u'BR_INST_EXEC.TAKEN',
-                 u'BR_INST_RETIRED.TAKEN_JCC',
-                 u'BR_INST_TYPE_RETIRED.COND_TAKEN')
+target_events = ('BR_INST_RETIRED.NEAR_TAKEN',
+                 'BR_INST_EXEC.TAKEN',
+                 'BR_INST_RETIRED.TAKEN_JCC',
+                 'BR_INST_TYPE_RETIRED.COND_TAKEN')
 
 ap = argparse.ArgumentParser()
 ap.add_argument('--all', '-a', help='Print for all CPUs', action='store_true')
@@ -71,47 +72,46 @@ def get_cpustr():
     return "%s-%d-%X" % tuple(cpu)[:3]
 
 def find_event(eventurl, model):
-    print >>sys.stderr, "Downloading", eventurl
-    u = urllib2.urlopen(eventurl)
+    print("Downloading", eventurl, file = sys.stderr)
+    u = urllib.request.urlopen(eventurl)
     events = json.loads(u.read())
     u.close()
 
     found = 0
     for j in events:
-        if j[u'EventName'] in target_events:
-            event = "cpu/event=%s,umask=%s/" % (j[u'EventCode'], j[u'UMask'])
-            if u'PEBS' in j and j[u'PEBS'] > 0:
+        if j['EventName'] in target_events:
+            event = "cpu/event=%s,umask=%s/" % (j['EventCode'], j['UMask'])
+            if 'PEBS' in j and int(j['PEBS']) > 0:
                 event += "p"
             if args.script:
                 eventmap[event].append(model)
             else:
-                print j[u'EventName'], "event for model", model, "is", event
+                print(j['EventName'], "event for model", model, "is", event)
             found += 1
     return found
 
 if not args.all:
-    cpu = get_cpu_str()
+    cpu = get_cpustr()
     if not cpu:
         sys.exit("Unknown CPU type")
 
 url = baseurl + "/mapfile.csv"
-print >>sys.stderr, "Downloading", url
-u = urllib2.urlopen(url)
+print("Downloading", url, file = sys.stderr)
+u = urllib.request.urlopen(url)
 found = 0
 cpufound = 0
 for j in u:
-    n = j.rstrip().split(',')
+    n = j.rstrip().decode().split(',')
     if len(n) >= 4 and (args.all or n[0] == cpu) and n[3] == "core":
-        if args.all:
-            components = n[0].split("-")
-            model = components[2]
-            model = int(model, 16)
+        components = n[0].split("-")
+        model = components[2]
+        model = int(model, 16)
         cpufound += 1
         found += find_event(baseurl + n[2], model)
 u.close()
 
 if args.script:
-    print '''#!/bin/sh
+    print('''#!/bin/sh
 # Profile workload for gcc profile feedback (autofdo) using Linux perf.
 # Auto generated. To regenerate for new CPUs run
 # contrib/gen_autofdo_event.py --script --all in gcc source
@@ -146,27 +146,27 @@ if grep -q hypervisor /proc/cpuinfo ; then
   echo >&2 "Warning: branch profiling may not be functional in VMs"
 fi
 
-case `egrep -q "^cpu family\s*: 6" /proc/cpuinfo &&
-  egrep "^model\s*:" /proc/cpuinfo | head -n1` in'''
-    for event, mod in eventmap.iteritems():
+case `grep -E -q "^cpu family\s*: 6" /proc/cpuinfo &&
+  grep -E "^model\s*:" /proc/cpuinfo | head -n1` in''')
+    for event, mod in eventmap.items():
         for m in mod[:-1]:
-            print "model*:\ %s|\\" % m
-        print 'model*:\ %s) E="%s$FLAGS" ;;' % (mod[-1], event)
-    print '''*)
+            print("model*:\ %s|\\" % m)
+        print('model*:\ %s) E="%s$FLAGS" ;;' % (mod[-1], event))
+    print('''*)
 echo >&2 "Unknown CPU. Run contrib/gen_autofdo_event.py --all --script to update script."
-	exit 1 ;;'''
-    print "esac"
-    print "set -x"
-    print 'if ! perf record -e $E -b "$@" ; then'
-    print '  # PEBS may not actually be working even if the processor supports it'
-    print '  # (e.g., in a virtual machine). Trying to run without /p.'
-    print '  set +x'
-    print '  echo >&2 "Retrying without /p."'
-    print '  E="$(echo "${E}" | sed -e \'s/\/p/\//\')"'
-    print '  set -x'
-    print '  exec perf record -e $E -b "$@"'
-    print ' set +x'
-    print 'fi'
+	exit 1 ;;''')
+    print("esac")
+    print("set -x")
+    print('if ! perf record -e $E -b "$@" ; then')
+    print('  # PEBS may not actually be working even if the processor supports it')
+    print('  # (e.g., in a virtual machine). Trying to run without /p.')
+    print('  set +x')
+    print('  echo >&2 "Retrying without /p."')
+    print('  E="$(echo "${E}" | sed -e \'s/\/p/\//\')"')
+    print('  set -x')
+    print('  exec perf record -e $E -b "$@"')
+    print(' set +x')
+    print('fi')
 
 if cpufound == 0 and not args.all:
     sys.exit('CPU %s not found' % cpu)
diff --git a/gcc/config/i386/gcc-auto-profile b/gcc/config/i386/gcc-auto-profile
index 56f64cbff1f..5ab224b041b 100755
--- a/gcc/config/i386/gcc-auto-profile
+++ b/gcc/config/i386/gcc-auto-profile
@@ -33,8 +33,15 @@ if grep -q hypervisor /proc/cpuinfo ; then
   echo >&2 "Warning: branch profiling may not be functional in VMs"
 fi
 
-case `egrep -q "^cpu family\s*: 6" /proc/cpuinfo &&
-  egrep "^model\s*:" /proc/cpuinfo | head -n1` in
+case `grep -E -q "^cpu family\s*: 6" /proc/cpuinfo &&
+  grep -E "^model\s*:" /proc/cpuinfo | head -n1` in
+model*:\ 46|\
+model*:\ 30|\
+model*:\ 31|\
+model*:\ 26|\
+model*:\ 47|\
+model*:\ 37|\
+model*:\ 44) E="cpu/event=0x88,umask=0x40/$FLAGS" ;;
 model*:\ 55|\
 model*:\ 77|\
 model*:\ 76|\
@@ -43,6 +50,11 @@ model*:\ 95|\
 model*:\ 87|\
 model*:\ 133|\
 model*:\ 122) E="cpu/event=0xC4,umask=0xFE/p$FLAGS" ;;
+model*:\ 28|\
+model*:\ 38|\
+model*:\ 39|\
+model*:\ 54|\
+model*:\ 53) E="cpu/event=0x88,umask=0x41/$FLAGS" ;;
 model*:\ 42|\
 model*:\ 45|\
 model*:\ 58|\
@@ -63,23 +75,14 @@ model*:\ 165|\
 model*:\ 166|\
 model*:\ 85|\
 model*:\ 85) E="cpu/event=0xC4,umask=0x20/p$FLAGS" ;;
-model*:\ 46|\
-model*:\ 30|\
-model*:\ 31|\
-model*:\ 26|\
-model*:\ 47|\
-model*:\ 37|\
-model*:\ 44) E="cpu/event=0x88,umask=0x40/p$FLAGS" ;;
-model*:\ 28|\
-model*:\ 38|\
-model*:\ 39|\
-model*:\ 54|\
-model*:\ 53) E="cpu/event=0x88,umask=0x41/p$FLAGS" ;;
 model*:\ 126|\
 model*:\ 140|\
 model*:\ 141|\
+model*:\ 143|\
 model*:\ 106|\
 model*:\ 108) E="cpu/event=0xc4,umask=0x20/p$FLAGS" ;;
+model*:\ 134|\
+model*:\ 150) E="cpu/event=0xc4,umask=0xfe/p$FLAGS" ;;
 *)
 echo >&2 "Unknown CPU. Run contrib/gen_autofdo_event.py --all --script to update script."
 	exit 1 ;;
-- 
2.36.1

RE: [EXTERNAL] [PATCH] contrib: modernize gen_autofdo_event.py

[Eugene Rozenfeld]

The changes look good to me. Also adding Andi, the author of the script.

Eugene

-----Original Message-----
From: Gcc-patches <gcc-patches-bounces+erozen=microsoft.com@gcc.gnu.org> On Behalf Of Xi Ruoyao via Gcc-patches
Sent: Sunday, June 26, 2022 11:15 PM
To: gcc-patches@gcc.gnu.org
Subject: [EXTERNAL] [PATCH] contrib: modernize gen_autofdo_event.py

Python 2 has been EOL'ed for two years.  egrep has been deprecated for many years and the next grep release will start to print warning if it is used.

-E option may be unsupported by some non-POSIX grep implementations, but gcc-auto-profile won't work on non-Linux systems anyway.

contrib/ChangeLog:

	* gen_autofdo_event.py: Port to Python 3, and use grep -E
	instead of egrep.

gcc/ChangeLog:

	* config/i386/gcc-auto-profile: Regenerate.
---
 contrib/gen_autofdo_event.py     | 80 ++++++++++++++++----------------
 gcc/config/i386/gcc-auto-profile | 31 +++++++------
 2 files changed, 57 insertions(+), 54 deletions(-)

diff --git a/contrib/gen_autofdo_event.py b/contrib/gen_autofdo_event.py index 1eb6f1d6d85..7da2876530d 100755
--- a/contrib/gen_autofdo_event.py
+++ b/contrib/gen_autofdo_event.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python3
 # Generate Intel taken branches Linux perf event script for autofdo profiling.
 
 # Copyright (C) 2016 Free Software Foundation, Inc.
@@ -26,18 +26,19 @@
 # Requires internet (https) access. This may require setting up a proxy  # with export https_proxy=...
 #
-import urllib2
+import urllib.request
 import sys
 import json
 import argparse
 import collections
+import os
 
 baseurl = "https://download.01.org/perfmon"
 
-target_events = (u'BR_INST_RETIRED.NEAR_TAKEN',
-                 u'BR_INST_EXEC.TAKEN',
-                 u'BR_INST_RETIRED.TAKEN_JCC',
-                 u'BR_INST_TYPE_RETIRED.COND_TAKEN')
+target_events = ('BR_INST_RETIRED.NEAR_TAKEN',
+                 'BR_INST_EXEC.TAKEN',
+                 'BR_INST_RETIRED.TAKEN_JCC',
+                 'BR_INST_TYPE_RETIRED.COND_TAKEN')
 
 ap = argparse.ArgumentParser()
 ap.add_argument('--all', '-a', help='Print for all CPUs', action='store_true') @@ -71,47 +72,46 @@ def get_cpustr():
     return "%s-%d-%X" % tuple(cpu)[:3]
 
 def find_event(eventurl, model):
-    print >>sys.stderr, "Downloading", eventurl
-    u = urllib2.urlopen(eventurl)
+    print("Downloading", eventurl, file = sys.stderr)
+    u = urllib.request.urlopen(eventurl)
     events = json.loads(u.read())
     u.close()
 
     found = 0
     for j in events:
-        if j[u'EventName'] in target_events:
-            event = "cpu/event=%s,umask=%s/" % (j[u'EventCode'], j[u'UMask'])
-            if u'PEBS' in j and j[u'PEBS'] > 0:
+        if j['EventName'] in target_events:
+            event = "cpu/event=%s,umask=%s/" % (j['EventCode'], j['UMask'])
+            if 'PEBS' in j and int(j['PEBS']) > 0:
                 event += "p"
             if args.script:
                 eventmap[event].append(model)
             else:
-                print j[u'EventName'], "event for model", model, "is", event
+                print(j['EventName'], "event for model", model, "is", 
+ event)
             found += 1
     return found
 
 if not args.all:
-    cpu = get_cpu_str()
+    cpu = get_cpustr()
     if not cpu:
         sys.exit("Unknown CPU type")
 
 url = baseurl + "/mapfile.csv"
-print >>sys.stderr, "Downloading", url
-u = urllib2.urlopen(url)
+print("Downloading", url, file = sys.stderr) u = 
+urllib.request.urlopen(url)
 found = 0
 cpufound = 0
 for j in u:
-    n = j.rstrip().split(',')
+    n = j.rstrip().decode().split(',')
     if len(n) >= 4 and (args.all or n[0] == cpu) and n[3] == "core":
-        if args.all:
-            components = n[0].split("-")
-            model = components[2]
-            model = int(model, 16)
+        components = n[0].split("-")
+        model = components[2]
+        model = int(model, 16)
         cpufound += 1
         found += find_event(baseurl + n[2], model)
 u.close()
 
 if args.script:
-    print '''#!/bin/sh
+    print('''#!/bin/sh
 # Profile workload for gcc profile feedback (autofdo) using Linux perf.
 # Auto generated. To regenerate for new CPUs run  # contrib/gen_autofdo_event.py --script --all in gcc source @@ -146,27 +146,27 @@ if grep -q hypervisor /proc/cpuinfo ; then
   echo >&2 "Warning: branch profiling may not be functional in VMs"
 fi
 
-case `egrep -q "^cpu family\s*: 6" /proc/cpuinfo &&
-  egrep "^model\s*:" /proc/cpuinfo | head -n1` in'''
-    for event, mod in eventmap.iteritems():
+case `grep -E -q "^cpu family\s*: 6" /proc/cpuinfo &&
+  grep -E "^model\s*:" /proc/cpuinfo | head -n1` in''')
+    for event, mod in eventmap.items():
         for m in mod[:-1]:
-            print "model*:\ %s|\\" % m
-        print 'model*:\ %s) E="%s$FLAGS" ;;' % (mod[-1], event)
-    print '''*)
+            print("model*:\ %s|\\" % m)
+        print('model*:\ %s) E="%s$FLAGS" ;;' % (mod[-1], event))
+    print('''*)
 echo >&2 "Unknown CPU. Run contrib/gen_autofdo_event.py --all --script to update script."
-	exit 1 ;;'''
-    print "esac"
-    print "set -x"
-    print 'if ! perf record -e $E -b "$@" ; then'
-    print '  # PEBS may not actually be working even if the processor supports it'
-    print '  # (e.g., in a virtual machine). Trying to run without /p.'
-    print '  set +x'
-    print '  echo >&2 "Retrying without /p."'
-    print '  E="$(echo "${E}" | sed -e \'s/\/p/\//\')"'
-    print '  set -x'
-    print '  exec perf record -e $E -b "$@"'
-    print ' set +x'
-    print 'fi'
+	exit 1 ;;''')
+    print("esac")
+    print("set -x")
+    print('if ! perf record -e $E -b "$@" ; then')
+    print('  # PEBS may not actually be working even if the processor supports it')
+    print('  # (e.g., in a virtual machine). Trying to run without /p.')
+    print('  set +x')
+    print('  echo >&2 "Retrying without /p."')
+    print('  E="$(echo "${E}" | sed -e \'s/\/p/\//\')"')
+    print('  set -x')
+    print('  exec perf record -e $E -b "$@"')
+    print(' set +x')
+    print('fi')
 
 if cpufound == 0 and not args.all:
     sys.exit('CPU %s not found' % cpu)
diff --git a/gcc/config/i386/gcc-auto-profile b/gcc/config/i386/gcc-auto-profile
index 56f64cbff1f..5ab224b041b 100755
--- a/gcc/config/i386/gcc-auto-profile
+++ b/gcc/config/i386/gcc-auto-profile
@@ -33,8 +33,15 @@ if grep -q hypervisor /proc/cpuinfo ; then
   echo >&2 "Warning: branch profiling may not be functional in VMs"
 fi
 
-case `egrep -q "^cpu family\s*: 6" /proc/cpuinfo &&
-  egrep "^model\s*:" /proc/cpuinfo | head -n1` in
+case `grep -E -q "^cpu family\s*: 6" /proc/cpuinfo &&
+  grep -E "^model\s*:" /proc/cpuinfo | head -n1` in model*:\ 46|\ 
+model*:\ 30|\ model*:\ 31|\ model*:\ 26|\ model*:\ 47|\ model*:\ 37|\ 
+model*:\ 44) E="cpu/event=0x88,umask=0x40/$FLAGS" ;;
 model*:\ 55|\
 model*:\ 77|\
 model*:\ 76|\
@@ -43,6 +50,11 @@ model*:\ 95|\
 model*:\ 87|\
 model*:\ 133|\
 model*:\ 122) E="cpu/event=0xC4,umask=0xFE/p$FLAGS" ;;
+model*:\ 28|\
+model*:\ 38|\
+model*:\ 39|\
+model*:\ 54|\
+model*:\ 53) E="cpu/event=0x88,umask=0x41/$FLAGS" ;;
 model*:\ 42|\
 model*:\ 45|\
 model*:\ 58|\
@@ -63,23 +75,14 @@ model*:\ 165|\
 model*:\ 166|\
 model*:\ 85|\
 model*:\ 85) E="cpu/event=0xC4,umask=0x20/p$FLAGS" ;; -model*:\ 46|\ -model*:\ 30|\ -model*:\ 31|\ -model*:\ 26|\ -model*:\ 47|\ -model*:\ 37|\ -model*:\ 44) E="cpu/event=0x88,umask=0x40/p$FLAGS" ;; -model*:\ 28|\ -model*:\ 38|\ -model*:\ 39|\ -model*:\ 54|\ -model*:\ 53) E="cpu/event=0x88,umask=0x41/p$FLAGS" ;;  model*:\ 126|\  model*:\ 140|\  model*:\ 141|\
+model*:\ 143|\
 model*:\ 106|\
 model*:\ 108) E="cpu/event=0xc4,umask=0x20/p$FLAGS" ;;
+model*:\ 134|\
+model*:\ 150) E="cpu/event=0xc4,umask=0xfe/p$FLAGS" ;;
 *)
 echo >&2 "Unknown CPU. Run contrib/gen_autofdo_event.py --all --script to update script."
 	exit 1 ;;
--
2.36.1

Re: [EXTERNAL] [PATCH] contrib: modernize gen_autofdo_event.py

[Andi Kleen]

On 8/6/2022 1:07 AM, Eugene Rozenfeld wrote:
> The changes look good to me. Also adding Andi, the author of the script.


Looks all good to me too.


-Andi

RE: [EXTERNAL] [PATCH] contrib: modernize gen_autofdo_event.py

[Eugene Rozenfeld]

The patch is approved.

Eugene

-----Original Message-----
From: Andi Kleen <ak@linux.intel.com> 
Sent: Friday, August 05, 2022 11:29 PM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>; Xi Ruoyao <xry111@xry111.site>; gcc-patches@gcc.gnu.org
Subject: Re: [EXTERNAL] [PATCH] contrib: modernize gen_autofdo_event.py


On 8/6/2022 1:07 AM, Eugene Rozenfeld wrote:
> The changes look good to me. Also adding Andi, the author of the script.


Looks all good to me too.


-Andi

Re: [EXTERNAL] [PATCH] contrib: modernize gen_autofdo_event.py

[Xi Ruoyao]

On Fri, 2022-08-26 at 22:20 +0000, Eugene Rozenfeld wrote:
> The patch is approved.

Thanks, pushed r13-2234.
-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University