From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa4.mentor.iphmx.com (esa4.mentor.iphmx.com [68.232.137.252]) by sourceware.org (Postfix) with ESMTPS id 691A73850416; Mon, 21 Jun 2021 13:26:16 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 691A73850416 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com IronPort-SDR: w9TQndqekkGGPFHT6d++EHHAHIZXK1dMi/4QbyKNYiBuUS6rFcWjE3G37J6DaBRlcpqoPd3Ia4 qmT7au7TyQwf+WjqeJMWWU23C431ChbfMzCtbn3Gek/Y/INbegVbOSBmO37gczgKns+BEUPR9J 5EtxcTDH/eBLbTxfUfqwh+1qLivCHyj2FgoBuZ346hoyO7ZqRbYYk7fwh1S+tj/b26LK/m6biA TK+8C3SlULhQp7v/e9OgZ01AjoKE6F31gu6HVnuvdYq4nXaJt/e1ONExVrKINZ1UcySz8vs/8E cE4= X-IronPort-AV: E=Sophos;i="5.83,289,1616486400"; d="diff'?scan'208";a="62818450" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa4.mentor.iphmx.com with ESMTP; 21 Jun 2021 05:26:15 -0800 IronPort-SDR: xcMxaWXLcnD9n3JEo417nn5P3/rpBoeNs2dzEYVmVWJ9/ocXBBpdAY7bZ2FPJoA2wq95wzB7uW XxfA4eO2+6XxLZhAQGkKorksTgysM4k7klPRHeYU47p79eCPvOcp7SB30+zVIgrSKGTHqDqJB0 d6QW7Q3kSf0lWf9gED5guKHMkLuN4RnfkaqtwZKpEnUjZeFIkzFoUOD0hAJopIh8z09HgCI8dA tNp6080tzure+KcIhpycm5fdAeby9bIzP/fpj7F0+L3UPTQI9tOQED68Io2zsFu/vCWDheD3ol zQY= Subject: Re: [Patch, v2] contrib/mklog.py: Improve PR handling (was: Re: git gcc-commit-mklog doesn't extract PR number to ChangeLog) To: =?UTF-8?Q?Martin_Li=c5=a1ka?= , Martin Sebor , Jason Merrill CC: Jakub Jelinek , gcc Mailing List , Jonathan Wakely , gcc-patches References: <20210610190941.GJ7746@tucnak> <58b63929-01f5-038c-931c-9ff8349d9f95@gmail.com> <71b4a023-efb2-6c6a-9ced-93cce7c96540@gmail.com> <3c66efd4-eb5e-f2bb-6138-4126b5909c9c@codesourcery.com> <80fee54a-c007-a62b-20e2-41bb54a2bd00@suse.cz> <870491ea-a0bf-3ac3-14d3-1aec951a356c@codesourcery.com> <0e604906-df5a-455b-9a26-4c5519e261b6@suse.cz> From: Tobias Burnus Message-ID: <26c6e47d-38e3-83c2-36ce-1786510e00ed@codesourcery.com> Date: Mon, 21 Jun 2021 15:26:07 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <0e604906-df5a-455b-9a26-4c5519e261b6@suse.cz> Content-Type: multipart/mixed; boundary="------------48C782EC3EDB44D031D6AD67" Content-Language: en-US X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) To svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, GIT_PATCH_0, HEADER_FROM_DIFFERENT_DOMAINS, KAM_DMARC_STATUS, KAM_SHORT, NICE_REPLY_A, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jun 2021 13:26:18 -0000 --------------48C782EC3EDB44D031D6AD67 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: quoted-printable Now committed as r12-1700-gedf0c3ffb59d75c11e05bc722432dc554e275c72 / as attached. (We had some follow-up discussion on IRC.) On 21.06.21 14:53, Martin Li=C5=A1ka wrote: >>>> + # PR number in the file name >>>> + fname =3D os.path.basename(file.path) >>> >>> This is a dead code. >>> >>>> + fname =3D os.path.splitext(fname)[0] >>>> + m =3D pr_filename_regex.search(fname) (Meant was the 'splitext' line - it is dead code as the re.search globs all digits after 'pr' and then stops, ignoring the rest, including file extensions. =E2=80=93 I first thought it referred to the basename line, whi= ch confused me.) >>>> + parser.add_argument('-b', '--pr-numbers', action=3D'append', >>>> + help=3D'Add the specified PRs (comma >>>> separated)') >>> >>> Do we really want to support '-b 1 -b 2' and also -b '1,2' formats? >>> Seems to me quite >>> complicated. > [...] > I would start with -b 1,2,3,4 syntax. It will be likely easier for git > alias integration. Done so. I note that argparse permits '-d dir1 -d dir2 -b bug1 -b bug2' which then only keeps the dir2 as directory and bug2 as bug without printing an error (or warning) for ignoring dir1 and bug1. Tobias ----------------- Mentor Graphics (Deutschland) GmbH, Arnulfstrasse 201, 80634 M=C3=BCnchen R= egistergericht M=C3=BCnchen HRB 106955, Gesch=C3=A4ftsf=C3=BChrer: Thomas H= eurung, Frank Th=C3=BCrauf --------------48C782EC3EDB44D031D6AD67 Content-Type: text/x-patch; charset="UTF-8"; name="committed.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="committed.diff" commit edf0c3ffb59d75c11e05bc722432dc554e275c72 Author: Tobias Burnus Date: Mon Jun 21 15:17:22 2021 +0200 contrib/mklog.py: Improve PR handling Co-authored-by: Martin Sebor contrib/ChangeLog: * mklog.py (bugzilla_url): Fetch also component. (pr_filename_regex): New. (get_pr_titles): Update PR string with correct format and component. (generate_changelog): Take additional PRs; extract PR from the filename. (__main__): Add -b/--pr-numbers argument. * test_mklog.py (EXPECTED4): Update to expect a PR for the new file. --- contrib/mklog.py | 38 +++++++++++++++++++++++++++++--------- contrib/test_mklog.py | 3 +++ 2 files changed, 32 insertions(+), 9 deletions(-) diff --git a/contrib/mklog.py b/contrib/mklog.py index 1f59055e723..0b434f67971 100755 --- a/contrib/mklog.py +++ b/contrib/mklog.py @@ -42,6 +42,7 @@ pr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PPR [a-z+-]+\/[0-9]+)') prnum_regex = re.compile(r'PR (?P[a-z+-]+)/(?P[0-9]+)') dr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PDR [0-9]+)') dg_regex = re.compile(r'{\s+dg-(error|warning)') +pr_filename_regex = re.compile(r'(^|[\W_])[Pp][Rr](?P\d{4,})') identifier_regex = re.compile(r'^([a-zA-Z0-9_#].*)') comment_regex = re.compile(r'^\/\*') struct_regex = re.compile(r'^(class|struct|union|enum)\s+' @@ -52,7 +53,7 @@ fn_regex = re.compile(r'([a-zA-Z_][^()\s]*)\s*\([^*]') template_and_param_regex = re.compile(r'<[^<>]*>') md_def_regex = re.compile(r'\(define.*\s+"(.*)"') bugzilla_url = 'https://gcc.gnu.org/bugzilla/rest.cgi/bug?id=%s&' \ - 'include_fields=summary' + 'include_fields=summary,component' function_extensions = {'.c', '.cpp', '.C', '.cc', '.h', '.inc', '.def', '.md'} @@ -118,20 +119,23 @@ def sort_changelog_files(changed_file): def get_pr_titles(prs): - output = '' - for pr in prs: + output = [] + for idx, pr in enumerate(prs): pr_id = pr.split('/')[-1] r = requests.get(bugzilla_url % pr_id) bugs = r.json()['bugs'] if len(bugs) == 1: - output += '%s - %s\n' % (pr, bugs[0]['summary']) - print(output) + prs[idx] = 'PR %s/%s' % (bugs[0]['component'], pr_id) + out = '%s - %s\n' % (prs[idx], bugs[0]['summary']) + if out not in output: + output.append(out) if output: - output += '\n' - return output + output.append('') + return '\n'.join(output) -def generate_changelog(data, no_functions=False, fill_pr_titles=False): +def generate_changelog(data, no_functions=False, fill_pr_titles=False, + additional_prs=None): changelogs = {} changelog_list = [] prs = [] @@ -139,6 +143,8 @@ def generate_changelog(data, no_functions=False, fill_pr_titles=False): diff = PatchSet(data) global firstpr + if additional_prs: + prs = [pr for pr in additional_prs if pr not in prs] for file in diff: # skip files that can't be parsed if file.path == '/dev/null': @@ -154,21 +160,32 @@ def generate_changelog(data, no_functions=False, fill_pr_titles=False): # Only search first ten lines as later lines may # contains commented code which a note that it # has not been tested due to a certain PR or DR. + this_file_prs = [] for line in list(file)[0][0:10]: m = pr_regex.search(line.value) if m: pr = m.group('pr') if pr not in prs: prs.append(pr) + this_file_prs.append(pr.split('/')[-1]) else: m = dr_regex.search(line.value) if m: dr = m.group('dr') if dr not in prs: prs.append(dr) + this_file_prs.append(dr.split('/')[-1]) elif dg_regex.search(line.value): # Found dg-warning/dg-error line break + # PR number in the file name + fname = os.path.basename(file.path) + m = pr_filename_regex.search(fname) + if m: + pr = m.group('pr') + pr2 = 'PR ' + pr + if pr not in this_file_prs and pr2 not in prs: + prs.append(pr2) if prs: firstpr = prs[0] @@ -286,6 +303,9 @@ if __name__ == '__main__': parser = argparse.ArgumentParser(description=help_message) parser.add_argument('input', nargs='?', help='Patch file (or missing, read standard input)') + parser.add_argument('-b', '--pr-numbers', action='store', + type=lambda arg: arg.split(','), nargs="?", + help='Add the specified PRs (comma separated)') parser.add_argument('-s', '--no-functions', action='store_true', help='Do not generate function names in ChangeLogs') parser.add_argument('-p', '--fill-up-bug-titles', action='store_true', @@ -309,7 +329,7 @@ if __name__ == '__main__': update_copyright(data) else: output = generate_changelog(data, args.no_functions, - args.fill_up_bug_titles) + args.fill_up_bug_titles, args.pr_numbers) if args.changelog: lines = open(args.changelog).read().split('\n') start = list(takewhile(lambda l: not l.startswith('#'), lines)) diff --git a/contrib/test_mklog.py b/contrib/test_mklog.py index a0670dac119..f5e9ecd577c 100755 --- a/contrib/test_mklog.py +++ b/contrib/test_mklog.py @@ -240,6 +240,9 @@ index 4ad78c1f77b..6687b368038 100644 ''' EXPECTED4 = '''\ + + PR 50209 + gcc/ChangeLog: * ipa-icf.c: --------------48C782EC3EDB44D031D6AD67--