From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa1.mentor.iphmx.com (esa1.mentor.iphmx.com [68.232.129.153]) by sourceware.org (Postfix) with ESMTPS id 660313894402; Mon, 21 Jun 2021 08:37:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 660313894402 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com IronPort-SDR: w6EEnZMH3/T6QCWEZ9qwmJGVmCl3756yx/pJI0Prlz8GP2wR8eDygPhmrLeq3JLpviwVf4Ehzn rGPrW5pYhWkG2cZdfiJqMZ5c1mPsFjYvSVhN4npN+h8OTeYV5Sq+XtiJtk0RmA00uumycOIUaU w3FQq8ryuUKPB/aVZnu3t9nlHxB4nX6Uw0AXHq9DK+lo5RNGryb6GigNTSVsbtCz7iOWaR2VlQ mdCfBz2eBx/izz8J2me89AtOT7QnsFd0JKaP9HqgGGjRTjlKJmCZCkuxzay7EvjrImtX8qAr2L C34= X-IronPort-AV: E=Sophos;i="5.83,289,1616486400"; d="diff'?scan'208";a="65044326" Received: from orw-gwy-02-in.mentorg.com ([192.94.38.167]) by esa1.mentor.iphmx.com with ESMTP; 21 Jun 2021 00:37:44 -0800 IronPort-SDR: AMAj8mzpzt9mSOTlkabZOWQzMq8eDiuMqOnPSI2MsFF67VQxI4PnfaOE1AjUpnO++ruf0cIkN6 K3zQCYQ5LdIA9YT0gFbSQz9tLAIQfwgCiuPnnUZq6QLH2wxJ4DVHJu34KH180LJquWvHOsL6hr ViCkw1bLdBMywRo+GgpYTDU/mUssPBgNIrS/pWDi1idpIV93fDGpvzn8sur0abS5kDUmInlZks 4r9Zgzf5/ibtKnW8jJQuNQbNBJOYlvZRV99fPywPvBu+ivp2WjKpgM05EjC7PfmO3T3qKNKdbo O/Y= Subject: Re: [Patch, v2] contrib/mklog.py: Improve PR handling (was: Re: git gcc-commit-mklog doesn't extract PR number to ChangeLog) To: =?UTF-8?Q?Martin_Li=c5=a1ka?= , Martin Sebor , Jason Merrill CC: Jakub Jelinek , gcc Mailing List , Jonathan Wakely , gcc-patches References: <5f89ddc0-aed4-2c20-0979-dfafb29046ee@gmail.com> <20210610173005.GI7746@tucnak> <20210610190941.GJ7746@tucnak> <58b63929-01f5-038c-931c-9ff8349d9f95@gmail.com> <71b4a023-efb2-6c6a-9ced-93cce7c96540@gmail.com> <3c66efd4-eb5e-f2bb-6138-4126b5909c9c@codesourcery.com> <80fee54a-c007-a62b-20e2-41bb54a2bd00@suse.cz> From: Tobias Burnus Message-ID: <870491ea-a0bf-3ac3-14d3-1aec951a356c@codesourcery.com> Date: Mon, 21 Jun 2021 10:37:37 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <80fee54a-c007-a62b-20e2-41bb54a2bd00@suse.cz> Content-Type: multipart/mixed; boundary="------------8CF229CBEBED5F46F3C6A101" Content-Language: en-US X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) To svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, GIT_PATCH_0, HEADER_FROM_DIFFERENT_DOMAINS, KAM_DMARC_STATUS, KAM_SHORT, NICE_REPLY_A, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jun 2021 08:37:47 -0000 --------------8CF229CBEBED5F46F3C6A101 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: quoted-printable On 21.06.21 10:09, Martin Li=C5=A1ka wrote: > $ pytest test_mklog.py > FAILED test_mklog.py::TestMklog::test_sorting - AssertionError: assert > '\n\tPR 50209...New test.\n\n' =3D=3D 'gcc/ChangeLo...New test.\n\n' Aha, missed that there is indeed a testsuite - nice! > $ flake8 mklog.py > mklog.py:187:23: Q000 Remove bad quotes I have now filled: https://bugs.launchpad.net/ubuntu/+source/python-pytest-flake8/+bug/1933075 >> + # PR number in the file name >> + fname =3D os.path.basename(file.path) > > This is a dead code. > >> + fname =3D os.path.splitext(fname)[0] >> + m =3D pr_filename_regex.search(fname) It does not look like dead code to me. >> + parser.add_argument('-b', '--pr-numbers', action=3D'append', >> + help=3D'Add the specified PRs (comma separated)= ') > > Do we really want to support '-b 1 -b 2' and also -b '1,2' formats? > Seems to me quite > complicated. I don't have a strong opinion. I started with '-b 123,245', believing that the syntax is fine. But then I realized that without '-p' specifying multiple '-b' looks better by having multiple '-b' if 'PR /' (needed for -p as the string is than taken as is). Thus, I ended up supporting either variant. But I also happily drop the ',' support. Change: One quote change, one test_mklog update. Tobias ----------------- Mentor Graphics (Deutschland) GmbH, Arnulfstrasse 201, 80634 M=C3=BCnchen R= egistergericht M=C3=BCnchen HRB 106955, Gesch=C3=A4ftsf=C3=BChrer: Thomas H= eurung, Frank Th=C3=BCrauf --------------8CF229CBEBED5F46F3C6A101 Content-Type: text/x-patch; charset="UTF-8"; name="mklog-p-v3.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="mklog-p-v3.diff" contrib/mklog.py: Improve PR handling Co-authored-by: Martin Sebor contrib/ChangeLog: * mklog.py (bugzilla_url): Fetch also component. (pr_filename_regex): New. (get_pr_titles): Update PR string with correct format and component. (generate_changelog): Take additional PRs; extract PR from the filename. (__main__): Add -b/--pr-numbers argument. * test_mklog.py (EXPECTED4): Update to expect a PR for the new file. contrib/mklog.py | 41 ++++++++++++++++++++++++++++++++--------- contrib/test_mklog.py | 3 +++ 2 files changed, 35 insertions(+), 9 deletions(-) diff --git a/contrib/mklog.py b/contrib/mklog.py index 1f59055e723..e49d14d0859 100755 --- a/contrib/mklog.py +++ b/contrib/mklog.py @@ -42,6 +42,7 @@ pr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PPR [a-z+-]+\/[0-9]+)') prnum_regex = re.compile(r'PR (?P[a-z+-]+)/(?P[0-9]+)') dr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PDR [0-9]+)') dg_regex = re.compile(r'{\s+dg-(error|warning)') +pr_filename_regex = re.compile(r'(^|[\W_])[Pp][Rr](?P\d{4,})') identifier_regex = re.compile(r'^([a-zA-Z0-9_#].*)') comment_regex = re.compile(r'^\/\*') struct_regex = re.compile(r'^(class|struct|union|enum)\s+' @@ -52,7 +53,7 @@ fn_regex = re.compile(r'([a-zA-Z_][^()\s]*)\s*\([^*]') template_and_param_regex = re.compile(r'<[^<>]*>') md_def_regex = re.compile(r'\(define.*\s+"(.*)"') bugzilla_url = 'https://gcc.gnu.org/bugzilla/rest.cgi/bug?id=%s&' \ - 'include_fields=summary' + 'include_fields=summary,component' function_extensions = {'.c', '.cpp', '.C', '.cc', '.h', '.inc', '.def', '.md'} @@ -118,20 +119,23 @@ def sort_changelog_files(changed_file): def get_pr_titles(prs): - output = '' - for pr in prs: + output = [] + for idx, pr in enumerate(prs): pr_id = pr.split('/')[-1] r = requests.get(bugzilla_url % pr_id) bugs = r.json()['bugs'] if len(bugs) == 1: - output += '%s - %s\n' % (pr, bugs[0]['summary']) - print(output) + prs[idx] = 'PR %s/%s' % (bugs[0]['component'], pr_id) + out = '%s - %s\n' % (prs[idx], bugs[0]['summary']) + if out not in output: + output.append(out) if output: - output += '\n' - return output + output.append('') + return '\n'.join(output) -def generate_changelog(data, no_functions=False, fill_pr_titles=False): +def generate_changelog(data, no_functions=False, fill_pr_titles=False, + additional_prs=None): changelogs = {} changelog_list = [] prs = [] @@ -139,6 +143,8 @@ def generate_changelog(data, no_functions=False, fill_pr_titles=False): diff = PatchSet(data) global firstpr + if additional_prs: + prs = [pr for pr in additional_prs if pr not in prs] for file in diff: # skip files that can't be parsed if file.path == '/dev/null': @@ -154,21 +160,33 @@ def generate_changelog(data, no_functions=False, fill_pr_titles=False): # Only search first ten lines as later lines may # contains commented code which a note that it # has not been tested due to a certain PR or DR. + this_file_prs = [] for line in list(file)[0][0:10]: m = pr_regex.search(line.value) if m: pr = m.group('pr') if pr not in prs: prs.append(pr) + this_file_prs.append(pr.split('/')[-1]) else: m = dr_regex.search(line.value) if m: dr = m.group('dr') if dr not in prs: prs.append(dr) + this_file_prs.append(dr.split('/')[-1]) elif dg_regex.search(line.value): # Found dg-warning/dg-error line break + # PR number in the file name + fname = os.path.basename(file.path) + fname = os.path.splitext(fname)[0] + m = pr_filename_regex.search(fname) + if m: + pr = m.group('pr') + pr2 = 'PR ' + pr + if pr not in this_file_prs and pr2 not in prs: + prs.append(pr2) if prs: firstpr = prs[0] @@ -286,6 +304,8 @@ if __name__ == '__main__': parser = argparse.ArgumentParser(description=help_message) parser.add_argument('input', nargs='?', help='Patch file (or missing, read standard input)') + parser.add_argument('-b', '--pr-numbers', action='append', + help='Add the specified PRs (comma separated)') parser.add_argument('-s', '--no-functions', action='store_true', help='Do not generate function names in ChangeLogs') parser.add_argument('-p', '--fill-up-bug-titles', action='store_true', @@ -308,8 +328,11 @@ if __name__ == '__main__': if args.update_copyright: update_copyright(data) else: + pr_numbers = args.pr_numbers + if pr_numbers: + pr_numbers = [b for i in args.pr_numbers for b in i.split(',')] output = generate_changelog(data, args.no_functions, - args.fill_up_bug_titles) + args.fill_up_bug_titles, pr_numbers) if args.changelog: lines = open(args.changelog).read().split('\n') start = list(takewhile(lambda l: not l.startswith('#'), lines)) diff --git a/contrib/test_mklog.py b/contrib/test_mklog.py index a0670dac119..f5e9ecd577c 100755 --- a/contrib/test_mklog.py +++ b/contrib/test_mklog.py @@ -240,6 +240,9 @@ index 4ad78c1f77b..6687b368038 100644 ''' EXPECTED4 = '''\ + + PR 50209 + gcc/ChangeLog: * ipa-icf.c: --------------8CF229CBEBED5F46F3C6A101--