From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa3.mentor.iphmx.com (esa3.mentor.iphmx.com [68.232.137.180]) by sourceware.org (Postfix) with ESMTPS id B57AA3886C6A; Mon, 21 Jun 2021 07:54:49 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B57AA3886C6A Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com IronPort-SDR: gpa0c2ujlPXS517ffleMYz8RvCx6mD+CFUQel2VboR3LH5p4tWGHrdXaarX5ksRCycm/MImjVH OkFEZ42u7ZHe1S6GOVOTFtbtp1uZ0G9zN2xvNh0obW+iKLj0FIbqVlW9vvWk2iW9BSftuIHJ9w 5CU3rtQwP/tPqDLoLro415m/WAEqAAMOGaxvC2fch60qxWNSU9JvY9hpxa5Moknkr/IMZDroz0 paKUFx29AtYm2MAWtdf2ekonapt92Shq7z4x1biwMNHZyO1v0aY1BBpCBOAWjwMcJVJo+oYKPg dnU= X-IronPort-AV: E=Sophos;i="5.83,289,1616486400"; d="diff'?scan'208";a="62592604" Received: from orw-gwy-02-in.mentorg.com ([192.94.38.167]) by esa3.mentor.iphmx.com with ESMTP; 20 Jun 2021 23:54:48 -0800 IronPort-SDR: WZRwT8j0SI0i7ENNUN0fBKeEhVvfSqEO2NucEmgJntnVD6qfb9lJgYeBroozG9zfhG/mmVUUUh 6/hDwzqgOrE4EmOasITELZeYytFnHxb83nqh0Z2cPxc3gtrQXm6UVIjzoUNXgN8sDI4D/JIAKW iGsAfTWKol+84KG8UZiwV1M642xw9vXwScTdJYuZDO+5qm9mFOtrKWLWG+giMMHvFfLgLTvCl4 4yof6oUgqWj2Jg+Hidir6VoZE2gIJ+rzHfOOwFGTNuI7XqXPJ7U1dSUKybeugZYAVnynGI2b6d rGQ= Subject: [Patch, v2] contrib/mklog.py: Improve PR handling (was: Re: git gcc-commit-mklog doesn't extract PR number to ChangeLog) To: Martin Sebor , Jason Merrill , =?UTF-8?Q?Martin_Li=c5=a1ka?= CC: Jakub Jelinek , gcc Mailing List , Jonathan Wakely , gcc-patches References: <3228435b-aba0-6157-3266-c0f025822829@gmail.com> <5f89ddc0-aed4-2c20-0979-dfafb29046ee@gmail.com> <20210610173005.GI7746@tucnak> <20210610190941.GJ7746@tucnak> <58b63929-01f5-038c-931c-9ff8349d9f95@gmail.com> <71b4a023-efb2-6c6a-9ced-93cce7c96540@gmail.com> From: Tobias Burnus Message-ID: <3c66efd4-eb5e-f2bb-6138-4126b5909c9c@codesourcery.com> Date: Mon, 21 Jun 2021 09:54:41 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <71b4a023-efb2-6c6a-9ced-93cce7c96540@gmail.com> Content-Type: multipart/mixed; boundary="------------40A3BEC54A2887EF85D639F7" Content-Language: en-US X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: SVR-IES-MBX-03.mgc.mentorg.com (139.181.222.3) To svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, GIT_PATCH_0, HEADER_FROM_DIFFERENT_DOMAINS, KAM_DMARC_STATUS, KAM_SHORT, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jun 2021 07:54:51 -0000 --------------40A3BEC54A2887EF85D639F7 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: quoted-printable On 17.06.21 02:17, Martin Sebor via Gcc wrote: > @@ -147,6 +152,12 @@ def generate_changelog(data, no_functions=3DFalse, f= ill_pr_titles=3DFalse): > > # Extract PR entries from newly added tests > if 'testsuite' in file.path and file.is_added_file: > + name =3D os.path.basename(file.path) > + name =3D os.path.splitext(name)[0] > + if name.startswith("pr"): > + name =3D name[2:] > + name =3D "PR " + name > + prs.append(name) I think you need a regular expression to extract the PR =E2=80=93 as it wil= l both match too much and to little. We have file names such as: * libstdc++-pr91488.C (other prefix) * PR37039.f90 (capitalized PR) * pr98218-1.C (suffix with '-') * pr40724_1.f (suffix with '_') * pr101023a.C (suffix with a letter) But otherwise, I like that idea. * * * Changes in my patch compared to v1: - (From Martin's patch:) Extract the PR from new-files file name (using pattern matching), but only take the PR if the PR wasn't found in the file as PR comment. (The latter happens, e.g., with b376b1ef389.) - Avoid printing the same PR multiple times as summary line (duplicates occur due to 'PR 134' vs. 'PR comp/123' vs. 'PR othercomp/123') =E2=80=94 This does not avoid all issues but at leas= t some. If this becomes a real world issue, we can try harder. OK to commit this one? =E2=80=94 Comments? * * * I did leave out other changes as they seem to be less clear cut, and which can be still be handled as follow up. Like: - Adding 'Resolves:' (as in some cases it only resolves part of the PR) - ... other changes/patches I missed. (This thread has too many emails.) In particular, if ^PR / - .... is accepted by gcc-commit/, then there is no need to list the PRs individually later on. But currently, it is still required. * * * Cross ref: * v1 of my patch was at https://gcc.gnu.org/pipermail/gcc/2021-June/236498.html * Discussion of the -b option is at https://gcc.gnu.org/pipermail/gcc/2021-June/236519.html * Martin S's patch (partially quoted above) is at https://gcc.gnu.org/pipermail/gcc/2021-June/236460.html Tobias ----------------- Mentor Graphics (Deutschland) GmbH, Arnulfstrasse 201, 80634 M=C3=BCnchen R= egistergericht M=C3=BCnchen HRB 106955, Gesch=C3=A4ftsf=C3=BChrer: Thomas H= eurung, Frank Th=C3=BCrauf --------------40A3BEC54A2887EF85D639F7 Content-Type: text/x-patch; charset="UTF-8"; name="mklog-p-v2.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="mklog-p-v2.diff" contrib/mklog.py: Improve PR handling Co-authored-by: Martin Sebor contrib/ChangeLog: * mklog.py (bugzilla_url): Fetch also component. (pr_filename_regex): New. (get_pr_titles): Update PR string with correct format and component. (generate_changelog): Take additional PRs; extract PR from the filename. (__main__): Add -b/--pr-numbers argument. contrib/mklog.py | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/contrib/mklog.py b/contrib/mklog.py index 1f59055e723..bba6c1a0e1a 100755 --- a/contrib/mklog.py +++ b/contrib/mklog.py @@ -42,6 +42,7 @@ pr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PPR [a-z+-]+\/[0-9]+)') prnum_regex = re.compile(r'PR (?P[a-z+-]+)/(?P[0-9]+)') dr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PDR [0-9]+)') dg_regex = re.compile(r'{\s+dg-(error|warning)') +pr_filename_regex = re.compile(r'(^|[\W_])[Pp][Rr](?P\d{4,})') identifier_regex = re.compile(r'^([a-zA-Z0-9_#].*)') comment_regex = re.compile(r'^\/\*') struct_regex = re.compile(r'^(class|struct|union|enum)\s+' @@ -52,7 +53,7 @@ fn_regex = re.compile(r'([a-zA-Z_][^()\s]*)\s*\([^*]') template_and_param_regex = re.compile(r'<[^<>]*>') md_def_regex = re.compile(r'\(define.*\s+"(.*)"') bugzilla_url = 'https://gcc.gnu.org/bugzilla/rest.cgi/bug?id=%s&' \ - 'include_fields=summary' + 'include_fields=summary,component' function_extensions = {'.c', '.cpp', '.C', '.cc', '.h', '.inc', '.def', '.md'} @@ -118,20 +119,23 @@ def sort_changelog_files(changed_file): def get_pr_titles(prs): - output = '' - for pr in prs: + output = [] + for idx, pr in enumerate(prs): pr_id = pr.split('/')[-1] r = requests.get(bugzilla_url % pr_id) bugs = r.json()['bugs'] if len(bugs) == 1: - output += '%s - %s\n' % (pr, bugs[0]['summary']) - print(output) + prs[idx] = 'PR %s/%s' % (bugs[0]['component'], pr_id) + out = '%s - %s\n' % (prs[idx], bugs[0]['summary']) + if out not in output: + output.append(out) if output: - output += '\n' - return output + output.append('') + return '\n'.join(output) -def generate_changelog(data, no_functions=False, fill_pr_titles=False): +def generate_changelog(data, no_functions=False, fill_pr_titles=False, + additional_prs=None): changelogs = {} changelog_list = [] prs = [] @@ -139,6 +143,8 @@ def generate_changelog(data, no_functions=False, fill_pr_titles=False): diff = PatchSet(data) global firstpr + if additional_prs: + prs = [pr for pr in additional_prs if pr not in prs] for file in diff: # skip files that can't be parsed if file.path == '/dev/null': @@ -154,21 +160,33 @@ def generate_changelog(data, no_functions=False, fill_pr_titles=False): # Only search first ten lines as later lines may # contains commented code which a note that it # has not been tested due to a certain PR or DR. + this_file_prs = [] for line in list(file)[0][0:10]: m = pr_regex.search(line.value) if m: pr = m.group('pr') if pr not in prs: prs.append(pr) + this_file_prs.append(pr.split('/')[-1]) else: m = dr_regex.search(line.value) if m: dr = m.group('dr') if dr not in prs: prs.append(dr) + this_file_prs.append(dr.split('/')[-1]) elif dg_regex.search(line.value): # Found dg-warning/dg-error line break + # PR number in the file name + fname = os.path.basename(file.path) + fname = os.path.splitext(fname)[0] + m = pr_filename_regex.search(fname) + if m: + pr = m.group('pr') + pr2 = "PR " + pr + if pr not in this_file_prs and pr2 not in prs: + prs.append(pr2) if prs: firstpr = prs[0] @@ -286,6 +304,8 @@ if __name__ == '__main__': parser = argparse.ArgumentParser(description=help_message) parser.add_argument('input', nargs='?', help='Patch file (or missing, read standard input)') + parser.add_argument('-b', '--pr-numbers', action='append', + help='Add the specified PRs (comma separated)') parser.add_argument('-s', '--no-functions', action='store_true', help='Do not generate function names in ChangeLogs') parser.add_argument('-p', '--fill-up-bug-titles', action='store_true', @@ -308,8 +328,11 @@ if __name__ == '__main__': if args.update_copyright: update_copyright(data) else: + pr_numbers = args.pr_numbers + if pr_numbers: + pr_numbers = [b for i in args.pr_numbers for b in i.split(',')] output = generate_changelog(data, args.no_functions, - args.fill_up_bug_titles) + args.fill_up_bug_titles, pr_numbers) if args.changelog: lines = open(args.changelog).read().split('\n') start = list(takewhile(lambda l: not l.startswith('#'), lines)) --------------40A3BEC54A2887EF85D639F7--