Re: [RFC] Updating patchwork patches on commit

[Girish Joshi <girish946@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1934 bytes --]

Hello all,
I tried a couple of very basic scripts for this. (I know that there
are a lot of improvements needed there.)
I was able to merge 336 series out of 1114.

As "git-pw patch apply <id>" gives "Resource not found" for the older
patches. So right now only series are applied to a branch.
Here is how the scripts work.
We have two scripts, "get-patches.py" and "apply-patches.py" (we can
change the names of course).
"get-patches.py" reads the patches/series starting from page1 to page
100 (currently) in csv format and dumps it to stdout. This output is
piped to the second script "apply-patches.py" which tries to apply
each series/patch to the branch.
In the end we get two files as an output "merged.txt" and
"unmerged.txt" containing the IDs for merged and unmerged series
respectively.
Currently these files are placed in the current directory, I'll change
it to /tmp or something else in the next patch.

Just to have it here, to apply patches using these two scripts

    $ python scripts/get-patches.py series | python
scripts/apply-patches.py series apply

I'm still not sure about what happens to the older patches, do they
get applied from "git-pw series apply" or not (I'm looking into it)
because the newer ones do get applied.

Is it going in the right direction? Please share your thoughts.
Thanks.

Girish Joshi
girishjoshi.io

On Tue, Dec 8, 2020 at 3:40 PM Siddhesh Poyarekar <siddhesh@gotplt.org> wrote:
>
> On 12/8/20 2:38 PM, Andreas Schwab wrote:
> >> It doesn't; it has it's own hashing function where it normalizes spaces
> >> and newline chars to avoid false negatives.
> >
> > Like git patch-id?
> >
>
> Yeah, except that it (AFAICT) doesn't order the diff input like git
> patch-id does :)  I suppose I could check if they're willing to add a
> dependency on git for this and drop their custom hasher or at least
> provide a supported way to add a different hashing function or program.
>
> Siddhesh

[-- Attachment #2: get-patches.py --]
[-- Type: text/x-python, Size: 286 bytes --]

#!python3
import os
import sys

type_ = sys.argv[1]

command = "git-pw {0} list --page {1} -f csv"
if type_ == 'patch':
    command+= " --state 'new'"

for i in range(1, 100):
    # print(command.format(type_, i))
    ret = os.system(command.format(type_, i))
    if ret:
        break

[-- Attachment #3: apply-patches.py --]
[-- Type: text/x-python, Size: 3382 bytes --]

#!python3
import re
import csv
import sys
import shlex
import subprocess as sp

# import time

prune_warining = "warning: There are too many unreachable loose objects; run 'git prune' to remove them."

# List for series entries
series = []

# These lists will contain merged and unmnerged series data.
merged = []
unmerged = []

# option that we will be operating upon, series or the patch
# this is the command line argument to git-pw
# for example "git-pw patch apply 12345" or "git-pw series apply "12356"
type_ = "series"


# Get the csv data from stdin
csv_data = []
for line in sys.stdin:
    if not '"ID","Date","Name","Version","Submitter"' in line:
        print(line)
        csv_data.append(line.strip())


# parse the csv entries
def read_rows(csvfile):
    spamreader = csv.reader(csvfile, delimiter=",", quotechar='"')
    for row in spamreader:
        # print(row)
        if not row:
            return
        if row and row[1] != "ID":
            series.append(row)


def get_output(cmnd):
    """
    Execute command and check the output, if git throws a warning saying
    "warning: There are too many unreachable loose objects; run 'git prune' to remove them."
    `git prune` will be executed. otherwise output will be printed and exit code
    will be returned.
    """
    try:
        output = sp.check_output(
            cmnd, stderr=sp.STDOUT, shell=True, universal_newlines=True
        )
    except sp.CalledProcessError as exc:
        print("Status : FAIL", exc.returncode, exc.output)
        return exc.returncode
    else:
        print("Output: \n{}\n".format(output))
    if prune_warining in output:
        print("running git prune")
        get_output("git prune")
    return 0


def write_file(filename, list_):
    """
    This function is used to write the IDs for patches/series that
    are merged/unmerged after we have processed everything.
    """
    with open(filename, "w") as f:
        for i in list_:
            f.write(i[0] + "\n")


if __name__ == "__main__":

    read_rows(csv_data)

    # this is crappy, it will be replaced by arg parser.
    if len(sys.argv) >= 3:
        if sys.argv[1] == "series" or sys.argv[1] == "patch":
            type_ = sys.argv[1]

        if sys.argv[2] == "apply":
            print("applying ", type_)
            if series:
                for i in series:
                    try:
                        print("trying to apply:", i[0], i[1], i[2])
                        if i[0] == "ID":
                            pass
                        ret = get_output(f"git-pw {type_} apply {i[0]}")
                        print("git exit code: ", ret)

                        # time.sleep(0.5)
                        if ret:
                            # if `git-pw patch/series apply <id>` fails
                            # resetting to HEAD
                            print("resetting...")
                            get_output("git reset --hard HEAD")
                            get_output("git am --abort")
                            unmerged.append(i)
                        else:
                            merged.append(i)
                    except KeyboardInterrupt as ke:
                        break

    print("total merged: {0}, total unmerged {1}".format(len(merged), len(unmerged)))
    write_file("merged.txt", merged)
    write_file("unmerged.txt", unmerged)