Re: [RFC] Updating patchwork patches on commit

[Girish Joshi <girish946@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1480 bytes --]

I've created this[1] script to go through all available series and get
the patches that do not belong to any one of them.
It dumps a json containing individual patch ids in /tmp directory.
This script can be merged with the previous one "apply-patches.py".
I'll do that soon.
Currently we have around 106 individual patches with the state "new"
that do not belong to any of the series.

Girish Joshi
girishjoshi.io

On Sat, Dec 19, 2020 at 6:55 PM Girish Joshi <girish946@gmail.com> wrote:
>
> On Fri, Dec 18, 2020 at 9:34 AM Siddhesh Poyarekar <siddhesh@gotplt.org> wrote:
> >
> > On 12/17/20 11:19 PM, Girish Joshi wrote:
> > > Hi Siddhesh,
> > > On Thu, Dec 17, 2020 at 12:19 AM Siddhesh Poyarekar <siddhesh@gotplt.org> wrote:
> > >> I'm surprised there are 1114 series that need action; maybe it's
> > >> including series that have already been committed and you need to filter
> > >> those out?
> > > Yeah, in the git output we can see that a lot of those are already applied.
> >
> > Are they marked as committed though?  If not then they should be.
> Yes, the status for (almost all of) those patches is "committed" on
> the patchwork instance.
> To verify it I'm writing down the IDs for such series in a separate file now.
> Although I did not find an option from the git-pw cli for checking if
> a series is already committed.
> The work around for that could be to go through all of the patches in
> that series and check if all of them are committed.
>
> Girish Joshi

[-- Attachment #2: check-series.py --]
[-- Type: text/x-python, Size: 2980 bytes --]

#!/usr/bin/env python
import csv
import sys
import os
import subprocess as sp
import _thread as thread


def read_file(file_name):
    file_data = []
    if os.path.exists(file_name):
        data = open(file_name).read().strip().split("\n")
        for line in data:
            if not '"ID"' in line:
                # print(line)
                file_data.append(line.strip())
    return file_data


# parse the csv entries
def read_rows(csvfile):

    # List for series entries
    series_data = []
    csvreader = csv.reader(csvfile, delimiter=",", quotechar='"')
    for row in csvreader:
        # print(row)
        if not row:
            return
        if row and row[1] != "ID":
            series_data.append(row)
    return series_data


def run_cmd(cmd, debug=False):
    """
    Execute command and return the exit code and output.
    """
    exit_code = 0
    output = ""
    try:
        output = sp.check_output(
            cmd, stderr=sp.STDOUT, shell=True, universal_newlines=True
        )
    except sp.CalledProcessError as exc:
        if debug:
            print("Status : FAIL", exc.returncode, exc.output)
        exit_code, output = exc.returncode, exc.output
    else:
        if debug:
            print("{}\n".format(output))

    return exit_code, output


def write_json(file_name, data):
    import json

    with open(file_name, "w") as f:
        f.write(json.dumps(data))


def check_data(list_, index):
    for i in list_:
        ret, op = run_cmd(f"git-pw series show {i} -f csv")
        print("****", index, "****")

        series_data = read_rows(op.strip().split("\n"))
        print(series_data[1])
        for j in series_data[11:]:
            patch_data = j[1].split()
            print(patch_data[0], patch_data[1])
            series_dict[i].append(patch_data[0])
            if patch_data[0] in patch_ids:
                patch_ids.remove(patch_data[0])

    done_lists[index] = True


if __name__ == "__main__":

    file_loc = "/tmp/pw-analysis"
    if not os.path.exists(file_loc):
        os.mkdir(file_loc)
    series_file = sys.argv[1]
    patches_file = sys.argv[2]

    series = [i for i in read_rows(read_file(series_file))]
    series_dict = {i[0]: [] for i in series}

    patches = read_rows(read_file(patches_file))
    patch_ids = [i[0] for i in patches]

    series_ids = [i for i in series_dict.keys()]

    chunk_size = 100
    all_lists = [
        series_ids[i : i + chunk_size] for i in range(0, len(series_ids), chunk_size)
    ]

    done_lists = [False for i in range(len(all_lists))]

    print(len(all_lists))
    for index, i in enumerate(all_lists):
        print(index)
        thread.start_new_thread(check_data, (i, index))
    while False in done_lists:
        pass

    print("Total individual patches: ", len(patch_ids))
    print("writing patch ids to", file_loc + "/patch_ids")
    write_json(file_loc + "/dict", series_dict)
    write_json(file_loc + "/patch_ids", {"patches": patch_ids})