public inbox for cygwin-apps-cvs@sourceware.org
help / color / mirror / Atom feed
* [calm - Cygwin server-side packaging maintenance script] branch master, updated. 20210626-51-g46d04fa
@ 2022-06-07 20:46 Jon TURNEY
  0 siblings, 0 replies; only message in thread
From: Jon TURNEY @ 2022-06-07 20:46 UTC (permalink / raw)
  To: cygwin-apps-cvs




https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=46d04fa775587ed9ab22bd92d9211d5abb00b999

commit 46d04fa775587ed9ab22bd92d9211d5abb00b999
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date:   Tue Jun 7 11:11:09 2022 +0100

    Avoid false-positive in warning about replace-version:
    
    Avoid warning about replace-version: listing an installable version when
    the replacable version is a test: version (and a non-test version
    exists), because the setup depsolver wil correctly downgrade in that
    case.

https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=867e172bf9c1095c0939551398b83ce47647d18f

commit 867e172bf9c1095c0939551398b83ce47647d18f
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date:   Tue Jun 7 10:59:18 2022 +0100

    Only mail leads log records of ERROR or higher
    
    Emulate the behaviour of retainLevel=ERROR for the leads email, as we
    had prior to a1cb1581.

https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=f2af73f6d9f220c3b3d211edfae4e69966f31beb

commit f2af73f6d9f220c3b3d211edfae4e69966f31beb
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date:   Sat Jun 4 14:00:40 2022 +0100

    Improve default for EMAIL when testing

https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=d4888ac0292dfc801da586e0873e923925396382

commit d4888ac0292dfc801da586e0873e923925396382
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date:   Mon May 30 14:25:14 2022 +0100

    CI: Drop python 3.5 and add python 3.10
    
    license_expression doesn't support python 3.5

https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=dd860396f2fa459872360704626443ecab000d48

commit dd860396f2fa459872360704626443ecab000d48
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date:   Fri May 27 15:47:15 2022 +0100

    Extend fix-missing-src-hint-info for license:

https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=f4e788e17bce36b1b4658646728fb877463c32f4

commit f4e788e17bce36b1b4658646728fb877463c32f4
Author: Jon Turney <jon.turney@dronecode.org.uk>
Date:   Fri May 27 15:34:49 2022 +0100

    Allow 'license:' key in source hint
    
    Allow 'license:' key in source hint, and check it contains a valid SPDX
    license expression.
    
    Show the key value in source package summary page, and in repology JSON
    output.
    
    Update tests appropriately
    
    Future work: Add 'license:' to the set of mandatory keys for a source
    package.


Diff:
---
 .flake8                                            |   2 +
 .github/workflows/calm.yaml                        |   2 +-
 calm/calm.py                                       |   7 +-
 calm/common_constants.py                           |   2 +-
 calm/fix-missing-src-hint-info.py                  |  10 +-
 calm/fixes.py                                      | 101 +++++++++++++++++----
 calm/hint.py                                       |  24 ++++-
 calm/package.py                                    |  13 ++-
 calm/pkg2html.py                                   |   5 +
 requirements.txt                                   |   1 +
 .../x86/release/cygwin/cygwin-2.2.1-1-src.expected |   5 +-
 .../htdocs.expected/summary/cygwin-src.html        |   3 +
 test/testdata/process_arch/packages.json.expected  |   1 +
 .../x86/release/cygwin/cygwin-2.2.1-1-src.hint     |   1 +
 14 files changed, 145 insertions(+), 32 deletions(-)

diff --git a/.flake8 b/.flake8
index 0da33f9..48e7f81 100644
--- a/.flake8
+++ b/.flake8
@@ -1,3 +1,5 @@
 [flake8]
 ignore=E741,E129,W504,A003,B020,B902
 max-line-length=240
+per-file-ignores =
+  calm/fixes.py:E241,E127
diff --git a/.github/workflows/calm.yaml b/.github/workflows/calm.yaml
index 69be9d2..90a532a 100644
--- a/.github/workflows/calm.yaml
+++ b/.github/workflows/calm.yaml
@@ -8,7 +8,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [3.5, 3.6, 3.7, 3.8, 3.9]
+        python-version: [3.6, 3.7, 3.8, 3.9, '3.10']
 
     steps:
       - uses: actions/checkout@v2
diff --git a/calm/calm.py b/calm/calm.py
index 62dc902..63cdc60 100755
--- a/calm/calm.py
+++ b/calm/calm.py
@@ -630,12 +630,13 @@ def mail_cb(state, loghandler):
     if not state.args.email:
         return
 
-    # if there are any log records of ERROR level or higher, send all records to
-    # leads
+    # if there are any log records of ERROR level or higher, send those records
+    # to leads
     if any([record.levelno >= logging.ERROR for record in loghandler.buffer]):
         leads_email = BufferingSMTPHandler(state.args.email, subject='%s' % (state.subject))
         for record in loghandler.buffer:
-            leads_email.handle(record)
+            if record.levelno >= logging.ERROR:
+                leads_email.handle(record)
         leads_email.close()
 
     # send each maintainer mail containing log entries caused by their actions,
diff --git a/calm/common_constants.py b/calm/common_constants.py
index b85c32c..1887681 100644
--- a/calm/common_constants.py
+++ b/calm/common_constants.py
@@ -74,7 +74,7 @@ DEFAULT_KEEP_DAYS = 0
 # different values to be used when we are not running on sourceware.org, but my
 # test system...
 if os.uname()[1] == 'tambora':
-    EMAILS = 'jon.turney@dronecode.org.uk'
+    EMAILS = 'debug'
     ALWAYS_BCC = ''
 
 # package compressions
diff --git a/calm/fix-missing-src-hint-info.py b/calm/fix-missing-src-hint-info.py
index 88142a6..3185083 100644
--- a/calm/fix-missing-src-hint-info.py
+++ b/calm/fix-missing-src-hint-info.py
@@ -32,7 +32,7 @@ from . import common_constants
 from . import fixes
 
 
-def fix_hints(relarea, packages):
+def fix_hints(relarea, packages, fixids):
     for (dirpath, _subdirs, files) in os.walk(relarea):
 
         # only apply to listed packages, if specified
@@ -50,7 +50,7 @@ def fix_hints(relarea, packages):
                     logging.error('hint %s missing' % hf)
                     continue
 
-                fixes.fix_hint(dirpath, hf, f, ['homepage'])
+                fixes.fix_hint(dirpath, hf, f, fixids)
 
 
 #
@@ -62,6 +62,7 @@ def main():
 
     parser = argparse.ArgumentParser(description='src hint improver')
     parser.add_argument('package', nargs='*', metavar='PACKAGE')
+    parser.add_argument('--fix', action='append', help='ids of fixes to perform')
     parser.add_argument('-v', '--verbose', action='count', dest='verbose', help='verbose output', default=0)
     parser.add_argument('--releasearea', action='store', metavar='DIR', help="release directory (default: " + relarea_default + ")", default=relarea_default, dest='relarea')
     (args) = parser.parse_args()
@@ -69,9 +70,12 @@ def main():
     if args.verbose:
         logging.getLogger().setLevel(logging.INFO)
 
+    if not args.fix:
+        args.fix = ['homepage', 'license']
+
     logging.basicConfig(format=os.path.basename(sys.argv[0]) + ': %(message)s')
 
-    fix_hints(args.relarea, args.package)
+    fix_hints(args.relarea, args.package, args.fix)
 
 
 #
diff --git a/calm/fixes.py b/calm/fixes.py
index 796805a..e7f8146 100644
--- a/calm/fixes.py
+++ b/calm/fixes.py
@@ -63,6 +63,29 @@ def read_cygport(dirpath, tf):
     return content
 
 
+def _parse_cygport_var(dirpath, tf, var):
+    # crack open corresponding -src.tar and parse var out from .cygport
+    logging.debug('examining %s' % tf)
+    content = read_cygport(dirpath, tf)
+
+    value = None
+    if content:
+        for l in content.splitlines():
+            match = re.match(r'^\s*' + var + r'\s*=\s*("|)([^"].*)\1', l)
+            if match:
+                if value:
+                    logging.warning('multiple %s lines in .cygport in srcpkg %s' % (var, tf))
+                pn = os.path.basename(dirpath)
+                value = match.group(2)
+                value = re.sub(r'\$({|)(PN|ORIG_PN|NAME)(}|)', pn, value)
+
+    if value and '$' in value:
+        logging.warning('unknown shell parameter expansions in %s="%s" in .cygport in srcpkg %s' % (var, value, tf))
+        value = None
+
+    return value
+
+
 class NoRedirection(urllib.request.HTTPErrorProcessor):
     def http_response(self, request, response):
         return response
@@ -89,24 +112,7 @@ def _fix_homepage_src_hint(hints, dirpath, _hf, tf):
     if 'homepage' in hints:
         homepage = hints['homepage']
     else:
-        # crack open corresponding -src.tar and parse homepage out from .cygport
-        logging.debug('examining %s' % tf)
-        content = read_cygport(dirpath, tf)
-
-        homepage = None
-        if content:
-            for l in content.splitlines():
-                match = re.match(r'^\s*HOMEPAGE\s*=\s*("|)([^"].*)\1', l)
-                if match:
-                    if homepage:
-                        logging.warning('multiple HOMEPAGE lines in .cygport in srcpkg %s', tf)
-                    pn = os.path.basename(dirpath)
-                    homepage = match.group(2)
-                    homepage = re.sub(r'\$({|)(PN|ORIG_PN|NAME)(}|)', pn, homepage)
-
-        if homepage and '$' in homepage:
-            logging.warning('unknown shell parameter expansions in HOMEPAGE="%s" in .cygport in srcpkg %s' % (homepage, tf))
-            homepage = None
+        homepage = _parse_cygport_var(dirpath, tf, 'HOMEPAGE')
 
         if not homepage:
             logging.info('cannot determine homepage: from srcpkg %s' % tf)
@@ -139,6 +145,63 @@ def _fix_homepage_src_hint(hints, dirpath, _hf, tf):
     return False
 
 
+# for specific packages, map some human-readable license texts to SPDX expressions
+licmap = [
+    ('Apache License, Version 2',             'Apache-2.0',                             ['meson', 'ninja']),
+    ('BSD 3-Clause',                          'BSD-3-Clause',                           ['libsolv', 'mingw64-i686-libsolv', 'mingw64-x86_64-libsolv']),
+    ('BSD3/GPLv2+',                           'BSD-3-Clause AND GPL-2.0-or-later',      ['dash']),
+    ('CC BY-SA 3.0',                          'CC-BY-SA-3.0',                           ['dmalloc']),
+    ('GNU General Public License, Version 2', 'GPL-2.0-only',                           ['buildbot-slave', 'buildbot-worker']),
+    ('GNU General Public License, Version 3', 'GPL-3.0-or-later',                       ['osslsigncode']),
+    ('GPL',                                   'GPL-2.0-or-later',                       ['cpuid']),
+    ('GPL',                                   'GPL-3.0-or-later',                       ['units']),
+    ('GPLv2+',                                'GPL-2.0-or-later',                       ['grep', 'gzip', 'readline']),
+    ('GPLv2+FontEmbExc/OFL-1.1',              'GPL-2.0-or-later WITH Font-exception-2.0 OR OFL-1.1',
+                                                                                        ['unifont']),
+    ('GPLv3+',                                'GPL-3.0-or-later',                       ['bison', 'wget']),
+    ('LGPLv2.1+/GPLv2+',                      'LGPL-2.1-or-later AND GPL-2.0-or-later', ['libgcrypt']),
+    ('LGPLv3+/GPLv2+/GPLv3+/LGPLv2+',         '(LGPL-3.0-or-later OR GPL-2.0-or-later) AND GPL-3.0-or-later',
+                                                                                        ['libidn', 'mingw64-i686-libidn', 'mingw64-x86_64-libidn']),
+    ('LGPLv3+/GPLv2+/GPLv3+/Unicode2016',     '(LGPL-3.0-or-later OR GPL-2.0-or-later) AND GPL-3.0-or-later AND Unicode-DFS-2016',
+                                                                                        ['libidn2', 'mingw64-i686-libidn2', 'mingw64-x86_64-libidn2']),
+    ('MIT License',                           'MIT',                                    ['python-future']),
+    ('MIT-like',                              'curl',                                   ['curl', 'mingw64-i686-curl', 'mingw64-x86_64-curl']),
+    ('MIT-like',                              'Linux-man-pages-copyleft',               ['man-pages-linux', 'man-pages-posix']),
+    ('MIT-like',                              'BSD-Source-Code',                        ['vttest']),
+    ('Public domain',                         'BSD-3-Clause AND Public-Domain',         ['tzdata', 'tzcode']),
+    ('SGI Free Software License B',           'SGI-B-2.0',                              ['khronos-opengl-registry']),
+    ('Sun OpenLook',                          'XVIEW',                                  ['xview']),
+]
+
+
+def _fix_license_src_hint(hints, dirpath, _hf, tf):
+    # already present?
+    if 'license' in hints:
+        lic = hints['license']
+    else:
+        lic = _parse_cygport_var(dirpath, tf, 'LICENSE')
+
+        if not lic:
+            logging.info('cannot determine license: from srcpkg %s' % tf)
+            return False
+
+        pn = dirpath.split(os.path.sep)[-1]
+        for (human, spdx, pl) in licmap:
+            if (pn in pl) and (lic.lower() == human.lower()):
+                lic = spdx
+                logging.info("converted license text '%s' to SPDX license expression '%s'" % (human, spdx))
+                break
+
+        logging.info('adding license:%s to hints for srcpkg %s' % (lic, tf))
+
+    # changed?
+    if lic != hints.get('license', None):
+        hints['license'] = lic
+        return True
+
+    return False
+
+
 def _fix_invalid_keys_hint(hints, _dirpath, hf, _tf):
     # eliminate keys that aren't appropriate to the package type
     if hf.endswith('-src.hint'):
@@ -170,6 +233,8 @@ def fix_hint(dirpath, hf, tf, problems):
         changed = _fix_homepage_src_hint(hints, dirpath, hf, tf)
     if 'invalid_keys' in problems:
         changed = _fix_invalid_keys_hint(hints, dirpath, hf, tf) or changed
+    if 'license' in problems:
+        changed = _fix_license_src_hint(hints, dirpath, hf, tf) or changed
 
     # write updated hints
     if changed:
diff --git a/calm/hint.py b/calm/hint.py
index 15c468f..99408d7 100755
--- a/calm/hint.py
+++ b/calm/hint.py
@@ -26,8 +26,20 @@
 #
 
 from collections import OrderedDict
-import re
 import argparse
+import license_expression
+import re
+
+# reach inside license_expression to add custom license ids we permit
+json = license_expression.get_license_index()
+extra_licenses = [
+    'Linux-man-pages-copyleft',  # requires SPDX license-list 3.15
+    'Public-Domain',
+    'XVIEW',
+]
+for l in extra_licenses:
+    json.append({"spdx_license_key": l})
+licensing = license_expression.build_spdx_licensing(json)
 
 # types of key:
 # 'multilineval' - always have a value, which may be multiline
@@ -67,6 +79,7 @@ hintkeys[spvr].update({
     'skip': 'noval',   # in all spvr hints, but ignored
     'homepage': 'val',
     'build-depends': 'optval',
+    'license': 'val',
 })
 
 hintkeys[override] = {
@@ -277,6 +290,15 @@ def hint_file_parse(fn, kind, strict=False):
                         if not re.match(r'(\S+)\s+(\S.*)', value):
                             errors.append('message value must have id and text')
 
+                    # license must be a valid spdx license expression
+                    if key == 'license':
+                        try:
+                            le = licensing.validate(value, strict=True)
+                        except (license_expression.ExpressionParseError, license_expression.ExpressionError) as e:
+                            errors.append('value for key %s not a valid license expression: %s' % (key, e))
+                        if le.original_expression != le.normalized_expression:
+                            errors.append("license expression: '%s' normalizes to '%s'" % (value, le.normalized_expression))
+
                     # warn if value starts with a quote followed by whitespace
                     if re.match(r'^"[ \t]+', value):
                         warnings.append('value for key %s starts with quoted whitespace' % (key))
diff --git a/calm/package.py b/calm/package.py
index 0b3dec8..482d042 100755
--- a/calm/package.py
+++ b/calm/package.py
@@ -698,14 +698,16 @@ def validate_packages(args, packages, valid_requires_extra=None):
                 # warn if replace-versions lists a version which is less than
                 # the current version (which is pointless as the current version
                 # will replace it anyhow)
-                if packages[p].best_version:
-                    if SetupVersion(rv) <= SetupVersion(packages[p].best_version):
-                        logging.warning("package '%s' replace-versions: uselessly lists version '%s', which is <= current version" % (p, rv))
+                bv = packages[p].best_version
+                if bv:
+                    if SetupVersion(rv) <= SetupVersion(bv):
+                        logging.warning("package '%s' replace-versions: uselessly lists version '%s', which is <= current version '%s'" % (p, rv, bv))
 
                 # warn if replace-versions lists a version which is also
                 # available to install (as this doesn't work as expected)
                 if rv in packages[p].version_hints:
-                    logging.warning("package '%s' replace-versions: lists version '%s', which is also available to install" % (p, rv))
+                    if ('test' in packages[p].version_hints[rv]) == ('test' in packages[p].version_hints[bv]):
+                        logging.warning("package '%s' replace-versions: lists version '%s', which is also available to install" % (p, rv))
 
         # If the install tarball is empty, we should probably either be marked
         # obsolete (if we have no dependencies) or virtual (if we do)
@@ -1212,6 +1214,9 @@ def write_repo_json(args, packages, f):
         if 'homepage' in po.version_hints[bv]:
             d['homepage'] = po.version_hints[bv]['homepage']
 
+        if 'license' in po.version_hints[bv]:
+            d['license'] = po.version_hints[bv]['license']
+
         if pkg_maintainers[po.orig_name] and ('ORPHANED' not in pkg_maintainers[po.orig_name]):
             d['maintainers'] = sorted(pkg_maintainers[po.orig_name])
 
diff --git a/calm/pkg2html.py b/calm/pkg2html.py
index ae6a6a7..565e515 100755
--- a/calm/pkg2html.py
+++ b/calm/pkg2html.py
@@ -242,6 +242,11 @@ def update_package_listings(args, packages):
                         homepage = po.version_hints[po.best_version].get('homepage', None)
                         if homepage:
                             print('<span class="detail">homepage</span>: <a href="%s">%s</a><br><br>' % (homepage, homepage), file=f)
+                        lic = po.version_hints[po.best_version].get('license', None)
+                        if lic:
+                            print('<span class="detail">license</span>: %s' % (lic), file=f)
+                            print('<span class="smaller">(<a href="https://spdx.org/licenses/">SPDX</a>)</span>', file=f)
+                            print('<br><br>', file=f)
                     else:
                         es = po.srcpackage(bv)
                         print('<span class="detail">source package</span>: %s<br><br>' % linkify_package(es), file=f)
diff --git a/requirements.txt b/requirements.txt
index bc5496c..1f4e625 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,6 +3,7 @@ flake8-blind-except
 flake8-bugbear ; python_version >= "3.5"
 flake8-builtins
 flake8-import-order == 0.14.1
+license_expression
 lockfile
 pycodestyle
 python-daemon
diff --git a/test/testdata/hints/x86/release/cygwin/cygwin-2.2.1-1-src.expected b/test/testdata/hints/x86/release/cygwin/cygwin-2.2.1-1-src.expected
index 9cf692b..3657b00 100644
--- a/test/testdata/hints/x86/release/cygwin/cygwin-2.2.1-1-src.expected
+++ b/test/testdata/hints/x86/release/cygwin/cygwin-2.2.1-1-src.expected
@@ -1 +1,4 @@
-OrderedDict([('sdesc', '"The UNIX emulation engine"'), ('ldesc', '"The UNIX emulation engine"'), ('category', 'Base')])
+{'sdesc': '"The UNIX emulation engine"',
+ 'ldesc': '"The UNIX emulation engine"',
+ 'category': 'Base',
+ 'license': 'LGPL-3.0-or-later WITH LGPL-3.0-linking-exception'}
diff --git a/test/testdata/htdocs.expected/summary/cygwin-src.html b/test/testdata/htdocs.expected/summary/cygwin-src.html
index a68217c..4720b17 100755
--- a/test/testdata/htdocs.expected/summary/cygwin-src.html
+++ b/test/testdata/htdocs.expected/summary/cygwin-src.html
@@ -14,6 +14,9 @@
 <span class="detail">description</span>: The UNIX emulation engine<br><br>
 <span class="detail">categories</span>: Base<br><br>
 <span class="detail">install package(s)</span>: <a href="cygwin.html">cygwin</a>, <a href="cygwin-debuginfo.html">cygwin-debuginfo</a>, <a href="cygwin-devel.html">cygwin-devel</a><br><br>
+<span class="detail">license</span>: LGPL-3.0-or-later WITH LGPL-3.0-linking-exception
+<span class="smaller">(<a href="https://spdx.org/licenses/">SPDX</a>)</span>
+<br><br>
 <span class="detail">maintainer(s)</span>: Corinna Vinschen, Yaakov Selkowitz 
 <span class="smaller">(Use <a href="/lists.html#cygwin">the mailing list</a> to report bugs or ask questions.
 <a href="/problems.html#personal-email">Do not contact the maintainer(s) directly</a>.)</span>
diff --git a/test/testdata/process_arch/packages.json.expected b/test/testdata/process_arch/packages.json.expected
index 512fcf2..be21cfa 100644
--- a/test/testdata/process_arch/packages.json.expected
+++ b/test/testdata/process_arch/packages.json.expected
@@ -51,6 +51,7 @@
  '            "arches": [\n'
  '                "x86"\n'
  '            ],\n'
+ '            "license": "LGPL-3.0-or-later WITH LGPL-3.0-linking-exception",\n'
  '            "maintainers": [\n'
  '                "Corinna Vinschen",\n'
  '                "Yaakov Selkowitz"\n'
diff --git a/test/testdata/relarea/x86/release/cygwin/cygwin-2.2.1-1-src.hint b/test/testdata/relarea/x86/release/cygwin/cygwin-2.2.1-1-src.hint
index d924e38..28bb602 100644
--- a/test/testdata/relarea/x86/release/cygwin/cygwin-2.2.1-1-src.hint
+++ b/test/testdata/relarea/x86/release/cygwin/cygwin-2.2.1-1-src.hint
@@ -1,3 +1,4 @@
 sdesc: "The UNIX emulation engine"
 ldesc: "The UNIX emulation engine"
 category: Base
+license: LGPL-3.0-or-later WITH LGPL-3.0-linking-exception



^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2022-06-07 20:46 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-07 20:46 [calm - Cygwin server-side packaging maintenance script] branch master, updated. 20210626-51-g46d04fa Jon TURNEY

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).