public inbox for cygwin-apps-cvs@sourceware.org help / color / mirror / Atom feed
From: jturney@sourceware.org To: cygwin-apps-cvs@sourceware.org Subject: [calm - Cygwin server-side packaging maintenance script] branch master, updated. ac6a34563121d96f19e7f4c88ea40c203ea84c19 Date: Tue, 29 Mar 2016 11:28:00 -0000 [thread overview] Message-ID: <20160329112826.48972.qmail@sourceware.org> (raw) https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=ac6a34563121d96f19e7f4c88ea40c203ea84c19 commit ac6a34563121d96f19e7f4c88ea40c203ea84c19 Author: Jon Turney <jon.turney@dronecode.org.uk> Date: Tue Mar 29 12:21:56 2016 +0100 Don't emit warnings about ignored upload files on every run Periodically emit warnings about apparently forgotten upload files which are being ignored, but not on every run https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=83dc7d12d415f8c2c2d941acc5b145bb8cbb28b8 commit 83dc7d12d415f8c2c2d941acc5b145bb8cbb28b8 Author: Jon Turney <jon.turney@dronecode.org.uk> Date: Tue Mar 29 12:15:59 2016 +0100 Add an 'X-Calm' header to mark mails from calm https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=71791009a682a83ca273c3c58998891d7052ea1d commit 71791009a682a83ca273c3c58998891d7052ea1d Author: Jon Turney <jon.turney@dronecode.org.uk> Date: Wed Mar 23 21:22:55 2016 +0000 Add a testing facility to Bcc: all emails sent https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=b245856fe5e286431d75bab81e8ce68786b8b2d3 commit b245856fe5e286431d75bab81e8ce68786b8b2d3 Author: Jon Turney <jon.turney@dronecode.org.uk> Date: Tue Mar 22 18:48:12 2016 +0000 Improve checking sdesc for package name - match case-insensitively - try harder to manufacture a 'package basename' - ignore whitespace before ':' - look for 'packagename -' as well https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=95ff7fea38070dbe9219172aecb043ddea659ead commit 95ff7fea38070dbe9219172aecb043ddea659ead Author: Jon Turney <jon.turney@dronecode.org.uk> Date: Tue Mar 22 23:50:40 2016 +0000 Fix sdesc which end with '.' Don't just warn, also fix any sdesc which end with '.' Update tests https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=13d133e800946d7b339f2e3bf82b36e12ef686ca commit 13d133e800946d7b339f2e3bf82b36e12ef686ca Author: Jon Turney <jon.turney@dronecode.org.uk> Date: Tue Mar 22 18:47:00 2016 +0000 Warn if sdesc contains ' ' Some sdesc erroneously contained multiple spaces between words. https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=35dcccec440f79e6f86bdaa7098ed922859de94b commit 35dcccec440f79e6f86bdaa7098ed922859de94b Author: Jon Turney <jon.turney@dronecode.org.uk> Date: Wed Mar 23 21:27:41 2016 +0000 Sort requires: when read This avoids having to treat requires: specially when showing differences, which wasn't doing quite the right thing anyhow, as that was writing a setup.ini based on the sorted requires, leading to a differences of ordering on the next run... https://sourceware.org/git/gitweb.cgi?p=cygwin-apps/calm.git;h=8713d4e83fe3b9646366073f84c970c4f61c64cf commit 8713d4e83fe3b9646366073f84c970c4f61c64cf Author: Jon Turney <jon.turney@dronecode.org.uk> Date: Mon Mar 21 20:14:06 2016 +0000 Add .xz compression of setup.ini Diff: --- TODO | 1 - buffering_smtp_handler.py | 2 + calm.py | 4 ++- common_constants.py | 3 ++ hint.py | 18 ++++++++-- package.py | 22 ++++++------ testdata/.gitignore | 1 + .../x86/base-cygwin/base-cygwin-3.6-1 | 2 +- .../x86/base-cygwin/base-cygwin-3.8-1 | 2 +- testdata/htdocs.expected/x86/packages.inc | 2 +- testdata/inifile/setup.ini.expected | 2 +- testdata/x86.hints/release/base-cygwin/expected | 2 +- .../release/libtextcat/libtextcat-devel/expected | 2 +- .../release/mingw64-i686-binutils/expected | 2 +- .../mingw64-i686-binutils-debuginfo/expected | 2 +- testdata/x86.hints/release/splint/expected | 2 +- uploads.py | 36 +++++++++++++++++-- 17 files changed, 75 insertions(+), 30 deletions(-) diff --git a/TODO b/TODO index 246aac2..f2c7bc7 100644 --- a/TODO +++ b/TODO @@ -1,6 +1,5 @@ * per-package-version hint files (e.g. named <pvr>.hint, or /.setup.hint inside tar file) * more than 2 versions possible (and automatically vault 'old' versions, where we have some mechanism to explicity say what is old) -* enable .xz compressed setup.ini (setup.exe checks for it and presumably can handle it) * automatically suppress 'empty source' packages from setup.ini ? * run more often, option to not do anything if no uploads (to avoid reading the release area if we don't need to), lockfile to avoid colliding runs * work out some way to package it as a replacement for genini diff --git a/buffering_smtp_handler.py b/buffering_smtp_handler.py index 7720588..2bcffcc 100644 --- a/buffering_smtp_handler.py +++ b/buffering_smtp_handler.py @@ -55,7 +55,9 @@ class BufferingSMTPHandler(logging.handlers.BufferingHandler): m = email.message.Message() m['From'] = self.fromaddr m['To'] = ','.join(self.toaddrs) + m['Bcc'] = common_constants.ALWAYS_BCC m['Subject'] = self.subject + m['X-Calm'] = '1' # use utf-8 only if the message can't be ascii encoded charset = 'ascii' diff --git a/calm.py b/calm.py index f1b6b2d..0dd001a 100755 --- a/calm.py +++ b/calm.py @@ -188,7 +188,7 @@ def main(args): shutil.move(tmpfile.name, inifile) # compress and re-sign - for ext in ['.ini', '.bz2']: + for ext in ['.ini', '.bz2', '.xz']: try: os.remove(os.path.join(basedir, 'setup' + ext + '.sig')) except FileNotFoundError: @@ -196,6 +196,8 @@ def main(args): if ext == '.bz2': os.system('/usr/bin/bzip2 <%s >%s' % (inifile, os.path.splitext(inifile)[0] + ext)) + elif ext == '.xz': + os.system('/usr/bin/xz -6e <%s >%s' % (inifile, os.path.splitext(inifile)[0] + ext)) os.system('/usr/bin/gpg --batch --yes -b ' + os.path.join(basedir, 'setup' + ext)) diff --git a/common_constants.py b/common_constants.py index 7d031c9..e40c346 100644 --- a/common_constants.py +++ b/common_constants.py @@ -38,6 +38,9 @@ FTP = '/var/ftp/pub/cygwin' # logs are always emailed to these addresses EMAILS = ','.join(map(lambda m: m + '@sourceware.org', ['corinna', 'yselkowitz', 'jturney'])) +# for testing purposes, every email we send is bcc'd to these addresses +ALWAYS_BCC = '' + # these maintainers can upload orphaned packages as well ORPHANMAINT = "Yaakov Selkowitz" diff --git a/hint.py b/hint.py index 4751466..2579fad 100755 --- a/hint.py +++ b/hint.py @@ -180,9 +180,6 @@ def setup_hint_parse(fn): if key in hints: errors.append('duplicate key %s' % (key)) - # store the key:value - hints[key] = value - # check the value meets any key-specific constraints if (key in valkeys) and (len(value) == 0): errors.append('%s has empty value' % (key)) @@ -202,10 +199,16 @@ def setup_hint_parse(fn): if not (value.startswith('"') and value.endswith('"')): errors.append("%s value '%s' should be quoted" % (key, value)) - # warn if sdesc ends with a '.' + # if sdesc ends with a '.', warn and fix it if key == 'sdesc': if re.search(r'\."$', value): warnings.append("sdesc ends with '.'") + value = re.sub(r'\."$', '"', value) + + # warn if sdesc contains ' ' + if key == 'sdesc': + if ' ' in value: + warnings.append("sdesc contains ' '") # only 'ldesc' and 'message' are allowed a multi-line value if (key not in multilinevalkeys) and (len(value.splitlines()) > 1): @@ -219,6 +222,9 @@ def setup_hint_parse(fn): # warn if value starts with a quote followed by whitespace if re.match(r'^"[ \t]+', value): warnings.append('value for key %s starts with quoted whitespace' % (key)) + + # store the key:value + hints[key] = value else: errors.append("unknown setup construct '%s' at line %d" % (item, i)) @@ -239,6 +245,10 @@ def setup_hint_parse(fn): if len(hints['sdesc']) > 2*len(hints['ldesc']): warnings.append('sdesc is much longer than ldesc') + # sort requires: as differences in ordering are uninteresting + if 'requires' in hints: + hints['requires'] = ' '.join(sorted(hints['requires'].split())) + except UnicodeDecodeError: errors.append('invalid UTF-8') diff --git a/package.py b/package.py index 0b1313a..b1ca42e 100755 --- a/package.py +++ b/package.py @@ -203,13 +203,18 @@ def read_package(packages, basedir, dirpath, files, strict=False): if p in past_mistakes.self_source: packages[p].hints['self-source'] = '' - # don't allow a redundant 'package-initial-substring:' at start of sdesc + # don't allow a redundant 'package:' or 'package - ' at start of sdesc + # + # match case-insensitively, and use a base package name (trim off any + # leading 'lib' from package name, remove any soversion or 'devel' + # suffix) + # if 'sdesc' in hints: - sdesc = re.sub(r'^"|"$', '', hints['sdesc']) - colon = sdesc.find(':') - if colon > -1: - if sdesc[:colon] == p[:colon]: - logging.warning("package '%s' sdesc starts with '%s:'; this is redundant as the UI will show both the package name and sdesc" % (p, sdesc[:colon])) + colon = re.match(r'^"(.*?)(\s*:|\s+-)', hints['sdesc']) + if colon: + package_basename = re.sub(r'^lib(.*?)(|-devel|\d*)$', r'\1', p) + if package_basename.upper().startswith(colon.group(1).upper()): + logging.warning("package '%s' sdesc starts with '%s'; this is redundant as the UI will show both the package name and sdesc" % (p, ''.join(colon.group(1, 2)))) warnings = True elif (len(files) > 0) and (relpath.count(os.path.sep) > 0): @@ -591,11 +596,6 @@ def merge(a, b): if a[p].hints != b[p].hints: c[p].hints = b[p].hints - # sort requires: as differences in ordering are uninteresting - for hints in [a[p].hints, b[p].hints]: - if 'requires' in hints: - hints['requires'] = ' '.join(sorted(hints['requires'].split())) - diff = '\n'.join(difflib.ndiff( pprint.pformat(a[p].hints).splitlines(), pprint.pformat(b[p].hints).splitlines())) diff --git a/testdata/.gitignore b/testdata/.gitignore index 22f46b8..2dada27 100644 --- a/testdata/.gitignore +++ b/testdata/.gitignore @@ -4,3 +4,4 @@ results setup.ini sha512.sum \!ready +\!reminder-timestamp diff --git a/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.6-1 b/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.6-1 index 42b3a8e..a6c9d6c 100644 --- a/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.6-1 +++ b/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.6-1 @@ -1,5 +1,5 @@ <html> -<h1>base-cygwin: Initial base installation helper script. (installed binaries and support files)</h1> +<h1>base-cygwin: Initial base installation helper script (installed binaries and support files)</h1> <tt><pre> 2015-10-11 14:45 26 test/test.1 2015-10-11 14:45 31 test/test.2 diff --git a/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.8-1 b/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.8-1 index 42b3a8e..a6c9d6c 100644 --- a/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.8-1 +++ b/testdata/htdocs.expected/x86/base-cygwin/base-cygwin-3.8-1 @@ -1,5 +1,5 @@ <html> -<h1>base-cygwin: Initial base installation helper script. (installed binaries and support files)</h1> +<h1>base-cygwin: Initial base installation helper script (installed binaries and support files)</h1> <tt><pre> 2015-10-11 14:45 26 test/test.1 2015-10-11 14:45 31 test/test.2 diff --git a/testdata/htdocs.expected/x86/packages.inc b/testdata/htdocs.expected/x86/packages.inc index 484f797..cd5e235 100755 --- a/testdata/htdocs.expected/x86/packages.inc +++ b/testdata/htdocs.expected/x86/packages.inc @@ -7,7 +7,7 @@ <br> <table class="pkglist"> <tr><td><a href="x86/arc">arc</a></td><td>The ARC archive utility</td></tr> -<tr><td><a href="x86/base-cygwin">base-cygwin</a></td><td>Initial base installation helper script.</td></tr> +<tr><td><a href="x86/base-cygwin">base-cygwin</a></td><td>Initial base installation helper script</td></tr> <tr><td><a href="x86/cygwin">cygwin</a></td><td>The UNIX emulation engine</td></tr> <tr><td><a href="x86/cygwin-debuginfo">cygwin-debuginfo</a></td><td>Debug info for cygwin</td></tr> <tr><td><a href="x86/cygwin-devel">cygwin-devel</a></td><td>Core development files</td></tr> diff --git a/testdata/inifile/setup.ini.expected b/testdata/inifile/setup.ini.expected index 6140c5d..6b241ef 100644 --- a/testdata/inifile/setup.ini.expected +++ b/testdata/inifile/setup.ini.expected @@ -23,7 +23,7 @@ '6de201dfed1d45412509c65deb34690dc2d09c6aafccfe491fd2f440f92842b9c755b61dc7bcdd4cc0c9f18cf46c2b3a1241e99c4c2a33fff5555e7b2f0b6348\n' '\n' '@ base-cygwin\n' - 'sdesc: "Initial base installation helper script."\n' + 'sdesc: "Initial base installation helper script"\n' 'ldesc: "Initial base installation helper script."\n' 'category: Base\n' 'version: 3.8-1\n' diff --git a/testdata/x86.hints/release/base-cygwin/expected b/testdata/x86.hints/release/base-cygwin/expected index e3409a1..7505632 100644 --- a/testdata/x86.hints/release/base-cygwin/expected +++ b/testdata/x86.hints/release/base-cygwin/expected @@ -1,4 +1,4 @@ -{'sdesc': '"Initial base installation helper script."', +{'sdesc': '"Initial base installation helper script"', 'ldesc': '"Initial base installation helper script."', 'category': 'Base', 'requires': '', diff --git a/testdata/x86.hints/release/libtextcat/libtextcat-devel/expected b/testdata/x86.hints/release/libtextcat/libtextcat-devel/expected index f60d34d..aca0db6 100644 --- a/testdata/x86.hints/release/libtextcat/libtextcat-devel/expected +++ b/testdata/x86.hints/release/libtextcat/libtextcat-devel/expected @@ -7,6 +7,6 @@ 'BSD License.\n' 'http://software.wise-guys.nl/libtextcat/"', 'category': 'Devel Text', - 'requires': 'libtextcat0 libtextcat', + 'requires': 'libtextcat libtextcat0', 'external-source': 'libtextcat', 'parse-errors': ['embedded quote at line 7']} diff --git a/testdata/x86.hints/release/mingw64-i686-binutils/expected b/testdata/x86.hints/release/mingw64-i686-binutils/expected index bc0368d..114e53c 100644 --- a/testdata/x86.hints/release/mingw64-i686-binutils/expected +++ b/testdata/x86.hints/release/mingw64-i686-binutils/expected @@ -1,6 +1,6 @@ {'category': 'Devel', 'requires': 'cygwin libintl8 zlib0', - 'sdesc': '"The GNU Binutils are a collection of binary tools. This package\n is capable of targeting win32."', + 'sdesc': '"The GNU Binutils are a collection of binary tools. This package\n is capable of targeting win32"', 'ldesc': '"Binutils for MinGW-w64 Win32 toolchain"', 'parse-errors': ['key sdesc has multi-line value'], 'parse-warnings': ["sdesc ends with '.'", 'sdesc is much longer than ldesc']} diff --git a/testdata/x86.hints/release/mingw64-i686-binutils/mingw64-i686-binutils-debuginfo/expected b/testdata/x86.hints/release/mingw64-i686-binutils/mingw64-i686-binutils-debuginfo/expected index d925953..c936ff6 100644 --- a/testdata/x86.hints/release/mingw64-i686-binutils/mingw64-i686-binutils-debuginfo/expected +++ b/testdata/x86.hints/release/mingw64-i686-binutils/mingw64-i686-binutils-debuginfo/expected @@ -2,6 +2,6 @@ 'requires': 'cygwin-debuginfo', 'external-source': 'mingw64-i686-binutils', 'ldesc': '"Debug info for mingw64-i686-binutils"', - 'sdesc': '"This package contains files necessary for debugging the\nmingw64-i686-binutils package with gdb."', + 'sdesc': '"This package contains files necessary for debugging the\nmingw64-i686-binutils package with gdb"', 'parse-errors': ['key sdesc has multi-line value'], 'parse-warnings': ["sdesc ends with '.'", 'sdesc is much longer than ldesc']} diff --git a/testdata/x86.hints/release/splint/expected b/testdata/x86.hints/release/splint/expected index 8c0b6c4..5667e4b 100644 --- a/testdata/x86.hints/release/splint/expected +++ b/testdata/x86.hints/release/splint/expected @@ -1,4 +1,4 @@ -{'sdesc': '"Check C programs for security vulnerabilities and programming\nmistakes."', +{'sdesc': '"Check C programs for security vulnerabilities and programming\nmistakes"', 'ldesc': '"Program does many of the traditional lint checks including\n' 'unused declarations, type inconsistencies, use before definition,\n' 'unreachable code, ignored return values, execution paths with no\n' diff --git a/uploads.py b/uploads.py index 050c494..e3f6d78 100644 --- a/uploads.py +++ b/uploads.py @@ -30,9 +30,13 @@ import filecmp import os import logging import re +import time import package +# reminders will be issued daily +REMINDER_INTERVAL = 60*60*24 + # # @@ -60,6 +64,14 @@ def scan(m, all_packages, args): logging.info('processing files with mtime older than %d' % (mtime)) remove.append(ready) + # the mtime of this file indicates when 'ignoring as there is no !ready' + # warnings were last emitted + reminder_file = os.path.join(basedir, '!reminder-timestamp') + if os.path.exists(reminder_file): + reminder_time = os.path.getmtime(reminder_file) + else: + reminder_time = 0 + # scan package directories for (dirpath, subdirs, files) in os.walk(os.path.join(basedir, 'release')): relpath = os.path.relpath(dirpath, basedir) @@ -83,10 +95,9 @@ def scan(m, all_packages, args): # shortest-to-longest order, since os.walk() walks the tree # top-down), and use the mtime of the first (longest) matching path. while True: - (path, time) = mtimes[-1] + (path, mtime) = mtimes[-1] if relpath.startswith(path): - logging.info("using mtime %d from subpath '%s' of '%s'" % (time, path, relpath)) - mtime = time + logging.info("using mtime %d from subpath '%s' of '%s'" % (mtime, path, relpath)) break else: mtimes.pop() @@ -116,7 +127,15 @@ def scan(m, all_packages, args): # only process files newer than !ready if os.path.getmtime(fn) > mtime: if mtime == 0: - logging.warning("ignoring %s as there is no !ready" % fn) + lvl = logging.INFO + + # if more than REMINDER_INTERVAL has elapsed since we warned + # about files being ignored, warn again + if time.time() > (reminder_time + REMINDER_INTERVAL): + lvl = logging.WARNING + touch(reminder_file) + + logging.log(lvl, "ignoring %s as there is no !ready" % fn) else: logging.warning("ignoring %s as it is newer than !ready" % fn) files.remove(f) @@ -159,6 +178,15 @@ def scan(m, all_packages, args): # # +def touch(fn, times=None): + with open(fn, 'a'): + os.utime(fn, times) + + +# +# +# + def remove(args, remove): for f in remove: logging.info("rm %s", f)
reply other threads:[~2016-03-29 11:28 UTC|newest] Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20160329112826.48972.qmail@sourceware.org \ --to=jturney@sourceware.org \ --cc=cygwin-apps-cvs@sourceware.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).