From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 45889 invoked by alias); 16 May 2018 13:53:05 -0000 Mailing-List: contact springfield-help@sourceware.org; run by ezmlm Precedence: bulk List-Post: List-Help: List-Subscribe: Sender: springfield-owner@sourceware.org Received: (qmail 45374 invoked by uid 89); 16 May 2018 13:53:04 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Checked: by ClamAV 0.99.4 on sourceware.org X-Virus-Found: No X-Spam-SWARE-Status: No, score=-0.9 required=5.0 tests=BAYES_00,KAM_LAZY_DOMAIN_SECURITY,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.2 spammy=Online, online, HX-HELO:sk:mail-pl X-Spam-Status: No, score=-0.9 required=5.0 tests=BAYES_00,KAM_LAZY_DOMAIN_SECURITY,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on sourceware.org X-Spam-Level: X-HELO: mail-pl0-f52.google.com X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition:user-agent; bh=3nVwbbfqeBS21kBkB35+Vt6p93Ag9LCFXzJbZdOBwhs=; b=QWAOHqqy9x9LN4M4/y5Nj/AzC5CcDwUoIqlyt+FJlA3HWNX0sunTAoHhdDtlRQMqas K9VaIEsTzG1SsYOc2RG/Rsfc6EqJ4reF6MMU7/lIZFEAzvJsMOvJWYf+B70mw65U0URj xpNOkaBlWRZ/g3dpX7U5YZ6s28ynU7ZACr/VnEhBVAadzfCxW+RKy0esnLITbp3MuleA Hkcrb55gD1GD1ob+Beff1lJTQpTvjRtj7uHlg41JqNlnc7LBSAyYoSj2FnLeXUi1cqLT Z9LRku1p/UnNaBS595YERYxho0XFdGH56IO2K54wYHjxLfyVcPyQp3aIroqttBea100b pcqw== X-Gm-Message-State: ALKqPwdDzJiodA1XaJYPxSZo4wEk03t/cuVOsT828WEZ+stgmCSZWLlL z6gIOJ3lYeDVQdO/4fi52t4DJw== X-Google-Smtp-Source: AB8JxZoczTa4v3zdZn4T/c7fzG2t2qlrrLgAPMd5QzoB6+tMFGtx2GZrGPGiTaGKoWlxu9J3/q4fvw== X-Received: by 2002:a17:902:7109:: with SMTP id a9-v6mr1054815pll.271.1526478781332; Wed, 16 May 2018 06:53:01 -0700 (PDT) Date: Mon, 01 Jan 2018 00:00:00 -0000 From: Gris Ge To: rhel-sysmgt , springfield@sourceware.org, eng-common-logging@redhat.com Subject: [RFC] Daemon to provide structured storage events: peripety Message-ID: <20180516135253.GA12920@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="SLDf9lqlvOQaIe6s" Content-Disposition: inline X-PGP-Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6CCDE58FE41E28FF User-Agent: Mutt/1.9.5 (2018-04-13) X-SW-Source: 2018-05/txt/msg00000.txt.bz2 --SLDf9lqlvOQaIe6s Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-length: 2524 Hi Guys, In order to provide structured event for storage subsystem. I created a daemon for append structured data and device ID to systems logs: https://github.com/cathay4t/peripety/ Features: * Use sysfs and devfs for information query only. Designed to handle log burst. * Saves logs in journald with structured data. Log are hide by default to eliminate the duplicate entry. Try `journalctl -o json-pretty IS_PERIPETY=3DTRUE` * Provides consistent id for device of the log. * Provides extra information like mount point on fs mount, FC/iSCSI path information on multipath path failure: https://github.com/cathay4t/peripety/blob/master/examples/fs/ext4_mount= _lv_mpath_scsi.json https://github.com/cathay4t/peripety/blob/master/examples/mpath/mpath_f= c_path_offline.json * Provides device dependency information. E.g. LV over mpath over SCSI. Could query all related logs(fs, lvm, mpath, scsi) of physical device by command: journalctl DEV_WWID=3D + OWNERS_WWIDS=3D * Allows user defined regex in /etc/peripetyd.conf. * TBD: C/python/rust lightweight library for query block information on all kind of dev string(major:minor, scsi_id, nvme ctrl_id+ns_id, etc). * TODO: Varlink(json) interface. * TODO: Handle user space tool logs like mulitpathd, iscsid. Example JSON event of multipath FC path failure: { "hostname": "storageqe-18.rhts.eng.bos.redhat.com", "severity": "Warning", "sub_system": "Multipath", "timestamp": 1526053021631595, "event_id": "", "event_type": "DM_MPATH_PATH_FAILED", "dev_wwid": "360a98000324669436c2b45666c567946", "dev_path": "/dev/mapper/360a98000324669436c2b45666c567946", "owners_wwids": [ "t10.NETAPP-LUN-2FiCl+EflVyF" ], "owners_paths": [ "/dev/sdd" ], "kdev": "8:48", "msg": "device-mapper: multipath: Failing path 8:48.", "extention": { "blk_major_minor": "8:48", "transport": "FC", "port_state": "Online", "target_wwpn": "0x500a0982891b8dc5", "host_wwpn": "0x10000000c9a02834", "speed": "8 Gbit", "driver_name": "lpfc" } } I have also created some kernel patches to improve this: * Provides WWID by kernel on every log to fix the race issue. * Provides event type by kernel to save the use space regex captures. https://github.com/cathay4t/linux/commits/structured_log Thank you for your time. Best regards. --=20 Gris Ge --SLDf9lqlvOQaIe6s Content-Type: application/pgp-signature; name="signature.asc" Content-length: 833 -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE8f1XsqXpyNthgIbGbM3lj+QeKP8FAlr8N7AACgkQbM3lj+Qe KP/MFg/8CDEn2a+TWjAGyumcxqInf8EHq43L9dzr89vLSS8uSb4GMAybjp0GPwI+ j5GDS4t6mNlMyinHBKbwcfcFY/np/BJX/WotVaolIu/WYxAHWGxBhaP+z5sJx7Eu ivRAb9R4lRTwFjzBEHkHPzemmN8tyg8xShAcl99p9epVtF9ckX3F3ocB+0iuoHwg uBXHSbCNy9sdiHWqgbKDKy08DQ9Kpq3D90KtdnkAh8pLv38S5YKEuIWvu2yTb6gb U9TwrgWVF/b3OvAeMlNeHBd+YFjyYWBeFlwS2VEFJO1pjnLCOUF7ixR6fqGV2P+h EPJsz/ckJ9tuLH+bpuRtUhZNoGhCYEDJ2PYDDAK2uq5U3ScX/fxjIGYuge5Vqovy ENlqxebu9L7HA1lr0yaF860FE+nvnj0PnZIjshcmPYeKLJ4HQJc9PzIcopOt+MQ3 zc3MGf5v0fT3MYM3AXZF8P7540WsB5SFvxOA42YFJM+npuD4OjFRjhvmPsFrhQsk nJRORzcbXW5l7bMBQYJrcWanGWt3umKtnitfrZhbq74JKWjnJzehcpd7fb/tGbip iDlFGOZFyeBx8b/17onZjKcX83rWQcdNFpo63nUogARumbS4Fp++4PqKU23e1OQ1 wS2l5q25MKkpMZHp4MMwxeOYfYPekeujU/qxbO1uT2JT0Gz4ghs= =iR5c -----END PGP SIGNATURE----- --SLDf9lqlvOQaIe6s--