From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 130850 invoked by alias); 16 May 2018 14:13:31 -0000 Mailing-List: contact springfield-help@sourceware.org; run by ezmlm Precedence: bulk List-Post: List-Help: List-Subscribe: Sender: springfield-owner@sourceware.org Received: (qmail 130838 invoked by uid 89); 16 May 2018 14:13:30 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Checked: by ClamAV 0.99.4 on sourceware.org X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=docker, physical, Online, online X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_PASS autolearn=ham version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on sourceware.org X-Spam-Level: X-HELO: mx1.redhat.com Subject: Re: [ECL] [RFC] Daemon to provide structured storage events: peripety To: Gris Ge , rhel-sysmgt , springfield@sourceware.org, eng-common-logging@redhat.com References: <20180516135253.GA12920@redhat.com> From: Rich Megginson Message-ID: <2895d882-5c78-8629-5a2f-8f674396f19e@redhat.com> Date: Mon, 01 Jan 2018 00:00:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <20180516135253.GA12920@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Wed, 16 May 2018 14:13:22 +0000 (UTC) X-SW-Source: 2018-05/txt/msg00001.txt.bz2 On 05/16/2018 07:52 AM, Gris Ge wrote: > Hi Guys, > > In order to provide structured event for storage subsystem. > I created a daemon for append structured data and device ID to > systems logs: > > https://github.com/cathay4t/peripety/ > > Features: > * Use sysfs and devfs for information query only. Designed to handle > log burst. > > * Saves logs in journald with structured data. Log are hide by default > to eliminate the duplicate entry. Try `journalctl -o json-pretty > IS_PERIPETY=TRUE` > > * Provides consistent id for device of the log. > > * Provides extra information like mount point on fs mount, FC/iSCSI > path information on multipath path failure: > https://github.com/cathay4t/peripety/blob/master/examples/fs/ext4_mount_lv_mpath_scsi.json > https://github.com/cathay4t/peripety/blob/master/examples/mpath/mpath_fc_path_offline.json > > * Provides device dependency information. E.g. LV over mpath over SCSI. > Could query all related logs(fs, lvm, mpath, scsi) of physical device > by command: > > journalctl DEV_WWID= + OWNERS_WWIDS= > > * Allows user defined regex in /etc/peripetyd.conf. > > * TBD: C/python/rust lightweight library for query block information > on all kind of dev string(major:minor, scsi_id, nvme ctrl_id+ns_id, > etc). > > * TODO: Varlink(json) interface. > > * TODO: Handle user space tool logs like mulitpathd, iscsid. > > > Example JSON event of multipath FC path failure: > { > "hostname": "storageqe-18.rhts.eng.bos.redhat.com", > "severity": "Warning", > "sub_system": "Multipath", > "timestamp": 1526053021631595, > "event_id": "", > "event_type": "DM_MPATH_PATH_FAILED", > "dev_wwid": "360a98000324669436c2b45666c567946", > "dev_path": "/dev/mapper/360a98000324669436c2b45666c567946", > "owners_wwids": [ > "t10.NETAPP-LUN-2FiCl+EflVyF" > ], > "owners_paths": [ > "/dev/sdd" > ], > "kdev": "8:48", > "msg": "device-mapper: multipath: Failing path 8:48.", > "extention": { > "blk_major_minor": "8:48", > "transport": "FC", > "port_state": "Online", > "target_wwpn": "0x500a0982891b8dc5", > "host_wwpn": "0x10000000c9a02834", > "speed": "8 Gbit", > "driver_name": "lpfc" > } > } > > I have also created some kernel patches to improve this: > * Provides WWID by kernel on every log to fix the race issue. > * Provides event type by kernel to save the use space regex captures. > https://github.com/cathay4t/linux/commits/structured_log > > Thank you for your time. > > Best regards. > Thanks!  For better integration with common logging, I suggest you develop your own common logging schema with namespace(s).  We will be happy to assist you.  See https://github.com/ViaQ/elasticsearch-templates/tree/master/namespaces for an example of how we break down different components (kubernetes, systemd, docker, aushape, etc.) into their own namespaces with their own fields. The common data model also requires certain fields at the "top level" to be in a certain format.  For example, instead of       "timestamp": 1526053021631595, We use       "timestamp": "2018-05-11T09:37:01.631595-06:00", See also https://github.com/ViaQ/elasticsearch-templates/releases/download/0.0.16/com.redhat.viaq-openshift-operations.asciidoc