[RFC] Daemon to provide structured storage events: peripety

[RFC] Daemon to provide structured storage events: peripety

[Gris Ge]

[-- Attachment #1: Type: text/plain, Size: 2592 bytes --]

Hi Guys,

In order to provide structured event for storage subsystem.
I created a daemon for append structured data and device ID to
systems logs:

    https://github.com/cathay4t/peripety/

Features:
 * Use sysfs and devfs for information query only. Designed to handle
   log burst.

 * Saves logs in journald with structured data. Log are hide by default
   to eliminate the duplicate entry. Try `journalctl  -o json-pretty
   IS_PERIPETY=TRUE`

 * Provides consistent id for device of the log.

 * Provides extra information like mount point on fs mount, FC/iSCSI
   path information on multipath path failure:
    https://github.com/cathay4t/peripety/blob/master/examples/fs/ext4_mount_lv_mpath_scsi.json
    https://github.com/cathay4t/peripety/blob/master/examples/mpath/mpath_fc_path_offline.json

 * Provides device dependency information. E.g. LV over mpath over SCSI.
   Could query all related logs(fs, lvm, mpath, scsi) of physical device
   by command:

       journalctl DEV_WWID=<WWID> + OWNERS_WWIDS=<WWID>

 * Allows user defined regex in /etc/peripetyd.conf.

 * TBD: C/python/rust lightweight library for query block information
   on all kind of dev string(major:minor, scsi_id, nvme ctrl_id+ns_id,
   etc).

 * TODO: Varlink(json) interface.

 * TODO: Handle user space tool logs like mulitpathd, iscsid.


Example JSON event of multipath FC path failure:
    {
      "hostname": "storageqe-18.rhts.eng.bos.redhat.com",
      "severity": "Warning",
      "sub_system": "Multipath",
      "timestamp": 1526053021631595,
      "event_id": "",
      "event_type": "DM_MPATH_PATH_FAILED",
      "dev_wwid": "360a98000324669436c2b45666c567946",
      "dev_path": "/dev/mapper/360a98000324669436c2b45666c567946",
      "owners_wwids": [
        "t10.NETAPP-LUN-2FiCl+EflVyF"
      ],
      "owners_paths": [
        "/dev/sdd"
      ],
      "kdev": "8:48",
      "msg": "device-mapper: multipath: Failing path 8:48.",
      "extention": {
        "blk_major_minor": "8:48",
        "transport": "FC",
        "port_state": "Online",
        "target_wwpn": "0x500a0982891b8dc5",
        "host_wwpn": "0x10000000c9a02834",
        "speed": "8 Gbit",
        "driver_name": "lpfc"
      }
    }

I have also created some kernel patches to improve this:
 * Provides WWID by kernel on every log to fix the race issue.
 * Provides event type by kernel to save the use space regex captures.
https://github.com/cathay4t/linux/commits/structured_log

Thank you for your time.

Best regards.

-- 
Gris Ge

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ECL] [RFC] Daemon to provide structured storage events: peripety

[Rich Megginson]

On 05/16/2018 07:52 AM, Gris Ge wrote:
> Hi Guys,
>
> In order to provide structured event for storage subsystem.
> I created a daemon for append structured data and device ID to
> systems logs:
>
>      https://github.com/cathay4t/peripety/
>
> Features:
>   * Use sysfs and devfs for information query only. Designed to handle
>     log burst.
>
>   * Saves logs in journald with structured data. Log are hide by default
>     to eliminate the duplicate entry. Try `journalctl  -o json-pretty
>     IS_PERIPETY=TRUE`
>
>   * Provides consistent id for device of the log.
>
>   * Provides extra information like mount point on fs mount, FC/iSCSI
>     path information on multipath path failure:
>      https://github.com/cathay4t/peripety/blob/master/examples/fs/ext4_mount_lv_mpath_scsi.json
>      https://github.com/cathay4t/peripety/blob/master/examples/mpath/mpath_fc_path_offline.json
>
>   * Provides device dependency information. E.g. LV over mpath over SCSI.
>     Could query all related logs(fs, lvm, mpath, scsi) of physical device
>     by command:
>
>         journalctl DEV_WWID=<WWID> + OWNERS_WWIDS=<WWID>
>
>   * Allows user defined regex in /etc/peripetyd.conf.
>
>   * TBD: C/python/rust lightweight library for query block information
>     on all kind of dev string(major:minor, scsi_id, nvme ctrl_id+ns_id,
>     etc).
>
>   * TODO: Varlink(json) interface.
>
>   * TODO: Handle user space tool logs like mulitpathd, iscsid.
>
>
> Example JSON event of multipath FC path failure:
>      {
>        "hostname": "storageqe-18.rhts.eng.bos.redhat.com",
>        "severity": "Warning",
>        "sub_system": "Multipath",
>        "timestamp": 1526053021631595,
>        "event_id": "",
>        "event_type": "DM_MPATH_PATH_FAILED",
>        "dev_wwid": "360a98000324669436c2b45666c567946",
>        "dev_path": "/dev/mapper/360a98000324669436c2b45666c567946",
>        "owners_wwids": [
>          "t10.NETAPP-LUN-2FiCl+EflVyF"
>        ],
>        "owners_paths": [
>          "/dev/sdd"
>        ],
>        "kdev": "8:48",
>        "msg": "device-mapper: multipath: Failing path 8:48.",
>        "extention": {
>          "blk_major_minor": "8:48",
>          "transport": "FC",
>          "port_state": "Online",
>          "target_wwpn": "0x500a0982891b8dc5",
>          "host_wwpn": "0x10000000c9a02834",
>          "speed": "8 Gbit",
>          "driver_name": "lpfc"
>        }
>      }
>
> I have also created some kernel patches to improve this:
>   * Provides WWID by kernel on every log to fix the race issue.
>   * Provides event type by kernel to save the use space regex captures.
> https://github.com/cathay4t/linux/commits/structured_log
>
> Thank you for your time.
>
> Best regards.
>

Thanks!  For better integration with common logging, I suggest you 
develop your own common logging schema with namespace(s).  We will be 
happy to assist you.  See 
https://github.com/ViaQ/elasticsearch-templates/tree/master/namespaces 
for an example of how we break down different components (kubernetes, 
systemd, docker, aushape, etc.) into their own namespaces with their own 
fields.

The common data model also requires certain fields at the "top level" to 
be in a certain format.  For example, instead of

       "timestamp": 1526053021631595,

We use

       "timestamp": "2018-05-11T09:37:01.631595-06:00",

See also 
https://github.com/ViaQ/elasticsearch-templates/releases/download/0.0.16/com.redhat.viaq-openshift-operations.asciidoc