From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by sourceware.org (Postfix) with ESMTPS id B35623858418 for ; Sun, 20 Mar 2022 02:59:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B35623858418 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=serhei.io Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=serhei.io Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 3489A3200A16; Sat, 19 Mar 2022 22:59:35 -0400 (EDT) Received: from imap50 ([10.202.2.100]) by compute5.internal (MEProxy); Sat, 19 Mar 2022 22:59:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=serhei.io; h=cc :cc:content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; bh=BGCOalD/STc2cWEu1FaPzYpDQ06v6IBWQysxTB N3Pr4=; b=loWAMX/w04/js4fQvqsXP515zUhtO20DA0aK5cebK4xA8Bb4HQcuVG teQKFhv02dfz/m2Z1nA7RWsTY66uVMJKkl9cFDT5ngX/y4krcz5yV9C6jfk+MDNS jnhyrCkX0Oomg9igYsjV3QzLAeNcmB4VzS5V9vlFN9Vp/nU9E3GQh93Lu6PGvFaT 4rfJ8VDaQwL4JcODCgnkwWZYoa5lUswxR6tK3bRX/aAOIV3Lga+BwDXadeChxUZ2 wRRHvxy2nGMAo8iX1cbqtmYZFEzLDAsbNo3ycKH8OrWcHDIaJe1kp0k3hrykRkoM HeuCfCGXPhBkW/OOjiUrHzABI7EM3O5Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=BGCOalD/STc2cWEu1 FaPzYpDQ06v6IBWQysxTBN3Pr4=; b=HT6kBkBtKB4jdSFfDpV+e7YLIRwflC/0a m8wRcV3a2VYGOxh13xg4Cu5NKLelVH0jVvDRyUan9W7gZk/APyMO+/6RyzeZqPE6 saKU73QO5jSwP6OMSBOB93Zb/5JIZWTFo+ouhLtVT73v4zfnRFPu0hd8301+wAcI AX4Uejq4piF09RiBhKiAmqViBhpuDxevYu2KnXMMBu98pCQMHvv+nAx6QeagLDWu VkXfLAmBlGgUdd2aPUeFEPSjXVPqlzZ6ZU1vF2fBvsIFir38Uv1V23+sD4uCG9aJ PlrpTUt+lSBBc1q49BFsidt22BUQsRJuYhLNuswrhUOKkA6f4Lk7A== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudefledgheefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpedfufgvrhhhvghiucforghkrghrohhvfdcuoehmvgesshgv rhhhvghirdhioheqnecuggftrfgrthhtvghrnhepfeejgffggeffudehiedtledtgeejtd ekgfeuleehuddtgfegvdfghfejkeeigfdtnecuvehluhhsthgvrhfuihiivgeptdenucfr rghrrghmpehmrghilhhfrhhomhepmhgvsehsvghrhhgvihdrihho X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 5ED1A192A40D; Sat, 19 Mar 2022 22:59:34 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-4907-g25ce6f34a9-fm-20220311.001-g25ce6f34 Mime-Version: 1.0 Message-Id: In-Reply-To: <1796a6e2-2b2a-49a7-b350-9d58700d3e30@www.fastmail.com> References: <1796a6e2-2b2a-49a7-b350-9d58700d3e30@www.fastmail.com> Date: Sat, 19 Mar 2022 22:59:13 -0400 From: "Serhei Makarov" To: Bunsen Subject: Re: bunsen (re)design discussion #2: Repository Layout, SQLite design idea + Django-esque API(?) Content-Type: text/plain X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, JMQ_SPF_NEUTRAL, RCVD_IN_DNSWL_LOW, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: bunsen@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bunsen mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Mar 2022 02:59:40 -0000 > This is a subject of ongoing discussion with fche, because we have > quite different opinions about what is convenient to keep in Git vs > SQLite, how flexible the Git layout should be etc. In principle, the > analysis and data representation will work more or less the same > regardless of what solution we settle on. Therefore, the repository > layout can be made configurable (and potentially the configurability > can be reduced down the line as we settle on what makes sense and what > doesn't). Git+JSON (the current format) - Good for cloning (git clone + git pull, options for shallow cloning, cloning subsets of branches) SQLite - Awful for cloning (essentially, requires redoing the parse on the receiving end, or returning data to the Git+JSON format) - Likely to be more efficient at certain queries - The 'sliding window' analyses that compare nearby points in a history would be possible but tedious to express - Designing a *stable* schema for analyses to query against would be tricky, requiring space-optimized tables to be munged into views following an unchanging schema. Otherwise any change to optimizations will require embedded SQL in analysis scripts to be rewritten. Given the toss-up and potential complementary strengths, it would be best to have a way to support both formats and experiment extensively. The configuration file could allow options as follows: [core] git_repo = ... // stores both testlogs and testruns testlogs_repo = ... // testlogs in Git format with properly named branches testruns_repo = ... // testruns in JSON format testruns_db = ... // enables data to be stored in SQLite DB cache_db = ... // can be the same as cache_db (Obviously, not all at once. This is meant to capture the possible use cases.) Then we could also add options per-project: [project "systemtap"] raw_testlogs_repo = ... // testlogs in Git format with any branches, for importing parse_module = systemtap.parse_dejagnu [project "systemtap-contrib"] testlogs_repo = ... // archive testruns in a separate repo (or db) from the main repo I'll think a bit more about the configuration format in light of the 'Makefile' model.