From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 59811 invoked by alias); 13 Dec 2019 19:25:13 -0000 Mailing-List: contact elfutils-devel-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Post: List-Help: List-Subscribe: Sender: elfutils-devel-owner@sourceware.org Received: (qmail 59800 invoked by uid 89); 13 Dec 2019 19:25:13 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Checked: by ClamAV 0.100.3 on sourceware.org X-Virus-Found: No X-Spam-SWARE-Status: No, score=-15.2 required=5.0 tests=AWL,BAYES_00,GIT_PATCH_0,GIT_PATCH_1,GIT_PATCH_2,GIT_PATCH_3,KAM_BADIPHTTP,KAM_LOTSOFHASH,SPF_PASS autolearn=ham version=3.3.1 spammy=UD:x86_64, rpm, deb, bank X-Spam-Status: No, score=-15.2 required=5.0 tests=AWL,BAYES_00,GIT_PATCH_0,GIT_PATCH_1,GIT_PATCH_2,GIT_PATCH_3,KAM_BADIPHTTP,KAM_LOTSOFHASH,SPF_PASS autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on sourceware.org X-Spam-Level: X-HELO: us-smtp-1.mimecast.com Received: from us-smtp-delivery-1.mimecast.com (HELO us-smtp-1.mimecast.com) (207.211.31.120) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 13 Dec 2019 19:25:11 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1576265109; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ro+9LtIZQyEo+2fH3eu4huScF3pioUZiBtKzDbkuFSA=; b=QK7zNhLPueZelpYYI3tTQq8tsPhAnjG5Nv72Kaxe8UA3LFA1Gy++vOvTeySyrRcUnQljJC 0OvghKSn9912y2KefbYAib+RN5vCgnQ+Cx7+KqNCLCycWAwSXCQOtgT37hAW+pqpst6RfY fNWlDN8IC+N7eExIYXrP7Un0SzbycN0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-175-pjRtdvHvM82vBvYutHTNEA-1; Fri, 13 Dec 2019 14:25:06 -0500 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 16D508005B5; Fri, 13 Dec 2019 19:25:04 +0000 (UTC) Received: from redhat.com (ovpn-116-64.phx2.redhat.com [10.3.116.64]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CB5E210013A7; Fri, 13 Dec 2019 19:25:03 +0000 (UTC) Received: from fche by redhat.com with local (Exim 4.92) (envelope-from ) id 1ifqYg-0000B1-8L; Fri, 13 Dec 2019 14:25:02 -0500 Date: Fri, 13 Dec 2019 19:25:00 -0000 From: "Frank Ch. Eigler" To: Mark Wielaard Cc: elfutils-devel@sourceware.org Subject: Re: RFCv2: debuginfod debian archive support Message-ID: <20191213192502.GA322@redhat.com> References: <20191202225401.GF4136@redhat.com> <20191207030313.GA23921@redhat.com> <617f80866c5e790aad6ac3b0e1cb54808fefaf58.camel@klomp.org> MIME-Version: 1.0 In-Reply-To: <617f80866c5e790aad6ac3b0e1cb54808fefaf58.camel@klomp.org> User-Agent: Mutt/1.12.0 (2019-05-25) X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-MC-Unique: pjRtdvHvM82vBvYutHTNEA-1 X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-IsSubscribed: yes X-SW-Source: 2019-q4/txt/msg00273.txt.bz2 Hi - > I see, I missed that both functions are only called after first > checking the archive type. I think it might be helpful/clearer if both > methods would be called with the intended archive type then, also > because that might make it simpler to... The archive subtype (rpm vs deb) is not stored in the database, as this would break the current schema, and is really not needed. In any case, it's easily recovered for purpose of separate metrics. Following sur-patch added to fche/debuginfod-deb branch: commit 02694cd29672d6912569a4bfe03b703bc134a821 Author: Frank Ch. Eigler Date: Fri Dec 13 14:21:12 2019 -0500 debuginfod deb support: review responses diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx index f022995f490f..70cb95fecd65 100644 --- a/debuginfod/debuginfod.cxx +++ b/debuginfod/debuginfod.cxx @@ -868,9 +868,13 @@ handle_buildid_r_match (int64_t b_mtime, } =20 string archive_decoder =3D "/dev/null"; + string archive_extension =3D ""; for (auto&& arch : scan_archives) if (string_endswith(b_source0, arch.first)) - archive_decoder =3D arch.second; + { + archive_extension =3D arch.first; + archive_decoder =3D arch.second; + } string popen_cmd =3D archive_decoder + " " + shell_escape(b_source0); FILE* fp =3D popen (popen_cmd.c_str(), "r"); // "e" O_CLOEXEC? if (fp =3D=3D NULL) @@ -922,7 +926,7 @@ handle_buildid_r_match (int64_t b_mtime, throw archive_exception(a, "cannot extract file"); } =20 - inc_metric ("http_responses_total","result","archive"); + inc_metric ("http_responses_total","result",archive_extension + " ar= chive"); struct MHD_Response* r =3D MHD_create_response_from_fd (archive_entr= y_size(e), fd); if (r =3D=3D 0) { @@ -1881,16 +1885,20 @@ thread_main_scan_source_file_path (void* arg) // Analyze given archive file of given age; record buildids / exec/debugin= fo-ness of its // constituent files with given upsert statements. static void -archive_classify (const string& rps, sqlite_ps& ps_upsert_buildids, sqlite= _ps& ps_upsert_files, - sqlite_ps& ps_upsert_de, sqlite_ps& ps_upsert_sref, sqlite_p= s& ps_upsert_sdef, - time_t mtime, - unsigned& fts_executable, unsigned& fts_debuginfo, unsigned&= fts_sref, unsigned& fts_sdef, - bool& fts_sref_complete_p) +archive_classify (const string& rps, string& archive_extension, + sqlite_ps& ps_upsert_buildids, sqlite_ps& ps_upsert_file= s, + sqlite_ps& ps_upsert_de, sqlite_ps& ps_upsert_sref, sqli= te_ps& ps_upsert_sdef, + time_t mtime, + unsigned& fts_executable, unsigned& fts_debuginfo, unsig= ned& fts_sref, unsigned& fts_sdef, + bool& fts_sref_complete_p) { string archive_decoder =3D "/dev/null"; for (auto&& arch : scan_archives) if (string_endswith(rps, arch.first)) - archive_decoder =3D arch.second; + { + archive_extension =3D arch.first; + archive_decoder =3D arch.second; + } string popen_cmd =3D archive_decoder + " " + shell_escape(rps); FILE* fp =3D popen (popen_cmd.c_str(), "r"); // "e" O_CLOEXEC? if (fp =3D=3D NULL) @@ -2176,18 +2184,19 @@ scan_source_archive_path (const string& dir) bool my_fts_sref_complete_p =3D true; try { - archive_classify (rps, + string archive_extension; + archive_classify (rps, archive_extension, ps_upsert_buildids, ps_upsert_files, ps_upsert_de, ps_upsert_sref, ps_upsert_= sdef, // dalt f->fts_statp->st_mtime, my_fts_executable, my_fts_debuginfo, my_= fts_sref, my_fts_sdef, my_fts_sref_complete_p); - inc_metric ("scanned_total","source","archive"); - add_metric("found_debuginfo_total","source","archive", + inc_metric ("scanned_total","source",archive_extension= + " archive"); + add_metric("found_debuginfo_total","source",archive_ex= tension + " archive", my_fts_debuginfo); - add_metric("found_executable_total","source","archive", + add_metric("found_executable_total","source",archive_e= xtension + " archive", my_fts_executable); - add_metric("found_sourcerefs_total","source","archive", + add_metric("found_sourcerefs_total","source",archive_e= xtension + " archive", my_fts_sref); } catch (const reportable_exception& e) diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh index cd31e30e0491..01c7e58e5ffc 100755 --- a/tests/run-debuginfod-find.sh +++ b/tests/run-debuginfod-find.sh @@ -81,7 +81,8 @@ wait_ready() done; =20 if [ $timeout -eq 0 ]; then - echo "metric $what never changed to $value on port $port" + echo "metric $what never changed to $value on port $port" + curl -s http://127.0.0.1:$port/metrics exit 1; fi } @@ -166,7 +167,7 @@ cp -rvp ${abs_srcdir}/debuginfod-rpms R kill -USR1 $PID1 # All rpms need to be in the index rpms=3D$(find R -name \*rpm | wc -l) -wait_ready $PORT1 'scanned_total{source=3D"archive"}' $rpms +wait_ready $PORT1 'scanned_total{source=3D".rpm archive"}' $rpms =20 kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->d= wz->srefs # Expect all source files found in the rpms (they are all called hello.c :) @@ -191,11 +192,11 @@ sourcefiles=3D$(find -name \*\\.debug \ cd .. rm -rf extracted =20 -wait_ready $PORT1 'found_sourcerefs_total{source=3D"archive"}' $sourcefiles +wait_ready $PORT1 'found_sourcerefs_total{source=3D".rpm archive"}' $sourc= efiles =20 -# Run a bank of queries against the debuginfod-rpms test cases +# Run a bank of queries against the debuginfod-rpms / debuginfod-debs test= cases =20 -rpm_test() { +archive_test() { __BUILDID=3D$1 __SOURCEPATH=3D$2 __SOURCESHA1=3D$3 @@ -221,14 +222,14 @@ rpm_test() { # common source file sha1 SHA=3Df4a1a8062be998ae93b8f1cd744a398c6de6dbb1 # fedora30 -rpm_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.= 0-2.x86_64/hello.c $SHA -rpm_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.= 0-2.x86_64/hello.c $SHA +archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello= 2-1.0-2.x86_64/hello.c $SHA +archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello= 2-1.0-2.x86_64/hello.c $SHA # rhel7 -rpm_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0= /hello.c $SHA -rpm_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0= /hello.c $SHA +archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello= -1.0/hello.c $SHA +archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello= -1.0/hello.c $SHA # rhel6 -rpm_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0= /hello.c $SHA -rpm_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0= /hello.c $SHA +archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello= -1.0/hello.c $SHA +archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello= -1.0/hello.c $SHA =20 RPM_BUILDID=3Dd44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir,= for a later test =20 @@ -276,11 +277,13 @@ if type dpkg-deb 2>/dev/null; then cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D kill -USR1 $PID2 # All debs need to be in the index - debs=3D$(find D -name \*deb | wc -l) - wait_ready $PORT2 'scanned_total{source=3D"archive"}' `expr $debs` + debs=3D$(find D -name \*.deb | wc -l) + wait_ready $PORT2 'scanned_total{source=3D".deb archive"}' `expr $debs` + ddebs=3D$(find D -name \*.ddeb | wc -l) + wait_ready $PORT2 'scanned_total{source=3D".ddeb archive"}' `expr $dde= bs` =20 # ubuntu - rpm_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" "" + archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" "" fi =20 rm -rf $DEBUGINFOD_CACHE_PATH