patch PR25448: more debuginfod prometheus metrics

patch PR25448: more debuginfod prometheus metrics

[Frank Ch. Eigler]

Hi -

commit 34d851344dbaa5fd00b4368742839f86203202a1 (HEAD -> fche/pr25448)
Author: Frank Ch. Eigler <fche@redhat.com>
Date:   Thu Mar 26 16:44:20 2020 -0400

    PR25448: debuginfod: add transfer performance metrics
    
    We now export metrics related to the time taken and data sent,
    from which prometheus type tools can compute nice time series
    with averages.
    
    http_responses_duration_milliseconds_count{code="200"} 63
    http_responses_duration_milliseconds_count{code="404"} 2
    http_responses_duration_milliseconds_count{code="503"} 1
    http_responses_duration_milliseconds_sum{code="200"} 66
    http_responses_duration_milliseconds_sum{code="404"} 2
    http_responses_duration_milliseconds_sum{code="503"} 0
    http_responses_transfer_bytes_count{code="200"} 63
    http_responses_transfer_bytes_count{code="404"} 2
    http_responses_transfer_bytes_count{code="503"} 1
    http_responses_transfer_bytes_sum{code="200"} 425177
    http_responses_transfer_bytes_sum{code="404"} 18
    http_responses_transfer_bytes_sum{code="503"} 37
    
    Signed-off-by: Frank Ch. Eigler <fche@redhat.com>

diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index 5e19db517472..ebb67b0df378 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,8 @@
+2020-03-26  Frank Ch. Eigler <fche@redhat.com>
+
+	* debuginfod.cxx (handler_cb): Export two families of metrics for
+	prometheus traffic analysis: response times and data amounts.
+
 2020-03-25  Frank Ch. Eigler <fche@redhat.com>
 
 	* debuginfod.cxx (parse_opt): Associate a bsdtar subshell with
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index c03bbf922f59..e17aec862676 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1759,6 +1759,7 @@ handler_cb (void * /*cls*/,
       inc_metric("http_responses_total","result","error");
       e.report(clog);
       http_code = e.code;
+      http_size = e.message.size();
       rc = e.mhd_send_response (connection);
     }
 
@@ -1770,6 +1771,17 @@ handler_cb (void * /*cls*/,
                  << ' ' << (int)(deltas*1000) << "ms"
                  << endl;
 
+  // related prometheus metrics
+  string http_code_str = to_string(http_code);
+  if (http_size >= 0)
+    add_metric("http_responses_transfer_bytes_sum","code",http_code_str,
+               http_size);
+  inc_metric("http_responses_transfer_bytes_count","code",http_code_str);
+
+  add_metric("http_responses_duration_milliseconds_sum","code",http_code_str,
+             deltas*1000); // prometheus prefers _seconds and floating point
+  inc_metric("http_responses_duration_milliseconds_count","code",http_code_str);
+
   return rc;
 }
 
diff --git a/tests/ChangeLog b/tests/ChangeLog
index c06d892440a0..ba88ecf6aeb9 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2020-03-26  Frank Ch. Eigler <fche@redhat.com>
+
+	* run-debuginfod-find.sh: Look for debuginfod's new
+	http_responses_* metrics.
+
 2020-03-26  Frank Ch. Eigler <fche@redhat.com>
 
 	* run-debuginfod-find.sh: Look for bsdtar instead of dpkg.
diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
index cc826f9607a4..db36420f75ec 100755
--- a/tests/run-debuginfod-find.sh
+++ b/tests/run-debuginfod-find.sh
@@ -416,6 +416,10 @@ curl -s http://127.0.0.1:$PORT2/metrics
 curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
 curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*fdcache'
 curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
 
 # And generate a few errors into the second debuginfod's logs, for analysis just below
 curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true

Re: patch PR25448: more debuginfod prometheus metrics

[Mark Wielaard]

Hi Frank,

On Thu, 2020-03-26 at 16:46 -0400, Frank Ch. Eigler via Elfutils-devel wrote:
> commit 34d851344dbaa5fd00b4368742839f86203202a1 (HEAD ->
> fche/pr25448)
> Author: Frank Ch. Eigler <fche@redhat.com>
> Date:   Thu Mar 26 16:44:20 2020 -0400
> 
>     PR25448: debuginfod: add transfer performance metrics
>     
>     We now export metrics related to the time taken and data sent,
>     from which prometheus type tools can compute nice time series
>     with averages.
>     
>     http_responses_duration_milliseconds_count{code="200"} 63
>     http_responses_duration_milliseconds_count{code="404"} 2
>     http_responses_duration_milliseconds_count{code="503"} 1
>     http_responses_duration_milliseconds_sum{code="200"} 66
>     http_responses_duration_milliseconds_sum{code="404"} 2
>     http_responses_duration_milliseconds_sum{code="503"} 0
>     http_responses_transfer_bytes_count{code="200"} 63
>     http_responses_transfer_bytes_count{code="404"} 2
>     http_responses_transfer_bytes_count{code="503"} 1
>     http_responses_transfer_bytes_sum{code="200"} 425177
>     http_responses_transfer_bytes_sum{code="404"} 18
>     http_responses_transfer_bytes_sum{code="503"} 37
>     
>     Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
> 
> diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
> index 5e19db517472..ebb67b0df378 100644
> --- a/debuginfod/ChangeLog
> +++ b/debuginfod/ChangeLog
> @@ -1,3 +1,8 @@
> +2020-03-26  Frank Ch. Eigler <fche@redhat.com>
> +
> +	* debuginfod.cxx (handler_cb): Export two families of metrics for
> +	prometheus traffic analysis: response times and data amounts.

This looks nice. Do we have to document these anywhere, or are
prometheus metrics "self documenting"? I assume tools just take what
they can get from the json export and create graphs based on the name
count/sum patterns?

Thanks,

Mark

Re: patch PR25448: more debuginfod prometheus metrics

[Frank Ch. Eigler]

Hi -

On Fri, Mar 27, 2020 at 03:47:27PM +0100, Mark Wielaard wrote:
> This looks nice. Do we have to document these anywhere, or are
> prometheus metrics "self documenting"? I assume tools just take what
> they can get from the json export and create graphs based on the name
> count/sum patterns?

Yeah, the naming convention of related metrics (_sum, _count) allows
averages to be auto-calculated.  No, we don't document any of these
formally (and note that in the man page).

Thanks for the review.

- FChE