kvserver,admission: per tenant metrics in the shared kv/storage server

[sumeerbhola]

We seem to lack per-tenant metrics in the shared server. This hinders troubleshooting.
Examples of useful metrics:

• Admission WorkQueue latency histograms per tenant
• Intent resolution rate per tenant
• TODO: add more here

Per-tenant metrics need to account for the fact that over the lifetime of a shared server process it may see 1000s of active tenants. Based on past experience with multi-tenant systems, the set of active tenants is often in the 10s (on a server). The approach that works is to have in-memory metric storage only for the active ones, and hence only those get exported as timeseries. This results in 1000s of timeseries over a long interval (say a 30d interval) but for the interval of interest (say an hour) the number of timeseries is small. Some approaches I have seen in the past:

• Gauge metrics: In-memory state for points with a 0 value is GC'd.
• Cumulative metrics: One of two approaches:
  • GC cumulative points that are not changing.
  • Export as deltas: Deltas that are 0 are GC'd and not exported.

cc: @dhartunian

Jira issue: CRDB-44829