🐛 [BUG] - Datadog exporter - Custom metrics quota

[snahelou]

SLO Generator Version

v2.3.3

Python Version

3+

What happened?

Hello everyone,

It look like the way you export data to Datadog is a little bit weird: the metrics events_count export data in label instead of value.

Example:

INFO - test-slo | 28 days | SLI: 99.9954 % | SLO: 99.95 % | Gap: +0.05 % | BR: 0.1 / 1.0 | Alert: 0 | Good: 8868457 | Bad: 406

Will give the following metric
events_count{good_events_count: 8868457,bad_events_count: 406,... }

The issue with this implementation is that having a big granularity in label will consume all custom metrics quota.

What did you expect?

I expected to have only 2 custom metrics by default

events_count{type: bad_event, ...} 406
events_count{type: good_event, ...} 8868457

Screenshots

![DESCRIPTION](LINK.png)

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[bkamin29]

Hi @snahelou,
a workaround i currently use is customize metrics and labels on exporters.
Have a look on this documentation https://github.com/google/slo-generator/blob/master/docs/shared/metrics.md

This is an example of my context :

  "cloud_monitoring/default":
    "metrics":
    - "error_budget_burn_rate"
    - "events_count"
    - "good_events_count"
    - "bad_events_count"
    - "sli_measurement"
    "project_id": "xxxxxxx"

The same configuration can be use with any other exporter like datadog

Maybe for your need you can replace events_count by good_events_count and bad_events_count

[snahelou]

To be honest, I don't use it anymore. The issue is just to "alert" people who want to try it with the default config :)