A way to have a parser automatically report non-critical errors

[q--]

For example, the Panama parser has a list of power plants it maps to generation type.

Currently it logs a warning when there's a new plant it cannot yet map to a generation type.

It would be nice to, for example, to have @electricitymapsbot open an issue automatically for this kind of thing. Of course you could get that by throwing an error, but then you'd break the parser for something non-critical.

[VIKTORVAV99]

We are currently looking into making it report exchange errors as well. So this could potentially be part of a larger restructuring of the bot.

cc @madsnedergaard

[madsnedergaard]

It's a good point, but how do we best determine when logs should create issues or not? For some parsers we have warnings that are not actionable where an issue would not make sense - here's a few examples:

# From FR.py
logger.warning(f"Setting small value of {key} ({value}) to 0.")
# From ENTSOE.py
logger.warning(f"Self-consumption data not yet available for {zone_key} at {dt}")
# From OPENNEM.py
logger.warning(f"Dropping columns {df.columns[is_duplicated_column]} that appear more than once")

[VIKTORVAV99]

Hmm the top example should probably be moved to the parser classes and not log anything at all since it is expected to work this way?

Not sure about the self consumption, this seems like a valid warning but I'm also not sure if we actually use the self consumption for anything.

The third example should probably be more like a info or debug since it's doing what is expected.

So maybe the actionable item here would be to clean up the parser logging a bit by either using debug or info instead of warnings for more trivial things and reserve warnings for things that deserve our attention ASAP? Like the unexpected inclusion of an additional json key in the response or similar that can impact our data quality.

[q--]

I could also imagine a separate method being added that one can call specifically for actionable-but-not-critical parser issues

[madsnedergaard]

I think that's probably the best solution, to somehow mark specific log statements as "issue-triggering". Otherwise individual contributors would need to know of this hidden side effect of using a .warning() :)

We could either create a separate method or add a parameter to warning, what makes most sense?

[VIKTORVAV99]

A new parameter to warning makes it easy to upgrade the parsers with minimal code changes. Perhaps something like raise_alert: bool?

[q--]

A new parameter to warning makes it easy to upgrade the parsers with minimal code changes. Perhaps something like raise_alert: bool?

My suggestion would be is_actionable