MN production parser down

[electricitymapsbot]

Description

This is an automatic error report generated for Mongolia (MN).

Issues:

• No recent data found for consumption parser
• No recent data found for production parser

Suggestions

• Try running the parser locally using the command poetry run test_parser MN production
• Explore the runtime logs

You can see an overview of all parser issues here.

[q--]

See #5905

[electricitymapsbot]

Closing this as the electricity data is flowing again.

[electricitymapsbot]

Reopening this as the electricity data is no longer flowing.

[q--]

The endpoint (used by the parser, but also the website) now redirects to a login page instead of returning the JSON.

This is probably a mistake, as the web page still prominently features the image in which the generation was displayed.

Their e-mail if someone wants to contact them: [email protected]

[electricitymapsbot]

Closing this as the electricity data is flowing again.

[electricitymapsbot]

Reopening this as the electricity data is no longer flowing.

[q--]

Same issue as before

The endpoint (used by the parser, but also the website) now redirects to a login page instead of returning the JSON.

This is probably a mistake, as the web page still prominently features the image in which the generation was displayed.

Their e-mail if someone wants to contact them: [email protected]

[tonypls]

Bump, still an issue. Would greatly appreciate any help on this one 🙏

[q--]

#5906 (comment)

I checked again, this time it isn't due to the endpoint being down as I can see the JSON response at https://disnews.energy.mn/convertt.php. Not sure what's causing it this time, then.

[tonypls]

Ah thanks! I had a look and it appears there's a key error:

File "/contrib/parsers/MN.py", line 96, in fetch_production
query_data = query(session, logger, zone_key)
File "/contrib/parsers/MN.py", line 82, in query
query_result = parse_json(response_json, logger, zone_key)
File "/contrib/parsers/MN.py", line 62, in parse_json
query_data[query_key] = float(web_json[src_key])
KeyError: 'sums'

Perhaps they've changed the data structure

[VIKTORVAV99]

Ah thanks! I had a look and it appears there's a key error:

File "/contrib/parsers/MN.py", line 96, in fetch_production

query_data = query(session, logger, zone_key)

File "/contrib/parsers/MN.py", line 82, in query

query_result = parse_json(response_json, logger, zone_key)

File "/contrib/parsers/MN.py", line 62, in parse_json

query_data[query_key] = float(web_json[src_key])

KeyError: 'sums'

Perhaps they've changed the data structure

I can take a look at it later today then if you wish?

[silimotion]

Hi!

It seems they have also updated the webpage recently:

Together with the new response from the endpoint:

{"date":"2024-06-11 17:00:00","syssum":"940.5","dts":630.4,"sumnar":29.2,"sumsalhit":45.3,"energyimport":"230.1","t":"23.5","Songino":"0.0"} 

In theory the only changes needed would be to update the keys that have changed, and add the new ones to the ignored list. These new ones would be "dts", which seems to be thermal power plants (they use the same icon to refer to these in the pdf below) and "Songino", which I'm quite unsure but may refer to a place in Mongolia.

[q--]

"Songino", which I'm quite unsure but may refer to a place in Mongolia.

https://www.adb.org/sites/default/files/project-documents/48030/48030-001-tacr-en_2.pdf page 42 (45 in the PDF) says that it's the name of a substation (and probably a place nearby as well, I'd assume).

[q--]

I've looked into the 'Songino' key. TL;DR it's battery storage.

---

How I've found this out:

1. Search in the Firefox Debugger for the word Songino => one match in the file https://ndc.energy.mn/_next/static/chunks/app/page-e5af86de40973fd7.js
2. Beautify the JavaScript with https://jsbeautifier.org
3. There are bits of code related to creating SVG images; these images are displayed on the web page. To be safe, I first tried to search for the key we know corresponds to solar (sumnar), extracted the image from the code fragment with ChatGPT and got, as expected, the solar panel icon .
4. Then I did the same for the Songino-related code and got the battery icon .

So, we can conclude it's battery storage.

Unfortunately I haven't actually seen values for this yet other than "0.0".

[silimotion]

I agree that "songino" is battery storage, but I remember that when I looked into it I found several problems which led me to believe then that getting useful information was impossible. Maybe someone can come up with a better idea though:

Firstly, there is an unknown quantity, which we assume that is hydro but may include random oil generators, etc that is not reported as a value but needs to be inferred from the sum of the other values and the total sum. Up to here there is no problem.

However, due to the battery system, the sum gets mangled. It seems that when the value is positive (the battery exports power) the value is added to the syssum. When it consumes power though, it appears to me that it is not added. I believe this because, if I recall correctly, I have seen the other values (dts, sumnar, sumsalhit, import) add more than syssum.

If there was only the battery we could just calculate consumption or subtract its value when positive to get the value as if there was no battery. With both unknown quantities this becomes impossible under the current data system. It is also impossible to consider together the unknown and the battery as they may add negative and production can't be negative.

All this is based on speculation as to what does the source really mean based on several measures. It is truly a black box and we could never be sure we are understanding correctly the meaning of all the data.

And add to these other shenanigans such as the fact that solar is sometimes negative (only by a mere 0.2 MW). It could be possible to produce a statistic for the coal production, or wind, etc, but it would be a feat of utmost difficulty to produce a meaningful carbon footprint, especially in a hourly basis.

Overall I think that this source is of very poor value, and that it would be best if someone could contact the source (maybe someone knows Mongolian or more probably Russian?) and ask them to adhere to the usual reporting conventions or at least explain the meaning of the values. The source is quite unofficial though (we're using here a random API call used as decoration on the page, not the main content of an official reporting page) so it is possible that they won't care enough or we discover after asking that this data has up to 5% error or whatever.

Otherwise the only two solutions I imagine are:

• Only using consumption and import and calling it a day
• Add unknown and battery, say it is battery, and put a notice somewhere on the app explaining that it is actually this lovecraftian sum. I don't know what the emission factor for this could be.

[q--]

However, due to the battery system, the sum gets mangled. It seems that when the value is positive (the battery exports power) the value is added to the syssum. When it consumes power though, it appears to me that it is not added. I believe this because, if I recall correctly, I have seen the other values (dts, sumnar, sumsalhit, import) add more than syssum.

Other possible explanations for more generation than consumption: transmission losses, either consumption or production data being somewhat delayed w.r.t. the other, or gross generation (so including self-consumption) being reported instead of net. Doesn't have to be related to the battery, and TBH it would surprise me a bit if that was the case.

Also, IIRC ElectricityMaps has other countries where we have generation but not consumption for hydro storage or battery, so I don't think it should be a blocker.

solar is sometimes negative (only by a mere 0.2 MW).

Self-consumption being reported is a pretty normal thing we've seen in other parsers too.