Added database feature.

[RaBa64]

The database ist stored persistently on LittleFS.
The AC total energy is written every hour to the database, together with a timestamp. Each entry to the database requires 8 bytes on the LittleFS partition. The database can be read with the API call /api/database

ToDo:
UI component is still missing.
Modification for chunked response needed.

Ralf Bauer

P.S.: Thanks for the great job on OpenDTU !

[ituri]

Looking excellent. Really hoping for this to get merged.🙏

Btw, your inverter's serial bumper is visible in the screenshot. You might want to edit that out.

[RaBa64]

I have one last problem with the code.
I get an error when building the distribution files with "yarn build".
It says that it cannot find the google library.
Everything works fine with "yarn dev".
Maybe somebody can help me with that (it is my first Vue project).

[RaBa64]

Looking excellent. Really hoping for this to get merged.🙏

Btw, your inverter's serial bumper is visible in the screenshot. You might want to edit that out.

Thank you for the notification about the serial number.
I will upload a new screenshot.

[tbnobody]

Just a few first remarks:

• Do not include libraries from external servers (maybe the google chart lib has to be replaced by something more lightweight, don't now how much space it requires now)
• You placed the chart in the inverter section of the live view. This is very confusing when having multiple inverters
• You edited parts of the vue code without changing any logic in this code sections
• Each hour consumes 9 byte of data. That means ~80kb / year. The size of the data partition is ~200kb. LittleFS is using copy on write. That means after ~1,5 years you cannot any additional data.
• Due to the copy on write behavior the writing duration of writing the data will increase over time. Also the wear out of the flash will increase over time as the whole file will be rewritten.

[RaBa64]

Thank you for your comments, let me clarify a few points.

• The Google Chart lib does not require any space on the ESP32, except for the Vue wrapper. The library is directly loaded and run by the browser. I also want to implement a calendar chart to show the enery for each day.
• Yes this is a bug, the chart must be outside the inverter section. I'll fix that.
• I'm working with Vue.js for 2 days now, so please be patient with me ;-)
• Yes its 8 bytes per hour with energy generation. So there will be no data stored during night hours. In summer this makes around 15-16 data points, in winter it will be much less. Calculating with an average of 10 data points per day, it will be 3650 data points or 29.200 bytes per year. So 200kB is enough for almost 7 years. It would also be possible to implement a purge of old data. There should also be no excessive wear on the FLASH. Calculating with 100.000 write cycles, the FLASH should at least last for 27 years, without taking the wear leveling into account.
• It is also possible to keep all data points for one day in RAM and write them only once a day to the filesystem. As long as you don't unplug the power supply this will work fine. All you would loose is one day of data.

[tbnobody]

• The Google Chart lib does not require any space on the ESP32, except for the Vue wrapper.

So it will not work offline or in AP mode. And you are transfering data to google in US (which will require a DSGVO hint and a Opt-Out)..... As I said... please no libraries from external servers....

So 200kB is enough for almost 7 years.

But you cannot extend a 150kb by 10bytes if you have only 50kb left... As I said, LittleFS is a copy-on-write filesystem. That means at each file operation, it creates a copy of the file, makes the changes and adjusts the file pointer in the last step to be 100% sure that there is no corrupted file system.

[RaBa64]

• Only the Google Chart library code (JS) is loaded in the browser. The rendering of the charts is done in the browser without sending any data to Google servers. But I agree, this feature can only be used online.
• You are right, LittleFS is a COW filesystem. But fortunately it is implemented very intelligently. If you append data to a file, it does not copy the already existing data (see here).

[tbnobody]

Only the Google Chart library code (JS) is loaded in the browser.

Thats enough to require a DSVGO hint because the IP address of the fetch operation is transmitted to google.

[t2ton]

Hi guys,
I think 1 hour database is not really usefull. Even for possible troubleshooting of the system it is way too rough and slow. In same way one could just saved a day total close to midnight time..
I would really like (as many others) an SD card offline logging feature. It shouldn't be hard to code. Simple saving "time: power" feature. Probably with option in GUI on time step. way more data, but who cares - SD capacities are huge nowadays. I personally would like to have a shortest step (5s?), as it may be used afterwards for our PhD students on simulation of solar light utilization in different off-grid systems.
One can of course do one another ESP32 which will just ask for data and save, but it is somehow an overkill - one has one MC already, which works 24/7 right?
So it would be great to have SD feature. Or you just dont want to steal the cookies from Ahoi guy? :)

[RaBa64]

@t2on, I have also thought about stoing the data on an SD-card, but I decided to stay with my current solution and store all high resolution data in my InfluxDB, which is much easier and faster to query. This is pretty easy to do by using MQTT and Telegraf.
I use the 25 hours chart in OpenDTU only for a brief overview of the energy production.
What is still on my todo list is to integrate a calendar chart, besides the bar chart, to have also an overview on a day by day basis for all years, similar as the one on the original Hoymiles website / App.

[stefan123t]

• Yes its 8 bytes per hour with energy generation. So there will be no data stored during night hours. In summer this makes around 15-16 data points, in winter it will be much less. Calculating with an average of 10 data points per day, it will be 3650 data points or 29.200 bytes per year. So 200kB is enough for almost 7 years. It would also be possible to implement a purge of old data. There should also be no excessive wear on the FLASH. Calculating with 100.000 write cycles, the FLASH should at least last for 27 years, without taking the wear leveling into account.

• It is also possible to keep all data points for one day in RAM and write them only once a day to the filesystem. As long as you don't unplug the power supply this will work fine. All you would loose is one day of data.

@RaBa64 thanks for your PR / fork of OpenDTU! I still could not get it to run / flash on my OpenDTU Fusion board for some unknown reason.

I come from a background / long time usage of the Round Robin DB (RRDtool) from Tobi Oetiker which is also available for ESP32 https://github.com/lbernstone/rrdtool_ESP32.

Have you considered storing your data in some RRD databases, i.e. you would need

• 31 x 24 x 8 bytes = 5952 bytes for a daily store of hourly data for the last month
• 12 x 31 x 8 bytes = 2976 bytes for a yearly store of daily data for the past year
• 10 x 12 x 8 bytes = 960 bytes for a store of monthly data for the past ten years/decade

That is a total of 9888 / <10kB which you need to store every time you update it, i.e. 24 x 365 = 8760 times or <10k times a year.
With our 100.000 write cycles this should allow for storing the three RRD files / chunks for about the next 10 years until your flash may wear out.

If you reduce that to only one daily writing it would require only ~365 writes a year, which would wear out after ~273 years 😄
So this should be the safe default setting and in case users want to cope with the additional wear and tear of their flash memory they can enable the hourly storage scheme for added hourly details across potential power cuts.

[stefan123t]

@RaBa64 how does this compare with #2125 which implements SD Card storage in a CSV file ?