WIP content-sqlite: preallocate space to avoid outside diskfull events #6217

Problem: Some new features will require a slightly larger tmpfs for testing than the current 1m one. Add an additionl 5m tmpfs mount for testing.

Problem: In the near future we may wish to open the sqlite multiple times within the content-sqlite module. The current content_sqlite_opendb() takes a "truncate" parameter that would truncate the db on every open. This is not what we want if we are closing/opening the database multiple times. Solution: Split out the truncate into a new "setup" function. This setup function will be called once when the module is loaded.

Problem: In the near future we may need to close the sqlite db and reopen it. It would be wise to re-initialize context variables when closing the db to avoid re-using older configs. Init all appropriate variables in the content-sqlite context when closing the sqlite db.

Problem: In the near future, we may need to open the sqlite db with different settings during setup. Solution: Have the content_sqlite_opendb() function take journal_mode and synchronous as input parameters.

Problem: When a disk runs out of space, the content-sqlite module can no longer work. It would be convenient if space could be reserved ahead of time to prevent these types of problems. Support a new preallocate module and config option that takes a byte maximum as input. This option will internally create a new special database and write data to that database until the byte max is reached. Then this database will be dropped. This internally reserves space for the sole use of sqlite. Note that this feature will not work if any type of journaling or write ahead log is used, as that will also require disk space.

Problem: There is no coverage for the new content-sqlite preallocate config. Add tests in t0012-content-sqlite.t and t0090-content-enospc.t.

Problem: If a disk fills up, sqlite may no longer be able to operate because journal files can no longer be written to disk. However, if space was pre-allocated, sqlite can still be used if we turn off journaling. If ENOSPC is hit and pre-allocated space was configured, turn off journaling in an attempt to keep the content-sqlite module functional, although with slower performance.

Problem: There is no coverage to test if content-sqlite preallocate works if journaling was initially enabled. Add coverage in t/t0090-content-enospc.t.

Problem: The new journal_mode, synchronous, and preallocate configs for the content-sqlite module are not documented. Add them in new doc/man5/flux-config-content-sqlite.rst.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP content-sqlite: preallocate space to avoid outside diskfull events #6217

WIP content-sqlite: preallocate space to avoid outside diskfull events #6217

Commits on Aug 20, 2024

WIP content-sqlite: preallocate space to avoid outside diskfull events #6217

Are you sure you want to change the base?

WIP content-sqlite: preallocate space to avoid outside diskfull events #6217

Commits on Aug 20, 2024