Add support for merging chunks #7433
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erimatnor
force-pushed
the
merge-chunks
branch
5 times, most recently
from
ce949f8
to
581388c
Compare
erimatnor
commented
Nov 12, 2024
| @@ -44,6 +44,10 @@ CREATE OR REPLACE FUNCTION @[email protected]_chunk( | |||
| if_compressed BOOLEAN = true | |||
| ) RETURNS REGCLASS AS '@MODULE_PATHNAME@', 'ts_decompress_chunk' LANGUAGE C STRICT VOLATILE; | |||
|
|
|||
| CREATE OR REPLACE PROCEDURE @[email protected]_chunks( | |||
There was a problem hiding this comment.
I put the function in the extension schema instead of _timescaledb_functions since it is a function in line with compress_chunk, etc. But we could move it to _timescaledb_functions if we want this to be less of a public feature. 🤷
There was a problem hiding this comment.
I think this is on the same level as compress_chunk, drop_chunk, and friends so we should keep it at the same place as these.
erimatnor
force-pushed
the
merge-chunks
branch
3 times, most recently
from
ba6ea6a
to
8d36316
Compare
erimatnor
force-pushed
the
merge-chunks
branch
from
8d36316
to
66850c7
Compare
A new procedure called `merge_chunks` is introduced that can merge an arbitrary number of chunks if the right conditions apply. Basic checks are done to ensure that the chunks can be merged from a partitioning perspective. Some more advanced cases that are potentially mergeable are not supported at this time (e.g., complicated merges of chunks with multi-dimensional partitioning). Currently, the merge defaults to taking an AccessExclusive lock on the merged chunks to prevent deadlocks and concurrent modifications. Weaker locking is supported via an anonymous settings variable, but this is mostly to prove in tests that these approaches can lead to deadlocks. The actual merging is done by rewriting all the data from multiple chunks into a (temporary) merged heap using the same approach as that implemented to support VACUUM FULL and CLUSTER. Then this new heap is swapped into one of the original relations while the rest are dropped. This approach is MVCC compliant and implements correct visibility under higher isolation levels, while also doing vacuum and leaving no garbage tuples.
erimatnor
force-pushed
the
merge-chunks
branch
from
66850c7
to
bd20e92
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A new procedure called
merge_chunksis introduced that can merge an arbitrary number of chunks if the right conditions apply. Basic checks are done to ensure that the chunks can be merged from a partitioningperspective. Some more advanced cases that are potentially mergeable are not supported at this time (e.g., complicated merges of chunks with multi-dimensional partitioning).
Currently, the merge defaults to taking an AccessExclusive lock on the merged chunks to prevent deadlocks and concurrent modifications. Weaker locking is supported via an anonymous settings variable, but this is mostly to prove in tests that these approaches can lead to deadlocks.
The actual merging is done by rewriting all the data from multiple chunks into a (temporary) merged heap using the same approach as that implemented to support VACUUM FULL and CLUSTER. Then this new heap is swapped into one of the original relations while the rest are dropped. This approach is MVCC compliant and implements correct visibility under higher isolation levels, while also doing vacuum and leaving no garbage tuples.