Is there a possibility to create a checkpoint file at custom location?

[ashishkaul01]

Major Question:
Is there a possibility to create a checkpoint file at custom location?

Explanation
We have a deltatable inside an AWS s3 bucket which would be accessible to multiple parties.
Since all the parties can call the checkpoint() function to create their checkpoint files we wanted to check if there was a possibility that each one could create the checkpoint file at a custom location instead of the default location.

Ideally, we would save the checkpoint files at the AWS s3 bucket of individual party.

[ion-elgreco]

The protocol states this:

Checkpoints are also stored in the _delta_log directory, and can be created at any time, for any committed version of the table

https://github.com/delta-io/delta/blob/master/PROTOCOL.md#checkpoints

[rtyler]

Checkpoints represent written state of the Delta table and are there to provide for more optimized reads. If writes are coming from different applications, I wouldn't worry about independent creation of checkpoint files since they're just a summation of the transaction log.

I would however be concerned about different applications not safely writing transactions on S3 since we require a dynamodb table for writer coordination currently

[ashishkaul01]

Thanks @rtyler for the response. The writing part is not a challenge as you also explained.

Just to provide a little more context. We have the following scenario:

We are currently reading the Change Data Feed from the Deltatble.

1. Deltatable/ Change Data Feed has 4 records.
2. Application 1 explicitly creates a checkpoint file (Checkpoint _0.parquet)
3. Additional 4 records are added.
4. Add 6 more records.
5. Application 2 creates a checkpoint file (Checkpoint_1.parquet)

Now Checkpoint_1 contains only the transactions which happened after Step-2.

If Application 1 wants to read the Change Data Feed again, it must either provide a starting version or starting timestamp. But the challenge is to find that version number. Currently there are 2 checkpoint files, with more than 2 applications there could be multiple checkpoint files.

This raises two questions for us:

_1. How would Application 1 know which checkpoint file was created by IT? My assumption is that if we are either able to provide a custom location to store the checkpoint files OR a custom name for the checkpoint files this would be possible.

2. Alternatively, would you know how Application 1 can read changes ONLY after checkpoint_0.parquet and Application 2 ONLY after checkpoint_1.parquet._

[ww917352]

@rtyler @ion-elgreco

We would be grateful to any guidance or suggestion from your side.

[ion-elgreco]

Checkpoints have a different goal then you are expecting it to be used for. They are used to improve the speed of determining the state of the table by having the add/remove/metadata/protocol actions stored in a more efficient manner.

None of the readers should call checkpoint, only a writer should. If you want to know what you reader has already read or not, just maintain the read_last_version elsewhere.

An approach I have done in the past is where you add a column "version" into the target table, so all records you write into that target you know came from a certain version in the source. By reading the max version of the target you can derive where to start reading the source from. But you could also just maintain this last_read_version in some sort of state file.

[ww917352]

Thanks for the explanation. This is extremely helpful!