delta-rs forward compatibility with Spark/Flink #2038
Answered
by
ion-elgreco
EldarSehayekZenity
asked this question in
Q&A
-
|
Hi, I'm strongly considering adopting delta lake as the OTF for our new data pipeline. Will Spark/Flink be able to handle the delta lake files created by delta-rs? |
Beta Was this translation helpful? Give feedback.
Answered by
ion-elgreco
Jan 5, 2024
Replies: 1 comment
-
|
Spark should be able to read them, we follow the protocol. But there are cases where spark-delta doesn't follow the protocol with for example parsing the transaction log to show the history. This is obviously incorrect but has not been addressed yet in spark-delta |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
rtyler
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Spark should be able to read them, we follow the protocol.
But there are cases where spark-delta doesn't follow the protocol with for example parsing the transaction log to show the history. This is obviously incorrect but has not been addressed yet in spark-delta