Does deltalake-rs support streaming?

[dadepo]

Hi. Does anyone knows if it is possible to use deltalake-rs for streaming with datafusion in Rust? I can’t seem to find anything on this.

Spark does have it. See https://docs.delta.io/latest/delta-streaming.html#delta-table-as-a-sink I am looking at reproducing at least the reading of the stream part using datafusion.

Any pointers?

[roeap]

Haven't worked so much with streaming sources in datafusion, so take anything that follows with a grain of salt :).

The write operation in delta-rs can be configured to take a datafusion ExecutionPlan as an input. So the missing piece would be an input execution plan that reads from whatever streaming source you are reading from. StreamingTable might be what you are looking for, it implements TableProvider and creates a corresponding execution plan when scan'ed.

[dadepo]

I am actually interested only in the reading of a stream source and not the write part. I have indeed seen StreamingTable but still not sure how to create one that successfully reads from a delta table as a stream.

I see the core of having the StreamingTable seems to be the execute function on the PartitionStream trait…and looking for an example implementation I found

https://github.com/apache/arrow-datafusion/blob/01eb72af4ccdc911ba3cfe22e41f2d71389c5eb9/datafusion/core/tests/memory_limit.rs#L259

Which is basically returning the record batch as a stream.

If I am going to provide an implementation of this for a delta table, that means I need to continually execute the query that fetches the data from the table? But not sure how to continually execute an SQL against a delta table, in other to have the record batches return as a stream while keeping track of where the last query ended so that it will be the start of the next query...