How do you use zstd (or others) compression for parquet?

[theelderbeever]

As the question states above I would like to use zstd instead of snappy for writing parquet files to delta lake using the rust crate. I haven't been able to find anything that mentions customizing this in the writer. Any clarity would be super helpful.

[wjones127]

For Rust, this is in passed in through WriterProperties.

In Python, this isn't exposed yet. It's being tracked in #1465

[theelderbeever]

@wjones127 thanks for pointing me towards that.

Looking into some of the writer code it seems like the only place that WriterProperties is passable is in the PartitionWriter here. Are there plans to expose this field elsewhere such as on the RecordBatchWriter::try_new()? From what I can tell you have to custom roll the RecordBatchWriter{ ... } (here) to set any properties.

Or am I missing something?

[wjones127]

Oh I think you are right. I though it would be exposed on the write operation here:

https://github.com/delta-io/delta-rs/blob/0dda99bbb080d5cf8a6ecc350d0a789e7fa56549/rust/src/operations/write.rs

But it isn't 😢

[theelderbeever]

Is this worth turning into an issue?

[wjones127]

Yes, feel free to file a new issue for that.