2024-07-18 kernel meeting notes

[zachschuermann]

summary

0.2.0 released! big thing: read parquet correctly #271

action items

• Push parquet select to leaves, add correct reordering #271
• First pass on IN/Not In #270

attendees

@zachschuermann @nicklan @roeap @hntd187

notes stream

• 0.2.0 released!
• types separation into its own crate is planned work
• PR prefixes are nice - tag with feat: bug: etc.?
• automation for changelog?
• versioning: major 0, minor bumps = breaking API changes, patch bumps = no breaking API
• release automation would be nice
• different versions for different crates in delta-rs? note: arrow takes drastic approach - release everything together and simultaneously bump versions
• [WIP] updates for delta-rs #189 integration plans/delta-rs plans? @roeap: first to adopt is expressions; remove some more code (partition filtering is small scale version of expr system). sql parsing to expressions (filters etc) useful. omission in delta protocol: SQL! (e.g. we say write invariant in sql but no details)
• incubate sql parser etc in delta-rs?
• after: parquet checkpoints - delta-rs has low level parquet code laying around. but now backend is all arrow. some code to remove (including checkpoint reading and writing - and v2 checkpoints). people want v2 checkpoint since liquid depends on it
• nick: v2 checkpoint is on the list. might bump up prio