2024-03-14 kernel meeting notes

[zachschuermann]

summary

PRs and reviews! matthew encouraging prioritizing docs.

action items

some PRs in flight:

• Switch to new data passing API #109
• start implementing FFI #91
• Fixed file path resolution so tests pass on Windows #131 (needs final review)
• remove arrow dependency from data skipping #132 : getting some checkpoint read bugs ironed out (maybe should have put in a separate pr..)
• Initial work to use maturin+pyo3 to get a python interface #130
• use spawn_blocking to try and send back read results #141 (waiting on another review)
• Use a proc macro for being able to derive schemas for our action structs #129 more reviews

attendees

@nicklan @zachschuermann @MrPowers @hntd187 @ryan-johnson-databricks

notes stream

matthew

• kernel rust: want pretty docs! (zach +100000000)
• spark-connect rust connector, matthew is meeting, trying to get them to summit
• consider spark-connect use case for something similar to kernel. kernel-like approach for native support in many languages for spark connect?
  nick
• in flight PRs, lots to review
• merge remove arrow dependency from data skipping #132 do checkpoint reads in a follow-up
  ryan
• current kernel rust doesn't push down filters etc. into reads. apply expressions on top
• to fix
  nick
• proc macro PR ready for review Use a proc macro for being able to derive schemas for our action structs #129
• how to move forward on FFI PR start implementing FFI #91
  ryan
• progress on writing an EngineInterface in duckdb (starting with parquet reading)
• got to compile on the rust side
• need to check C++ side
• starting to write EngineData wrapper for duckdb columnar chunk
• Nick: duckdb plan for not is to use default client (arrow) then do arrow --> custom duckdb in-memory format
• easy doing EngineData wrapper since APIs are similar - just need to do some unpacking during 'schema explode' and they have all the getters/setters/etc. this is just plumbing.
  tidbit: duckdb released their own file format! https://motherduck.com/blog/differential-storage-building-block-for-data-warehouse/