Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: upgrade to datafusion 43 #2886

Merged
merged 14 commits into from
Nov 21, 2024
Merged

chore: upgrade to datafusion 43 #2886

merged 14 commits into from
Nov 21, 2024

Commits on Nov 20, 2024

  1. chore: bump kernel

    @ion-elgreco @rtyler
    ion-elgreco authored and rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    f57d86e View commit details
    Browse the repository at this point in the history

  2. chore: adopt new datafusion crate 

    Signed-off-by: R. Tyler Croy <[email protected]>
    @rtyler
    rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    e45561b View commit details
    Browse the repository at this point in the history

  3. fix: upgrade python ABI to 3.9 from 3.8 

    The release of pyo3 0.22.3 compells this since we cannot otherwise
compile. The choice is between pinning 0.22.2 and upgrading our ABI, and
I think it's better to upgrade the ABI

Signed-off-by: R. Tyler Croy <[email protected]>
    @rtyler
    rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    acc1817 View commit details
    Browse the repository at this point in the history

  4. fix: adopt the right array item name which changed in kernel 0.3.1 

    see delta-io/delta-kernel-rs#301

Signed-off-by: R. Tyler Croy <[email protected]>
    @rtyler
    rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    5c26eeb View commit details
    Browse the repository at this point in the history

  5. chore: drop python 3.8 support

    @ion-elgreco @rtyler
    ion-elgreco authored and rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    48f8db1 View commit details
    Browse the repository at this point in the history

  6. chore: add a rust test demonstrating challenges with updates and lists 

    Signed-off-by: R. Tyler Croy <[email protected]>
    @rtyler
    rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    57703d7 View commit details
    Browse the repository at this point in the history

  7. chore: upgrade to the latest datafusion 43 

    Signed-off-by: R. Tyler Croy <[email protected]>
    @rtyler
    rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    22012d3 View commit details
    Browse the repository at this point in the history

  8. chore: upgrade all version ranges for the datafusion 43 ABI change 

    Signed-off-by: R. Tyler Croy <[email protected]>
    @rtyler
    rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    7b4845d View commit details
    Browse the repository at this point in the history

  9. chore: add trait annotations for compilation with datafusion 43 

    Signed-off-by: R. Tyler Croy <[email protected]>
    @rtyler
    rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    e831776 View commit details
    Browse the repository at this point in the history

  10. chore: bump kernel

    @ion-elgreco @rtyler
    ion-elgreco authored and rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    dfdf463 View commit details
    Browse the repository at this point in the history

  11. fix: import DataType for the datafusion tests

    @rtyler
    rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    0d02b26 View commit details
    Browse the repository at this point in the history

  12. fix: address build failures with the newer kernel

    @rtyler
    rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    6dc38ef View commit details
    Browse the repository at this point in the history

  13. chore: datafusion 43 updates, use projected schema in some places 

    Signed-off-by: Stephen Carman <[email protected]>
    @hntd187 @rtyler
    hntd187 authored and rtyler committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    9c46985 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2024

  1. fix: disable datafusion optimizers which perform list field name erasure 

    Today the make_array function from Datafusion uses "item" as the list
element's field name. With recent changes in delta-kernel-rs we have
switched to calling it "element" which is more conventional related to
how Apache Parquet handles things

This change introduces a test which helps isolate the behavior seen in
Python tests within the core crate for easier regression testing

Signed-off-by: R. Tyler Croy <[email protected]>
    @rtyler
    rtyler committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    b144cf1 View commit details
    Browse the repository at this point in the history