Reading Delta Table written by Spark with list field.

[alexwilcoxson-rel]

Hey there,

We have a table with a field of type List. When written with pyspark the list's entries are named "element". delta-rs is using "item" as the name. This makes it so we cannot read this table using delta-rs.

I found this discussion: https://github.com/delta-io/delta-rs/pull/684/files#r940790524

It references an upstream change? Is that something still pending or would there be further investigation/work required to support this?

Test and data to reproduce

use deltalake::DeltaOps;
use futures::StreamExt;

#[tokio::test]
async fn test_read_array_table() {
    let ops = DeltaOps::try_from_uri("./tests/data/array_table").await.unwrap();
    let (_, mut stream) = ops.load().await.unwrap();
    while let Some(maybe_batch) = stream.next().await {
        let batch = maybe_batch.unwrap();
        println!("{:?}", batch);
    }
}

array_table.zip

output

thread 'test_read_array_table' panicked at 'called `Result::unwrap()` on an `Err` value: ExternalError(Execution("Failed to map column projection for field array. Incompatible data types List(Field { name: \"element\", data_type: Float32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) and List(Field { name: \"item\", data_type: Float32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })"))', rust/tests/read_array_table.rs:11:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test test_read_array_table ... FAILED

this is the spark code (run on databricks) that created the table

from pyspark.sql import functions as F
from pyspark.sql import types as T

data = [
    [0, [1.0, 2.0, 3.0]],
    [1, [4.0, 5.0, 6.0]],
]

schema = T.StructType([
    T.StructField("id", T.IntegerType()),
    T.StructField("array", T.ArrayType(T.FloatType())),
])

df = spark.createDataFrame(data, schema=schema)
df.write.format("delta").mode("overwrite").save(PATH)
df = spark.read.format("delta").load(PATH)
df.printSchema()
df.show()

output

root
 |-- id: integer (nullable = true)
 |-- array: array (nullable = true)
 |    |-- element: float (containsNull = true)

+---+---------------+
| id|          array|
+---+---------------+
|  0|[1.0, 2.0, 3.0]|
|  1|[4.0, 5.0, 6.0]|
+---+---------------+