write_deltalake fails writing a simple dataset with categorical columns

[torshind]

Minimal test case to reproduce it:

from deltalake.writer import write_deltalake
from sklearn.datasets import fetch_openml

df = fetch_openml(name="arrhythmia", version=1, as_frame=True)["data"]
write_deltalake("./table_test", data=df, mode="overwrite")

Error:

Traceback (most recent call last):
  File "/home/mvm/gitlab/lakehouse/bug.py", line 5, in <module>
    write_deltalake("./table_test", data=df, mode="overwrite")
  File "/home/mvm/.pyenv/versions/studio-py3.10/lib/python3.10/site-packages/deltalake/writer.py", line 312, in write_deltalake
    _write_new_deltalake(
deltalake.PyDeltaTableError: Schema error: Invalid data type for Delta Lake: Dictionary(Int8, Utf8)

Is this a bug? Is there a way around the problem without losing the categorical variables?

[torshind]

#1326