Can I replace a partition ?

[dridk]

Hi,

I have a deltaTable partitioning monthly.

How Can I overwrite a specific partition? For instance, recreate all data for January without removing all other month?

[JHibbard]

Hi dridk,

This is possible in v0.8.0 of delta-rs and newer, see this commit for discussion and docs for details. Example code:

from deltalake.writer import write_deltalake


df = pd.DataFrame({'x': [1, 2, 3], 'y': ['a', 'a', 'b']})
write_deltalake('path/to/table', df, partition_by=['y'])

table = DeltaTable('path/to/table')
df2 = pd.DataFrame({'x': [100], 'y': ['b']})
write_deltalake(table, df2, partition_filters=[('y', '=', 'b')], mode="overwrite")

table.to_pandas()
     x  y
0    1  a
1    2  a
2  100  b

You'll have to write your partition filter so that it only affects the partition you want to overwrite.

Hope that helps!
Jim