Adding more fields to the History

[bnewman-tech]

I am trying to enhance the history. I would like to add username and userMetadata.

https://docs.delta.io/latest/delta-utility.html#history-schema

I am currently doing this.

    write_deltalake(
        table_or_uri=file_path,
        data=data_df,
        mode=mode,
        storage_options=storage_options,
        overwrite_schema=overwrite_schema,
    )

Currently, the dt.history() is giving the following.

	{
		"timestamp": 1701364256524,
		"operation": "WRITE",
		"operationParameters": {
			"partitionBy": "[]",
			"mode": "Overwrite"
		},
		"clientVersion": "delta-rs.0.17.0",
		"version": 1
	}

Is there any way to add more to the history log?
Appreciate any insight yall can provide.

[roeap]

the commit info is actually a quite free structure. the info in the link is actually what spark is adding, and delta-rs tries to be similar to that.

In principle it is possible to add custom metadata to a commit - not sure if we expose that in the python write command though. It is however important to know, that commit provenance information is not retained across checkpoints and will be completely lost once the table is vacuumed.

There are potentially other means though to track some application metadata depending on the use case.

[bnewman-tech]

Thank you for your reply.

My main interest is in capturing the username of the person performing each action.

Do you know if there is a plan to add the Python Binding for this?

[bnewman-tech]

Good morning,
Wanted to follow up and see if there was any way to add any custom information to the history log via the Python Bindings?

[bnewman-tech]

This is complete! Thank you!!

You can use the custom_metadata property and pass any key:value pair.

#2032

https://github.com/delta-io/delta-rs/releases/tag/python-v0.15.1