[ISSUE] Issue with databricks_sql_table resource - perpetual diff on effective_options for Delta tables

[beccar97]

Configuration

# Copy-paste your Terraform configuration here
resource "databricks_sql_table" "my_table" {
  name = "my_table"
  catalog_name ="my_catalog"
  schema_name = "my_schema"
  table_type = "MANAGED" 
  data_source_format = "DELTA"

  properties = {
    "delta.autoOptimize.autoCompact"   = "true"
    "delta.autoOptimize.optimizeWrite" = "true"
    "delta.enableChangeDataFeed"       = "true"
  }

 column {
   name = "name"
   type = "string"
 }

 comment = "This is a managed delta table"
}

Note I'm not confident on whether enabling change data feed or not is significant as we use that on all our tables.

Expected Behavior

When terraform apply is run, and no changes have been made to the table config, there should be no changes shown for the table in the plan.

Actual Behavior

If the data in the table has been updated then an entry will be shown in the plan making changes to the effective_properties attribute, like

  # databricks_sql_table.my_table will be updated in-place
  ~ resource "databricks_sql_table" "my_table" {
      ~ effective_properties = {
          ~ "delta.lastCommitTimestamp"                                = "1731693595000" -> "1711545334000"
          ~ "delta.lastUpdateVersion"                                  = "40" -> "7"
            # (21 unchanged elements hidden)
        }
        id                   = "my_catalog._raw.my_schema.my_table"
        name                 = "my_table"
        # (10 unchanged attributes hidden)

        # (35 unchanged blocks hidden)
    }

Steps to Reproduce

1. Create a managed delta table, which is managed by terraform
2. Update the data in the table, e.g. add a new row
3. Run a terraform plan

Terraform and provider versions

$ tf version
Terraform v1.7.5
on windows_amd64
+ provider registry.terraform.io/azure/azapi v1.15.0
+ provider registry.terraform.io/databricks/databricks v1.59.0
+ provider registry.terraform.io/hashicorp/azurerm v3.109.0
+ provider registry.terraform.io/hashicorp/null v3.2.3

Is it a regression?

This definitely didn't used to be an issue, we started seeing it a little while ago. I'm not sure if it was introduced in a particular provider - I tried to go back to figure it out but seemed to see the issue everywhere

Important Factoids

To get round the spurious changes this adds to plans, we've added effective_properties to the ignore_changes attribute of the lifecycle block which leads to "Warning: Redundant ignore_changes element" warnings on terraform commands.