Generic deepmerge() function

[Zordrak]

Terraform Version

1.9.8

Use Cases

Merging of files read via jsondecode and yamldecode, merge of nested maps and objects tree typically used for configuration or lookups.

This request was previously submitted by @scholli as #31815 and rejected by @apparentlymart

The rejection of this ticket came as surprise and brought sadness to all concerned in the community. The reason given was that as there are multiple ways to offer a deep merge, terraform should not provide any at all, and custom providers should be developed by anyone who has need of deep merge functionality.

While I understand the desire to encourage community-driven development, I respectfully disagree with the reasons provided for rejecting this request and ask that it be reconsidered.

There are numerous scenarios where even a basic deep merge routine would significantly reduce code complexity. Consider the following example:

locals {
  rest_api_body_source = yamldecode(file(local.api_body_source))

  rest_api_components_securityschemes_authorizer = var.b2c == null ? {} : {
    authorizer = {
      in   = "header"
      name = "Authorization"
      type = "apiKey"

      x-amazon-apigateway-authtype = "custom"

      x-amazon-apigateway-authorizer = {
        authorizerCredentials        = module.b2c[0].authorizer_lambda_invoke_role_arn
        ...
      }
    }
  }

  rest_api_components_securityschemes = {
    securitySchemes = merge(
      try(local.rest_api_body_source["components"]["securitySchemes"], {}),
      local.rest_api_components_securityschemes_authorizer,
    )
  }

  rest_api_components = {
    components = merge(
      try(local.rest_api_body_source["components"], {}),
      local.rest_api_components_securityschemes,
    )
  }

  rest_api_body = yamlencode(merge(
    local.rest_api_body_source,
    local.rest_api_components,
  ))
}

EDIT: I have had to update this example since posting it - because I made a mistake in it - which is really easy to do when you're manually writing a deep merge instead of using a proven function to do it.

This multi-step approach is necessary because Terraform's shallow merge would overwrite the entire components section of the API spec if performed in one step. The requirement is simple: “insert an additional key-value pair into an existing object at a level below the object root.” The solution, however, is complex and unnecessarily so. Providing even a basic deep merge function as a native feature would greatly improve this situation.

A simple parameter could allow for multiple deep merge approaches, but even a single solution permitting insertion at depth would be a major improvement and simplify workarounds for more complex needs.

The suggestion that all users requiring a deep merge should use a custom provider is problematic for several reasons:

1. Supply-Chain Security: In enterprise environments, the choice to delegate authority to a provider is not made arbitrarily. Only providers with a certain level of commercial trust and provenance are used, limiting the feasibility of using unknown 3rd party providers.
2. Development and Maintenance: Creating, maintaining, and distributing a custom provider within an enterprise is challenging, and even more so across multiple organisations. Many workload pipelines cannot feasibly implement a custom provider compared to writing more complex code that, while inefficient, gets the job done.

Many of us work with JSON and YAML data structures that need processing and occasional alteration. A deep merge function would significantly improve the code we deliver to our customers and demonstrate that Terraform is not more complicated than alternatives, but rather a superior solution.

Thank you for considering this request.

Attempted Solutions

• merge(): does not recursively merge values that are objects and maps

• Kyle Kotowik's deepmerge module: is computationally intensive (since the tree must be flattened, merged, then re-created) and is currently not usable if infracost is used for PR costing

• Provider: lots of unnecessary implementation work to create, maintain and use in every case requiring the function

Proposal

The deepmerge(list_of_items_to_merge, merge_config}) would behave like this:

Takes a list of items to merge [item1, item2, ...], and an optional merge-configuration object {list_merge_strategy="replace", type_mismatch_strategy="complex_wins"}, and returns a single item that is a merge of all items, based on the merge configuration.

The merge algorithm for list_of_items_to_merge is as follows:

• items of different types are handled based on the type_mismatch_strategy:
  • complex_wins: object and map win over lists which win over literals; the losers are removed from the merge;
  • abort: abort the terraform plan or apply
• if all items are literals, the rightmost item is used
• if the remaining items are lists, the list_merge_strategy specified (which defaults to "replace") is applied:
  • replace: the list of the right most argument applies
  • concat: the sequence of lists is concatenated in the same order as arguments
  • merge: each list is appended with null so they all have the same length, then the deepmerge() is applied to each slice through the lists
• if the remaining items are maps or objects (the most common case), if more than one defines the same key or attribute, the deepmerge() is applied to the sequence of associated values across all the items

This is backwards compatible with all versions of terraform, and should cover 99% of use cases.

Examples:

deepmerge([a, b, c, d]):

• if a, b, c, d are all literals (string, bool, number), then d is return
• if all items are lists, then they are all merged according to list_merge_strategy
• if a and b are lists and c and d are literals, then c and d are discarded and a and b are merged according to list_merge_strategy
• if a and b are maps or objects then c and d are discarded and for each key in a, b, deepmerge() is called on the corresponding list of associated values, and objects that don't have the key are not included. So if a has key1 and key2, and b has key1 and key3, then the set of keys of the result is (key1, key2, key3) and there will be 3 calls to deepmerge(): deepmerge([a.key1, b.key1]), deepmerge([a.key2]), and deepmerge([b.key3]) (the merge-config is passed to these but left out here for simplicity)
• if all objects are maps or objects then the previous rule is applied to all items instead of just a and b

References

• Generic deepmerge() function #31815
• Many threads of discussion about this on community forum and github issues related to merge(). Also, the table in Support recursive merging of maps - deepmerge function #24987 (comment).

[crw]

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!