google_monitoring_dashboard: resource still has perma diffs

[IchordeDionysos]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

I hope this helps reproduce the issue and I hope we don't need to wait until the next major version again to get this fixed 😅

This is a long-standing annoyance, polluting the changes when applying Terraform, leading to important differences needing to be noticed more easily.

Terraform Version

v1.6.0

Affected Resource(s)

• google_monitoring_dashboard

Terraform Configuration Files

Full Example reproducing the issue:

terraform {
  required_version = ">= 1.3.0, < 2.0.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

variable "project" {
  type = string
}

variable "region" {
  type = string
}

provider "google" {
  project = var.project
  region  = var.region
}

resource "google_monitoring_dashboard" "dashboard" {

  dashboard_json = <<EOF
  {
  "dashboardFilters": [],
  "displayName": "Perma diff dashboard",
  "labels": {},
  "mosaicLayout": {
    "columns": 12,
    "tiles": [
      {
        "height": 4,
        "widget": {
          "title": "SLO Compliance: 99.9% - Good/Total Ratio - Rolling 28 days",
          "xyChart": {
            "chartOptions": {
              "mode": "COLOR"
            },
            "dataSets": [
              {
                "breakdowns": [],
                "dimensions": [],
                "measures": [],
                "plotType": "LINE",
                "targetAxis": "Y1",
                "timeSeriesQuery": {
                  "timeSeriesFilter": {
                    "aggregation": {
                      "perSeriesAligner": "ALIGN_NEXT_OLDER"
                    },
                    "filter": "select_slo_compliance(\"projects/${var.project}/services/run-service/serviceLevelObjectives/run-service-rolling-availability-slo\")"
                  },
                  "unitOverride": "10^2.%"
                }
              }
            ],
            "thresholds": [
              {
                "label": "",
                "targetAxis": "Y1",
                "value": 0.999
              }
            ]
          }
        },
        "width": 12
      },
      {
        "height": 4,
        "widget": {
          "title": "SLO Compliance: 95% - 90th percentile latency under 1000ms - Rolling 28 days",
          "xyChart": {
            "chartOptions": {
              "mode": "COLOR"
            },
            "dataSets": [
              {
                "breakdowns": [],
                "dimensions": [],
                "measures": [],
                "plotType": "LINE",
                "targetAxis": "Y1",
                "timeSeriesQuery": {
                  "timeSeriesFilter": {
                    "aggregation": {
                      "perSeriesAligner": "ALIGN_NEXT_OLDER"
                    },
                    "filter": "select_slo_compliance(\"projects/${var.project}/services/run-service/serviceLevelObjectives/run-service-latency-slo\")"
                  },
                  "unitOverride": "10^2.%"
                }
              }
            ],
            "thresholds": [
              {
                "label": "",
                "targetAxis": "Y1",
                "value": 0.95
              }
            ]
          }
        },
        "width": 12,
        "yPos": 4
      },
      {
        "height": 4,
        "widget": {
          "title": "Requests per response code class",
          "xyChart": {
            "chartOptions": {
              "mode": "COLOR"
            },
            "dataSets": [
              {
                "breakdowns": [],
                "dimensions": [],
                "measures": [],
                "minAlignmentPeriod": "60s",
                "plotType": "LINE",
                "targetAxis": "Y1",
                "timeSeriesQuery": {
                  "timeSeriesFilter": {
                    "aggregation": {
                      "alignmentPeriod": "60s",
                      "perSeriesAligner": "ALIGN_RATE"
                    },
                    "filter": "metric.type=\"loadbalancing.googleapis.com/https/request_count\" resource.type=\"https_lb_rule\" (resource.label.backend_target_name=\"api-lb-backend-run-service\")",
                    "secondaryAggregation": {
                      "alignmentPeriod": "60s",
                      "crossSeriesReducer": "REDUCE_MEAN",
                      "groupByFields": [
                        "metric.label.\"response_code_class\""
                      ],
                      "perSeriesAligner": "ALIGN_MEAN"
                    }
                  }
                }
              }
            ],
            "thresholds": [],
            "timeshiftDuration": "0s",
            "yAxis": {
              "label": "y1Axis",
              "scale": "LINEAR"
            }
          }
        },
        "width": 12,
        "yPos": 8
      },
      {
        "height": 4,
        "widget": {
          "title": "Latency 50th & 95th percentile",
          "xyChart": {
            "chartOptions": {
              "mode": "COLOR"
            },
            "dataSets": [
              {
                "breakdowns": [],
                "dimensions": [],
                "measures": [],
                "minAlignmentPeriod": "60s",
                "plotType": "LINE",
                "targetAxis": "Y1",
                "timeSeriesQuery": {
                  "timeSeriesFilter": {
                    "aggregation": {
                      "alignmentPeriod": "60s",
                      "crossSeriesReducer": "REDUCE_PERCENTILE_50",
                      "groupByFields": [],
                      "perSeriesAligner": "ALIGN_DELTA"
                    },
                    "filter": "metric.type=\"loadbalancing.googleapis.com/https/total_latencies\" resource.type=\"https_lb_rule\" (resource.label.backend_target_name=\"api-lb-backend-run-service\")"
                  }
                }
              },
              {
                "breakdowns": [],
                "dimensions": [],
                "measures": [],
                "minAlignmentPeriod": "60s",
                "plotType": "LINE",
                "targetAxis": "Y1",
                "timeSeriesQuery": {
                  "timeSeriesFilter": {
                    "aggregation": {
                      "alignmentPeriod": "60s",
                      "crossSeriesReducer": "REDUCE_PERCENTILE_95",
                      "groupByFields": [],
                      "perSeriesAligner": "ALIGN_DELTA"
                    },
                    "filter": "metric.type=\"loadbalancing.googleapis.com/https/total_latencies\" resource.type=\"https_lb_rule\" (resource.label.backend_target_name=\"api-lb-backend-run-service\")"
                  }
                }
              }
            ],
            "thresholds": [],
            "timeshiftDuration": "0s",
            "yAxis": {
              "label": "y1Axis",
              "scale": "LINEAR"
            }
          }
        },
        "width": 12,
        "yPos": 12
      },
      {
        "height": 4,
        "widget": {
          "title": "Cache Hit Ratio",
          "xyChart": {
            "chartOptions": {
              "mode": "COLOR"
            },
            "dataSets": [
              {
                "breakdowns": [],
                "dimensions": [],
                "measures": [],
                "plotType": "LINE",
                "targetAxis": "Y1",
                "timeSeriesQuery": {
                  "timeSeriesFilterRatio": {
                    "denominator": {
                      "aggregation": {
                        "alignmentPeriod": "60s",
                        "crossSeriesReducer": "REDUCE_SUM",
                        "groupByFields": [
                          "resource.label.\"backend_target_name\""
                        ],
                        "perSeriesAligner": "ALIGN_RATE"
                      },
                      "filter": "metric.type=\"loadbalancing.googleapis.com/https/backend_request_count\" resource.type=\"https_lb_rule\" (resource.label.backend_target_name=\"api-lb-backend-run-service\")"
                    },
                    "numerator": {
                      "aggregation": {
                        "alignmentPeriod": "60s",
                        "crossSeriesReducer": "REDUCE_SUM",
                        "groupByFields": [],
                        "perSeriesAligner": "ALIGN_RATE"
                      },
                      "filter": "metric.type=\"loadbalancing.googleapis.com/https/backend_request_count\" resource.type=\"https_lb_rule\" metric.label.\"cache_result\"=\"HIT\" (resource.label.backend_target_name=\"api-lb-backend-run-service\")"
                    }
                  }
                }
              }
            ],
            "thresholds": [],
            "timeshiftDuration": "0s",
            "yAxis": {
              "label": "y1Axis",
              "scale": "LINEAR"
            }
          }
        },
        "width": 12,
        "yPos": 16
      },
      {
        "height": 7,
        "widget": {
          "logsPanel": {
            "filter": "resource.type=\"cloud_function\"\nlog_name=\"projects/${var.project}/logs/cloudaudit.googleapis.com%2Factivity\"\nprotoPayload.request.function.labels.domain=\"run-service\"",
            "resourceNames": [
              "projects/${var.project}"
            ]
          },
          "title": "API deployments"
        },
        "width": 12,
        "yPos": 20
      }
    ]
  }
}
EOF
}

Minimal example reproducing the issue

terraform {
  required_version = ">= 1.3.0, < 2.0.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

variable "project" {
  type    = string
  default = "simpleclub-stage"
}

variable "region" {
  type    = string
  default = "europe-west1"
}

provider "google" {
  project = var.project
  region  = var.region
}

resource "google_monitoring_dashboard" "dashboard" {

  dashboard_json = <<EOF
  {
  "dashboardFilters": [],
  "displayName": "Perma diff dashboard",
  "labels": {},
  "mosaicLayout": {
    "columns": 12,
    "tiles": [
      {
        "height": 4,
        "widget": {
          "title": "SLO Compliance: 99.9% - Good/Total Ratio - Rolling 28 days",
          "xyChart": {
            "chartOptions": {
              "mode": "COLOR"
            },
            "dataSets": [
              {
                "breakdowns": [],
                "dimensions": [],
                "measures": [],
                "plotType": "LINE",
                "targetAxis": "Y1",
                "timeSeriesQuery": {
                  "timeSeriesFilter": {
                    "aggregation": {
                      "perSeriesAligner": "ALIGN_NEXT_OLDER"
                    },
                    "filter": "select_slo_compliance(\"projects/${var.project}/services/run-service/serviceLevelObjectives/run-service-rolling-availability-slo\")"
                  },
                  "unitOverride": "10^2.%"
                }
              }
            ],
            "thresholds": [
              {
                "label": "",
                "targetAxis": "Y1",
                "value": 0.999
              }
            ]
          }
        },
        "width": 12
      }
    ]
  }
}
EOF
}

Terraform output after re-planning (with perma diffs):

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_monitoring_dashboard.dashboard will be updated in-place
  ~ resource "google_monitoring_dashboard" "dashboard" {
      ~ dashboard_json = jsonencode(
          ~ {
              + dashboardFilters = []
              - etag             = "dashboard-etag"
              + labels           = {}
              ~ mosaicLayout     = {
                  ~ tiles   = [
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets     = [
                                      ~ {
                                          + breakdowns      = []
                                          + dimensions      = []
                                          + measures        = []
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                  ~ thresholds   = [
                                      ~ {
                                          + label      = ""
                                            # (2 unchanged attributes hidden)
                                        },
                                    ]
                                    # (1 unchanged attribute hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (2 unchanged attributes hidden)
                        },
                    ]
                    # (1 unchanged attribute hidden)
                }
              - name             = "projects/projectA/dashboards/dashboard-id"
                # (1 unchanged attribute hidden)
            }
        )
        id             = "projects/projectA/dashboards/dashboard-id"
        # (1 unchanged attribute hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

Debug Output

Expected Behavior

Re-planning shows "No changes".

Actual Behavior

Re-planning shows changes needed for Terraform.

Steps to Reproduce

1. terraform apply
2. terraform plan

Important Factoids

It seems like using the mosaicLayout or some specific fields leads to this issue, but I'm not sure if that's indeed the case.

References

• google_monitoring_dashboard: resource always need to be updated. #7242
• monitoring: Add tests for perma-diffs with mosaic layout GoogleCloudPlatform/magic-modules#9197

b/304483210

[IchordeDionysos]

@c2thorn here is the dedicated issue

[IchordeDionysos]

@c2thorn did you have time to look into this?

[philipp-wahala-db]

We face the same issue. Would be really helpful to get this resolved!

[devshree-cohere]

same issue, please fix

[pjohnston]

still an issue in 1.7.0 and provider v5.18.0

[alexandra-buica]

Hello! This issue still persists - tf version 1.7.0, provider v.5.20.0
Could someone please take a look over it? Many thanks in advance!

[tm185187]

The issue is still presists. Tested in Terraform Version 1.8.2 and 5.26.0 for google terraform provider.

[IchordeDionysos]

Given no progress have been made on this, we switched from mosaicLayout to row layout and were able to get rid of the issues.

Also removing fields that are defaults helped.

This shouldn't be necessary though...

[radsec]

++ this still exists

[danielsiwiec]

still an issue with mosaic layout. tf 1.9.5, provider 6.0.1

[danielsiwiec]

I found a solution to the problem. I used to always get the dashboard JSON from the GCP UI, using the 'Copy JSON' option. Turns out this gets a bunch of garbage that throws terraform off. Instead, we do this now and the diffs aren't showing anymore:

gcloud monitoring dashboards list --format=json | jq '.[] | select(.displayName == "YOU DASHBOARD NAME") | del(.name, .etag)'

[skyuchukov-bp]

I found a solution to the problem. I used to always get the dashboard JSON from the GCP UI, using the 'Copy JSON' option. Turns out this gets a bunch of garbage that throws terraform off. Instead, we do this now and the diffs aren't showing anymore:

gcloud monitoring dashboards list --format=json | jq '.[] | select(.displayName == "YOU DASHBOARD NAME") | del(.name, .etag)'

This fixed the problem for us too, thanks!

[cpl-alanb]

What also works:

• Download the dashboards over the GCP UI or using gcloud
• Run in bash: cat "my_dashboard.json" | jq 'del(..|nulls)' > my_dashboard_cleared.json
• Use the "cleared" version in your deployments

jq 'del(..|nulls)' removes all null and empty values which are causing the issues in the first place

[IchordeDionysos]

@cpl-alanb that's a bit difficult to achieve if you need to dynamically and conditionally construct the JSON

[laura-lindberg-sap]

Hello! Is there an official solution for this issue?

[IchordeDionysos]

@laura-lindberg-sap no there is none.
You may be lucky with using different layout types or otherwise change your dashboards, though it's quite unpredictable what works and what doesn't...