Running the "Example Usage - Cloud Run Anthos" on google_cloud_run_service throws an error

[amerello-otto]

Terraform, Provider, Kubernetes versions

The example in the official documentation doesn't work

Terraform version:
Terraform v0.14.8

    provider registry.terraform.io/hashicorp/google v3.65.0
    provider registry.terraform.io/hashicorp/google-beta v3.70.0
    provider registry.terraform.io/hashicorp/kubernetes v2.3.1
    provider registry.terraform.io/hashicorp/kubernetes-alpha v0.4.1
Kubernetes version: 1.19.9-gke

Affected Resource(s)

kubernetes_manifest

Terraform Configuration Files

data "google_client_config" "provider" {}

data "google_container_cluster" "my_cluster" {
  name     = "cluster-1"
  location = "us-central1-c"
}

provider "kubernetes-alpha" {
  host  = "https://${data.google_container_cluster.my_cluster.endpoint}"
  token = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(
    data.google_container_cluster.my_cluster.master_auth.0.cluster_ca_certificate,
  )
}

resource "kubernetes_manifest" "test-configmap" {
  provider = kubernetes-alpha

  manifest = {
    "apiVersion" = "serving.knative.dev/v1"
    "kind"       = "Service"
    "metadata" = {
      "name"      = "helloworld-go"
      "namespace" = "default"
    }
    "spec" = {
      "template" = {
        "spec" = {
          "containers" = [
            {
              "image" = "us-docker.pkg.dev/cloudrun/container/hello"
            }
          ]
        }
      }
    }
  }
}

Debug Output

Panic Output

Steps to Reproduce

1. terraform apply

Expected Behavior

Resource should be created

Actual Behavior

Step #2 - "apply": kubernetes_manifest.test-configmap: Refreshing state...
Step #2 - "apply": kubernetes_manifest.test-configmap: Destroying...
Step #2 - "apply": kubernetes_manifest.test-configmap: Destruction complete after 1s
Step #2 - "apply": kubernetes_manifest.test-configmap: Creating...
Step #2 - "apply":
Step #2 - "apply": Error: Provider produced inconsistent result after apply
Step #2 - "apply":
Step #2 - "apply": When applying changes to kubernetes_manifest.test-configmap, provider
Step #2 - "apply": "registry.terraform.io/hashicorp/kubernetes-alpha" produced an unexpected new
Step #2 - "apply": value: .object: wrong final value type: incorrect object attributes.
Step #2 - "apply":
Step #2 - "apply": This is a bug in the provider, which should be reported in the provider's own
Step #2 - "apply": issue tracker.
Step #2 - "apply":

Important Factoids

References

hashicorp/terraform-provider-google#9320

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[alexsomesan]

Hi! I was able to successfully apply your example CR using provider v0.5.0 and minikube.
I think the error you are seeing is caused by GKE adding annotations to the resource. But that would only cause conflicts if user annotations would also be set on the manifest, which I'm not seeing here. Are you sure this is the exact configuration applied? If not, can you share the complete / exact configuration that caused this?

[amerello-otto]

Hi @alexsomesan, thank you for looking into this. This is most likely about the annotations added by GKE. The way I was able to work around the issue, was by adding the annotations from the created resource back into the terraform configuration:

resource "kubernetes_manifest" "cloudrun_on_gke" {
  provider = kubernetes-alpha

  manifest = {
    "apiVersion" = "serving.knative.dev/v1"
    "kind" = "Service"
    "metadata" = {
      "annotations" = {
        "serving.knative.dev/creator" = "${var.project_number}@cloudbuild.gserviceaccount.com"
        "serving.knative.dev/lastModifier" = "${var.project_number}@cloudbuild.gserviceaccount.com"
      }
      "name" = "helloworld-go"
      "namespace" = "default"
    }
    "spec" = {
      "template" = {
        "metadata" = {
          "creationTimestamp" = null
        }
        "spec" = {
          "containerConcurrency" = 0
          "containers" = [
            {
              "image" = "us-docker.pkg.dev/cloudrun/container/hello"
              "name" = "user-container"
              "readinessProbe" = {
                "successThreshold" = 1
                "tcpSocket" = {
                  "port" = 0
                }
              }
              "resources" = {}
            },
          ]
          "enableServiceLinks" = false
          "timeoutSeconds" = 300
        }
      }
      "traffic" = [
        {
          "latestRevision" = true
          "percent" = 100
        },
      ]
    }
  }
}

This way it doesn't throw an error anymore, but I find this unfortunate, because the configuration becomes unnecessarily verbose. I specially don't like having the serving.knative.dev/lastModifier and creationTimestamp annotations there. I did confirm by trial and error that they are necessary for the apply to return successfully.