Cannot Delete DynamoDB Table

[rubenhak]

What happened?

Using Crossplane+ArgoCD. I have successfully created DynamoDB Table. Then I tried to delete it. Process gets stuck with errors.
See the error in the logs:

• DEBUG events cannot run refresh: refresh failed: Missing required argument: The argument "enabled" is required, but no definition was found.
• Missing required argument: The argument "attribute_name" is required, but no definition was found.

Table Definition:

apiVersion: dynamodb.aws.upbound.io/v1beta1
kind: Table
metadata:
  annotations:
    meta.upbound.io/example-id: dynamodb/v1beta1/table
  name: dynamodb-games
spec:
  forProvider:
    attribute:
      - name: UserId
        type: "S"
      - name: GameTitle
        type: "S"
      - name: TopScore
        type: "N"
    billingMode: PROVISIONED
    globalSecondaryIndex:
      - hashKey: GameTitle
        name: GameTitleIndex
        nonKeyAttributes:
          - UserId
        projectionType: INCLUDE
        rangeKey: TopScore
        readCapacity: 1
        writeCapacity: 1
    hashKey: UserId
    rangeKey: GameTitle
    readCapacity: 1
    region: us-east-1
    tags:
      Environment: production
      Name: games
    writeCapacity: 1

Logs for 'provider-aws-dynamodb':

2023-08-18T18:28:53Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/dynamodb.aws.upbound.io/v1beta1, kind=table", "request": {"name":"games-table"}}
2023-08-18T18:28:53Z    DEBUG   provider-aws    Reusing the provider runner     {"handle": "b008969b4bb6ace78cd0a5a67f0dc1189926631c853ad9512cf31a42bcb73977", "ttl": 100, "ttlMargin": 0.1, "invocationCount": 6}
2023-08-18T18:28:53Z    DEBUG   provider-aws    Shared gRPC server is running...        {"handle": "b008969b4bb6ace78cd0a5a67f0dc1189926631c853ad9512cf31a42bcb73977", "ttl": 100, "ttlMargin": 0.1, "nativeProviderPath": "/terraform/provider-mirror/registry.terraform.io/hashicorp/aws/4.67.0/linux_amd64/terraform-provider-aws_v4.67.0_x5", "nativeProviderArgs": [], "reattachConfig": "{\"registry.terraform.io/hashicorp/aws\":{\"Protocol\":\"grpc\",\"ProtocolVersion\":5,\"Pid\":1,\"Test\": true,\"Addr\":{\"Network\": \"unix\",\"String\": \"/tmp/plugin3587366727\"}}}"}
2023-08-18T18:28:53Z    DEBUG   provider-aws    Running terraform       {"workspace": "/tmp/c6ca62e5-f18a-4c3a-bfe6-523b4585df92", "args": ["apply", "-refresh-only", "-auto-approve", "-input=false", "-lock=false", "-json"]}
2023-08-18T18:28:53Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/dynamodb.aws.upbound.io/v1beta1, kind=table", "request": {"name":"dynamodb-campaigns"}}
2023-08-18T18:28:53Z    DEBUG   provider-aws    Reusing the provider runner     {"handle": "b008969b4bb6ace78cd0a5a67f0dc1189926631c853ad9512cf31a42bcb73977", "ttl": 100, "ttlMargin": 0.1, "invocationCount": 7}
2023-08-18T18:28:53Z    DEBUG   provider-aws    Shared gRPC server is running...        {"handle": "b008969b4bb6ace78cd0a5a67f0dc1189926631c853ad9512cf31a42bcb73977", "ttl": 100, "ttlMargin": 0.1, "nativeProviderPath": "/terraform/provider-mirror/registry.terraform.io/hashicorp/aws/4.67.0/linux_amd64/terraform-provider-aws_v4.67.0_x5", "nativeProviderArgs": [], "reattachConfig": "{\"registry.terraform.io/hashicorp/aws\":{\"Protocol\":\"grpc\",\"ProtocolVersion\":5,\"Pid\":1,\"Test\": true,\"Addr\":{\"Network\": \"unix\",\"String\": \"/tmp/plugin3587366727\"}}}"}
2023-08-18T18:28:53Z    DEBUG   provider-aws    Running terraform       {"workspace": "/tmp/3f60614d-fe42-4b6b-a1f0-c5a60a940ec9", "args": ["apply", "-refresh-only", "-auto-approve", "-input=false", "-lock=false", "-json"]}
2023-08-18T18:28:54Z    DEBUG   provider-aws    refresh ended   {"workspace": "/tmp/c6ca62e5-f18a-4c3a-bfe6-523b4585df92", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.2.1\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-08-18T18:28:53.174163Z\",\"terraform\":\"1.2.1\",\"type\":\"version\",\"ui\":\"1.0\"}\n{\"@level\":\"error\",\"@message\":\"Error: Missing required argument\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-08-18T18:28:54.129149Z\",\"diagnostic\":{\"severity\":\"error\",\"summary\":\"Missing required argument\",\"detail\":\"The argument \\\"enabled\\\" is required, but no definition was found.\",\"range\":{\"filename\":\"main.tf.json\",\"start\":{\"line\":1,\"column\":1401,\"byte\":1400},\"end\":{\"line\":1,\"column\":1402,\"byte\":1401}},\"snippet\":{\"context\":\"resource.aws_dynamodb_table.games-table.point_in_time_recovery[0]\",\"code\":\"{\\\"provider\\\":{\\\"aws\\\":{\\\"access_key\\\":\\\"REDACTED\\\",\\\"region\\\":\\\"REDACTED\\\",\\\"secret_key\\\":\\\"REDACTED\\\",\\\"token\\\":\\\"REDACTED\\\"}},\\\"resource\\\":{\\\"aws_dynamodb_table\\\":{\\\"games-table\\\":{\\\"attribute\\\":[{\\\"name\\\":\\\"UserId\\\",\\\"type\\\":\\\"S\\\"},{\\\"name\\\":\\\"GameTitle\\\",\\\"type\\\":\\\"S\\\"},{\\\"name\\\":\\\"TopScore\\\",\\\"type\\\":\\\"N\\\"}],\\\"billing_mode\\\":\\\"PROVISIONED\\\",\\\"global_secondary_index\\\":[{\\\"hash_key\\\":\\\"GameTitle\\\",\\\"name\\\":\\\"GameTitleIndex\\\",\\\"non_key_attributes\\\":[\\\"UserId\\\"],\\\"projection_type\\\":\\\"INCLUDE\\\",\\\"range_key\\\":\\\"TopScore\\\",\\\"read_capacity\\\":1,\\\"write_capacity\\\":1}],\\\"hash_key\\\":\\\"UserId\\\",\\\"lifecycle\\\":{\\\"prevent_destroy\\\":false},\\\"name\\\":\\\"games-table\\\",\\\"point_in_time_recovery\\\":[{}],\\\"range_key\\\":\\\"GameTitle\\\",\\\"read_capacity\\\":1,\\\"table_class\\\":\\\"STANDARD\\\",\\\"tags\\\":{\\\"Environment\\\":\\\"production\\\",\\\"Name\\\":\\\"games-table\\\",\\\"crossplane-kind\\\":\\\"table.dynamodb.aws.upbound.io\\\",\\\"crossplane-name\\\":\\\"games-table\\\",\\\"crossplane-providerconfig\\\":\\\"default\\\"},\\\"ttl\\\":[{}],\\\"write_capacity\\\":1}}},\\\"terraform\\\":{\\\"required_providers\\\":{\\\"aws\\\":{\\\"source\\\":\\\"hashicorp/aws\\\",\\\"version\\\":\\\"4.67.0\\\"}}}}\",\"start_line\":1,\"highlight_start_offset\":1400,\"highlight_end_offset\":1401,\"values\":[]}},\"type\":\"diagnostic\"}\n{\"@level\":\"error\",\"@message\":\"Error: Missing required argument\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-08-18T18:28:54.129609Z\",\"diagnostic\":{\"severity\":\"error\",\"summary\":\"Missing required argument\",\"detail\":\"The argument \\\"attribute_name\\\" is required, but no definition was found.\",\"range\":{\"filename\":\"main.tf.json\",\"start\":{\"line\":1,\"column\":1660,\"byte\":1659},\"end\":{\"line\":1,\"column\":1661,\"byte\":1660}},\"snippet\":{\"context\":\"resource.aws_dynamodb_table.games-table.ttl[0]\",\"code\":\"{\\\"provider\\\":{\\\"aws\\\":{\\\"access_key\\\":\\\"REDACTED\\\",\\\"region\\\":\\\"REDACTED\\\",\\\"secret_key\\\":\\\"REDACTED\\\",\\\"token\\\":\\\"REDACTED\\\"}},\\\"resource\\\":{\\\"aws_dynamodb_table\\\":{\\\"games-table\\\":{\\\"attribute\\\":[{\\\"name\\\":\\\"UserId\\\",\\\"type\\\":\\\"S\\\"},{\\\"name\\\":\\\"GameTitle\\\",\\\"type\\\":\\\"S\\\"},{\\\"name\\\":\\\"TopScore\\\",\\\"type\\\":\\\"N\\\"}],\\\"billing_mode\\\":\\\"PROVISIONED\\\",\\\"global_secondary_index\\\":[{\\\"hash_key\\\":\\\"GameTitle\\\",\\\"name\\\":\\\"GameTitleIndex\\\",\\\"non_key_attributes\\\":[\\\"UserId\\\"],\\\"projection_type\\\":\\\"INCLUDE\\\",\\\"range_key\\\":\\\"TopScore\\\",\\\"read_capacity\\\":1,\\\"write_capacity\\\":1}],\\\"hash_key\\\":\\\"UserId\\\",\\\"lifecycle\\\":{\\\"prevent_destroy\\\":false},\\\"name\\\":\\\"games-table\\\",\\\"point_in_time_recovery\\\":[{}],\\\"range_key\\\":\\\"GameTitle\\\",\\\"read_capacity\\\":1,\\\"table_class\\\":\\\"STANDARD\\\",\\\"tags\\\":{\\\"Environment\\\":\\\"production\\\",\\\"Name\\\":\\\"games-table\\\",\\\"crossplane-kind\\\":\\\"table.dynamodb.aws.upbound.io\\\",\\\"crossplane-name\\\":\\\"games-table\\\",\\\"crossplane-providerconfig\\\":\\\"default\\\"},\\\"ttl\\\":[{}],\\\"write_capacity\\\":1}}},\\\"terraform\\\":{\\\"required_providers\\\":{\\\"aws\\\":{\\\"source\\\":\\\"hashicorp/aws\\\",\\\"version\\\":\\\"4.67.0\\\"}}}}\",\"start_line\":1,\"highlight_start_offset\":1659,\"highlight_end_offset\":1660,\"values\":[]}},\"type\":\"diagnostic\"}\n"}
2023-08-18T18:28:54Z    DEBUG   provider-aws    Cannot observe external resource        {"controller": "managed/dynamodb.aws.upbound.io/v1beta1, kind=table", "request": {"name":"games-table"}, "uid": "c6ca62e5-f18a-4c3a-bfe6-523b4585df92", "version": "289999", "external-name": "games-table", "error": "cannot run refresh: refresh failed: Missing required argument: The argument \"enabled\" is required, but no definition was found.\nMissing required argument: The argument \"attribute_name\" is required, but no definition was found.", "errorVerbose": "refresh failed: Missing required argument: The argument \"enabled\" is required, but no definition was found.\nMissing required argument: The argument \"attribute_name\" is required, but no definition was found.\ncannot run refresh\ngithub.com/upbound/upjet/pkg/controller.(*external).Observe\n\tgithub.com/upbound/[email protected]/pkg/controller/external.go:215\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:858\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226\nruntime.goexit\n\truntime/asm_amd64.s:1594"}
2023-08-18T18:28:54Z    DEBUG   events  cannot run refresh: refresh failed: Missing required argument: The argument "enabled" is required, but no definition was found.
Missing required argument: The argument "attribute_name" is required, but no definition was found.      {"type": "Warning", "object": {"kind":"Table","name":"games-table","uid":"c6ca62e5-f18a-4c3a-bfe6-523b4585df92","apiVersion":"dynamodb.aws.upbound.io/v1beta1","resourceVersion":"289999"}, "reason": "CannotObserveExternalResource"}
2023-08-18T18:28:54Z    DEBUG   provider-aws    refresh ended   {"workspace": "/tmp/3f60614d-fe42-4b6b-a1f0-c5a60a940ec9", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.2.1\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-08-18T18:28:53.521654Z\",\"terraform\":\"1.2.1\",\"type\":\"version\",\"ui\":\"1.0\"}\n{\"@level\":\"error\",\"@message\":\"Error: Missing required argument\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-08-18T18:28:54.449160Z\",\"diagnostic\":{\"severity\":\"error\",\"summary\":\"Missing required argument\",\"detail\":\"The argument \\\"enabled\\\" is required, but no definition was found.\",\"range\":{\"filename\":\"main.tf.json\",\"start\":{\"line\":1,\"column\":1411,\"byte\":1410},\"end\":{\"line\":1,\"column\":1412,\"byte\":1411}},\"snippet\":{\"context\":\"resource.aws_dynamodb_table.dynamodb-campaigns.point_in_time_recovery[0]\",\"code\":\"{\\\"provider\\\":{\\\"aws\\\":{\\\"access_key\\\":\\\"REDACTED\\\",\\\"region\\\":\\\"REDACTED\\\",\\\"secret_key\\\":\\\"REDACTED\\\",\\\"token\\\":\\\"REDACTED\\\"}},\\\"resource\\\":{\\\"aws_dynamodb_table\\\":{\\\"dynamodb-campaigns\\\":{\\\"attribute\\\":[{\\\"name\\\":\\\"UserId\\\",\\\"type\\\":\\\"S\\\"},{\\\"name\\\":\\\"GameTitle\\\",\\\"type\\\":\\\"S\\\"},{\\\"name\\\":\\\"TopScore\\\",\\\"type\\\":\\\"N\\\"}],\\\"billing_mode\\\":\\\"PROVISIONED\\\",\\\"global_secondary_index\\\":[{\\\"hash_key\\\":\\\"GameTitle\\\",\\\"name\\\":\\\"GameTitleIndex\\\",\\\"non_key_attributes\\\":[\\\"UserId\\\"],\\\"projection_type\\\":\\\"INCLUDE\\\",\\\"range_key\\\":\\\"TopScore\\\",\\\"read_capacity\\\":1,\\\"write_capacity\\\":1}],\\\"hash_key\\\":\\\"UserId\\\",\\\"lifecycle\\\":{\\\"prevent_destroy\\\":false},\\\"name\\\":\\\"dynamodb-campaigns\\\",\\\"point_in_time_recovery\\\":[{}],\\\"range_key\\\":\\\"GameTitle\\\",\\\"read_capacity\\\":1,\\\"table_class\\\":\\\"STANDARD\\\",\\\"tags\\\":{\\\"Environment\\\":\\\"production\\\",\\\"Name\\\":\\\"campaigns\\\",\\\"crossplane-kind\\\":\\\"table.dynamodb.aws.upbound.io\\\",\\\"crossplane-name\\\":\\\"dynamodb-campaigns\\\",\\\"crossplane-providerconfig\\\":\\\"default\\\"},\\\"ttl\\\":[{}],\\\"write_capacity\\\":1}}},\\\"terraform\\\":{\\\"required_providers\\\":{\\\"aws\\\":{\\\"source\\\":\\\"hashicorp/aws\\\",\\\"version\\\":\\\"4.67.0\\\"}}}}\",\"start_line\":1,\"highlight_start_offset\":1410,\"highlight_end_offset\":1411,\"values\":[]}},\"type\":\"diagnostic\"}\n{\"@level\":\"error\",\"@message\":\"Error: Missing required argument\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-08-18T18:28:54.449608Z\",\"diagnostic\":{\"severity\":\"error\",\"summary\":\"Missing required argument\",\"detail\":\"The argument \\\"attribute_name\\\" is required, but no definition was found.\",\"range\":{\"filename\":\"main.tf.json\",\"start\":{\"line\":1,\"column\":1671,\"byte\":1670},\"end\":{\"line\":1,\"column\":1672,\"byte\":1671}},\"snippet\":{\"context\":\"resource.aws_dynamodb_table.dynamodb-campaigns.ttl[0]\",\"code\":\"{\\\"provider\\\":{\\\"aws\\\":{\\\"access_key\\\":\\\"REDACTED\\\",\\\"region\\\":\\\"REDACTED\\\",\\\"secret_key\\\":\\\"REDACTED\\\",\\\"token\\\":\\\"REDACTED\\\"}},\\\"resource\\\":{\\\"aws_dynamodb_table\\\":{\\\"dynamodb-campaigns\\\":{\\\"attribute\\\":[{\\\"name\\\":\\\"UserId\\\",\\\"type\\\":\\\"S\\\"},{\\\"name\\\":\\\"GameTitle\\\",\\\"type\\\":\\\"S\\\"},{\\\"name\\\":\\\"TopScore\\\",\\\"type\\\":\\\"N\\\"}],\\\"billing_mode\\\":\\\"PROVISIONED\\\",\\\"global_secondary_index\\\":[{\\\"hash_key\\\":\\\"GameTitle\\\",\\\"name\\\":\\\"GameTitleIndex\\\",\\\"non_key_attributes\\\":[\\\"UserId\\\"],\\\"projection_type\\\":\\\"INCLUDE\\\",\\\"range_key\\\":\\\"TopScore\\\",\\\"read_capacity\\\":1,\\\"write_capacity\\\":1}],\\\"hash_key\\\":\\\"UserId\\\",\\\"lifecycle\\\":{\\\"prevent_destroy\\\":false},\\\"name\\\":\\\"dynamodb-campaigns\\\",\\\"point_in_time_recovery\\\":[{}],\\\"range_key\\\":\\\"GameTitle\\\",\\\"read_capacity\\\":1,\\\"table_class\\\":\\\"STANDARD\\\",\\\"tags\\\":{\\\"Environment\\\":\\\"production\\\",\\\"Name\\\":\\\"campaigns\\\",\\\"crossplane-kind\\\":\\\"table.dynamodb.aws.upbound.io\\\",\\\"crossplane-name\\\":\\\"dynamodb-campaigns\\\",\\\"crossplane-providerconfig\\\":\\\"default\\\"},\\\"ttl\\\":[{}],\\\"write_capacity\\\":1}}},\\\"terraform\\\":{\\\"required_providers\\\":{\\\"aws\\\":{\\\"source\\\":\\\"hashicorp/aws\\\",\\\"version\\\":\\\"4.67.0\\\"}}}}\",\"start_line\":1,\"highlight_start_offset\":1670,\"highlight_end_offset\":1671,\"values\":[]}},\"type\":\"diagnostic\"}\n"}
2023-08-18T18:28:54Z    DEBUG   provider-aws    Cannot observe external resource        {"controller": "managed/dynamodb.aws.upbound.io/v1beta1, kind=table", "request": {"name":"dynamodb-campaigns"}, "uid": "3f60614d-fe42-4b6b-a1f0-c5a60a940ec9", "version": "290003", "external-name": "dynamodb-campaigns", "error": "cannot run refresh: refresh failed: Missing required argument: The argument \"enabled\" is required, but no definition was found.\nMissing required argument: The argument \"attribute_name\" is required, but no definition was found.", "errorVerbose": "refresh failed: Missing required argument: The argument \"enabled\" is required, but no definition was found.\nMissing required argument: The argument \"attribute_name\" is required, but no definition was found.\ncannot run refresh\ngithub.com/upbound/upjet/pkg/controller.(*external).Observe\n\tgithub.com/upbound/[email protected]/pkg/controller/external.go:215\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:858\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226\nruntime.goexit\n\truntime/asm_amd64.s:1594"}
2023-08-18T18:28:54Z    DEBUG   events  cannot run refresh: refresh failed: Missing required argument: The argument "enabled" is required, but no definition was found.
Missing required argument: The argument "attribute_name" is required, but no definition was found.      {"type": "Warning", "object": {"kind":"Table","name":"dynamodb-campaigns","uid":"3f60614d-fe42-4b6b-a1f0-c5a60a940ec9","apiVersion":"dynamodb.aws.upbound.io/v1beta1","resourceVersion":"290003"}, "reason": "CannotObserveExternalResource"}

How can we reproduce it?

• Create DynamoDB Table resource
• Wait for Sync
• The DynamoDB Table is create successfully in AWS
• Delete DynamoDB Table resource
• Wait for Sync. See the error.

What environment did it happen in?

• Crossplane Version: v1.13.2
• Provider Version: v0.38.0
• Kubernetes Version: v1.27.4
• Kubernetes Distribution: k0s

[turkenf]

Hi @rubenhak,

Thank you for raising this issue. Fixed this issue and cannot reproduce in v0.39.0(related PR). Can you please update the provider version and try again?

[rubenhak]

@turkenf, same behavior with v.0.39.

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

This issue is being closed since there has been no activity for 14 days since marking it as stale. If you still need help, feel free to comment or reopen the issue!