Pod stuck in ContainerCreating state with no logs or errors when running a job

[csvlrs-lak]

What's the issue?

When triggering a run for the below Dagster asset job, the Kubernetes pod remains stuck in the ContainerCreating state indefinitely.

What did you expect to happen?

The pod should transition from ContainerCreating to Running and execute the job successfully.

How to reproduce?

1. Define the job using the provided code sample.
2. Trigger a job run manually or via the schedule.(pvc:mypvc is available in the namespace )
3. Observe the pod state.

import pandas as pd
from dagster import asset, AssetIn, define_asset_job, ScheduleDefinition, DefaultScheduleStatus, AssetKey, AssetSelection, job
import time

@asset(group_name="check", key_prefix="check")
def Assest_1():
    return 0

check_job = define_asset_job(
    "check_job",
    selection=AssetSelection.keys(
        *[AssetKey(["check", node["id"]]) for node in [{'id': 'Assest_1'}]]
    ),
    tags={
        "dagster-k8s/config": {
            "container_config": {
                "volumeMounts": [
                    {
                        "name": "data-volume",
                        "mountPath": "/mnt/data"
                    }
                ]
            },
            "pod_spec_config": {
                "volumes": [
                    {
                        "name": "data-volume",
                        "persistentVolumeClaim": {
                            "claimName": "my-pvc"
                        }
                    }
                ],
                "affinity": {
                    "node_affinity": {
                        "required_during_scheduling_ignored_during_execution": {
                            "nodeSelectorTerms": [
                                {
                                    "matchExpressions": [
                                        {
                                            "key": "poolname",
                                            "operator": "In",
                                            "values": ["agentpool"]
                                        }
                                    ]
                                }
                            ]
                        }
                    }
                }
            },
        }
    }
)

check_schedule= ScheduleDefinition(
    job=check_job,
    cron_schedule="30 22 * * *",
    default_status=DefaultScheduleStatus.RUNNING
)

Dagster version

1.9.2

Deployment type

None

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
By submitting this issue, you agree to follow Dagster's Code of Conduct.