Failed to initialize nvml unknown error inside docker

[HassanAsghar1]

Describe the bug
Nvidia-smi is not working after few hours inside docker(NVML error)

To Reproduce
docker : bladedisc/bladedisc:latest-runtime-torch1.13.1-cu116.
GPU : T4 (nvidia-drivers 515 in host machine)
OS : Linux -ubuntu 22.04
On the start of docker , nvidia-smi shows GPU information .
After few hours , nvidia-smi gives failed to initialize nvml unknown error which results in blocking the GPU usage inside the docker.

Expected behavior

nvidia-smi inside the docker should always displays GPU information