Verifying GPU Access with nvidia-smi

Why this matters

A pod can schedule onto a GPU node and still not actually see the GPU, because of a runtime misconfiguration, a missing capability, or a driver mismatch. The single fastest way to prove a container has real GPU access is to run nvidia-smi inside it — if it lists the device, you have a working stack end to end. Making this verification a reflex saves hours: instead of debugging why your training job is mysteriously slow, you run one command and instantly learn whether the container even sees the hardware. This is the 'is it plugged in' check for every GPU workload.

Demo

Exec into a GPU pod and run nvidia-smi. A correct setup prints the GPU model, driver and CUDA versions, memory, and any running processes. If it errors with 'command not found' or 'no devices', the runtime isn't injecting the GPU.

Try it yourself

1. Run the gpu-check pod and confirm nvidia-smi prints a populated GPU table.
2. Read off the driver and CUDA versions and note them — version skew is a common failure cause.
3. Remove the nvidia.com/gpu limit from the override and confirm nvidia-smi now reports 'No devices were found'.
4. Run nvidia-smi inside an actual workload pod (not a throwaway) and confirm your process appears in its process list.

References

• NVIDIA — Container Toolkit
• NVIDIA — k8s-device-plugin