You have deployed an AI training job on a GPU cluster, but the training time has not decreased as expected after adding more GPUs. Upon further investigation, you observe that the GPU utilization is low, and the CPU utilization is very high. What is the most likely cause of this issue?
The data preprocessing being bottlenecked by the CPU is the most likely cause. High CPU utilization and low GPU utilization suggest the GPUs are idle, waiting for data, a common issue when preprocessing (e.g., data loading) is CPU-bound. NVIDIA recommends GPU-accelerated preprocessing (e.g., DALI) to mitigate this. Option A (model incompatibility) would show errors, not low utilization. Option B (connection issues) would disrupt communication, not CPU load. Option C (software version) is less likely without specific errors. NVIDIA's performance guides highlight preprocessing bottlenecks.
Loise
2 days agoMargurite
3 days agoTamala
6 days ago