Repository navigation
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request aims to address a critical hanging issue in the metrics endpoint by introducing a thread-safe lock for NVIDIA GPU detection. However, the current implementation is vulnerable to Denial of Service (DoS) due to incomplete locking. The threading.Lock used can lead to deadlocks in nested calls and should be replaced with threading.RLock. Furthermore, the fix is incomplete as other methods like is_supported() and get_topology() that interact with the NVML library are not protected, leaving them susceptible to the same concurrency issues.
| with self._detect_lock: | ||
| result = self._detect_impl() | ||
|
|
||
| return result |
There was a problem hiding this comment.
This block can be simplified by returning the result of _detect_impl() directly from within the with statement. This makes the code more concise and avoids an unnecessary intermediate variable.
| with self._detect_lock: | |
| result = self._detect_impl() | |
| return result | |
| with self._detect_lock: | |
| return self._detect_impl() |
|
6f14109 to
0618ac2
Compare
2 participants
Problem:
The metrics endpoint would hang after repeated calls, particularly under concurrent load.
Analysis revealed that pynvml.nvmlInit() was getting stuck,
causing the generate_latest() function to block indefinitely.
Root Cause:
The NVML library is not designed to handle frequent initialization/shutdown cycles.
The original detect() method invoked pynvml.nvmlInit() and pynvml.nvmlShutdown() on every request,
which destabilized the library and led to hanging behavior under load.
Solution:
This prevents concurrent initialization of the NVML library and resolves the hanging issue in the metrics endpoint.