... | ... | @@ -92,4 +92,97 @@ $ nvidia-smi pmon |
|
|
```
|
|
|
*in this case, two different python processes are running; one on each GPU; only 2 over 4 GPU are used*
|
|
|
|
|
|
## Monitoring and Managing GPU Boost
|
|
|
|
|
|
The GPU Boost feature which NVIDIA has included with more recent GPUs allows the GPU clocks to vary depending upon load (achieving maximum performance so long as power and thermal headroom are available). However, the amount of available headroom will vary by application (and even by input file!) so users should keep their eyes on the status of the GPUs. A listing of available clock speeds can be shown for each GPU on Turing with V100:
|
|
|
|
|
|
```
|
|
|
$ nvidia-smi -q -d SUPPORTED_CLOCKS
|
|
|
==============NVSMI LOG==============
|
|
|
Timestamp : Mon Nov 23 18:48:39 2020
|
|
|
Driver Version : 450.51.06
|
|
|
CUDA Version : 11.0
|
|
|
|
|
|
Attached GPUs : 4
|
|
|
GPU 00000000:18:00.0
|
|
|
Supported Clocks
|
|
|
Memory : 877 MHz
|
|
|
Graphics : 1530 MHz
|
|
|
Graphics : 1522 MHz
|
|
|
Graphics : 1515 MHz
|
|
|
Graphics : 1507 MHz
|
|
|
[...180 additional clock speeds omitted...]
|
|
|
Graphics : 150 MHz
|
|
|
Graphics : 142 MHz
|
|
|
Graphics : 135 MHz
|
|
|
```
|
|
|
|
|
|
As shown, the Tesla V100 GPU supports 187 different clock speeds (from 135 MHz to 1530 MHz). However, only one memory clock speed is supported (877 MHz). Some GPUs support two different memory clock speeds (one high speed and one power-saving speed). Typically, such GPUs only support a single GPU clock speed when the memory is in the power-saving speed (which is the idle GPU state). On all recent Tesla and Quadro GPUs, GPU Boost automatically manages these speeds and runs the clocks as fast as possible (within the thermal/power limits and any limits set by the administrator).
|
|
|
|
|
|
To review the current GPU clock speed (here we display 1 GPU), default clock speed, and maximum possible clock speed, run:
|
|
|
|
|
|
```
|
|
|
$ nvidia-smi -q -d CLOCK
|
|
|
==============NVSMI LOG==============
|
|
|
Timestamp : Mon Nov 23 18:56:48 2020
|
|
|
Driver Version : 450.51.06
|
|
|
CUDA Version : 11.0
|
|
|
|
|
|
Attached GPUs : 4
|
|
|
GPU 00000000:18:00.0
|
|
|
Clocks
|
|
|
Graphics : 1290 MHz
|
|
|
SM : 1290 MHz
|
|
|
Memory : 877 MHz
|
|
|
Video : 1170 MHz
|
|
|
Applications Clocks
|
|
|
Graphics : 1290 MHz
|
|
|
Memory : 877 MHz
|
|
|
Default Applications Clocks
|
|
|
Graphics : 1290 MHz
|
|
|
Memory : 877 MHz
|
|
|
Max Clocks
|
|
|
Graphics : 1530 MHz
|
|
|
SM : 1530 MHz
|
|
|
Memory : 877 MHz
|
|
|
Video : 1372 MHz
|
|
|
Max Customer Boost Clocks
|
|
|
Graphics : 1530 MHz
|
|
|
SM Clock Samples
|
|
|
Duration : 0.01 sec
|
|
|
Number of Samples : 4
|
|
|
Max : 1290 MHz
|
|
|
Min : 135 MHz
|
|
|
Avg : 870 MHz
|
|
|
Memory Clock Samples
|
|
|
Duration : 0.01 sec
|
|
|
Number of Samples : 4
|
|
|
Max : 877 MHz
|
|
|
Min : 877 MHz
|
|
|
Avg : 877 MHz
|
|
|
Clock Policy
|
|
|
Auto Boost : N/A
|
|
|
Auto Boost Default : N/A
|
|
|
...
|
|
|
```
|
|
|
|
|
|
Ideally, you’d like all clocks to be running at the highest speed all the time. However, this will not be possible for all applications. To review the current state of each GPU and any reasons for clock slowdowns, use the `PERFORMANCE` flag:
|
|
|
|
|
|
```
|
|
|
$ nvidia-smi -q -d PERFORMANCE
|
|
|
|
|
|
Attached GPUs : 4
|
|
|
GPU 00000000:18:00.0
|
|
|
Performance State : P0
|
|
|
Clocks Throttle Reasons
|
|
|
Idle : Not Active
|
|
|
Applications Clocks Setting : Not Active
|
|
|
SW Power Cap : Not Active
|
|
|
HW Slowdown : Not Active
|
|
|
HW Thermal Slowdown : Not Active
|
|
|
HW Power Brake Slowdown : Not Active
|
|
|
Sync Boost : Not Active
|
|
|
SW Thermal Slowdown : Not Active
|
|
|
Display Clock Setting : Not Active
|
|
|
...
|
|
|
``` |