Reducing LLM Inference Power Consumption by 40% with GPU Serverless Functions
Serverless ≠ FaaS. FaaS typically refers to CPU functions with a shared pool. Serverless GPU means **ephemeral, 1-instance-per-request** allocations—critical for preventing VRAM thrashing and CUDA context switches.
The 12 minutes at full power include **both token computation and model load time**. Modal’s cold-start for a GGUF model on a L4 is 1.7 seconds (± 0.1 s) measured with `time curl`.
Keep-warm costs: 1 instance × 85 W × 24 h = 2.04 kWh/day. Baseline idle cost: 40 W × 24 h = 9.6 kWh/day. Still a 79 % reduction versus always-on.
If you lack hardware meters, use `nvidia-smi dmon` and `psutil` to log GPU and CPU power draw. Correlate against wall power with a 15-point calibration curve. Remember: `nvidia-smi` reports *GPU board power* (≈ 85 % of system power) so scale readings by 1.18 to match Kill-A-Watt.