AI model distillation techniquescomputational costsmachine learning
Reducing Model Inference Costs 60% with Knowledge Distillation in PyTorch
9 min read · 1,737 wordsBy Orandi Felix
The softmax function with temperature is: σ(z_i) = e^(z_i / T) / Σ_j e^(z_j / T) When T=1, it’s the standard softmax. As T→∞, probabilities become uniform.
Note the `(temperature ** 2)` scaling factor. This ensures the gradient magnitude of the distillation loss stays consistent as temperature changes.
Cost calculation: Assuming $0.35/hr for a cloud GPU instance, student inference is 2.6x faster → 2.6x fewer instance-hours → 61.5% cost reduction.
Use this rule of thumb: start with `T` between 3–10 and `alpha` between 0.5–0.7. Plot the distillation loss vs. temperature (log scale) to find a sweet spot.
If "both_wrong" > 5%, your teacher may have blind spots. Consider ensemble distillation (average logits from multiple teachers).