neural networkscontinual learningdebuggingcatastrophic forgettingregularization techniquesoptimization methods
When Your Neural Networks Need Therapy: Debugging Catastrophic Forgetting Using State-of-the-Art Regularization Techniques
8 min read · 1,401 wordsBy Orandi Felix
Catastrophic forgetting isn’t memory loss—it’s gradient-based amnesia. The network hasn’t lost the ability to store information; the loss landscape has changed, and the optimizer follows the steepest descent.
The Fisher information is the second derivative of the loss—practically, it tells us which weights were *most important* for the old task. EWC anchors those weights, allowing others to adapt freely.
All runs used a 1-layer MLP (512 hidden units) on an RTX 4090 with 10k examples per task. Memory excludes replay buffers. Training time is wall-clock for 10 epochs per task.