Catastrophic Forgetting

Catastrophic forgetting is a phenomenon in machine learning and artificial intelligence where a neural network or model loses previously learned information when trained on new data or tasks. It occurs primarily in sequential learning scenarios, where the model's parameters are updated to accommodate new knowledge, causing it to 'forget' old patterns or skills. This is a significant challenge in continual or lifelong learning systems that aim to adapt over time without degrading performance on earlier tasks.

Also known as: Catastrophic Interference, Forgetting in Neural Networks, Sequential Learning Problem, CF, Catastrophic Forgetting Issue

Why learn Catastrophic Forgetting?

Developers should learn about catastrophic forgetting when working on AI systems that require incremental learning, such as robotics, autonomous vehicles, or personalized recommendation engines, to prevent performance drops on prior tasks. Understanding this concept is crucial for implementing techniques like regularization, rehearsal, or architectural changes to mitigate forgetting and build more robust, adaptable models. It's particularly relevant in fields like natural language processing or computer vision where models need to handle evolving data streams.

Learning Resources

• docs: Catastrophic Forgetting in Connectionist Networks
• docs: Overcoming Catastrophic Forgetting in Neural Networks
• video: Continual Learning with Deep Neural Networks
• tutorial: Catastrophic Forgetting Tutorial on Towards Data Science
• book: Deep Learning Book Chapter on Regularization