Mini toolsDefinition
A technique that gradually reduces the learning rate during training to fine-tune model parameters more precisely. This helps achieve better final model performance.
Detailed Explanation
Learning rate decay also known as learning rate scheduling systematically reduces the learning rate according to a predetermined schedule or based on validation performance. Common approaches include step decay exponential decay and cosine decay. This technique helps models converge to better optima by taking smaller steps as training progresses.
Use Cases
Deep neural network training Optimization problems Fine-tuning procedures Transfer learning
