Mini toolsDefinition
A neural network mechanism that allows models to weigh the importance of different parts of the input sequence dynamically. This is a key component of transformer architectures.
Detailed Explanation
Self-attention computes relationships between all positions in a sequence by calculating attention scores based on queries keys and values derived from the input. Each position attends to all positions allowing the model to capture long-range dependencies and complex relationships in the data. Multiple attention heads allow the model to capture different types of relationships simultaneously.
Use Cases
Language models Image processing Speech recognition Sequential data analysis
