Research delineated in ‘Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models’ investigates the superiority of the Adam optimizer over traditional gradient descent for training language transformers. The study attributes this to the heavy-tailed class imbalance prevalent in language modeling tasks, which leads to slower loss reduction for infrequent words when using gradient descent.

Core Findings:

• Language modeling tasks feature a heavy-tailed distribution of word frequencies.
• Adam and sign-based methods achieve better loss reduction across all classes.
• The study extends beyond language transformers to include vision CNNs and linear models.

Expert Commentary: This research profoundly explains the variation in optimizer performance on different machine learning tasks. It also provides practical insights for AI practitioners when designing and training models, particularly those dealing with classification in natural language processing.

• Discover More