Introduction

• Language Ranker is introduced to benchmark and rank different languages based on LLMs’ performance comparisons with the English baseline.

Findings

• Performance rankings of various LLM sizes remain consistent across different languages.
• A strong correlation is noted between the performance in different languages and the proportion of text corpus used during pre-training. This highlights the imbalance in text corpus distribution across languages and its impact on model effectiveness.

Significance

This research addresses a crucial gap by quantifying the performance of LLMs across less-resourced languages. It sheds light on the necessity to develop more balanced training datasets to enhance model performance globally.

Future Work

The study serves as a baseline for future research into improving LLM capabilities across a broad spectrum of languages, potentially guiding more equitable AI development practices.