Rigorous analysis in A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets introduces a mechanism to gauge the linguistic diversity of data sets used in multilingual NLP:

• The current diversity metrics often fail to consider the structural variety of languages included in a dataset.
• The authors offer a structured approach to compare datasets against a reference language sample, maximizing linguistic diversity.
• By representing languages as feature sets and utilizing an optimized Jaccard index, the method offers a quantitative diversity score that’s feature-informed.
• Various popular NLP datasets were evaluated, revealing notable absences like (poly)synthetic languages.

This methodology opens doors to more inclusive linguistic representation in NLP datasets, emphasizing the need for data sets that mirror the true expanse of global linguistic diversity.