In the pursuit of efficient information retrieval, speed matters as much as accuracy. The paper ‘Shallow Cross-Encoders for Low-Latency Retrieval’ explores the trade-off between the two by employing shallow transformer models that score high in both effectiveness and speed (Read more):

• The main discovery is that shallow transformer models with fewer layers are able to process more documents within the same time budget while constrained by practical low-latency settings.
• Researchers observed that training these shallow transformers with the generalized Binary Cross-Entropy training scheme can further improve their performance.
• In a comparison using the TREC Deep Learning passage ranking query sets, a shallow cross-encoder based on TinyBERT trained with this approach outperformed a full-scale model in terms of NDCG@10 within a 25ms per query latency limit, achieving a +51% gain.
• Importantly, the study finds that shallow Cross-Encoders can run effectively on CPUs as well, demonstrating their practical applicability without specialized hardware acceleration.

The findings from this research offer new insights into achieving both practical and effective text retrieval. With low-latency and budget-friendly characteristics, these shallow transformer models could very well shape the future of search technologies and other applications where quick response times are crucial.