Gecko emerges as a potent and versatile text embedding model detailed in this publication. The model’s strong retrieval capabilities are achieved through a novel two-step distillation process. Initially, synthetic paired data are created using an LLM. This data is then refined by retrieval and relabeling using the same LLM, enhancing data quality. The main points discussed are:

• The compactness of Gecko which surpasses existing models of larger size and embedding dimensions.
• Gecko’s performance on the Massive Text Embedding Benchmark (MTEB).
• The unique approach of using an LLM to generate and refine synthetic data for distillation purposes.
• Its capability to offer powerful text embeddings while maintaining model efficiency.

Gecko’s development is pivotal in the information retrieval domain, offering enhanced performance with reduced computational overhead. The process could be applied to other areas requiring efficient, high-quality embeddings, such as search engines and recommendation systems. Future exploration may utilize Gecko in varied domains to capitalize on its efficient knowledge distillation. Explore the details.