Researchers recently introduced FoFo, a benchmark to evaluate large language models’ (LLMs) format-following capabilities. This benchmark presents a variety of real-world scenarios to test models’ adherence to domain-specific formats. The key findings are:

• Open-source LLMs like Llama 2 and WizardLM are trailing behind closed-source models such as GPT-4 and PALM2 in this capacity.
• The format-following proficiency of an LLM is not tied to its content generation quality.
• Performance varies across different domains, suggesting a need for domain-specific tuning.

FoFo marks an important step in selecting AI agents for specialized tasks. The study underlines the potential and necessity of developing LLMs with strong format-following skills.

Read more about FoFo on arXiv