The research paper GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations establishes GTBench, a set of language-driven game-theoretic tasks evaluating the strategic reasoning of LLMs. It offers insights into LLM behaviors across different types of games, highlighting:

• LLMs perform poorly in complete and deterministic games but show competencies in probabilistic scenarios.
• Open-source models such as CodeLlama-34b-Instruct are outperformed by commercial alternatives like GPT-4 in complex tasks.
• Code-pretraining is beneficial for strategic reasoning, while advanced reasoning methods like Chain-of-Thought don’t consistently provide an advantage.

This work furthers our understanding of LLM’s limitations and capabilities within competitive and logical strategic environments, prompting the need for targeted advancements in AI strategic reasoning.