CogBench: a large language model walks into a psychology lab by Coda-Forno et al. presents a unique approach to evaluating large language models (LLMs) via cognitive psychology experiments. CogBench introduces ten behavioral metrics derived from seven cognitive experiments, providing an innovative toolkit for ‘phenotyping’ LLM behavior.

• The application of CogBench to 35 LLMs produced a rich dataset showing variances in behavior.
• The study employs statistical modeling to distill the influences of model size and reinforcement learning from human feedback.
• Results show that open-source models tend to be less risk-prone compared to proprietary models, and that models trained on code do not automatically exhibit enhanced behavior.
• Prompt-engineering techniques like chain-of-thought prompting were found to improve probabilistic reasoning and foster model-based behaviors.

My Opinion: The convergence of AI and cognitive psychology through CogBench is a novel and exciting domain that can yield insights into the ‘mind’ of AI. By understanding LLM behavior more deeply, we enhance our capacity to predict, manipulate, and interpret AI responses in a variety of contexts, from casual interactions to critical decision-making environments.