LLMs with Chain-of-Thought Are Non-Causal Reasoners investigates the perplexing phenomenon of LLMs producing correct answers after incorrect reasoning chains and vice versa. This study introduces causal analysis to unravel the intricate relationship between CoTs, instructions, and final answers in LLMs, uncovering the causal structure they approximate and how it differs from human reasoning.

Highlights:

• The study critically analyzes the Chain of Thought (CoT) approach and reveals instances of correct outcomes following flawed reasoning.
• A Structural Causal Model (SCM) is proposed to dissect the cause-effect relationship in LLM reasoning, contrasting it with human cognition.
• It was found that in-context learning, supervised fine-tuning, and reinforcement learning profoundly affect the LLM’s implied causal structure.
• Such exploration might guide improvements in LLM reasoning fidelity and trustworthiness.

Evaluating the causality in LLM reasoning raises essential questions about the models’ reliability and interpretability. By probing into the SCM of LLMs, the paper urges the AI community to critically assess and refine the reasoning processes of these sophisticated systems. Understanding and addressing the discrepancies with human reasoning are crucial for advancing AI agents capable of more accurate and transparent cognition. Explore More