Summary:

• Examines neural sub-structures within LLMs that facilitate step-by-step reasoning.
• Observes functional differences in LLM layers pertaining to pretraining and in-context reasoning.
• Highlights the role of specific attention heads in generating chain-of-thought (CoT).

Opinion: This study sheds light on the complex internal structures that enable complex reasoning in LLMs, offering a detailed analysis that demystifies the process. The insights provided can guide future model design and optimization for enhanced reasoning capabilities.