100

Define compositional depth.

 title: 'Figure 2: Comparative analysis of thinking versus non-thinking models across math benchmarks reveals inconsistent performance patterns. While results on the MATH-500 dataset show comparable performance between both model types, the thinking models demonstrate superior performance on AIME24 and AIME25 benchmarks. Additionally, the observed performance degradation from AIME24 to AIME25 highlights the vulnerability of these benchmarks to data contamination issues.'

Compositional depth refers to the number of sequential operations or moves required to reach a full solution in a given puzzle environment. In the context of the paper, the compositional depth scales with the problem size across different puzzle environments. For example, the Tower of Hanoi puzzle exhibits exponential growth in compositional depth, while the Checker Jumping puzzle displays quadratic scaling. Other puzzles, like River Crossing and Blocks World, show more moderate, near-linear growth with the number of elements involved in the problem. This characterization of compositional depth is crucial for evaluating how language reasoning models handle various sequential reasoning challenges and whether their accuracy correlates with the compositional depth needed to solve the puzzle[1].


Related Content From The Pandipedia