State-Driven Reasoning · NeurIPS 2026

State of Thought Enables Endogenous Reasoning

SoT reframes test-time reasoning as a closed loop: the model's endogenous state selects the right historical evidence for the next step and controls when to stop, rather than following fixed external reasoning programs.

Zhiren Gong^1,2, Yikun Hou¹, Zihao Zeng¹, Ming Xiao⁴, Chau Yuen³, Wei Yang Bryan Lim¹

¹ College of Computing and Data Science, Nanyang Technological University ² Interdisciplinary Graduate Programme, Nanyang Technological University ³ School of Electrical and Electronic Engineering, Nanyang Technological University ⁴ Information Science & Engineering, KTH Royal Institute of Technology, Sweden

Across 3 LLMs and 2 VLM scales (16 text benchmarks + 3 multimodal tasks), a 582-parameter controller on frozen backbones improves accuracy while cutting generated tokens by 62.6% and latency by 44.6%.

▶ Watch the tutorial Paper (arXiv) Code Cite (BibTeX)

Reasoning, from the inside

AskWhich bag for a rainy hiking trip?

Each step, the endogenous state re-selects only the past steps that matter for this step (green links) — then raises p(stop) as the answer locks in.

Chain-of-Thought520 tok

State of Thought198 tok

—

fewer tokens generated

—

lower end-to-end latency

582

param controller, frozen backbone

One tiny controller reads the endogenous state to organize evidence and stop early — consistent gains across 3 LLMs + 2 VLMs.

From external control to endogenous state-conditioned reasoning — Figure. Paradigm shift: from external control to endogenous state-conditioned reasoning.

Why This Paradigm Matters

External scripts are rigid: fixed reasoning formats are brittle across heterogeneous tasks.
Search-heavy methods are costly: quality gains often depend on large sampling and high latency.
SoT changes the control variable: reasoning is driven by endogenous state, not external templates.
Result: better quality-efficiency frontier with a reusable closed-loop controller.

Figure. SoT performance across models, tasks, and efficiency metrics.

3 + 2

LLMs + VLM scales

16

Text benchmarks (+3 multimodal)

62.6%

Token reduction

44.6%

Latency reduction

1.34×

Quantitative gain factor

1.62×

General gain factor

1.76×

Symbolic/code gain factor

2.51×

Long-context gain factor

582

Controller parameters

84.1%

Black-box judge agreement

Abstract

Test-time compute improves LLMs, but existing reasoning paradigms rely on externally imposed control — fixed reasoning programs or costly search in constrained spaces — which limits both generalization and efficiency. We propose State of Thought (SoT), a paradigm that enables endogenous reasoning, with the model's own internal reasoning state governing how reasoning unfolds.

Concretely, SoT extracts a compact dynamics-geometric state from the model's internal information transfer and uses a 582-parameter controller on frozen backbones to selectively activate the historical reasoning support useful under the current state — framing reasoning as a state-conditioned process over evidence rather than an externally prescribed token chain.

Across quantitative (1.34×), general (1.62×), symbolic-and-code (1.76×), and long-context (2.51×) reasoning on 3 LLMs and 16 datasets, SoT consistently improves accuracy while reducing generated tokens by 62.6% and latency by 44.6%. Across 2 VLM scales and 3 tasks, it adds 3.8 points over reasoning baselines with 74.9% fewer completion tokens and 73.5% lower latency than search-based methods.

Method

1) Read the endogenous state

At each sentence-level step, SoT reads a compact dynamics-geometric state m_t ∈ ℝ⁴ from the backbone's internal information transfer: geometry (within-step dispersion), progress (step displacement), direction (consistency between steps), and uncertainty (predictive entropy).

Output: a control signal reflecting the current reasoning regime.

2) Select state-matched evidence (𝒮)

The evidence-organization operator re-selects only the sparse subset of past sentence-level reasoning units that match the current state, and re-inserts them in order — sharpening the evidence carried into the next step, rather than broadening search.

Output: a sparse active context for the next reasoning step.

3) State-conditioned stopping (𝒯)

The same state feeds a stopping operator, p_stop = σ(𝒯(m_t)): reasoning ends when enough support has accumulated, so depth emerges from state rather than a fixed schedule.

Output: a closed loop balancing quality and efficiency.

State trajectories across reasoning paradigms — Figure. Different reasoning paradigms occupy distinguishable endogenous state regions; SoT spans broader adaptive regimes.

State-conditioned activation and stopping patterns — Figure. Different endogenous states induce different sparse evidence activation patterns and stop tendencies.

Mechanism Insight

Tiny, frozen-backbone controller: only a 582-parameter interface is trained (a 16×32 selection head + a 4→1 stopping head); the backbone stays frozen. The selector is stable across problem-held-out refits (AUC 0.779 ± 0.005).
Generalization source: SoT transfers as a control principle, not as a dataset-specific prompt recipe.
Efficiency source: compute is redirected to state-matched support, rather than uniformly longer chains.
Interpretability: state clusters align with distinct evidence-selection and stop behaviors.

Main Results

Main Result Table (Llama-3.1-8B · Quantitative + Symbolic/Code)

Category	Method	GSM8K	MATH	DROP	QS avg	FOLIO	ProofWriter	BBH-Temporal	HumanEval	MBPP	S&C avg
Greedy	Vanilla	79.0	58.2	6.2	53.9	29.6	26.0	26.0	25.0	55.2	33.0
Reasoning	CoT	80.8	64.2	1.7	55.5	4.9	24.5	70.0	25.0	51.6	31.2
Reasoning	PS	71.2	40.0	1.4	43.4	4.9	15.2	55.0	10.0	22.0	17.6
Reasoning	SR	75.8	44.8	15.6	50.4	32.5	31.0	46.0	25.0	36.4	32.9
Reasoning	SC	83.6	54.0	1.6	53.2	16.8	28.5	52.0	10.0	35.2	27.3
Reasoning	CB	48.0	36.5	19.8	37.1	40.4	35.8	34.0	30.0	49.2	38.6
Reasoning	MCTS	55.2	33.5	13.2	37.5	36.0	31.5	30.0	35.0	49.2	36.7
Memory	H2O	79.8	59.2	2.4	53.6	6.4	27.8	48.0	10.0	42.0	26.3
Memory	SNAP	80.4	60.2	2.3	54.1	8.4	26.2	55.0	15.0	41.2	27.3
Memory	STREAM	70.4	44.2	1.3	44.4	1.5	31.8	1.0	0.0	5.6	13.0
Latent	COCO	81.4	60.0	3.7	54.8	45.3	45.2	51.0	20.0	47.2	42.5
RL-Based	GRPO	0.0	0.5	0.3	0.2	2.0	2.2	7.0	0.0	0.0	1.8
Ours	SoT	81.8	64.8	40.5	65.8	49.8	64.8	78.0	59.1	46.8	58.4

Main Result Table (Llama-3.1-8B · General + Long-Context)

Category	Method	CommonsenseQA	StrategyQA	BoolQ	MMLU	RACE	GU avg	HotpotQA	NarrativeQA	LongBench MultiFieldQA	LCR avg
Greedy	Vanilla	48.0	68.8	59.2	45.6	50.7	54.2	7.0	20.1	32.7	16.2
Reasoning	CoT	32.8	31.5	37.2	34.0	50.3	36.3	2.0	4.5	26.9	7.1
Reasoning	PS	30.5	18.2	21.5	31.2	42.0	28.1	1.7	3.0	20.4	5.3
Reasoning	SR	57.0	64.0	72.2	62.4	62.7	63.6	17.9	21.7	25.5	20.6
Reasoning	SC	28.5	34.5	43.2	31.0	45.7	35.8	1.9	4.3	25.8	6.8
Reasoning	CB	46.0	71.5	82.2	51.2	56.0	61.1	15.5	27.5	26.2	21.8
Reasoning	MCTS	42.0	65.5	82.2	46.4	49.7	57.0	13.5	27.1	27.8	21.0
Memory	H2O	33.5	30.8	39.2	35.0	19.7	32.4	2.0	2.0	0.0	1.7
Memory	SNAP	33.5	31.0	38.5	34.6	16.7	31.8	2.0	2.0	0.0	1.7
Memory	STREAM	34.8	26.2	38.8	22.4	0.0	25.6	2.5	0.0	0.0	1.1
Latent	COCO	40.5	63.5	70.0	42.8	43.3	52.0	4.4	10.8	34.6	11.9
RL-Based	GRPO	10.5	3.8	9.8	8.2	12.0	8.7	0.6	1.1	5.3	1.6
Ours	SoT	69.5	68.8	82.3	65.0	67.0	70.4	24.9	39.0	34.7	31.9

Extended Experimental Modules

Click each module to show the corresponding experimental table and interpretation.

Backbone	Quantitative Reasoning avg	Symbolic and Code avg	General Understanding avg	Long-Context avg
Qwen2.5-14B (SoT)	66.1	76.8	81.4	25.2
Mixtral-8x7B (SoT)	46.3	51.7	72.6	23.5

Qwen2.5-14B shows especially strong General Understanding performance across all five GU datasets.
Mixtral-8x7B remains near-best in quantitative tasks and leads strongly in symbolic, GU, and long-context averages.
The same SoT controller transfers across markedly different backbone architectures.

Scale	Task	SoT acc.	SoT tokens	SoT latency (s)	BoN acc.	BoN tokens
Qwen2.5-VL-7B	A-OKVQA	87.5	191	4.4	84.5	691
Qwen2.5-VL-7B	AI2D	86.7	215	5.4	88.7	961
Qwen2.5-VL-7B	M³CoT	87.3	198	4.5	86.7	919
Qwen2.5-VL-32B	A-OKVQA	85.5	425	23.2	85.0	1322
Qwen2.5-VL-32B	AI2D	90.0	364	19.2	87.3	1680
Qwen2.5-VL-32B	M³CoT	88.0	409	20.1	85.3	1633

SoT is best in 5 of 6 model–task settings, averaging 87.5% accuracy.
On 7B AI2D it trails BoN by 2.0 points but uses 4.5× fewer completion tokens; at 32B it exceeds the strongest alternative by 0.5–0.7 points on all three tasks with 3.1–4.6× fewer tokens.
Across both scales: +3.8 accuracy points over reasoning baselines, with 74.9% fewer completion tokens and 73.5% lower latency than search-based methods.

Variant	Quantitative avg	Symbolic and Code avg	General Understanding avg	Long-Context avg
Top-3 baseline average	53.2	44.0	61.5	24.2
SoT-Training-free	59.0	50.1	52.0	26.9
SoT-Embedding	59.0	49.8	51.9	27.9

Even without full internal access, SoT variants remain competitive and often exceed Top-3 baseline averages in key domains.
The largest relative resilience appears in long-context tasks, where trajectory-level organization is critical.
These results support that the mechanism is not tied to one specific implementation interface.

Performance Interpretation

SoT consistently outperforms strong baselines across heterogeneous reasoning regimes. This pattern suggests the gain is structural: changing the control variable from external token programs to endogenous state-conditioned evidence organization.

On Llama-3.1-8B, SoT leads all four domain averages and is best or tied-best on 13 of 16 datasets, with an average domain-level gain of +10.8 points over the strongest competitor. Against the best non-SoT baseline per domain it gains +10.3 / +15.9 / +6.8 / +10.1 points (quantitative / symbolic-and-code / general / long-context) — robust gains without larger search budgets.

Efficiency & Trade-off

Tradeoff on Qwen2.5-14B — Figure. Trade-off on Qwen2.5-14B.

Tradeoff on Mixtral-8x7B — Figure. Trade-off on Mixtral-8x7B.

Accuracy-efficiency trade-off on Llama-3.1-8B by method family — Figure. Accuracy–efficiency trade-off on Llama-3.1-8B (by method family).

Tradeoff on Qwen2.5-VL 7B and 32B — Figure. Trade-off on Qwen2.5-VL (7B / 32B) — SoT stays at high accuracy with far fewer tokens.

Llama Efficiency Summary (Domain Average)

Method	QS Tok / Lat	S&C Tok / Lat	GU Tok / Lat	LCR Tok / Lat	Avg Tok	Avg Lat (s)
CoT	271.6 / 17.5	369.8 / 27.5	212.9 / 12.8	198.1 / 19.6	263.1	19.4
Self-Consistency	736.8 / 43.1	953.9 / 63.5	626.5 / 37.6	533.8 / 48.9	712.8	48.3
Constrained Beam	423.9 / 22.0	363.9 / 19.4	341.7 / 16.8	283.7 / 19.5	353.3	19.4
MCTS	422.1 / 21.4	364.7 / 19.4	335.2 / 16.5	286.6 / 20.5	352.1	19.4
H2O	226.0 / 16.7	264.4 / 34.6	185.6 / 15.8	164.6 / 21.5	210.2	22.1
COCO	171.5 / 11.5	214.6 / 20.4	87.2 / 6.7	131.5 / 30.3	151.2	17.2
SoT	255.7 / 5.8	302.2 / 9.5	74.9 / 1.8	263.0 / 22.0	223.9	9.8

Efficiency Insight

Search-heavy baselines often move to high-token and high-latency zones.
Memory-only compression may cut tokens but can lose state-relevant support and hurt quality.
SoT frontier shift: quality gains and cost reduction are achieved jointly, not by a simple trade.

Case Studies

Case-Level Insight

Across arithmetic, long-context QA, and multimodal chart reasoning, SoT exhibits a shared pattern: it keeps only support still needed for the next decision, suppresses obsolete context, and raises stop readiness when sufficient evidence has accumulated.

Representative SoT case trajectory — Figure. Representative SoT case with step-level active evidence and stop probability.

Arithmetic case study — Figure. Arithmetic case: staged decomposition with non-monotone evidence carry.

Long-context case study — Figure. Long-context case: sparse retrieval after exploration stabilizes contrastive reasoning.

VLM case study — Figure. Multimodal case: the same state-conditioned interface transfers to chart-grounded reasoning.

Contact

For collaboration or questions, contact zhiren001@e.ntu.edu.sg.