EMNLP 2025 MAIN · LLM/NLP

ThinkSLM: A Comprehensive Audit of Small-Language-Model Reasoning

A six-axis matrix of training origins, prompting, compression, and robustness

ACL Anthology · 2025.emnlp-main.1659 2026-09-14 Reasoning · Compression · Evaluation

Eight-bit quantization usually preserves existing reasoning, while structured pruning can push intermediate reasoning from usable to failure.

Core paper: Gaurav Srivastava, Shuxiang Cao, Xuan Wang. ThinkSLM: Towards Reasoning in Small Language Models. EMNLP 2025 Main,ACL Anthology 2025.emnlp-main.1659,pp. 32612–32662。
Report date: 2026-09-14 Field: LLM/NLP (reasoning, post-training, evaluation) Evidence convention: Prioritize the final ACL version; all recomputations are marked as this report's calculations.

0. Verified links and version audit

All URLs were accessed on 2026-09-14. “Not public/not found” reflects public-page status that day, not inference.

Work Conference / official page arXiv version and date Code Data / weights Project page / videos
ThinkSLM EMNLP 2025 Main / ACL;PDF 2502.11569v3, 2025-09-30; v1 dated 2025-02-17; arXiv retains the old title The project's Code link targets GitHub URL and returns 404, so it is marked not public No dedicated weights released; complete results/scripts repository not found Leaderboard project page; video not found
GSM-Plus ACL 2024 Long;PDF 2402.19255v1,2024-02-29 qtli/GSM-Plus Repository provides data/evaluation; weights not applicable Project page/video not found
MR-GSM8K No formal conference page seen 2312.17080v4,2024-06-05 JIA-Lab-research/MR-GSM8K Repository provides data/evaluation; weights not applicable Project page/video not found
DeepSeek-R1 Nature 2025 DOI 2501.12948v2, 2026-01-04; v1 dated 2025-01-22 Official DeepSeek-R1 repository Official weights and six distilled models Repository/model cards double as project pages; paper video not found
Minitron No formal conference page seen 2408.11796v4,2024-12-09 NVlabs/Minitron Repository links public 4B/8B weights Repository includes blog/demo links; paper video not found

Version risk. The final ACL abstract states 72 models, six families, 17 benchmarks, and 1,000 human-validated examples(ACL PDF pp.1,4), while the project page still says 14 benchmarks and 800 human examplesand its “Code (Will be available soon)” link returns 404. More seriously, §3.3 describes majority voting by three annotators, while Appendix G (p.50) describes one graduate student repeating each judgment three times. This report treats final-PDF tables as final results, but the human-evaluation design is not fully reproducible.

1. Candidate pool, deduplication, and random selection

First check ml-report-index.md: recently covered ATLANTIS (SFT data weighting), DEFAME (multimodal retrieval evidence), and Emergent Misalignment (safety fine-tuning) were excluded. Candidates all came from EMNLP 2025 Main Proceedings, with arXiv, Hugging Face Daily Papers, and Semantic Scholar checking titles/versions. OpenReview is not EMNLP's formal publication venue, and some Papers with Code records are unstable, so missing entries were not grounds for exclusion.

There were 24 candidates:

  1. Thinking Out Loud: Do Reasoning Models Know When They’re Right?
  2. Learning Like Humans: Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation
  3. How Far Can LLMs Improve from Experience?
  4. LaMDAgent: An Autonomous Framework for Post-training Large Language Models
  5. Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings
  6. Table-R1: Inference-Time Scaling for Table Reasoning
  7. Forget What You Know about LLMs Evaluations—LLMs Are Like a Chameleon
  8. Order Doesn’t Matter, But Reasoning Does: Training LLMs with Order-Centric Augmentation
  9. ThinkSLM: Towards Reasoning in Small Language Models
  10. Let’s Reason Formally: Natural–Formal Hybrid Reasoning Enhances LLM Math Capability
  11. CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation
  12. Parallel Continuous Chain-of-Thought with Jacobi Iteration
  13. Direct Judgement Preference Optimization
  14. reWordBench: Benchmarking and Improving Instruction-Level Reward Models
  15. Critique Fine-Tuning on One Problem Is Enough to Improve Math Reasoning
  16. TokenSkip: Controllable Chain-of-Thought Compression in LLMs
  17. Diverse Reasoning Paths are Better than One
  18. Two Heads Are Better Than One: Test-time Scaling of Verbal Reflection
  19. Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
  20. AdaptThink: Reasoning Models Can Learn When to Think
  21. T²: An Adaptive Test-Time Scaling Strategy for Contextual Question Answering
  22. A Systematic Analysis of Base Model Choice for Reward Modeling
  23. From Generation to Judgment: Opportunities and Risks of LLM-as-a-Judge
  24. LLMs Have Intrinsic Meta-Cognition but Need a Good Lens

The system command executed was shuf -i 1-24 -n 1returned 9, selecting ThinkSLM. No redraw was needed: the 51-page final paper, complete appendices, main results, and robustness tables were accessible and sufficient.

2. Motivation: the problem is confounded comparison axes, not simply small models

“More parameters mean better reasoning” often confounds scale with more pretraining data, stronger post-training, longer output budgets, and different evaluators. Existing work separately examines distillation, quantization, CoT, or mathematics datasets, rarely in one matrix. ThinkSLM evaluates 72 models, six families, and 17 benchmarks across six axes: capacity/family, training/architecture, prompting, compression, task domain, and robustness (abstract, §3.1, pp.1,3).

Three quantitative observations motivate this.

First,family differences at equal size can dwarf some scaling gains. Qwen2.5-7B reaches 91.76% on GSM8K versus Mistral-v0.3-7B at 55.98%, a 35.78 pp or 63.91% relative gap. Scaling Qwen2.5 from 7B to 14B adds only 2.53 pp, 91.76→94.29, or 2.76% relative (Table 3, pp.5–6; percentages recomputed here). Training recipes are at least as consequential as scale.

Second,common automatic scoring systematically underestimates small models. On 1,000 human-judged examples, agreement is 0.51 for lm-eval-harness, 0.36 for regex parsing, and 0.60 for direct-answer extraction, versus 0.99 for GPT-4-turbo and 0.98 for GPT-4o (Table 1, p.4). With unstable formatting, evaluations may measure instruction following rather than solution correctness.

Third,compression is not one operation. Qwen2.5-32B GPTQ 8-bit reduces measured GPU memory from 125 GB to 33.81 GB (−72.95%) while Overall moves 92.64→92.65. Conversely, 2:4 pruning of Sparse-Llama-3.1-8B drops MR-GSM8K from 24.2 to zero (Tables 3,8, pp.5,33; reduction recomputed). The question becomes which capabilities survive under which training origins, compression routes, and evaluation definitions.

3. Argument, assumptions, and trade-offs

The study proceeds in four steps. First, select scorers against human judgments rather than assuming exact match: compare lm-eval, parsers, direct answers, and four GPT judges. Second, build a 72-model matrix across family, scale, training/architecture, and prompting. Third, compare quantization, pruning, and distillation. Fourth, extend static GSM8K accuracy to GSM-Plus perturbations, MR-GSM8K intermediate steps, and MR-Ben error recognition.

Three assumptions underlie the design:

  • High judge–human agreement permits scaled evaluation, at the cost of embedding closed-source judge bias and expense in the metric.
  • Changing size/bitwidth within one family and prompt approximately isolates capacity/compression, although community checkpoints, defaults, and hardware paths are not fully controlled.
  • Sorting tasks unlikely to occur in pretraining can proxy low-contamination algorithmic reasoning, but generating correct sequences also demands format compliance.

The strength is a broad comparison matrix, not causal identification. It offers practical safe operating regions, but observational rankings cannot identify whether 18T tokens, RLHF, or architecture caused gains.

4. Equation-level formalization and evaluation flow

Given model output (y) and reference answer (y^), evaluator (m) produces binary score (S_m(y,y^){0,1}). The authors choose the evaluator with greatest agreement with human score (S_h):

[ m^*=_{mM}(S_m,S_h). ]

The parsing baseline is (S_{parse}(y,y^*)=[extract(y)=y^*]). Paired judge–human differences use McNemar's statistic:

[ ^2=, ]

Here (b) counts human-correct/method-wrong cases and (c) human-wrong/method-correct cases (Eqs.1–2, p.4). The authors also define (C_{total}=n(c_{token}l_{avg}+c_{base})) and use (Utility=-Cost) to explain GPT-4-turbo for GSM8K and GPT-4o for ARC/CommonsenseQA.

MR-GSM8K combines correctness judgments, first-error localization, and error explanations:

[ MR=w_1(0,MCC)+w_2ACC_{step}+w_3ACC_{reason}. ]

Weights are chosen empirically, without sensitivity analysis (Appendix A.5, p.16).

Evaluation pseudocode:

Input: model checkpoint, task, prompt template, compression variant
1. Generate outputs; repeat three times unless otherwise stated
2. GSM8K → GPT-4-turbo; ARC-E/C and CSQ → GPT-4o
3. Sorting → deterministic parser covering 13 output formats
4. Robustness → original benchmark scripts + GPT-4o
5. Aggregate task accuracies, Sorting_Avg, and equally weighted five-column Overall
6. Slice by family/scale/prompt/compression/robustness rather than only overall ranking
Output: performance, GPU memory, disk space, means/selected standard deviations

5. Key mechanisms and engineering details

Module Implementation Key settings Reproducibility assessment
Generation backend vLLM v0.7.3; Accelerate for multiple GPUs Version reported, but sampling temperatures/seed lists not centrally specified (Appendix F, p.50)
Model collection Qwen2/2.5, Llama3.x, Mistral, Phi, SmolLM2, Hymba, Minitron, etc. Default Hugging Face settings; maximum multi-shot input 4096 Defaults introduce model-specific template differences
Hardware H100-80GB、A100-80GB、L40-48GB、A40-48GB GPU memory reportedly measured on A100-80GB; about 24,000 GPU-hours total No per-model wall-clock, energy, or throughput (Appendix F, p.50)
Judge Mixed GPT-4-turbo / GPT-4o judging GPT-4-turbo for mathematics; GPT-4o for science/common sense High agreement, but evaluator regime changes by task
Sorting Thirteen regex formats Lengths 8/16/32; positive-only and mixed-sign numbers 98.8% agreement on 400 human-checked examples (Table 2, p.4)
Quantization GPTQ-8/4、W8A8、W8A16、W4A16、FP8 Checkpoints from Qwen/Neural Magic community collections Not a controlled ablation using one quantization toolchain
Pruning/distillation SparseGPT, 2:4, Minitron, etc. Varying retraining/distillation status and data quantities Heterogeneous methods suit risk screening, not causal ranking

6. Datasets, benchmarks, and metrics

Mathematics includes GSM8K, MATH, MathQA; science ARC-Easy/Challenge; common sense CommonsenseQA, OpenBookQA, HellaSwag; algorithms six custom Sorting tasks; robustness GSM-Plus, MR-GSM8K, MR-Ben (§3.2 and appendices, pp.3,14–17). The main table shows GSM8K, ARC-E, ARC-C, CSQ, and Sorting_Avg with a simple equal-weight average:

[ Overall=. ]

For Qwen2.5-0.5B, ((46.80+62.50+44.28+46.90+2.61)/5=40.62), matching Table 3 (recomputed here). This is convenient for ranking, but averages six sorting subsets into one column weighted equally with individual datasets. It is neither sample-weighted nor justified by task importance.

Table 19 (p.51) lists test sizes: GSM8K 1,319; ARC-E 2,376; ARC-C 1,172; CommonsenseQA 1,140; HellaSwag 10,003; MathQA 2,985; OpenBookQA 500; MATH 5,000. Sorting subset counts are not clearly listed there, preventing overall-CI recomputation.

7. Complete main-results matrix: six scales within one family

The table fixes Qwen2.5, no quantization, and Direct I/O to avoid mixing family/compression with scale. Bold marks column bests; Overall Δ is versus the preceding size. Pp and relative changes are recomputed here; original values come from Table 3 (p.5).

Model GSM8K ARC-E ARC-C CSQ Sorting Avg Overall Overall Δ
Qwen2.5-0.5B 46.80±0.77 62.50±0.21 44.28±1.03 46.90±1.49 2.61 40.62 —
Qwen2.5-1.5B 70.00±0.53 87.58±0.21 73.81±0.30 71.85±0.48 29.11 66.47 +25.85 pp / +63.64%
Qwen2.5-3B 84.74±0.28 93.49±0.07 83.73±0.38 76.25±0.29 39.89 75.62 +9.15 pp / +13.77%
Qwen2.5-7B 91.76±0.20 96.03±0.17 90.53±0.12 82.66±0.14 57.67 83.73 +8.11 pp / +10.72%
Qwen2.5-14B 94.29±0.40 97.87±0.05 93.37±0.22 84.08±0.54 74.61 88.84 +5.11 pp / +6.10%
Qwen2.5-32B 95.40±0.29 98.26±0.10 95.25±0.18 87.11±0.37 87.39 92.64 +3.80 pp / +4.28%

Saturation is clear but asynchronous: GSM8K gains only another 3.64 pp beyond 7B, whereas Sorting rises from 57.67 to 87.39, another 29.72 pp. Diminishing scale returns on an established math set do not automatically transfer to longer algorithmic execution.

8. Quantization: 8-bit is stable; 4-bit risk concentrates in small models and long execution

Qwen2.5 Variant GPU GB Overall Δ versus full precision Sorting Avg Δ
0.5B GPTQ-8 0.71 40.36 −0.26 pp / −0.64% 0.00 pp
0.5B GPTQ-4 1.12 32.55 −8.07 pp / −19.87% −1.11 pp
3B GPTQ-8 4.21 75.72 +0.10 pp / +0.13% −0.33 pp
3B GPTQ-4 2.88 71.23 −4.39 pp / −5.81% −10.45 pp
7B GPTQ-8 9.63 83.97 +0.24 pp / +0.29% +1.11 pp
7B GPTQ-4 6.48 78.40 −5.33 pp / −6.37% −23.78 pp
14B GPTQ-8 17.24 89.02 +0.18 pp / +0.20% +0.17 pp
14B GPTQ-4 10.65 87.63 −1.21 pp / −1.36% −5.05 pp
32B GPTQ-8 33.81 92.65 +0.01 pp / +0.01% −0.22 pp
32B GPTQ-4 52.42 92.67 +0.03 pp / +0.03% −0.11 pp

Original scores: Table 3 (p.5); differences recomputed here. The ±0.24 pp 8-bit fluctuations are near individual-task run variability and should not be interpreted as improved reasoning, only no detectable substantial loss. Four-bit shows a scale transition: 0.5B Overall loses 19.87% relatively, 7B mainly loses Sorting performance, and 32B barely changes. Notably, measured 32B GPTQ-4 memory is 52.42 GB, exceeding GPTQ-8's 33.81 GB: bitwidth is not end-to-end memory, so kernels, caches, and implementation paths require measurement.

Qwen2.5 scale–8-bit capability frontier

Overall accuracy (%); curves nearly overlap, but this is not proof of statistical equivalence.

Full precisionGPTQ 8-bit
Full-precision versus GPTQ 8-bit Overall across six Qwen2.5 sizes From 0.5B to 32B, both series rise from about 40 to about 93, with small gaps throughout. 4060801000.5B1.5B3B7B14B32B
Source: ThinkSLM Table 3 (p.5); single y-axis is Overall accuracy (%). Hover or keyboard-focus points for exact values.

9. Robustness, layered ablations, and transitions

9.1 Training origins

Qwen2.5-3B Base→Instruct: GSM8K 77.91→84.74 (+6.83 pp), GSM-Plus 60.44→68.33 (+7.89 pp), MR-GSM8K 8.1→11.0 (+2.9), but MR-Ben average only 6.9→8.8 (+1.9). Instruction tuning helps solving and surface perturbations more than detecting others' reasoning errors (Table 4, p.8; differences recomputed).

9.2 Quantization and scale

Qwen2.5-32B and GPTQ-8 both score 55.6 on MR-GSM8K; GSM-Plus is 82.71 versus 82.78, MR-Ben average 21.4 versus 21.3 (Table 4, p.8). This is relatively clean evidence of preservation, but no equivalence test was performed; matching decimal values do not prove statistical equality.

9.3 Pruning

Llama-3.1-8B MR-GSM8K is 24.2 unpruned, 23.3 with W8A16, and zero after 2:4 pruning; GSM-Plus is 67.10, 66.78, and 35.17 (Table 4, p.8). Appendix A.6 (p.17) admits skipping ARC-E/C and CSQ for some pruned models because outputs are nonsensical and accuracy near zero. This is a discrete transition into unscorable responses, not merely a few lost average points.

9.4 Prompting

Figure 1c and §4.4 (pp.6–7) report only about 2% gains from CoT for post-2024 models, with complex prompts sometimes hurting. Yet Appendix Table 9 gives pruned models 10–20 pp gains from CoT over Direct I/O: Llama-3.1-8B 2:4 rises 51.86%→60.27%, then falls to 1.95% with five-shot CoT (p.33). Prompting matters less only for stronger aligned models; after structural damage, it becomes an unstable compensator.

9.5 Hyperparameter sensitivity

The paperNot reportedsystematic sweeps of quantization calibration size, GPTQ block/group settings, temperature/sampling, or MR-Score weights, nor continuous pruning-rate curves. Observed transitions use discrete checkpoints: stable 8-bit, degraded small-model 4-bit, and collapsed 2:4 pruning. Missing intermediate points prevent identifying critical bitwidths or sparsity rates.

10. Statistical significance: extensive reporting, but an incomplete evidence chain

Positively, experiments reportedly repeat three times. Table 1 gives McNemar tests, MAE, and 1,000-bootstrap CIs for judges: GPT-4-turbo agreement 0.99, 95% CI [0.98,1.00], p=0.249; GPT-4o 0.98, [0.97,0.99], p=0.156 (p.4).

Four boundaries remain. First, p>0.05 means failure to reject zero difference, not passing an equivalence test. Second, main results lack paired between-model significance, CIs, and multiple-comparison correction. Third, §4 says three-run variance is always below 1.49%, yet Table 7 includes standard deviations ±3.25 and ±2.79, and Table 10 includes ±5.67, inconsistent with that statement (pp.5,32,34). Fourth, Sorting_Avg and Overall lack standard deviations, so the ordering of 32B scores 92.64, 92.65, and 92.67 may not reproduce.

This report therefore treats large effects such as +35.78 pp or MR 24.2→0 as credible directions, while interpreting 0.01–0.24 pp quantization differences as preservation, not improvement.

11. Genuine contributions, reusable design, and transfer potential

There are three contributions. First, calibrate evaluators before scaling the model matrix, avoiding exact-match confusion between formatting and reasoning errors. Second, show capability–memory frontiers within families across sizes/bitwidths: 8-bit is a more conservative engineering starting point than structured pruning. Third, extend final-answer accuracy to perturbations, first-error localization, and explanations, exposing substantial metareasoning gaps despite saturation on older benchmarks.

The reusable design is a three-stage evaluation, not another overall leaderboard:

  1. Output judgeability: anchor judges/parsers to humans and record unparseable rates;
  2. Task capability: fix prompts/budgets and test mathematics, science, common sense, and algorithmic execution;
  3. Stress testing: paired perturbations, process-error localization, and post-compression regression tests.

For code models, replace Sorting with unit-test sequences and bug localization; for medical models, replace MR-Ben with first-error diagnosis-chain analysis. For edge models, add TTFT, TPOT, energy, and peak RAM to GPU memory, then analyze joint bitwidth–kernel Pareto frontiers.

12. Related work: problem origins and technical lineage

12.1 GSM-Plus(ACL 2024 Long)

Problem addressed: Generates GSM8K perturbations involving value substitution, digit expansion, integer/decimal/fraction conversion, added/inverse operations, paraphrases, distractors, and critical thinking. The original evaluates 25 models and four prompts, showing that solving originals does not imply perturbation robustness (GSM-Plus abstract/text).Remaining gap: It targets larger LLMs, without systematically testing small or compressed models' robustness retention.ThinkSLM's response: place GSM-Plus alongside GSM8K and report drops: Qwen2.5-32B falls 95.40→82.71, still a 12.69 pp decline (Table 4, p.8).Transferred design: Paired changes to surface form or operational conditions become compression regression tests beyond static accuracy.

12.2 MR-GSM8K(arXiv 2312.17080v4)

Problem addressed: Turns models from students into teachers: judge correctness, find the first wrong step, and explain it, combining MCC, step accuracy, and reason accuracy into MR-Score. Some models approach GPT-4 on GSM8K yet trail by over 20 absolute points on MR-GSM8K (original abstract).Remaining gap: The original does not cover a broad SLM compression lineage.ThinkSLM's response: Tests 3B/8B/32B/70B and quantized/pruned versions: Qwen32B reaches 55.6 versus the cited GPT-4-turbo 53.0; Llama8B scores 24.2 and zero after pruning (Table 8, p.33).Transferred design: Moving from correct results to process auditing magnifies the pruning fragility found by ThinkSLM.

12.3 DeepSeek-R1(Nature 2025 / arXiv 2501.12948)

Problem addressed: RL incentivizes reflection, verification, and adaptive strategies; large-model reasoning traces are distilled into six dense checkpoints: 1.5B, 7B, 8B, 14B, 32B, 70B (official paper/model cards).Remaining gap: A single model report cannot broadly separate distillation gains from family, prompts, and compression.ThinkSLM's response: Treats teacher-driven distillation as a training-origin axis alongside Minitron, Qwen, Llama, and others in one task matrix.Transferred design: Small-model reasoning arising from transferable training signals, not only spontaneous scale emergence, becomes a central ThinkSLM hypothesis.

12.4 Minitron(arXiv 2408.11796v4)

Problem addressed: Depth/width pruning of Llama-3.1-8B and Mistral-NeMo-12B, followed by distillation recovery to 4B/8B; the official repository releases implementation and weights.Remaining gap: The original mainly uses common LM Evaluation Harness benchmarks, with limited long algorithmic execution/metareasoning checks.ThinkSLM's response: Evaluates Minitron alongside direct SparseGPT/2:4 on Sorting, GSM-Plus, and MR tasks. Phi-3.5-mini 3.8B scores 85.47% on GSM8K versus Minitron-4B at 27.95 (Table 3, pp.6–7), showing similar parameter counts do not erase training-history differences.Transferred design: Whether pruning is followed by recovery training, and which distillation data are used, must be part of checkpoint identity.

12.5 Nonlinear technical evolution tree

GSM8K: final-answer correctness
├─ GSM-Plus: robustness to paired perturbations ──────────┐
├─ MR-GSM8K: auditing intermediate reasoning ───────┐     │
└─ CoT / RL: explicit processes and self-verification │     │
       └─ DeepSeek-R1: large-model RL → small-model distillation ─┐
                                                                ├─ ThinkSLM: unified measurement matrix
Parameter compression                                           │   ├─ 8-bit: mostly preserved performance
├─ GPTQ / FP8: numerical approximation ─────────────────────────┤   ├─ 4-bit: scale-dependent risk
└─ Minitron / SparseGPT / 2:4: remove structure → retrain ────────┘   └─ pruning: process-capability transition

Two shifts drive this evolution: from answer correctness to stable perturbation/process behavior, and from parameter count alone to training origins and compression paths. ThinkSLM is a convergence point, not a new training algorithm.

12.6 Cross-paper comparison

Work Core object Smallest evaluation unit Compression coverage Process-error coverage Design adopted by ThinkSLM
GSM-Plus Meaning-preserving mathematical perturbations Paired-question accuracy No No Original-to-perturbed performance drop
MR-GSM8K Solution-chain auditing Correctness / first error / reason No Yes MR-Score stress test
DeepSeek-R1 RL and distilled model families benchmark accuracy/pass@k Distillation Partial Training origin takes priority over parameter labels
Minitron Pruning + distillation Compressed-model benchmarks Yes No Recovery training is part of the compression path
ThinkSLM 72-model multi-axis matrix Task/prompt/compression slices Yes Yes Integrates and audits interactions

13. Author-stated limitations

  1. Judge bias. The authors acknowledge GPT-4 is not perfectly accurate; Appendix D.8 even shows nonsensical outputs marked correct. Full human supervision is impractical (Limitations p.14; Appendix D.8 pp.48–49).
  2. Sorting parser errors. Despite 98.8% agreement on 400 checks, regex can reject correct answers in unusual formats (Limitations p.14; Table 2 p.4).
  3. Narrow task coverage. Mostly standard academic benchmarks, lacking real-world and domain reasoning, so findings do not directly transfer to medicine, law, or software engineering (Limitations p.14).

14. Independent critique

  1. [Internal consistency] Because §3.3 describes three-annotator majority voting while Appendix G describes one graduate student repeating judgments three times, κ=0.87's participants and independence cannot be reconstructed, weakening judge calibration's foundation.
  2. [Version governance] Because the final paper uses 17 benchmarks/1,000 examples while the project page says 14/800 and links to nonexistent code, users cannot identify the leaderboard's experimental version. The dynamic leaderboard is not an equivalent copy of final tables.
  3. [Causal attribution] Because Qwen2.5-7B and Mistral-7B differ by 35.78 pp while also differing in pretraining tokens, quality, architecture, and alignment, training data is a plausible explanation, not a controlled causal conclusion.
  4. [Statistical inference] Because GPT-4o's p=0.156 against humans merely fails to reject a difference, comparable/optimal claims require equivalence tests with prespecified tolerances. Nonsignificance does not establish equivalence.
  5. [Multiple comparisons] Because 72 models across tasks/prompts generate many comparisons without correction, and Overall lacks variability, 32B scores 92.64/92.65/92.67 cannot establish real superiority.
  6. [Selection bias] Because ARC-E/C and CSQ are skipped precisely when pruned models produce near-zero, nonsensical results, missingness is nonrandom. Showing only remaining tasks underestimates pruning's overall risk.
  7. [Evaluation contamination] Because GSM8K and ARC have been public for years and training/distillation data may contain their patterns or answers, 95% static accuracy alone does not prove transferable reasoning. Sorting reduces contamination but also measures formatting execution.
  8. [System extrapolation] Because about 24,000 GPU-hours are reported without per-model throughput, latency, energy, batch, or KV-cache definitions, deployability is only approximated by peak memory/disk. End-to-end cost advantages cannot be inferred.

15. Open challenges: actionable head-to-head gaps

Theory

  • Equivalence rather than difference testing: Fix Qwen2.5-7B/14B/32B and compare BF16, GPTQ-8, GPTQ-4. Apply paired per-question TOST with ±1 pp margins, stratifying GSM8K and Sorting. Does 8-bit meet a prespecified engineering tolerance?
  • Scale–bitwidth transitions: With one calibration set and GPTQ implementation, sweep 0.5/1.5/3/7/14/32B × 3/4/6/8 bits and fit distortion–reasoning-length interactions. Does 4-bit risk arise from capacity or accumulated execution error?
  • Process-score weights: Publish MR-GSM8K (w_1,w_2,w_3) and compare ranking stability under equal, expert, and IRT weights. Does Qwen32B's 2.6-point lead over GPT-4-turbo depend on weighting?

Engineering applications

  • Latency-matched comparison: On A100/H100/L40, run BF16, GPTQ-8, GPTQ-4, and 2:4 sparsity under equal TTFT/TPOT budgets. Report peak memory, tokens/s, energy, and accuracy to test whether memory gains become serving gains.
  • Unified checkpoint pipeline: Generate every quantized/pruned variant from one Qwen2.5 checkpoint with fixed calibration data/tokenizer, then compare community checkpoints. How much observed behavior comes from implementation differences?
  • Explicit failure rates: Do not skip nonsensical outputs. Add no-response, parse-fail, and semantic-invalid categories to Overall, distinguishing generation failure from reasoning failure.

Emerging directions

  • Adaptive reasoning budgets: For the same 3B/7B models, compare Direct I/O, fixed CoT, AdaptThink, and TokenSkip at equal tokens/energy, grouped by difficulty. Is thinking less more economical than compressing weights?
  • Contamination resistance: Compare freshly generated, program-verified mathematics and hidden Sorting rules against GSM8K/ARC. Do family advantages survive decontamination?
  • Process faithfulness: For unchanged final answers with counterfactually replaced intermediate chains, compare MR-GSM8K judges, executable verifiers, and humans. Does high MR-Score reflect process tracking or judge-style matching?

16. Recent-work radar

  1. Thinking Out Loud: Do Reasoning Models Know When They’re Right?(EMNLP 2025 Main): explicit confidence rationales support calibration, addressing ThinkSLM's correctness-only evaluation without self-knowledge measurement.
  2. Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time(EMNLP 2025 Main): token-level reasoning-chain memorization diagnostics offer finer tools for contamination in old public benchmarks.
  3. TokenSkip: Controllable Chain-of-Thought Compression in LLMs(EMNLP 2025 Main): compresses output tokens rather than weights. The official abstract reports Qwen2.5-14B GSM8K outputs reduced from 313 to 181 tokens with under 0.4% performance loss.
  4. AdaptThink: Reasoning Models Can Learn When to Think(EMNLP 2025 Main): adaptively chooses Thinking/NoThinking, providing a serving-budget comparison beyond static CoT.
  5. Parallel Continuous Chain-of-Thought with Jacobi Iteration(EMNLP 2025 Main): replaces serial continuous-thought-token dependencies with Jacobi parallel iterations, addressing ThinkSLM's missing latency dimension.
  6. DeepSeek-R1(Nature 2025, arXiv v2 2026-01-04): distills large-model RL reasoning into six dense sizes, remaining key upstream evidence for SLM training-origin effects.
  7. LLMThinkBench: Towards Basic Math Reasoning and Overthinking(2025 arXiv): studies overthinking on simple mathematics, a mechanistic comparison for ThinkSLM's finding that complex prompts sometimes hurt.

17. Final assessment

ThinkSLM turns the slogan-like question of whether small models reason into a six-axis matrix with clear engineering signals: Qwen2.5-32B GPTQ-8 scores 92.65 Overall versus full precision 92.64 while reducing GPU memory 125→33.81 GB; Llama-3.1-8B 2:4 pruning drops MR-GSM8K 24.2→0. Eight-bit often preserves capability, while structural deletion can cross a reasoning-integrity threshold.

It is a risk map rather than causal proof of training laws. Final paper, project page, and appendix conflict on benchmark/sample/annotator counts and code status; overall significance and budget-matched system metrics are missing. Deployment decisions should first reproduce common-source 8-bit/4-bit/2:4 comparisons and include no-response rate, TTFT, TPOT, and energy in one Pareto frontier.


Footnotes and index status: This report cites only the read final ThinkSLM PDF/appendices and public upstream-paper pages. Recomputations are explicitly marked.ml-report-index.md This entry was appended. HTML passed the Warm Editorial static validator (0 errors, 0 warnings), with responsive checks for light/dark/mobile structure. The environment lacks a /usr/bin/chromium or /opt/pw-browsers executable, and direct Playwright launch confirmed no browser, soactual screenshot checks were not completed, nor was the prohibited playwright install。