PARQ: Making “Continuous First, Discrete Later” Quantization Training a Provable Optimization Process

0. Link verification table

All URLs below were accessed during preparation. “Not found/not public” means no separate resource was verified in the paper, official pages, or author repositories, not definitive nonexistence.

1. Selection process and candidate pool

The deduplication index shows DuoAttention (sparse KV-cache retention) and SimbaV2 (deep-RL optimization geometry) covered in the past 14 days, but not proximal optimization and convex regularization for QAT. A 20-paper pool spans efficiency, systems, quantization, long context, and low rank from ICML 2025 and NeurIPS 2025: Star Attention, RAPID, Reward-Guided Speculative Decoding, GPTAQ, Q-VDiT, LRA-QViT, SeedLoRA, BDIA-Transformer, BlockDialect, RWKVQuant, xLSTM 7B, SketchTune, KVTuner, OrthoRank, PARQ, Learned Prefix Caching, MARS, ClusterFusion, HiFC, and LLM numerical nondeterminism analysis. The candidate list and verified homepages are saved in the same directory at candidates_2026-08-21.txt.

Executed as requested shuf -i 1-20 -n 1 returned 15, corresponding to PARQ. The full text, three complete result tables, three-seed statistics, algorithm, open implementation, and theoretical proofs were accessible, so zero redraws.

2. In-depth analysis of the core paper

2.1 Motivation: the real difficulty is that discrete objectives have no usable gradient

Quantized inference restricts weights to a finite set (\mathcal Q), such as 1-bit ({-q,+q}), ternary ({-q,0,+q}), or four symmetric 2-bit values, reducing storage and multiply-add costs. But (f(Q(u))) has zero true derivative with respect to latent variables (u) almost everywhere because staircase quantizer (Q) does. STE treats (dQ/du) as 1 during backpropagation, using (\nabla f(Q(u))) to update full-precision latent variables. It works but leaves two issues: replacing zero with one is not an ordinary approximation, and stability/usefulness of final iterates under biased gradients lacks guarantees directly matching large-network training (page 2, Equations 2–3).

This is not merely theoretical fastidiousness. Tables 1–3 show differences amplified in the most constrained 1-bit setting: PARQ's average gain over STE across six models is 0.615 pp, with mean relative gain 0.91%; at 4 bits, the average shrinks to 0.043 pp. The strongest case is CIFAR-10/ResNet-20 at 1 bit: STE's 89.56% rises to 90.48%, absolute +0.92pp and relative +1.03%. Optimization matters not because more quantization is faster, but because a stable continuous-to-discrete path matters when tight feasible sets make hard projections abrupt.

The novel challenge combines three demands: genuine discrete structure requires nonsmooth regularization; avoiding wrong valleys of W-shaped nonconvex penalties requires convexity; and convergence must concern the final iterate rather than an average, because averaging discrete points generally produces continuous values. PARQ's convex + nonsmooth + last-iterate combination addresses all three.

2.2 Narrative: from challenging STE's gradient interpretation to an executable alternative

The paper first rewrites BinaryConnect as a two-variable update: full-precision (u_t) accumulates gradients computed at quantized weights (w_t=Q(u_t)). Attention shifts from fabricated derivatives to mapping historical gradients into structured parameters. If nonsmooth (\ell_1) kinks induce sparsity, can kinks be placed at every target quantization value? Piecewise-affine regularization (PAR) answers this.

Next, ordinary Prox-SGD scales regularization by the shrinking step size (\eta_t), making soft-threshold dead zones disappear and weakening quantization late in training. AProx instead uses cumulative step size (\gamma_t=\sum_{s=1}^t\eta_s). The proximal map progressively changes from near-identity with slope near 1 to staircase hard quantization.

The third turn bridges theory and engineering: real models cannot hand-tune centers (q_k), slopes (a_k), and (\lambda) per layer. PARQ estimates the target set (Q_t) online from latent weights using LSBQ, while independently scheduling inverse slope (\rho_t^{-1}:1\rightarrow0). This sacrifices exact equivalence to theoretical AProx for an optimizer-only interface requiring no changes to model forward/backward code.

The tradeoffs are clear: full-precision latent variables and online center estimation remain, so the method primarily lowers deployment weight/arithmetic costs rather than QAT training memory. Continuous annealing reduces hard-quantization jumps but introduces start/end/steepness hyperparameters. Theory covers convex losses while experiments use nonconvex networks, connected by evidence rather than a theorem.

2.3 Formal definitions and mechanism analysis

Let the target set be (\mathcal Q={0,\pm q_1,\ldots,\pm q_m}), with (0=q_0<q_1<\cdots<q_m). PAR regularization is defined by Equation 6, page 4:

where (0\le a_0<a_1<\cdots<a_m=+\infty), (b_0=0), and

A maximum of affine functions ensures convexity; slope jumps at (\pm q_k) create nonsmooth breakpoints, allowing optimality conditions to place many coordinates exactly at quantization values. The objective is

Ordinary Prox-SGD uses (\mathrm{prox}_{\eta_t\lambda\Psi}), approaching identity as (\eta_t\to0). AProx replaces it with (Equation 11, page 7):

Under convexity and Lipschitz conditions, Theorem 3.2 gives a last-iterate bound (page 9):

The key is not a dramatically new rate but that it concerns (w_t), not (\bar w_t). Even if each (w_t) is discrete, the average (\bar w_t) often returns to continuous space; last-iterate guarantees therefore match deployable quantized models.

Practical PARQ pseudocode (Algorithm 1, page 10):

Inputs: initial w1, bit count n, learning rate ηt, inverse-slope schedule ρt^{-1}
u1 ← w1
for t = 1 ... T-1:
    ut+1 ← ut - ηt ∇f(wt, zt)
    Qt+1 ← LSBQ(ut+1, n)             # Estimate 2^n quantization values online
    wt+1 ← prox_PARQ(ut+1, Qt+1, ρt) # Gradually change soft maps into hard stairs
return wT

Full-precision latent u ── gradients at quantized weights w ──► Gradient accumulation
       │                                                       │
       └──── LSBQ estimates Q_t online ◄────────────────────────┘
                         │
              PARQ proximal map, ρ^{-1}: 1 → 0
                         │
      Near-full precision (early) → soft clustering (middle) → hard quantization (late)

2.4 Engineering details

CIFAR-10 experiments quantize all ResNet-20/56 weights, including final projection, for 200 epochs: SGD momentum 0.9, weight decay (2\times10^{-4}), initial LR 0.1, reduced 10× at epochs 80/120/150 (page 11). ImageNet/ResNet-50 quantizes residual-block weights per channel, using SGD LR 0.1, momentum 0.9, weight decay (10^{-4}), and 10× LR decay every 30 epochs.

DeiT-Ti/S/B have approximately 5M/22M/86M parameters and train for 300 epochs with AdamW LR (5\times10^{-4}) and weight decay 0.05. The last 20 epochs fix LR at (10^{-8}), explicitly reported as better than default minimum (10^{-5}); PARQ and BinaryRelax are already hard-quantized then. Code examples use 8 GPUs, per-channel LSBQ, and steepness 50 for 2-bit DeiT; ResNet-50 uses steepness 75. No equal-compute training times, per-step LSBQ overhead, or kernel-level inference speeds are reported, so efficiency is demonstrated mainly as feasible low-bit representation rather than end-to-end throughput.

2.5 Experimental design, benchmarks, and statistics

The paper has only three head-to-head QAT methods, so a six-method matrix cannot honestly be created. The complete three-method × six-model results from Tables 1–3 are retained. To limit width, the most informative 1-bit and 2-bit results appear first, followed by aggregates over five bit widths.

The strongest quantitative evidence in the paper is not peak accuracy but the bit-width gradient: PARQ's mean advantage over STE monotonically shrinks from +0.615pp at 1 bit to +0.043pp at 4 bits (recomputed from Tables 1–3). This fits the idea that soft-to-hard paths primarily relieve extreme discrete constraints. PARQ is not universally best: ternary DeiT-Ti scores 60.99%, below BinaryRelax's 62.18% by 1.19 pp; 4-bit DeiT-B scores 80.35%, below BinaryRelax's 80.76% by 0.41 pp.

2.6 Ablations, hyperparameter sensitivity, and phase transitions

The paper does not provide a progressive ablation table for LSBQ → PAR regularization → aggregated step sizes → independent slope schedules, so final gains cannot be assigned quantitatively to individual components. STE/BinaryRelax/PARQ comparisons change regularization geometry and annealing direction, making them method-level rather than single-variable ablations. Figures 10–13 show dynamics: early 2-bit DeiT-Ti PARQ loss tracks FP, rises during rapid slope changes, and approaches STE late. Figure 11 shows proximal maps changing from lines to stairs at epochs 6/150/300; Figure 12 shows PARQ avoiding abrupt accuracy drops seen in STE/BinaryRelax on 1-bit DeiT-S; Figure 13 shows (q_1,q_2) expanding quickly then contracting slowly.

There is a qualitative transition window: near (\rho^{-1}=1), the model approximates full precision; slope changes raise training loss; as (\rho^{-1}\to0), hard quantization begins. Figure 10 lacks machine-readable per-epoch values, and no anneal_start/end/steepness grid is published, preventing precise transition thresholds or CIs. Repository steepness values of 50 for DeiT and 75 for ResNet-50 show architecture-specific tuning, not established robustness ranges.

Tables 1–3 report mean±standard deviation over 3 seeds, without CIs, significance tests, or raw seed values. ResNet-20 1-bit's PARQ–STE gap is 0.92pp, versus standard deviations 0.26/0.18, clearly larger than either individual deviation. DeiT-B ternary's 0.01pp advantage, however, is comparable to 0.01/0.08 deviations and does not establish stable superiority.

2.7 Genuine contributions, reusable designs, and transfer potential

The contribution is not another quantizer, but connecting engineering STEs, optimization's dual averaging/proximal maps, and structural learning's nonsmooth regularization. STE becomes a limit of soft maps rather than an isolated trick, while quantization values, regularization geometry, and annealing paths become inspectable design choices.

The optimizer-only abstraction is most reusable: retain model definitions, forward passes, losses, and backward passes; specify bit counts for parameter groups and replace an optimizer wrapper. Second, estimate discrete centers online while independently scheduling discretization strength: (Q_t) adapts to within-layer distributions, while (\rho_t) controls hardening, no longer entangled in one (\lambda). Third, track last iterates: averaging can destroy intended discrete structure, so theory/logging should follow final points rather than average losses alone.

Transfer opportunities include sparsification, clustered weights, mixed precision, and vector quantization. Structures expressible through nonsmooth breakpoints or grouped proximal maps may reuse AProx. Asymmetric/nonuniform quantization, activations, and LLMs still require fresh validation of compute overhead and nonconvexity.

2.8 Limitations: author-stated boundaries and additional critique

There is no separate Limitations section. The following author-stated boundaries come from explicit textual qualifications, not an author-enumerated risk list.

1. Theoretical scope: Sections 2–3 assume sample losses convex in weights and a Lipschitz objective, whereas §5 trains nonconvex ResNet/DeiT models. The authors state last-iterate guarantees for convex optimization, not deep networks.
2. Approximate higher-bit LSBQ: Page 10 explicitly states that (n>2) lacks closed forms and uses greedy foldable representations. Thus 3/4-bit centers have no global least-squares guarantee.
3. Finite training differs from theoretical cumulative strength: Page 10 acknowledges that theory requires (\gamma_t\to\infty), while finite steps and decaying learning rates yield only a “not very large constant.” A separate practical (\rho) schedule absorbs this discrepancy.

Additional critique:

1. [Argument gap] Theory–experiment disconnect. Theorem 3.2 assumes convex losses, while all main experiments use nonconvex ResNet/DeiT. Last-iterate convergence is therefore a mechanistic inspiration, not a proof of practical superiority. Compare convex logistic regression and small nonconvex networks under the same schedule, recording objective gaps, discretization rates, and final iterates.
2. [Missing key competitors] Only two baselines. Tables 1–3 compare only STE and BinaryRelax, although related work discusses ProxQuant/ProxConnect. It is unclear whether gains over the closest proximal methods come from convex PAR or tuning differences. Add ProxQuant and ProxConnect under identical code, LSBQ, and augmentation.
3. [Experimental design] No component-level ablations. Online LSBQ, aggregate scaling, soft-to-hard slopes, and a final 20 epochs at LR (10^{-8}) coexist, so the +0.615pp 1-bit mean cannot be attributed to convex regularization alone. A 2×2×2 factorial experiment should separate center updates, cumulative step sizes, and annealing.
4. [Failing subtasks] Advantages are not monotonic. Ternary DeiT-Ti's 60.99% trails BinaryRelax's 62.18% by 1.19pp, and 4-bit DeiT-B trails by 0.41pp. Greater stability/better local minima mainly applies to low bits or selected models, not dominance across bit widths.
5. [Metric validity] No measured efficiency. Only Top-1 accuracy is reported, without model bytes, training throughput, inference latency, energy, or dedicated low-bit kernels. n-bit weights do not directly imply end-to-end speed. DeiT embeddings, LayerNorm, and final projections remain FP, so compression is not simply 32/n either.
6. [Statistical evidence] Three seeds cannot establish tiny gaps. DeiT-B ternary gains only +0.01pp and DeiT-S 4-bit +0.04pp, while some standard deviations reach 0.08/0.12. These rankings may reflect noise; paired seeds, 95% CIs, and multiple-comparison corrections are needed.
7. [Reproducibility] Hyperparameter reporting is incomplete. The text allows cosine/sigmoid schedules, but repository examples use steepness 50 for DeiT and 75 for ResNet-50 without sensitivity curves, so architecture transfer still requires search. Public code substantially reduces risk but does not eliminate tuning-budget bias.
8. [External validity] No language models, activation quantization, or deployment. All 30 configurations concern image-classification weight quantization, so results do not extend to LLM activation outliers, KV caches, MoE experts, or accumulated generation errors. A direct next test compares weight-only versus W+A QAT on the same 1B–7B language model.

3. Related-work retrospective: problem and technical lineage

3.1 BinaryConnect — 2015, NeurIPS

What it solved. BinaryConnect uses binary weights in forward/backward passes while retaining full-precision latent weights to accumulate gradients, showing deployable models need not maintain binary training states throughout. It popularized hardware motivation for replacing multiplication with addition/subtraction and the latent–quantized two-variable structure.

Remaining gap. Backpropagation bypasses sign/round's zero derivatives, effectively using STE. Empirical success does not explain why replacing 0 with 1 avoids uncontrolled bias; hard maps from the first step can also cause abrupt optimization changes.

PARQ's response. PARQ retains the two-variable framework but replaces hard (Q(u)) with a progressively steeper proximal map: near FP early, returning to BinaryConnect late. BinaryConnect becomes PARQ's asymptotic endpoint rather than a separately justified heuristic.

Inherited design. Full-precision latents accumulating gradients evaluated at quantized weights carry over unchanged; only the geometry of the latent-to-quantized mapping changes.

3.2 BinaryRelax — 2018, SIAM Journal on Imaging Sciences

What it solved. BinaryRelax avoids imposing discrete constraints immediately, using squared distance to the quantized set (a Moreau envelope) as a continuous relaxation, increasing regularization gradually, and finally switching to exact quantization at small learning rates. This establishes soft-to-hard continuation.

Remaining gap. Squared distance yields smooth, nonconvex W-shaped geometry. Smoothness does not readily pull weights exactly onto discrete points, while nonconvex hills can prevent movement from poor valleys to better ones. The two-stage switch can also be abrupt.

PARQ's response. PARQ inherits continuation but replaces its geometry with convex, nonsmooth PAR: kinks attract weights, while convexity removes incorrect W-shaped valleys. BinaryRelax is a main baseline in Tables 1–3, yet PARQ exceeds it in only 17/30 configurations (recomputed here), so geometric advantages do not automatically improve every model's accuracy.

Inherited design. Gradually increasing regularization toward hard constraints becomes a proximal slope increasing from 1 to infinity; both treat training time as a quantization degree of freedom.

3.3 ProxQuant — 2019, ICLR

What it solved. ProxQuant formulates quantized training as regularized learning, inserting proximal operators between stochastic gradients and relating STE/BinaryConnect to dual averaging. It systematically moves QAT from pseudogradients into proximal-optimization language.

Remaining gap. ProxQuant commonly uses nonconvex distance-to-discrete-set penalties, and ordinary Prox-SGD weakens regularization as LR shrinks. It does not jointly address discrete final iterates and convex regularizers inducing multiple discrete clusters.

PARQ's response. PARQ constructs convex PAR with multiple kinks and scales it by cumulative rather than current step sizes, preventing late-stage quantization from disappearing. Theory advances from average/function-value convergence to last iterates.

Inherited design. The prox-between-gradients structure stays; the regularizer, scaling, and target guarantee change.

3.4 ProxConnect — 2021, NeurIPS

What it solved. ProxConnect further identifies BinaryConnect as a special case of dual averaging, proposes easy-to-design/analyze proximal quantizers, and supplies convergence results, unifying BinaryConnect, PTQ, and other quantization algorithms.

Remaining gap. PARQ page 3 explicitly notes that ProxConnect's convex-regularization results lack a concrete convex construction inducing multivalue quantization and mainly guarantee average iterates, which need not stay quantized.

PARQ's response. AProx is algorithmically equivalent to ProxConnect, but PARQ adds a concrete convex nonsmooth PAR and last-iterate bounds of the same order as average-iterate bounds.

Inherited design. Dual-averaging/aggregate-gradient interpretations feed directly into AProx; PARQ turns an abstract proximal family into an online-center-estimating implementation for ResNet/DeiT.

4. Technical evolution

2015 BinaryConnect [hard-quantized forward passes; FP latents; STE]
        │
        ├── 2018 BinaryRelax [continuous relaxation; nonconvex smooth distance; two stages]
        │          │
        │          └── Soft-to-hard training, but W-shaped valleys and abrupt switching
        │
        └── 2019 ProxQuant [regularization + proximal gradients]
                   │
                   └── 2021 ProxConnect [dual-averaging unification; proximal family; average-iterate guarantees]
                                  │
                                  ├── No concrete convex multivalue quantization regularizer
                                  └── Averages of quantized points need not be quantized
                                           │
2025 PARQ [convex+nonsmooth PAR; AProx; last iterates; online LSBQ; soft→hard schedules]

The driver shifts from bypassing nondifferentiability to designing optimization geometry. Early work asked whether binary weights could train; later work relaxed hard constraints, then unified STE with proximal/dual averaging. PARQ focuses on local kinks, global convexity, and final-iterate structure. Attention expands beyond accuracy to stability, guarantees, and integration, but real low-bit kernels and end-to-end latency remain unfinished.

5. Open challenges and actionable research opportunities

5.1 Theory

1. Nonconvex last-iterate bounds. Under matched networks/noise assumptions, derive final-iterate gradient-mapping or local-stability bounds for AProx, ProxConnect, and STE. Measure objective gaps, discretization rates, and basin switches to test whether theory predicts accuracy.
2. Coupled analysis of online (Q_t). Theory fixes (\Psi), while practice updates (Q_t) with LSBQ every step. Compare frozen Q, epoch-wise Q updates, and step-wise updates on fixed models, studying bounds linking drift speed and convergence error.
3. Necessity of multivalue convex nonsmooth regularization. Under identical proximal kernels/schedules, compare convex PAR, nonconvex L1 distance, and smooth L2 distance. Measure incorrect-valley visits, Hessian spectra, and final discretization rates to isolate convexity and nonsmoothness causally.

5.2 Engineering and Applications

1. Equal-GPU-hour QAT. On fixed 8×H100 hardware, batches, and wall-clock time, compare STE, BinaryRelax, ProxQuant, and PARQ on images/s, extra memory, LSBQ time share, and final accuracy, addressing the current accuracy-only evidence gap.
2. Real-inference Pareto curves. Export INT1/T/INT2 models to one backend and report model size, peak memory, p50/p99 latency, throughput, and energy. Account separately for FP DeiT layers rather than equating theoretical bit width with actual compression.
3. Fair annealing-schedule comparisons. With fixed training steps and tuning budgets, compare cosine, sigmoid, linear, and validation-triggered schedules head-to-head, reporting 10 seeds and loss spikes during annealing.
4. LLM scaling. On identical 1B/7B models and token budgets, compare PARQ, STE, and GPTQ-style QAT for W1/W2-only and W2A8. Report perplexity, zero-shot accuracy, generation stability, and tokens/s.

5.3 Emerging Directions

1. Adaptive quantization for MoE experts. Compare shared (Q), expert-specific (Q_e), and routing-frequency-grouped (Q_g) with fixed total bit budgets, testing whether sparse samples cause center drift in long-tail experts.
2. Time-varying activation/KV-cache quantization. Extend PAR breakpoints to token/channel-conditioned forms and compare with static per-channel quantization at equal context length, batch, and memory budgets, focusing on outlier-token error propagation.
3. Combining structures. At equal compression, compare PARQ only, structured sparsity only, and PARQ+sparsity, testing competition between nonsmooth structural regularizers. Report actual kernel speedups, not just theoretical FLOPs.

6. Other recent work worth watching

1. GPTAQ(2025.07, ICML 2025): Uses asymmetric calibration against full-precision layer outputs to reduce accumulated interlayer quantization errors, complementing PARQ's training-time geometry.
2. KVTuner(2025.07, ICML 2025): Assigns mixed-precision KV caches by layer sensitivity, representing bit budgets as learnable system resources.
3. BlockDialect(2025.07, ICML 2025): Fine-grained mixed formats for all LLM matrix-multiplication paths; the project reports accuracy advantages over MXFP4, motivating hardware-format/QAT coupling.
4. RWKVQuant(2025.07, ICML 2025): Proxy-guided PTQ handles RWKV nonlinear operators that obstruct smoothing/rotation fusion, showing quantization must respect architectural operators.
5. Q-VDiT(2025.07, ICML 2025): Token-aware error compensation and distillation for video DiTs, focusing on temporal quantization errors.
6. xLSTM 7B(2025.07, ICML 2025): Uses recurrent architecture rather than compression for efficient long generation, suitable for equal-quality throughput comparisons with low-bit Transformers.
7. Learned Prefix Caching(2025.12, NeurIPS 2025): Learns session continuation to guide prefix-cache eviction, reporting 18–47% cache reduction at equal hit rates and +11% simulated prefill throughput.

7. Conclusion

PARQ turns QAT heuristics into three inspectable objects: target centers, structure-inducing regularizers, and time-dependent annealing paths. Recomputed Tables 1–3 are strongest at 1–2 bits: 1-bit gains average +0.615pp over STE across six models, while 2-bit wins against the strongest per-model baseline in 5/6 models. It does not win every bit width or report real latency, energy, or LLM results. The precise conclusion is: PARQ offers a more principled and more robust training path at extremely low bit widths, but has not established faster end-to-end systems or universal cross-architecture quantization.

Data provenance: core formulas, algorithms, setups, and results come from PARQ arXiv PDF pages 2–13, Tables 1–3, Algorithm 1, and Figures 10–13. All derived means, absolute pp, and relative percentages are recomputed from original tables. Unreported component ablations, hyperparameter grids, CIs, inference latency, and energy are not invented. HTML passed Warm Editorial structural, dependency, semantic, and placeholder checks (0 errors / 0 warnings). The runtime supplied neither /usr/bin/chromium, /opt/pw-browsers nor another executable Chromium, and browser installation was not run under the constraints, preventing Playwright light/dark/mobile screenshot verification. Responsive, print, and theme-switch rules remain, with this validation gap explicitly recorded.