Diffusion Large Language Models (dLLMs) break the rigid left-to-right constraint of traditional LLMs, enabling token generation in arbitrary orders. Intuitively, this flexibility implies a solution space that strictly supersets the fixed autoregressive trajectory, theoretically unlocking superior reasoning potential.
However, we find that for general reasoning tasks (e.g., mathematics and coding), arbitrary-order generation may in fact limit the reasoning potential of dLLMs. We observe that dLLMs tend to exploit this order flexibility to bypass high-uncertainty tokens that are crucial for exploration, which can lead to a premature collapse of solution coverage. This observation motivates a rethink of RL approaches for dLLMs, where considerable complexities—such as handling combinatorial trajectories and intractable likelihoods—are often devoted to preserving this flexibility.
We show that effective reasoning can be elicited by simply forgoing arbitrary order and applying standard Group Relative Policy Optimization (GRPO) instead. Our approach, JustGRPO, is minimalist yet surprisingly effective (e.g., 89.1% accuracy on GSM8K) while fully retaining the parallel decoding ability of dLLMs.
Central to the appeal of dLLMs is their theoretical flexibility: the capability for arbitrary-order generation. Theoretically, this unconstrained generation order constitutes a superset of the fixed autoregressive trajectory, and prior work has shown its benefits on specific tasks such as sudoku and zebra puzzles. Driven by this promise, the community has increasingly adopted RL to elicit similar advantages for general reasoning tasks like mathematics and coding.
We challenge this intuition. Our findings reveal that, for these general reasoning tasks, arbitrary-order generation may in fact narrow rather than expand the reasoning potential elicitable by RL.
To rigorously assess reasoning potential, we employ Pass@k—a standard proxy for the solution space coverage and the achievable reasoning upper bound under RL. We compare two decoding modes—Arbitrary Order (standard diffusion decoding with low-confidence remasking) and AR Order (strictly left-to-right decoding)—across three dLLMs (LLaDA-Instruct, Dream-Instruct, and LLaDA 1.5) and four benchmarks.
Reasoning potential measured by Pass@k. While arbitrary order is competitive in single-shot settings (k=1), it exhibits notably flatter scaling curves than AR Order. As k increases, AR mode uncovers more correct solutions—consistently across the three dLLMs and four benchmarks.
One might hypothesize that arbitrary order explores a different solution space, albeit less efficiently. We test this by analyzing solution coverage at k=1024. The result reveals a stark reality: the reasoning traces generated by arbitrary order are largely a strict subset of those generated by AR.
Solution space coverage measured by Pass@1024. Solutions found by arbitrary order (AO) are largely a subset of those found by AR. On HumanEval, AR solves 21.3% of problems that arbitrary order misses, whereas the reverse is only 0.6%. The flexible decoding process rarely unlocks genuinely new solutions.
Why does the theoretically superior solution space of dLLMs collapse in practice? We attribute this phenomenon to how these two modes handle uncertainty.
AR order constrains the model to strictly resolve the left-most unresolved token at each step, forcing the model to confront uncertainty as it arises. By sampling exactly at the fork, the model commits to a specific rationale, thereby preserving the exploration space.
Arbitrary order adaptively selects tokens to update based on model confidence, preferentially generating "easy" tokens with high certainty while bypassing "hard" ones. By the time the model returns to fill in the bypassed forks, the established bidirectional context has already severely constrained the potential branches.
Inspecting the frequently bypassed tokens reveals a clear pattern: the diffusion sampler disproportionately defers logical connectives and transition markers such as "Therefore", "Thus", and "Since". Prior work has shown that such tokens often function as "reasoning sparks" or "logical forks" that determine subsequent reasoning directions. Keeping these tokens in a high-entropy state is critical for effective exploration.
Left: Frequently bypassed tokens are typically logical connectors and transition words. Right: While the global average token entropy remains comparable (dashed lines), the entropy at logical forks drops significantly in arbitrary order (blue bars). We term this entropy degradation.
Key Insight: The flexibility of arbitrary order serves as a mechanism for inference-time exploitation rather than reasoning exploration. By bypassing high-uncertainty tokens, the model trades the exploration of diverse reasoning paths for greedy optimization of local consistency.
Our findings suggest that, for general reasoning tasks, arbitrary order can limit the reasoning potential accessible to RL. Despite this, current RL methods for dLLMs remain heavily burdened by the need to preserve this flexibility—grappling with combinatorial trajectories, intractable marginal likelihoods, and sampler-learner mismatches.
We propose a return to simplicity: since pure autoregressive order yields better reasoning potential, we explicitly forgo arbitrary-order generation during RL training. This transforms the dLLM from a chaotic sequence denoiser into a well-defined autoregressive policy.
To obtain the probability of the next token ot given history o<t, we construct an input state where the past is observed and the future is masked:
x̃t = [o1, ..., ot-1, [MASK], ..., [MASK]]
The autoregressive policy is then defined as:
πARθ(ot | o<t, q) = Softmax(fθ(x̃t))t
This formulation enables the direct application of standard GRPO to diffusion language models—no trajectory approximations or marginal likelihood estimation needed.
We evaluate JustGRPO on LLaDA-Instruct across four standard benchmarks (GSM8K, MATH-500, HumanEval, and MBPP) at three generation lengths. Despite using just standard GRPO with no diffusion-specific adaptations, it compares favorably with methods specifically designed for dLLMs.
| Model / Seq Len | GSM8K | MATH-500 | HumanEval | MBPP | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 128 | 256 | 512 | 128 | 256 | 512 | 128 | 256 | 512 | 128 | 256 | 512 | |
| d1 | 73.2 | 81.1 | 82.1 | 33.8 | 38.6 | 40.2 | – | – | – | – | – | – |
| LLaDOU†| – | 88.1 | – | – | 41.1 | – | – | 59.1 | – | – | 51.6 | – |
| LLaDA-1.5‡ | – | 83.3 | – | – | – | – | 29.3 | 39.6 | 51.9 | 39.6 | 39.9 | 38.8 |
| wd1 | 77.2 | 80.8 | 82.3 | 33.3 | 34.4 | 39.0 | – | – | – | – | – | – |
| d-TreeRPO | – | 81.2 | 82.6 | – | 37.7 | 38.9 | – | – | – | – | – | – |
| ESPO | 80.0 | 82.3 | 83.7 | 36.0 | 39.0 | 43.4 | 28.1 | 42.1 | 50.0 | 47.4 | 44.6 | 44.2 |
| GDPO | 78.4 | 82.8 | 84.5 | 33.2 | 39.6 | 41.4 | 26.2 | 39.6 | 39.0 | 43.6 | 50.6 | 47.1 |
| SPG | 78.5 | 86.1 | 84.5 | 33.4 | 40.0 | 41.8 | – | – | – | – | – | – |
| JustGRPO | 83.8 | 89.1 | 89.8 | 39.0 | 45.1 | 45.2 | 37.8 | 49.4 | 48.7 | 50.6 | 52.4 | 49.0 |
System-level comparison on LLaDA-Instruct; baseline numbers are mostly quoted from their original papers under heterogeneous settings (JustGRPO uses full fine-tuning and unmasks one token per step). †LLaDOU modifies the base architecture with an auxiliary module. ‡ LLaDA-1.5 is trained on a privately collected dataset at a significantly larger scale.
A natural concern: does training with AR constraints compromise the model's inherent parallel decoding capabilities? We evaluate inference performance under varying degrees of parallelism using the training-free EB-Sampler.
JustGRPO preserves parallel decoding. Using the training-free EB-Sampler, accuracy gains become more pronounced as parallelism increases—on MBPP, the gap widens from +10.6% (1 token/step) to +25.5% (~5 tokens/step). The AR-trained model learns a more robust distribution that is resilient to the approximation errors inherent in parallel sampling.
The intuitive appeal of diffusion language models lies in their order arbitrariness, often perceived as a promising mechanism for navigating complex reasoning paths. However, our study suggests that for general reasoning tasks such as mathematics and coding, this unrestricted flexibility may inadvertently constrain the reasoning potential—arbitrary-order generation appears to prioritize refining a single trajectory over broader solution coverage.
This observation points toward a simpler recipe for eliciting the reasoning capability of dLLMs. By aligning the model with a standard autoregressive objective, we enable the direct application of Group Relative Policy Optimization (GRPO) without any complex adaptations tailored for order arbitrariness. Despite its simplicity, this approach—JustGRPO—yields notable improvements in reasoning while fully preserving the parallel decoding capabilities of dLLMs at inference. By returning to the basic left-to-right order for alignment, we hope to encourage a reconsideration of the trade-offs between arbitrary and AR order in the development of next-generation diffusion models.
@inproceedings{ni2026flexibility,
title = {The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models},
author = {Ni, Zanlin and Wang, Shenzhi and Yue, Yang and Yu, Tianyu and Zhao, Weilin and Hua, Yeguo and Chen, Tianyi and Song, Jun and Yu, Cheng and Zheng, Bo and Huang, Gao},
booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
year = {2026},
}