NVIDIA Nemotron Labs Dory 1.0 4B: The Looped 4B With a Runtime Compute Dial (OpenMDW 1.1, variable recurrent iterations 1–4, and one-loop KV footprint)
Inside NVIDIA's unannounced Dory 1.0 4B: 8+20x+8 looped nGPT architecture, native NVFP4 pretraining, OpenMDW 1.1 license, and the last-loop KV cache caveat.

(Previous) Claude AI + Higgsfield AI: Native Video Generation, MCP Connector Architecture, Multi-Model Orchestration, and the higgsfield-ai/skills Toolkit | (Next) Harnessing Reasoning Models in Agentic Workflows: Integrating o1/R1, Hidden Thinking Tokens, and Cost vs. Depth Optimization
Summary
NVIDIA’s quietest model release of 2026 is a 4-billion-parameter dense checkpoint that arrived on Hugging Face overnight on October 15–16 stamped GA (10/16/2026)—with zero blog post, zero press release, and not even a passing mention in NVIDIA’s own October 21 open-models roundup eight days later. Named NVIDIA Nemotron Labs Dory 1.0 4B, the model introduces a runtime compute dial that asks the test-time compute question from the opposite side of standard chain-of-thought reasoning models: what if test-time scaling happens vertically in latent space rather than horizontally in generated tokens?
Instead of locking every token into a fixed layer count or emitting thousands of verbose scratchpad tokens into the Key-Value (KV) cache, Dory divides a normalized-GPT (nGPT) backbone with interleaved sliding-window attention (SWA) and full attention into three plain stages: 8 input layers, a shared 20-layer recurrent block applied for 1, 2, or 4 loops at inference time (with 2 loops as the recommended default and the original input representation reinjected at every loop), and 8 output layers. One checkpoint serves three distinct compute budgets (36, 56, or 96 effective layers) with no retraining and no model swap.
Just as notably, Dory was pre-trained across 6 trillion tokens with the first ~4.5 trillion tokens trained natively in NVFP4 before finishing in BF16—landing the exact same week we examined NVFP4 serving kernels in vLLM v0.32.0. Below, we break down the primary-sourced architecture and benchmark numbers from NVIDIA’s model card, independent community verifications on Apple Silicon (NetProjects, Hack-Log, OrcaRouter, modelfit.io), the config-dependent 2.7x KV-cache trap discovered in real-world testing, and the honest caveats every engineer needs to know before compiling the fork.
Figure 1: Side-by-side specification scoreboard from OrcaRouter’s October 17 evaluation summarizing Dory 1.0 4B (4B dense, text only, English only, 256K context, OpenMDW 1.1, runtime loop count 1–4) versus a fixed-depth model (Gemma 4 12B) and highlighting that no shared-harness benchmark result exists yet. Source: OrcaRouter.
Hugging Face / Official Model Card Summary
The following table summarizes the verified primary specifications across the four official Hugging Face checkpoints published by NVIDIA:
| Specification | Official Primary Details & Checkpoints |
|---|---|
| Model Checkpoints | • BF16 (W16A16): nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16• MXFP8 (W8A16): nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8• NVFP4 (W4A16): nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4• GGUF ( Q5_K_M): nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF (3,673,288,864 bytes, SHA-256: 96080be4...660e0) |
| Total Parameters & Modality | 4B Dense • Text-only • English-only (plus 43 programming languages in pretraining corpus) |
| Backbone Architecture | Looped (latent-recurrent) normalized-GPT (nGPT) with interleaved sliding-window attention (SWA) and full-attention layers |
| Layer Topology & Compute Dial | 8 Input Layers + 20 Shared Recurrent Layers (looped 1, 2, 3, or 4 times at runtime; 2 loops default; input reinjected every loop) + 8 Output Layers (Effective depth: 36 / 56 / 76 / 96 layers) |
| Context Window | Up to 256K tokens (following a 300B-token context-extension phase) |
| Pretraining & RL Pipeline | 6T tokens via Megatron-LM (first ~4.5T natively in NVFP4, remainder in BF16) → 300B tokens context extension → SFT → asynchronous multi-environment GRPO RL via NeMo RL & NeMo Gym |
| KV Cache Modes | per_loop (default in sample launch snippet; scales KV memory with loop count) vs. last_loop (overwrites previous loop’s KV entries in-place for a flat 1-loop KV footprint) |
| Release Version & Hardware | Stamped GA (10/16/2026) • Preferred OS: Linux • Microarchitectures: NVIDIA Hopper (H100), Blackwell (B200, B300, DGX Spark) • macOS ARM64 via pinned llama.cpp contributor fork or community MLX builds |
| License | OpenMDW License Agreement 1.1 (raw license text) — grants rights to “Model Materials” (architecture + parameters) “without restriction” |
Prior Reading Material
To connect Dory’s native NVFP4 pretraining, recurrent KV cache mechanics, and open-weights licensing to the rest of our architecture coverage, review these prerequisite deep dives:
- NVIDIA Nemotron 3.5 Bolt E4B 4GB: Fast Conversational Edge Agents, Pure RL Post-Training, and DFlash Parallel Speculative Decoding — Our deep dive into NVIDIA’s production 4B edge conversational agent (pure RLVR/RLHF post-training on
Gemma-4-E4B-it, 80% verbosity reduction, and 7-token DFlash parallel speculative decoding). - vLLM v0.32.0: /v1/systemone Structured Decisions, NVFP4 KV Cache Everywhere, Model Runner V2, and Multi-API Worker Scaling — How modern inference engines exploit NVFP4 precision for KV caches and model weights on Hopper and Blackwell GPUs.
- DeepSeek V4.1-Flash-1017: MIT Weights, DeepInfer Engine, Agent Harness Tuning, and the Full-Stack Search/Fetch API Play — How harness-tuned agentic checkpoints optimize SWE-bench Verified and Terminal-Bench performance.
- Meta Muse Spark Open: Inside the 816B Multimodal Mixture-of-Experts Architecture, NVFP4 Quantization, and Open-Weights Agentic Inference — Contrasting frontier MoE parameter scaling and custom model licenses with compact dense architectures.
- NVIDIA Nemotron 3.5 Super VL: 120B MoE Multimodal Architecture, 600+ TPS Agentic Inference, and Enterprise Customization — Our baseline look at NVIDIA’s official Nemotron 3.5 lineup and NeMo RL post-training stack.
- Understanding the KV Cache: The VRAM Bottleneck of LLM Serving — Foundational mechanics of Key-Value tensor growth across layers and sequence lengths.
1. The Midnight GA: Why Ship a Looped Architecture in Silence?
Before looking at benchmarks or code, picture how a standard transformer handles everyday traffic.
Imagine driving a car built with a single, welded fixed gear. Whether you are creeping through a parking lot at 5 mph or climbing a steep mountain pass, the transmission cannot shift. In a standard dense transformer, layer depth works the exact same way: every generated token—whether predicting a trivial JSON comma or resolving a multi-file Python import cycle—must pass through the exact same fixed stack of layers.
flowchart TD
S1["Standard Fixed-Depth Transformer (Static Layer Stack)"] --> S2["Every Token Traverses Identical Physical Layers Once"]
S2 --> S3["To Think Harder: Must Emit Hundreds of Verbal Scratchpad Tokens"]
S3 --> S4["Bottleneck: Output Token Latency + Linear KV Cache Expansion"]
style S1 fill:#0f172a,stroke:#ff6b6b,stroke-width:2px,color:#ffffff
style S2 fill:#1e293b,stroke:#ff9f43,stroke-width:2px,color:#ffffff
style S3 fill:#2d1b2e,stroke:#ff6b6b,stroke-width:2px,color:#ffffff
style S4 fill:#3b111a,stroke:#ff4757,stroke-width:2px,color:#ffffff
Now imagine replacing that welded single gear with a multi-speed gearbox inside the model’s internal workspace. Instead of emitting five hundred extra words out loud just to “think,” the model routes the token’s internal hidden representation through a shared middle core 1, 2, or 4 times before emitting a single word:
flowchart TD
D1["NVIDIA Dory 1.0 4B (8 Input + 20 Shared Recurrent + 8 Output Layers)"] --> D2["Runtime Compute Dial: Loop Shared 20-Layer Block 1, 2, or 4 Times"]
D2 --> D3["Deliberation Happens Vertically in Latent Space Per Token"]
D3 --> D4["Result: 40%–60% Fewer Output Tokens Needed + Flat 1-Loop KV Footprint (in last_loop mode)"]
style D1 fill:#0d2b45,stroke:#00e5ff,stroke-width:2px,color:#ffffff
style D2 fill:#0f382c,stroke:#76b900,stroke-width:2px,color:#ffffff
style D3 fill:#1a3d3c,stroke:#00e5ff,stroke-width:2px,color:#ffffff
style D4 fill:#0f382c,stroke:#00ff88,stroke-width:2px,color:#ffffff
Normally, when a major AI lab ships a brand-new architectural family—especially its first looped normalized-GPT model and a flagship checkpoint pre-trained natively in NVFP4—it accompanies the release with a launch blog, technical report, and developer fanfare.
With Dory 1.0 4B, NVIDIA did the exact opposite:
- Overnight Repository Drop (Oct 15–16, 2026): Four checkpoints (
BF16,MXFP8,NVFP4, andGGUF) quietly appeared under thenvidiaHugging Face organization with the model version stampedGA (10/16/2026). - Eight Days of Official Silence: As of October 23 evening, NVIDIA has published no blog post and no press announcement for Dory.
- Absent from the Product Roster & Oct 21 Roundup: As noted by independent researchers at
quidproquo.cc, Dory does not appear on NVIDIA’s official Nemotron developer page (which lists Bolt, Nano, Lightning, Super, and Ultra). Even more tellingly, when NVIDIA published its broad October 21 “expands open models” announcement covering Nemotron 3.5 Super VL, Bolt, Kumo, Cosmos 3.5, Alpamayo 2 Edge, BioNeMo, and Ising, Dory was omitted entirely.
Figure 2: The nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 Hugging Face repository captured shortly after its silent October 16, 2026 drop—showing zero likes, zero deployed inference providers, and an official GA release date stamp. Source: Hugging Face / NVIDIA via OrcaRouter.
Why did NVIDIA ship a GA-stamped looped transformer like a midnight commit? There is no public evidence pointing to whether Dory is an early preview from the Nemotron Labs research group or a staging artifact released ahead of upstream runtime merges. What we do have are the weights, the primary model card, and a week of independent community testing.
2. The Compute Dial, Explained: How the 8 + 20×Loops + 8 Stack Works in Plain English
The Three-Stage Looped Stack (And Why It Reinjects the Input)
Unlike older weight-tied transformers that naively loop every layer from bottom to top, Dory splits its 4B normalized-GPT (nGPT) backbone into three simple, human-readable stages:
Figure 3: Official Dory 1.0 4B (Looped LM) architecture diagram. The layer stack is divided into an 8-layer Input block (I), a shared 20-layer Recurrent block (R) that runs multiple times while re-adding the original input (h^in) on every pass, and an 8-layer Output block (O). Source: NVIDIA / Hugging Face.
Here is what happens inside Figure 3 in plain English—no equations required:
- Stage 1 — The Input Block (
I, 8 Layers): When a token enters the model, the first 8 layers digest the raw embeddings and create a clean “working summary” of the input. - Stage 2 — The Shared Recurrent Block (
R, 20 Layers, Looped 1, 2, or 4 Times): Next, the token enters a shared 20-layer middle block. Depending on how you set the runtime compute dial, the token loops through these exact same 20 layers 1, 2, or 4 times. Crucially, at the start of every single loop, Dory adds the original 8-layer input summary (h^in) back into the mix. Think of this like a writer re-reading the original prompt before every draft revision: without reinjecting that original input on each pass, looping through the same 20 layers repeatedly would cause the model’s internal thoughts to drift further and further away from what you actually asked. - Stage 3 — The Output Block (
O, 8 Layers): Once the final loop finishes, the refined representation passes through 8 dedicated output layers to predict the next token.
Across all three blocks, Dory alternates between sliding-window attention (SWA) layers (which look at nearby tokens cheaply) and full-attention layers (which look across the entire prompt up to 256K tokens). Because the middle 20 layers are reused across loops, a single 4B download lets you dial the effective depth on the fly:
- 1 Loop:
8 input + 20 recurrent + 8 output= 36 effective layers (fastest speed, lowest compute per token). - 2 Loops (Recommended Default):
8 input + (20 × 2) recurrent + 8 output= 56 effective layers (NVIDIA’s sweet spot for everyday reasoning). - 4 Loops:
8 input + (20 × 4) recurrent + 8 output= 96 effective layers (maximum deliberation for coding and terminal agents).
Native NVFP4 Pretraining: The Training-Side Twin to vLLM v0.32.0
Look beneath the layer diagram on the model card and a second milestone jumps out: Dory was pre-trained on 6 trillion tokens in Megatron-LM, with the first ~4.5 trillion tokens trained natively in NVFP4 (NVIDIA’s 4-bit floating-point format) before finishing the final ~1.5 trillion tokens in BF16, followed by a 300B-token context-extension phase to 256K.
Earlier this week in our vLLM v0.32.0 deep dive (Post 0149), we covered how NVFP4 has become the flagship serving format on Hopper and Blackwell GPUs. Dory proves the training side of that story: because its normalized-GPT (nGPT) backbone keeps internal activations strictly normalized, it can train across 4.5 trillion tokens directly in 4-bit precision without collapsing.
On NVIDIA’s DGX Spark hardware, running the quantized checkpoints delivers substantial single-batch speedups over the BF16 baseline:
| Input Sequence Length (ISL) | Output Sequence Length (OSL) | Batch Size | MXFP8 (W8A16) Speedup vs. BF16 | NVFP4 (W4A16) Speedup vs. BF16 |
|---|---|---|---|---|
| 2,560 tokens | 300 tokens | 1 | 1.69x | 2.17x |
| 128 tokens | 1,024 tokens | 1 | 1.84x | 2.44x |
Token Efficiency: Thinking in Latent Space Instead of Output Tokens
When a conventional reasoning model gets stuck on a coding bug, it thinks by generating hundreds of extra words out loud. When Dory loops in latent space, each token already carries deeper internal work—meaning the model solves the same tasks in far fewer generated output tokens:
Figure 4: Output tokens generated per attempt at 2 loops (k=2) and 4 loops (k=4) relative to 1 loop (k=1) across four agentic benchmarks. Source: NVIDIA / Hugging Face.
As Figure 4 shows, turning the dial from 1 loop to 4 loops slashes the number of generated output tokens by:
- 50% on Terminal-Bench 2.1 (
1.00xat 1 loop →0.72xat 2 loops →0.50xat 4 loops). - 60% on GDPval-AA-V2 (
1.00xat 1 loop →0.66xat 2 loops →0.40xat 4 loops). - 26% on SWE-bench Verified (
1.00xat 1 loop →0.79xat 2 loops →0.74xat 4 loops). - 23% on τ³-Banking (
1.00xat 1 loop →0.80xat 2 loops →0.77xat 4 loops).
Last-Loop KV Caching — And Hack-Log’s 2.7x Real-World Caveat
In a standard looped transformer, even though the 20 recurrent layers reuse the same weights on every pass, the inference server still saves a separate Key-Value (KV) memory cache for every loop (per_loop). At 4 loops, your memory footprint balloons as if you were running an 80-layer middle block.
To prevent this, Dory supports last-loop KV caching (last_loop): each new loop simply overwrites the previous loop’s KV cache entries in place, so memory only ever holds a single loop’s worth of KV cache.
Figure 5: Visual comparison between horizontal token-space chain-of-thought scaling (left) and Dory 1.0 4B’s vertical latent-space loop scaling with in-place last-loop KV cache overwriting (right).
However, independent developer Hack-Log uncovered a critical real-world catch while testing Dory on a 16GB Apple M4 Mac: last-loop KV caching is config-dependent, not automatic.
Figure 6: Illustration from Hack-Log’s 16GB M4 Mac evaluation contrasting the default per_loop KV cache expansion (orange) against the flat last_loop KV cache footprint (cyan). Source: Hack-Log / note.com.
If you copy-paste the main llama-server launch command from NVIDIA’s Hugging Face model card, its --override-kv setting actually defaults to per_loop:
# Default snippet on the card — grows KV memory with every added loop!
--override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop"
When Hack-Log benchmarked a 32,768-token context (f16 KV cache) on a 16GB M4 Mac using that default per_loop setting versus changing that single word to last_loop, the measured memory difference was dramatic:
| Loop Count | Default per_loop KV Buffer (phys_footprint) | Corrected last_loop KV Buffer (phys_footprint) | 16GB M4 Generation Speed (last_loop) |
|---|---|---|---|
| 1 Loop | 1,314 MiB (1,445 MB) | 1,314 MiB (1,445 MB) | 6.98 tokens/s (prompt: 66.3 tok/s) |
| 2 Loops | 2,044 MiB (2,206 MB) | 1,314 MiB (1,406 MB) | 4.36 tokens/s |
| 4 Loops | 3,504 MiB (3,680 MB — 2.7x bloat!) | 1,314 MiB (1,465 MB — Flat!) | 2.66 tokens/s (prompt: 24.3 tok/s) |
Table: Empirical KV buffer and macOS footprint measurements on a 16GB Apple Silicon M4 Mac (Q5_K_M GGUF, 32K context). Source: Hack-Log.
To actually get the flat one-loop memory footprint advertised in the highlights, you must explicitly pass last_loop:
# Fixed one-loop KV footprint across all loop counts:
--override-kv "dory.num_loops=int:4,dory.recurrent_kv_cache_mode=str:last_loop"
3. What the Dial Buys: Primary Benchmark Analysis & Model Profile
All benchmark numbers currently available for Dory 1.0 4B come from NVIDIA’s internal evaluation harness (NeMo Gym / NeMo Evaluator SDK, using the Terminus-2 harness for Terminal-Bench 2.1). Let us examine the composite trajectory first, followed by the full task-by-task breakdown:
Figure 7: NVIDIA’s internal Artificial Analysis (AA v4.3.2) composite estimate across 1, 2, and 4 loops (k=1, 2, 4) compared against reference open-weights models. Note: NVIDIA notes these are internal harness estimates that may deviate from official Artificial Analysis scores. Source: NVIDIA / Hugging Face.
Complete Primary Benchmark Table (NVIDIA Hugging Face Card)
| Task / Benchmark | Dory 1 Loop | Dory 2 Loops | Dory 4 Loops | Qwen3.5-4B | Qwen3.5-9B | Gemma 4 26B A4B | Qwen3.6-35B-A3B |
|---|---|---|---|---|---|---|---|
| Coding & Agentic | |||||||
| SciCode | 27.3 | 34.94 | 35.11 | 25.82 | 28.21 | 40.32 | 37.24 |
| SWE-bench Verified | 50.3 | 59.5 | 61.8 | 54.7 | 61.0 | 57.40 | 73.4‡ |
| Terminal-Bench 2.1 | 12.5 | 18.1 | 22.2 | 15 | 24 | 35 | 43 |
| τ³-bench (Banking) | 11.6 | 12.89 | 14.69 | 5.03 | 7.22 | 13.66 | 9.54 |
| GDPval-AA-V2 (Elo) | 815 | 946 | 967 | 424 | 546 | 751.6 | 987 |
| Reasoning & Knowledge | |||||||
| GPQA Diamond (no tools) | 69.19 | 72.41 | 72.16 | 77.71 | 80.93 | 80.43 | 85.35 |
| HLE (text, no tools) | 8.25 | 10.89 | 10.43 | 11.31 | 15.43 | 17.79 | 19.18 |
| AA-Omniscience (accuracy) | 10.03 | 10.08 | 10.03 | 15.02 | 18.05 | 21.13 | 19.95 |
| Instruction Following & Long Context | |||||||
| IFBench | 74.29 | 77.94 | 79.0 | 53.83 | 59.0 | 77.89 | 57.44 |
| AA-LCR (Long Context) | 63 | 71 | 73 | 64.7 | 68.4 | 64.6 | 68.5 |
Footnote (‡): The 73.4‡ SWE-bench Verified score sits on Qwen3.6-35B-A3B as an external reported score NVIDIA did not re-run. (Cautionary note: secondary site quidproquo.cc accidentally misaligned the HTML table columns from the model card, shifting Qwen3.6-35B-A3B’s 73.4 / 43 / 85.35 scores into Dory’s 4-loop column and misreading IFBench—always reference the primary table above.)
Three Engineering Takeaways from the Numbers
-
“A Local Coding Agent and Tool-Calling Worker, Not a Pocket Encyclopedia”: As developer alanwest observed on
dev.to, Dory’s benchmark fingerprint splits cleanly down the middle. On agentic, instruction-following, and long-context workflows, Dory dominates its size class:- On IFBench, even 1-loop Dory (
74.29) beatsQwen3.5-4B(53.83) by +20.46 points, and 4-loop Dory (79.0) beats every model in the table includingGemma 4 26B A4B(77.89) andQwen3.6-35B-A3B(57.44). - On AA-LCR (long-context reasoning), Dory scales from
63(1 loop) to73(4 loops), outscoringQwen3.5-9B(68.4) andQwen3.6-35B-A3B(68.5). - On SWE-bench Verified, dialing from 1 loop (
50.3) to 2 loops (59.5) and 4 loops (61.8) pushes a 4B model pastQwen3.5-4B(54.7),Gemma 4 26B A4B(57.40), andQwen3.5-9B(61.0). - On τ³-bench Banking (
14.69at 4 loops) and GDPval-AA-V2 (967 Eloat 4 loops vs.Qwen3.5-4B’s424 Elo), multi-environment GRPO reinforcement learning clearly shines. - Conversely, on pure parametric knowledge and encyclopedia tests (
GPQA Diamond,HLE, andAA-Omniscience), Dory trailsQwen3.5-4Bacross every single loop setting.
- On IFBench, even 1-loop Dory (
-
The Dial Is Not Monotonic Goodness—Loop 2 Is the Sweet Spot: Look at where the gains actually land. Moving from 1 loop to 2 loops delivers massive jumps: +9.2 points on SWE-bench Verified (
50.3 → 59.5), +7.64 points on SciCode (27.3 → 34.94), +131 Elo on GDPval-AA-V2 (815 → 946), and +3.9 points on the AA v4.3.2 composite (13.8 → 17.7). Moving from 2 loops to 4 loops yields much smaller marginal gains on coding/agentic tasks (59.5 → 61.8on SWE-bench), while GPQA Diamond (72.41 → 72.16) and HLE (10.89 → 10.43) peak at 2 loops and slightly dip at 4 loops. Meanwhile, AA-Omniscience stays completely flat (10.03 → 10.08 → 10.03): looping through shared weights four times cannot invent factual trivia that never fit into 4B parameters during pretraining. Treat the loop count as per-task tuning starting at 2 loops, not a naive “crank to 4 for everything” knob. -
Independent Verification of the Loop Dial (
NetProjectsApple Silicon MLX): While no third party has independently reproduced the full NeMo Gym benchmark table in a shared harness yet, independent verification that the loop dial genuinely improves model predictions arrived on October 17–23 via Paris Michalas at NetProjects (netprojects/nemotron-dory1.0-4b-mlx-bf16anddory-mlx). Validating their Apple Silicon MLX port against an independent FP32 PyTorch implementation, NetProjects confirmed 100% top-1 logit agreement (KL divergence0.001–0.003) and measured both WikiText-2 perplexity and MacBook Pro M5 Max (128 GB) speed across 1, 2, and 4 loops:
| Loop Count | WikiText-2 Perplexity (BF16 / 8-bit) | M5 Max Short-Prompt Speed (8-bit / BF16) | M5 Max After 32K Prompt (8-bit / BF16) | M5 Max 32K Prompt Reading (8-bit) | M5 Max 32K Peak Memory (8-bit / BF16) |
|---|---|---|---|---|---|
| 1 Loop | 12.17 / 12.18 | 101 tok/s / 58 tok/s | 82 tok/s / 55 tok/s | 3,200 tok/s (10 s) | 7.7 GB / 11.3 GB |
| 2 Loops | 11.76 / 11.77 | 69 tok/s / 39 tok/s | 54 tok/s / 36 tok/s | 1,980 tok/s (17 s) | 9.0 GB / 12.6 GB |
| 4 Loops | 11.69 / 11.70 | 39 tok/s / 23 tok/s | 31 tok/s / 22 tok/s | 1,070 tok/s (31 s) | 11.5 GB / 15.1 GB |
Table: Independent Apple Silicon MLX perplexity verification and throughput/memory benchmarks on a MacBook Pro M5 Max (128 GB). Source: NetProjects / Paris Michalas.
4. Running Dory 1.0 4B Today: Pinned Forks, Apple Silicon Setup, and Runnable Client
Ecosystem Reality Check (As of October 23 Evening)
If you try ollama run dory or load the checkpoint in a stock pip install vllm transformers environment today, it will fail. Here is the exact state of runtime support eight days after GA:
- No Stock Upstream Merges Yet: Upstream
llama.cppsupport (ggml-org/llama.cpp#30306) remains open in draft; upstreamvLLMsupport (vllm-project/vllm#60149) remains open and unmerged; upstreammlx-lmsupport (ml-explore/mlx-lm#1988) is still in PR review; stock Hugging Facetransformerslacks merged Dory classes; andollama.com/library/doryreturns a404. - No Hosted API Endpoints: Dory is currently open-weights only; neither NVIDIA NIM nor third-party inference providers offer a hosted endpoint yet.
- Working Local Paths Today:
- NVIDIA’s Pinned
llama.cppContributor Fork (brianchmielNV/llama.cpp): Confirmed on the official Hugging Face GGUF card, NVIDIA pins commit98a74c97ea09ac72f6ceda1559808fc0a2f447cefor native ARM64 macOS Metal builds. Paired with the 3.67 GBdory-q5_k_m.ggufcheckpoint, Dory builds in ~44 seconds on an M4 Mac and runs locally within 16GB unified memory (modelfit.io,dev.to/alanwest,Hack-Log). - Community Apple Silicon MLX Builds: Available via
internet-projects/dory-mlx(netprojects/nemotron-dory1.0-4b-mlx-8bitat 4.2 GB andmlx-bf16at 8.0 GB) andsnsnc’s 4-bit MLX build.
- NVIDIA’s Pinned
1. Building & Serving Dory Locally on Apple Silicon (scripts/build_and_serve_dory_llamacpp.sh)
The headless shell script below (scripts/build_and_serve_dory_llamacpp.sh) compiles the exact pinned brianchmielNV/llama.cpp revision (98a74c97ea09ac72f6ceda1559808fc0a2f447ce) with Apple Metal acceleration, verifies the SHA-256 checksum of dory-q5_k_m.gguf, and launches llama-server on http://127.0.0.1:8080 with last_loop KV caching enabled so your memory stays flat even at 4 loops:
Click to expand runnable code script (scripts/build_and_serve_dory_llamacpp.sh)
#!/usr/bin/env bash
# File: scripts/build_and_serve_dory_llamacpp.sh
# Mode: Headless build & local server startup for macOS ARM64 (Apple Silicon)
# Purpose: Compiles NVIDIA's pinned llama.cpp Dory fork, verifies Q5_K_M GGUF SHA-256,
# and starts llama-server with last_loop KV caching.
set -euo pipefail
LOOPS="${DORY_LOOPS:-2}"
KV_MODE="${DORY_KV_MODE:-last_loop}" # Use 'last_loop' to prevent the 2.7x KV cache growth!
WORK_DIR="/Users/username/models/llama.cpp-dory"
PINNED_COMMIT="98a74c97ea09ac72f6ceda1559808fc0a2f447ce"
GGUF_FILE="dory-q5_k_m.gguf"
EXPECTED_SHA256="96080be49ffd67b916018b7bdef8c484762576fb52780e10ab98ac4978a660e0"
echo "[1/4] Checking Homebrew dependencies (cmake, openssl@3)..."
brew list cmake >/dev/null 2>&1 || brew install cmake
brew list openssl@3 >/dev/null 2>&1 || brew install openssl@3
mkdir -p "${WORK_DIR}"
cd "${WORK_DIR}"
if [ ! -x "./build/bin/llama-server" ]; then
echo "[2/4] Fetching pinned Dory llama.cpp fork (${PINNED_COMMIT})..."
git init .
git remote remove origin 2>/dev/null || true
git remote add origin https://github.com/brianchmielNV/llama.cpp.git
git fetch --depth 1 origin "${PINNED_COMMIT}"
git checkout --detach FETCH_HEAD
echo "[2/4] Building llama-server with Apple Metal & Accelerate BLAS..."
HF_UI_VERSION=b11551 cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_OSX_ARCHITECTURES=arm64 \
-DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON \
-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Apple \
-DLLAMA_BUILD_SERVER=ON \
-DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_APP=OFF \
-DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=ON \
-DLLAMA_OPENSSL=ON -DOPENSSL_ROOT_DIR="$(brew --prefix openssl@3)"
cmake --build build --parallel 4 --target llama-server
fi
echo "[3/4] Verifying ${GGUF_FILE} SHA-256 checksum (3,673,288,864 bytes)..."
if [ ! -f "./${GGUF_FILE}" ]; then
echo "Downloading ${GGUF_FILE} from nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF..."
curl -L "https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF/resolve/main/${GGUF_FILE}" -o "./${GGUF_FILE}"
fi
echo "${EXPECTED_SHA256} ${GGUF_FILE}" | shasum -a 256 -c -
echo "[4/4] Launching llama-server (dory.num_loops=${LOOPS}, dory.recurrent_kv_cache_mode=${KV_MODE})..."
./build/bin/llama-server \
-m "./${GGUF_FILE}" \
--alias dory \
--host 127.0.0.1 --port 8080 \
-ngl 999 \
--ctx-size 4096 --parallel 1 \
--batch-size 512 --ubatch-size 512 \
--cache-type-k f16 --cache-type-v f16 \
--flash-attn off \
--no-context-shift \
--jinja --reasoning on \
--temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 --repeat-penalty 1 \
--override-kv "dory.num_loops=int:${LOOPS},dory.recurrent_kv_cache_mode=str:${KV_MODE}"
2. Querying Dory’s OpenAI-Compatible Endpoint with Tool Calling (scripts/dory_agent_client.py)
Once llama-server is listening on http://127.0.0.1:8080/v1 (or if you are running the unmerged vLLM PR #60149 with --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser nemotron_v3), note two model-card specifics for coding agents:
- Recommended sampling parameters are
temperature=1.0andtop_p=0.95. - For coding agents and tool calls, pass
extra_body={"chat_template_kwargs": {"force_nonempty_content": True}}so the Jinja template does not emit empty content blocks alongside tool invocations.
The standalone script below (scripts/dory_agent_client.py) connects to a local llama-server instance—and includes a deterministic offline fallback mode so you can inspect the loop-scaling telemetry anywhere:
Click to expand runnable code script (scripts/dory_agent_client.py)
"""
File: scripts/dory_agent_client.py
Mode: Headless CLI script (connects to local llama-server on port 8080 or runs offline telemetry check)
Purpose: Demonstrates querying NVIDIA Nemotron Labs Dory 1.0 4B for agentic tool-calling
and comparing 1-loop, 2-loop, and 4-loop behavior.
"""
import os
import json
import urllib.request
import urllib.error
from typing import Dict, Any, List
DORY_BASE_URL = os.getenv("DORY_BASE_URL", "http://127.0.0.1:8080/v1/chat/completions")
def call_dory_agent(
prompt: str,
tools: List[Dict[str, Any]],
max_tokens: int = 1024,
) -> Dict[str, Any]:
"""
Sends an OpenAI-compatible tool-calling request to a local Dory 1.0 4B llama-server
using NVIDIA's recommended sampling and coding-agent chat_template_kwargs.
"""
payload = {
"model": "dory",
"messages": [{"role": "user", "content": prompt}],
"tools": tools,
"max_tokens": max_tokens,
"temperature": 1.0,
"top_p": 0.95,
"extra_body": {
"chat_template_kwargs": {
"force_nonempty_content": True
}
}
}
req = urllib.request.Request(
DORY_BASE_URL,
data=json.dumps(payload).encode("utf-8"),
headers={
"Content-Type": "application/json",
"Authorization": "Bearer EMPTY"
},
method="POST"
)
try:
with urllib.request.urlopen(req, timeout=15.0) as response:
return json.loads(response.read().decode("utf-8"))
except (urllib.error.URLError, TimeoutError):
# Fallback telemetry summary from verified card & NetProjects/Hack-Log measurements
return {
"mode": "offline_reference_profile",
"architectural_stack": "8 Input (I) + 20 Recurrent (R) * Loops + 8 Output (O)",
"loop_profiles": {
1: {"effective_layers": 36, "swebench_verified": 50.3, "ifbench": 74.29, "wikitext2_ppl": 12.17, "terminal_bench_tokens": "1.00x"},
2: {"effective_layers": 56, "swebench_verified": 59.5, "ifbench": 77.94, "wikitext2_ppl": 11.76, "terminal_bench_tokens": "0.72x"},
4: {"effective_layers": 96, "swebench_verified": 61.8, "ifbench": 79.00, "wikitext2_ppl": 11.69, "terminal_bench_tokens": "0.50x"},
}
}
def main() -> None:
tools = [
{
"type": "function",
"function": {
"name": "run_pytest_target",
"description": "Execute a specific pytest module inside the repository workspace",
"parameters": {
"type": "object",
"properties": {
"test_path": {"type": "string"},
"pytest_flags": {"type": "string"}
},
"required": ["test_path"]
}
}
}
]
result = call_dory_agent(
prompt="Run the unit tests in tests/test_kv_cache.py with verbose output.",
tools=tools
)
print("=" * 78)
print("NVIDIA NEMOTRON LABS DORY 1.0 4B — AGENT HARNESS & LOOP DIAL SUMMARY")
print("=" * 78)
if result.get("mode") == "offline_reference_profile":
print(f"Architecture : {result['architectural_stack']}")
print("-" * 78)
print(f"{'Loops':<12} | {'Eff. Layers':<12} | {'SWE-bench':<11} | {'IFBench':<9} | {'WikiText PPL':<13} | {'Output Tokens'}")
print("-" * 78)
for loops, stats in result["loop_profiles"].items():
print(
f"Loops = {loops:<4} | {stats['effective_layers']:<12} | "
f"{stats['swebench_verified']:>8.1f}% | {stats['ifbench']:>7.2f}% | "
f"{stats['wikitext2_ppl']:>12.2f} | {stats['terminal_bench_tokens']}"
)
else:
print(json.dumps(result["choices"][0]["message"], indent=2))
print("=" * 78)
if __name__ == "__main__":
main()
5. The Honest Caveats: Six Realities to Check Before Adopting
Because Dory 1.0 4B dropped silently without a technical paper or independent third-party leaderboard run, separating verified engineering facts from marketing summaries is essential:
- Every Benchmark Score Is Vendor-Run (Zero Independent Shared-Harness Reproductions Yet):
As of October 23 evening (
quidproquo.cc,lapaasvoice.com,dev.to,orcarouter.ai), every single score in the benchmark table comes from NVIDIA’s own NeMo Gym / NeMo Evaluator harness. Furthermore, NVIDIA explicitly notes on the card that itsAA-LCR,AA-Omniscience, andAA v4.3.2charts are internal estimates that may deviate from official Artificial Analysis scores (for instance, NVIDIA’sAA-LCRevaluation uses the olderQwen3-235B-A22Bjudge rather thanGPT-5.6 Luna). WhileNetProjectsindependently verified the model’s logits and WikiText-2 perplexity scaling (12.17 → 11.76 → 11.69), no independent shared-harness reproduction of the agentic benchmark table exists yet. - The Compute Dial Is Not Monotonic:
Most of Dory’s accuracy leap lands on the second loop. Pushing to 4 loops helps multi-step coding and terminal tasks (
SWE-bench Verified59.5 → 61.8,Terminal-Bench 2.118.1 → 22.2), but slightly degradesGPQA Diamond(72.41 → 72.16) andHLE(10.89 → 10.43), while doing nothing forAA-Omniscience(10.03). Looping refines reasoning trajectories; it cannot hallucinate world knowledge that never fit into 4B weights. - Flat KV Cache Requires
last_loop(per_loopGrows KV by 2.7x): As Hack-Log demonstrated on a 16GB M4 Mac, copy-pasting the firstllama-servercommand on the model card leavesdory.recurrent_kv_cache_mode=str:per_loopactive, inflating KV memory 2.7x at 4 loops (1,314 MiB → 3,504 MiB). Always switch tostr:last_loopin memory-constrained environments. (Hack-Log also noted that on a base 16GB M4 running at ~4.36 tok/s, leaving--reasoning oncan cause long<think>chains to run for several minutes—so disabling--reasoningat 1–2 loops is much more practical on base M-series Macs, whereas an M5 Max runs 2 loops at 69 tok/s in 8-bit MLX.) - Stock Tooling Still Cannot Load It:
Until
ggml-org/llama.cpp#30306,vllm-project/vllm#60149, andml-explore/mlx-lm#1988are merged and tagged upstream, running Dory requires compiling the pinned contributor fork or installingdory-mlx. - Context Window Precision (“Up to 256K”):
The model card states “Up to 256K tokens” (and
lapaasvoice.comissued an October 18 correction noting that “up to 256K” is the honest specification rather than a hard 256,000 guarantee across all quantized exports). Note also that NVIDIA’s local macOSllama-serverreference command defaults to--ctx-size 4096. - Watch Out for Secondary Table Misreads:
If you come across secondary articles claiming Dory hits
73.4%on SWE-bench or that itsIFBenchscore collapses at 4 loops, those stem fromquidproquo.cc’s column-alignment error when transcribing NVIDIA’s HTMLrowspan/colspantable. Stick to the primary Hugging Face card.
6. The License Trilogy (0147 → 0151 → 0152/0154) & Forward Pointer to Part 11
Step back and look at the open-weights releases we have analyzed over the past week—together, they form a snapshot of how frontier labs are answering “what does open mean in late 2026?”:
- Meta’s Strings-Attached Custom License (Post 0147 — Meta Muse Spark Open): 816B MoE weights released under the bespoke Muse Model License, complete with commercial active-user thresholds and custom acceptable-use covenants.
- DeepSeek’s Pure OSI Standard (Post 0151 — DeepSeek V4.1-Flash-1017): Unconditional MIT License weights paired with a commercial Search/Fetch API funnel.
- NVIDIA’s House-Backed OpenMDW 1.1 (Post 0152 — Bolt E4B & Post 0154 — Dory 1.0 4B): Both Bolt E4B and Dory ship under the OpenMDW License Agreement 1.1 (raw license text). The agreement explicitly grants rights to “Model Materials”—defined to include both the model architecture and parameters—“without restriction” for commercial and non-commercial deployment. While independent analysts (
quidproquo.cc) read OpenMDW 1.1 as functionally MIT-equivalent with no extra field-of-use restrictions, enterprise counsel should still review the primary license text directly since OpenMDW is a custom model license rather than an OSI-ratified MIT or Apache-2.0 template.
What Comes Next: Latent Loops vs. Reasoning Harnesses (Part 11)
Ultimately, Dory 1.0 4B’s biggest contribution is architectural: paired alongside NVIDIA’s recent production-ready NVIDIA Nemotron 3.5 Bolt E4B 4GB, it proves that a single 4B checkpoint pre-trained natively in NVFP4 can expose a runtime compute dial that trades latent recurrent passes (1, 2, or 4 loops) for accuracy and a 40%–60% reduction in generated output tokens.
In our next deep dive—Part 11 of our Agentic Workflows series: Harnessing Reasoning Models in Agentic Workflows: Integrating o1/R1, Hidden Thinking Tokens, and Cost vs. Depth Optimization—we analyze how production agent harnesses budget, route, and prune reasoning traces when the thinking happens in tokens rather than latent loops.
(Previous) Claude AI + Higgsfield AI: Native Video Generation, MCP Connector Architecture, Multi-Model Orchestration, and the higgsfield-ai/skills Toolkit | (Next) Harnessing Reasoning Models in Agentic Workflows: Integrating o1/R1, Hidden Thinking Tokens, and Cost vs. Depth Optimization
