NVIDIA Nemotron Labs Dory 1.0 4B: The Looped 4B With a Runtime Compute Dial (OpenMDW 1.1, variable recurrent iterations 1–4, and one-loop KV footprint)

Inside NVIDIA's unannounced Dory 1.0 4B: 8+20x+8 looped nGPT architecture, native NVFP4 pretraining, OpenMDW 1.1 license, and the last-loop KV cache caveat.

NVIDIA Nemotron Labs Dory 1.0 4B: The Looped 4B With a Runtime Compute Dial (OpenMDW 1.1, variable recurrent iterations 1–4, and one-loop KV footprint)

(Previous) Claude AI + Higgsfield AI: Native Video Generation, MCP Connector Architecture, Multi-Model Orchestration, and the higgsfield-ai/skills Toolkit | (Next) Harnessing Reasoning Models in Agentic Workflows: Integrating o1/R1, Hidden Thinking Tokens, and Cost vs. Depth Optimization


Summary

NVIDIA’s quietest model release of 2026 is a 4-billion-parameter dense checkpoint that arrived on Hugging Face overnight on October 15–16 stamped GA (10/16/2026)—with zero blog post, zero press release, and not even a passing mention in NVIDIA’s own October 21 open-models roundup eight days later. Named NVIDIA Nemotron Labs Dory 1.0 4B, the model introduces a runtime compute dial that asks the test-time compute question from the opposite side of standard chain-of-thought reasoning models: what if test-time scaling happens vertically in latent space rather than horizontally in generated tokens?

Instead of locking every token into a fixed layer count or emitting thousands of verbose scratchpad tokens into the Key-Value (KV) cache, Dory divides a normalized-GPT (nGPT) backbone with interleaved sliding-window attention (SWA) and full attention into three plain stages: 8 input layers, a shared 20-layer recurrent block applied for 1, 2, or 4 loops at inference time (with 2 loops as the recommended default and the original input representation reinjected at every loop), and 8 output layers. One checkpoint serves three distinct compute budgets (36, 56, or 96 effective layers) with no retraining and no model swap.

Just as notably, Dory was pre-trained across 6 trillion tokens with the first ~4.5 trillion tokens trained natively in NVFP4 before finishing in BF16—landing the exact same week we examined NVFP4 serving kernels in vLLM v0.32.0. Below, we break down the primary-sourced architecture and benchmark numbers from NVIDIA’s model card, independent community verifications on Apple Silicon (NetProjects, Hack-Log, OrcaRouter, modelfit.io), the config-dependent 2.7x KV-cache trap discovered in real-world testing, and the honest caveats every engineer needs to know before compiling the fork.

OrcaRouter Scoreboard Comparing Dory 1.0 4B vs. Gemma 4 12B Figure 1: Side-by-side specification scoreboard from OrcaRouter’s October 17 evaluation summarizing Dory 1.0 4B (4B dense, text only, English only, 256K context, OpenMDW 1.1, runtime loop count 1–4) versus a fixed-depth model (Gemma 4 12B) and highlighting that no shared-harness benchmark result exists yet. Source: OrcaRouter.


Hugging Face / Official Model Card Summary

The following table summarizes the verified primary specifications across the four official Hugging Face checkpoints published by NVIDIA:

SpecificationOfficial Primary Details & Checkpoints
Model Checkpoints• BF16 (W16A16): nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16
• MXFP8 (W8A16): nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8
• NVFP4 (W4A16): nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4
• GGUF (Q5_K_M): nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF (3,673,288,864 bytes, SHA-256: 96080be4...660e0)
Total Parameters & Modality4B Dense • Text-only • English-only (plus 43 programming languages in pretraining corpus)
Backbone ArchitectureLooped (latent-recurrent) normalized-GPT (nGPT) with interleaved sliding-window attention (SWA) and full-attention layers
Layer Topology & Compute Dial8 Input Layers + 20 Shared Recurrent Layers (looped 1, 2, 3, or 4 times at runtime; 2 loops default; input reinjected every loop) + 8 Output Layers (Effective depth: 36 / 56 / 76 / 96 layers)
Context WindowUp to 256K tokens (following a 300B-token context-extension phase)
Pretraining & RL Pipeline6T tokens via Megatron-LM (first ~4.5T natively in NVFP4, remainder in BF16) → 300B tokens context extension → SFT → asynchronous multi-environment GRPO RL via NeMo RL & NeMo Gym
KV Cache Modesper_loop (default in sample launch snippet; scales KV memory with loop count) vs. last_loop (overwrites previous loop’s KV entries in-place for a flat 1-loop KV footprint)
Release Version & HardwareStamped GA (10/16/2026) • Preferred OS: Linux • Microarchitectures: NVIDIA Hopper (H100), Blackwell (B200, B300, DGX Spark) • macOS ARM64 via pinned llama.cpp contributor fork or community MLX builds
LicenseOpenMDW License Agreement 1.1 (raw license text) — grants rights to “Model Materials” (architecture + parameters) “without restriction”

Prior Reading Material

To connect Dory’s native NVFP4 pretraining, recurrent KV cache mechanics, and open-weights licensing to the rest of our architecture coverage, review these prerequisite deep dives:


1. The Midnight GA: Why Ship a Looped Architecture in Silence?

Before looking at benchmarks or code, picture how a standard transformer handles everyday traffic.

Imagine driving a car built with a single, welded fixed gear. Whether you are creeping through a parking lot at 5 mph or climbing a steep mountain pass, the transmission cannot shift. In a standard dense transformer, layer depth works the exact same way: every generated token—whether predicting a trivial JSON comma or resolving a multi-file Python import cycle—must pass through the exact same fixed stack of layers.

flowchart TD
    S1["Standard Fixed-Depth Transformer (Static Layer Stack)"] --> S2["Every Token Traverses Identical Physical Layers Once"]
    S2 --> S3["To Think Harder: Must Emit Hundreds of Verbal Scratchpad Tokens"]
    S3 --> S4["Bottleneck: Output Token Latency + Linear KV Cache Expansion"]

    style S1 fill:#0f172a,stroke:#ff6b6b,stroke-width:2px,color:#ffffff
    style S2 fill:#1e293b,stroke:#ff9f43,stroke-width:2px,color:#ffffff
    style S3 fill:#2d1b2e,stroke:#ff6b6b,stroke-width:2px,color:#ffffff
    style S4 fill:#3b111a,stroke:#ff4757,stroke-width:2px,color:#ffffff

Now imagine replacing that welded single gear with a multi-speed gearbox inside the model’s internal workspace. Instead of emitting five hundred extra words out loud just to “think,” the model routes the token’s internal hidden representation through a shared middle core 1, 2, or 4 times before emitting a single word:

flowchart TD
    D1["NVIDIA Dory 1.0 4B (8 Input + 20 Shared Recurrent + 8 Output Layers)"] --> D2["Runtime Compute Dial: Loop Shared 20-Layer Block 1, 2, or 4 Times"]
    D2 --> D3["Deliberation Happens Vertically in Latent Space Per Token"]
    D3 --> D4["Result: 40%–60% Fewer Output Tokens Needed + Flat 1-Loop KV Footprint (in last_loop mode)"]

    style D1 fill:#0d2b45,stroke:#00e5ff,stroke-width:2px,color:#ffffff
    style D2 fill:#0f382c,stroke:#76b900,stroke-width:2px,color:#ffffff
    style D3 fill:#1a3d3c,stroke:#00e5ff,stroke-width:2px,color:#ffffff
    style D4 fill:#0f382c,stroke:#00ff88,stroke-width:2px,color:#ffffff

Normally, when a major AI lab ships a brand-new architectural family—especially its first looped normalized-GPT model and a flagship checkpoint pre-trained natively in NVFP4—it accompanies the release with a launch blog, technical report, and developer fanfare.

With Dory 1.0 4B, NVIDIA did the exact opposite:

  • Overnight Repository Drop (Oct 15–16, 2026): Four checkpoints (BF16, MXFP8, NVFP4, and GGUF) quietly appeared under the nvidia Hugging Face organization with the model version stamped GA (10/16/2026).
  • Eight Days of Official Silence: As of October 23 evening, NVIDIA has published no blog post and no press announcement for Dory.
  • Absent from the Product Roster & Oct 21 Roundup: As noted by independent researchers at quidproquo.cc, Dory does not appear on NVIDIA’s official Nemotron developer page (which lists Bolt, Nano, Lightning, Super, and Ultra). Even more tellingly, when NVIDIA published its broad October 21 “expands open models” announcement covering Nemotron 3.5 Super VL, Bolt, Kumo, Cosmos 3.5, Alpamayo 2 Edge, BioNeMo, and Ising, Dory was omitted entirely.

NVIDIA Nemotron Labs Dory 1.0 4B Hugging Face Model Card Capture Figure 2: The nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 Hugging Face repository captured shortly after its silent October 16, 2026 drop—showing zero likes, zero deployed inference providers, and an official GA release date stamp. Source: Hugging Face / NVIDIA via OrcaRouter.

Why did NVIDIA ship a GA-stamped looped transformer like a midnight commit? There is no public evidence pointing to whether Dory is an early preview from the Nemotron Labs research group or a staging artifact released ahead of upstream runtime merges. What we do have are the weights, the primary model card, and a week of independent community testing.


2. The Compute Dial, Explained: How the 8 + 20×Loops + 8 Stack Works in Plain English

The Three-Stage Looped Stack (And Why It Reinjects the Input)

Unlike older weight-tied transformers that naively loop every layer from bottom to top, Dory splits its 4B normalized-GPT (nGPT) backbone into three simple, human-readable stages:

Dory 1.0 4B Looped LM Architecture Figure 3: Official Dory 1.0 4B (Looped LM) architecture diagram. The layer stack is divided into an 8-layer Input block (I), a shared 20-layer Recurrent block (R) that runs multiple times while re-adding the original input (h^in) on every pass, and an 8-layer Output block (O). Source: NVIDIA / Hugging Face.

Here is what happens inside Figure 3 in plain English—no equations required:

  1. Stage 1 — The Input Block (I, 8 Layers): When a token enters the model, the first 8 layers digest the raw embeddings and create a clean “working summary” of the input.
  2. Stage 2 — The Shared Recurrent Block (R, 20 Layers, Looped 1, 2, or 4 Times): Next, the token enters a shared 20-layer middle block. Depending on how you set the runtime compute dial, the token loops through these exact same 20 layers 1, 2, or 4 times. Crucially, at the start of every single loop, Dory adds the original 8-layer input summary (h^in) back into the mix. Think of this like a writer re-reading the original prompt before every draft revision: without reinjecting that original input on each pass, looping through the same 20 layers repeatedly would cause the model’s internal thoughts to drift further and further away from what you actually asked.
  3. Stage 3 — The Output Block (O, 8 Layers): Once the final loop finishes, the refined representation passes through 8 dedicated output layers to predict the next token.

Across all three blocks, Dory alternates between sliding-window attention (SWA) layers (which look at nearby tokens cheaply) and full-attention layers (which look across the entire prompt up to 256K tokens). Because the middle 20 layers are reused across loops, a single 4B download lets you dial the effective depth on the fly:

  • 1 Loop: 8 input + 20 recurrent + 8 output = 36 effective layers (fastest speed, lowest compute per token).
  • 2 Loops (Recommended Default): 8 input + (20 × 2) recurrent + 8 output = 56 effective layers (NVIDIA’s sweet spot for everyday reasoning).
  • 4 Loops: 8 input + (20 × 4) recurrent + 8 output = 96 effective layers (maximum deliberation for coding and terminal agents).

Native NVFP4 Pretraining: The Training-Side Twin to vLLM v0.32.0

Look beneath the layer diagram on the model card and a second milestone jumps out: Dory was pre-trained on 6 trillion tokens in Megatron-LM, with the first ~4.5 trillion tokens trained natively in NVFP4 (NVIDIA’s 4-bit floating-point format) before finishing the final ~1.5 trillion tokens in BF16, followed by a 300B-token context-extension phase to 256K.

Earlier this week in our vLLM v0.32.0 deep dive (Post 0149), we covered how NVFP4 has become the flagship serving format on Hopper and Blackwell GPUs. Dory proves the training side of that story: because its normalized-GPT (nGPT) backbone keeps internal activations strictly normalized, it can train across 4.5 trillion tokens directly in 4-bit precision without collapsing.

On NVIDIA’s DGX Spark hardware, running the quantized checkpoints delivers substantial single-batch speedups over the BF16 baseline:

Input Sequence Length (ISL)Output Sequence Length (OSL)Batch SizeMXFP8 (W8A16) Speedup vs. BF16NVFP4 (W4A16) Speedup vs. BF16
2,560 tokens300 tokens11.69x2.17x
128 tokens1,024 tokens11.84x2.44x

Token Efficiency: Thinking in Latent Space Instead of Output Tokens

When a conventional reasoning model gets stuck on a coding bug, it thinks by generating hundreds of extra words out loud. When Dory loops in latent space, each token already carries deeper internal work—meaning the model solves the same tasks in far fewer generated output tokens:

Output Token Reduction from Looping Figure 4: Output tokens generated per attempt at 2 loops (k=2) and 4 loops (k=4) relative to 1 loop (k=1) across four agentic benchmarks. Source: NVIDIA / Hugging Face.

As Figure 4 shows, turning the dial from 1 loop to 4 loops slashes the number of generated output tokens by:

  • 50% on Terminal-Bench 2.1 (1.00x at 1 loop → 0.72x at 2 loops → 0.50x at 4 loops).
  • 60% on GDPval-AA-V2 (1.00x at 1 loop → 0.66x at 2 loops → 0.40x at 4 loops).
  • 26% on SWE-bench Verified (1.00x at 1 loop → 0.79x at 2 loops → 0.74x at 4 loops).
  • 23% on τ³-Banking (1.00x at 1 loop → 0.80x at 2 loops → 0.77x at 4 loops).

Last-Loop KV Caching — And Hack-Log’s 2.7x Real-World Caveat

In a standard looped transformer, even though the 20 recurrent layers reuse the same weights on every pass, the inference server still saves a separate Key-Value (KV) memory cache for every loop (per_loop). At 4 loops, your memory footprint balloons as if you were running an 80-layer middle block.

To prevent this, Dory supports last-loop KV caching (last_loop): each new loop simply overwrites the previous loop’s KV cache entries in place, so memory only ever holds a single loop’s worth of KV cache.

Test-Time Compute Scaling: Token-Space CoT vs. Dory 1.0 4B Latent-Space Looping Figure 5: Visual comparison between horizontal token-space chain-of-thought scaling (left) and Dory 1.0 4B’s vertical latent-space loop scaling with in-place last-loop KV cache overwriting (right).

However, independent developer Hack-Log uncovered a critical real-world catch while testing Dory on a 16GB Apple M4 Mac: last-loop KV caching is config-dependent, not automatic.

Hack-Log 16GB M4 Mac Looped KV Cache Illustration Figure 6: Illustration from Hack-Log’s 16GB M4 Mac evaluation contrasting the default per_loop KV cache expansion (orange) against the flat last_loop KV cache footprint (cyan). Source: Hack-Log / note.com.

If you copy-paste the main llama-server launch command from NVIDIA’s Hugging Face model card, its --override-kv setting actually defaults to per_loop:

# Default snippet on the card — grows KV memory with every added loop!
--override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop"

When Hack-Log benchmarked a 32,768-token context (f16 KV cache) on a 16GB M4 Mac using that default per_loop setting versus changing that single word to last_loop, the measured memory difference was dramatic:

Loop CountDefault per_loop KV Buffer (phys_footprint)Corrected last_loop KV Buffer (phys_footprint)16GB M4 Generation Speed (last_loop)
1 Loop1,314 MiB (1,445 MB)1,314 MiB (1,445 MB)6.98 tokens/s (prompt: 66.3 tok/s)
2 Loops2,044 MiB (2,206 MB)1,314 MiB (1,406 MB)4.36 tokens/s
4 Loops3,504 MiB (3,680 MB — 2.7x bloat!)1,314 MiB (1,465 MB — Flat!)2.66 tokens/s (prompt: 24.3 tok/s)

Table: Empirical KV buffer and macOS footprint measurements on a 16GB Apple Silicon M4 Mac (Q5_K_M GGUF, 32K context). Source: Hack-Log.

To actually get the flat one-loop memory footprint advertised in the highlights, you must explicitly pass last_loop:

# Fixed one-loop KV footprint across all loop counts:
--override-kv "dory.num_loops=int:4,dory.recurrent_kv_cache_mode=str:last_loop"

3. What the Dial Buys: Primary Benchmark Analysis & Model Profile

All benchmark numbers currently available for Dory 1.0 4B come from NVIDIA’s internal evaluation harness (NeMo Gym / NeMo Evaluator SDK, using the Terminus-2 harness for Terminal-Bench 2.1). Let us examine the composite trajectory first, followed by the full task-by-task breakdown:

Dory 1.0 4B AA v4.3.2 Estimate vs. Loop Count Figure 7: NVIDIA’s internal Artificial Analysis (AA v4.3.2) composite estimate across 1, 2, and 4 loops (k=1, 2, 4) compared against reference open-weights models. Note: NVIDIA notes these are internal harness estimates that may deviate from official Artificial Analysis scores. Source: NVIDIA / Hugging Face.

Complete Primary Benchmark Table (NVIDIA Hugging Face Card)

Task / BenchmarkDory 1 LoopDory 2 LoopsDory 4 LoopsQwen3.5-4BQwen3.5-9BGemma 4 26B A4BQwen3.6-35B-A3B
Coding & Agentic
SciCode27.334.9435.1125.8228.2140.3237.24
SWE-bench Verified50.359.561.854.761.057.4073.4‡
Terminal-Bench 2.112.518.122.215243543
τ³-bench (Banking)11.612.8914.695.037.2213.669.54
GDPval-AA-V2 (Elo)815946967424546751.6987
Reasoning & Knowledge
GPQA Diamond (no tools)69.1972.4172.1677.7180.9380.4385.35
HLE (text, no tools)8.2510.8910.4311.3115.4317.7919.18
AA-Omniscience (accuracy)10.0310.0810.0315.0218.0521.1319.95
Instruction Following & Long Context
IFBench74.2977.9479.053.8359.077.8957.44
AA-LCR (Long Context)63717364.768.464.668.5

Footnote (‡): The 73.4‡ SWE-bench Verified score sits on Qwen3.6-35B-A3B as an external reported score NVIDIA did not re-run. (Cautionary note: secondary site quidproquo.cc accidentally misaligned the HTML table columns from the model card, shifting Qwen3.6-35B-A3B’s 73.4 / 43 / 85.35 scores into Dory’s 4-loop column and misreading IFBench—always reference the primary table above.)

Three Engineering Takeaways from the Numbers

  1. “A Local Coding Agent and Tool-Calling Worker, Not a Pocket Encyclopedia”: As developer alanwest observed on dev.to, Dory’s benchmark fingerprint splits cleanly down the middle. On agentic, instruction-following, and long-context workflows, Dory dominates its size class:

    • On IFBench, even 1-loop Dory (74.29) beats Qwen3.5-4B (53.83) by +20.46 points, and 4-loop Dory (79.0) beats every model in the table including Gemma 4 26B A4B (77.89) and Qwen3.6-35B-A3B (57.44).
    • On AA-LCR (long-context reasoning), Dory scales from 63 (1 loop) to 73 (4 loops), outscoring Qwen3.5-9B (68.4) and Qwen3.6-35B-A3B (68.5).
    • On SWE-bench Verified, dialing from 1 loop (50.3) to 2 loops (59.5) and 4 loops (61.8) pushes a 4B model past Qwen3.5-4B (54.7), Gemma 4 26B A4B (57.40), and Qwen3.5-9B (61.0).
    • On τ³-bench Banking (14.69 at 4 loops) and GDPval-AA-V2 (967 Elo at 4 loops vs. Qwen3.5-4B’s 424 Elo), multi-environment GRPO reinforcement learning clearly shines.
    • Conversely, on pure parametric knowledge and encyclopedia tests (GPQA Diamond, HLE, and AA-Omniscience), Dory trails Qwen3.5-4B across every single loop setting.
  2. The Dial Is Not Monotonic Goodness—Loop 2 Is the Sweet Spot: Look at where the gains actually land. Moving from 1 loop to 2 loops delivers massive jumps: +9.2 points on SWE-bench Verified (50.3 → 59.5), +7.64 points on SciCode (27.3 → 34.94), +131 Elo on GDPval-AA-V2 (815 → 946), and +3.9 points on the AA v4.3.2 composite (13.8 → 17.7). Moving from 2 loops to 4 loops yields much smaller marginal gains on coding/agentic tasks (59.5 → 61.8 on SWE-bench), while GPQA Diamond (72.41 → 72.16) and HLE (10.89 → 10.43) peak at 2 loops and slightly dip at 4 loops. Meanwhile, AA-Omniscience stays completely flat (10.03 → 10.08 → 10.03): looping through shared weights four times cannot invent factual trivia that never fit into 4B parameters during pretraining. Treat the loop count as per-task tuning starting at 2 loops, not a naive “crank to 4 for everything” knob.

  3. Independent Verification of the Loop Dial (NetProjects Apple Silicon MLX): While no third party has independently reproduced the full NeMo Gym benchmark table in a shared harness yet, independent verification that the loop dial genuinely improves model predictions arrived on October 17–23 via Paris Michalas at NetProjects (netprojects/nemotron-dory1.0-4b-mlx-bf16 and dory-mlx). Validating their Apple Silicon MLX port against an independent FP32 PyTorch implementation, NetProjects confirmed 100% top-1 logit agreement (KL divergence 0.001–0.003) and measured both WikiText-2 perplexity and MacBook Pro M5 Max (128 GB) speed across 1, 2, and 4 loops:

Loop CountWikiText-2 Perplexity (BF16 / 8-bit)M5 Max Short-Prompt Speed (8-bit / BF16)M5 Max After 32K Prompt (8-bit / BF16)M5 Max 32K Prompt Reading (8-bit)M5 Max 32K Peak Memory (8-bit / BF16)
1 Loop12.17 / 12.18101 tok/s / 58 tok/s82 tok/s / 55 tok/s3,200 tok/s (10 s)7.7 GB / 11.3 GB
2 Loops11.76 / 11.7769 tok/s / 39 tok/s54 tok/s / 36 tok/s1,980 tok/s (17 s)9.0 GB / 12.6 GB
4 Loops11.69 / 11.7039 tok/s / 23 tok/s31 tok/s / 22 tok/s1,070 tok/s (31 s)11.5 GB / 15.1 GB

Table: Independent Apple Silicon MLX perplexity verification and throughput/memory benchmarks on a MacBook Pro M5 Max (128 GB). Source: NetProjects / Paris Michalas.


4. Running Dory 1.0 4B Today: Pinned Forks, Apple Silicon Setup, and Runnable Client

Ecosystem Reality Check (As of October 23 Evening)

If you try ollama run dory or load the checkpoint in a stock pip install vllm transformers environment today, it will fail. Here is the exact state of runtime support eight days after GA:

  • No Stock Upstream Merges Yet: Upstream llama.cpp support (ggml-org/llama.cpp#30306) remains open in draft; upstream vLLM support (vllm-project/vllm#60149) remains open and unmerged; upstream mlx-lm support (ml-explore/mlx-lm#1988) is still in PR review; stock Hugging Face transformers lacks merged Dory classes; and ollama.com/library/dory returns a 404.
  • No Hosted API Endpoints: Dory is currently open-weights only; neither NVIDIA NIM nor third-party inference providers offer a hosted endpoint yet.
  • Working Local Paths Today:
    1. NVIDIA’s Pinned llama.cpp Contributor Fork (brianchmielNV/llama.cpp): Confirmed on the official Hugging Face GGUF card, NVIDIA pins commit 98a74c97ea09ac72f6ceda1559808fc0a2f447ce for native ARM64 macOS Metal builds. Paired with the 3.67 GB dory-q5_k_m.gguf checkpoint, Dory builds in ~44 seconds on an M4 Mac and runs locally within 16GB unified memory (modelfit.io, dev.to/alanwest, Hack-Log).
    2. Community Apple Silicon MLX Builds: Available via internet-projects/dory-mlx (netprojects/nemotron-dory1.0-4b-mlx-8bit at 4.2 GB and mlx-bf16 at 8.0 GB) and snsnc’s 4-bit MLX build.

1. Building & Serving Dory Locally on Apple Silicon (scripts/build_and_serve_dory_llamacpp.sh)

The headless shell script below (scripts/build_and_serve_dory_llamacpp.sh) compiles the exact pinned brianchmielNV/llama.cpp revision (98a74c97ea09ac72f6ceda1559808fc0a2f447ce) with Apple Metal acceleration, verifies the SHA-256 checksum of dory-q5_k_m.gguf, and launches llama-server on http://127.0.0.1:8080 with last_loop KV caching enabled so your memory stays flat even at 4 loops:

Click to expand runnable code script (scripts/build_and_serve_dory_llamacpp.sh)
#!/usr/bin/env bash
# File: scripts/build_and_serve_dory_llamacpp.sh
# Mode: Headless build & local server startup for macOS ARM64 (Apple Silicon)
# Purpose: Compiles NVIDIA's pinned llama.cpp Dory fork, verifies Q5_K_M GGUF SHA-256,
#          and starts llama-server with last_loop KV caching.

set -euo pipefail

LOOPS="${DORY_LOOPS:-2}"
KV_MODE="${DORY_KV_MODE:-last_loop}" # Use 'last_loop' to prevent the 2.7x KV cache growth!
WORK_DIR="/Users/username/models/llama.cpp-dory"
PINNED_COMMIT="98a74c97ea09ac72f6ceda1559808fc0a2f447ce"
GGUF_FILE="dory-q5_k_m.gguf"
EXPECTED_SHA256="96080be49ffd67b916018b7bdef8c484762576fb52780e10ab98ac4978a660e0"

echo "[1/4] Checking Homebrew dependencies (cmake, openssl@3)..."
brew list cmake >/dev/null 2>&1 || brew install cmake
brew list openssl@3 >/dev/null 2>&1 || brew install openssl@3

mkdir -p "${WORK_DIR}"
cd "${WORK_DIR}"

if [ ! -x "./build/bin/llama-server" ]; then
    echo "[2/4] Fetching pinned Dory llama.cpp fork (${PINNED_COMMIT})..."
    git init .
    git remote remove origin 2>/dev/null || true
    git remote add origin https://github.com/brianchmielNV/llama.cpp.git
    git fetch --depth 1 origin "${PINNED_COMMIT}"
    git checkout --detach FETCH_HEAD

    echo "[2/4] Building llama-server with Apple Metal & Accelerate BLAS..."
    HF_UI_VERSION=b11551 cmake -S . -B build \
        -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_OSX_ARCHITECTURES=arm64 \
        -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON \
        -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Apple \
        -DLLAMA_BUILD_SERVER=ON \
        -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_APP=OFF \
        -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=ON \
        -DLLAMA_OPENSSL=ON -DOPENSSL_ROOT_DIR="$(brew --prefix openssl@3)"
    cmake --build build --parallel 4 --target llama-server
fi

echo "[3/4] Verifying ${GGUF_FILE} SHA-256 checksum (3,673,288,864 bytes)..."
if [ ! -f "./${GGUF_FILE}" ]; then
    echo "Downloading ${GGUF_FILE} from nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF..."
    curl -L "https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF/resolve/main/${GGUF_FILE}" -o "./${GGUF_FILE}"
fi
echo "${EXPECTED_SHA256}  ${GGUF_FILE}" | shasum -a 256 -c -

echo "[4/4] Launching llama-server (dory.num_loops=${LOOPS}, dory.recurrent_kv_cache_mode=${KV_MODE})..."
./build/bin/llama-server \
    -m "./${GGUF_FILE}" \
    --alias dory \
    --host 127.0.0.1 --port 8080 \
    -ngl 999 \
    --ctx-size 4096 --parallel 1 \
    --batch-size 512 --ubatch-size 512 \
    --cache-type-k f16 --cache-type-v f16 \
    --flash-attn off \
    --no-context-shift \
    --jinja --reasoning on \
    --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 --repeat-penalty 1 \
    --override-kv "dory.num_loops=int:${LOOPS},dory.recurrent_kv_cache_mode=str:${KV_MODE}"

2. Querying Dory’s OpenAI-Compatible Endpoint with Tool Calling (scripts/dory_agent_client.py)

Once llama-server is listening on http://127.0.0.1:8080/v1 (or if you are running the unmerged vLLM PR #60149 with --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser nemotron_v3), note two model-card specifics for coding agents:

  1. Recommended sampling parameters are temperature=1.0 and top_p=0.95.
  2. For coding agents and tool calls, pass extra_body={"chat_template_kwargs": {"force_nonempty_content": True}} so the Jinja template does not emit empty content blocks alongside tool invocations.

The standalone script below (scripts/dory_agent_client.py) connects to a local llama-server instance—and includes a deterministic offline fallback mode so you can inspect the loop-scaling telemetry anywhere:

Click to expand runnable code script (scripts/dory_agent_client.py)
"""
File: scripts/dory_agent_client.py
Mode: Headless CLI script (connects to local llama-server on port 8080 or runs offline telemetry check)
Purpose: Demonstrates querying NVIDIA Nemotron Labs Dory 1.0 4B for agentic tool-calling
         and comparing 1-loop, 2-loop, and 4-loop behavior.
"""

import os
import json
import urllib.request
import urllib.error
from typing import Dict, Any, List

DORY_BASE_URL = os.getenv("DORY_BASE_URL", "http://127.0.0.1:8080/v1/chat/completions")


def call_dory_agent(
    prompt: str,
    tools: List[Dict[str, Any]],
    max_tokens: int = 1024,
) -> Dict[str, Any]:
    """
    Sends an OpenAI-compatible tool-calling request to a local Dory 1.0 4B llama-server
    using NVIDIA's recommended sampling and coding-agent chat_template_kwargs.
    """
    payload = {
        "model": "dory",
        "messages": [{"role": "user", "content": prompt}],
        "tools": tools,
        "max_tokens": max_tokens,
        "temperature": 1.0,
        "top_p": 0.95,
        "extra_body": {
            "chat_template_kwargs": {
                "force_nonempty_content": True
            }
        }
    }

    req = urllib.request.Request(
        DORY_BASE_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer EMPTY"
        },
        method="POST"
    )

    try:
        with urllib.request.urlopen(req, timeout=15.0) as response:
            return json.loads(response.read().decode("utf-8"))
    except (urllib.error.URLError, TimeoutError):
        # Fallback telemetry summary from verified card & NetProjects/Hack-Log measurements
        return {
            "mode": "offline_reference_profile",
            "architectural_stack": "8 Input (I) + 20 Recurrent (R) * Loops + 8 Output (O)",
            "loop_profiles": {
                1: {"effective_layers": 36, "swebench_verified": 50.3, "ifbench": 74.29, "wikitext2_ppl": 12.17, "terminal_bench_tokens": "1.00x"},
                2: {"effective_layers": 56, "swebench_verified": 59.5, "ifbench": 77.94, "wikitext2_ppl": 11.76, "terminal_bench_tokens": "0.72x"},
                4: {"effective_layers": 96, "swebench_verified": 61.8, "ifbench": 79.00, "wikitext2_ppl": 11.69, "terminal_bench_tokens": "0.50x"},
            }
        }


def main() -> None:
    tools = [
        {
            "type": "function",
            "function": {
                "name": "run_pytest_target",
                "description": "Execute a specific pytest module inside the repository workspace",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "test_path": {"type": "string"},
                        "pytest_flags": {"type": "string"}
                    },
                    "required": ["test_path"]
                }
            }
        }
    ]

    result = call_dory_agent(
        prompt="Run the unit tests in tests/test_kv_cache.py with verbose output.",
        tools=tools
    )

    print("=" * 78)
    print("NVIDIA NEMOTRON LABS DORY 1.0 4B — AGENT HARNESS & LOOP DIAL SUMMARY")
    print("=" * 78)
    if result.get("mode") == "offline_reference_profile":
        print(f"Architecture : {result['architectural_stack']}")
        print("-" * 78)
        print(f"{'Loops':<12} | {'Eff. Layers':<12} | {'SWE-bench':<11} | {'IFBench':<9} | {'WikiText PPL':<13} | {'Output Tokens'}")
        print("-" * 78)
        for loops, stats in result["loop_profiles"].items():
            print(
                f"Loops = {loops:<4} | {stats['effective_layers']:<12} | "
                f"{stats['swebench_verified']:>8.1f}%  | {stats['ifbench']:>7.2f}% | "
                f"{stats['wikitext2_ppl']:>12.2f} | {stats['terminal_bench_tokens']}"
            )
    else:
        print(json.dumps(result["choices"][0]["message"], indent=2))
    print("=" * 78)


if __name__ == "__main__":
    main()

5. The Honest Caveats: Six Realities to Check Before Adopting

Because Dory 1.0 4B dropped silently without a technical paper or independent third-party leaderboard run, separating verified engineering facts from marketing summaries is essential:

  1. Every Benchmark Score Is Vendor-Run (Zero Independent Shared-Harness Reproductions Yet): As of October 23 evening (quidproquo.cc, lapaasvoice.com, dev.to, orcarouter.ai), every single score in the benchmark table comes from NVIDIA’s own NeMo Gym / NeMo Evaluator harness. Furthermore, NVIDIA explicitly notes on the card that its AA-LCR, AA-Omniscience, and AA v4.3.2 charts are internal estimates that may deviate from official Artificial Analysis scores (for instance, NVIDIA’s AA-LCR evaluation uses the older Qwen3-235B-A22B judge rather than GPT-5.6 Luna). While NetProjects independently verified the model’s logits and WikiText-2 perplexity scaling (12.17 → 11.76 → 11.69), no independent shared-harness reproduction of the agentic benchmark table exists yet.
  2. The Compute Dial Is Not Monotonic: Most of Dory’s accuracy leap lands on the second loop. Pushing to 4 loops helps multi-step coding and terminal tasks (SWE-bench Verified 59.5 → 61.8, Terminal-Bench 2.1 18.1 → 22.2), but slightly degrades GPQA Diamond (72.41 → 72.16) and HLE (10.89 → 10.43), while doing nothing for AA-Omniscience (10.03). Looping refines reasoning trajectories; it cannot hallucinate world knowledge that never fit into 4B weights.
  3. Flat KV Cache Requires last_loop (per_loop Grows KV by 2.7x): As Hack-Log demonstrated on a 16GB M4 Mac, copy-pasting the first llama-server command on the model card leaves dory.recurrent_kv_cache_mode=str:per_loop active, inflating KV memory 2.7x at 4 loops (1,314 MiB → 3,504 MiB). Always switch to str:last_loop in memory-constrained environments. (Hack-Log also noted that on a base 16GB M4 running at ~4.36 tok/s, leaving --reasoning on can cause long <think> chains to run for several minutes—so disabling --reasoning at 1–2 loops is much more practical on base M-series Macs, whereas an M5 Max runs 2 loops at 69 tok/s in 8-bit MLX.)
  4. Stock Tooling Still Cannot Load It: Until ggml-org/llama.cpp#30306, vllm-project/vllm#60149, and ml-explore/mlx-lm#1988 are merged and tagged upstream, running Dory requires compiling the pinned contributor fork or installing dory-mlx.
  5. Context Window Precision (“Up to 256K”): The model card states “Up to 256K tokens” (and lapaasvoice.com issued an October 18 correction noting that “up to 256K” is the honest specification rather than a hard 256,000 guarantee across all quantized exports). Note also that NVIDIA’s local macOS llama-server reference command defaults to --ctx-size 4096.
  6. Watch Out for Secondary Table Misreads: If you come across secondary articles claiming Dory hits 73.4% on SWE-bench or that its IFBench score collapses at 4 loops, those stem from quidproquo.cc’s column-alignment error when transcribing NVIDIA’s HTML rowspan/colspan table. Stick to the primary Hugging Face card.

6. The License Trilogy (0147 → 0151 → 0152/0154) & Forward Pointer to Part 11

Step back and look at the open-weights releases we have analyzed over the past week—together, they form a snapshot of how frontier labs are answering “what does open mean in late 2026?”:

  • Meta’s Strings-Attached Custom License (Post 0147 — Meta Muse Spark Open): 816B MoE weights released under the bespoke Muse Model License, complete with commercial active-user thresholds and custom acceptable-use covenants.
  • DeepSeek’s Pure OSI Standard (Post 0151 — DeepSeek V4.1-Flash-1017): Unconditional MIT License weights paired with a commercial Search/Fetch API funnel.
  • NVIDIA’s House-Backed OpenMDW 1.1 (Post 0152 — Bolt E4B & Post 0154 — Dory 1.0 4B): Both Bolt E4B and Dory ship under the OpenMDW License Agreement 1.1 (raw license text). The agreement explicitly grants rights to “Model Materials”—defined to include both the model architecture and parameters—“without restriction” for commercial and non-commercial deployment. While independent analysts (quidproquo.cc) read OpenMDW 1.1 as functionally MIT-equivalent with no extra field-of-use restrictions, enterprise counsel should still review the primary license text directly since OpenMDW is a custom model license rather than an OSI-ratified MIT or Apache-2.0 template.

What Comes Next: Latent Loops vs. Reasoning Harnesses (Part 11)

Ultimately, Dory 1.0 4B’s biggest contribution is architectural: paired alongside NVIDIA’s recent production-ready NVIDIA Nemotron 3.5 Bolt E4B 4GB, it proves that a single 4B checkpoint pre-trained natively in NVFP4 can expose a runtime compute dial that trades latent recurrent passes (1, 2, or 4 loops) for accuracy and a 40%–60% reduction in generated output tokens.

In our next deep dive—Part 11 of our Agentic Workflows series: Harnessing Reasoning Models in Agentic Workflows: Integrating o1/R1, Hidden Thinking Tokens, and Cost vs. Depth Optimization—we analyze how production agent harnesses budget, route, and prune reasoning traces when the thinking happens in tokens rather than latent loops.


(Previous) Claude AI + Higgsfield AI: Native Video Generation, MCP Connector Architecture, Multi-Model Orchestration, and the higgsfield-ai/skills Toolkit | (Next) Harnessing Reasoning Models in Agentic Workflows: Integrating o1/R1, Hidden Thinking Tokens, and Cost vs. Depth Optimization